diff options
Diffstat (limited to 'doc/development/telemetry')
| -rw-r--r-- | doc/development/telemetry/index.md | 100 | ||||
| -rw-r--r-- | doc/development/telemetry/snowplow.md | 16 | ||||
| -rw-r--r-- | doc/development/telemetry/usage_ping.md | 898 |
3 files changed, 704 insertions, 310 deletions
diff --git a/doc/development/telemetry/index.md b/doc/development/telemetry/index.md index 32f63d5221e..aee16e4049a 100644 --- a/doc/development/telemetry/index.md +++ b/doc/development/telemetry/index.md @@ -1,3 +1,9 @@ +--- +stage: Growth +group: Telemetry +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Telemetry Guide At GitLab, we collect telemetry for the purpose of helping us build a better GitLab. Data about how GitLab is used is collected to better understand what parts of GitLab needs improvement and what features to build next. Telemetry also helps our team better understand the reasons why people use GitLab and with this knowledge we are able to make better product decisions. @@ -19,11 +25,11 @@ Telemetry Guide: 1. [What is Usage Ping](usage_ping.md#what-is-usage-ping) 1. [Usage Ping payload](usage_ping.md#usage-ping-payload) - 1. [Disabling Usage Ping](usage_ping.md#disabling-usage-ping) + 1. [Disable Usage Ping](usage_ping.md#disable-usage-ping) 1. [Usage Ping request flow](usage_ping.md#usage-ping-request-flow) 1. [How Usage Ping works](usage_ping.md#how-usage-ping-works) 1. [Implementing Usage Ping](usage_ping.md#implementing-usage-ping) - 1. [Developing and testing usage ping](usage_ping.md#developing-and-testing-usage-ping) + 1. [Developing and testing Usage Ping](usage_ping.md#developing-and-testing-usage-ping) [Snowplow Guide](snowplow.md) @@ -44,27 +50,27 @@ More useful links: ## Our tracking tools -In this section we will explain the six different technologies we use to gather product usage data. +We use several different technologies to gather product usage data. -**Snowplow JS (Frontend)** +### Snowplow JS (Frontend) Snowplow is an enterprise-grade marketing and product analytics platform which helps track the way users engage with our website and application. [Snowplow JS](https://github.com/snowplow/snowplow/wiki/javascript-tracker) is a frontend tracker for client-side events. -**Snowplow Ruby (Backend)** +### Snowplow Ruby (Backend) Snowplow is an enterprise-grade marketing and product analytics platform which helps track the way users engage with our website and application. [Snowplow Ruby](https://github.com/snowplow/snowplow/wiki/ruby-tracker) is a backend tracker for server-side events. -**Usage Ping** +### Usage Ping Usage Ping is a method for GitLab Inc to collect usage data on a GitLab instance. Usage Ping is primarily composed of row counts for different tables in the instance’s database. By comparing these counts month over month (or week over week), we can get a rough sense for how an instance is using the different features within the product. This high-level data is used to help our product, support, and sales teams. -Read more about how this works in the [Usage Ping guide](usage_ping.md) +For more details, read the [Usage Ping](usage_ping.md) guide. -**Database import** +### Database import Database imports are full imports of data into GitLab's data warehouse. For GitLab.com, the PostgreSQL database is loaded into Snowflake data warehouse every 6 hours. For more details, see the [data team handbook](https://about.gitlab.com/handbook/business-ops/data-team/#extract-and-load). -**Log system** +### Log system System logs are the application logs generated from running the GitLab Rails application. For more details, see the [log system](../../administration/logs.md) and [logging infrastructure](https://gitlab.com/gitlab-com/runbooks/tree/master/logging/doc#logging-infrastructure-overview). @@ -72,63 +78,63 @@ System logs are the application logs generated from running the GitLab Rails app Our different tracking tools allows us to track different types of events. The event types and examples of what data can be tracked are outlined below. -| Event Type | Snowplow JS (Frontend) | Snowplow Ruby (Backend) | Usage Ping | Database import | Log system | -| ------ | ------ | ------ | ------ | ------ | ------ | -| Database counts | ❌ | ❌ | ✅ | ✅ | ❌ | -| Pageview events | ✅ | ✅ | ❌ | ❌ | ❌ | -| UI events | ✅ | ❌ | ❌ | ❌ | ❌ | -| CRUD and API events | ❌ | ✅ | ❌ | ❌ | ❌ | -| Event funnels | ✅ | ✅ | ❌ | ❌ | ❌ | -| PostgreSQL Data | ❌ | ❌ | ❌ | ✅ | ❌ | -| Logs | ❌ | ❌ | ❌ | ❌ | ✅ | -| External services | ❌ | ❌ | ❌ | ❌ | ❌ | +| Event Type | Snowplow JS (Frontend) | Snowplow Ruby (Backend) | Usage Ping | Database import | Log system | +|---------------------|------------------------|-------------------------|---------------------|---------------------|---------------------| +| Database counts | **{dotted-circle}** | **{dotted-circle}** | **{check-circle}** | **{check-circle}** | **{dotted-circle}** | +| Pageview events | **{check-circle}** | **{check-circle}** | **{dotted-circle}** | **{dotted-circle}** | **{dotted-circle}** | +| UI events | **{check-circle}** | **{dotted-circle}** | **{dotted-circle}** | **{dotted-circle}** | **{dotted-circle}** | +| CRUD and API events | **{dotted-circle}** | **{check-circle}** | **{dotted-circle}** | **{dotted-circle}** | **{dotted-circle}** | +| Event funnels | **{check-circle}** | **{check-circle}** | **{dotted-circle}** | **{dotted-circle}** | **{dotted-circle}** | +| PostgreSQL Data | **{dotted-circle}** | **{dotted-circle}** | **{dotted-circle}** | **{check-circle}** | **{dotted-circle}** | +| Logs | **{dotted-circle}** | **{dotted-circle}** | **{dotted-circle}** | **{dotted-circle}** | **{check-circle}** | +| External services | **{dotted-circle}** | **{dotted-circle}** | **{dotted-circle}** | **{dotted-circle}** | **{dotted-circle}** | -**Database counts** +### Database counts -- How many Projects have been created by unique users -- How many users logged in the past 28 day +- Number of Projects created by unique users +- Number of users logged in the past 28 day -Database counts are row counts for different tables in an instance’s database. These are SQL count queries which have been filtered, grouped, or aggregated which provide high level usage data. The full list of available tables can be found in [structure.sql](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/structure.sql) +Database counts are row counts for different tables in an instance’s database. These are SQL count queries which have been filtered, grouped, or aggregated which provide high level usage data. The full list of available tables can be found in [structure.sql](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/structure.sql). -**Pageview events** +### Pageview events -- How many sessions visited the /dashboard/groups page +- Number of sessions that visited the /dashboard/groups page -**UI Events** +### UI Events -- How many sessions clicked on a button or link -- How many sessions closed a modal +- Number of sessions that clicked on a button or link +- Number of sessions that closed a modal UI events are any interface-driven actions from the browser including click data. -**CRUD or API events** +### CRUD or API events -- How many Git pushes were made -- How many GraphQL queries were made -- How many requests were made to a Rails action or controller. +- Number of Git pushes +- Number of GraphQL queries +- Number of requests to a Rails action or controller -These are backend events that include the creation, read, update, deletion of records and other events that might be triggered from layers that aren't necessarily only available in the interface. +These are backend events that include the creation, read, update, deletion of records, and other events that might be triggered from layers other than those available in the interface. -**Event funnels** +### Event funnels -- How many sessions performed action A, B, then C -- What is our conversion rate from step A to B? +- Number of sessions that performed action A, B, then C +- Conversion rate from step A to B -**PostgreSQL data** +### PostgreSQL data -These are raw database records which can be explored using business intelligence tools like Sisense. The full list of available tables can be found in [structure.sql](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/structure.sql) +These are raw database records which can be explored using business intelligence tools like Sisense. The full list of available tables can be found in [structure.sql](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/structure.sql). -**Logs** +### Logs These are raw logs such as the [Production logs](../../administration/logs.md#production_jsonlog), [API logs](../../administration/logs.md#api_jsonlog), or [Sidekiq logs](../../administration/logs.md#sidekiqlog). See the [overview of Logging Infrastructure](https://gitlab.com/gitlab-com/runbooks/tree/master/logging/doc#logging-infrastructure-overview) for more details. -**External services** +### External services These are external services a GitLab instance interacts with such as an [external storage provider](../../administration/static_objects_external_storage.md) or an [external container registry](../../administration/packages/container_registry.md#use-an-external-container-registry-with-gitlab-as-an-auth-endpoint). These services must be able to send data back into a GitLab instance for data to be tracked. ## Telemetry systems overview -The systems overview is a simplified diagram showing the interactions between GitLab Inc and self-managed nstances. +The systems overview is a simplified diagram showing the interactions between GitLab Inc and self-managed instances.  @@ -140,7 +146,7 @@ For Telemetry purposes, GitLab Inc has three major components: 1. [Data Infrastructure](https://about.gitlab.com/handbook/business-ops/data-team/data-infrastructure/): This contains everything managed by our data team including Sisense Dashboards for visualization, Snowflake for Data Warehousing, incoming data sources such as PostgreSQL Pipeline and S3 Bucket, and lastly our data collectors [GitLab.com's Snowplow Collector](https://about.gitlab.com/handbook/engineering/infrastructure/library/snowplow/) and GitLab's Versions Application. 1. GitLab.com: This is the production GitLab application which is made up of a Client and Server. On the Client or browser side, a Snowplow JS Tracker (Frontend) is used to track client-side events. On the Server or application side, a Snowplow Ruby Tracker (Backend) is used to track server-side events. The server also contains Usage Ping which leverages a PostgreSQL database and a Redis in-memory data store to report on usage data. Lastly, the server also contains System Logs which are generated from running the GitLab application. -1. [Monitoring infrastructure](https://about.gitlab.com/handbook/engineering/monitoring/): This is the infrastructure used to ensure GitLab.com is operating smoothly. System Logs are sent from GitLab.com to our monitoring infrastructure and collected by a FluentD collector. From FluentD, logs are either sent to long term Google Cloud Services cold storage via Stackdriver, or, they are sent to our Elastic Cluster via Cloud Pub/Sub which can be explored in real-time using Kibana +1. [Monitoring infrastructure](https://about.gitlab.com/handbook/engineering/monitoring/): This is the infrastructure used to ensure GitLab.com is operating smoothly. System Logs are sent from GitLab.com to our monitoring infrastructure and collected by a FluentD collector. From FluentD, logs are either sent to long term Google Cloud Services cold storage via Stackdriver, or, they are sent to our Elastic Cluster via Cloud Pub/Sub which can be explored in real-time using Kibana. ### Self-managed @@ -151,15 +157,15 @@ For Telemetry purposes, self-managed instances have two major components: ### Differences between GitLab Inc and Self-managed -As shown by the orange lines, on GitLab.com Snowplow JS, Snowplow Ruby, Usage Ping, and PostgreSQL database imports all flow into GitLab Inc's data fnfrastructure. However, on self-managed, only Usage Ping flows into GitLab Inc's data infrastructure. +As shown by the orange lines, on GitLab.com Snowplow JS, Snowplow Ruby, Usage Ping, and PostgreSQL database imports all flow into GitLab Inc's data infrastructure. However, on self-managed, only Usage Ping flows into GitLab Inc's data infrastructure. As shown by the green lines, on GitLab.com system logs flow into GitLab Inc's monitoring infrastructure. On self-managed, there are no logs sent to GitLab Inc's monitoring infrastructure. The differences between GitLab.com and self-managed are summarized below: -| Environment | Snowplow JS (Frontend) | Snowplow Ruby (Backend) | Usage Ping | Database import | Logs system | -| ------ | ------ | ------ | ------ | ------ | ------ | -| GitLab.com | ✅ | ✅ | ✅ | ✅ | ✅ | -| Self-Managed | ❌(1) | ❌(1) | ✅ | ❌ | ❌ | +| Environment | Snowplow JS (Frontend) | Snowplow Ruby (Backend) | Usage Ping | Database import | Logs system | +|--------------|------------------------|-------------------------|--------------------|---------------------|---------------------| +| GitLab.com | **{check-circle}** | **{check-circle}** | **{check-circle}** | **{check-circle}** | **{check-circle}** | +| Self-Managed | **{dotted-circle}**(1) | **{dotted-circle}**(1) | **{check-circle}** | **{dotted-circle}** | **{dotted-circle}** | Note (1): Snowplow JS and Snowplow Ruby are available on self-managed, however, the Snowplow Collector endpoint is set to a self-managed Snowplow Collector which GitLab Inc does not have access to. diff --git a/doc/development/telemetry/snowplow.md b/doc/development/telemetry/snowplow.md index aeaad6e5624..b7090ee4d20 100644 --- a/doc/development/telemetry/snowplow.md +++ b/doc/development/telemetry/snowplow.md @@ -1,3 +1,9 @@ +--- +stage: Growth +group: Telemetry +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Snowplow Guide This guide provides a details about how Snowplow works. It includes the following sections: @@ -40,7 +46,7 @@ From [Snowplow's documentation](https://github.com/snowplow/snowplow), Snowplow ## Snowplow schema -We currently have many definitions of Snowplow's schema. We have an active issue to [standardize this schema](https://gitlab.com/gitlab-org/gitlab/issues/207930) including the following definitions: +We currently have many definitions of Snowplow's schema. We have an active issue to [standardize this schema](https://gitlab.com/gitlab-org/gitlab/-/issues/207930) including the following definitions: - Frontend and backend taxonomy as listed below - [Feature instrumentation taxonomy](https://about.gitlab.com/handbook/product/feature-instrumentation/#taxonomy) @@ -121,7 +127,7 @@ Below is an example of `data-track-*` attributes assigned to a button: /> ``` -Event listeners are bound at the document level to handle click events on or within elements with these data attributes. This allows for them to be properly handled on rerendering and changes to the DOM, but it's important to know that because of the way these events are bound, click events shouldn't be stopped from propagating up the DOM tree. If for any reason click events are being stopped from propagating, you'll need to implement your own listeners and follow the instructions in [Tracking in raw JavaScript](#tracking-in-raw-javascript). +Event listeners are bound at the document level to handle click events on or within elements with these data attributes. This allows for them to be properly handled on re-rendering and changes to the DOM, but it's important to know that because of the way these events are bound, click events shouldn't be stopped from propagating up the DOM tree. If for any reason click events are being stopped from propagating, you'll need to implement your own listeners and follow the instructions in [Tracking in raw JavaScript](#tracking-in-raw-javascript). Below is a list of supported `data-track-*` attributes: @@ -213,7 +219,7 @@ button.addEventListener('click', () => { ### Tests and test helpers -In Jest particularly in vue tests, you can use the following: +In Jest particularly in Vue tests, you can use the following: ```javascript import { mockTracking } from 'helpers/tracking_helper'; @@ -330,10 +336,10 @@ Snowplow Inspector Chrome Extension is a browser extension for testing frontend Snowplow Micro is a very small version of a full Snowplow data collection pipeline: small enough that it can be launched by a test suite. Events can be recorded into Snowplow Micro just as they can a full Snowplow pipeline. Micro then exposes an API that can be queried. -Snowplow Micro is a docker-based solution for testing frontend and backend events in a local development environment. You need to modify GDK using the instructions below to set this up. +Snowplow Micro is a Docker-based solution for testing frontend and backend events in a local development environment. You need to modify GDK using the instructions below to set this up. - Read [Introducing Snowplow Micro](https://snowplowanalytics.com/blog/2019/07/17/introducing-snowplow-micro/) -- Look at the [Snowplow Micro repo](https://github.com/snowplow-incubator/snowplow-micro) +- Look at the [Snowplow Micro repository](https://github.com/snowplow-incubator/snowplow-micro) - Watch our [installation guide recording](https://www.youtube.com/watch?v=OX46fo_A0Ag) 1. Install [Snowplow Micro](https://github.com/snowplow-incubator/snowplow-micro) diff --git a/doc/development/telemetry/usage_ping.md b/doc/development/telemetry/usage_ping.md index e9b959eaa96..0f438e02772 100644 --- a/doc/development/telemetry/usage_ping.md +++ b/doc/development/telemetry/usage_ping.md @@ -1,19 +1,17 @@ -# Usage Ping Guide +--- +stage: Growth +group: Telemetry +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- -> - [Introduced][ee-557] in GitLab Enterprise Edition 8.10. -> - More statistics [were added][ee-735] in GitLab Enterprise Edition 8.12. -> - [Moved to GitLab Core][ce-23361] in 9.1. -> - More statistics [were added][ee-6602] in GitLab Ultimate 11.2. +# Usage Ping Guide -This guide provides a details about how usage ping works. It includes the following sections: +> - Introduced in GitLab Enterprise Edition 8.10. +> - More statistics were added in GitLab Enterprise Edition 8.12. +> - Moved to GitLab Core in 9.1. +> - More statistics were added in GitLab Ultimate 11.2. -1. [What is Usage Ping](#what-is-usage-ping) -1. [Usage Ping payload](#usage-ping-payload) -1. [Disabling Usage Ping](#disabling-usage-ping) -1. [Usage Ping request flow](#usage-ping-request-flow) -1. [How Usage Ping works](#how-usage-ping-works) -1. [Implementing Usage Ping](#implementing-usage-ping) -1. [Developing and testing usage ping](#developing-and-testing-usage-ping) +This guide describes Usage Ping's purpose and how it's implemented. For more information about Telemetry, see: @@ -27,237 +25,50 @@ More useful links: - [Data for Product Managers](https://about.gitlab.com/handbook/business-ops/data-team/data-for-product-managers/) - [Data Infrastructure](https://about.gitlab.com/handbook/business-ops/data-team/data-infrastructure/) -## What is Usage Ping +## What is Usage Ping? -- GitLab sends a weekly payload containing usage data to GitLab Inc. The usage ping uses high-level data to help our product, support, and sales teams. It does not send any project names, usernames, or any other specific data. The information from the usage ping is not anonymous, it is linked to the hostname of the instance. Sending usage ping is optional, and any instance can disable analytics. -- The usage data is primarily composed of row counts for different tables in the instance’s database. By comparing these counts month over month (or week over week), we can get a rough sense for how an instance is using the different features within the product. -- Usage ping is important to GitLab as we use it to calculate our and Stage Monthly Active Users (SMAU) which helps us measure the success of our stages and features. +- GitLab sends a weekly payload containing usage data to GitLab Inc. Usage Ping provides high-level data to help our product, support, and sales teams. It does not send any project names, usernames, or any other specific data. The information from the usage ping is not anonymous, it is linked to the hostname of the instance. Sending usage ping is optional, and any instance can disable analytics. +- The usage data is primarily composed of row counts for different tables in the instance’s database. By comparing these counts month over month (or week over week), we can get a rough sense for how an instance is using the different features within the product. In addition to counts, other facts + that help us classify and understand GitLab installations are collected. +- Usage ping is important to GitLab as we use it to calculate our Stage Monthly Active Users (SMAU) which helps us measure the success of our stages and features. - Once usage ping is enabled, GitLab will gather data from the other instances and will be able to show usage statistics of your instance to your users. -### Why Should We Enable Usage Ping? +### Why should we enable Usage Ping? -- The main purpose of Usage Ping is to build a better GitLab. Data about how GitLab is used is collected to better understand feature/stage adoption and usage, which helps us understand how GitLab is adding value and helps our team better understand the reasons why people use GitLab and with this knowledge we are able to make better product decisions. +- The main purpose of Usage Ping is to build a better GitLab. Data about how GitLab is used is collected to better understand feature/stage adoption and usage, which helps us understand how GitLab is adding value and helps our team better understand the reasons why people use GitLab and with this knowledge we're able to make better product decisions. - As a benefit of having the usage ping active, GitLab lets you analyze the users’ activities over time of your GitLab installation. - As a benefit of having the usage ping active, GitLab provides you with The DevOps Score,which gives you an overview of your entire instance’s adoption of Concurrent DevOps from planning to monitoring. - You will get better, more proactive support. (assuming that our TAMs and support organization used the data to deliver more value) - You will get insight and advice into how to get the most value out of your investment in GitLab. Wouldn't you want to know that a number of features or values are not being adopted in your organization? - You get a report that illustrates how you compare against other similar organizations (anonymized), with specific advice and recommendations on how to improve your DevOps processes. +- Usage Ping is enabled by default. To disable it, see [Disable Usage Ping](#disable-usage-ping). ### Limitations -- Usage Ping does not track frontend events things like page views, link clicks, or user sessions and only focuses on aggregated backend events. +- Usage Ping does not track frontend events things like page views, link clicks, or user sessions, and only focuses on aggregated backend events. - Because of these limitations we recommend instrumenting your products with Snowplow for more detailed analytics on GitLab.com and use Usage Ping to track aggregated backend events on self-managed. ## Usage Ping payload You can view the exact JSON payload sent to GitLab Inc. in the administration panel. To view the payload: -1. Navigate to the **Admin Area > Settings > Metrics and profiling**. +1. Navigate to **Admin Area > Settings > Metrics and profiling**. 1. Expand the **Usage statistics** section. 1. Click the **Preview payload** button. -Here is an example of the payload structure - -``` json -{ - "uuid": "0000000-0000-0000-0000-000000000000", - "hostname": "example.com", - "version": "12.10.0-pre", - "installation_type": "omnibus-gitlab", - "active_user_count": 999, - "recorded_at": "2020-04-17T07:43:54.162+00:00", - "edition": "EEU", - "license_md5": "00000000000000000000000000000000", - "license_id": null, - "historical_max_users": 999, - "licensee": { - "Name": "ABC, Inc.", - "Email": "email@example.com", - "Company": "ABC, Inc." - }, - "license_user_count": 999, - "license_starts_at": "2020-01-01", - "license_expires_at": "2021-01-01", - "license_plan": "ultimate", - "license_add_ons": { - }, - "license_trial": false, - "counts": { - "assignee_lists": 999, - "boards": 999, - "ci_builds": 999, - ... - }, - "container_registry_enabled": true, - "dependency_proxy_enabled": false, - "gitlab_shared_runners_enabled": true, - "gravatar_enabled": true, - "influxdb_metrics_enabled": true, - "ldap_enabled": false, - "mattermost_enabled": false, - "omniauth_enabled": true, - "prometheus_metrics_enabled": false, - "reply_by_email_enabled": "incoming+%{key}@incoming.gitlab.com", - "signup_enabled": true, - "web_ide_clientside_preview_enabled": true, - "ingress_modsecurity_enabled": true, - "projects_with_expiration_policy_disabled": 999, - "projects_with_expiration_policy_enabled": 999, - ... - "elasticsearch_enabled": true, - "license_trial_ends_on": null, - "geo_enabled": false, - "git": { - "version": { - "major": 2, - "minor": 26, - "patch": 1 - } - }, - "gitaly": { - "version": "12.10.0-rc1-93-g40980d40", - "servers": 56, - "filesystems": [ - "EXT_2_3_4" - ] - }, - "gitlab_pages": { - "enabled": true, - "version": "1.17.0" - }, - "database": { - "adapter": "postgresql", - "version": "9.6.15" - }, - "app_server": { - "type": "console" - }, - "avg_cycle_analytics": { - "issue": { - "average": 999, - "sd": 999, - "missing": 999 - }, - "plan": { - "average": null, - "sd": 999, - "missing": 999 - }, - "code": { - "average": null, - "sd": 999, - "missing": 999 - }, - "test": { - "average": null, - "sd": 999, - "missing": 999 - }, - "review": { - "average": null, - "sd": 999, - "missing": 999 - }, - "staging": { - "average": null, - "sd": 999, - "missing": 999 - }, - "production": { - "average": null, - "sd": 999, - "missing": 999 - }, - "total": 999 - }, - "usage_activity_by_stage": { - "configure": { - "project_clusters_enabled": 999, - ... - }, - "create": { - "merge_requests": 999, - ... - }, - "manage": { - "events": 999, - ... - }, - "monitor": { - "clusters": 999, - ... - }, - "package": { - "projects_with_packages": 999 - }, - "plan": { - "issues": 999, - ... - }, - "release": { - "deployments": 999, - ... - }, - "secure": { - "user_container_scanning_jobs": 999, - ... - }, - "verify": { - "ci_builds": 999, - ... - } - }, - "usage_activity_by_stage_monthly": { - "configure": { - "project_clusters_enabled": 999, - ... - }, - "create": { - "merge_requests": 999, - ... - }, - "manage": { - "events": 999, - ... - }, - "monitor": { - "clusters": 999, - ... - }, - "package": { - "projects_with_packages": 999 - }, - "plan": { - "issues": 999, - ... - }, - "release": { - "deployments": 999, - ... - }, - "secure": { - "user_container_scanning_jobs": 999, - ... - }, - "verify": { - "ci_builds": 999, - ... - } - } -} -``` +For an example payload, see [Example Usage Ping payload](#example-usage-ping-payload). -## Disabling usage ping +## Disable Usage Ping -The usage ping is opt-out. If you want to deactivate this feature, go to the Settings page of your administration panel and uncheck the Usage Ping checkbox. +To disable Usage Ping in the GitLab UI, go to the **Settings** page of your administration panel and uncheck the **Usage Ping** checkbox. -To disable the usage ping and prevent it from being configured in future through the administration panel, Omnibus installs can set the following in [`gitlab.rb`](https://docs.gitlab.com/omnibus/settings/configuration.html#configuration-options): +To disable Usage Ping and prevent it from being configured in the future through the administration panel, Omnibus installs can set the following in [`gitlab.rb`](https://docs.gitlab.com/omnibus/settings/configuration.html#configuration-options): ```ruby gitlab_rails['usage_ping_enabled'] = false ``` -And source installs can set the following in `gitlab.yml`: +Source installations can set the following in `gitlab.yml`: ```yaml production: &base @@ -267,9 +78,9 @@ production: &base usage_ping_enabled: false ``` -## Usage Ping Request Flow +## Usage Ping request flow -The following example shows a basic request/response flow between a GitLab Instance, the Versions Application, the License Application, Salesforce, GitLab's S3 Bucket, GitLab's Snowflake Data Warehouse, and Sisense.: +The following example shows a basic request/response flow between a GitLab instance, the Versions Application, the License Application, Salesforce, GitLab's S3 Bucket, GitLab's Snowflake Data Warehouse, and Sisense: ```mermaid sequenceDiagram @@ -303,34 +114,40 @@ sequenceDiagram ## How Usage Ping works 1. The Usage Ping [cron job](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/gitlab_usage_ping_worker.rb#L30) is set in Sidekiq to run weekly. -1. When the cron job runs, it calls [GitLab::UsageData.to_json](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/submit_usage_ping_service.rb#L22). -1. GitLab::UsageData.to_json [cascades down](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data.rb#L22) to ~400+ other counter method calls. -1. The response of all methods calls are [merged together](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data.rb#L14) into a single JSON payload in GitLab::UsageData.to_json. +1. When the cron job runs, it calls [`GitLab::UsageData.to_json`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/submit_usage_ping_service.rb#L22). +1. `GitLab::UsageData.to_json` [cascades down](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data.rb#L22) to ~400+ other counter method calls. +1. The response of all methods calls are [merged together](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data.rb#L14) into a single JSON payload in `GitLab::UsageData.to_json`. 1. The JSON payload is then [posted to the Versions application]( https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/submit_usage_ping_service.rb#L20). ## Implementing Usage Ping -Usage Ping consists of four types of counters which are all found in `usage_data.rb`: +Usage Ping consists of two kinds of data, counters and observations. Counters track how often a certain event +happened over time, such as how many CI pipelines have run. They are monotonic and always trend up. +Observations are facts collected from one or more GitLab instances and can carry arbitrary data. There are no +general guidelines around how to collect those, due to the individual nature of that data. + +There are four types of counters which are all found in `usage_data.rb`: - **Ordinary Batch Counters:** Simple count of a given ActiveRecord_Relation - **Distinct Batch Counters:** Distinct count of a given ActiveRecord_Relation on given column - **Alternative Counters:** Used for settings and configurations - **Redis Counters:** Used for in-memory counts. This method is being deprecated due to data inaccuracies and will be replaced with a persistent method. -Note: Only use the provided counter methods. Each counter method contains a built in fail safe to isolate each counter to avoid breaking the entire Usage Ping. +NOTE: **Note:** +Only use the provided counter methods. Each counter method contains a built in fail safe to isolate each counter to avoid breaking the entire Usage Ping. ### Why batch counting For large tables, PostgreSQL can take a long time to count rows due to MVCC [(Multi-version Concurrency Control)](https://en.wikipedia.org/wiki/Multiversion_concurrency_control). Batch counting is a counting method where a single large query is broken into multiple smaller queries. For example, instead of a single query querying 1,000,000 records, with batch counting, you can execute 100 queries of 10,000 records each. Batch counting is useful for avoiding database timeouts as each batch query is significantly shorter than one single long running query. -For GitLab.com, there are extremely large tables with 15 second query timeouts, so, we use batch counting to avoid encountering timeouts. Here are the sizes of some GitLab.com tables: +For GitLab.com, there are extremely large tables with 15 second query timeouts, so we use batch counting to avoid encountering timeouts. Here are the sizes of some GitLab.com tables: -| Table | Row counts in millions | -| ------ | ------ | -| merge_request_diff_commits | 2280 | -| ci_build_trace_sections | 1764 | -| merge_request_diff_files | 1082 | -| events | 514 | +| Table | Row counts in millions | +|------------------------------|------------------------| +| `merge_request_diff_commits` | 2280 | +| `ci_build_trace_sections` | 1764 | +| `merge_request_diff_files` | 1082 | +| `events` | 514 | There are two batch counting methods provided, `Ordinary Batch Counters` and `Distinct Batch Counters`. Batch counting requires indexes on columns to calculate max, min, and range queries. In some cases, a specialized index may need to be added on the columns involved in a counter. @@ -393,7 +210,7 @@ Method: `redis_usage_data(counter, &block)` Arguments: - `counter`: a counter from `Gitlab::UsageDataCounters`, that has `fallback_totals` method implemented -- or a `block`: wich is evaluated +- or a `block`: which is evaluated Example of usage: @@ -413,8 +230,8 @@ Method: `alt_usage_data(value = nil, fallback: -1, &block)` Arguments: -- `value`: a simple static value in wich case the value is simply returned. -- or a `block`: wich is evaluated +- `value`: a simple static value in which case the value is simply returned. +- or a `block`: which is evaluated - `fallback: -1`: the common value used for any metrics that are failing. Example of usage: @@ -425,6 +242,29 @@ alt_usage_data { Gitlab::CurrentSettings.uuid } alt_usage_data(999) ``` +### Prometheus Queries + +In those cases where operational metrics should be part of Usage Ping, a database or Redis query is unlikely +to provide useful data. Instead, Prometheus might be more appropriate, since most of GitLab's architectural +components publish metrics to it that can be queried back, aggregated, and included as usage data. + +NOTE: **Note:** +Prometheus as a data source for Usage Ping is currently only available for single-node Omnibus installations +that are running the [bundled Prometheus](../../administration/monitoring/prometheus/index.md) instance. + +In order to query Prometheus for metrics, a helper method is available that will `yield` a fully configured +`PrometheusClient`, given it is available as per the note above: + +```ruby +with_prometheus_client do |client| + response = client.query('<your query>') + ... +end +``` + +Please refer to [the `PrometheusClient` definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/prometheus_client.rb) +for how to use its API to query for data. + ## Developing and testing Usage Ping ### 1. Use your Rails console to manually test counters @@ -441,49 +281,591 @@ Gitlab::UsageData.distinct_count(::Note.with_suggestions.where(time_period), :au ### 2. Generate the SQL query -Your Rails console will give back the generated SQL queries. +Your Rails console will return the generated SQL queries. Example: ```ruby - pry(main)> Gitlab::UsageData.count(User.active) - (0.4ms) SELECT "features"."key" FROM "features" - (0.7ms) SELECT MIN("users"."id") FROM "users" WHERE ("users"."state" IN ('active')) AND (ghost IS NOT TRUE) AND ("users"."user_type" IS NULL OR "users"."user_type" NOT IN (2, 1, 3)) - (0.6ms) SELECT MAX("users"."id") FROM "users" WHERE ("users"."state" IN ('active')) AND (ghost IS NOT TRUE) AND ("users"."user_type" IS NULL OR "users"."user_type" NOT IN (2, 1, 3)) - (0.5ms) SELECT COUNT("users"."id") FROM "users" WHERE ("users"."state" IN ('active')) AND (ghost IS NOT TRUE) AND ("users"."user_type" IS NULL OR "users"."user_type" NOT IN (2, 1, 3)) AND "users"."id" BETWEEN 0 AND 99999 +pry(main)> Gitlab::UsageData.count(User.active) + (2.6ms) SELECT "features"."key" FROM "features" + (15.3ms) SELECT MIN("users"."id") FROM "users" WHERE ("users"."state" IN ('active')) AND ("users"."user_type" IS NULL OR "users"."user_type" IN (6, 4)) + (2.4ms) SELECT MAX("users"."id") FROM "users" WHERE ("users"."state" IN ('active')) AND ("users"."user_type" IS NULL OR "users"."user_type" IN (6, 4)) + (1.9ms) SELECT COUNT("users"."id") FROM "users" WHERE ("users"."state" IN ('active')) AND ("users"."user_type" IS NULL OR "users"."user_type" IN (6, 4)) AND "users"."id" BETWEEN 1 AND 100000 ``` ### 3. Optimize queries with #database-lab Paste the SQL query into `#database-lab` to see how the query performs at scale. -- #database-lab is a Slack channel which uses a production-sized environment to test your queries +- `#database-lab` is a Slack channel which uses a production-sized environment to test your queries. - GitLab.com’s production database has a 15 second timeout. -- For each query we require an execution time of under 1 second due do cold caches which can 10x this time. -- Add a specialized index on columns involved to reduce your the execution time. - -In order to have an understanding of the queries execution we add in the MR description the following information - -For counters that have a `time_period` test and add information for both cases. - -- with `time_period = {}` for all time period -- and `time_period = { created_at: 28.days.ago..Time.current }` for last 28 days period - -Execution plan and query time before and after optimization - -Using database-lab and [explain.depesz.com](https://explain.depesz.com/) see more details in [database review guide](../database_review.md#preparation-when-adding-or-modifying-queries) +- For each query we require an execution time of under 1 second due to cold caches which can 10x this time. +- Add a specialized index on columns involved to reduce the execution time. -Query generated for the index and time +In order to have an understanding of the query's execution we add in the MR description the following information: -Using database-lab +- For counters that have a `time_period` test we add information for both cases: + - `time_period = {}` for all time periods + - `time_period = { created_at: 28.days.ago..Time.current }` for last 28 days period +- Execution plan and query time before and after optimization +- Query generated for the index and time +- Migration output for up and down execution -Migration output for up and down execution +We also use `#database-lab` and [explain.depesz.com](https://explain.depesz.com/). For more details, see the [database review guide](../database_review.md#preparation-when-adding-or-modifying-queries). Examples of query optimization work: - [Example 1](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26445) - [Example 2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26871) -### 4. Ask for a Telemetry Review - -On GitLab.com, we have DangerBot setup to monitor Telemetry related files and DangerBot will recommend a Telemetry review. Simply `@gitlab-org/growth/telemetry/engineers` in your MR for a review. +### 4. Add the metric definition + +When adding, changing, or updating metrics, please update the [Usage Statistics definition table](#usage-statistics-definitions). + +### 5. Add new metric to Versions Application + +Check if new metrics need to be added to the Versions Application. See `usage_data` [schema](https://gitlab.com/gitlab-services/version-gitlab-com/-/blob/master/db/schema.rb#L147) and usage data [parameters accepted](https://gitlab.com/gitlab-services/version-gitlab-com/-/blob/master/app/services/usage_ping.rb). Any metrics added under the `counts` key are saved in the `counts` column. + +### 6. Ask for a Telemetry Review + +On GitLab.com, we have DangerBot setup to monitor Telemetry related files and DangerBot will recommend a Telemetry review. Mention `@gitlab-org/growth/telemetry/engineers` in your MR for a review. + +### Optional: Test Prometheus based Usage Ping + +If the data submitted includes metrics [queried from Prometheus](#prometheus-queries) that you would like to inspect and verify, +then you need to ensure that a Prometheus server is running locally, and that furthermore the respective GitLab components +are exporting metrics to it. If you do not need to test data coming from Prometheus, no further action +is necessary, since Usage Ping should degrade gracefully in the absence of a running Prometheus server. + +There are currently three kinds of components that may export data to Prometheus, and which are included in Usage Ping: + +- [`node_exporter`](https://github.com/prometheus/node_exporter) - Exports node metrics from the host machine +- [`gitlab-exporter`](https://gitlab.com/gitlab-org/gitlab-exporter) - Exports process metrics from various GitLab components +- various GitLab services such as Sidekiq and the Rails server that export their own metrics + +#### Test with an Omnibus container + +This is the recommended approach to test Prometheus based Usage Ping. + +The easiest way to verify your changes is to build a new Omnibus image from your code branch via CI, then download the image +and run a local container instance: + +1. From your merge request, click on the `qa` stage, then trigger the `package-and-qa` job. This job will trigger an Omnibus +build in a [downstream pipeline of the `omnibus-gitlab-mirror` project](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/-/pipelines). +1. In the downstream pipeline, wait for the `gitlab-docker` job to finish. +1. Open the job logs and locate the full container name including the version. It will take the following form: `registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:<VERSION>`. +1. On your local machine, make sure you are logged in to the GitLab Docker registry. You can find the instructions for this in +[Authenticating to the GitLab Container Registry](../../user/packages/container_registry/index.md#authenticating-to-the-gitlab-container-registry). +1. Once logged in, download the new image via `docker pull registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:<VERSION>` +1. For more information about working with and running Omnibus GitLab containers in Docker, please refer to [GitLab Docker images](https://docs.gitlab.com/omnibus/docker/README.html) in the Omnibus documentation. + +#### Test with GitLab development toolkits + +This is the less recommended approach, since it comes with a number of difficulties when emulating a real GitLab deployment. + +The [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit) is not currently set up to run a Prometheus server or `node_exporter` alongside other GitLab components. If you would +like to do so, [Monitoring the GDK with Prometheus](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/master/doc/howto/prometheus/index.md#monitoring-the-gdk-with-prometheus) is a good start. + +The [GCK](https://gitlab.com/gitlab-org/gitlab-compose-kit) has limited support for testing Prometheus based Usage Ping. +By default, it already comes with a fully configured Prometheus service that is set up to scrape a number of components, +but with the following limitations: + +- It does not currently run a `gitlab-exporter` instance, so several `process_*` metrics from services such as Gitaly may be missing. +- While it runs a `node_exporter`, `docker-compose` services emulate hosts, meaning that it would normally report itself to not be associated +with any of the other services that are running. That is not how node metrics are reported in a production setup, where `node_exporter` +always runs as a process alongside other GitLab components on any given node. From Usage Ping's perspective none of the node data would therefore +appear to be associated to any of the services running, since they all appear to be running on different hosts. To alleviate this problem, the `node_exporter` in GCK was arbitrarily "assigned" to the `web` service, meaning only for this service `node_*` metrics will appear in Usage Ping. + +## Usage Statistics definitions + +| Statistic | Section | Stage | Tier | Description | +|:--------------------------------------------------------|:-----------------------------------|:------------|:---------------|:--------------------------------------------------| +| `uuid` | | | | | +| `hostname` | | | | | +| `version` | | | | | +| `installation_type` | | | | | +| `active_user_count` | | | | | +| `recorded_at` | | | | | +| `edition` | | | | | +| `license_md5` | | | | | +| `license_id` | | | | | +| `historical_max_users` | | | | | +| `Name` | `licensee` | | | | +| `Email` | `licensee` | | | | +| `Company` | `licensee` | | | | +| `license_user_count` | | | | | +| `license_starts_at` | | | | | +| `license_expires_at` | | | | | +| `license_plan` | | | | | +| `license_trial` | | | | | +| `assignee_lists` | `counts` | | | | +| `boards` | `counts` | | | | +| `ci_builds` | `counts` | `verify` | | Unique builds in project | +| `ci_internal_pipelines` | `counts` | `verify` | | Total pipelines in GitLab repositories | +| `ci_external_pipelines` | `counts` | `verify` | | Total pipelines in external repositories | +| `ci_pipeline_config_auto_devops` | `counts` | `verify` | | Total pipelines from an Auto DevOps template | +| `ci_pipeline_config_repository` | `counts` | `verify` | | Total Pipelines from templates in repository | +| `ci_runners` | `counts` | `verify` | | Total configured Runners in project | +| `ci_triggers` | `counts` | `verify` | | Total configured Triggers in project | +| `ci_pipeline_schedules` | `counts` | `verify` | | Pipeline schedules in GitLab | +| `auto_devops_enabled` | `counts` |`configure` | | Projects with Auto DevOps template enabled | +| `auto_devops_disabled` | `counts` |`configure` | | Projects with Auto DevOps template disabled | +| `deploy_keys` | `counts` | | | | +| `deployments` | `counts` |`release` | | Total deployments | +| `dast_jobs` | `counts` | | | | +| `successful_deployments` | `counts` |`release` | | Total successful deployments | +| `failed_deployments` | `counts` |`release` | | Total failed deployments | +| `environments` | `counts` |`release` | | Total available and stopped environments | +| `clusters` | `counts` |`configure` | | Total GitLab Managed clusters both enabled and disabled | +| `clusters_enabled` | `counts` |`configure` | | Total GitLab Managed clusters currently enabled | +| `project_clusters_enabled` | `counts` |`configure` | | Total GitLab Managed clusters attached to projects| +| `group_clusters_enabled` | `counts` |`configure` | | Total GitLab Managed clusters attached to groups | +| `instance_clusters_enabled` | `counts` |`configure` | | Total GitLab Managed clusters attached to the instance | +| `clusters_disabled` | `counts` |`configure` | | Total GitLab Managed disabled clusters | +| `project_clusters_disabled` | `counts` |`configure` | | Total GitLab Managed disabled clusters previously attached to projects | +| `group_clusters_disabled` | `counts` |`configure` | | Total GitLab Managed disabled clusters previously attached to groups | +| `instance_clusters_disabled` | `counts` |`configure` | | Total GitLab Managed disabled clusters previously attached to the instance | +| `clusters_platforms_eks` | `counts` |`configure` | | Total GitLab Managed clusters provisioned with GitLab on AWS EKS | +| `clusters_platforms_gke` | `counts` |`configure` | | Total GitLab Managed clusters provisioned with GitLab on GCE GKE | +| `clusters_platforms_user` | `counts` |`configure` | | Total GitLab Managed clusters that are user provisioned | +| `clusters_applications_helm` | `counts` |`configure` | | Total GitLab Managed clusters with Helm enabled | +| `clusters_applications_ingress` | `counts` |`configure` | | Total GitLab Managed clusters with Ingress enabled | +| `clusters_applications_cert_managers` | `counts` |`configure` | | Total GitLab Managed clusters with Cert Manager enabled | +| `clusters_applications_crossplane` | `counts` |`configure` | | Total GitLab Managed clusters with Crossplane enabled | +| `clusters_applications_prometheus` | `counts` |`configure` | | Total GitLab Managed clusters with Prometheus enabled | +| `clusters_applications_runner` | `counts` |`configure` | | Total GitLab Managed clusters with Runner enabled | +| `clusters_applications_knative` | `counts` |`configure` | | Total GitLab Managed clusters with Knative enabled | +| `clusters_applications_elastic_stack` | `counts` |`configure` | | Total GitLab Managed clusters with Elastic Stack enabled | +| `clusters_management_project` | `counts` |`configure` | | Total GitLab Managed clusters with defined cluster management project | +| `in_review_folder` | `counts` | | | | +| `grafana_integrated_projects` | `counts` | | | | +| `groups` | `counts` | | | | +| `issues` | `counts` | | | | +| `issues_created_from_gitlab_error_tracking_ui` | `counts` | `monitor` | | | +| `issues_with_associated_zoom_link` | `counts` | `monitor` | | | +| `issues_using_zoom_quick_actions` | `counts` | `monitor` | | | +| `issues_with_embedded_grafana_charts_approx` | `counts` | `monitor` | | | +| `issues_with_health_status` | `counts` | | | | +| `keys` | `counts` | | | | +| `label_lists` | `counts` | | | | +| `lfs_objects` | `counts` | | | | +| `milestone_lists` | `counts` | | | | +| `milestones` | `counts` | | | | +| `pages_domains` | `counts` |`release` | | Total GitLab Pages domains | +| `pool_repositories` | `counts` | | | | +| `projects` | `counts` | | | | +| `projects_imported_from_github` | `counts` | | | | +| `projects_with_repositories_enabled` | `counts` | | | | +| `projects_with_error_tracking_enabled` | `counts` | `monitor` | | | +| `protected_branches` | `counts` | | | | +| `releases` | `counts` |`release` | | Unique release tags | +| `remote_mirrors` | `counts` | | | | +| `requirements_created` | `counts` | | | | +| `snippets` | `counts` | | | | +| `suggestions` | `counts` | | | | +| `todos` | `counts` | | | | +| `uploads` | `counts` | | | | +| `web_hooks` | `counts` | | | | +| `projects_alerts_active` | `counts` | | | | +| `projects_asana_active` | `counts` | | | | +| `projects_assembla_active` | `counts` | | | | +| `projects_bamboo_active` | `counts` | | | | +| `projects_bugzilla_active` | `counts` | | | | +| `projects_buildkite_active` | `counts` | | | | +| `projects_campfire_active` | `counts` | | | | +| `projects_custom_issue_tracker_active` | `counts` | | | | +| `projects_discord_active` | `counts` | | | | +| `projects_drone_ci_active` | `counts` | | | | +| `projects_emails_on_push_active` | `counts` | | | | +| `projects_external_wiki_active` | `counts` | | | +| `projects_flowdock_active` | `counts` | | | | +| `projects_github_active` | `counts` | | | | +| `projects_hangouts_chat_active` | `counts` | | | | +| `projects_hipchat_active` | `counts` | | | | +| `projects_irker_active` | `counts` | | | | +| `projects_jenkins_active` | `counts` | | | | +| `projects_jira_active` | `counts` | | | | +| `projects_mattermost_active` | `counts` | | | | +| `projects_mattermost_slash_commands_active` | `counts` | | | | +| `projects_microsoft_teams_active` | `counts` | | | | +| `projects_packagist_active` | `counts` | | | | +| `projects_pipelines_email_active` | `counts` | | | | +| `projects_pivotaltracker_active` | `counts` | | | | +| `projects_prometheus_active` | `counts` | | | | +| `projects_pushover_active` | `counts` | | | | +| `projects_redmine_active` | `counts` | | | | +| `projects_slack_active` | `counts` | | | | +| `projects_slack_slash_commands_active` | `counts` | | | | +| `projects_teamcity_active` | `counts` | | | | +| `projects_unify_circuit_active` | `counts` | | | | +| `projects_webex_teams_active` | `counts` | | | | +| `projects_youtrack_active` | `counts` | | | | +| `projects_slack_notifications_active` | `counts` | | | | +| `projects_slack_slash_active` | `counts` | | | | +| `projects_jira_server_active` | `counts` | | | | +| `projects_jira_cloud_active` | `counts` | | | | +| `projects_jira_dvcs_cloud_active` | `counts` | | | | +| `projects_jira_dvcs_server_active` | `counts` | | | | +| `labels` | `counts` | | | | +| `merge_requests` | `counts` | | | | +| `merge_requests_users` | `counts` | | | | +| `notes` | `counts` | | | | +| `wiki_pages_create` | `counts` | | | | +| `wiki_pages_update` | `counts` | | | | +| `wiki_pages_delete` | `counts` | | | | +| `web_ide_commits` | `counts` | | | | +| `web_ide_views` | `counts` | | | | +| `web_ide_merge_requests` | `counts` | | | | +| `web_ide_previews` | `counts` | | | | +| `snippet_comment` | `counts` | | | | +| `commit_comment` | `counts` | | | | +| `merge_request_comment` | `counts` | | | | +| `snippet_create` | `counts` | | | | +| `snippet_update` | `counts` | | | | +| `navbar_searches` | `counts` | | | | +| `cycle_analytics_views` | `counts` | | | | +| `productivity_analytics_views` | `counts` | | | | +| `source_code_pushes` | `counts` | | | | +| `merge_request_create` | `counts` | | | | +| `design_management_designs_create` | `counts` | | | | +| `design_management_designs_update` | `counts` | | | | +| `design_management_designs_delete` | `counts` | | | | +| `licenses_list_views` | `counts` | | | | +| `user_preferences_group_overview_details` | `counts` | | | | +| `user_preferences_group_overview_security_dashboard` | `counts` | | | | +| `ingress_modsecurity_logging` | `counts` | | | | +| `ingress_modsecurity_blocking` | `counts` | | | | +| `ingress_modsecurity_disabled` | `counts` | | | | +| `ingress_modsecurity_not_installed` | `counts` | | | | +| `dependency_list_usages_total` | `counts` | | | | +| `epics` | `counts` | | | | +| `feature_flags` | `counts` | | | | +| `geo_nodes` | `counts` | `geo` | | Number of sites in a Geo deployment | +| `geo_event_log_max_id` | `counts` | `geo` | | Number of replication events on a Geo primary | +| `incident_issues` | `counts` | `monitor` | | Issues created by the alert bot | +| `alert_bot_incident_issues` | `counts` | `monitor` | | Issues created by the alert bot | +| `incident_labeled_issues` | `counts` | `monitor` | | Issues with the incident label | +| `issues_created_gitlab_alerts` | `counts` | `monitor` | | Issues created from alerts by non-alert bot users | +| `issues_created_manually_from_alerts` | `counts` | `monitor` | | Issues created from alerts by non-alert bot users | +| `issues_created_from_alerts` | `counts` | `monitor` | | Issues created from Prometheus and alert management alerts | +| `ldap_group_links` | `counts` | | | | +| `ldap_keys` | `counts` | | | | +| `ldap_users` | `counts` | | | | +| `pod_logs_usages_total` | `counts` | | | | +| `projects_enforcing_code_owner_approval` | `counts` | | | | +| `projects_mirrored_with_pipelines_enabled` | `counts` |`release` | | Projects with repository mirroring enabled | +| `projects_reporting_ci_cd_back_to_github` | `counts` |`verify` | | Projects with a GitHub service pipeline enabled | +| `projects_with_packages` | `counts` |`package` | | Projects with package registry configured | +| `projects_with_prometheus_alerts` | `counts` |`monitor` | | Projects with Prometheus alerting enabled | +| `projects_with_tracing_enabled` | `counts` |`monitor` | | Projects with tracing enabled | +| `projects_with_alerts_service_enabled` | `counts` |`monitor` | | Projects with alerting service enabled | +| `template_repositories` | `counts` | | | | +| `container_scanning_jobs` | `counts` | | | | +| `dependency_scanning_jobs` | `counts` | | | | +| `license_management_jobs` | `counts` | | | | +| `sast_jobs` | `counts` | | | | +| `status_page_projects` | `counts` | `monitor` | | Projects with status page enabled | +| `status_page_issues` | `counts` | `monitor` | | Issues published to a Status Page | +| `status_page_incident_publishes` | `counts` | `monitor` | | Cumulative count of usages of publish operation | +| `status_page_incident_unpublishes` | `counts` | `monitor` | | Cumulative count of usages of unpublish operation | +| `epics_deepest_relationship_level` | `counts` | | | | +| `operations_dashboard_default_dashboard` | `counts` | `monitor` | | Active users with enabled operations dashboard | +| `operations_dashboard_users_with_projects_added` | `counts` | `monitor` | | Active users with projects on operations dashboard| +| `container_registry_enabled` | | | | | +| `dependency_proxy_enabled` | | | | | +| `gitlab_shared_runners_enabled` | | | | | +| `gravatar_enabled` | | | | | +| `ldap_enabled` | | | | | +| `mattermost_enabled` | | | | | +| `omniauth_enabled` | | | | | +| `prometheus_metrics_enabled` | | | | | +| `reply_by_email_enabled` | | | | | +| `average` | `avg_cycle_analytics - code` | | | | +| `sd` | `avg_cycle_analytics - code` | | | | +| `missing` | `avg_cycle_analytics - code` | | | | +| `average` | `avg_cycle_analytics - test` | | | | +| `sd` | `avg_cycle_analytics - test` | | | | +| `missing` | `avg_cycle_analytics - test` | | | | +| `average` | `avg_cycle_analytics - review` | | | | +| `sd` | `avg_cycle_analytics - review` | | | | +| `missing` | `avg_cycle_analytics - review` | | | | +| `average` | `avg_cycle_analytics - staging` | | | | +| `sd` | `avg_cycle_analytics - staging` | | | | +| `missing` | `avg_cycle_analytics - staging` | | | | +| `average` | `avg_cycle_analytics - production` | | | | +| `sd` | `avg_cycle_analytics - production` | | | | +| `missing` | `avg_cycle_analytics - production` | | | | +| `total` | `avg_cycle_analytics` | | | | +| `clusters_applications_cert_managers` | `usage_activity_by_stage` | `configure` | | Unique clusters with certificate managers enabled | +| `clusters_applications_helm` | `usage_activity_by_stage` | `configure` | | Unique clusters with Helm enabled | +| `clusters_applications_ingress` | `usage_activity_by_stage` | `configure` | | Unique clusters with Ingress enabled | +| `clusters_applications_knative` | `usage_activity_by_stage` | `configure` | | Unique clusters with Knative enabled | +| `clusters_management_project` | `usage_activity_by_stage` | `configure` | | Unique clusters with project management enabled | +| `clusters_disabled` | `usage_activity_by_stage` | `configure` | | Total non-"GitLab Managed clusters" | +| `clusters_enabled` | `usage_activity_by_stage` | `configure` | | Total GitLab Managed clusters | +| `clusters_platforms_gke` | `usage_activity_by_stage` | `configure` | | Unique clusters with Google Cloud installed | +| `clusters_platforms_eks` | `usage_activity_by_stage` | `configure` | | Unique clusters with AWS installed | +| `clusters_platforms_user` | `usage_activity_by_stage` | `configure` | | Unique clusters that are user provided | +| `instance_clusters_disabled` | `usage_activity_by_stage` | `configure` | | Unique clusters disabled on instance | +| `instance_clusters_enabled` | `usage_activity_by_stage` | `configure` | | Unique clusters enabled on instance | +| `group_clusters_disabled` | `usage_activity_by_stage` | `configure` | | Unique clusters disabled on group | +| `group_clusters_enabled` | `usage_activity_by_stage` | `configure` | | Unique clusters enabled on group | +| `project_clusters_disabled` | `usage_activity_by_stage` | `configure` | | Unique clusters disabled on project | +| `project_clusters_enabled` | `usage_activity_by_stage` | `configure` | | Unique clusters enabled on project | +| `projects_slack_notifications_active` | `usage_activity_by_stage` | `configure` | | Unique projects with Slack service enabled | +| `projects_slack_slash_active` | `usage_activity_by_stage` | `configure` | | Unique projects with Slack '/' commands enabled | +| `projects_with_prometheus_alerts: 0` | `usage_activity_by_stage` | `monitor` | | Projects with Prometheus enabled and no alerts | +| `deploy_keys` | `usage_activity_by_stage` | `create` | | | +| `keys` | `usage_activity_by_stage` | `create` | | | +| `projects_jira_dvcs_server_active` | `usage_activity_by_stage` | `plan` | | | +| `service_desk_enabled_projects` | `usage_activity_by_stage` | `plan` | | | +| `service_desk_issues` | `usage_activity_by_stage` | `plan` | | | +| `todos: 0` | `usage_activity_by_stage` | `plan` | | | +| `deployments` | `usage_activity_by_stage` | `release` | | Total deployments | +| `failed_deployments` | `usage_activity_by_stage` | `release` | | Total failed deployments | +| `projects_mirrored_with_pipelines_enabled` | `usage_activity_by_stage` | `release` | | Projects with repository mirroring enabled | +| `releases` | `usage_activity_by_stage` | `release` | | Unique release tags in project | +| `successful_deployments: 0` | `usage_activity_by_stage` | `release` | | Total successful deployments | +| `user_preferences_group_overview_security_dashboard: 0` | `usage_activity_by_stage` | `secure` | | | +| `ci_builds` | `usage_activity_by_stage` | `verify` | | Unique builds in project | +| `ci_external_pipelines` | `usage_activity_by_stage` | `verify` | | Total pipelines in external repositories | +| `ci_internal_pipelines` | `usage_activity_by_stage` | `verify` | | Total pipelines in GitLab repositories | +| `ci_pipeline_config_auto_devops` | `usage_activity_by_stage` | `verify` | | Total pipelines from an Auto DevOps template | +| `ci_pipeline_config_repository` | `usage_activity_by_stage` | `verify` | | Pipelines from templates in repository | +| `ci_pipeline_schedules` | `usage_activity_by_stage` | `verify` | | Pipeline schedules in GitLab | +| `ci_pipelines` | `usage_activity_by_stage` | `verify` | | Total pipelines | +| `ci_triggers` | `usage_activity_by_stage` | `verify` | | Triggers enabled | +| `clusters_applications_runner` | `usage_activity_by_stage` | `verify` | | Unique clusters with Runner enabled | +| `projects_reporting_ci_cd_back_to_github: 0` | `usage_activity_by_stage` | `verify` | | Unique projects with a GitHub pipeline enabled | +| `nodes` | `topology` | `enablement`| | The list of server nodes on which GitLab components are running | +| `duration_s` | `topology` | `enablement`| | Time it took to collect topology data | +| `node_memory_total_bytes` | `topology > nodes` | `enablement`| | The total available memory of this node | +| `node_cpus` | `topology > nodes` | `enablement`| | The number of CPU cores of this node | +| `node_services` | `topology > nodes` | `enablement`| | The list of GitLab services running on this node | +| `name` | `topology > nodes > node_services` | `enablement`| | The name of the GitLab service running on this node | +| `process_count` | `topology > nodes > node_services` | `enablement`| | The number of processes running for this service | +| `process_memory_rss` | `topology > nodes > node_services` | `enablement`| | The average Resident Set Size of a service process | +| `process_memory_uss` | `topology > nodes > node_services` | `enablement`| | The average Unique Set Size of a service process | +| `process_memory_pss` | `topology > nodes > node_services` | `enablement`| | The average Proportional Set Size of a service process | + +## Example Usage Ping payload + +The following is example content of the Usage Ping payload. + +```json +{ + "uuid": "0000000-0000-0000-0000-000000000000", + "hostname": "example.com", + "version": "12.10.0-pre", + "installation_type": "omnibus-gitlab", + "active_user_count": 999, + "recorded_at": "2020-04-17T07:43:54.162+00:00", + "edition": "EEU", + "license_md5": "00000000000000000000000000000000", + "license_id": null, + "historical_max_users": 999, + "licensee": { + "Name": "ABC, Inc.", + "Email": "email@example.com", + "Company": "ABC, Inc." + }, + "license_user_count": 999, + "license_starts_at": "2020-01-01", + "license_expires_at": "2021-01-01", + "license_plan": "ultimate", + "license_add_ons": { + }, + "license_trial": false, + "counts": { + "assignee_lists": 999, + "boards": 999, + "ci_builds": 999, + ... + }, + "container_registry_enabled": true, + "dependency_proxy_enabled": false, + "gitlab_shared_runners_enabled": true, + "gravatar_enabled": true, + "influxdb_metrics_enabled": true, + "ldap_enabled": false, + "mattermost_enabled": false, + "omniauth_enabled": true, + "prometheus_metrics_enabled": false, + "reply_by_email_enabled": "incoming+%{key}@incoming.gitlab.com", + "signup_enabled": true, + "web_ide_clientside_preview_enabled": true, + "ingress_modsecurity_enabled": true, + "projects_with_expiration_policy_disabled": 999, + "projects_with_expiration_policy_enabled": 999, + ... + "elasticsearch_enabled": true, + "license_trial_ends_on": null, + "geo_enabled": false, + "git": { + "version": { + "major": 2, + "minor": 26, + "patch": 1 + } + }, + "gitaly": { + "version": "12.10.0-rc1-93-g40980d40", + "servers": 56, + "clusters": 14, + "filesystems": [ + "EXT_2_3_4" + ] + }, + "gitlab_pages": { + "enabled": true, + "version": "1.17.0" + }, + "database": { + "adapter": "postgresql", + "version": "9.6.15" + }, + "app_server": { + "type": "console" + }, + "avg_cycle_analytics": { + "issue": { + "average": 999, + "sd": 999, + "missing": 999 + }, + "plan": { + "average": null, + "sd": 999, + "missing": 999 + }, + "code": { + "average": null, + "sd": 999, + "missing": 999 + }, + "test": { + "average": null, + "sd": 999, + "missing": 999 + }, + "review": { + "average": null, + "sd": 999, + "missing": 999 + }, + "staging": { + "average": null, + "sd": 999, + "missing": 999 + }, + "production": { + "average": null, + "sd": 999, + "missing": 999 + }, + "total": 999 + }, + "usage_activity_by_stage": { + "configure": { + "project_clusters_enabled": 999, + ... + }, + "create": { + "merge_requests": 999, + ... + }, + "manage": { + "events": 999, + ... + }, + "monitor": { + "clusters": 999, + ... + }, + "package": { + "projects_with_packages": 999 + }, + "plan": { + "issues": 999, + ... + }, + "release": { + "deployments": 999, + ... + }, + "secure": { + "user_container_scanning_jobs": 999, + ... + }, + "verify": { + "ci_builds": 999, + ... + } + }, + "usage_activity_by_stage_monthly": { + "configure": { + "project_clusters_enabled": 999, + ... + }, + "create": { + "merge_requests": 999, + ... + }, + "manage": { + "events": 999, + ... + }, + "monitor": { + "clusters": 999, + ... + }, + "package": { + "projects_with_packages": 999 + }, + "plan": { + "issues": 999, + ... + }, + "release": { + "deployments": 999, + ... + }, + "secure": { + "user_container_scanning_jobs": 999, + ... + }, + "verify": { + "ci_builds": 999, + ... + } + }, + "topology": { + "nodes": [ + { + "node_memory_total_bytes": 33269903360, + "node_cpus": 16, + "node_services": [ + { + "name": "web", + "process_count": 16, + "process_memory_pss": 233349888, + "process_memory_rss": 788220927, + "process_memory_uss": 195295487 + }, + { + "name": "sidekiq", + "process_count": 1, + "process_memory_pss": 734080000, + "process_memory_rss": 750051328, + "process_memory_uss": 731533312 + }, + ... + ], + ... + }, + ... + ], + "duration_s": 0.013836685999194742 + } +} +``` |