diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2021-04-20 23:50:22 +0000 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2021-04-20 23:50:22 +0000 |
| commit | 9dc93a4519d9d5d7be48ff274127136236a3adb3 (patch) | |
| tree | 70467ae3692a0e35e5ea56bcb803eb512a10bedb /doc/development | |
| parent | 4b0f34b6d759d6299322b3a54453e930c6121ff0 (diff) | |
| download | gitlab-ce-9dc93a4519d9d5d7be48ff274127136236a3adb3.tar.gz | |
Add latest changes from gitlab-org/gitlab@13-11-stable-eev13.11.0-rc43
Diffstat (limited to 'doc/development')
106 files changed, 4964 insertions, 3733 deletions
diff --git a/doc/development/README.md b/doc/development/README.md index 43ceb737dde..3b6eb068c13 100644 --- a/doc/development/README.md +++ b/doc/development/README.md @@ -176,6 +176,7 @@ In these cases, use the following workflow: - [Working with the GitHub importer](github_importer.md) - [Import/Export development documentation](import_export.md) - [Test Import Project](import_project.md) +- [Group migration](bulk_import.md) - [Elasticsearch integration docs](elasticsearch.md) - [Working with Merge Request diffs](diffs.md) - [Kubernetes integration guidelines](kubernetes.md) @@ -262,7 +263,7 @@ See [database guidelines](database/index.md). - [Product Intelligence guide](https://about.gitlab.com/handbook/product/product-intelligence-guide/) - [Usage Ping guide](usage_ping/index.md) -- [Snowplow guide](snowplow.md) +- [Snowplow guide](snowplow/index.md) ## Experiment guide diff --git a/doc/development/agent/identity.md b/doc/development/agent/identity.md index 49d20d2fd87..83af4318de9 100644 --- a/doc/development/agent/identity.md +++ b/doc/development/agent/identity.md @@ -92,3 +92,15 @@ GitLab provides the following information in its response for a given Agent acce - Agent configuration Git repository. (The agent doesn't support per-folder authorization.) - Agent name. + +## Create an agent + +You can create an agent by following the [user documentation](../../user/clusters/agent/index.md#create-an-agent-record-in-gitlab), or via Rails console: + +```ruby +project = ::Project.find_by_full_path("path-to/your-configuration-project") +# agent-name should be the same as specified above in the config.yaml +agent = ::Clusters::Agent.create(name: "<agent-name>", project: project) +token = ::Clusters::AgentToken.create(agent: agent) +token.token # this will print out the token you need to use on the next step +``` diff --git a/doc/development/agent/local.md b/doc/development/agent/local.md index 603364567bf..670315db3a8 100644 --- a/doc/development/agent/local.md +++ b/doc/development/agent/local.md @@ -98,3 +98,58 @@ bazel test //internal/module/gitops/server:server_test - Bazel documentation about [specifying targets to build](https://docs.bazel.build/versions/master/guide.html#specifying-targets-to-build). - [The Bazel query](https://docs.bazel.build/versions/master/query.html) - [Bazel query how to](https://docs.bazel.build/versions/master/query-how-to.html) + +## KAS QA tests + +This section describes how to run KAS tests against different GitLab environments based on the +[GitLab QA orchestrator](https://gitlab.com/gitlab-org/gitlab-qa). + +### Status + +The `kas` QA tests currently have some limitations. You can run them manually on GDK, but they don't +run automatically with the nightly jobs against the live environment. See the section below +to learn how to run them against different environments. + +### Prepare + +Before performing any of these tests, if you have a `k3s` instance running, make sure to +stop it manually before running them. Otherwise, the tests might fail with the message +`failed to remove k3s cluster`. + +You might need to specify the correct Agent image version that matches the `kas` image version. You can use the `GITLAB_AGENTK_VERSION` local env for this. + +### Against `staging` + +1. Go to your local `qa/qa/service/cluster_provider/k3s.rb` and comment out + [this line](https://gitlab.com/gitlab-org/gitlab/-/blob/5b15540ea78298a106150c3a1d6ed26416109b9d/qa/qa/service/cluster_provider/k3s.rb#L8) and + [this line](https://gitlab.com/gitlab-org/gitlab/-/blob/5b15540ea78298a106150c3a1d6ed26416109b9d/qa/qa/service/cluster_provider/k3s.rb#L36). + We don't allow local connections on `staging` as they require an admin user. +1. Ensure you don't have an `EE_LICENSE` env var set as this would force an admin login. +1. Go to your GDK root folder and `cd gitlab/qa`. +1. Login with your user in staging and create a group to be used as sandbox. + Something like: `username-qa-sandbox`. +1. Create an access token for your user with the `api` permission. +1. Replace the values given below with your own and run: + + ```shell + GITLAB_SANDBOX_NAME="<THE GROUP ID YOU CREATED ON STEP 2>" \ + GITLAB_QA_ACCESS_TOKEN="<THE ACCESS TOKEN YOU CREATED ON STEP 3>" \ + GITLAB_USERNAME="<YOUR STAGING USERNAME>" \ + GITLAB_PASSWORD="<YOUR STAGING PASSWORD>" \ + bundle exec bin/qa Test::Instance::All https://staging.gitlab.com -- --tag quarantine qa/specs/features/ee/api/7_configure/kubernetes/kubernetes_agent_spec.rb + ``` + +### Against GDK + +1. Go to your `qa/qa/fixtures/kubernetes_agent/agentk-manifest.yaml.erb` and comment out [this line](https://gitlab.com/gitlab-org/gitlab/-/blob/a55b78532cfd29426cf4e5b4edda81407da9d449/qa/qa/fixtures/kubernetes_agent/agentk-manifest.yaml.erb#L27) and uncomment [this line](https://gitlab.com/gitlab-org/gitlab/-/blob/a55b78532cfd29426cf4e5b4edda81407da9d449/qa/qa/fixtures/kubernetes_agent/agentk-manifest.yaml.erb#L28). + GDK's `kas` listens on `grpc`, not on `wss`. +1. Go to the GDK's root folder and `cd gitlab/qa`. +1. On the contrary to staging, run the QA test in GDK as admin, which is the default choice. To do so, use the default sandbox group and run the command below. Make sure to adjust your credentials if necessary, otherwise, the test might fail: + + ```shell + GITLAB_USERNAME=root \ + GITLAB_PASSWORD="5iveL\!fe" \ + GITLAB_ADMIN_USERNAME=root \ + GITLAB_ADMIN_PASSWORD="5iveL\!fe" \ + bundle exec bin/qa Test::Instance::All http://gdk.test:3000 -- --tag quarantine qa/specs/features/ee/api/7_configure/kubernetes/kubernetes_agent_spec.rb + ``` diff --git a/doc/development/api_graphql_styleguide.md b/doc/development/api_graphql_styleguide.md index 8bac02c99af..6256610ae6a 100644 --- a/doc/development/api_graphql_styleguide.md +++ b/doc/development/api_graphql_styleguide.md @@ -392,6 +392,28 @@ field :blob, type: Types::Snippets::BlobType, This will increment the [`complexity` score](#field-complexity) of the field by `1`. +If a resolver calls Gitaly, it can be annotated with +`BaseResolver.calls_gitaly!`. This passes `calls_gitaly: true` to any +field that uses this resolver. + +For example: + +```ruby +class BranchResolver < BaseResolver + type ::Types::BranchType, null: true + calls_gitaly! + + argument name: ::GraphQL::STRING_TYPE, required: true + + def resolve(name:) + object.branch(name) + end +end +``` + +Then when we use it, any field that uses `BranchResolver` has the correct +value for `calls_gitaly:`. + ### Exposing permissions for a type To expose permissions the current user has on a resource, you can call @@ -750,113 +772,7 @@ argument :title, GraphQL::STRING_TYPE, ## Authorization -Authorizations can be applied to both types and fields using the same -abilities as in the Rails app. - -If the: - -- Currently authenticated user fails the authorization, the authorized - resource is returned as `null`. -- Resource is part of a collection, the collection is filtered to - exclude the objects that the user's authorization checks failed against. - -Also see [authorizing resources in a mutation](#authorizing-resources). - -NOTE: -Try to load only what the currently authenticated user is allowed to -view with our existing finders first, without relying on authorization -to filter the records. This minimizes database queries and unnecessary -authorization checks of the loaded records. - -### Type authorization - -Authorize a type by passing an ability to the `authorize` method. All -fields with the same type is authorized by checking that the -currently authenticated user has the required ability. - -For example, the following authorization ensures that the currently -authenticated user can only see projects that they have the -`read_project` ability for (so long as the project is returned in a -field that uses `Types::ProjectType`): - -```ruby -module Types - class ProjectType < BaseObject - authorize :read_project - end -end -``` - -You can also authorize against multiple abilities, in which case all of -the ability checks must pass. - -For example, the following authorization ensures that the currently -authenticated user must have `read_project` and `another_ability` -abilities to see a project: - -```ruby -module Types - class ProjectType < BaseObject - authorize [:read_project, :another_ability] - end -end -``` - -### Field authorization - -Fields can be authorized with the `authorize` option. - -For example, the following authorization ensures that the currently -authenticated user must have the `owner_access` ability to see the -project: - -```ruby -module Types - class MyType < BaseObject - field :project, Types::ProjectType, null: true, resolver: Resolvers::ProjectResolver, authorize: :owner_access - end -end -``` - -Fields can also be authorized against multiple abilities, in which case -all of ability checks must pass. This requires explicitly -passing a block to `field`: - -```ruby -module Types - class MyType < BaseObject - field :project, Types::ProjectType, null: true, resolver: Resolvers::ProjectResolver do - authorize [:owner_access, :another_ability] - end - end -end -``` - -If the field's type already [has a particular -authorization](#type-authorization) then there is no need to add that -same authorization to the field. - -### Type and Field authorizations together - -Authorizations are cumulative, so where authorizations are defined on -a field, and also on the field's type, then the currently authenticated -user would need to pass all ability checks. - -In the following simplified example the currently authenticated user -would need both `first_permission` and `second_permission` abilities in -order to see the author of the issue. - -```ruby -class UserType - authorize :first_permission -end -``` - -```ruby -class IssueType - field :author, UserType, authorize: :second_permission -end -``` +See: [GraphQL Authorization](graphql_guide/authorization.md) ## Resolvers @@ -1098,6 +1014,26 @@ class MyThingResolver < BaseResolver end ``` +By default, fields defined in `#preloads` will be preloaded if that field +is selected in the query. Occasionally, finer control may be +needed to avoid preloading too much or incorrect content. + +Extending the above example, we might want to preload a different +association if certain fields are requested together. This can +be done by overriding `#filtered_preloads`: + +```ruby +class MyThingResolver < BaseResolver + # ... + + def filtered_preloads + return [:alternate_attribute] if lookahead.selects?(:field_one) && lookahead.selects?(:field_two) + + super + end +end +``` + The final thing that is needed is that every field that uses this resolver needs to advertise the need for lookahead: @@ -1137,9 +1073,10 @@ When using resolvers, they can and should serve as the SSoT for field metadata. All field options (apart from the field name) can be declared on the resolver. These include: -- `type` (this is particularly important, and is planned to be mandatory) +- `type` (required - all resolvers must include a type annotation) - `extras` - `description` +- Gitaly annotations (with `calls_gitaly!`) Example: @@ -1149,6 +1086,7 @@ module Resolvers type Types::MyType, null: true extras [:lookahead] description 'Retrieve a single MyType' + calls_gitaly! end end ``` diff --git a/doc/development/application_limits.md b/doc/development/application_limits.md index c42e9224105..3c1c91e0d2e 100644 --- a/doc/development/application_limits.md +++ b/doc/development/application_limits.md @@ -43,8 +43,6 @@ It's recommended to create two separate migration script files. class InsertProjectHooksPlanLimits < ActiveRecord::Migration[5.2] include Gitlab::Database::MigrationHelpers - DOWNTIME = false - def up create_or_update_plan_limit('project_hooks', 'default', 0) create_or_update_plan_limit('project_hooks', 'free', 10) diff --git a/doc/development/application_secrets.md b/doc/development/application_secrets.md index 92b18f5ad78..06a38eb238a 100644 --- a/doc/development/application_secrets.md +++ b/doc/development/application_secrets.md @@ -15,7 +15,7 @@ This page is a development guide for application secrets. |`secret_key_base` | The base key to be used for generating a various secrets | | `otp_key_base` | The base key for One Time Passwords, described in [User management](../raketasks/user_management.md#rotate-two-factor-authentication-encryption-key) | |`db_key_base` | The base key to encrypt the data for `attr_encrypted` columns | -|`openid_connect_signing_key` | The singing key for OpenID Connect | +|`openid_connect_signing_key` | The signing key for OpenID Connect | | `encrypted_settings_key_base` | The base key to encrypt settings files with | ## Where the secrets are stored diff --git a/doc/development/avoiding_downtime_in_migrations.md b/doc/development/avoiding_downtime_in_migrations.md new file mode 100644 index 00000000000..d8981ce0999 --- /dev/null +++ b/doc/development/avoiding_downtime_in_migrations.md @@ -0,0 +1,415 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Avoiding downtime in migrations + +When working with a database certain operations may require downtime. Since we +cannot have downtime in migrations we need to use a set of steps to get the +same end result without downtime. This guide describes various operations that +may appear to need downtime, their impact, and how to perform them without +requiring downtime. + +## Dropping Columns + +Removing columns is tricky because running GitLab processes may still be using +the columns. To work around this safely, you will need three steps in three releases: + +1. Ignoring the column (release M) +1. Dropping the column (release M+1) +1. Removing the ignore rule (release M+2) + +The reason we spread this out across three releases is that dropping a column is +a destructive operation that can't be rolled back easily. + +Following this procedure helps us to make sure there are no deployments to GitLab.com +and upgrade processes for self-managed installations that lump together any of these steps. + +### Step 1: Ignoring the column (release M) + +The first step is to ignore the column in the application code. This is +necessary because Rails caches the columns and re-uses this cache in various +places. This can be done by defining the columns to ignore. For example, to ignore +`updated_at` in the User model you'd use the following: + +```ruby +class User < ApplicationRecord + include IgnorableColumns + ignore_column :updated_at, remove_with: '12.7', remove_after: '2020-01-22' +end +``` + +Multiple columns can be ignored, too: + +```ruby +ignore_columns %i[updated_at created_at], remove_with: '12.7', remove_after: '2020-01-22' +``` + +We require indication of when it is safe to remove the column ignore with: + +- `remove_with`: set to a GitLab release typically two releases (M+2) after adding the + column ignore. +- `remove_after`: set to a date after which we consider it safe to remove the column + ignore, typically after the M+1 release date, during the M+2 development cycle. + +This information allows us to reason better about column ignores and makes sure we +don't remove column ignores too early for both regular releases and deployments to GitLab.com. For +example, this avoids a situation where we deploy a bulk of changes that include both changes +to ignore the column and subsequently remove the column ignore (which would result in a downtime). + +In this example, the change to ignore the column went into release 12.5. + +### Step 2: Dropping the column (release M+1) + +Continuing our example, dropping the column goes into a _post-deployment_ migration in release 12.6: + +```ruby + remove_column :user, :updated_at +``` + +### Step 3: Removing the ignore rule (release M+2) + +With the next release, in this example 12.7, we set up another merge request to remove the ignore rule. +This removes the `ignore_column` line and - if not needed anymore - also the inclusion of `IgnoreableColumns`. + +This should only get merged with the release indicated with `remove_with` and once +the `remove_after` date has passed. + +## Renaming Columns + +Renaming columns the normal way requires downtime as an application may continue +using the old column name during/after a database migration. To rename a column +without requiring downtime we need two migrations: a regular migration, and a +post-deployment migration. Both these migration can go in the same release. + +### Step 1: Add The Regular Migration + +First we need to create the regular migration. This migration should use +`Gitlab::Database::MigrationHelpers#rename_column_concurrently` to perform the +renaming. For example + +```ruby +# A regular migration in db/migrate +class RenameUsersUpdatedAtToUpdatedAtTimestamp < ActiveRecord::Migration[4.2] + include Gitlab::Database::MigrationHelpers + + DOWNTIME = false + + disable_ddl_transaction! + + def up + rename_column_concurrently :users, :updated_at, :updated_at_timestamp + end + + def down + undo_rename_column_concurrently :users, :updated_at, :updated_at_timestamp + end +end +``` + +This will take care of renaming the column, ensuring data stays in sync, and +copying over indexes and foreign keys. + +If a column contains one or more indexes that don't contain the name of the +original column, the previously described procedure will fail. In that case, +you'll first need to rename these indexes. + +### Step 2: Add A Post-Deployment Migration + +The renaming procedure requires some cleaning up in a post-deployment migration. +We can perform this cleanup using +`Gitlab::Database::MigrationHelpers#cleanup_concurrent_column_rename`: + +```ruby +# A post-deployment migration in db/post_migrate +class CleanupUsersUpdatedAtRename < ActiveRecord::Migration[4.2] + include Gitlab::Database::MigrationHelpers + + disable_ddl_transaction! + + def up + cleanup_concurrent_column_rename :users, :updated_at, :updated_at_timestamp + end + + def down + undo_cleanup_concurrent_column_rename :users, :updated_at, :updated_at_timestamp + end +end +``` + +If you're renaming a [large table](https://gitlab.com/gitlab-org/gitlab/-/blob/master/rubocop/rubocop-migrations.yml#L3), please carefully consider the state when the first migration has run but the second cleanup migration hasn't been run yet. +With [Canary](https://gitlab.com/gitlab-com/gl-infra/readiness/-/tree/master/library/canary/) it is possible that the system runs in this state for a significant amount of time. + +## Changing Column Constraints + +Adding or removing a `NOT NULL` clause (or another constraint) can typically be +done without requiring downtime. However, this does require that any application +changes are deployed _first_. Thus, changing the constraints of a column should +happen in a post-deployment migration. + +Avoid using `change_column` as it produces an inefficient query because it re-defines +the whole column type. + +You can check the following guides for each specific use case: + +- [Adding foreign-key constraints](migration_style_guide.md#adding-foreign-key-constraints) +- [Adding `NOT NULL` constraints](database/not_null_constraints.md) +- [Adding limits to text columns](database/strings_and_the_text_data_type.md) + +## Changing Column Types + +Changing the type of a column can be done using +`Gitlab::Database::MigrationHelpers#change_column_type_concurrently`. This +method works similarly to `rename_column_concurrently`. For example, let's say +we want to change the type of `users.username` from `string` to `text`. + +### Step 1: Create A Regular Migration + +A regular migration is used to create a new column with a temporary name along +with setting up some triggers to keep data in sync. Such a migration would look +as follows: + +```ruby +# A regular migration in db/migrate +class ChangeUsersUsernameStringToText < ActiveRecord::Migration[4.2] + include Gitlab::Database::MigrationHelpers + + disable_ddl_transaction! + + def up + change_column_type_concurrently :users, :username, :text + end + + def down + undo_change_column_type_concurrently :users, :username + end +end +``` + +### Step 2: Create A Post Deployment Migration + +Next we need to clean up our changes using a post-deployment migration: + +```ruby +# A post-deployment migration in db/post_migrate +class ChangeUsersUsernameStringToTextCleanup < ActiveRecord::Migration[4.2] + include Gitlab::Database::MigrationHelpers + + disable_ddl_transaction! + + def up + cleanup_concurrent_column_type_change :users, :username + end + + def down + undo_cleanup_concurrent_column_type_change :users, :username, :string + end +end +``` + +And that's it, we're done! + +### Casting data to a new type + +Some type changes require casting data to a new type. For example when changing from `text` to `jsonb`. +In this case, use the `type_cast_function` option. +Make sure there is no bad data and the cast will always succeed. You can also provide a custom function that handles +casting errors. + +Example migration: + +```ruby + def up + change_column_type_concurrently :users, :settings, :jsonb, type_cast_function: 'jsonb' + end +``` + +## Changing The Schema For Large Tables + +While `change_column_type_concurrently` and `rename_column_concurrently` can be +used for changing the schema of a table without downtime, it doesn't work very +well for large tables. Because all of the work happens in sequence the migration +can take a very long time to complete, preventing a deployment from proceeding. +They can also produce a lot of pressure on the database due to it rapidly +updating many rows in sequence. + +To reduce database pressure you should instead use +`change_column_type_using_background_migration` or `rename_column_using_background_migration` +when migrating a column in a large table (e.g. `issues`). These methods work +similarly to the concurrent counterparts but uses background migration to spread +the work / load over a longer time period, without slowing down deployments. + +For example, to change the column type using a background migration: + +```ruby +class ExampleMigration < ActiveRecord::Migration[4.2] + include Gitlab::Database::MigrationHelpers + + disable_ddl_transaction! + + class Issue < ActiveRecord::Base + self.table_name = 'issues' + + include EachBatch + + def self.to_migrate + where('closed_at IS NOT NULL') + end + end + + def up + change_column_type_using_background_migration( + Issue.to_migrate, + :closed_at, + :datetime_with_timezone + ) + end + + def down + change_column_type_using_background_migration( + Issue.to_migrate, + :closed_at, + :datetime + ) + end +end +``` + +This would change the type of `issues.closed_at` to `timestamp with time zone`. + +Keep in mind that the relation passed to +`change_column_type_using_background_migration` _must_ include `EachBatch`, +otherwise it will raise a `TypeError`. + +This migration then needs to be followed in a separate release (_not_ a patch +release) by a cleanup migration, which should steal from the queue and handle +any remaining rows. For example: + +```ruby +class MigrateRemainingIssuesClosedAt < ActiveRecord::Migration[4.2] + include Gitlab::Database::MigrationHelpers + + DOWNTIME = false + + disable_ddl_transaction! + + class Issue < ActiveRecord::Base + self.table_name = 'issues' + include EachBatch + end + + def up + Gitlab::BackgroundMigration.steal('CopyColumn') + Gitlab::BackgroundMigration.steal('CleanupConcurrentTypeChange') + + migrate_remaining_rows if migrate_column_type? + end + + def down + # Previous migrations already revert the changes made here. + end + + def migrate_remaining_rows + Issue.where('closed_at_for_type_change IS NULL AND closed_at IS NOT NULL').each_batch do |batch| + batch.update_all('closed_at_for_type_change = closed_at') + end + + cleanup_concurrent_column_type_change(:issues, :closed_at) + end + + def migrate_column_type? + # Some environments may have already executed the previous version of this + # migration, thus we don't need to migrate those environments again. + column_for('issues', 'closed_at').type == :datetime # rubocop:disable Migration/Datetime + end +end +``` + +The same applies to `rename_column_using_background_migration`: + +1. Create a migration using the helper, which will schedule background + migrations to spread the writes over a longer period of time. +1. In the next monthly release, create a clean-up migration to steal from the + Sidekiq queues, migrate any missing rows, and cleanup the rename. This + migration should skip the steps after stealing from the Sidekiq queues if the + column has already been renamed. + +For more information, see [the documentation on cleaning up background +migrations](background_migrations.md#cleaning-up). + +## Adding Indexes + +Adding indexes does not require downtime when `add_concurrent_index` +is used. + +See also [Migration Style Guide](migration_style_guide.md#adding-indexes) +for more information. + +## Dropping Indexes + +Dropping an index does not require downtime. + +## Adding Tables + +This operation is safe as there's no code using the table just yet. + +## Dropping Tables + +Dropping tables can be done safely using a post-deployment migration, but only +if the application no longer uses the table. + +## Renaming Tables + +Renaming tables requires downtime as an application may continue +using the old table name during/after a database migration. + +## Adding Foreign Keys + +Adding foreign keys usually works in 3 steps: + +1. Start a transaction +1. Run `ALTER TABLE` to add the constraint(s) +1. Check all existing data + +Because `ALTER TABLE` typically acquires an exclusive lock until the end of a +transaction this means this approach would require downtime. + +GitLab allows you to work around this by using +`Gitlab::Database::MigrationHelpers#add_concurrent_foreign_key`. This method +ensures that no downtime is needed. + +## Removing Foreign Keys + +This operation does not require downtime. + +## Data Migrations + +Data migrations can be tricky. The usual approach to migrate data is to take a 3 +step approach: + +1. Migrate the initial batch of data +1. Deploy the application code +1. Migrate any remaining data + +Usually this works, but not always. For example, if a field's format is to be +changed from JSON to something else we have a bit of a problem. If we were to +change existing data before deploying application code we'll most likely run +into errors. On the other hand, if we were to migrate after deploying the +application code we could run into the same problems. + +If you merely need to correct some invalid data, then a post-deployment +migration is usually enough. If you need to change the format of data (e.g. from +JSON to something else) it's typically best to add a new column for the new data +format, and have the application use that. In such a case the procedure would +be: + +1. Add a new column in the new format +1. Copy over existing data to this new column +1. Deploy the application code +1. In a post-deployment migration, copy over any remaining data + +In general there is no one-size-fits-all solution, therefore it's best to +discuss these kind of migrations in a merge request to make sure they are +implemented in the best way possible. diff --git a/doc/development/background_migrations.md b/doc/development/background_migrations.md index 3ef5bf382b8..a96606719d0 100644 --- a/doc/development/background_migrations.md +++ b/doc/development/background_migrations.md @@ -142,6 +142,14 @@ migration performing the scheduling. Otherwise the background migration would be scheduled multiple times on systems that are upgrading multiple patch releases at once. +When you start the second post-deployment migration, you should delete any +previously queued jobs from the initial migration with the provided +helper: + +```ruby +delete_queued_jobs('BackgroundMigrationClassName') +``` + ## Cleaning Up NOTE: diff --git a/doc/development/bulk_import.md b/doc/development/bulk_import.md index 40e4af923ea..e70880635e6 100644 --- a/doc/development/bulk_import.md +++ b/doc/development/bulk_import.md @@ -42,7 +42,7 @@ step to generate the file to be imported. GitLab Group migration leverages on [GitLab API](../api/README.md) to speed the migration. -And, because we're on the road to [GraphQL](../api/README.md#road-to-graphql), +And, because we're on the road to [GraphQL](../api/README.md#graphql-api), GitLab Group Migration will be contributing towards to expand the GraphQL API coverage, which benefits both GitLab and its users. diff --git a/doc/development/changelog.md b/doc/development/changelog.md index 98a3e75bb3c..ee80d998c14 100644 --- a/doc/development/changelog.md +++ b/doc/development/changelog.md @@ -46,18 +46,18 @@ the `author` field. GitLab team members **should not**. and with `type` set to `security`. - Any user-facing change **must** have a changelog entry. This includes both visual changes (regardless of how minor), and changes to the rendered DOM which impact how a screen reader may announce the content. - Any client-facing change to our REST and GraphQL APIs **must** have a changelog entry. See the [complete list what comprises a GraphQL breaking change](api_graphql_styleguide.md#breaking-changes). +- Any change that introduces an [Advanced Search migration](elasticsearch.md#creating-a-new-advanced-search-migration) **must** have a changelog entry. - Performance improvements **should** have a changelog entry. -- Changes that need to be documented in the Product Intelligence [Event Dictionary](https://about.gitlab.com/handbook/product/product-intelligence-guide/#event-dictionary) also require a changelog entry. - _Any_ contribution from a community member, no matter how small, **may** have a changelog entry regardless of these guidelines if the contributor wants one. Example: "Fixed a typo on the search results page." - Any docs-only changes **should not** have a changelog entry. -- Any change behind a disabled feature flag **should not** have a changelog entry. -- Any change behind an enabled feature flag **should** have a changelog entry. -- Any change that adds new usage data metrics and changes that needs to be documented in Product Intelligence [Event Dictionary](https://about.gitlab.com/handbook/product/product-intelligence-guide/#event-dictionary) **should** have a changelog entry. +- Any change behind a feature flag **disabled** by default **should not** have a changelog entry. +- Any change behind a feature flag that is **enabled** by default **should** have a changelog entry. +- Any change that adds new Usage Data metrics and changes that needs to be documented in Product Intelligence [Metrics Dictionary](usage_ping/dictionary.md) **should** have a changelog entry. - A change that adds snowplow events **should** have a changelog entry - -- A change that [removes a feature flag](feature_flags/index.md) **must** have a changelog entry. +- A change that [removes a feature flag, or removes a feature and its feature flag](feature_flags/index.md) **must** have a changelog entry. - A fix for a regression introduced and then fixed in the same release (i.e., fixing a bug introduced during a monthly release candidate) **should not** have a changelog entry. diff --git a/doc/development/chaos_endpoints.md b/doc/development/chaos_endpoints.md index 85c93f521ac..56e91acbc4a 100644 --- a/doc/development/chaos_endpoints.md +++ b/doc/development/chaos_endpoints.md @@ -146,10 +146,10 @@ curl "http://localhost:3000/-/chaos/sleep?duration_s=60&token=secret" ## Kill -This endpoint simulates the unexpected death of a worker process using a `kill` signal. +This endpoint simulates the unexpected death of a worker process using the `KILL` signal. -Because this endpoint uses the `KILL` signal, the worker isn't given an -opportunity to cleanup or shutdown. +Because this endpoint uses the `KILL` signal, the process isn't given an +opportunity to clean up or shut down. ```plaintext GET /-/chaos/kill @@ -158,13 +158,33 @@ GET /-/chaos/kill?async=true | Attribute | Type | Required | Description | | ------------ | ------- | -------- | ---------------------------------------------------------------------- | -| `async` | boolean | no | Set to true to kill a Sidekiq background worker process | +| `async` | boolean | no | Set to true to signal a Sidekiq background worker process | ```shell curl "http://localhost:3000/-/chaos/kill" --header 'X-Chaos-Secret: secret' curl "http://localhost:3000/-/chaos/kill?token=secret" ``` +## Quit + +This endpoint simulates the unexpected death of a worker process using the `QUIT` signal. +Unlike `KILL`, the `QUIT` signal will also attempt to write a core dump. +See [core(5)](https://man7.org/linux/man-pages/man5/core.5.html) for more information. + +```plaintext +GET /-/chaos/quit +GET /-/chaos/quit?async=true +``` + +| Attribute | Type | Required | Description | +| ------------ | ------- | -------- | ---------------------------------------------------------------------- | +| `async` | boolean | no | Set to true to signal a Sidekiq background worker process | + +```shell +curl "http://localhost:3000/-/chaos/quit" --header 'X-Chaos-Secret: secret' +curl "http://localhost:3000/-/chaos/quit?token=secret" +``` + ## Run garbage collector This endpoint triggers a GC run on the worker handling the request and returns its worker ID diff --git a/doc/development/cicd/index.md b/doc/development/cicd/index.md index eb2224d710a..35c4cc0694e 100644 --- a/doc/development/cicd/index.md +++ b/doc/development/cicd/index.md @@ -23,7 +23,7 @@ On the left side we have the events that can trigger a pipeline based on various - A `git push` is the most common event that triggers a pipeline. - The [Web API](../../api/pipelines.md#create-a-new-pipeline). -- A user clicking the "Run Pipeline" button in the UI. +- A user clicking the "Run pipeline" button in the UI. - When a [merge request is created or updated](../../ci/merge_request_pipelines/index.md#pipelines-for-merge-requests). - When an MR is added to a [Merge Train](../../ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md#merge-trains). - A [scheduled pipeline](../../ci/pipelines/schedules.md#pipeline-schedules). @@ -182,3 +182,17 @@ Watch a walkthrough of this feature in details in the video below. <figure class="video-container"> <iframe src="https://www.youtube.com/embed/NmdWRGT8kZg" frameborder="0" allowfullscreen="true"> </iframe> </figure> + +## External pipeline validation service + +The [external CI/CD pipeline validation service](../../administration/external_pipeline_validation.md) +is available for use on self-managed GitLab instances, but is not in use on GitLab.com. +It is configured with [environment variables](../../administration/environment_variables.md) +on the instance. + +To enable the feature on GitLab.com, enable the `ci_external_validation_service` +[feature flag](../feature_flags/index.md). The valid "Not accepted" response code +for GitLab.com is `406` only. + +For more details, see the linked issues and MRs in the +[feature flag rollout issue](https://gitlab.com/gitlab-org/gitlab/-/issues/325982). diff --git a/doc/development/cicd/templates.md b/doc/development/cicd/templates.md index ed0d217c247..fd1456a1676 100644 --- a/doc/development/cicd/templates.md +++ b/doc/development/cicd/templates.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Verify +group: Pipeline Authoring info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: index, concepts, howto --- @@ -9,27 +9,214 @@ type: index, concepts, howto This document explains how to develop [GitLab CI/CD templates](../../ci/examples/README.md). -## Place the template file in a relevant directory +## Requirements for CI/CD templates + +Before submitting a merge request with a new or updated CI/CD template, you must: + +- Place the template in the correct [directory](#template-directories). +- Follow the [CI/CD template authoring guidelines](#template-authoring-guidelines). +- Name the template following the `*.gitlab-ci.yml` format. +- Use valid [`.gitlab-ci.yml` syntax](../../ci/yaml/README.md). Verify it's valid + with the [CI/CD lint tool](../../ci/lint.md). +- Include [a changelog](../changelog.md) if the merge request introduces a user-facing change. +- Follow the [template review process](#contribute-cicd-template-merge-requests). +- (Optional but highly recommended) Test the template in an example GitLab project + that reviewers can access. Reviewers might not be able to create the data or configuration + that the template requires, so an example project helps the reviewers ensure the + template is correct. The example project pipeline should succeed before submitting + the merge request for review. + +## Template directories + +All template files are saved in `lib/gitlab/ci/templates`. Save general templates +in this directory, but certain template types have a specific directory reserved for +them. The ability to [select a template in new file UI](#make-sure-the-new-template-can-be-selected-in-ui) +is determined by the directory it is in: + +| Sub-directory | Selectable in UI | Template type | +|----------------|------------------|---------------| +| `/*` (root) | Yes | General templates. | +| `/AWS/*` | No | Templates related to Cloud Deployment (AWS). | +| `/Jobs/*` | No | Templates related to Auto DevOps. | +| `/Pages/*` | Yes | Sample templates for using Static site generators with GitLab Pages. | +| `/Security/*` | Yes | Templates related to Security scanners. | +| `/Terraform/*` | No | Templates related to infrastructure as Code (Terraform). | +| `/Verify/*` | Yes | Templates related to Testing features. | +| `/Workflows/*` | No | Sample templates for using the `workflow:` keyword. | + +## Template authoring guidelines + +Use the following guidelines to ensure your template submission follows standards: + +### Template types + +Templates have two different types that impact the way the template should be written +and used. The style in a template should match one of these two types: + +A **pipeline template** provides an end-to-end CI/CD workflow that matches a project's +structure, language, and so on. It usually should be used by itself in projects that +don't have any other `.gitlab-ci.yml` files. + +When authoring pipeline templates: + +- Place any [global keywords](../../ci/yaml/README.md#global-keywords) like `image` + or `before_script` in a [`default`](../../ci/yaml/README.md#custom-default-keyword-values) + section at the top of the template. +- Note clearly in the [code comments](#explain-the-template-with-comments) if the + template is designed to be used with the `includes` keyword in an existing + `.gitlab-ci.yml` file or not. + +A **job template** provides specific jobs that can be added to an existing CI/CD +workflow to accomplish specific tasks. It usually should be used by adding it to +an existing `.gitlab-ci.yml` file by using the [`includes`](../../ci/yaml/README.md#global-keywords) +keyword. You can also copy and paste the contents into an existing `.gitlab-ci.yml` file. + +Configure job templates so that users can add them to their current pipeline with very +few or no modifications. It must be configured to reduce the risk of conflicting with +other pipeline configuration. + +When authoring job templates: + +- Do not use [global](../../ci/yaml/README.md#global-keywords) or [`default`](../../ci/yaml/README.md#custom-default-keyword-values) + keywords. When a root `.gitlab-ci.yml` includes a template, global or default keywords + might be overridden and cause unexpected behavior. If a job template requires a + specific stage, explain in the code comments that users must manually add the stage + to the main `.gitlab-ci.yml` configuration. +- Note clearly in [code comments](#explain-the-template-with-comments) that the template + is designed to be used with the `includes` keyword or copied into an existing configuration. +- Consider [versioning](#versioning) the template with latest and stable versions + to avoid [backward compatibility](#backward-compatibility) problems. + Maintenance of this type of template is more complex, because changes to templates + imported with `includes` can break pipelines for all projects using the template. + +Additional points to keep in mind when authoring templates: + +| Template design points | Pipeline templates | Job templates | +|------------------------------------------------------|--------------------|---------------| +| Can use global keywords, including `stages`. | Yes | No | +| Can define jobs. | Yes | Yes | +| Can be selected in the new file UI | Yes | Yes | +| Can include other job templates with `include` | Yes | No | +| Can include other pipeline templates with `include`. | No | No | + +### Syntax guidelines + +To make templates easier to follow, templates should all use clear syntax styles, +with a consistent format. + +#### Do not hardcode the default branch + +Use [`$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH`](../../ci/variables/predefined_variables.md) +instead of a hardcoded `main` branch, and never use `master`: -All template files reside in the `lib/gitlab/ci/templates` directory, and are categorized by the following sub-directories: +```yaml +job1: + rules: + if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + script: + echo "example job 1" + +job2: + only: + variables: + - $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + script: + echo "example job 2" + +``` + +#### Use `rules` instead of `only` or `except` + +Avoid using [`only` or `except`](../../ci/yaml/README.md#onlyexcept-basic) if possible. +Only and except is not being developed any more, and [`rules`](../../ci/yaml/README.md#rules) +is now the preferred syntax: + +```yaml +job2: + script: + - echo + rules: + - if: $CI_COMMIT_BRANCH +``` + +#### Break up long commands + +If a command is very long, or has many command line flags, like `-o` or `--option`: + +- Split these up into a multi-line command to make it easier to see every part of the command. +- Use the long name for the flags, when available. + +For example, with a long command with short CLI flags like +`docker run --e SOURCE_CODE="$PWD" -v "$PWD":/code -v /var/run/docker.sock:/var/run/docker.sock "$CODE_QUALITY_IMAGE" /code`: + +```yaml +job1: + script: + - docker run + --env SOURCE_CODE="$PWD" + --volume "$PWD":/code + --volume /var/run/docker.sock:/var/run/docker.sock + "$CODE_QUALITY_IMAGE" /code +``` -| Sub-directory | Content | [Selectable in UI](#make-sure-the-new-template-can-be-selected-in-ui) | -|----------------|----------------------------------------------------|-----------------------------------------------------------------------| -| `/AWS/*` | Cloud Deployment (AWS) related jobs | No | -| `/Jobs/*` | Auto DevOps related jobs | No | -| `/Pages/*` | Static site generators for GitLab Pages (for example Jekyll) | Yes | -| `/Security/*` | Security related jobs | Yes | -| `/Terraform/*` | Infrastructure as Code related templates | No | -| `/Verify/*` | Verify/testing related jobs | Yes | -| `/Workflows/*` | Common uses of the `workflow:` keyword | No | -| `/*` (root) | General templates | Yes | +You can also use the `|` and `>` YAML operators to [split up multi-line commands](../../ci/yaml/script.md#split-long-commands). -## Criteria +### Explain the template with comments -The file must follow the [`.gitlab-ci.yml` syntax](../../ci/yaml/README.md). -Verify it's valid by pasting it into the CI lint tool at `https://gitlab.com/gitlab-org/gitlab/-/ci/lint`. +You can access template contents from the new file menu, and this might be the only +place users see information about the template. It's important to clearly document +the behavior of the template directly in the template itself. -Also, all templates must be named with the `*.gitlab-ci.yml` suffix. +The following guidelines cover the basic comments expected in all template submissions. +Add additional comments as needed if you think the comments can help users or +[template reviewers](#contribute-cicd-template-merge-requests). + +#### Explain requirements and expectations + +Give the details on how to use the template in `#` comments at the top of the file. +This includes: + +- Repository/project requirements. +- Expected behavior. +- Any places that need to be edited by users before using the template. +- If the template should be used by copy pasting it into a configuration file, or + by using it with the `include` keyword in an existing pipeline. +- If any variables need to be saved in the project's CI/CD settings. + +```yaml +# Use this template to publish an application that uses the ABC server. +# You can copy and paste this template into a new `.gitlab-ci.yml` file. +# You should not add this template to an existing `.gitlab-ci.yml` file by using the `include:` keyword. +# +# Requirements: +# - An ABC project with content saved in /content and tests in /test +# - A CI/CD variable named ABC-PASSWORD saved in the project CI/CD settings. The value +# should be the password used to deploy to your ABC server. +# - An ABC server configured to listen on port 12345. +# +# You must change the URL on line 123 to point to your ABC server and port. +# +# For more information, see https://gitlab.com/example/abcserver/README.md + +job1: + ... +``` + +#### Explain how variables affect template behavior + +If the template uses variables, explain them in `#` comments where they are first +defined. You can skip the comment when the variable is trivially clear: + +```yaml +variables: # Good to have a comment here, for example: + TEST_CODE_PATH: <path/to/code> # Update this variable with the relative path to your Ruby specs + +job1: + variables: + ERROR_MESSAGE: "The $TEST_CODE_PATH path is invalid" # (No need for a comment here, it's already clear) + script: + - echo ${ERROR_MESSAGE} +``` ### Backward compatibility @@ -65,18 +252,6 @@ users have to fix their `.gitlab-ci.yml` that could annoy their workflow. Please read [versioning](#versioning) section for introducing breaking change safely. -### Best practices - -- Avoid using [global keywords](../../ci/yaml/README.md#global-keywords), - such as `image`, `stages` and `variables` at top-level. - When a root `.gitlab-ci.yml` [includes](../../ci/yaml/README.md#include) - multiple templates, these global keywords could be overridden by the - others and cause an unexpected behavior. -- Include [a changelog](../changelog.md) if your merge request introduces a user-facing change. -- Use [`$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH`](../../ci/variables/predefined_variables.md) - instead of a hardcoded `main` branch, and never use `master`. -- Use [`rules`](../../ci/yaml/README.md#rules) instead of [`only` or `except`](../../ci/yaml/README.md#onlyexcept-basic), if possible. - ## Versioning Versioning allows you to introduce a new template without modifying the existing @@ -135,7 +310,7 @@ include: ### Further reading There is an [open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/17716) about -introducing versioning concepts in GitLab CI Templates. You can check that issue to +introducing versioning concepts in GitLab CI/CD templates. You can check that issue to follow the progress. ## Testing @@ -157,7 +332,7 @@ This is useful information for reviewers to make sure the template is safe to be ### Make sure the new template can be selected in UI -Templates located under some directories are also [selectable in the **New file** UI](#place-the-template-file-in-a-relevant-directory). +Templates located under some directories are also [selectable in the **New file** UI](#template-directories). When you add a template into one of those directories, make sure that it correctly appears in the dropdown:  @@ -187,6 +362,10 @@ A template could contain malicious code. For example, a template that contains t might accidentally expose secret project CI/CD variables in a job log. If you're unsure if it's secure or not, you need to ask security experts for cross-validation. -## Contribute CI/CD Template Merge Requests +## Contribute CI/CD template merge requests -After your CI/CD Template MR is created and labeled with `ci::templates`, DangerBot suggests one reviewer and one maintainer that can review your code. When your merge request is ready for review, please `@mention` the reviewer and ask them to review your CI/CD Template changes. See details in the merge request that added [a DangerBot task for CI/CD Template MRs](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/44688). +After your CI/CD template MR is created and labeled with `ci::templates`, DangerBot +suggests one reviewer and one maintainer that can review your code. When your merge +request is ready for review, please `@mention` the reviewer and ask them to review +your CI/CD template changes. See details in the merge request that added +[a DangerBot task for CI/CD template MRs](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/44688). diff --git a/doc/development/code_review.md b/doc/development/code_review.md index 0a811ceba65..731fec98933 100644 --- a/doc/development/code_review.md +++ b/doc/development/code_review.md @@ -38,14 +38,15 @@ Depending on the areas your merge request touches, it must be **approved** by on or more [maintainers](https://about.gitlab.com/handbook/engineering/workflow/code-review/#maintainer): For approvals, we use the approval functionality found in the merge request -widget. Reviewers can add their approval by [approving additionally](../user/project/merge_requests/merge_request_approvals.md#adding-or-removing-an-approval). +widget. For reviewers, we use the [reviewer functionality](../user/project/merge_requests/getting_started.md#reviewer) in the sidebar. +Reviewers can add their approval by [approving additionally](../user/project/merge_requests/merge_request_approvals.md#adding-or-removing-an-approval). Getting your merge request **merged** also requires a maintainer. If it requires more than one approval, the last maintainer to review and approve merges it. ### Domain experts -Domain experts are team members who have substantial experience with a specific technology, product feature or area of the codebase. Team members are encouraged to self-identify as domain experts and add it to their [team profile](https://gitlab.com/gitlab-com/www-gitlab-com/-/blob/master/data/team.yml) +Domain experts are team members who have substantial experience with a specific technology, product feature or area of the codebase. Team members are encouraged to self-identify as domain experts and add it to their [team profile](https://gitlab.com/gitlab-com/www-gitlab-com/-/blob/master/data/team.yml). When self-identifying as a domain expert, it is recommended to assign the MR changing the `team.yml` to be merged by an already established Domain Expert or a corresponding Engineering Manager. @@ -99,8 +100,9 @@ with [domain expertise](#domain-experts). Read the [database review guidelines](database_review.md) for more details. 1. If your merge request includes frontend changes (*1*), it must be **approved by a [frontend maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_maintainers_frontend)**. -1. If your merge request includes UX changes (*1*), it must be - **approved by a [UX team member](https://about.gitlab.com/company/team/)**. +1. If your merge request includes user-facing changes (*3*), it must be + **approved by a [Product Designer](https://about.gitlab.com/handbook/engineering/ux/product-design/)**, + based on assignments in the appropriate [DevOps stage group](https://about.gitlab.com/handbook/product/categories/#devops-stages). 1. If your merge request includes adding a new JavaScript library (*1*)... - If the library significantly increases the [bundle size](https://gitlab.com/gitlab-org/frontend/playground/webpack-memory-metrics/-/blob/master/doc/report.md), it must @@ -109,16 +111,14 @@ with [domain expertise](#domain-experts). GitLab, the license must be **approved by a [legal department member](https://about.gitlab.com/handbook/legal/)**. More information about license compatibility can be found in our [GitLab Licensing and Compatibility documentation](licensing.md). -1. If your merge request includes adding a new UI/UX paradigm (*1*), it must be - **approved by a [UX lead](https://about.gitlab.com/company/team/)**. 1. If your merge request includes a new dependency or a file system change, it must be **approved by a [Distribution team member](https://about.gitlab.com/company/team/)**. See how to work with the [Distribution team](https://about.gitlab.com/handbook/engineering/development/enablement/distribution/#how-to-work-with-distribution) for more details. 1. If your merge request includes documentation changes, it must be **approved by a [Technical writer](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments)**, based on the appropriate [product category](https://about.gitlab.com/handbook/product/categories/). -1. If your merge request includes end-to-end **and** non-end-to-end changes (*3*), it must be **approved +1. If your merge request includes end-to-end **and** non-end-to-end changes (*4*), it must be **approved by a [Software Engineer in Test](https://about.gitlab.com/handbook/engineering/quality/#individual-contributors)**. -1. If your merge request only includes end-to-end changes (*3*) **or** if the MR author is a [Software Engineer in Test](https://about.gitlab.com/handbook/engineering/quality/#individual-contributors), it must be **approved by a [Quality maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_maintainers_qa)** +1. If your merge request only includes end-to-end changes (*4*) **or** if the MR author is a [Software Engineer in Test](https://about.gitlab.com/handbook/engineering/quality/#individual-contributors), it must be **approved by a [Quality maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_maintainers_qa)** 1. If your merge request includes a new or updated [application limit](https://about.gitlab.com/handbook/product/product-processes/#introducing-application-limits), it must be **approved by a [product manager](https://about.gitlab.com/company/team/)**. 1. If your merge request includes Product Intelligence (telemetry or analytics) changes, it should be reviewed and approved by a [Product Intelligence engineer](https://gitlab.com/gitlab-org/growth/product_intelligence/engineers). 1. If your merge request includes an addition of, or changes to a [Feature spec](testing_guide/testing_levels.md#frontend-feature-tests), it must be **approved by a [Quality maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_maintainers_qa) or [Quality reviewer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_reviewers_qa)**. @@ -128,7 +128,10 @@ with [domain expertise](#domain-experts). - (*2*): We encourage you to seek guidance from a database maintainer if your merge request is potentially introducing expensive queries. It is most efficient to comment on the line of code in question with the SQL queries so they can give their advice. -- (*3*): End-to-end changes include all files within the `qa` directory. +- (*3*): User-facing changes include both visual changes (regardless of how minor), + and changes to the rendered DOM which impact how a screen reader may announce + the content. +- (*4*): End-to-end changes include all files within the `qa` directory. #### Security requirements @@ -137,9 +140,11 @@ View the updated documentation regarding [internal application security reviews] ### The responsibility of the merge request author The responsibility to find the best solution and implement it lies with the -merge request author. +merge request author. The author or [directly responsible individual](https://about.gitlab.com/handbook/people-group/directly-responsible-individuals/) +will stay assigned to the merge request as the assignee throughout +the code review lifecycle. If you are unable to set yourself as an assignee, ask a [reviewer](https://about.gitlab.com/handbook/engineering/workflow/code-review/#reviewer) to do this for you. -Before assigning a merge request to a maintainer for approval and merge, they +Before requesting a review from a maintainer to approve and merge, they should be confident that: - It actually solves the problem it was meant to solve. @@ -184,12 +189,11 @@ Avoid: [include a link to the relevant issue](code_comments.md). - Adding comments which only explain what the code is doing. If non-TODO comments are added, they should [_explain why, not what_](https://blog.codinghorror.com/code-tells-you-how-comments-tell-you-why/). -- Assigning merge requests with failed tests to maintainers. If the tests are failing and you have to assign, ensure you leave a comment with an explanation. +- Requesting maintainer reviews of merge requests with failed tests. If the tests are failing and you have to request a review, ensure you leave a comment with an explanation. - Excessively mentioning maintainers through email or Slack (if the maintainer is reachable -through Slack). If you can't assign a merge request, `@` mentioning a maintainer in a comment is acceptable and in all other cases assigning the merge request is sufficient. +through Slack). If you can't add a reviewer for a merge request, `@` mentioning a maintainer in a comment is acceptable and in all other cases adding a reviewer is sufficient. -This -[saves reviewers time and helps authors catch mistakes earlier](https://www.ibm.com/developerworks/rational/library/11-proven-practices-for-peer-review/index.html#__RefHeading__97_174136755). +This saves reviewers time and helps authors catch mistakes earlier. ### The responsibility of the reviewer @@ -197,9 +201,10 @@ This that it meets all requirements, you should: - Click the Approve button. -- Advise the author their merge request has been reviewed and approved. -- Assign the merge request to a maintainer. Default to assigning it to a maintainer with [domain expertise](#domain-experts), +- `@` mention the author to generate a to-do notification, and advise them that their merge request has been reviewed and approved. +- Request a review from a maintainer. Default to requests for a maintainer with [domain expertise](#domain-experts), however, if one isn't available or you think the merge request doesn't need a review by a [domain expert](#domain-experts), feel free to follow the [Reviewer roulette](#reviewer-roulette) suggestion. +- Remove yourself as a reviewer. ### The responsibility of the maintainer @@ -227,7 +232,7 @@ If a developer who happens to also be a maintainer was involved in a merge reque as a reviewer, it is recommended that they are not also picked as the maintainer to ultimately approve and merge it. Maintainers should check before merging if the merge request is approved by the -required approvers. +required approvers. If still awaiting further approvals from others, remove yourself as a reviewer then `@` mention the author and explain why in a comment. Stay as reviewer if you're merging the code. Maintainers must check before merging if the merge request is introducing new vulnerabilities, by inspecting the list in the Merge Request @@ -245,6 +250,19 @@ Note that certain Merge Requests may target a stable branch. These are rare events. These types of Merge Requests cannot be merged by the Maintainer. Instead these should be sent to the [Release Manager](https://about.gitlab.com/community/release-managers/). +After merging, a maintainer should stay as the reviewer listed on the merge request. + +### Dogfooding the Reviewers feature + +On March 18th 2021, an updated process was put in place aimed at efficiently and consistently dogfooding the Reviewers feature. + +Here is a summary of the changes, also reflected in this section above. + +- Merge request authors and DRIs stay as Assignees +- Authors request a review from Reviewers when they are expected to review +- Reviewers remove themselves after they're done reviewing/approving +- The last approver stays as Reviewer upon merging + ## Best practices ### Everyone @@ -304,11 +322,11 @@ first time. - Push commits based on earlier rounds of feedback as isolated commits to the branch. Do not squash until the branch is ready to merge. Reviewers should be able to read individual updates based on their earlier feedback. -- Assign the merge request back to the reviewer once you are ready for another round of - review. If you do not have the ability to assign merge requests, `@` +- Request a new review from the reviewer once you are ready for another round of + review. If you do not have the ability to request a review, `@` mention the reviewer instead. -### Assigning a merge request for a review +### Requesting a review When you are ready to have your merge request reviewed, you should request an initial review by assigning it to a reviewer from your group or team. @@ -319,14 +337,14 @@ You can also use `workflow::ready for review` label. That means that your merge When your merge request receives an approval from the first reviewer it can be passed to a maintainer. You should default to choosing a maintainer with [domain expertise](#domain-experts), and otherwise follow the Reviewer Roulette recommendation or use the label `ready for merge`. Sometimes, a maintainer may not be available for review. They could be out of the office or [at capacity](#review-response-slo). -You can and should check the maintainer’s availability in their profile. If the maintainer recommended by +You can and should check the maintainer's availability in their profile. If the maintainer recommended by the roulette is not available, choose someone else from that list. -It is responsibility of the author of a merge request that the merge request is reviewed. If it stays in `ready for review` state too long it is recommended to assign it to a specific reviewer. +It is the responsibility of the author for the merge request to be reviewed. If it stays in the `ready for review` state too long it is recommended to request a review from a specific reviewer. #### List of merge requests ready for review -Developers who have capacity can regularly check the list of [merge requests to review](https://gitlab.com/groups/gitlab-org/-/merge_requests?state=opened&label_name%5B%5D=workflow%3A%3Aready%20for%20review) and assign any merge request they want to review. +Developers who have capacity can regularly check the list of [merge requests to review](https://gitlab.com/groups/gitlab-org/-/merge_requests?state=opened&label_name%5B%5D=workflow%3A%3Aready%20for%20review) and add themselves as a reviewer for any merge request they want to review. ### Reviewing a merge request @@ -347,12 +365,11 @@ experience, refactors the existing code). Then: - For non-mandatory suggestions, decorate with (non-blocking) so the author knows they can optionally resolve within the merge request or follow-up at a later stage. - There's a [Chrome/Firefox add-on](https://gitlab.com/conventionalcomments/conventional-comments-button) which you can use to apply [Conventional Comment](https://conventionalcomments.org/) prefixes. -- Ensure there are no open dependencies. Check [related issues](../user/project/issues/related_issues.md) for blockers. Clarify with the author(s) +- Ensure there are no open dependencies. Check [linked issues](../user/project/issues/related_issues.md) for blockers. Clarify with the author(s) if necessary. If blocked by one or more open MRs, set an [MR dependency](../user/project/merge_requests/merge_request_dependencies.md). - After a round of line notes, it can be helpful to post a summary note such as "Looks good to me", or "Just a couple things to address." -- Assign the merge request to the author if changes are required following your - review. +- Let the author know if changes are required following your review. ### Merging a merge request @@ -371,8 +388,7 @@ those changes directly without going back to the author. You can do this by using the [suggest changes](../user/discussions/index.md#suggest-changes) feature to apply your own suggestions to the merge request. Note that: -- If the changes are not straightforward, please prefer assigning the merge request back - to the author. +- If the changes are not straightforward, please prefer allowing the author to make the change. - **Before applying suggestions**, edit the merge request to make sure [squash and merge](../user/project/merge_requests/squash_and_merge.md#squash-and-merge) @@ -392,17 +408,26 @@ When ready to merge: circling back with the author about that. Otherwise, if the MR only has a few commits, we'll be respecting the author's setting by not squashing them. -- **Start a new merge request pipeline with the `Run Pipeline` button in the merge - request's "Pipelines" tab, and enable "Merge When Pipeline Succeeds" (MWPS).** Note that: +WARNING: +**If the merge request is from a fork, review all changes thoroughly for malicious code before +starting a [Pipeline for Merged Results](../ci/merge_request_pipelines/index.md#run-pipelines-in-the-parent-project-for-merge-requests-from-a-forked-project).** +Pay particular attention to new dependencies and dependency updates, such as Ruby gems and Node packages. +While changes to files like `Gemfile.lock` or `yarn.lock` might appear trivial, they could lead to the +fetching of malicious packages. +If the MR source branch is more than 100 commits behind the target branch, ask the author to rebase it. +Review links and images, especially in documentation MRs. +When in doubt, ask someone from `@gitlab-com/gl-security/appsec` to review the merge request **before starting any merge request pipeline**. + +- Start a new merge request pipeline with the `Run pipeline` button in the merge + request's "Pipelines" tab, and enable "Merge When Pipeline Succeeds" (MWPS). + Note that: - If **[master is broken](https://about.gitlab.com/handbook/engineering/workflow/#broken-master), - do not merge the merge request**. Follow these specific [handbook instructions](https://about.gitlab.com/handbook/engineering/workflow/#maintaining-throughput-during-broken-master). + do not merge the merge request** except for + [very specific cases](https://about.gitlab.com/handbook/engineering/workflow/#criteria-for-merging-during-broken-master). + For other cases, follow these [handbook instructions](https://about.gitlab.com/handbook/engineering/workflow/#merging-during-broken-master). - If the **latest [Pipeline for Merged Results](../ci/merge_request_pipelines/pipelines_for_merged_results/#pipelines-for-merged-results)** finished less than 2 hours ago, you might merge without starting a new pipeline as the merge request is close enough to `master`. - - If the **merge request is from a fork**, we can use [Pipelines for Merged Results from a forked project](../ci/merge_request_pipelines/index.md#run-pipelines-in-the-parent-project-for-merge-requests-from-a-forked-project) with caution. - Before triggering the pipeline, review all changes for **malicious code**. - If you cannot trigger the pipeline, review the status of the fork relative to `master`. - If it's more than 100 commits behind, ask the author to rebase it before merging. - When you set the MR to "Merge When Pipeline Succeeds", you should take over subsequent revisions for anything that would be spotted after that. @@ -500,7 +525,7 @@ Enterprise Edition instance. This has some implications: ### Review turnaround time Because [unblocking others is always a top priority](https://about.gitlab.com/handbook/values/#global-optimization), -reviewers are expected to review assigned merge requests in a timely manner, +reviewers are expected to review merge requests in a timely manner, even when this may negatively impact their other tasks and priorities. Doing so allows everyone involved in the merge request to iterate faster as the @@ -515,7 +540,7 @@ To ensure swift feedback to ready-to-review code, we maintain a `Review-response If you don't think you can review a merge request in the `Review-response` SLO time frame, let the author know as soon as possible and try to help them find another reviewer or maintainer who is able to, so that they can be unblocked -and get on with their work quickly. +and get on with their work quickly. Remove yourself as a reviewer. If you think you are at capacity and are unable to accept any more reviews until some have been completed, communicate this through your GitLab status by setting @@ -529,7 +554,7 @@ this through your GitLab.com Status, authors are expected to realize this and find a different reviewer themselves. When a merge request author has been blocked for longer than -the `Review-response` SLO, they are free to remind the reviewer through Slack or assign +the `Review-response` SLO, they are free to remind the reviewer through Slack or add another reviewer. ### Customer critical merge requests @@ -539,7 +564,7 @@ A merge request may benefit from being considered a customer critical priority b Properties of customer critical merge requests: - The [Senior Director of Development](https://about.gitlab.com/job-families/engineering/engineering-management/#senior-director-engineering) ([@clefelhocz1](https://gitlab.com/clefelhocz1)) is the DRI for deciding if a merge request is customer critical. -- The DRI assigns the `customer-critical-merge-request` label to the merge request. +- The DRI applies the `customer-critical-merge-request` label to the merge request. - It is required that the reviewer(s) and maintainer(s) involved with a customer critical merge request are engaged as soon as this decision is made. - It is required to prioritize work for those involved on a customer critical merge request so that they have the time available necessary to focus on it. - It is required to adhere to GitLab [values](https://about.gitlab.com/handbook/values/) and processes when working on customer critical merge requests, taking particular note of family and friends first/work second, definition of done, iteration, and release when it's ready. diff --git a/doc/development/contributing/index.md b/doc/development/contributing/index.md index 7a1c189e087..227923b324e 100644 --- a/doc/development/contributing/index.md +++ b/doc/development/contributing/index.md @@ -149,7 +149,7 @@ Keep the following in mind when submitting merge requests: - [Documentation](../documentation/styleguide/index.md) style guide. - [Code style guides](style_guides.md). - Sometimes style guides will be followed but the code will lack structural integrity, or the - reviewer will have reservations about the code’s overall quality. When there is a reservation, + reviewer will have reservations about the code's overall quality. When there is a reservation, the reviewer will inform the author and provide some guidance. - Though GitLab generally allows anyone to indicate [approval](../../user/project/merge_requests/merge_request_approvals.md) of merge requests, the diff --git a/doc/development/contributing/issue_workflow.md b/doc/development/contributing/issue_workflow.md index 7029c8a8384..c505b8cf467 100644 --- a/doc/development/contributing/issue_workflow.md +++ b/doc/development/contributing/issue_workflow.md @@ -412,7 +412,7 @@ in the regression issue as fixes are addressed. In order to track things that can be improved in the GitLab codebase, we use the ~"technical debt" label in the [GitLab issue tracker](https://gitlab.com/gitlab-org/gitlab/-/issues). -For missed user experience requirements, we use the ~"UX debt" label. +We use the ~"UX debt" label when we choose to deviate from the MVC, in a way that harms the user experience. These labels should be added to issues that describe things that can be improved, shortcuts that have been taken, features that need additional attention, and all diff --git a/doc/development/contributing/merge_request_workflow.md b/doc/development/contributing/merge_request_workflow.md index 9051b621bcd..32b0ff45847 100644 --- a/doc/development/contributing/merge_request_workflow.md +++ b/doc/development/contributing/merge_request_workflow.md @@ -131,9 +131,9 @@ Commit messages should follow the guidelines below, for reasons explained by Chr - The commit subject must not be longer than 72 characters. - The commit subject must not end with a period. - The commit body must not contain more than 72 characters per line. -- Commits that change 30 or more lines across at least 3 files must - describe these changes in the commit body. - The commit subject or body must not contain Emojis. +- Commits that change 30 or more lines across at least 3 files should + describe these changes in the commit body. - Use issues and merge requests' full URLs instead of short references, as they are displayed as plain text outside of GitLab. - The merge request should not contain more than 10 commit messages. @@ -178,7 +178,7 @@ Example commit message template that can be used on your machine that embodies t # Do not end the subject line with a period # Subject must contain at least 3 words # Separate subject from body with a blank line -# Commits that change 30 or more lines across at least 3 files must +# Commits that change 30 or more lines across at least 3 files should # describe these changes in the commit body # Do not use Emojis # Use the body to explain what and why vs. how @@ -249,6 +249,7 @@ requirements. 1. [Black-box tests/end-to-end tests](../testing_guide/testing_levels.md#black-box-tests-at-the-system-level-aka-end-to-end-tests) added if required. Please contact [the quality team](https://about.gitlab.com/handbook/engineering/quality/#teams) with any questions. +1. The new feature does not degrade the user experience of the product. ## Dependencies diff --git a/doc/development/creating_enums.md b/doc/development/creating_enums.md index d0f04c30c7b..1a72d99112f 100644 --- a/doc/development/creating_enums.md +++ b/doc/development/creating_enums.md @@ -114,3 +114,41 @@ class Pipeline < ApplicationRecord } end ``` + +## Add new values in the gap + +After merging some EE and FOSS enums, there might be a gap between the two groups of values: + +```ruby +module Enums + module Ci + module CommitStatus + def self.failure_reasons + { + # ... + data_integrity_failure: 12, + forward_deployment_failure: 13, + insufficient_bridge_permissions: 1_001, + downstream_bridge_project_not_found: 1_002, + # ... + } + end + end + end +end +``` + +To add new values, you should fill the gap first. +In the example above add `14` instead of `1_003`: + +```ruby +{ + # ... + data_integrity_failure: 12, + forward_deployment_failure: 13, + a_new_value: 14, + insufficient_bridge_permissions: 1_001, + downstream_bridge_project_not_found: 1_002, + # ... +} +``` diff --git a/doc/development/dangerbot.md b/doc/development/dangerbot.md index 413c0a31eec..7fb1e11b303 100644 --- a/doc/development/dangerbot.md +++ b/doc/development/dangerbot.md @@ -150,16 +150,13 @@ at GitLab so far: ## Limitations -- Danger output is not added to a merge request comment if working on - a fork. This happens because the secret variable from the canonical - project is not shared to forks. - To work around this, you can add an [environment - variable](../ci/variables/README.md) called - `DANGER_GITLAB_API_TOKEN` with a personal API token to your - fork. That way the danger comments are made from CI using that - API token instead. - Making the variable - [masked](../ci/variables/README.md#mask-a-custom-variable) makes sure - it doesn't show up in the job logs. The variable cannot be - [protected](../ci/variables/README.md#protect-a-custom-variable), - as it needs to be present for all feature branches. +Danger is run but its output is not added to a merge request comment if working +on a fork. This happens because the secret variable from the canonical project +is not shared to forks. To work around this, you can add an [environment +variable](../ci/variables/README.md) called `DANGER_GITLAB_API_TOKEN` with a +personal API token to your fork. That way the danger comments are made from CI +using that API token instead. Making the variable +[masked](../ci/variables/README.md#mask-a-cicd-variable) makes sure it +doesn't show up in the job logs. The variable cannot be +[protected](../ci/variables/README.md#protect-a-cicd-variable), as it needs +to be present for all feature branches. diff --git a/doc/development/database/add_foreign_key_to_existing_column.md b/doc/development/database/add_foreign_key_to_existing_column.md index a5d40d455d8..f83dc35b4a6 100644 --- a/doc/development/database/add_foreign_key_to_existing_column.md +++ b/doc/development/database/add_foreign_key_to_existing_column.md @@ -58,6 +58,9 @@ emails = Email.where(user_id: 1) # returns emails for the deleted user Add a `NOT VALID` foreign key constraint to the table, which enforces consistency on the record changes. +[Using the `with_lock_retries` helper method is advised when performing operations on high-traffic tables](../migration_style_guide.md#when-to-use-the-helper-method), +in this case, if the table or the foreign table is a high-traffic table, we should use the helper method. + In the example above, you'd be still able to update records in the `emails` table. However, when you'd try to update the `user_id` with non-existent value, the constraint causes a database error. Migration file for adding `NOT VALID` foreign key: @@ -66,8 +69,6 @@ Migration file for adding `NOT VALID` foreign key: class AddNotValidForeignKeyToEmailsUser < ActiveRecord::Migration[5.2] include Gitlab::Database::MigrationHelpers - DOWNTIME = false - def up # safe to use: it requires short lock on the table since we don't validate the foreign key add_foreign_key :emails, :users, on_delete: :cascade, validate: false @@ -94,8 +95,6 @@ Example for cleaning up records in the `emails` table within a database migratio class RemoveRecordsWithoutUserFromEmailsTable < ActiveRecord::Migration[5.2] include Gitlab::Database::MigrationHelpers - DOWNTIME = false - disable_ddl_transaction! class Email < ActiveRecord::Base @@ -130,8 +129,6 @@ Migration file for validating the foreign key: class ValidateForeignKeyOnEmailUsers < ActiveRecord::Migration[5.2] include Gitlab::Database::MigrationHelpers - DOWNTIME = false - def up validate_foreign_key :emails, :user_id end diff --git a/doc/development/database/database_reviewer_guidelines.md b/doc/development/database/database_reviewer_guidelines.md index a242a3d5fd0..de131ddffbc 100644 --- a/doc/development/database/database_reviewer_guidelines.md +++ b/doc/development/database/database_reviewer_guidelines.md @@ -66,7 +66,7 @@ Finally, you can find various guides in the [Database guides](index.md) page tha topics and use cases. The most frequently required during database reviewing are the following: - [Migrations style guide](../migration_style_guide.md) for creating safe SQL migrations. -- [What requires downtime?](../what_requires_downtime.md). +- [Avoiding downtime in migrations](../avoiding_downtime_in_migrations.md). - [SQL guidelines](../sql.md) for working with SQL queries. ## How to apply for becoming a database maintainer diff --git a/doc/development/database/index.md b/doc/development/database/index.md index 870ae1542bd..01f6753e7a0 100644 --- a/doc/development/database/index.md +++ b/doc/development/database/index.md @@ -18,11 +18,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w - [Understanding EXPLAIN plans](../understanding_explain_plans.md) - [explain.depesz.com](https://explain.depesz.com/) or [explain.dalibo.com](https://explain.dalibo.com/) for visualizing the output of `EXPLAIN` -- [pgFormatter](http://sqlformat.darold.net/) a PostgreSQL SQL syntax beautifier +- [pgFormatter](https://sqlformat.darold.net/) a PostgreSQL SQL syntax beautifier ## Migrations -- [What requires downtime?](../what_requires_downtime.md) +- [Avoiding downtime in migrations](../avoiding_downtime_in_migrations.md) - [SQL guidelines](../sql.md) for working with SQL queries - [Migrations style guide](../migration_style_guide.md) for creating safe SQL migrations - [Testing Rails migrations](../testing_guide/testing_migrations_guide.md) guide diff --git a/doc/development/database/not_null_constraints.md b/doc/development/database/not_null_constraints.md index 01d5b7af7f1..9d1850f5504 100644 --- a/doc/development/database/not_null_constraints.md +++ b/doc/development/database/not_null_constraints.md @@ -26,8 +26,6 @@ For example, consider a migration that creates a table with two `NOT NULL` colum ```ruby class CreateDbGuides < ActiveRecord::Migration[6.0] - DOWNTIME = false - def change create_table :db_guides do |t| t.bigint :stars, default: 0, null: false @@ -47,8 +45,6 @@ For example, consider a migration that adds a new `NOT NULL` column `active` to ```ruby class AddExtendedTitleToSprints < ActiveRecord::Migration[6.0] - DOWNTIME = false - def change add_column :db_guides, :active, :boolean, default: true, null: false end @@ -117,7 +113,6 @@ with `validate: false` in a post-deployment migration, ```ruby class AddNotNullConstraintToEpicsDescription < ActiveRecord::Migration[6.0] include Gitlab::Database::MigrationHelpers - DOWNTIME = false disable_ddl_transaction! @@ -191,7 +186,6 @@ migration helper in a final post-deployment migration, ```ruby class ValidateNotNullConstraintOnEpicsDescription < ActiveRecord::Migration[6.0] include Gitlab::Database::MigrationHelpers - DOWNTIME = false disable_ddl_transaction! @@ -207,7 +201,7 @@ end ## `NOT NULL` constraints on large tables -If you have to clean up a text column for a really [large table](https://gitlab.com/gitlab-org/gitlab/-/blob/master/rubocop/migration_helpers.rb#L12) +If you have to clean up a text column for a [high-traffic table](../migration_style_guide.md#high-traffic-tables) (for example, the `artifacts` in `ci_builds`), your background migration will go on for a while and it will need an additional [background migration cleaning up](../background_migrations.md#cleaning-up) in the release after adding the data migration. diff --git a/doc/development/database/strings_and_the_text_data_type.md b/doc/development/database/strings_and_the_text_data_type.md index f338520c6ca..6fbb95c4f60 100644 --- a/doc/development/database/strings_and_the_text_data_type.md +++ b/doc/development/database/strings_and_the_text_data_type.md @@ -52,8 +52,6 @@ For example, consider a migration that creates a table with two text columns, class CreateDbGuides < ActiveRecord::Migration[6.0] include Gitlab::Database::MigrationHelpers - DOWNTIME = false - def up create_table_with_constraints :db_guides do |t| t.bigint :stars, default: 0, null: false @@ -89,7 +87,6 @@ For example, consider a migration that adds a new text column `extended_title` t ```ruby class AddExtendedTitleToSprints < ActiveRecord::Migration[6.0] - DOWNTIME = false # rubocop:disable Migration/AddLimitToTextColumns # limit is added in 20200501000002_add_text_limit_to_sprints_extended_title @@ -106,8 +103,6 @@ A second migration should follow the first one with a limit added to `extended_t ```ruby class AddTextLimitToSprintsExtendedTitle < ActiveRecord::Migration[6.0] include Gitlab::Database::MigrationHelpers - DOWNTIME = false - disable_ddl_transaction! def up @@ -185,8 +180,6 @@ in a post-deployment migration, class AddTextLimitMigration < ActiveRecord::Migration[6.0] include Gitlab::Database::MigrationHelpers - DOWNTIME = false - disable_ddl_transaction! def up @@ -266,7 +259,6 @@ helper in a final post-deployment migration, ```ruby class ValidateTextLimitMigration < ActiveRecord::Migration[6.0] include Gitlab::Database::MigrationHelpers - DOWNTIME = false disable_ddl_transaction! diff --git a/doc/development/database_debugging.md b/doc/development/database_debugging.md index dc64b59018b..67ec1b3c4f1 100644 --- a/doc/development/database_debugging.md +++ b/doc/development/database_debugging.md @@ -29,10 +29,10 @@ If you just want to delete everything and start over with sample data (approxima also does `db:reset` and runs DB-specific migrations: ```shell -bundle exec rake dev:setup RAILS_ENV=development +bundle exec rake db:setup RAILS_ENV=development ``` -If your test DB is giving you problems, it is safe to nuke it because it doesn't contain important +If your test DB is giving you problems, it is safe to delete everything because it doesn't contain important data: ```shell diff --git a/doc/development/database_query_comments.md b/doc/development/database_query_comments.md index 39b14074073..e4133633a77 100644 --- a/doc/development/database_query_comments.md +++ b/doc/development/database_query_comments.md @@ -20,9 +20,8 @@ and its application source from the comments. Queries generated from **Rails** include the following metadata in comments: - `application` -- `controller` -- `action` - `correlation_id` +- `endpoint_id` - `line` Queries generated from **Sidekiq** workers will include the following metadata @@ -30,20 +29,34 @@ in comments: - `application` - `jid` -- `job_class` - `correlation_id` +- `endpoint_id` - `line` -Examples of queries with comments as observed in `development.log`: +`endpoint_id` is a single field that can represent any endpoint in the application: -1. Rails: +- For Rails controllers, it's the controller and action. For example, `Projects::BlobController#show`. +- For Grape API endpoints, it's the route. For example, `/api/:version/users/:id`. +- For Sidekiq workers, it's the worker class name. For example, `UserStatusCleanup::BatchWorker`. + +`line` is not present in production logs due to the additional overhead required. + +Examples of queries with comments: + +- Rails: + + ```sql + /*application:web,controller:blob,action:show,correlation_id:01EZVMR923313VV44ZJDJ7PMEZ,endpoint_id:Projects::BlobController#show*/ SELECT "routes".* FROM "routes" WHERE "routes"."source_id" = 75 AND "routes"."source_type" = 'Namespace' LIMIT 1 + ``` + +- Grape: ```sql - /*application:web,controller:jobs,action:trace,correlation_id:rYF4mey9CH3,line:/app/policies/project_policy.rb:504:in `feature_available?'*/ SELECT "project_features".* FROM "project_features" WHERE "project_features"."project_id" = $1 LIMIT $2 [["project_id", 5], ["LIMIT", 1]] + /*application:web,correlation_id:01EZVN0DAYGJF5XHG9N4VX8FAH,endpoint_id:/api/:version/users/:id*/ SELECT COUNT(*) FROM "users" INNER JOIN "user_follow_users" ON "users"."id" = "user_follow_users"."followee_id" WHERE "user_follow_users"."follower_id" = 1 ``` -1. Sidekiq: +- Sidekiq: ```sql - /*application:sidekiq,jid:e7d6668a39a991e323009833,job_class:ExpireJobCacheWorker,correlation_id:rYF4mey9CH3,line:/app/workers/expire_job_cache_worker.rb:14:in `perform'*/ SELECT "ci_pipelines".* FROM "ci_pipelines" WHERE "ci_pipelines"."id" = $1 LIMIT $2 [["id", 64], ["LIMIT", 1]] + /*application:sidekiq,correlation_id:df643992563683313bc0a0288fb55e23,jid:15fbc506590c625d7664b074,endpoint_id:UserStatusCleanup::BatchWorker,line:/app/workers/user_status_cleanup/batch_worker.rb:19:in `perform'*/ SELECT $1 AS one FROM "user_statuses" WHERE "user_statuses"."clear_status_at" <= $2 LIMIT $3 ``` diff --git a/doc/development/database_review.md b/doc/development/database_review.md index a19f46b2198..a25714ca6cf 100644 --- a/doc/development/database_review.md +++ b/doc/development/database_review.md @@ -30,9 +30,9 @@ A database review is required for: See the [Product Intelligence Guide](https://about.gitlab.com/handbook/product/product-intelligence-guide/) for implementation details. -A database reviewer is expected to look out for obviously complex +A database reviewer is expected to look out for overly complex queries in the change and review those closer. If the author does not -point out specific queries for review and there are no obviously +point out specific queries for review and there are no overly complex queries, it is enough to concentrate on reviewing the migration only. @@ -67,8 +67,8 @@ A database **reviewer**'s role is to: - Ensure the [required](#required) artifacts are provided and in the proper format. If they are not, reassign the merge request back to the author. - Perform a first-pass review on the MR and suggest improvements to the author. - Once satisfied, relabel the MR with ~"database::reviewed", approve it, and - reassign MR to the database **maintainer** suggested by Reviewer - Roulette. + request a review from the database **maintainer** suggested by Reviewer + Roulette. Remove yourself as a reviewer once this has been done. A database **maintainer**'s role is to: @@ -78,12 +78,13 @@ A database **maintainer**'s role is to: - Finally approve the MR and relabel the MR with ~"database::approved" - Merge the MR if no other approvals are pending or pass it on to other maintainers as required (frontend, backend, docs). + - If not merging, remove yourself as a reviewer. ### Distributing review workload Review workload is distributed using [reviewer roulette](code_review.md#reviewer-roulette) ([example](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/25181#note_147551725)). -The MR author should then co-assign the suggested database +The MR author should request a review from the suggested database **reviewer**. When they give their sign-off, they will hand over to the suggested database **maintainer**. @@ -175,7 +176,7 @@ test its execution using `CREATE INDEX CONCURRENTLY` in the `#database-lab` Slac #### Preparation when removing columns, tables, indexes, or other structures -- Follow the [guidelines on dropping columns](what_requires_downtime.md#dropping-columns). +- Follow the [guidelines on dropping columns](avoiding_downtime_in_migrations.md#dropping-columns). - Generally it's best practice (but not a hard rule) to remove indexes and foreign keys in a post-deployment migration. - Exceptions include removing indexes and foreign keys for small tables. - If you're adding a composite index, another index might become redundant, so remove that in the same migration. @@ -198,7 +199,7 @@ test its execution using `CREATE INDEX CONCURRENTLY` in the `#database-lab` Slac - Check that the relevant version files under `db/schema_migrations` were added or removed. - Check queries timing (If any): In a single transaction, cumulative query time executed in a migration needs to fit comfortably within `15s` - preferably much less than that - on GitLab.com. - - For column removals, make sure the column has been [ignored in a previous release](what_requires_downtime.md#dropping-columns) + - For column removals, make sure the column has been [ignored in a previous release](avoiding_downtime_in_migrations.md#dropping-columns) - Check [background migrations](background_migrations.md): - Establish a time estimate for execution on GitLab.com. For historical purposes, it's highly recommended to include this estimation on the merge request description. @@ -220,7 +221,7 @@ test its execution using `CREATE INDEX CONCURRENTLY` in the `#database-lab` Slac - Data migrations should be reversible too or come with a description of how to reverse, when possible. This applies to all types of migrations (regular, post-deploy, background). - Query performance - - Check for any obviously complex queries and queries the author specifically + - Check for any overly complex queries and queries the author specifically points out for review (if any) - If not present yet, ask the author to provide SQL queries and query plans (for example, by using [ChatOps](understanding_explain_plans.md#chatops) or direct diff --git a/doc/development/deprecation_guidelines/index.md b/doc/development/deprecation_guidelines/index.md index c2fea3c6053..2d092b24d65 100644 --- a/doc/development/deprecation_guidelines/index.md +++ b/doc/development/deprecation_guidelines/index.md @@ -26,4 +26,8 @@ A feature can be deprecated at any time, provided there is a viable alternative. ## When can a feature be removed/changed? -See our [Release and Maintenance policy](../../policy/maintenance.md). +For API removals, see the [GraphQL](../../api/graphql/index.md#deprecation-and-removal-process) and [GitLab API](../../api/README.md#compatibility-guidelines) guidelines. + +For configuration removals, see the [Omnibus deprecation policy](https://docs.gitlab.com/omnibus/package-information/deprecation_policy.html). + +For versioning and upgrade details, see our [Release and Maintenance policy](../../policy/maintenance.md). diff --git a/doc/development/distributed_tracing.md b/doc/development/distributed_tracing.md index 17967c5f63c..e5293c0804c 100644 --- a/doc/development/distributed_tracing.md +++ b/doc/development/distributed_tracing.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -157,7 +157,7 @@ This should start the process with the default listening ports. Once you have Jaeger running, configure the `GITLAB_TRACING` variable with the appropriate configuration string. -**TL;DR:** If you are running everything on the same host, use the following value: +If you're running everything on the same host, use the following value: ```shell export GITLAB_TRACING="opentracing://jaeger?http_endpoint=http%3A%2F%2Flocalhost%3A14268%2Fapi%2Ftraces&sampler=const&sampler_param=1" diff --git a/doc/development/documentation/feature_flags.md b/doc/development/documentation/feature_flags.md index 8fe3f0cbf8e..7dd02c44afa 100644 --- a/doc/development/documentation/feature_flags.md +++ b/doc/development/documentation/feature_flags.md @@ -20,7 +20,7 @@ must be documented. For context, see the ## Criteria -According to the process of [deploying GitLab features behind feature flags](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle): +According to the process of [deploying GitLab features behind feature flags](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/): > - _By default, feature flags should be off._ > - _Feature flags should remain in the codebase for a short period as possible to reduce the need for feature flag accounting._ @@ -62,13 +62,14 @@ be enabled for a single project, and is not ready for production use: # Feature Name > - [Introduced](link-to-issue) in GitLab 12.0. -> - It's [deployed behind a feature flag](<replace with path to>/user/feature_flags.md), disabled by default. -> - It's disabled on GitLab.com. -> - It's not recommended for production use. -> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#anchor-to-section). **(FREE SELF)** +> - [Deployed behind a feature flag](<replace with path to>/user/feature_flags.md), disabled by default. +> - Disabled on GitLab.com. +> - Not recommended for production use. +> - To use in GitLab self-managed instances, ask a GitLab administrator to [enable it](#anchor-to-section). **(FREE SELF)** -WARNING: -This feature might not be available to you. Check the **version history** note above for details. +This in-development feature might not be available for your use. There can be +[risks when enabling features still in development](<replace with path to>/user/feature_flags.md#risks-when-enabling-features-still-in-development). +Refer to this feature's version history for more details. (...Regular content goes here...) @@ -120,14 +121,15 @@ use: # Feature Name > - [Introduced](link-to-issue) in GitLab 12.0. -> - It was [deployed behind a feature flag](<replace with path to>/user/feature_flags.md), disabled by default. -> - [Became enabled by default](link-to-issue) on GitLab 12.1. -> - It's enabled on GitLab.com. -> - It's recommended for production use. +> - [Deployed behind a feature flag](<replace with path to>/user/feature_flags.md), disabled by default. +> - [Enabled by default](link-to-issue) in GitLab 12.1. +> - Enabled on GitLab.com. +> - Recommended for production use. > - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#anchor-to-section). **(FREE SELF)** -WARNING: -This feature might not be available to you. Check the **version history** note above for details. +There can be +[risks when disabling released features](<replace with path to>/user/feature_flags.md#risks-when-disabling-released-features). +Refer to this feature's version history for more details. (...Regular content goes here...) @@ -177,13 +179,14 @@ cannot be enabled for a single project, and is ready for production use: # Feature Name > - [Introduced](link-to-issue) in GitLab 12.0. -> - It's [deployed behind a feature flag](<replace with path to>/user/feature_flags.md), enabled by default. -> - It's enabled on GitLab.com. -> - It's recommended for production use. +> - [Deployed behind a feature flag](<replace with path to>/user/feature_flags.md), enabled by default. +> - Enabled on GitLab.com. +> - Recommended for production use. > - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#anchor-to-section). **(FREE SELF)** -WARNING: -This feature might not be available to you. Check the **version history** note above for details. +There can be +[risks when disabling released features](<replace with path to>/user/feature_flags.md#risks-when-disabling-released-features). +Refer to this feature's version history for more details. (...Regular content goes here...) @@ -249,14 +252,15 @@ be enabled by project, and is ready for production use: # Feature Name > - [Introduced](link-to-issue) in GitLab 12.0. -> - It's [deployed behind a feature flag](<replace with path to>/user/feature_flags.md), enabled by default. -> - It's enabled on GitLab.com. -> - It can be enabled or disabled for a single project. -> - It's recommended for production use. +> - [Deployed behind a feature flag](<replace with path to>/user/feature_flags.md), enabled by default. +> - Enabled on GitLab.com. +> - Can be enabled or disabled for a single project. +> - Recommended for production use. > - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#anchor-to-section). **(FREE SELF)** -WARNING: -This feature might not be available to you. Check the **version history** note above for details. +There can be +[risks when disabling released features](<replace with path to>/user/feature_flags.md#risks-when-disabling-released-features). +Refer to this feature's version history for more details. (...Regular content goes here...) diff --git a/doc/development/documentation/index.md b/doc/development/documentation/index.md index 53e7ba35831..f10396570a2 100644 --- a/doc/development/documentation/index.md +++ b/doc/development/documentation/index.md @@ -183,20 +183,52 @@ file. To add a redirect: -1. Create a merge request in one of the internal docs projects (`gitlab`, - `gitlab-runner`, `omnibus-gitlab`, or `charts`), depending on the location of - the file that's being moved, renamed, or removed. -1. To move or rename the documentation file, create a new file with the new - name or location, but don't delete the existing documentation file. -1. In the original documentation file, add the redirect code for - `/help`. Use the following template exactly, and change only the links and - date. Use relative paths and `.md` for a redirect to another documentation - page. Use the full URL (with `https://`) to redirect to a different project or - site: +1. In the repository (`gitlab`, `gitlab-runner`, `omnibus-gitlab`, or `charts`), + create a new documentation file. Don't delete the old one. The easiest + way is to copy it. For example: + + ```shell + cp doc/user/search/old_file.md doc/api/new_file.md + ``` + +1. Add the redirect code to the old documentation file by running the + following Rake task. The first argument is the path of the old file, + and the second argument is the path of the new file: + + - To redirect to a page in the same project, use relative paths and + the `.md` extension. Both old and new paths start from the same location. + In the following example, both paths are relative to `doc/`: + + ```shell + bundle exec rake "gitlab:docs:redirect[doc/user/search/old_file.md, doc/api/new_file.md]" + ``` + + - To redirect to a page in a different project or site, use the full URL (with `https://`) : + + ```shell + bundle exec rake "gitlab:docs:redirect[doc/user/search/old_file.md, https://example.com]" + ``` + + Alternatively, you can omit the arguments, and you'll be asked to enter + their values: + + ```shell + bundle exec rake gitlab:docs:redirect + ``` + + If you don't want to use the Rake task, you can use the following template. + However, the file paths must be relative to the `doc` or `docs` directory. + + Replace the value of `redirect_to` with the new file path and `YYYY-MM-DD` + with the date the file should be removed. + + Redirect files that link to docs in internal documentation projects + are removed after three months. Redirect files that link to external sites are + removed after one year: ```markdown --- - redirect_to: '../path/to/file/index.md' + redirect_to: '../newpath/to/file/index.md' --- This document was moved to [another location](../path/to/file/index.md). @@ -205,27 +237,24 @@ To add a redirect: <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> ``` - Redirect files linking to docs in any of the internal documentations projects - are removed after three months. Redirect files linking to external sites are - removed after one year. - 1. If the documentation page being moved has any Disqus comments, follow the steps described in [Redirections for pages with Disqus comments](#redirections-for-pages-with-disqus-comments). -1. If a documentation page you're removing includes images that aren't used +1. Open a merge request with your changes. If a documentation page + you're removing includes images that aren't used with any other documentation pages, be sure to use your merge request to delete those images from the repository. 1. Assign the merge request to a technical writer for review and merge. -1. Search for links to the original documentation file. You must find and update all - links that point to the original documentation file: +1. Search for links to the old documentation file. You must find and update all + links that point to the old documentation file: - In <https://gitlab.com/gitlab-com/www-gitlab-com>, search for full URLs: `grep -r "docs.gitlab.com/ee/path/to/file.html" .` - In <https://gitlab.com/gitlab-org/gitlab-docs/-/tree/master/content/_data>, search the navigation bar configuration files for the path with `.html`: `grep -r "path/to/file.html" .` - - In any of the four internal projects. This includes searching for links in the docs + - In any of the four internal projects, search for links in the docs and codebase. Search for all variations, including full URL and just the path. - In macOS for example, go to the root directory of the `gitlab` project and run: + For example, go to the root directory of the `gitlab` project and run: ```shell grep -r "docs.gitlab.com/ee/path/to/file.html" . @@ -387,11 +416,11 @@ To preview your changes to documentation locally, follow this The live preview is currently enabled for the following projects: - [`gitlab`](https://gitlab.com/gitlab-org/gitlab) +- [`omnibus-gitlab`](https://gitlab.com/gitlab-org/omnibus-gitlab) - [`gitlab-runner`](https://gitlab.com/gitlab-org/gitlab-runner) If your merge request has docs changes, you can use the manual `review-docs-deploy` job to deploy the docs review app for your merge request. -You need at least Maintainer permissions to be able to run it.  @@ -399,11 +428,6 @@ You must push a branch to those repositories, as it doesn't work for forks. The `review-docs-deploy*` job: -1. Creates a new branch in the [`gitlab-docs`](https://gitlab.com/gitlab-org/gitlab-docs) - project named after the scheme: `docs-preview-$DOCS_GITLAB_REPO_SUFFIX-$CI_MERGE_REQUEST_IID`, - where `DOCS_GITLAB_REPO_SUFFIX` is the suffix for each product, e.g, `ee` for - EE, `omnibus` for Omnibus GitLab, etc, and `CI_MERGE_REQUEST_IID` is the ID - of the respective merge request. 1. Triggers a cross project pipeline and build the docs site with your changes. In case the review app URL returns 404, this means that either the site is not @@ -412,11 +436,6 @@ minutes and it should appear online, otherwise you can check the status of the remote pipeline from the link in the merge request's job output. If the pipeline failed or got stuck, drop a line in the `#docs` chat channel. -Make sure that you always delete the branch of the merge request you were -working on. If you don't, the remote docs branch isn't removed either, -and the server where the Review Apps are hosted can eventually run out of -disk space. - NOTE: Someone with no merge rights to the GitLab projects (think of forks from contributors) cannot run the manual job. In that case, you can @@ -440,19 +459,11 @@ If you want to know the in-depth details, here's what's really happening: 1. You manually run the `review-docs-deploy` job in a merge request. 1. The job runs the [`scripts/trigger-build`](https://gitlab.com/gitlab-org/gitlab/blob/master/scripts/trigger-build) - script with the `docs deploy` flag, which in turn: - 1. Takes your branch name and applies the following: - - The `docs-preview-` prefix is added. - - The product slug is used to know the project the review app originated - from. - - The number of the merge request is added so that you can know by the - `gitlab-docs` branch name the merge request it originated from. - 1. The remote branch is then created if it doesn't exist (meaning you can - re-run the manual job as many times as you want and this step is skipped). - 1. A new cross-project pipeline is triggered in the docs project. - 1. The preview URL is shown both at the job output and in the merge request - widget. You also get the link to the remote pipeline. -1. In the docs project, the pipeline is created and it + script with the `docs deploy` flag, which triggers the "Triggered from `gitlab-org/gitlab` 'review-docs-deploy' job" + pipeline trigger in the `gitlab-org/gitlab-docs` project for the `$DOCS_BRANCH` (defaults to `master`). +1. The preview URL is shown both at the job output and in the merge request + widget. You also get the link to the remote pipeline. +1. In the `gitlab-org/gitlab-docs` project, the pipeline is created and it [skips the test jobs](https://gitlab.com/gitlab-org/gitlab-docs/blob/8d5d5c750c602a835614b02f9db42ead1c4b2f5e/.gitlab-ci.yml#L50-55) to lower the build time. 1. Once the docs site is built, the HTML files are uploaded as artifacts. @@ -471,7 +482,8 @@ The following GitLab features are used among others: ## Testing -For more information on documentation testing, see [Documentation testing](testing.md) +For more information about documentation testing, see the [Documentation testing](testing.md) +guide. ## Danger Bot diff --git a/doc/development/documentation/site_architecture/global_nav.md b/doc/development/documentation/site_architecture/global_nav.md index f66b0543ad1..7175a9f1a5c 100644 --- a/doc/development/documentation/site_architecture/global_nav.md +++ b/doc/development/documentation/site_architecture/global_nav.md @@ -10,6 +10,7 @@ description: "Learn how GitLab docs' global navigation works and how to add new > - [Introduced](https://gitlab.com/gitlab-org/gitlab-docs/-/merge_requests/362) in GitLab 11.6. > - [Updated](https://gitlab.com/gitlab-org/gitlab-docs/-/merge_requests/482) in GitLab 12.1. > - [Per-project](https://gitlab.com/gitlab-org/gitlab-docs/-/merge_requests/498) navigation added in GitLab 12.2. +> - [Unified global navigation](https://gitlab.com/gitlab-org/gitlab-docs/-/merge_requests/1482) added in GitLab 13.11. Global navigation (the left-most pane in our three pane documentation) provides: @@ -19,24 +20,12 @@ Global navigation (the left-most pane in our three pane documentation) provides: - The ability to refine landing pages, so they don't have to do all the work of surfacing every page contained within the documentation. -## Quick start - -To add a topic to the global nav, go to the directory that contains -[navigation files](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/content/_data/) -and edit the `yaml` file for your product area. You can copy an existing nav entry and -edit it to point to your topic. - -The files are: - -| File | Document | Location | -|-----------------------|--------------------------------------------------------------------|-------------------------------------------------------| -| `charts-nav.yaml` | GitLab cloud native Helm Chart | `https://docs.gitlab.com/charts/` | -| `default-nav.yaml` | GitLab Docs | `https://docs.gitlab.com/ee/` | -| `omnibus-nav.yaml` | Omnibus GitLab Docs | `https://docs.gitlab.com/omnibus/` | -| `runner-nav.yaml` | GitLab Runner Docs | `https://docs.gitlab.com/runner/` | - ## Adding new items +To add a topic to the global nav, edit +[`navigation.yaml`](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/content/_data/navigation.yaml) +and add your item. + All new pages need a new navigation item. Without a navigation, the page becomes "orphaned". That is: @@ -84,28 +73,16 @@ mechanics of what is required is [documented below](#data-file) but, in principl - Avoid jargon or terms of art, unless ubiquitous. For example, **CI** is an acceptable substitution for **Continuous Integration**. - Navigation links must follow the rules documented in the [data file](#data-file). -- EE badging is subject to the following: - - Required when linking to an EE-only page. - - Not required when linking to a page that is a mix of CE and EE-only content. - - Required when all sub-items are EE-only. In this case, no sub-items are EE badged. - - Not required when sub-items are a mix of CE and EE-only items. In this case, each item is - badged appropriately. ## How it works -The global nav has 3 components: +The global nav has five levels: - **Section** - Category - Doc - -The available sections are described on the table below: - -| Section | Description | -| ------------- | ------------------------------------ | -| User | Documentation for the GitLab UI. | -| Administrator | Documentation for the Admin Area. | -| Contributor | Documentation for developing GitLab. | + - Doc + - Doc The majority of the links available on the nav were added according to the UI. The match is not perfect, as for some UI nav items the documentation doesn't @@ -114,7 +91,7 @@ documentation. The docs under **Administration** are ordered alphabetically for clarity. To see the improvements planned, check the -[global nav epic](https://gitlab.com/groups/gitlab-com/-/epics/21). +[global nav epic](https://gitlab.com/groups/gitlab-org/-/epics/1599). **Do not** [add items](#adding-new-items) to the global nav without the consent of one of the technical writers. @@ -131,9 +108,9 @@ the data among the nav in containers properly [styled](#css-classes). ### Data file -The data file describes the structure of the navigation for the applicable project. All data files -are stored at <https://gitlab.com/gitlab-org/gitlab-docs/blob/master/content/_data/> and comprise -three components: +The data file describes the structure of the navigation for the applicable project. +It is stored at <https://gitlab.com/gitlab-org/gitlab-docs/blob/master/content/_data/navigation.yaml> +and comprises of three main components: - Sections - Categories @@ -183,30 +160,12 @@ Example of section with two stand-alone categories: For clarity, **always** add a blank line between categories. -If a category URL is not present in CE (it's an EE-only document), add the -attribute `ee_only: true` below the category link. Example: - -```yaml -- category_title: Category title - category_url: 'category-link' - ee_only: true -``` - -If the category links to an external URL, e.g., [GitLab Design System](https://design.gitlab.com), -add the attribute `external_url: true` below the category title. Example: - -```yaml -- category_title: GitLab Design System - category_url: 'https://design.gitlab.com' - external_url: true -``` - #### Docs -Each doc represents the third level of nav links. They must be always +Each doc represents the third, fourth, and fifth level of nav links. They must be always added within a category. -Example with one doc link: +Example with three doc links, one at each level: ```yaml - category_title: Category title @@ -214,10 +173,17 @@ Example with one doc link: docs: - doc_title: Document title doc_url: 'doc-link' + docs: + - doc_title: Document title + doc_url: 'doc-link' + docs: + - doc_title: Document title + doc_url: 'doc-link' ``` A category supports as many docs as necessary, but, for clarity, try to not -overpopulate a category. +overpopulate a category. Also, do not use more than three levels of docs, it +is not supported. Example with multiple docs: @@ -231,15 +197,6 @@ Example with multiple docs: doc_url: 'doc-2-link' ``` -Whenever a document is only present in EE, add the attribute `ee-only: true` -below the doc link. Example: - -```yaml -- doc_title: Document 2 title - doc_url: 'doc-2-link' - ee_only: true -``` - If you need to add a document in an external URL, add the attribute `external_url` below the doc link: @@ -258,12 +215,12 @@ Example: ```yaml - category_title: Operations - category_url: 'user/project/integrations/prometheus_library/' + category_url: 'ee/user/project/integrations/prometheus_library/' # until we have a link to operations, the first doc link is # repeated in the category link docs: - doc_title: Metrics - doc_url: 'user/project/integrations/prometheus_library/' + doc_url: 'ee/user/project/integrations/prometheus_library/' ``` #### Syntax @@ -274,43 +231,39 @@ and the following syntax rules. ##### Titles - Use sentence case, capitalizing feature names. -- There's no need to wrap the titles, unless there's a special char in it. E.g., +- There's no need to wrap the titles, unless there's a special char in it. For example, in `GitLab CI/CD`, there's a `/` present, therefore, it must be wrapped in quotes. As convention, wrap the titles in double quotes: `category_title: "GitLab CI/CD"`. ##### URLs -- As convention, always wrap URLs in single quotes `'url'`. -- Always use relative paths against the home of CE and EE. Examples: - - For `https://docs.gitlab.com/ee/README.html`, the relative URL is `README.html`. - - For `https://docs.gitlab.com/ee/user/analytics/value_stream_analytics.md`, the relative - URL is `user/analytics/value_stream_analytics.html`. -- For `README.html` files, add the complete path `path/to/README.html`. -- For `index.html` files, use the clean (canonical) URL: `path/to/`. -- For EE-only docs, use the same relative path, but add the attribute `ee_only: true` below - the `doc_url` or `category_url`, as explained above. This displays - an "information" icon on the nav to make the user aware that the feature is - EE-only. - -WARNING: -All links present on the data file must end in `.html`, not `.md`. Do not -start any relative link with a forward slash `/`. - -Examples: - -```yaml -- category_title: Issues - category_url: 'user/project/issues/' - # note that the above URL does not start with a slash and - # does not include index.html at the end +URLs can be either relative or external, and the following apply: - docs: - - doc_title: Container Scanning - doc_url: 'user/application_security/container_scanning/' - ee_only: true - # note that the URL above ends in html and, as the - # document is EE-only, the attribute ee_only is set to true. -``` +- All links in the data file must end with `.html` (with the exception + of `index.html` files), and not `.md`. +- For `index.html` files, use the clean (canonical) URL: `path/to/`. For example, `https://docs.gitlab.com/ee/install/index.html` becomes `ee/install/`. +- Do not start any relative link with a forward slash `/`. +- As convention, always wrap URLs in single quotes `'url'`. +- Always use the project prefix depending on which project the link you add + lives in. To find the global nav link, from the full URL remove `https://docs.gitlab.com/`. +- If a URL links to an external URL, like the [GitLab Design System](https://design.gitlab.com), + add the attribute `external_url: true`, for example: + + ```yaml + - category_title: GitLab Design System + category_url: 'https://design.gitlab.com' + external_url: true + ``` + +Examples of relative URLs: + +| Full URL | Global nav URL | +| -------------------------------------------------------------- | ------------------------------------- | +| `https://docs.gitlab.com/ee/api/avatar.html` | `ee/api/avatar.html` | +| `https://docs.gitlab.com/ee/install/index.html` | `ee/install/` | +| `https://docs.gitlab.com/omnibus/settings/database.html` | `omnibus/settings/database.html` | +| `https://docs.gitlab.com/charts/installation/deployment.html` | `charts/installation/deployment.html` | +| `https://docs.gitlab.com/runner/install/docker.html` | `runner/install/docker.html` | ### Layout file (logic) @@ -318,74 +271,15 @@ The [layout](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/layouts/globa is fed by the [data file](#data-file), builds the global nav, and is rendered by the [default](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/layouts/default.html) layout. -There are three main considerations on the logic built for the nav: - -- [Path](#path): first-level directories underneath `docs.gitlab.com/`: - - `https://docs.gitlab.com/ce/` - - `https://docs.gitlab.com/ee/` - - `https://docs.gitlab.com/omnibus/` - - `https://docs.gitlab.com/runner/` - - `https://docs.gitlab.com/*` -- [EE-only](#ee-only-docs): documentation only available in `/ee/`, not on `/ce/`, e.g.: - - `https://docs.gitlab.com/ee/user/group/epics/` - - `https://docs.gitlab.com/ee/user/application_security/security_dashboard/index.html` -- [Default URL](#default-url): between CE and EE docs, the default is `ee`, therefore, all docs - should link to `/ee/` unless if on `/ce/` linking internally to `ce`. - -#### Path - -To use relative paths in the data file, we defined the variable `dir` -from the root's first-child directory, which defines the path to build -all the nav links to other pages: - -```html -<% dir = @item.identifier.to_s[%r{(?<=/)[^/]+}] %> -``` - -For instance, for `https://docs.gitlab.com/ee/user/index.html`, -`dir` == `ee`, and for `https://docs.gitlab.com/omnibus/README.html`, -`dir` == `omnibus`. - -#### Default URL - -The default and canonical URL for GitLab documentation is -`https://docs.gitlab.com/ee/`, thus, all links -in the docs site should link to `/ee/` except when linking -among `/ce/` docs themselves. - -Therefore, if the user is looking at `/ee/`, `/omnibus/`, -`/runner/`, or any other highest-level dir, the nav should -point to `/ee/` docs. - -On the other hand, if the user is looking at `/ce/` docs, -all the links in the CE nav should link internally to `/ce/` -files. - -```html -<% if dir != 'ce' %> - <a href="/ee/<%= sec[:section_url] %>">...</a> - <% else %> - <a href="/<%= dir %>/<%= sec[:section_url] %>">...</a> - <% end %> - ... -<% end %> -``` - -This also allows the nav to be displayed on other -highest-level directories (`/omnibus/`, `/runner/`, etc), -linking them back to `/ee/`. - -The same logic is applied to all sections (`sec[:section_url]`), -categories (`cat[:category_url]`), and docs (`doc[:doc_url]`) URLs. - -#### `ee-only` docs - -Docs for features present only in GitLab EE are tagged -in the data file by `ee-only` and an icon is displayed on the nav -link indicating that the `ee-only` feature is not available in CE. +The global nav contains links from all [four upstream projects](index.md#architecture). +The [global nav URL](#urls) has a different prefix depending on the documentation file you change. -The `ee-only` attribute is available for `categories` (`<% if cat[:ee_only] %>`) -and `docs` (`<% if doc[:ee_only] %>`), but not for `sections`. +| Repository | Link prefix | Final URL | +|----------------------------------------------------------------|-------------|-----------| +| <https://gitlab.com/gitlab-org/gitlab/tree/master/doc> | `ee/` |`https://docs.gitlab.com/ee/` | +| <https://gitlab.com/gitlab-org/omnibus-gitlab/tree/master/doc> | `omnibus/` |`https://docs.gitlab.com/omnibus/` | +| <https://gitlab.com/gitlab-org/gitlab-runner/tree/master/docs> | `runner/` |`https://docs.gitlab.com/runner/` | +| <https://gitlab.com/charts/gitlab/tree/master/doc> | `charts/` |`https://docs.gitlab.com/charts/` | ### CSS classes diff --git a/doc/development/documentation/site_architecture/index.md b/doc/development/documentation/site_architecture/index.md index 70fa80b3306..35e9ab5157b 100644 --- a/doc/development/documentation/site_architecture/index.md +++ b/doc/development/documentation/site_architecture/index.md @@ -119,7 +119,7 @@ pipeline in the main `gitlab` repository as well as in `gitlab-docs`. Create an a different name first and test it to ensure you do not break the pipelines. 1. In [`gitlab-docs`](https://gitlab.com/gitlab-org/gitlab-docs), go to **{rocket}** **CI/CD > Pipelines**. -1. Click the **Run Pipeline** button. +1. Click the **Run pipeline** button. 1. See that a new pipeline is running. The jobs that build the images are in the first stage, `build-images`. You can click the pipeline number to see the larger pipeline graph, or click the first (`build-images`) stage in the mini pipeline graph to @@ -231,7 +231,7 @@ for its search function. This is how it works: ### Algolia notes for GitLab team members -If you’re a GitLab team member, find credentials for the Algolia dashboard +If you're a GitLab team member, find credentials for the Algolia dashboard in the shared [GitLab 1Password account](https://about.gitlab.com/handbook/security/#1password-for-teams). To receive weekly reports of the search usage, search the Google doc with title `Email, Slack, and GitLab Groups and Aliases`, search for `docsearch`, diff --git a/doc/development/documentation/site_architecture/release_process.md b/doc/development/documentation/site_architecture/release_process.md index 7bdf3fbdcf8..9329b93bb26 100644 --- a/doc/development/documentation/site_architecture/release_process.md +++ b/doc/development/documentation/site_architecture/release_process.md @@ -1,168 +1,8 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: 'https://about.gitlab.com/handbook/engineering/ux/technical-writing/workflow/#monthly-documentation-releases' --- -# Monthly release process +This file was moved to [another location](https://about.gitlab.com/handbook/engineering/ux/technical-writing/workflow/#monthly-documentation-releases). -When a new GitLab version is released on the 22nd, we need to release the published documentation -for the new version. - -This should be done as soon as possible after the GitLab version is announced, so that: - -- The published documentation includes the three most recent minor releases of the current major - version, and the most recent minor releases of the last two major versions. For example 13.9, - 13.8, 13.7, 12.10, and 11.11. -- Documentation updates after the 22nd are for the next release. The versions drop down - should have the current milestone with `-pre` appended to it, for example `13.10-pre`. - -Each documentation release: - -- Has a dedicated branch, named in the format `XX.yy`. -- Has a Docker image that contains a build of that branch. - -For example: - -- For [GitLab 13.9](https://docs.gitlab.com/13.9/index.html), the - [stable branch](https://gitlab.com/gitlab-org/gitlab-docs/-/tree/13.9) and Docker image: - [`registry.gitlab.com/gitlab-org/gitlab-docs:13.9`](https://gitlab.com/gitlab-org/gitlab-docs/container_registry/631635). -- For [GitLab 13.8](https://docs.gitlab.com/13.8/index.html), the - [stable branch](https://gitlab.com/gitlab-org/gitlab-docs/-/tree/13.8) and Docker image: - [`registry.gitlab.com/gitlab-org/gitlab-docs:13.8`](https://gitlab.com/gitlab-org/gitlab-docs/container_registry/631635). - -To set up a documentation release, follow these steps: - -1. [Add the charts version](#add-chart-version), so that the documentation is built using the - [version of the charts project that maps to](https://docs.gitlab.com/charts/installation/version_mappings.html) - the GitLab release. This step may have been completed already. -1. [Create a stable branch and Docker image](#create-stable-branch-and-docker-image-for-release) for - the new version. -1. [Create a release merge request](#create-release-merge-request) for the new version, which - updates the version dropdown menu for the current documentation and adds the release to the - Docker configuration. For example, the - [release merge request for 13.9](https://gitlab.com/gitlab-org/gitlab-docs/-/merge_requests/1555). -1. [Update the three online versions](#update-dropdown-for-online-versions), so that they display the new release on their - version dropdown menus. For example: - - The merge request to [update the 13.9 version dropdown menu for the 13.9 release](https://gitlab.com/gitlab-org/gitlab-docs/-/merge_requests/1556). - - The merge request to [update the 13.8 version dropdown menu for the 13.9 release](https://gitlab.com/gitlab-org/gitlab-docs/-/merge_requests/1557). - - The merge request to [update the 13.7 version dropdown menu for the 13.9 release](https://gitlab.com/gitlab-org/gitlab-docs/-/merge_requests/1558). -1. [Merge the release merge request and run the necessary Docker image builds](#merge-release-merge-request-and-run-docker-image-builds). - -## Add chart version - -To add a new charts version for the release: - -1. Make sure you're in the root path of the `gitlab-docs` repository. -1. Open `content/_data/chart_versions.yaml` and add the new stable branch version using the - [version mapping](https://docs.gitlab.com/charts/installation/version_mappings.html). Only the - `major.minor` version is needed. -1. Create a new merge request and merge it. - -NOTE: -If you have time, add anticipated future mappings to `content/_data/chart_versions.yaml`. This saves -a step for the next GitLab release. - -## Create stable branch and Docker image for release - -To create a stable branch and Docker image for the release: - -1. Make sure you're in the root path of the `gitlab-docs` repository. -1. Run the Rake task to create the single version. For example, to create the 13.9 release branch - and perform others tasks: - - ```shell - ./bin/rake "release:single[13.9]" - ``` - - A branch for the release is created, a new `Dockerfile.13.9` is created, and `.gitlab-ci.yml` - has branches variables updated into a new branch. These files are automatically committed. - -1. Push the newly created branch, but **don't create a merge request**. After you push, the - `image:docs-single` job creates a new Docker image tagged with the name of the branch you created - earlier. You can see the Docker image in the `registry` environment at - <https://gitlab.com/gitlab-org/gitlab-docs/-/environments/folders/registry>. - -For example, see [the 13.9 release pipeline](https://gitlab.com/gitlab-org/gitlab-docs/-/pipelines/260288747). - -Optionally, you can test locally by: - -1. Building the image and running it. For example, for GitLab 13.9 documentation: - - ```shell - docker build -t docs:13.9 -f Dockerfile.13.9 . - docker run -it --rm -p 4000:4000 docs:13.9 - ``` - -1. Visiting <http://localhost:4000/13.9/> to see if everything works correctly. - -## Create release merge request - -NOTE: -An [epic is open](https://gitlab.com/groups/gitlab-org/-/epics/4361) to automate this step. - -To create the release merge request for the release: - -1. Make sure you're in the root path of the `gitlab-docs` repository. -1. Create a branch `release-X-Y`. For example: - - ```shell - git checkout master - git checkout -b release-13-9 - ``` - -1. Edit `content/_data/versions.yaml` and update the lists of versions to reflect the new release: - - - Add the latest version to the `online:` section. - - Move the oldest version in `online:` to the `offline:` section. There should now be three - versions in `online:`. - -1. Update these Dockerfiles: - - - `dockerfiles/Dockerfile.archives`: Add the latest version to the top of the list. - - `Dockerfile.master`: Remove the oldest version, and add the newest version to the - top of the list. - -1. Commit and push to create the merge request. For example: - - ```shell - git add content/ Dockerfile.master dockerfiles/Dockerfile.archives - git commit -m "Release 13.9" - git push origin release-13-9 - ``` - -Do not merge the release merge request yet. - -## Update dropdown for online versions - -To update`content/_data/versions.yaml` for all online versions (stable branches `X.Y` of the -`gitlab-docs` project): - -1. Run the Rake task that creates all of the necessary merge requests to update the dropdowns. For - example, for the 13.9 release: - - ```shell - git checkout release-13-9 - ./bin/rake release:dropdowns - ``` - - These merge requests are set to automatically merge. - -1. [Visit the merge requests page](https://gitlab.com/gitlab-org/gitlab-docs/-/merge_requests?label_name%5B%5D=release) - to check that their pipelines pass. After all MRs are merged, proceed to the following and final - step. - -## Merge release merge request and run Docker image builds - -The merge requests for the dropdowns should now all be merged into their respective stable branches. -Each merge triggers a new pipeline for each stable branch. Wait for the stable branch pipelines to -complete, then: - -1. Check the [pipelines page](https://gitlab.com/gitlab-org/gitlab-docs/pipelines) - and make sure all stable branches have green pipelines. -1. After all the pipelines succeed, merge the [release merge request](#create-release-merge-request). -1. Finally, run the - [`Build docker images weekly` pipeline](https://gitlab.com/gitlab-org/gitlab-docs/pipeline_schedules) - that builds the `:latest` and `:archives` Docker images. - -As the last step in the scheduled pipeline, the documentation site deploys with all new versions. +<!-- This redirect file can be deleted after <2021-07-12>. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/documentation/structure.md b/doc/development/documentation/structure.md index 65949d5b5f5..33bfc040bb6 100644 --- a/doc/development/documentation/structure.md +++ b/doc/development/documentation/structure.md @@ -52,9 +52,9 @@ A concept should answer the questions: - What is this? - Why would I use it? -Think of everything someone might want to know if they’ve never heard of this topic before. +Think of everything someone might want to know if they've never heard of this topic before. -Don’t tell them **how** to do this thing. Tell them **what it is**. +Don't tell them **how** to do this thing. Tell them **what it is**. If you start describing another topic, start a new concept and link to it. @@ -113,7 +113,7 @@ To create an issue: 1. Go to **Issues > List**. 1. In the top right, click **New issue**. 1. Complete the fields. (If you have a reference topic that lists each field, link to it here.) -1. Click **Submit issue**. +1. Click **Create issue**. The issue is created. You can view it by going to **Issues > List**. ``` @@ -153,6 +153,14 @@ This issue occurs when... The workaround is... ``` +For the topic title: + +- Consider including at least a partial error message in the title. +- Use fewer than 70 characters. + +Remember to include the complete error message in the topics content if it is +not complete in the title. + ## Other information on a topic Topics include other information. diff --git a/doc/development/documentation/styleguide/index.md b/doc/development/documentation/styleguide/index.md index 4831e5bbce5..25cdbaf75ba 100644 --- a/doc/development/documentation/styleguide/index.md +++ b/doc/development/documentation/styleguide/index.md @@ -608,7 +608,7 @@ Follow these guidelines for punctuation: | Do not use tabs for indentation. Use spaces instead. You can configure your code editor to output spaces instead of tabs when pressing the tab key. | --- | | Use serial commas (_Oxford commas_) before the final _and_ or _or_ in a list of three or more items. (Tested in [`OxfordComma.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/OxfordComma.yml).) | _You can create new issues, merge requests, and milestones._ | | Always add a space before and after dashes when using it in a sentence (for replacing a comma, for example). | _You should try this - or not._ | -| Always use lowercase after a colon. | _Related Issues: a way to create a relationship between issues._ | +| Always use lowercase after a colon. | Linked issues: a way to create a relationship between issues._ | <!-- vale gitlab.Repetition = YES --> diff --git a/doc/development/documentation/testing.md b/doc/development/documentation/testing.md index eed544911cb..aee3144e6de 100644 --- a/doc/development/documentation/testing.md +++ b/doc/development/documentation/testing.md @@ -166,7 +166,7 @@ You can use markdownlint: ### Vale -[Vale](https://errata-ai.gitbook.io/vale/) is a grammar, style, and word usage linter for the +[Vale](https://docs.errata.ai/vale/about/) is a grammar, style, and word usage linter for the English language. Vale's configuration is stored in the [`.vale.ini`](https://gitlab.com/gitlab-org/gitlab/blob/master/.vale.ini) file located in the root directory of projects. @@ -188,7 +188,7 @@ This configuration is also used in build pipelines, where You can use Vale: -- [On the command line](https://errata-ai.gitbook.io/vale/getting-started/usage). +- [On the command line](https://docs.errata.ai/vale/cli). - [In a code editor](#configure-editors). - [In a Git hook](#configure-pre-push-hooks). Vale only reports errors in the Git hook (the same configuration as the CI/CD pipelines), and does not report suggestions or warnings. @@ -333,4 +333,4 @@ document: Whenever possible, exclude only the problematic rule and line(s). For more information, see -[Vale's documentation](https://errata-ai.gitbook.io/vale/getting-started/markup#markup-based-configuration). +[Vale's documentation](https://docs.errata.ai/vale/scoping#markup-based-configuration). diff --git a/doc/development/ee_features.md b/doc/development/ee_features.md index 81014b7624c..35e7824721a 100644 --- a/doc/development/ee_features.md +++ b/doc/development/ee_features.md @@ -23,6 +23,8 @@ when no license is active. So EE features always should be guarded by `project.feature_available?` or `group.feature_available?` (or `License.feature_available?` if it is a system-wide feature). +Frontend features should be guarded by pushing a flag from the backend by [using `push_licensed_feature`](licensed_feature_availability.md#restricting-frontend-features), and checked using `this.glFeatures.someFeature` in the frontend. + CE specs should remain untouched as much as possible and extra specs should be added for EE. Licensed features can be stubbed using the spec helper `stub_licensed_features` in `EE::LicenseHelpers`. diff --git a/doc/development/elasticsearch.md b/doc/development/elasticsearch.md index 3f14ca454fe..705a69765f7 100644 --- a/doc/development/elasticsearch.md +++ b/doc/development/elasticsearch.md @@ -198,7 +198,10 @@ filename format, which is similar to Rails database migrations: # frozen_string_literal: true class MigrationName < Elastic::Migration - # Important: Any update to the Elastic index mappings should be replicated in Elastic::Latest::Config + # Important: Any updates to the Elastic index mappings must be replicated in the respective + # configuration files: + # - `Elastic::Latest::Config`, for the main index. + # - `Elastic::Latest::<Type>Config`, for standalone indices. def migrate end @@ -214,7 +217,10 @@ Applied migrations are stored in `gitlab-#{RAILS_ENV}-migrations` index. All mig are applied by the [`Elastic::MigrationWorker`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/app/workers/elastic/migration_worker.rb) cron worker sequentially. -Any update to the Elastic index mappings should be replicated in [`Elastic::Latest::Config`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/elastic/latest/config.rb). +To update Elastic index mappings, apply the configuration to the respective files: + +- For the main index: [`Elastic::Latest::Config`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/elastic/latest/config.rb). +- For standalone indices: `Elastic::Latest::<Type>Config`. Migrations can be built with a retry limit and have the ability to be [failed and marked as halted](https://gitlab.com/gitlab-org/gitlab/-/blob/66e899b6637372a4faf61cfd2f254cbdd2fb9f6d/ee/lib/elastic/migration.rb#L40). Any data or index cleanup needed to support migration retries should be handled within the migration. @@ -235,6 +241,10 @@ cron worker runs. Default value is 5 minutes. - `pause_indexing!` - Pause indexing while the migration runs. This setting will record the indexing setting before the migration runs and set it back to that value when the migration is completed. +- `space_requirements!` - Verify that enough free space is available in the cluster when the migration runs. This setting + will halt the migration if the storage required is not available when the migration runs. The migration must provide + the space required in bytes by defining a `space_required_bytes` method. + ```ruby # frozen_string_literal: true @@ -242,7 +252,9 @@ class BatchedMigrationName < Elastic::Migration # Declares a migration should be run in batches batched! throttle_delay 10.minutes - + pause_indexing! + space_requirements! + # ... end ``` diff --git a/doc/development/emails.md b/doc/development/emails.md index 1de1da33dc2..3e651a6efb8 100644 --- a/doc/development/emails.md +++ b/doc/development/emails.md @@ -54,9 +54,12 @@ See the [Rails guides](https://guides.rubyonrails.org/action_mailer_basics.html# incoming_email: enabled: true - # The email address including the %{key} placeholder that will be replaced to reference the item being replied to. This %{key} should be included in its entirety within the email address and not replaced by another value. - # For example: emailadress+%key@gmail.com. - # The placeholder can be omitted but if present, it must appear in the "user" part of the address (before the `@`). + # The email address including the %{key} placeholder that will be replaced to reference the + # item being replied to. This %{key} should be included in its entirety within the email + # address and not replaced by another value. + # For example: emailadress+%{key}@gmail.com. + # The placeholder must appear in the "user" part of the address (before the `@`). It can be omitted but some features, + # including Service Desk, may not work properly. address: "gitlab-incoming+%{key}@gmail.com" # Email account username diff --git a/doc/development/experiment_guide/gitlab_experiment.md b/doc/development/experiment_guide/gitlab_experiment.md index 6b15449b812..4b93105ea50 100644 --- a/doc/development/experiment_guide/gitlab_experiment.md +++ b/doc/development/experiment_guide/gitlab_experiment.md @@ -17,8 +17,8 @@ You're strongly encouraged to read and understand the [Feature flags in development of GitLab](../feature_flags/index.md) portion of the documentation before considering running experiments. Experiments add additional concepts which may seem confusing or advanced without understanding the underpinnings -of how GitLab uses feature flags in development. One concept: GLEX supports multivariate -experiments, which are sometimes referred to as A/B/n tests. +of how GitLab uses feature flags in development. One concept: GLEX supports +experiments with multiple variants, which are sometimes referred to as A/B/n tests. The [`gitlab-experiment` project](https://gitlab.com/gitlab-org/gitlab-experiment) exists in a separate repository, so it can be shared across any GitLab property that uses @@ -49,7 +49,7 @@ graph TD Running? -->|No| Excluded[Control / No Tracking] Cached? -->|No| Excluded? Cached? -->|Yes| Cached[Cached Value] - Excluded? -->|Yes / Cached| Excluded + Excluded? -->|Yes| Excluded Excluded? -->|No| Segmented? Segmented? -->|Yes / Cached| VariantA Segmented? -->|No| Included?[Experiment Group?] @@ -92,7 +92,7 @@ end ``` When this code executes, the experiment is run, a variant is assigned, and (if within a -controller or view) a `window.gon.experiment.pillColor` object will be available in the +controller or view) a `window.gon.experiment.pill_color` object will be available in the client layer, with details like: - The assigned variant. @@ -367,7 +367,7 @@ about contexts now. We can assume we run the experiment in one or a few places, but track events potentially in many places. The tracking call remains the same, with the arguments you would normally use when -[tracking events using snowplow](../snowplow.md). The easiest example +[tracking events using snowplow](../snowplow/index.md). The easiest example of tracking an event in Ruby would be: ```ruby @@ -501,6 +501,69 @@ Any experiment that's been run in the request lifecycle surfaces in `window.gon. and matches [this schema](https://gitlab.com/gitlab-org/iglu/-/blob/master/public/schemas/com.gitlab/gitlab_experiment/jsonschema/1-0-0) so you can use it when resolving some concepts around experimentation in the client layer. +### Use experiments in Vue + +With the `experiment` component, you can define slots that match the name of the +variants pushed to `window.gon.experiment`. For example, if we alter the `pill_color` +experiment to just use the default variants of `control` and `candidate` like so: + +```ruby +def show + experiment(:pill_color) do |e| + e.use { } # control + e.try { } # candidate + end.run +end +``` + +We can make use of the named slots `control` and `candidate` in the Vue component: + +```vue +<script> +import Experiment from '~/experimentation/components/experiment.vue'; + +export default { + components: { Experiment } +} +</script> + +<template> + <experiment name="pill_color"> + <template #control> + <button class="bg-default">Click default button</button> + </template> + + <template #candidate> + <button class="bg-red">Click red button</button> + </template> + </experiment> +</template> +``` + +When you're coding for an experiment with multiple variants, you can use the variant names. +For example, the Vue component for the previously-defined `pill_color` experiment with `red` and `blue` variants would look like this: + +```vue +<template> + <experiment name="pill_color"> + <template #control> + <button class="bg-default">Click default button</button> + </template> + + <template #red> + <button class="bg-red">Click red button</button> + </template> + + <template #blue> + <button class="bg-blue">Click blue button</button> + </template> + </experiment> +</template> +``` + +NOTE: +When there is no experiment data in the `window.gon.experiment` object for the given experiment name, the `control` slot will be used, if it exists. + ## Notes on feature flags NOTE: @@ -525,7 +588,7 @@ Conditional means that it returns `true` in some situations, but not all situati When a feature flag is disabled (meaning the state is `off`), the experiment is considered _inactive_. You can visualize this in the [decision tree diagram](#how-it-works) -as reaching the first [Running?] node, and traversing the negative path. +as reaching the first `Running?` node, and traversing the negative path. When a feature flag is rolled out to a `percentage_of_actors` or similar (meaning the state is `conditional`) the experiment is considered to be _running_ diff --git a/doc/development/fe_guide/accessibility.md b/doc/development/fe_guide/accessibility.md index 5f22c13ca06..ab1325c67a9 100644 --- a/doc/development/fe_guide/accessibility.md +++ b/doc/development/fe_guide/accessibility.md @@ -15,39 +15,264 @@ This page contains guidelines we should follow. Since [no ARIA is better than bad ARIA](https://www.w3.org/TR/wai-aria-practices/#no_aria_better_bad_aria), review the following recommendations before using `aria-*`, `role`, and `tabindex`. -Use semantic HTML, which typically has accessibility semantics baked in, but always be sure to test with +Use semantic HTML, which has accessibility semantics baked in, and ideally test with [relevant combinations of screen readers and browsers](https://www.accessibility-developer-guide.com/knowledge/screen-readers/relevant-combinations/). In [WebAIM's accessibility analysis of the top million home pages](https://webaim.org/projects/million/#aria), they found that "ARIA correlated to higher detectable errors". It is likely that *misuse* of ARIA is a big cause of increased errors, -so when in doubt don't use `aria-*`, `role`, and `tabindex`, and stick with semantic HTML. - -## Provide accessible names to screen readers +so when in doubt don't use `aria-*`, `role`, and `tabindex` and stick with semantic HTML. + +## Quick checklist + +- [Text](#text-inputs-with-accessible-names), + [select](#select-inputs-with-accessible-names), + [checkbox](#checkbox-inputs-with-accessible-names), + [radio](#radio-inputs-with-accessible-names), + [file](#file-inputs-with-accessible-names), + and [toggle](#gltoggle-components-with-an-accessible-names) inputs have accessible names. +- [Buttons](#buttons-and-links-with-descriptive-accessible-names), + [links](#buttons-and-links-with-descriptive-accessible-names), + and [images](#images-with-accessible-names) have descriptive accessible names. +- Icons + - [Non-decorative icons](#icons-that-convey-information) have an `aria-label`. + - [Clickable icons](#icons-that-are-clickable) are buttons, that is, `<gl-button icon="close" />` is used and not `<gl-icon />`. + - Icon-only buttons have an `aria-label`. +- Interactive elements can be [accessed with the Tab key](#support-keyboard-only-use) and have a visible focus state. +- Are any `role`, `tabindex` or `aria-*` attributes unnecessary? +- Can any `div` or `span` elements be replaced with a more semantic [HTML element](https://developer.mozilla.org/en-US/docs/Web/HTML/Element) like `p`, `button`, or `time`? + +## Provide accessible names for screen readers To provide markup with accessible names, ensure every: - `input` has an associated `label`. -- `button` and `a` have child text, or `aria-label` when text isn’t present. - For example, an icon button with no visible text. +- `button` and `a` have child text, or `aria-label` when child text isn't present, such as for an icon button with no content. - `img` has an `alt` attribute. - `fieldset` has `legend` as its first child. - `figure` has `figcaption` as its first child. - `table` has `caption` as its first child. +Groups of checkboxes and radio inputs should be grouped together in a `fieldset` with a `legend`. +`legend` gives the group of checkboxes and radio inputs a label. + If the `label`, child text, or child element is not visually desired, use `.gl-sr-only` to hide the element from everything but screen readers. -Ensure the accessible name is descriptive enough to be understood in isolation. +### Examples of providing accessible names + +The following subsections contain examples of markup that render HTML elements with accessible names. + +Note that [when using `GlFormGroup`](https://bootstrap-vue.org/docs/components/form-group#accessibility): + +- Passing only a `label` prop renders a `fieldset` with a `legend` containing the `label` value. +- Passing both a `label` and a `label-for` prop renders a `label` that points to the form input with the same `label-for` ID. + +#### Text inputs with accessible names + +When using `GlFormGroup`, the `label` prop alone does not give the input an accessible name. +The `label-for` prop must also be provided to give the input an accessible name. + +Text input examples: + +```html +<!-- Input with label --> +<gl-form-group :label="__('Issue title')" label-for="issue-title"> + <gl-form-input id="issue-title" v-model="title" name="title" /> +</gl-form-group> + +<!-- Input with hidden label --> +<gl-form-group :label="__('Issue title')" label-for="issue-title" :label-sr-only="true"> + <gl-form-input id="issue-title" v-model="title" name="title" /> +</gl-form-group> +``` + +Textarea examples: + +```html +<!-- Textarea with label --> +<gl-form-group :label="__('Issue description')" label-for="issue-description"> + <gl-form-textarea id="issue-description" v-model="description" name="description" /> +</gl-form-group> + +<!-- Textarea with hidden label --> +<gl-form-group :label="__('Issue description')" label-for="issue-description" :label-sr-only="true"> + <gl-form-textarea id="issue-description" v-model="description" name="description" /> +</gl-form-group> +``` + +Alternatively, you can use a plain `label` element: + +```html +<!-- Input with label using `label` --> +<label for="issue-title">{{ __('Issue title') }}</label> +<gl-form-input id="issue-title" v-model="title" name="title" /> + +<!-- Input with hidden label using `label` --> +<label for="issue-title" class="gl-sr-only">{{ __('Issue title') }}</label> +<gl-form-input id="issue-title" v-model="title" name="title" /> +``` + +#### Select inputs with accessible names + +Select input examples: + +```html +<!-- Select input with label --> +<gl-form-group :label="__('Issue status')" label-for="issue-status"> + <gl-form-select id="issue-status" v-model="status" name="status" :options="options" /> +</gl-form-group> + +<!-- Select input with hidden label --> +<gl-form-group :label="__('Issue status')" label-for="issue-status" :label-sr-only="true"> + <gl-form-select id="issue-status" v-model="status" name="status" :options="options" /> +</gl-form-group> +``` + +#### Checkbox inputs with accessible names + +Single checkbox: + +```html +<!-- Single checkbox with label --> +<gl-form-checkbox v-model="status" name="status" value="task-complete"> + {{ __('Task complete') }} +</gl-form-checkbox> + +<!-- Single checkbox with hidden label --> +<gl-form-checkbox v-model="status" name="status" value="task-complete"> + <span class="gl-sr-only">{{ __('Task complete') }}</span> +</gl-form-checkbox> +``` + +Multiple checkboxes: + +```html +<!-- Multiple labeled checkboxes grouped within a fieldset --> +<gl-form-group :label="__('Task list')"> + <gl-form-checkbox name="task-list" value="task-1">{{ __('Task 1') }}</gl-form-checkbox> + <gl-form-checkbox name="task-list" value="task-2">{{ __('Task 2') }}</gl-form-checkbox> +</gl-form-group> + +<!-- Or --> +<gl-form-group :label="__('Task list')"> + <gl-form-checkbox-group v-model="selected" :options="options" name="task-list" /> +</gl-form-group> + +<!-- Multiple labeled checkboxes grouped within a fieldset with hidden legend --> +<gl-form-group :label="__('Task list')" :label-sr-only="true"> + <gl-form-checkbox name="task-list" value="task-1">{{ __('Task 1') }}</gl-form-checkbox> + <gl-form-checkbox name="task-list" value="task-2">{{ __('Task 2') }}</gl-form-checkbox> +</gl-form-group> + +<!-- Or --> +<gl-form-group :label="__('Task list')" :label-sr-only="true"> + <gl-form-checkbox-group v-model="selected" :options="options" name="task-list" /> +</gl-form-group> +``` + +#### Radio inputs with accessible names + +Single radio input: + +```html +<!-- Single radio with a label --> +<gl-form-radio v-model="status" name="status" value="opened"> + {{ __('Opened') }} +</gl-form-radio> + +<!-- Single radio with a hidden label --> +<gl-form-radio v-model="status" name="status" value="opened"> + <span class="gl-sr-only">{{ __('Opened') }}</span> +</gl-form-radio> +``` + +Multiple radio inputs: + +```html +<!-- Multiple labeled radio inputs grouped within a fieldset --> +<gl-form-group :label="__('Issue status')"> + <gl-form-radio name="status" value="opened">{{ __('Opened') }}</gl-form-radio> + <gl-form-radio name="status" value="closed">{{ __('Closed') }}</gl-form-radio> +</gl-form-group> + +<!-- Or --> +<gl-form-group :label="__('Issue status')"> + <gl-form-radio-group v-model="selected" :options="options" name="status" /> +</gl-form-group> + +<!-- Multiple labeled radio inputs grouped within a fieldset with hidden legend --> +<gl-form-group :label="__('Issue status')" :label-sr-only="true"> + <gl-form-radio name="status" value="opened">{{ __('Opened') }}</gl-form-radio> + <gl-form-radio name="status" value="closed">{{ __('Closed') }}</gl-form-radio> +</gl-form-group> + +<!-- Or --> +<gl-form-group :label="__('Issue status')" :label-sr-only="true"> + <gl-form-radio-group v-model="selected" :options="options" name="status" /> +</gl-form-group> +``` + +#### File inputs with accessible names + +File input examples: + +```html +<!-- File input with a label --> +<label for="attach-file">{{ __('Attach a file') }}</label> +<input id="attach-file" type="file" name="attach-file" /> + +<!-- File input with a hidden label --> +<label for="attach-file" class="gl-sr-only">{{ __('Attach a file') }}</label> +<input id="attach-file" type="file" name="attach-file" /> +``` + +#### GlToggle components with an accessible names + +`GlToggle` examples: ```html -// bad -<button>Submit</button> -<a href="url">page</a> +<!-- GlToggle with label --> +<gl-toggle v-model="notifications" :label="__('Notifications')" /> -// good -<button>Submit review</button> -<a href="url">GitLab's accessibility page</a> +<!-- GlToggle with hidden label --> +<gl-toggle v-model="notifications" :label="__('Notifications')" label-position="hidden" /> +``` + +#### GlFormCombobox components with an accessible names + +`GlFormCombobox` examples: + +```html +<!-- GlFormCombobox with label --> +<gl-form-combobox :label-text="__('Key')" :token-list="$options.tokenList" /> +``` + +#### Images with accessible names + +Image examples: + +```html +<img :src="imagePath" :alt="__('A description of the image')" /> + +<!-- SVGs implicitly have a graphics role so if it is semantically an image we should apply `role="img"` --> +<svg role="img" :alt="__('A description of the image')" /> +``` + +#### Buttons and links with descriptive accessible names + +Buttons and links should have accessible names that are descriptive enough to be understood in isolation. + +```html +<!-- bad --> +<gl-button @click="handleClick">{{ __('Submit') }}</gl-button> + +<gl-link :href="url">{{ __('page') }}</gl-link> + +<!-- good --> +<gl-button @click="handleClick">{{ __('Submit review') }}</gl-button> + +<gl-link :href="url">{{ __("GitLab's accessibility page") }}</gl-link> ``` ## Role @@ -81,31 +306,37 @@ element is interactive you must ensure: Use semantic HTML, such as `a` and `button`, which provides these behaviours by default. +Keep in mind that: + +- <kbd>Tab</kbd> and <kbd>Shift-Tab</kbd> should only move between interactive elements, not static content. +- When you add `:hover` styles, in most cases you should add `:focus` styles too so that the styling is applied for both mouse **and** keyboard users. +- If you remove an interactive element's `outline`, make sure you maintain visual focus state in another way such as with `box-shadow`. + See the [Pajamas Keyboard-only page](https://design.gitlab.com/accessibility-audits/2-keyboard-only/) for more detail. ## Tabindex Prefer **no** `tabindex` to using `tabindex`, since: -- Using semantic HTML such as `button` implicitly provides `tabindex="0"` -- Tabbing order should match the visual reading order and positive `tabindex`s interfere with this +- Using semantic HTML such as `button` implicitly provides `tabindex="0"`. +- Tabbing order should match the visual reading order and positive `tabindex`s interfere with this. ### Avoid using `tabindex="0"` to make an element interactive -Use interactive elements instead of `div`s and `span`s. +Use interactive elements instead of `div` and `span` tags. For example: -- If the element should be clickable, use a `button` -- If the element should be text editable, use an `input` or `textarea` +- If the element should be clickable, use a `button`. +- If the element should be text editable, use an `input` or `textarea`. Once the markup is semantically complete, use CSS to update it to its desired visual state. ```html -// bad +<!-- bad --> <div role="button" tabindex="0" @click="expand">Expand</div> -// good -<button @click="expand">Expand</button> +<!-- good --> +<gl-button @click="expand">Expand</gl-button> ``` ### Do not use `tabindex="0"` on interactive elements @@ -113,13 +344,13 @@ Once the markup is semantically complete, use CSS to update it to its desired vi Interactive elements are already tab accessible so adding `tabindex` is redundant. ```html -// bad -<a href="help" tabindex="0">Help</a> -<button tabindex="0">Submit</button> +<!-- bad --> +<gl-link href="help" tabindex="0">Help</gl-link> +<gl-button tabindex="0">Submit</gl-button> -// good -<a href="help">Help</a> -<button>Submit</button> +<!-- good --> +<gl-link href="help">Help</gl-link> +<gl-button>Submit</gl-button> ``` ### Do not use `tabindex="0"` on elements for screen readers to read @@ -129,10 +360,10 @@ The use of `tabindex="0"` is unnecessary and can cause problems, as screen reader users then expect to be able to interact with it. ```html -// bad -<span tabindex="0" :aria-label="message">{{ message }}</span> +<!-- bad --> +<p tabindex="0" :aria-label="message">{{ message }}</p> -// good +<!-- good --> <p>{{ message }}</p> ``` @@ -141,6 +372,57 @@ as screen reader users then expect to be able to interact with it. [Always avoid using `tabindex="1"`](https://webaim.org/techniques/keyboard/tabindex#overview) or greater. +## Icons + +Icons can be split into three different types: + +- Icons that are decorative +- Icons that convey meaning +- Icons that are clickable + +### Icons that are decorative + +Icons are decorative when there's no loss of information to the user when they are removed from the UI. + +As the majority of icons within GitLab are decorative, `GlIcon` automatically hides its rendered icons from screen readers. +Therefore, you do not need to add `aria-hidden="true"` to `GlIcon`, as this is redundant. + +```html +<!-- unnecessary — gl-icon hides icons from screen readers by default --> +<gl-icon name="rocket" aria-hidden="true" />` + +<!-- good --> +<gl-icon name="rocket" />` +``` + +### Icons that convey information + +Icons convey information if there is loss of information to the user when they are removed from the UI. + +An example is a confidential icon that conveys the issue is confidential, and does not have the text "Confidential" next to it. + +Icons that convey information must have an accessible name so that the information is conveyed to screen reader users too. + +```html +<!-- bad --> +<gl-icon name="eye-slash" />` + +<!-- good --> +<gl-icon name="eye-slash" :aria-label="__('Confidential issue')" />` +``` + +### Icons that are clickable + +Icons that are clickable are semantically buttons, so they should be rendered as buttons, with an accessible name. + +```html +<!-- bad --> +<gl-icon name="close" :aria-label="__('Close')" @click="handleClick" /> + +<!-- good --> +<gl-button icon="close" category="tertiary" :aria-label="__('Close')" @click="handleClick" /> +``` + ## Hiding elements Use the following table to hide elements from users, when appropriate. @@ -158,22 +440,24 @@ If the image is not an `img` element, such as an inline SVG, you can hide it by unnecessary when using `gl-icon`. ```html -// good - decorative images hidden from screen readers +<!-- good - decorative images hidden from screen readers --> + <img src="decorative.jpg" alt=""> -<svg role="img" alt=""> -<gl-icon name="epic"/> + +<svg role="img" alt="" /> + +<gl-icon name="epic" /> ``` -## When should ARIA be used +## When to use ARIA -No ARIA is required when using semantic HTML because it incorporates accessibility. +No ARIA is required when using semantic HTML, because it already incorporates accessibility. -However, there are some UI patterns and widgets that do not have semantic HTML equivalents. +However, there are some UI patterns that do not have semantic HTML equivalents. +General examples of these are dialogs (modals) and tabs. +GitLab-specific examples are assignee and label dropdowns. Building such widgets require ARIA to make them understandable to screen readers. -Proper research and testing should be done to ensure compliance with ARIA. - -Ideally, these widgets would exist only in [GitLab UI](https://gitlab-org.gitlab.io/gitlab-ui/). -Use of ARIA would then only occur in [GitLab UI](https://gitlab.com/gitlab-org/gitlab-ui/) and not [GitLab](https://gitlab.com/gitlab-org/gitlab/). +Proper research and testing should be done to ensure compliance with [WCAG](https://www.w3.org/WAI/standards-guidelines/wcag/). ## Resources diff --git a/doc/development/fe_guide/dependencies.md b/doc/development/fe_guide/dependencies.md index bf46e8e16ce..e8e251baafc 100644 --- a/doc/development/fe_guide/dependencies.md +++ b/doc/development/fe_guide/dependencies.md @@ -26,7 +26,7 @@ production assets post-compile. We use the [Renovate GitLab Bot](https://gitlab.com/gitlab-org/frontend/renovate-gitlab-bot) to automatically create merge requests for updating dependencies of several projects. -You can find the up-to-date list of projects managed by the renovate bot in the project’s README. +You can find the up-to-date list of projects managed by the renovate bot in the project's README. Some key dependencies updated using renovate are: diff --git a/doc/development/fe_guide/design_anti_patterns.md b/doc/development/fe_guide/design_anti_patterns.md index d230e413879..ee4fceff927 100644 --- a/doc/development/fe_guide/design_anti_patterns.md +++ b/doc/development/fe_guide/design_anti_patterns.md @@ -30,7 +30,7 @@ const createStore = () => new Vuex.Store({ // Notice that we are forcing all references to this module to use the same single instance of the store. // We are also creating the store at import-time and there is nothing which can automatically dispose of it. // -// As an alternative, we should simply export the `createStore` and let the client manage the +// As an alternative, we should export the `createStore` and let the client manage the // lifecycle and instance of the store. export default createStore(); ``` @@ -129,7 +129,7 @@ Here are some ills that Singletons often produce: This is because of the limitations of languages like Java where everything has to be wrapped in a class. In JavaScript we have things like object and function literals where we can solve -many problems with a module that simply exports utility functions. +many problems with a module that exports utility functions. ### When could the Singleton pattern be actually appropriate?** diff --git a/doc/development/fe_guide/graphql.md b/doc/development/fe_guide/graphql.md index 2e812d9fa0a..e87b4197269 100644 --- a/doc/development/fe_guide/graphql.md +++ b/doc/development/fe_guide/graphql.md @@ -43,13 +43,9 @@ can help you learn how to integrate Vue Apollo. For other use cases, check out the [Usage outside of Vue](#usage-outside-of-vue) section. -<!-- vale gitlab.Spelling = NO --> - -We use [Immer](https://immerjs.github.io/immer/docs/introduction) for immutable cache updates; +We use [Immer](https://immerjs.github.io/immer/) for immutable cache updates; see [Immutability and cache updates](#immutability-and-cache-updates) for more information. -<!-- vale gitlab.Spelling = YES --> - ### Tooling <!-- vale gitlab.Spelling = NO --> @@ -173,14 +169,10 @@ const primaryKeyId = getIdFromGraphQLId(data.id); From Apollo version 3.0.0 all the cache updates need to be immutable. It needs to be replaced entirely with a **new and updated** object. -<!-- vale gitlab.Spelling = NO --> - To facilitate the process of updating the cache and returning the new object we -use the library [Immer](https://immerjs.github.io/immer/docs/introduction). +use the library [Immer](https://immerjs.github.io/immer/). When possible, follow these conventions: -<!-- vale gitlab.Spelling = YES --> - - The updated cache is named `data`. - The original cache data is named `sourceData`. @@ -238,17 +230,33 @@ Read more about [Vue Apollo](https://github.com/vuejs/vue-apollo) in the [Vue Ap It is possible to manage an application state with Apollo by passing in a resolvers object when creating the default client. The default state can be set by writing -to the cache after setting up the default client. +to the cache after setting up the default client. In the example below, we are using query with `@client` Apollo directive to write the initial data to Apollo cache and then get this state in the Vue component: + +```javascript +// user.query.graphql + +query User { + user @client { + name + surname + age + } +} +``` ```javascript +// index.js + import Vue from 'vue'; import VueApollo from 'vue-apollo'; import createDefaultClient from '~/lib/graphql'; +import userQuery from '~/user/user.query.graphql' Vue.use(VueApollo); const defaultClient = createDefaultClient(); -defaultClient.cache.writeData({ +defaultClient.cache.writeQuery({ + query: userQuery, data: { user: { name: 'John', @@ -263,16 +271,15 @@ const apolloProvider = new VueApollo({ }); ``` -We can query local data with `@client` Apollo directive: - ```javascript -// user.query.graphql +// App.vue +import userQuery from '~/user/user.query.graphql' -query User { - user @client { - name - surname - age +export default { + apollo: { + user: { + query: userQuery + } } } ``` @@ -767,6 +774,66 @@ export default { }; ``` +#### Polling and Performance + +While the Apollo client has support for simple polling, for performance reasons, our [Etag-based caching](../polling.md) is preferred to hitting the database each time. + +Once the backend is set up, there are a few changes to make on the frontend. + +First, get your resource Etag path from the backend. In the example of the pipelines graph, this is called the `graphql_resource_etag`. This will be used to create new headers to add to the Apollo context: + +```javascript +/* pipelines/components/graph/utils.js */ + +/* eslint-disable @gitlab/require-i18n-strings */ +const getQueryHeaders = (etagResource) => { + return { + fetchOptions: { + method: 'GET', + }, + headers: { + /* This will depend on your feature */ + 'X-GITLAB-GRAPHQL-FEATURE-CORRELATION': 'verify/ci/pipeline-graph', + 'X-GITLAB-GRAPHQL-RESOURCE-ETAG': etagResource, + 'X-REQUESTED-WITH': 'XMLHttpRequest', + }, + }; +}; +/* eslint-enable @gitlab/require-i18n-strings */ + +/* component.vue */ + +apollo: { + pipeline: { + context() { + return getQueryHeaders(this.graphqlResourceEtag); + }, + query: getPipelineDetails, + pollInterval: 10000, + .. + }, +}, +``` + +Then, because Etags depend on the request being a `GET` instead of GraphQL's usual `POST`, but our default link library does not support `GET` we need to let our defaut Apollo client know to use a different library. + +```javascript +/* componentMountIndex.js */ + +const apolloProvider = new VueApollo({ + defaultClient: createDefaultClient( + {}, + { + useGet: true, + }, + ), +}); +``` + +Keep in mind, this means your app will not batch queries. + +Once subscriptions are mature, this process can be replaced by using them and we can remove the separate link library and return to batching queries. + ### Testing #### Generating the GraphQL schema @@ -1377,6 +1444,34 @@ describe('My Index test with `createMockApollo`', () => { }); ``` +When you need to configure the mocked apollo client's caching behavior, +provide additional cache options when creating a mocked client instance and the provided options will merge with the default cache option: + +```javascript +const defaultCacheOptions = { + fragmentMatcher: { match: () => true }, + addTypename: false, +}; +``` + +```javascript +function createMockApolloProvider({ props = {}, requestHandlers } = {}) { + Vue.use(VueApollo); + + const mockApollo = createMockApollo( + requestHandlers, + {}, + { + dataIdFromObject: (object) => + // eslint-disable-next-line no-underscore-dangle + object.__typename === 'Requirement' ? object.iid : defaultDataIdFromObject(object), + }, + ); + + return mockApollo; +} +``` + ## Handling errors The GitLab GraphQL mutations have two distinct error modes: [Top-level](#top-level-errors) and [errors-as-data](#errors-as-data). diff --git a/doc/development/fe_guide/icons.md b/doc/development/fe_guide/icons.md index a7b62fbb267..ad344c00efd 100644 --- a/doc/development/fe_guide/icons.md +++ b/doc/development/fe_guide/icons.md @@ -104,7 +104,7 @@ by the examples that follow: **Example 1:** -The following HAML expression generates a loading icon’s markup and +The following HAML expression generates a loading icon's markup and centers the icon by wrapping it in a `gl-spinner-container` element. ```haml @@ -121,7 +121,7 @@ centers the icon by wrapping it in a `gl-spinner-container` element. **Example 2:** -The following HAML expression generates a loading icon’s markup +The following HAML expression generates a loading icon's markup with a custom size. It also appends a margin utility class. ```haml @@ -137,7 +137,7 @@ with a custom size. It also appends a margin utility class. ### Usage in Vue The [GitLab UI](https://gitlab-org.gitlab.io/gitlab-ui/) components library provides a -`GlLoadingIcon` component. See the component’s +`GlLoadingIcon` component. See the component's [storybook](https://gitlab-org.gitlab.io/gitlab-ui/?path=/story/base-loading-icon--default) for more information about its usage. diff --git a/doc/development/fe_guide/img/editor_lite_create_ext.png b/doc/development/fe_guide/img/editor_lite_create_ext.png Binary files differindex 73941cf5d62..9092c4b725c 100644 --- a/doc/development/fe_guide/img/editor_lite_create_ext.png +++ b/doc/development/fe_guide/img/editor_lite_create_ext.png diff --git a/doc/development/fe_guide/index.md b/doc/development/fe_guide/index.md index 1315520342e..0f3754c29e7 100644 --- a/doc/development/fe_guide/index.md +++ b/doc/development/fe_guide/index.md @@ -11,7 +11,7 @@ across the GitLab frontend team. ## Overview -GitLab is built on top of [Ruby on Rails](https://rubyonrails.org). It uses [Haml](https://haml.info/) and a JavaScript0based frontend with [Vue.js](https://vuejs.org). +GitLab is built on top of [Ruby on Rails](https://rubyonrails.org). It uses [Haml](https://haml.info/) and a JavaScript-based frontend with [Vue.js](https://vuejs.org). <!-- vale gitlab.Spelling = NO --> Be wary of [the limitations that come with using Hamlit](https://github.com/k0kubun/hamlit/blob/master/REFERENCE.md#limitations). <!-- vale gitlab.Spelling = YES --> diff --git a/doc/development/fe_guide/performance.md b/doc/development/fe_guide/performance.md index 795de87d309..b6130335654 100644 --- a/doc/development/fe_guide/performance.md +++ b/doc/development/fe_guide/performance.md @@ -35,7 +35,7 @@ performance.getEntriesByName('my-component-start') - The start of navigation and a mark - The start of navigation and the moment the measurement is taken -It takes several arguments of which the measurement’s name is the only one required. Examples: +It takes several arguments of which the measurement's name is the only one required. Examples: - Duration between the start and end marks: @@ -185,16 +185,16 @@ To access stored measurements, you can use either: ### Naming convention All the marks and measures should be instantiated with the constants from -`app/assets/javascripts/performance/constants.js`. When you’re ready to add a new mark’s or -measurement’s label, you can follow the pattern. +`app/assets/javascripts/performance/constants.js`. When you're ready to add a new mark's or +measurement's label, you can follow the pattern. NOTE: This pattern is a recommendation and not a hard rule. ```javascript -app-*-start // for a start ‘mark’ -app-*-end // for an end ‘mark’ -app-* // for ‘measure’ +app-*-start // for a start 'mark' +app-*-end // for an end 'mark' +app-* // for 'measure' ``` For example, `'webide-init-editor-start`, `mr-diffs-mark-file-tree-end`, and so on. We do it to @@ -381,7 +381,7 @@ Use `webpackChunkName` when generating dynamic imports as it provides a deterministic filename for the chunk which can then be cached in the browser across GitLab versions. -More information is available in [webpack's code splitting documentation](https://webpack.js.org/guides/code-splitting/#dynamic-imports). +More information is available in [webpack's code splitting documentation](https://webpack.js.org/guides/code-splitting/#dynamic-imports) and [vue's dynamic component documentation](https://vuejs.org/v2/guide/components-dynamic-async.html). ### Minimizing page size diff --git a/doc/development/fe_guide/principles.md b/doc/development/fe_guide/principles.md index 4952d023d90..1bf37c8d008 100644 --- a/doc/development/fe_guide/principles.md +++ b/doc/development/fe_guide/principles.md @@ -18,4 +18,4 @@ There are multiple ways of writing code to accomplish the same results. We shoul ## Improve code [iteratively](https://about.gitlab.com/handbook/values/#iteration) -Whenever you see existing code that does not follow our current style guide, update it proactively. You don’t need to fix everything, but each merge request should iteratively improve our codebase, and reduce technical debt where possible. +Whenever you see existing code that does not follow our current style guide, update it proactively. You don't need to fix everything, but each merge request should iteratively improve our codebase, and reduce technical debt where possible. diff --git a/doc/development/fe_guide/security.md b/doc/development/fe_guide/security.md index 1a6646df877..79452327673 100644 --- a/doc/development/fe_guide/security.md +++ b/doc/development/fe_guide/security.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w ## Resources -[Mozilla’s HTTP Observatory CLI](https://github.com/mozilla/http-observatory-cli) and +[Mozilla's HTTP Observatory CLI](https://github.com/mozilla/http-observatory-cli) and [Qualys SSL Labs Server Test](https://www.ssllabs.com/ssltest/analyze.html) are good resources for finding potential problems and ensuring compliance with security best practices. @@ -41,7 +41,7 @@ Security Policy headers in the GitLab Rails app. Some resources on implementing Content Security Policy: - [MDN Article on CSP](https://developer.mozilla.org/en-US/docs/Web/Security/CSP) -- [GitHub’s CSP Journey on the GitHub Engineering Blog](http://githubengineering.com/githubs-csp-journey/) +- [GitHub's CSP Journey on the GitHub Engineering Blog](http://githubengineering.com/githubs-csp-journey/) - The Dropbox Engineering Blog's series on CSP: [1](https://blogs.dropbox.com/tech/2015/09/on-csp-reporting-and-filtering/), [2](https://blogs.dropbox.com/tech/2015/09/unsafe-inline-and-nonce-deployment/), [3](https://blogs.dropbox.com/tech/2015/09/csp-the-unexpected-eval/), [4](https://blogs.dropbox.com/tech/2015/09/csp-third-party-integrations-and-privilege-separation/) ### Subresource Integrity (SRI) diff --git a/doc/development/fe_guide/style/javascript.md b/doc/development/fe_guide/style/javascript.md index 334372af1f4..a2035eb1b55 100644 --- a/doc/development/fe_guide/style/javascript.md +++ b/doc/development/fe_guide/style/javascript.md @@ -98,16 +98,27 @@ class a { } ``` -## Use ParseInt +## Converting Strings to Integers -Use `ParseInt` when converting a numeric string into a number. +When converting strings to integers, `parseInt` has a slight performance advantage over `Number`, but `Number` is semantic and can be more readable. Prefer `parseInt`, but do not discourage `Number` if it significantly helps readability. + +**WARNING:** `parseInt` **must** include the [radix argument](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/parseInt). ```javascript // bad -Number('10') +parseInt('10'); + +// bad +things.map(parseInt) + +// ok +Number("106") + +// good +things.map(Number) // good -parseInt('10', 10); +parseInt("106", 10) ``` ## CSS Selectors - Use `js-` prefix @@ -297,7 +308,7 @@ Strive to write many small pure functions and minimize where mutations occur ## Export constants as primitives -Prefer exporting constant primitives with a common namespace over exporting objects. This allows for better compile-time reference checks and helps to avoid accidential `undefined`s at runtime. In addition, it helps in reducing bundle sizes. +Prefer exporting constant primitives with a common namespace over exporting objects. This allows for better compile-time reference checks and helps to avoid accidental `undefined`s at runtime. In addition, it helps in reducing bundle sizes. Only export the constants as a collection (array, or object) when there is a need to iterate over them, for instance, for a prop validator. diff --git a/doc/development/fe_guide/style/vue.md b/doc/development/fe_guide/style/vue.md index d62145b4a4c..93e4f234ccb 100644 --- a/doc/development/fe_guide/style/vue.md +++ b/doc/development/fe_guide/style/vue.md @@ -615,7 +615,7 @@ component state wherever possible. Instead, set the component's ``` 1. When asserting multiple props, check the deep equality of the `props()` object with -[`toEqual`](https://jestjs.io/docs/en/expect#toequalvalue): +[`toEqual`](https://jestjs.io/docs/expect#toequalvalue): ```javascript // good @@ -632,8 +632,8 @@ component state wherever possible. Instead, set the component's ``` 1. If you are only interested in some of the props, you can use -[`toMatchObject`](https://jestjs.io/docs/en/expect#tomatchobjectobject). Prefer `toMatchObject` -over [`expect.objectContaining`](https://jestjs.io/docs/en/expect#expectobjectcontainingobject): +[`toMatchObject`](https://jestjs.io/docs/expect#tomatchobjectobject). Prefer `toMatchObject` +over [`expect.objectContaining`](https://jestjs.io/docs/expect#expectobjectcontainingobject): ```javascript // good diff --git a/doc/development/fe_guide/troubleshooting.md b/doc/development/fe_guide/troubleshooting.md index 250fe5106d3..1b3991ee80d 100644 --- a/doc/development/fe_guide/troubleshooting.md +++ b/doc/development/fe_guide/troubleshooting.md @@ -66,3 +66,29 @@ TypeError: $ is not a function ``` **Remedy - Try moving the script into a separate repository and point to it to files in the GitLab repository** + +## Using Vue component issues + +### When rendering a component that uses GlFilteredSearch and the component or its parent uses Vue Apollo + +When trying to render our component GlFilteredSearch, you might get an error in the component's `provide` function: + +`cannot read suggestionsListClass of undefined` + +Currently, `vue-apollo` tries to [manually call a component's `provide()` in the `beforeCreate` part](https://github.com/vuejs/vue-apollo/blob/35e27ec398d844869e1bbbde73c6068b8aabe78a/packages/vue-apollo/src/mixin.js#L149) of the component lifecycle. This means that when a `provide()` references props, which aren't actually setup until after `created`, it will blow up. + +See this [closed MR](https://gitlab.com/gitlab-org/gitlab-ui/-/merge_requests/2019#note_514671251) for more context. + +**Remedy - try providing `apolloProvider` to the top-level Vue instance options** + +VueApollo will skip manually running `provide()` if it sees that an `apolloProvider` is provided in the `$options`. + +```patch + new Vue( + el, ++ apolloProvider: {}, + render(h) { + return h(App); + }, + ); +``` diff --git a/doc/development/fe_guide/vue.md b/doc/development/fe_guide/vue.md index 220a4a107aa..574faa0898f 100644 --- a/doc/development/fe_guide/vue.md +++ b/doc/development/fe_guide/vue.md @@ -99,6 +99,86 @@ return new Vue({ > When adding an `id` attribute to mount a Vue application, please make sure this `id` is unique across the codebase. +#### Providing Rails form fields to Vue applications + +When composing a form with Rails, the `name`, `id`, and `value` attributes of form inputs are generated +to match the backend. It can be helpful to have access to these generated attributes when converting +a Rails form to Vue, or when [integrating components (datepicker, project selector, etc)](https://gitlab.com/gitlab-org/gitlab/-/blob/8956ad767d522f37a96e03840595c767de030968/app/assets/javascripts/access_tokens/index.js#L15) into it. +The [`parseRailsFormFields`](https://gitlab.com/gitlab-org/gitlab/-/blob/fe88797f682c7ff0b13f2c2223a3ff45ada751c1/app/assets/javascripts/lib/utils/forms.js#L107) utility can be used to parse the generated form input attributes so they can be passed to the Vue application. +This allows us to easily integrate Vue components without changing how the form submits. + +```haml +-# form.html.haml += form_for user do |form| + .js-user-form + = form.text_field :name, class: 'form-control gl-form-input', data: { js_name: 'name' } + = form.text_field :email, class: 'form-control gl-form-input', data: { js_name: 'email' } +``` + +> The `js_name` data attribute is used as the key in the resulting JavaScript object. +For example `= form.text_field :email, data: { js_name: 'fooBarBaz' }` would be translated +to `{ fooBarBaz: { name: 'user[email]', id: 'user_email', value: '' } }` + +```javascript +// index.js +import Vue from 'vue'; +import { parseRailsFormFields } from '~/lib/utils/forms'; +import UserForm from './components/user_form.vue'; + +export const initUserForm = () => { + const el = document.querySelector('.js-user-form'); + + if (!el) { + return null; + } + + const fields = parseRailsFormFields(el); + + return new Vue({ + el, + render(h) { + return h(UserForm, { + props: { + fields, + }, + }); + }, + }); +}; +``` + +```vue +<script> +// user_form.vue +import { GlButton, GlFormGroup, GlFormInput } from '@gitlab/ui'; + +export default { + name: 'UserForm', + components: { GlButton, GlFormGroup, GlFormInput }, + props: { + fields: { + type: Object, + required: true, + }, + }, +}; +</script> + +<template> + <div> + <gl-form-group :label-for="fields.name.id" :label="__('Name')"> + <gl-form-input v-bind="fields.name" size="lg" /> + </gl-form-group> + + <gl-form-group :label-for="fields.email.id" :label="__('Email')"> + <gl-form-input v-bind="fields.email" type="email" size="lg" /> + </gl-form-group> + + <gl-button type="submit" category="primary" variant="confirm">{{ __('Update') }}</gl-button> + </div> +</template> +``` + #### Accessing the `gl` object We query the `gl` object for data that doesn't change during the application's life @@ -197,7 +277,7 @@ Check this [page](vuex.md) for more details. In the [Vue documentation](https://vuejs.org/v2/api/#Options-Data) the Data function/object is defined as follows: > The data object for the Vue instance. Vue recursively converts its properties into getter/setters -to make it “reactive”. The object must be plain: native objects such as browser API objects and +to make it "reactive". The object must be plain: native objects such as browser API objects and prototype properties are ignored. A rule of thumb is that data should just be data - it is not recommended to observe objects with their own stateful behavior. diff --git a/doc/development/fe_guide/vue3_migration.md b/doc/development/fe_guide/vue3_migration.md index ee25e97ab6e..d06e93da0f3 100644 --- a/doc/development/fe_guide/vue3_migration.md +++ b/doc/development/fe_guide/vue3_migration.md @@ -6,6 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Migration to Vue 3 +Preparations for a Vue 3 migration are tracked in epic [&3174](https://gitlab.com/groups/gitlab-org/-/epics/3174) + In order to prepare for the eventual migration to Vue 3.x, we should be wary about adding the following features to the codebase: ## Vue filters diff --git a/doc/development/feature_flags/controls.md b/doc/development/feature_flags/controls.md index fc327a2defc..c9538c38850 100644 --- a/doc/development/feature_flags/controls.md +++ b/doc/development/feature_flags/controls.md @@ -36,7 +36,7 @@ easier to measure the impact of both separately. The GitLab feature library (using [Flipper](https://github.com/jnunemaker/flipper), and covered in the [Feature -Flags process](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle) guide) supports rolling out changes to a percentage of +Flags process](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/) guide) supports rolling out changes to a percentage of time to users. This in turn can be controlled using [GitLab ChatOps](../../ci/chatops/index.md). For an up to date list of feature flag commands please see [the source @@ -281,7 +281,7 @@ To remove a feature flag, open **one merge request** to make the changes. In the 1. Add the ~"feature flag" label so release managers are aware the changes are hidden behind a feature flag. 1. If the merge request has to be picked into a stable branch, add the appropriate `~"Pick into X.Y"` label, for example `~"Pick into 13.0"`. - See [the feature flag process](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle#including-a-feature-behind-feature-flag-in-the-final-release) + See [the feature flag process](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#including-a-feature-behind-feature-flag-in-the-final-release) for further details. 1. Remove all references to the feature flag from the codebase, including tests. 1. Remove the YAML definition for the feature from the repository. diff --git a/doc/development/feature_flags/index.md b/doc/development/feature_flags/index.md index 5c98dc2e473..560e4f8cb90 100644 --- a/doc/development/feature_flags/index.md +++ b/doc/development/feature_flags/index.md @@ -16,7 +16,7 @@ in the GitLab codebase to conditionally enable features and test them. Features that are developed and merged behind a feature flag -should not include a changelog entry. The entry should be added either in the merge +should not include a changelog entry. A changelog entry with `type: added` should be included in the merge request removing the feature flag or the merge request where the default value of the feature flag is set to enabled. If the feature contains any database migrations, it *should* include a changelog entry for the database changes. @@ -292,8 +292,7 @@ end ### Frontend -Use the `push_frontend_feature_flag` method for frontend code, which is -available to all controllers that inherit from `ApplicationController`. You can use +Use the `push_frontend_feature_flag` method which is available to all controllers that inherit from `ApplicationController`. You can use this method to expose the state of a feature flag, for example: ```ruby @@ -424,7 +423,7 @@ Feature.enabled?(:licensed_feature_feature_flag, project) && ### Feature groups Feature groups must be defined statically in `lib/feature.rb` (in the -`.register_feature_groups` method), but their implementation can obviously be +`.register_feature_groups` method), but their implementation can be dynamic (querying the DB, for example). Once defined in `lib/feature.rb`, you can to activate a diff --git a/doc/development/geo/framework.md b/doc/development/geo/framework.md index 055b9ce4ad6..2b3e4826de5 100644 --- a/doc/development/geo/framework.md +++ b/doc/development/geo/framework.md @@ -160,880 +160,16 @@ the Geo team if you are unsure. ### Blob Replicator Strategy -Models that use -[CarrierWave's](https://github.com/carrierwaveuploader/carrierwave) `Uploader::Base` -can be easily supported by Geo with the `Geo::BlobReplicatorStrategy` module. +Models that use [CarrierWave's](https://github.com/carrierwaveuploader/carrierwave) `Uploader::Base` are supported by Geo with the `Geo::BlobReplicatorStrategy` module. For example, see how [Geo replication was implemented for Pipeline Artifacts](https://gitlab.com/gitlab-org/gitlab/-/issues/238464). -First, each file should have its own primary ID and model. Geo strongly -recommends treating *every single file* as a first-class citizen, because in -our experience this greatly simplifies tracking replication and verification -state. +Each file is expected to have its own primary ID and model. Geo strongly recommends treating *every single file* as a first-class citizen, because in our experience this greatly simplifies tracking replication and verification state. -For example, to add support for files referenced by a `Widget` model with a -`widgets` table, you would perform the following steps: - -#### Replication - -1. Include `Gitlab::Geo::ReplicableModel` in the `Widget` class, and specify - the Replicator class `with_replicator Geo::WidgetReplicator`. - - At this point the `Widget` class should look like this: - - ```ruby - # frozen_string_literal: true - - class Widget < ApplicationRecord - include ::Gitlab::Geo::ReplicableModel - - with_replicator Geo::WidgetReplicator - - mount_uploader :file, WidgetUploader - - # @param primary_key_in [Range, Widget] arg to pass to primary_key_in scope - # @return [ActiveRecord::Relation<Widget>] everything that should be synced to this node, restricted by primary key - def self.replicables_for_current_secondary(primary_key_in) - # Should be implemented. The idea of the method is to restrict - # the set of synced items depending on synchronization settings - end - ... - end - ``` - - If there is a common constraint for records to be available for replication, - make sure to also overwrite the `available_replicables` scope. - -1. Create `ee/app/replicators/geo/widget_replicator.rb`. Implement the - `#carrierwave_uploader` method which should return a `CarrierWave::Uploader`, - and implement the class method `.model` to return the `Widget` class: - - ```ruby - # frozen_string_literal: true - - module Geo - class WidgetReplicator < Gitlab::Geo::Replicator - include ::Geo::BlobReplicatorStrategy - - def self.model - ::Widget - end - - def carrierwave_uploader - model_record.file - end - - # The feature flag follows the format `geo_#{replicable_name}_replication`, - # so here it would be `geo_widget_replication` - def self.replication_enabled_by_default? - false - end - end - end - ``` - -1. Add this replicator class to the method `replicator_classes` in - `ee/lib/gitlab/geo.rb`: - - ```ruby - REPLICATOR_CLASSES = [ - ::Geo::PackageFileReplicator, - ::Geo::WidgetReplicator - ] - end - ``` - -1. Create `ee/spec/replicators/geo/widget_replicator_spec.rb` and perform - the necessary setup to define the `model_record` variable for the shared - examples: - - ```ruby - # frozen_string_literal: true - - require 'spec_helper' - - RSpec.describe Geo::WidgetReplicator do - let(:model_record) { build(:widget) } - - it_behaves_like 'a blob replicator' - end - ``` - -1. Create the `widget_registry` table, with columns ordered according to [our guidelines](../ordering_table_columns.md) so Geo secondaries can track the sync and - verification state of each Widget's file. This migration belongs in `ee/db/geo/migrate`: - - ```ruby - # frozen_string_literal: true - - class CreateWidgetRegistry < ActiveRecord::Migration[6.0] - include Gitlab::Database::MigrationHelpers - - DOWNTIME = false - - disable_ddl_transaction! - - def up - unless table_exists?(:widget_registry) - ActiveRecord::Base.transaction do - create_table :widget_registry, id: :bigserial, force: :cascade do |t| - t.bigint :widget_id, null: false - t.datetime_with_timezone :created_at, null: false - t.datetime_with_timezone :last_synced_at - t.datetime_with_timezone :retry_at - t.datetime_with_timezone :verified_at - t.datetime_with_timezone :verification_started_at - t.datetime_with_timezone :verification_retry_at - t.integer :state, default: 0, null: false, limit: 2 - t.integer :verification_state, default: 0, null: false, limit: 2 - t.integer :retry_count, default: 0, limit: 2 - t.integer :verification_retry_count, default: 0, limit: 2 - t.boolean :checksum_mismatch - t.binary :verification_checksum - t.binary :verification_checksum_mismatched - t.string :verification_failure, limit: 255 - t.string :last_sync_failure, limit: 255 - - t.index :widget_id, name: :index_widget_registry_on_widget_id, unique: true - t.index :retry_at - t.index :state - # To optimize performance of WidgetRegistry.verification_failed_batch - t.index :verification_retry_at, name: :widget_registry_failed_verification, order: "NULLS FIRST", where: "((state = 2) AND (verification_state = 3))" - # To optimize performance of WidgetRegistry.needs_verification_count - t.index :verification_state, name: :widget_registry_needs_verification, where: "((state = 2) AND (verification_state = ANY (ARRAY[0, 3])))" - # To optimize performance of WidgetRegistry.verification_pending_batch - t.index :verified_at, name: :widget_registry_pending_verification, order: "NULLS FIRST", where: "((state = 2) AND (verification_state = 0))" - end - end - end - end - - def down - drop_table :widget_registry - end - end - ``` - -1. Create `ee/app/models/geo/widget_registry.rb`: - - ```ruby - # frozen_string_literal: true - - class Geo::WidgetRegistry < Geo::BaseRegistry - include ::Geo::ReplicableRegistry - include ::Geo::VerifiableRegistry - - MODEL_CLASS = ::Widget - MODEL_FOREIGN_KEY = :widget_id - - belongs_to :widget, class_name: 'Widget' - end - ``` - -1. Update `REGISTRY_CLASSES` in `ee/app/workers/geo/secondary/registry_consistency_worker.rb`. -1. Add `widget_registry` to `ActiveSupport::Inflector.inflections` in `config/initializers_before_autoloader/000_inflections.rb`. -1. Create `ee/spec/factories/geo/widget_registry.rb`: - - ```ruby - # frozen_string_literal: true - - FactoryBot.define do - factory :geo_widget_registry, class: 'Geo::WidgetRegistry' do - widget - state { Geo::WidgetRegistry.state_value(:pending) } - - trait :synced do - state { Geo::WidgetRegistry.state_value(:synced) } - last_synced_at { 5.days.ago } - end - - trait :failed do - state { Geo::WidgetRegistry.state_value(:failed) } - last_synced_at { 1.day.ago } - retry_count { 2 } - last_sync_failure { 'Random error' } - end - - trait :started do - state { Geo::WidgetRegistry.state_value(:started) } - last_synced_at { 1.day.ago } - retry_count { 0 } - end - end - end - ``` - -1. Create `ee/spec/models/geo/widget_registry_spec.rb`: - - ```ruby - # frozen_string_literal: true - - require 'spec_helper' - - RSpec.describe Geo::WidgetRegistry, :geo, type: :model do - let_it_be(:registry) { create(:geo_widget_registry) } - - specify 'factory is valid' do - expect(registry).to be_valid - end - - include_examples 'a Geo framework registry' - include_examples 'a Geo verifiable registry' - - describe '.find_registry_differences' do - ... # To be implemented - end - end - ``` - -Widgets should now be replicated by Geo. - -#### Verification - -There are two ways to add verification related fields so that the Geo primary -can track verification state. - -##### Option 1: Add verification state fields to the existing `widgets` table itself - -1. Add a migration to add columns ordered according to [our guidelines](../ordering_table_columns.md) - for verification state to the widgets table: - - ```ruby - # frozen_string_literal: true - - class AddVerificationStateToWidgets < ActiveRecord::Migration[6.0] - DOWNTIME = false - - def change - change_table(:widgets) do |t| - t.integer :verification_state, default: 0, limit: 2, null: false - t.column :verification_started_at, :datetime_with_timezone - t.integer :verification_retry_count, limit: 2 - t.column :verification_retry_at, :datetime_with_timezone - t.column :verified_at, :datetime_with_timezone - t.binary :verification_checksum, using: 'verification_checksum::bytea' - - # rubocop:disable Migration/AddLimitToTextColumns - t.text :verification_failure - # rubocop:enable Migration/AddLimitToTextColumns - end - end - end - ``` - -1. Adding a `text` column also [requires](../database/strings_and_the_text_data_type.md#add-a-text-column-to-an-existing-table) - setting a limit: - - ```ruby - # frozen_string_literal: true - - class AddVerificationFailureLimitToWidgets < ActiveRecord::Migration[6.0] - include Gitlab::Database::MigrationHelpers - - DOWNTIME = false - - disable_ddl_transaction! - - CONSTRAINT_NAME = 'widget_verification_failure_text_limit' - - def up - add_text_limit :widget, :verification_failure, 255, constraint_name: CONSTRAINT_NAME - end - - def down - remove_check_constraint(:widget, CONSTRAINT_NAME) - end - end - ``` - -1. Add indexes on verification fields to ensure verification can be performed efficiently: - - Some or all of these indexes can be omitted if the table is guaranteed to be small. Ask a database expert if you are unsure. - - ```ruby - # frozen_string_literal: true - - class AddVerificationIndexesToWidgets < ActiveRecord::Migration[6.0] - include Gitlab::Database::MigrationHelpers - - DOWNTIME = false - VERIFICATION_STATE_INDEX_NAME = "index_widgets_verification_state" - PENDING_VERIFICATION_INDEX_NAME = "index_widgets_pending_verification" - FAILED_VERIFICATION_INDEX_NAME = "index_widgets_failed_verification" - NEEDS_VERIFICATION_INDEX_NAME = "index_widgets_needs_verification" - - disable_ddl_transaction! - - def up - add_concurrent_index :widgets, :verification_state, name: VERIFICATION_STATE_INDEX_NAME - add_concurrent_index :widgets, :verified_at, where: "(verification_state = 0)", order: { verified_at: 'ASC NULLS FIRST' }, name: PENDING_VERIFICATION_INDEX_NAME - add_concurrent_index :widgets, :verification_retry_at, where: "(verification_state = 3)", order: { verification_retry_at: 'ASC NULLS FIRST' }, name: FAILED_VERIFICATION_INDEX_NAME - add_concurrent_index :widgets, :verification_state, where: "(verification_state = 0 OR verification_state = 3)", name: NEEDS_VERIFICATION_INDEX_NAME - end - - def down - remove_concurrent_index_by_name :widgets, VERIFICATION_STATE_INDEX_NAME - remove_concurrent_index_by_name :widgets, PENDING_VERIFICATION_INDEX_NAME - remove_concurrent_index_by_name :widgets, FAILED_VERIFICATION_INDEX_NAME - remove_concurrent_index_by_name :widgets, NEEDS_VERIFICATION_INDEX_NAME - end - end - ``` - -1. Add the `Gitlab::Geo::VerificationState` concern to the `widget` model if it is not already included in `Gitlab::Geo::ReplicableModel`: - - ```ruby - class Widget < ApplicationRecord - ... - include ::Gitlab::Geo::VerificationState - ... - end - ``` - -##### Option 2: Create a separate `widget_states` table with verification state fields - -1. Create a `widget_states` table and add an index on `verification_state` to ensure verification can be performed efficiently. Order the columns according to [the guidelines](../ordering_table_columns.md): - - ```ruby - # frozen_string_literal: true - - class CreateWidgetStates < ActiveRecord::Migration[6.0] - include Gitlab::Database::MigrationHelpers - - DOWNTIME = false - - disable_ddl_transaction! - - def up - unless table_exists?(:widget_states) - with_lock_retries do - create_table :widget_states, id: false do |t| - t.references :widget, primary_key: true, null: false, foreign_key: { on_delete: :cascade } - t.integer :verification_state, default: 0, limit: 2, null: false - t.column :verification_started_at, :datetime_with_timezone - t.datetime_with_timezone :verification_retry_at - t.datetime_with_timezone :verified_at - t.integer :verification_retry_count, limit: 2 - t.binary :verification_checksum, using: 'verification_checksum::bytea' - t.text :verification_failure - - t.index :verification_state, name: "index_widget_states_on_verification_state" - end - end - end - - add_text_limit :widget_states, :verification_failure, 255 - end - - def down - drop_table :widget_states - end - end - ``` - -1. Add the following lines to the `widget_state.rb` model: - - ```ruby - class WidgetState < ApplicationRecord - ... - self.primary_key = :widget_id - - include ::Gitlab::Geo::VerificationState - - belongs_to :widget, inverse_of: :widget_state - ... - end - ``` - -1. Add the following lines to the `widget` model: - - ```ruby - class Widget < ApplicationRecord - ... - has_one :widget_state, inverse_of: :widget - - delegate :verification_retry_at, :verification_retry_at=, - :verified_at, :verified_at=, - :verification_checksum, :verification_checksum=, - :verification_failure, :verification_failure=, - :verification_retry_count, :verification_retry_count=, - to: :widget_state - ... - end - ``` - -To do: Add verification on secondaries. This should be done as part of -[Geo: Self Service Framework - First Implementation for Package File verification](https://gitlab.com/groups/gitlab-org/-/epics/1817) - -Widgets should now be verified by Geo. - -#### Metrics - -Metrics are gathered by `Geo::MetricsUpdateWorker`, persisted in -`GeoNodeStatus` for display in the UI, and sent to Prometheus: - -1. Add fields `widgets_count`, `widgets_checksummed_count`, - `widgets_checksum_failed_count`, `widgets_synced_count`, - `widgets_failed_count`, and `widgets_registry_count` to - `GET /geo_nodes/status` example response in - `doc/api/geo_nodes.md`. -1. Add the same fields to `GET /geo_nodes/status` example response in - `ee/spec/fixtures/api/schemas/public_api/v4/geo_node_status.json`. -1. Add fields `geo_widgets`, `geo_widgets_checksummed`, - `geo_widgets_checksum_failed`, `geo_widgets_synced`, - `geo_widgets_failed`, and `geo_widgets_registry` to - `Sidekiq metrics` table in - `doc/administration/monitoring/prometheus/gitlab_metrics.md`. -1. Add the following to the parameterized table in - `ee/spec/models/geo_node_status_spec.rb`: - - ```ruby - Geo::WidgetReplicator | :widget | :geo_widget_registry - ``` - -1. Add the following to `spec/factories/widgets.rb`: - - ```ruby - trait(:verification_succeeded) do - with_file - verification_checksum { 'abc' } - verification_state { Widget.verification_state_value(:verification_succeeded) } - end - - trait(:verification_failed) do - with_file - verification_failure { 'Could not calculate the checksum' } - verification_state { Widget.verification_state_value(:verification_failed) } - end - ``` - -1. Make sure the factory also allows setting a `project` attribute. If the model - does not have a direct relation to a project, you can use a `transient` - attribute. Check out `spec/factories/merge_request_diffs.rb` for an example. - -Widget replication and verification metrics should now be available in the API, -the Admin Area UI, and Prometheus. - -#### GraphQL API - -1. Add a new field to `GeoNodeType` in - `ee/app/graphql/types/geo/geo_node_type.rb`: - - ```ruby - field :widget_registries, ::Types::Geo::WidgetRegistryType.connection_type, - null: true, - resolver: ::Resolvers::Geo::WidgetRegistriesResolver, - description: 'Find widget registries on this Geo node', - feature_flag: :geo_widget_replication - ``` - -1. Add the new `widget_registries` field name to the `expected_fields` array in - `ee/spec/graphql/types/geo/geo_node_type_spec.rb`. -1. Create `ee/app/graphql/resolvers/geo/widget_registries_resolver.rb`: - - ```ruby - # frozen_string_literal: true - - module Resolvers - module Geo - class WidgetRegistriesResolver < BaseResolver - include RegistriesResolver - end - end - end - ``` - -1. Create `ee/spec/graphql/resolvers/geo/widget_registries_resolver_spec.rb`: - - ```ruby - # frozen_string_literal: true - - require 'spec_helper' - - RSpec.describe Resolvers::Geo::WidgetRegistriesResolver do - it_behaves_like 'a Geo registries resolver', :geo_widget_registry - end - ``` - -1. Create `ee/app/finders/geo/widget_registry_finder.rb`: - - ```ruby - # frozen_string_literal: true - - module Geo - class WidgetRegistryFinder - include FrameworkRegistryFinder - end - end - ``` - -1. Create `ee/spec/finders/geo/widget_registry_finder_spec.rb`: - - ```ruby - # frozen_string_literal: true - - require 'spec_helper' - - RSpec.describe Geo::WidgetRegistryFinder do - it_behaves_like 'a framework registry finder', :geo_widget_registry - end - ``` - -1. Create `ee/app/graphql/types/geo/widget_registry_type.rb`: - - ```ruby - # frozen_string_literal: true - - module Types - module Geo - # rubocop:disable Graphql/AuthorizeTypes because it is included - class WidgetRegistryType < BaseObject - include ::Types::Geo::RegistryType - - graphql_name 'WidgetRegistry' - description 'Represents the Geo sync and verification state of a widget' - - field :widget_id, GraphQL::ID_TYPE, null: false, description: 'ID of the Widget' - end - end - end - ``` - -1. Create `ee/spec/graphql/types/geo/widget_registry_type_spec.rb`: - - ```ruby - # frozen_string_literal: true - - require 'spec_helper' - - RSpec.describe GitlabSchema.types['WidgetRegistry'] do - it_behaves_like 'a Geo registry type' - - it 'has the expected fields (other than those included in RegistryType)' do - expected_fields = %i[widget_id] - - expect(described_class).to have_graphql_fields(*expected_fields).at_least - end - end - ``` - -1. Add integration tests for providing Widget registry data to the frontend via - the GraphQL API, by duplicating and modifying the following shared examples - in `ee/spec/requests/api/graphql/geo/registries_spec.rb`: - - ```ruby - it_behaves_like 'gets registries for', { - field_name: 'widgetRegistries', - registry_class_name: 'WidgetRegistry', - registry_factory: :geo_widget_registry, - registry_foreign_key_field_name: 'widgetId' - } - ``` - -1. Update the GraphQL reference documentation: - - ```shell - bundle exec rake gitlab:graphql:compile_docs - ``` - -Individual widget synchronization and verification data should now be available -via the GraphQL API. - -Make sure to replicate the "update" events. Geo Framework does not currently support -replicating "update" events because all entities added to the framework, by this time, -are immutable. If this is the case -for the entity you're going to add, follow <https://gitlab.com/gitlab-org/gitlab/-/issues/118743> -and <https://gitlab.com/gitlab-org/gitlab/-/issues/118745> as examples to add the new event type. -Also, remove this notice when you've added it. - -#### Admin UI - -To do: This should be done as part of -[Geo: Implement frontend for Self-Service Framework replicables](https://gitlab.com/groups/gitlab-org/-/epics/2525) - -Widget sync and verification data (aggregate and individual) should now be -available in the Admin UI. - -#### Releasing the feature - -1. In `ee/config/feature_flags/development/geo_widget_replication.yml`, set `default_enabled: true` - -1. In `ee/app/replicators/geo/widget_replicator.rb`, delete the `self.replication_enabled_by_default?` method: - - ```ruby - module Geo - class WidgetReplicator < Gitlab::Geo::Replicator - ... - - # REMOVE THIS METHOD - def self.replication_enabled_by_default? - false - end - # REMOVE THIS METHOD - - ... - end - end - ``` - -1. In `ee/app/graphql/types/geo/geo_node_type.rb`, remove the `feature_flag` option for the released type: - - ```ruby - field :widget_registries, ::Types::Geo::WidgetRegistryType.connection_type, - null: true, - resolver: ::Resolvers::Geo::WidgetRegistriesResolver, - description: 'Find widget registries on this Geo node', - feature_flag: :geo_widget_replication # REMOVE THIS LINE - ``` +To implement Geo replication of a new blob-type Model, [open an issue with the provided issue template](https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Geo%3A%20Replicate%20a%20new%20blob%20type). ### Repository Replicator Strategy -Models that refer to any repository on the disk -can be easily supported by Geo with the `Geo::RepositoryReplicatorStrategy` module. - -For example, to add support for files referenced by a `Gizmos` model with a -`gizmos` table, you would perform the following steps. - -#### Replication - -1. Include `Gitlab::Geo::ReplicableModel` in the `Gizmo` class, and specify - the Replicator class `with_replicator Geo::GizmoReplicator`. - - At this point the `Gizmo` class should look like this: - - ```ruby - # frozen_string_literal: true - - class Gizmo < ApplicationRecord - include ::Gitlab::Geo::ReplicableModel - - with_replicator Geo::GizmoReplicator - - # @param primary_key_in [Range, Gizmo] arg to pass to primary_key_in scope - # @return [ActiveRecord::Relation<Gizmo>] everything that should be synced to this node, restricted by primary key - def self.replicables_for_current_secondary(primary_key_in) - # Should be implemented. The idea of the method is to restrict - # the set of synced items depending on synchronization settings - end - - # Geo checks this method in FrameworkRepositorySyncService to avoid - # snapshotting repositories using object pools - def pool_repository - nil - end - ... - end - ``` - - Pay some attention to method `pool_repository`. Not every repository type uses - repository pooling. As Geo prefers to use repository snapshotting, it can lead to data loss. - Make sure to overwrite `pool_repository` so it returns nil for repositories that do not - have pools. - - If there is a common constraint for records to be available for replication, - make sure to also overwrite the `available_replicables` scope. - -1. Create `ee/app/replicators/geo/gizmo_replicator.rb`. Implement the - `#repository` method which should return a `<Repository>` instance, - and implement the class method `.model` to return the `Gizmo` class: - - ```ruby - # frozen_string_literal: true - - module Geo - class GizmoReplicator < Gitlab::Geo::Replicator - include ::Geo::RepositoryReplicatorStrategy - - def self.model - ::Gizmo - end - - def repository - model_record.repository - end - - def self.git_access_class - ::Gitlab::GitAccessGizmo - end - - # The feature flag follows the format `geo_#{replicable_name}_replication`, - # so here it would be `geo_gizmo_replication` - def self.replication_enabled_by_default? - false - end - end - end - ``` - -1. Generate the feature flag definition file by running the feature flag command - and running through the steps: - - ```shell - bin/feature-flag --ee geo_gizmo_replication --type development --group 'group::geo' - ``` - -1. Make sure Geo push events are created. Usually it needs some - change in the `app/workers/post_receive.rb` file. Example: - - ```ruby - def replicate_gizmo_changes(gizmo) - if ::Gitlab::Geo.primary? - gizmo.replicator.handle_after_update if gizmo - end - end - ``` - - See `app/workers/post_receive.rb` for more examples. - -1. Make sure the repository removal is also handled. You may need to add something - like the following in the destroy service of the repository: - - ```ruby - gizmo.replicator.handle_after_destroy if gizmo.repository - ``` - -1. Add this replicator class to the method `replicator_classes` in - `ee/lib/gitlab/geo.rb`: - - ```ruby - REPLICATOR_CLASSES = [ - ... - ::Geo::PackageFileReplicator, - ::Geo::GizmoReplicator - ] - end - ``` - -1. Create `ee/spec/replicators/geo/gizmo_replicator_spec.rb` and perform - the necessary setup to define the `model_record` variable for the shared - examples: - - ```ruby - # frozen_string_literal: true - - require 'spec_helper' - - RSpec.describe Geo::GizmoReplicator do - let(:model_record) { build(:gizmo) } - - include_examples 'a repository replicator' - end - ``` - -1. Create the `gizmo_registry` table, with columns ordered according to [our guidelines](../ordering_table_columns.md) so Geo secondaries can track the sync and - verification state of each Gizmo. This migration belongs in `ee/db/geo/migrate`: - - ```ruby - # frozen_string_literal: true - - class CreateGizmoRegistry < ActiveRecord::Migration[6.0] - include Gitlab::Database::MigrationHelpers - - DOWNTIME = false - - disable_ddl_transaction! - - def up - create_table :gizmo_registry, id: :bigserial, force: :cascade do |t| - t.datetime_with_timezone :retry_at - t.datetime_with_timezone :last_synced_at - t.datetime_with_timezone :created_at, null: false - t.bigint :gizmo_id, null: false - t.integer :state, default: 0, null: false, limit: 2 - t.integer :retry_count, default: 0, limit: 2 - t.string :last_sync_failure, limit: 255 - t.boolean :force_to_redownload - t.boolean :missing_on_primary - - t.index :gizmo_id, name: :index_gizmo_registry_on_gizmo_id, unique: true - t.index :retry_at - t.index :state - end - end - - def down - drop_table :gizmo_registry - end - end - ``` - -1. Create `ee/app/models/geo/gizmo_registry.rb`: - - ```ruby - # frozen_string_literal: true - - class Geo::GizmoRegistry < Geo::BaseRegistry - include Geo::ReplicableRegistry - - MODEL_CLASS = ::Gizmo - MODEL_FOREIGN_KEY = :gizmo_id - - belongs_to :gizmo, class_name: 'Gizmo' - end - ``` - -1. Update `REGISTRY_CLASSES` in `ee/app/workers/geo/secondary/registry_consistency_worker.rb`. -1. Add `gizmo_registry` to `ActiveSupport::Inflector.inflections` in `config/initializers_before_autoloader/000_inflections.rb`. -1. Create `ee/spec/factories/geo/gizmo_registry.rb`: - - ```ruby - # frozen_string_literal: true - - FactoryBot.define do - factory :geo_gizmo_registry, class: 'Geo::GizmoRegistry' do - gizmo - state { Geo::GizmoRegistry.state_value(:pending) } - - trait :synced do - state { Geo::GizmoRegistry.state_value(:synced) } - last_synced_at { 5.days.ago } - end - - trait :failed do - state { Geo::GizmoRegistry.state_value(:failed) } - last_synced_at { 1.day.ago } - retry_count { 2 } - last_sync_failure { 'Random error' } - end - - trait :started do - state { Geo::GizmoRegistry.state_value(:started) } - last_synced_at { 1.day.ago } - retry_count { 0 } - end - end - end - ``` - -1. Create `ee/spec/models/geo/gizmo_registry_spec.rb`: - - ```ruby - # frozen_string_literal: true - - require 'spec_helper' - - RSpec.describe Geo::GizmoRegistry, :geo, type: :model do - let_it_be(:registry) { create(:geo_gizmo_registry) } - - specify 'factory is valid' do - expect(registry).to be_valid - end - - include_examples 'a Geo framework registry' - end - ``` - -1. Make sure the newly added repository type can be accessed by a secondary. - You may need to make some changes to one of the Git access classes. - - Gizmos should now be replicated by Geo. - -#### Metrics - -You need to make the same changes as for Blob Replicator Strategy. -You need to make the same changes for the [metrics as in the Blob Replicator Strategy](#metrics). - -#### GraphQL API - -You need to make the same changes for the GraphQL API [as in the Blob Replicator Strategy](#graphql-api). +Models that refer to any Git repository on disk are supported by Geo with the `Geo::RepositoryReplicatorStrategy` module. For example, see how [Geo replication was implemented for Group-level Wikis](https://gitlab.com/gitlab-org/gitlab/-/issues/208147). Note that this issue does not implement verification, since verification of Git repositories was not yet added to the Geo self-service framework. An example implementing verification can be found in the merge request to [Add Snippet repository verification](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/56596). -#### Releasing the feature +Each Git repository is expected to have its own primary ID and model. -You need to make the same changes for [releasing the feature as in the Blob Replicator Strategy](#releasing-the-feature). +To implement Geo replication of a new Git repository-type Model, [open an issue with the provided issue template](https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Geo%3A%20Replicate%20a%20new%20Git%20repository%20type). diff --git a/doc/development/gitaly.md b/doc/development/gitaly.md index 87b9d35f99b..2e814a9c41b 100644 --- a/doc/development/gitaly.md +++ b/doc/development/gitaly.md @@ -303,16 +303,19 @@ Here are the steps to gate a new feature in Gitaly behind a feature flag. ### GitLab Rails -1. Test in a Rails console by setting the feature flag: +Test in a Rails console by setting the feature flag: - NOTE: - Pay attention to the name of the flag and the one used in the Rails console. - There is a difference between them (dashes replaced by underscores and name - prefix is changed). Make sure to prefix all flags with `gitaly_`. +```ruby +Feature.enable('gitaly_go_find_all_tags') +``` - ```ruby - Feature.enable('gitaly_go_find_all_tags') - ``` +Pay attention to the name of the flag and the one used in the Rails console. There is a difference +between them (dashes replaced by underscores and name prefix is changed). Make sure to prefix all +flags with `gitaly_`. + +NOTE: +If not set in GitLab, feature flags are read as false from the console and Gitaly uses their +default value. The default value depends on the GitLab version. ### Testing with GDK diff --git a/doc/development/go_guide/index.md b/doc/development/go_guide/index.md index 745ec50bdcd..e8c0c751af9 100644 --- a/doc/development/go_guide/index.md +++ b/doc/development/go_guide/index.md @@ -322,7 +322,7 @@ A few things to keep in mind when adding context: - [Go 1.13 errors](https://blog.golang.org/go1.13-errors). - [Programing with errors](https://peter.bourgon.org/blog/2019/09/11/programming-with-errors.html). -- [Don’t just check errors, handle them +- [Don't just check errors, handle them gracefully](https://dave.cheney.net/2016/04/27/dont-just-check-errors-handle-them-gracefully). ## CLIs @@ -499,6 +499,12 @@ of the Code Review Comments page on the Go wiki for more details. Most editors/IDEs allow you to run commands before/after saving a file, you can set it up to run `goimports -local gitlab.com/gitlab-org` so that it's applied to every file when saving. +### Analyzer Tests + +The conventional Secure [analyzer](https://gitlab.com/gitlab-org/security-products/analyzers/) has a [`convert` function](https://gitlab.com/gitlab-org/security-products/analyzers/command/-/blob/main/convert.go#L15-17) that converts SAST/DAST scanner reports into [GitLab Security Reports](https://gitlab.com/gitlab-org/security-products/security-report-schemas). When writing tests for the `convert` function, we should make use of [test fixtures](https://dave.cheney.net/2016/05/10/test-fixtures-in-go) using a `testdata` directory at the root of the analyzer's repo. The `testdata` directory should contain two subdirectories: `expect` and `reports`. The `reports` directory should contain sample SAST/DAST scanner reports which are passed into the `convert` function during the test setup. The `expect` directory should contain the expected GitLab Security Report that the `convert` returns. See Secret Detection for an [example](https://gitlab.com/gitlab-org/security-products/analyzers/secrets/-/blob/160424589ef1eed7b91b59484e019095bc7233bd/convert_test.go#L13-66). + +If the scanner report is small, less than 35 lines, then feel free to [inline the report](https://gitlab.com/gitlab-org/security-products/analyzers/sobelow/-/blob/8bd2428a/convert/convert_test.go#L13-77) rather than use a `testdata` directory. + --- [Return to Development documentation](../README.md). diff --git a/doc/development/graphql_guide/authorization.md b/doc/development/graphql_guide/authorization.md new file mode 100644 index 00000000000..ee5713f6fda --- /dev/null +++ b/doc/development/graphql_guide/authorization.md @@ -0,0 +1,223 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# GraphQL Authorization + +Authorizations can be applied in these places: + +- Types: + - Objects (all classes descending from `::Types::BaseObject`) + - Enums (all classes descending from `::Types::BaseEnum`) +- Resolvers: + - Field resolvers (all classes descending from `::Types::BaseResolver`) + - Mutations (all classes descending from `::Types::BaseMutation`) +- Fields (all fields declared using the `field` DSL method) + +Authorizations cannot be specified for abstract types (interfaces and +unions). Abstract types delegate to their member types. +Basic built in scalars (such as integers) do not have authorizations. + +Our authorization system uses the same [`DeclarativePolicy`](../policies.md) +system as throughout the rest of the application. + +- For single values (such as `Query.project`), if the currently authenticated + user fails the authorization, the field resolves to `null`. +- For collections (such as `Project.issues`), the collection is filtered to + exclude the objects that the user's authorization checks failed against. This + process of filtering (also known as _redaction_) happens after pagination, so + some pages may be smaller than the requested page size, due to redacted + objects being removed. + +Also see [authorizing resources in a mutation](../api_graphql_styleguide.md#authorizing-resources). + +NOTE: +The best practice is to load only what the currently authenticated user is allowed to +view with our existing finders first, without relying on authorization +to filter the records. This minimizes database queries and unnecessary +authorization checks of the loaded records. It also avoids situations, +such as short pages, which can expose the presence of confidential resources. + +See [`authorization_spec.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/spec/graphql/features/authorization_spec.rb) +for examples of all the authorization schemes discussed here. + +## Type authorization + +Authorize a type by passing an ability to the `authorize` method. All +fields with the same type is authorized by checking that the +currently authenticated user has the required ability. + +For example, the following authorization ensures that the currently +authenticated user can only see projects that they have the +`read_project` ability for (so long as the project is returned in a +field that uses `Types::ProjectType`): + +```ruby +module Types + class ProjectType < BaseObject + authorize :read_project + end +end +``` + +You can also authorize against multiple abilities, in which case all of +the ability checks must pass. + +For example, the following authorization ensures that the currently +authenticated user must have `read_project` and `another_ability` +abilities to see a project: + +```ruby +module Types + class ProjectType < BaseObject + authorize [:read_project, :another_ability] + end +end +``` + +## Resolver authorization + +Resolvers can have their own authorizations, which can be applied either to the +parent object or to the resolved values. + +An example of a resolver that authorizes against the parent is +`Resolvers::BoardListsResolver`, which requires that the parent +satisfy `:read_list` before it runs. + +An example which authorizes against the resolved resource is +`Resolvers::Ci::ConfigResolver`, which requires that the resolved value satisfy +`:read_pipeline`. + +To authorize against the parent, the resolver must _opt in_ (because this +was not the default value initially), by declaring this with `authorizes_object!`: + +```ruby +module Resolvers + class MyResolver < BaseResolver + authorizes_object! + + authorize :some_permission + end +end +``` + +To authorize against the resolved value, the resolver must apply the +authorization at some point, typically by using `#authorized_find!(**args)`: + +```ruby +module Resolvers + class MyResolver < BaseResolver + authorize :some_permission + + def resolve(**args) + authorized_find!(**args) # calls find_object + end + + def find_object(id:) + MyThing.find(id) + end + end +end +``` + +Of the two approaches, authorizing the object is more efficient, because it +helps avoid unnecessary queries. + +## Field authorization + +Fields can be authorized with the `authorize` option. + +Fields authorization is checked against the current object, and +authorization happens _before_ resolution, which means that +fields do not have access to the resolved resource. If you need to +apply an authorization check to a field, you probably want to add +authorization to the resolver, or ideally to the type. + +For example, the following authorization ensures that the +authenticated user must have administrator level access to the project +to view the `secretName` field: + +```ruby +module Types + class ProjectType < BaseObject + field :secret_name, ::GraphQL::STRING_TYPE, null: true, authorize: :owner_access + end +end +``` + +In this example, we use field authorization (such as +`Ability.allowed?(current_user, :read_transactions, bank_account)`) to avoid +a more expensive query: + +```ruby +module Types + class BankAccountType < BaseObject + field :transactions, ::Types::TransactionType.connection_type, null: true, + authorize: :read_transactions + end +end +``` + +Field authorization is recommended for: + +- Scalar fields (strings, booleans, or numbers) that should have different levels + of access controls to other fields. +- Object and collection fields where an access check can be applied to the + parent to save the field resolution, and avoid individual policy checks + on each resolved object. + +Field authorization does not replace object level checks, unless the object +precisely matches the access level of the parent project. For example, issues +can be confidential, independent of the access level of the parent. Therefore, +we should not use field authorization for `Project.issue`. + +You can also authorize fields against multiple abilities. Pass the abilities +as an array instead of as a single value: + +```ruby +module Types + class MyType < BaseObject + field :hidden_field, ::GraphQL::INT_TYPE, + null: true, + authorize: [:owner_access, :another_ability] + end +end +``` + +The field authorization on `MyType.hiddenField` implies the following tests: + +```ruby +Ability.allowed?(current_user, :owner_access, object_of_my_type) && + Ability.allowed?(current_user, :another_ability, object_of_my_type) +``` + +## Type and Field authorizations together + +Authorizations are cumulative. In other words, the currently authenticated +user may need to pass authorization requirements on both a field and a field's +type. + +In the following simplified example the currently authenticated user +needs both `first_permission` on the user and `second_permission` on the +issue to see the author of the issue. + +```ruby +class UserType + authorize :first_permission +end +``` + +```ruby +class IssueType + field :author, UserType, authorize: :second_permission +end +``` + +The combination of the object authorization on `UserType` and the field authorization on `IssueType.author` implies the following tests: + +```ruby +Ability.allowed?(current_user, :second_permission, issue) && + Ability.allowed?(current_user, :first_permission, issue.author) +``` diff --git a/doc/development/graphql_guide/index.md b/doc/development/graphql_guide/index.md index 4ecb34835aa..b8d4b53992e 100644 --- a/doc/development/graphql_guide/index.md +++ b/doc/development/graphql_guide/index.md @@ -17,6 +17,7 @@ feedback, and suggestions. - [GraphQL API documentation style guide](../documentation/graphql_styleguide.md): documentation style guide for GraphQL. - [GraphQL API](../../api/graphql/index.md): user documentation for the GitLab GraphQL API. +- [GraphQL authorization](authorization.md): guide to using authorization in GraphQL. - [GraphQL BatchLoader](batchloader.md): development documentation on the BatchLoader. - [GraphQL pagination](pagination.md): development documentation on pagination. - [GraphQL Pro](graphql_pro.md): information on our GraphQL Pro subscription. diff --git a/doc/development/i18n/externalization.md b/doc/development/i18n/externalization.md index 90355e1cccb..0f7b8078933 100644 --- a/doc/development/i18n/externalization.md +++ b/doc/development/i18n/externalization.md @@ -255,7 +255,45 @@ For example use `%{created_at}` in Ruby but `%{createdAt}` in JavaScript. Make s - In Vue: - See the section on [Vue component interpolation](#vue-components-interpolation). + Use the [`GlSprintf`](https://gitlab-org.gitlab.io/gitlab-ui/?path=/docs/utilities-sprintf--sentence-with-link) component if: + + - you need to include child components in the translation string. + - you need to include HTML in your translation string. + - you are using `sprintf` and need to pass `false` as the third argument to + prevent it from escaping placeholder values. + + For example: + + ```html + <gl-sprintf :message="s__('ClusterIntegration|Learn more about %{linkStart}zones%{linkEnd}')"> + <template #link="{ content }"> + <gl-link :href="somePath">{{ content }}</gl-link> + </template> + </gl-sprintf> + ``` + + In other cases it may be simpler to use `sprintf`, perhaps in a computed + property. For example: + + ```html + <script> + import { __, sprintf } from '~/locale'; + + export default { + ... + computed: { + userWelcome() { + sprintf(__('Hello %{username}'), { username: this.user.name }); + } + } + ... + } + </script> + + <template> + <span>{{ userWelcome }}</span> + </template> + ``` - In JavaScript (when Vue cannot be used): @@ -265,12 +303,10 @@ For example use `%{created_at}` in Ruby but `%{createdAt}` in JavaScript. Make s sprintf(__('Hello %{username}'), { username: 'Joe' }); // => 'Hello Joe' ``` - If you want to use markup within the translation and are using Vue, you - **must** use the [`gl-sprintf`](#vue-components-interpolation) component. If - for some reason you cannot use Vue, use `sprintf` and stop it from escaping - placeholder values by passing `false` as its third argument. You **must** - escape any interpolated dynamic values yourself, for instance using - `escape` from `lodash`. + If you need to use markup within the translation, use `sprintf` and stop it + from escaping placeholder values by passing `false` as its third argument. + You **must** escape any interpolated dynamic values yourself, for instance + using `escape` from `lodash`. ```javascript import { escape } from 'lodash'; @@ -589,7 +625,7 @@ This also applies when using links in between translated sentences, otherwise th ```haml - zones_link_url = 'https://cloud.google.com/compute/docs/regions-zones/regions-zones' - zones_link_start = '<a href="%{url}" target="_blank" rel="noopener noreferrer">'.html_safe % { url: zones_link_url } - = s_('ClusterIntegration|Learn more about %{zones_link_start}zones%{zones_link_end}').html_safe % { zones_link_start: zones_link_start, zones_link_end: '</a>'.html_safe } + = html_escape(s_('ClusterIntegration|Learn more about %{zones_link_start}zones%{zones_link_end}')) % { zones_link_start: zones_link_start, zones_link_end: '</a>'.html_safe } ``` - In Vue, instead of: @@ -632,7 +668,7 @@ This also applies when using links in between translated sentences, otherwise th {{ sprintf(s__("ClusterIntegration|Learn more about %{link}"), { link: '<a href="https://cloud.google.com/compute/docs/regions-zones/regions-zones" target="_blank" rel="noopener noreferrer">zones</a>' - }) + }, false) }} ``` @@ -643,7 +679,7 @@ This also applies when using links in between translated sentences, otherwise th sprintf(s__("ClusterIntegration|Learn more about %{linkStart}zones%{linkEnd}"), { linkStart: '<a href="https://cloud.google.com/compute/docs/regions-zones/regions-zones" target="_blank" rel="noopener noreferrer">', linkEnd: '</a>', - }) + }, false) }} ``` @@ -652,45 +688,6 @@ The reasoning behind this is that in some languages words change depending on co When in doubt, try to follow the best practices described in this [Mozilla Developer documentation](https://developer.mozilla.org/en-US/docs/Mozilla/Localization/Localization_content_best_practices#Splitting). -##### Vue components interpolation - -When translating UI text in Vue components, you might want to include child components inside -the translation string. -You could not use a JavaScript-only solution to render the translation, -because Vue would not be aware of the child components and would render them as plain text. - -For this use case, you should use the `gl-sprintf` component which is maintained -in **GitLab UI**. - -The `gl-sprintf` component accepts a `message` property, which is the translatable string, -and it exposes a named slot for every placeholder in the string, which lets you include Vue -components easily. - -Assume you want to print the translatable string -`Pipeline %{pipelineId} triggered %{timeago} by %{author}`. To replace the `%{timeago}` and -`%{author}` placeholders with Vue components, here's how you would do that with `gl-sprintf`: - -```html -<template> - <div> - <gl-sprintf :message="__('Pipeline %{pipelineId} triggered %{timeago} by %{author}')"> - <template #pipelineId>{{ pipeline.id }}</template> - <template #timeago> - <timeago :time="pipeline.triggerTime" /> - </template> - <template #author> - <gl-avatar-labeled - :src="pipeline.triggeredBy.avatarPath" - :label="pipeline.triggeredBy.name" - /> - </template> - </gl-sprintf> - </div> -</template> -``` - -For more information, see the [`gl-sprintf`](https://gitlab-org.gitlab.io/gitlab-ui/?path=/story/base-sprintf--default) documentation. - ## Updating the PO files with the new content Now that the new content is marked for translation, we need to update @@ -702,7 +699,7 @@ bin/rake gettext:regenerate This command updates `locale/gitlab.pot` file with the newly externalized strings and remove any strings that aren't used anymore. You should check this -file in. Once the changes are on master, they are picked up by +file in. Once the changes are on the default branch, they are picked up by [CrowdIn](https://translate.gitlab.com) and be presented for translation. diff --git a/doc/development/i18n/merging_translations.md b/doc/development/i18n/merging_translations.md index e7d25942143..553820733e7 100644 --- a/doc/development/i18n/merging_translations.md +++ b/doc/development/i18n/merging_translations.md @@ -42,10 +42,11 @@ page](https://translate.gitlab.com/project/gitlab-ee/settings#integration). ## Merging translations -When all translations are found good and pipelines pass the -translations can be merged into the master branch. When merging the translations, -make sure to check the **Remove source branch** checkbox, so CrowdIn recreates the -`master-i18n` from master after the new translation was merged. +After all translations are determined to be appropriate and the pipelines pass, +you can merge the translations into the default branch. When merging translations, +be sure to select the **Remove source branch** check box, which causes CrowdIn +to recreate the `master-i18n` from the default branch after merging the new +translation. We are discussing [automating this entire process](https://gitlab.com/gitlab-org/gitlab/-/issues/19896). @@ -59,7 +60,7 @@ and delete the [`master-18n`](https://gitlab.com/gitlab-org/gitlab/-/branches/all?utf8=✓&search=master-i18n). This might be needed when the merge request contains failures that -have been fixed on master. +have been fixed on the default branch. ## Recreate the GitLab integration in CrowdIn diff --git a/doc/development/i18n/translation.md b/doc/development/i18n/translation.md index 7fb49521106..f3d02e180e7 100644 --- a/doc/development/i18n/translation.md +++ b/doc/development/i18n/translation.md @@ -29,7 +29,6 @@ GitLab is being translated into many languages. - If the language you are looking for is not available, [open an issue](https://gitlab.com/gitlab-org/gitlab/-/issues?scope=all&utf8=✓&state=all&label_name[]=Category%3AInternationalization). Notify our Crowdin administrators by including `@gitlab-org/manage/import` in your issue. - in the issue. - After the issue/Merge Request is complete, restart this procedure. 1. Next, you can view list of files and folders. Select `gitlab.pot` to open the translation editor. @@ -109,6 +108,6 @@ To propose additions to the glossary please <!-- vale gitlab.Spelling = NO --> In French, the "écriture inclusive" is now over (see on [Legifrance](https://www.legifrance.gouv.fr/jorf/id/JORFTEXT000036068906/)). -So, to include both genders, write “Utilisateurs et utilisatrices” instead of “Utilisateur·rice·s”. +So, to include both genders, write "Utilisateurs et utilisatrices" instead of "Utilisateur·rice·s". When space is missing, the male gender should be used alone. <!-- vale gitlab.Spelling = YES --> diff --git a/doc/development/instrumentation.md b/doc/development/instrumentation.md index 338ad76d414..83e7444bb1f 100644 --- a/doc/development/instrumentation.md +++ b/doc/development/instrumentation.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/integrations/secure.md b/doc/development/integrations/secure.md index fda75dad119..ae4e952d063 100644 --- a/doc/development/integrations/secure.md +++ b/doc/development/integrations/secure.md @@ -68,7 +68,7 @@ so the [`allow_failure`](../../ci/yaml/README.md#allow_failure) parameter should ### Artifacts Scanning jobs must declare a report that corresponds to the type of scanning they perform, -using the [`artifacts:reports`](../../ci/pipelines/job_artifacts.md#artifactsreports) keyword. +using the [`artifacts:reports`](../../ci/yaml/README.md#artifactsreports) keyword. Valid reports are: `dependency_scanning`, `container_scanning`, `dast`, and `sast`. For example, here is the definition of a SAST job that generates a file named `gl-sast-report.json`, @@ -209,7 +209,7 @@ It is recommended to name the output file after the type of scanning, and to use Since all Secure reports are JSON files, it is recommended to use `.json` as a file extension. For instance, a suggested filename for a Dependency Scanning report is `gl-dependency-scanning.json`. -The [`artifacts:reports`](../../ci/pipelines/job_artifacts.md#artifactsreports) keyword +The [`artifacts:reports`](../../ci/yaml/README.md#artifactsreports) keyword of the job definition must be consistent with the file path where the Security report is written. For instance, if a Dependency Scanning analyzer writes its report to the CI project directory, and if this report filename is `depscan.json`, diff --git a/doc/development/integrations/secure_partner_integration.md b/doc/development/integrations/secure_partner_integration.md index 17bce13583c..3154dc83ea4 100644 --- a/doc/development/integrations/secure_partner_integration.md +++ b/doc/development/integrations/secure_partner_integration.md @@ -71,19 +71,19 @@ best place to integrate your own product and its results into GitLab. This section describes the steps you need to complete to onboard as a partner and complete an integration with the Secure stage. -1. Read about our [partnerships](https://about.gitlab.com/partners/integrate/). +1. Read about our [partnerships](https://about.gitlab.com/partners/technology-partners/integrate/). 1. [Create an issue](https://gitlab.com/gitlab-com/alliances/alliances/-/issues/new?issuable_template=new_partner) using our new partner issue template to begin the discussion. 1. Get a test account to begin developing your integration. You can - request a [GitLab.com Subscription Sandbox](https://about.gitlab.com/partners/integrate/#gitlabcom-subscription-sandbox-request) - or an [EE Developer License](https://about.gitlab.com/partners/integrate/#requesting-ee-dev-license-for-rd). + request a [GitLab.com Subscription Sandbox](https://about.gitlab.com/partners/technology-partners/integrate/#gitlabcom-subscription-sandbox-request) + or an [EE Developer License](https://about.gitlab.com/partners/technology-partners/integrate/#requesting-ultimate-dev-license-for-rd). 1. Provide a [pipeline job](../../development/pipelines.md) template that users could integrate into their own GitLab pipelines. 1. Create a report artifact with your pipeline jobs. 1. Ensure your pipeline jobs create a report artifact that GitLab can process to successfully display your own product's results with the rest of GitLab. - See detailed [technical directions](secure.md) for this step. - - Read more about [job report artifacts](../../ci/pipelines/job_artifacts.md#artifactsreports). + - Read more about [job report artifacts](../../ci/yaml/README.md#artifactsreports). - Read about [job artifacts](../../ci/pipelines/job_artifacts.md). - Your report artifact must be in one of our currently supported formats. For more information, see the [documentation on reports](secure.md#report). @@ -105,10 +105,10 @@ and complete an integration with the Secure stage. interface. 1. Demo the integration to GitLab: - After you have tested and are ready to demo your integration please - [reach out](https://about.gitlab.com/partners/integrate/) to us. If you - skip this step you won’t be able to do supported marketing. + [reach out](https://about.gitlab.com/partners/technology-partners/integrate/) to us. If you + skip this step you won't be able to do supported marketing. 1. Begin doing supported marketing of your GitLab integration. - - Work with our [partner team](https://about.gitlab.com/partners/integrate/) + - Work with our [partner team](https://about.gitlab.com/partners/technology-partners/integrate/) to support your go-to-market as appropriate. - Examples of supported marketing could include being listed on our [Security Partner page](https://about.gitlab.com/partners/#security), doing an [Unfiltered blog post](https://about.gitlab.com/handbook/marketing/blog/unfiltered/), diff --git a/doc/development/licensed_feature_availability.md b/doc/development/licensed_feature_availability.md index a9fc0414297..10e6d717a18 100644 --- a/doc/development/licensed_feature_availability.md +++ b/doc/development/licensed_feature_availability.md @@ -41,3 +41,16 @@ the instance license. ```ruby License.feature_available?(:feature_symbol) ``` + +## Restricting frontend features + +To restrict frontend features based on the license, use `push_licensed_feature`. +The frontend can then access this via `this.glFeatures`: + +```ruby +before_action do + push_licensed_feature(:feature_symbol) + # or by project/namespace + push_licensed_feature(:feature_symbol, project) +end +``` diff --git a/doc/development/licensing.md b/doc/development/licensing.md index c467fe81755..1ab56e60a0b 100644 --- a/doc/development/licensing.md +++ b/doc/development/licensing.md @@ -18,9 +18,6 @@ Some gems may not include their license information in their `gemspec` file, and ### License Finder commands -NOTE: -License Finder currently uses GitLab misused terms of `whitelist` and `blacklist`. As a result, the commands below reference those terms. We've created an [issue on their project](https://github.com/pivotal/LicenseFinder/issues/745) to propose that they rename their commands. - There are a few basic commands License Finder provides that you need in order to manage license detection. To verify that the checks are passing, and/or to see what dependencies are causing the checks to fail: @@ -32,13 +29,13 @@ bundle exec license_finder To allowlist a new license: ```shell -license_finder whitelist add MIT +license_finder permitted_licenses add MIT ``` To denylist a new license: ```shell -license_finder blacklist add Unlicense +license_finder restricted_licenses add Unlicense ``` To tell License Finder about a dependency's license if it isn't auto-detected: diff --git a/doc/development/logging.md b/doc/development/logging.md index 30398eb87a1..88ae3950f1a 100644 --- a/doc/development/logging.md +++ b/doc/development/logging.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/merge_request_performance_guidelines.md b/doc/development/merge_request_performance_guidelines.md index 50362269c1b..0f696d39092 100644 --- a/doc/development/merge_request_performance_guidelines.md +++ b/doc/development/merge_request_performance_guidelines.md @@ -18,7 +18,7 @@ To measure the impact of a merge request you can use the following guides: - [Performance Guidelines](performance.md) -- [What requires downtime?](what_requires_downtime.md) +- [Avoiding downtime in migrations](avoiding_downtime_in_migrations.md) ## Definition @@ -158,6 +158,27 @@ objects_to_update.update_all(some_field: some_value) This uses ActiveRecord's `update_all` method to update all rows in a single query. This in turn makes it much harder for this code to overload a database. +## Use read replicas when possible + +In a DB cluster we have many read replicas and one primary. A classic use of scaling the DB is to have read-only actions be performed by the replicas. We use [load balancing](../administration/database_load_balancing.md) to distribute this load. This allows for the replicas to grow as the pressure on the DB grows. + +By default, queries use read-only replicas, but due to [primary sticking](../administration/database_load_balancing.md#primary-sticking), GitLab sticks to using the primary for a certain period of time and reverts back to secondaries after they have either caught up or after 30 seconds, which can lead to a considerable amount of unnecessary load on the primary. In this [merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/56849) we introduced the `without_sticky_writes` block to prevent switching to the primary. This [merge request example](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/57328) provides a good use case for when queries can stick to the primary and how to prevent this by using `without_sticky_writes`. + +Internally, our database load balancer classifies the queries based on their main statement (`select`, `update`, `delete`, etc.). When in doubt, it redirects the queries to the primary database. Hence, there are some common cases the load balancer sends the queries to the primary unnecessarily: + +- Custom queries (via `exec_query`, `execute_statement`, `execute`, etc.) +- Read-only transactions +- In-flight connection configuration set +- Sidekiq background jobs + +Worse, after the above queries are executed, GitLab [sticks to the primary](../administration/database_load_balancing.md#primary-sticking). In [this merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/56476), we introduced `use_replica_if_possible` to make the inside queries prefer to use the replicas. That MR is also an example how we redirected a costly, time-consuming query to the replicas. + +## Use CTEs wisely + +Read about [complex queries on the relation object](iterating_tables_in_batches.md#complex-queries-on-the-relation-object) for considerations on how to use CTEs. We have found in some situations that CTEs can become problematic in use (similar to the n+1 problem above). In particular, hierarchical recursive CTE queries such as the CTE in [AuthorizedProjectsWorker](https://gitlab.com/gitlab-org/gitlab/-/issues/325688) are very difficult to optimize and don't scale. We should avoid them when implementing new features that require any kind of hierarchical structure. + +However, in many simpler cases, such as this [example](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/43242#note_61416277), CTEs can be quite effective as an optimization fence. + ## Cached Queries **Summary:** a merge request **should not** execute duplicated cached queries. @@ -459,7 +480,7 @@ Performance deficiencies should be addressed right away after we merge initial changes. Read more about when and how feature flags should be used in -[Feature flags in GitLab development](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle#feature-flags-in-gitlab-development). +[Feature flags in GitLab development](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#feature-flags-in-gitlab-development). ## Storage diff --git a/doc/development/migration_style_guide.md b/doc/development/migration_style_guide.md index fcecc556052..40457dbb533 100644 --- a/doc/development/migration_style_guide.md +++ b/doc/development/migration_style_guide.md @@ -16,16 +16,12 @@ migrations are written carefully, can be applied online, and adhere to the style guide below. Migrations are **not** allowed to require GitLab installations to be taken -offline unless _absolutely necessary_. - -When downtime is necessary the migration has to be approved by: - -1. The VP of Engineering -1. A Backend Maintainer -1. A Database Maintainer - -An up-to-date list of people holding these titles can be found at -<https://about.gitlab.com/company/team/>. +offline ever. Migrations always must be written in such a way to avoid +downtime. In the past we had a process for defining migrations that allowed for +downtime by setting a `DOWNTIME` constant. You may see this when looking at +older migrations. This process was in place for 4 years without every being +used and as such we've learnt we can always figure out how to write a migration +differently to avoid downtime. When writing your migrations, also consider that databases might have stale data or inconsistencies and guard for that. Try to make as few assumptions as @@ -65,47 +61,16 @@ scripts/regenerate-schema TARGET=12-9-stable-ee scripts/regenerate-schema ``` -## What Requires Downtime? - -The document ["What Requires Downtime?"](what_requires_downtime.md) specifies -various database operations, such as - -- [dropping and renaming columns](what_requires_downtime.md#dropping-columns) -- [changing column constraints and types](what_requires_downtime.md#changing-column-constraints) -- [adding and dropping indexes, tables, and foreign keys](what_requires_downtime.md#adding-indexes) - -and whether they require downtime and how to work around that whenever possible. - -## Downtime Tagging - -Every migration must specify if it requires downtime or not, and if it should -require downtime it must also specify a reason for this. This is required even -if 99% of the migrations don't require downtime as this makes it easier to find -the migrations that _do_ require downtime. - -To tag a migration, add the following two constants to the migration class' -body: - -- `DOWNTIME`: a boolean that when set to `true` indicates the migration requires - downtime. -- `DOWNTIME_REASON`: a String containing the reason for the migration requiring - downtime. This constant **must** be set when `DOWNTIME` is set to `true`. - -For example: +## Avoiding downtime -```ruby -class MyMigration < ActiveRecord::Migration[6.0] - DOWNTIME = true - DOWNTIME_REASON = 'This migration requires downtime because ...' +The document ["Avoiding downtime in migrations"](avoiding_downtime_in_migrations.md) specifies +various database operations, such as: - def change - ... - end -end -``` +- [dropping and renaming columns](avoiding_downtime_in_migrations.md#dropping-columns) +- [changing column constraints and types](avoiding_downtime_in_migrations.md#changing-column-constraints) +- [adding and dropping indexes, tables, and foreign keys](avoiding_downtime_in_migrations.md#adding-indexes) -It is an error (that is, CI fails) if the `DOWNTIME` constant is missing -from a migration class. +and explains how to perform them without requiring downtime. ## Reversibility @@ -153,7 +118,7 @@ and therefore it does not have any records yet. When using a single-transaction migration, a transaction holds a database connection for the duration of the migration, so you must make sure the actions in the migration -do not take too much time: GitLab.com’s production database has a `15s` timeout, so +do not take too much time: GitLab.com's production database has a `15s` timeout, so in general, the cumulative execution time in a migration should aim to fit comfortably in that limit. Singular query timings should fit within the [standard limit](query_performance.md#timing-guidelines-for-queries) @@ -254,7 +219,7 @@ end **Creating a new table with a foreign key:** -We can simply wrap the `create_table` method with `with_lock_retries`: +We can wrap the `create_table` method with `with_lock_retries`: ```ruby def up @@ -289,10 +254,10 @@ def up t.bigint :project_id, null: false t.bigint :user_id, null: false t.string :jid, limit: 255 - end - add_index :imports, :project_id - add_index :imports, :user_id + t.index :project_id + t.index :user_id + end end def down @@ -302,7 +267,7 @@ end Adding foreign key to `projects`: -We can use the `add_concurrenct_foreign_key` method in this case, as this helper method +We can use the `add_concurrent_foreign_key` method in this case, as this helper method has the lock retries built into it. ```ruby @@ -512,8 +477,6 @@ class like so: class MyMigration < ActiveRecord::Migration[6.0] include Gitlab::Database::MigrationHelpers - DOWNTIME = false - disable_ddl_transaction! INDEX_NAME = 'index_name' @@ -640,8 +603,6 @@ Take the following migration as an example: ```ruby class DefaultRequestAccessGroups < ActiveRecord::Migration[5.2] - DOWNTIME = false - def change change_column_default(:namespaces, :request_access_enabled, from: false, to: true) end @@ -715,7 +676,7 @@ the `DROP TABLE` statement is likely to stall concurrent traffic until it fails Table **has no records** (feature was never in use) and **no foreign keys**: -- Simply use the `drop_table` method in your migration. +- Use the `drop_table` method in your migration. ```ruby def change @@ -849,8 +810,6 @@ Example migration adding this column: ```ruby class AddOptionsToBuildMetadata < ActiveRecord::Migration[5.0] - DOWNTIME = false - def change add_column :ci_builds_metadata, :config_options, :jsonb end @@ -1038,7 +997,7 @@ To identify a high-traffic table for GitLab.com the following measures are consi Note that the metrics linked here are GitLab-internal only: - [Read operations](https://thanos.gitlab.net/graph?g0.range_input=2h&g0.max_source_resolution=0s&g0.expr=topk(500%2C%20sum%20by%20(relname)%20(rate(pg_stat_user_tables_seq_tup_read%7Benvironment%3D%22gprd%22%7D%5B12h%5D)%20%2B%20rate(pg_stat_user_tables_idx_scan%7Benvironment%3D%22gprd%22%7D%5B12h%5D)%20%2B%20rate(pg_stat_user_tables_idx_tup_fetch%7Benvironment%3D%22gprd%22%7D%5B12h%5D)))&g0.tab=1) -- [Number of records](https://thanos.gitlab.net/graph?g0.range_input=2h&g0.max_source_resolution=0s&g0.expr=topk(500%2C%20sum%20by%20(relname)%20(rate(pg_stat_user_tables_n_live_tup%7Benvironment%3D%22gprd%22%7D%5B12h%5D)))&g0.tab=1) -- [Size](https://thanos.gitlab.net/graph?g0.range_input=2h&g0.max_source_resolution=0s&g0.expr=topk(500%2C%20sum%20by%20(relname)%20(rate(pg_total_relation_size_bytes%7Benvironment%3D%22gprd%22%7D%5B12h%5D)))&g0.tab=1) is greater than 10 GB +- [Number of records](https://thanos.gitlab.net/graph?g0.range_input=2h&g0.max_source_resolution=0s&g0.expr=topk(500%2C%20max%20by%20(relname)%20(pg_stat_user_tables_n_live_tup%7Benvironment%3D%22gprd%22%7D))&g0.tab=1) +- [Size](https://thanos.gitlab.net/graph?g0.range_input=2h&g0.max_source_resolution=0s&g0.expr=topk(500%2C%20max%20by%20(relname)%20(pg_total_relation_size_bytes%7Benvironment%3D%22gprd%22%7D))&g0.tab=1) is greater than 10 GB Any table which has some high read operation compared to current [high-traffic tables](https://gitlab.com/gitlab-org/gitlab/-/blob/master/rubocop/rubocop-migrations.yml#L4) might be a good candidate. diff --git a/doc/development/new_fe_guide/tips.md b/doc/development/new_fe_guide/tips.md index c60d70b3b16..5d4c0fc019f 100644 --- a/doc/development/new_fe_guide/tips.md +++ b/doc/development/new_fe_guide/tips.md @@ -27,7 +27,7 @@ Your feature flag can now be: ### More on feature flags - [Deleting a feature flag](../../api/features.md#delete-a-feature) -- [Manage feature flags](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle) +- [Manage feature flags](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/) - [Feature flags API](../../api/features.md) ## Running tests locally diff --git a/doc/development/packages.md b/doc/development/packages.md index e8c326da974..d4982473d67 100644 --- a/doc/development/packages.md +++ b/doc/development/packages.md @@ -133,7 +133,7 @@ During this phase, the idea is to collect as much information as possible about - **Authentication**: What authentication mechanisms are available (OAuth, Basic Authorization, other). Keep in mind that GitLab users often want to use their [Personal Access Tokens](../user/profile/personal_access_tokens.md). - Although not needed for the MVC first iteration, the [CI job tokens](../user/project/new_ci_build_permissions_model.md#job-token) + Although not needed for the MVC first iteration, the [CI/CD job tokens](../api/README.md#gitlab-cicd-job-token) have to be supported at some point in the future. - **Requests**: Which requests are needed to have a working MVC. Ideally, produce a list of all the requests needed for the MVC (including required actions). Further diff --git a/doc/development/permissions.md b/doc/development/permissions.md index 35f0941b756..2af451840d6 100644 --- a/doc/development/permissions.md +++ b/doc/development/permissions.md @@ -120,3 +120,31 @@ into different features like Merge Requests and CI flow. | View | Vulnerability feedback | Merge Request | Can read security findings | | View | Dependency List page | Project | Can access Dependency information | | View | License Compliance page | Project | Can access License information| + +## Where should permissions be checked? + +By default, controllers, API endpoints, and GraphQL types/fields are responsible for authorization. See [Secure Coding Guidelines > Permissions](secure_coding_guidelines.md#permissions). + +### Considerations + +- Many actions are completely or partially extracted to services, finders, and other classes, so it is normal to do permission checks "downstream". +- Often, authorization logic must be incorporated in DB queries to filter records. +- `DeclarativePolicy` rules are relatively performant, but conditions may perform database calls. +- Multiple permission checks across layers can be difficult to reason about, which is its own security risk. For example, duplicate authorization logic could diverge. +- Should we apply defense-in-depth with permission checks? [Join the discussion](https://gitlab.com/gitlab-org/gitlab/-/issues/324135) + +### Tips + +If a class accepts `current_user`, then it may be responsible for authorization. + +### Example: Adding a new API endpoint + +By default, we authorize at the endpoint. Checking an existing ability may make sense; if not, then we probably need to add one. + +As an aside, most endpoints can be cleanly categorized as a CRUD (create, read, update, destroy) action on a resource. The services and abilities follow suit, which is why many are named like `Projects::CreateService` or `:read_project`. + +Say, for example, we extract the whole endpoint into a service. The `can?` check will now be in the service. Say the service reuses an existing finder, which we are modifying for our purposes. Should we make the finder check an ability? + +- If the finder doesn't accept `current_user`, and therefore doesn't check permissions, then probably no. +- If the finder accepts `current_user`, and doesn't check permissions, then it would be a good idea to double check other usages of the finder, and we might consider adding authorization. +- If the finder accepts `current_user`, and already checks permissions, then either we need to add our case, or the existing checks are appropriate. diff --git a/doc/development/pipelines.md b/doc/development/pipelines.md index aa3f2e6791a..8a93a46247e 100644 --- a/doc/development/pipelines.md +++ b/doc/development/pipelines.md @@ -497,18 +497,20 @@ request, be sure to start the `dont-interrupt-me` job before pushing. 1. We currently have several different caches defined in [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/global.gitlab-ci.yml), with fixed keys: - - `.setup-test-env-cache`. - - `.rails-cache`. - - `.static-analysis-cache`. + - `.setup-test-env-cache` + - `.rails-cache` + - `.static-analysis-cache` - `.coverage-cache` + - `.danger-review-cache` - `.qa-cache` - - `.yarn-cache`. + - `.yarn-cache` - `.assets-compile-cache` (the key includes `${NODE_ENV}` so it's actually two different caches). 1. Only 6 specific jobs, running in 2-hourly scheduled pipelines, are pushing (i.e. updating) to the caches: - `update-setup-test-env-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/rails.gitlab-ci.yml). - `update-rails-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/rails.gitlab-ci.yml). - `update-static-analysis-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/rails.gitlab-ci.yml). - `update-coverage-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/rails.gitlab-ci.yml). + - `update-danger-review-cache`, defined in [`.gitlab/ci/review.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/review.gitlab-ci.yml). - `update-qa-cache`, defined in [`.gitlab/ci/qa.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/qa.gitlab-ci.yml). - `update-assets-compile-production-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). - `update-assets-compile-test-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). @@ -534,21 +536,23 @@ The pre-clone step works by using the `CI_PRE_CLONE_SCRIPT` variable The `CI_PRE_CLONE_SCRIPT` is currently defined as a project CI/CD variable: ```shell -echo "Downloading archived master..." -wget -O /tmp/gitlab.tar.gz https://storage.googleapis.com/gitlab-ci-git-repo-cache/project-278964/gitlab-master-shallow.tar.gz - -if [ ! -f /tmp/gitlab.tar.gz ]; then - echo "Repository cache not available, cloning a new directory..." - exit -fi - -rm -rf $CI_PROJECT_DIR -echo "Extracting tarball into $CI_PROJECT_DIR..." -mkdir -p $CI_PROJECT_DIR -cd $CI_PROJECT_DIR -tar xzf /tmp/gitlab.tar.gz -rm -f /tmp/gitlab.tar.gz -chmod a+w $CI_PROJECT_DIR +( + echo "Downloading archived master..." + wget -O /tmp/gitlab.tar.gz https://storage.googleapis.com/gitlab-ci-git-repo-cache/project-278964/gitlab-master-shallow.tar.gz + + if [ ! -f /tmp/gitlab.tar.gz ]; then + echo "Repository cache not available, cloning a new directory..." + exit + fi + + rm -rf $CI_PROJECT_DIR + echo "Extracting tarball into $CI_PROJECT_DIR..." + mkdir -p $CI_PROJECT_DIR + cd $CI_PROJECT_DIR + tar xzf /tmp/gitlab.tar.gz + rm -f /tmp/gitlab.tar.gz + chmod a+w $CI_PROJECT_DIR +) ``` The first step of the script downloads `gitlab-master.tar.gz` from @@ -576,7 +580,7 @@ overwrites the Git configuration with the appropriate settings to fetch from the GitLab repository. `CI_REPO_CACHE_CREDENTIALS` contains the Google Cloud service account -JSON for uploading to the `gitlab-ci-git-repo-cache` bucket. (If you’re a +JSON for uploading to the `gitlab-ci-git-repo-cache` bucket. (If you're a GitLab Team Member, find credentials in the [GitLab shared 1Password account](https://about.gitlab.com/handbook/security/#1password-for-teams). @@ -614,6 +618,7 @@ that is deployed in stage `review`. [`coverage-javascript`](https://gitlab-org.gitlab.io/gitlab/coverage-javascript/), and `webpack-report` (found at `https://gitlab-org.gitlab.io/gitlab/webpack-report/`, but there is [an issue with the deployment](https://gitlab.com/gitlab-org/gitlab/-/issues/233458)). +- `notify`: This stage includes jobs that notify various failures to Slack. ### Default image diff --git a/doc/development/polymorphic_associations.md b/doc/development/polymorphic_associations.md index cabd2e3fb41..b71e66c8671 100644 --- a/doc/development/polymorphic_associations.md +++ b/doc/development/polymorphic_associations.md @@ -74,7 +74,7 @@ different columns set) in the same table. ## The Solution -Fortunately there is a very simple solution to these problems: simply use a +Fortunately there is a very simple solution to these problems: use a separate table for every type you would otherwise store in the same table. Using a separate table allows you to use everything a database may provide to ensure consistency and query data efficiently, without any additional application logic diff --git a/doc/development/product_analytics/snowplow.md b/doc/development/product_analytics/snowplow.md index ff91e450e1e..4e2f6530126 100644 --- a/doc/development/product_analytics/snowplow.md +++ b/doc/development/product_analytics/snowplow.md @@ -1,8 +1,8 @@ --- -redirect_to: '../snowplow.md' +redirect_to: '../snowplow/index.md' --- -This document was moved to [another location](../snowplow.md). +This document was moved to [another location](../snowplow/index.md). <!-- This redirect file can be deleted after April 1, 2021. --> <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/prometheus_metrics.md b/doc/development/prometheus_metrics.md index 05a623448bf..09efb70f279 100644 --- a/doc/development/prometheus_metrics.md +++ b/doc/development/prometheus_metrics.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -39,8 +39,6 @@ Or, you can create a database migration: class ImportCommonMetrics < ActiveRecord::Migration[4.2] include Gitlab::Database::MigrationHelpers - DOWNTIME = false - def up ::Gitlab::DatabaseImporters::CommonMetrics::Importer.new.execute end diff --git a/doc/development/query_count_limits.md b/doc/development/query_count_limits.md index a9569fee8cc..fec6f9022ee 100644 --- a/doc/development/query_count_limits.md +++ b/doc/development/query_count_limits.md @@ -15,30 +15,30 @@ When a test fails because it executes more than 100 SQL queries there are two solutions to this problem: - Reduce the number of SQL queries that are executed. -- Whitelist the controller or API endpoint. +- Disable query limiting for the controller or API endpoint. -You should only resort to whitelisting when an existing controller or endpoint +You should only resort to disabling query limits when an existing controller or endpoint is to blame as in this case reducing the number of SQL queries can take a lot of effort. Newly added controllers and endpoints are not allowed to execute more than 100 SQL queries and no exceptions are made for this rule. _If_ a large number of SQL queries is necessary to perform certain work it's best to have this work performed by Sidekiq instead of doing this directly in a web request. -## Whitelisting +## Disable query limiting -In the event that you _have_ to whitelist a controller you must first +In the event that you _have_ to disable query limits for a controller, you must first create an issue. This issue should (preferably in the title) mention the controller or endpoint and include the appropriate labels (`database`, `performance`, and at least a team specific label such as `Discussion`). -After the issue has been created you can whitelist the code in question. For +After the issue has been created, you can disable query limits on the code in question. For Rails controllers it's best to create a `before_action` hook that runs as early as possible. The called method in turn should call -`Gitlab::QueryLimiting.whitelist('issue URL here')`. For example: +`Gitlab::QueryLimiting.disable!('issue URL here')`. For example: ```ruby class MyController < ApplicationController - before_action :whitelist_query_limiting, only: [:show] + before_action :disable_query_limiting, only: [:show] def index # ... @@ -48,8 +48,8 @@ class MyController < ApplicationController # ... end - def whitelist_query_limiting - Gitlab::QueryLimiting.whitelist('https://gitlab.com/gitlab-org/...') + def disable_query_limiting + Gitlab::QueryLimiting.disable!('https://gitlab.com/gitlab-org/...') end end ``` @@ -63,7 +63,7 @@ call directly into the endpoint like so: ```ruby get '/projects/:id/foo' do - Gitlab::QueryLimiting.whitelist('...') + Gitlab::QueryLimiting.disable!('...') # ... end diff --git a/doc/development/rake_tasks.md b/doc/development/rake_tasks.md index 98b386497df..f88424287b1 100644 --- a/doc/development/rake_tasks.md +++ b/doc/development/rake_tasks.md @@ -152,6 +152,24 @@ To run several tests inside one directory: - `bin/rspec spec/requests/api/` for the RSpec tests if you want to test API only +### Run RSpec tests which failed in Merge Request pipeline on your machine + +If your Merge Request pipeline failed with RSpec test failures, +you can run all the failed tests on your machine with the following Rake task: + +```shell +bin/rake spec:merge_request_rspec_failure +``` + +There are a few caveats for this Rake task: + +- You need to be on the same branch on your machine as the source branch of the Merge Request. +- The pipeline must have been completed. +- You may need to wait for the test report to be parsed and retry again. + +This Rake task depends on the [unit test reports](../ci/unit_test_reports.md) feature, +which only gets parsed when it is requested for the first time. + ### Speed up tests, Rake tasks, and migrations [Spring](https://github.com/rails/spring) is a Rails application pre-loader. It diff --git a/doc/development/scalability.md b/doc/development/scalability.md index a9b8fb4389f..8ee6e57e4d1 100644 --- a/doc/development/scalability.md +++ b/doc/development/scalability.md @@ -36,7 +36,7 @@ application starts, Rails queries the database schema, caching the tables and column types for the data requested. Because of this schema cache, dropping a column or table while the application is running can produce 500 errors to the user. This is why we have a [process for dropping columns and other -no-downtime changes](what_requires_downtime.md). +no-downtime changes](avoiding_downtime_in_migrations.md). #### Multi-tenancy diff --git a/doc/development/secure_coding_guidelines.md b/doc/development/secure_coding_guidelines.md index e9c95a14236..62cc2543fc4 100644 --- a/doc/development/secure_coding_guidelines.md +++ b/doc/development/secure_coding_guidelines.md @@ -565,7 +565,7 @@ In some scenarios such as [this one](https://gitlab.com/gitlab-org/gitlab/-/issu return unless user # Sessions are enforced to be unavailable for API calls, so ignore them for admin mode - Gitlab::Auth::CurrentUserMode.bypass_session!(user.id) if Feature.enabled?(:user_mode_in_session) + Gitlab::Auth::CurrentUserMode.bypass_session!(user.id) if Gitlab::CurrentSettings.admin_mode unless api_access_allowed?(user) forbidden!(api_access_denied_message(user)) @@ -581,7 +581,7 @@ In order to prevent this from happening, it is recommended to use the method `us user = find_user_from_sources return unless user - if user.is_a?(User) && Feature.enabled?(:user_mode_in_session) + if user.is_a?(User) && Gitlab::CurrentSettings.admin_mode # Sessions are enforced to be unavailable for API calls, so ignore them for admin mode Gitlab::Auth::CurrentUserMode.bypass_session!(user.id) end diff --git a/doc/development/shell_commands.md b/doc/development/shell_commands.md index f28828c2e4e..6f56e60f619 100644 --- a/doc/development/shell_commands.md +++ b/doc/development/shell_commands.md @@ -223,4 +223,4 @@ When importing, GitLab would execute the following command, passing the `import_ git clone file://git:/tmp/lol ``` -Git would simply ignore the `git:` part, interpret the path as `file:///tmp/lol`, and import the repository into the new project. This action could potentially give the attacker access to any repository in the system, whether private or not. +Git ignores the `git:` part, interpret the path as `file:///tmp/lol`, and imports the repository into the new project. This action could potentially give the attacker access to any repository in the system, whether private or not. diff --git a/doc/development/sidekiq_style_guide.md b/doc/development/sidekiq_style_guide.md index cff199c8b1d..4bcd5e50fae 100644 --- a/doc/development/sidekiq_style_guide.md +++ b/doc/development/sidekiq_style_guide.md @@ -551,7 +551,7 @@ does not account for weights. As we are [moving towards using `sidekiq-cluster` in Free](https://gitlab.com/gitlab-org/gitlab/-/issues/34396), newly-added -workers do not need to have weights specified. They can simply use the +workers do not need to have weights specified. They can use the default weight, which is 1. ## Worker context @@ -831,8 +831,6 @@ as shown in this example: class MigrateTheRenamedSidekiqQueue < ActiveRecord::Migration[5.0] include Gitlab::Database::MigrationHelpers - DOWNTIME = false - def up sidekiq_queue_migrate 'old_queue_name', to: 'new_queue_name' end diff --git a/doc/development/snowplow.md b/doc/development/snowplow.md index f5689068654..b5d3be5b2dd 100644 --- a/doc/development/snowplow.md +++ b/doc/development/snowplow.md @@ -1,634 +1,6 @@ --- -stage: Growth -group: Product Intelligence -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: 'snowplow/index.md' --- - -# Snowplow Guide - -This guide provides an overview of how Snowplow works, and implementation details. - -For more information about Product Intelligence, see: - -- [Product Intelligence Guide](https://about.gitlab.com/handbook/product/product-intelligence-guide/) -- [Usage Ping Guide](usage_ping/index.md) - -More useful links: - -- [Product Intelligence Direction](https://about.gitlab.com/direction/product-intelligence/) -- [Data Analysis Process](https://about.gitlab.com/handbook/business-ops/data-team/#data-analysis-process/) -- [Data for Product Managers](https://about.gitlab.com/handbook/business-ops/data-team/programs/data-for-product-managers/) -- [Data Infrastructure](https://about.gitlab.com/handbook/business-ops/data-team/platform/infrastructure/) - -## What is Snowplow - -Snowplow is an enterprise-grade marketing and Product Intelligence platform which helps track the way users engage with our website and application. - -[Snowplow](https://github.com/snowplow/snowplow) consists of the following loosely-coupled sub-systems: - -- **Trackers** fire Snowplow events. Snowplow has 12 trackers, covering web, mobile, desktop, server, and IoT. -- **Collectors** receive Snowplow events from trackers. We have three different event collectors, synchronizing events either to Amazon S3, Apache Kafka, or Amazon Kinesis. -- **Enrich** cleans up the raw Snowplow events, enriches them and puts them into storage. We have an Hadoop-based enrichment process, and a Kinesis-based or Kafka-based process. -- **Storage** is where the Snowplow events live. We store the Snowplow events in a flat file structure on S3, and in the Redshift and PostgreSQL databases. -- **Data modeling** is where event-level data is joined with other data sets and aggregated into smaller data sets, and business logic is applied. This produces a clean set of tables which make it easier to perform analysis on the data. We have data models for Redshift and Looker. -- **Analytics** are performed on the Snowplow events or on the aggregate tables. - - - -## Snowplow schema - -We have many definitions of Snowplow's schema. We have an active issue to [standardize this schema](https://gitlab.com/gitlab-org/gitlab/-/issues/207930) including the following definitions: - -- Frontend and backend taxonomy as listed below -- [Structured event taxonomy](#structured-event-taxonomy) -- [Self describing events](https://github.com/snowplow/snowplow/wiki/Custom-events#self-describing-events) -- [Iglu schema](https://gitlab.com/gitlab-org/iglu/) -- [Snowplow authored events](https://github.com/snowplow/snowplow/wiki/Snowplow-authored-events) - -## Enabling Snowplow - -Tracking can be enabled at: - -- The instance level, which enables tracking on both the frontend and backend layers. -- User level, though user tracking can be disabled on a per-user basis. GitLab tracking respects the [Do Not Track](https://www.eff.org/issues/do-not-track) standard, so any user who has enabled the Do Not Track option in their browser is not tracked at a user level. - -We use Snowplow for the majority of our tracking strategy and it is enabled on GitLab.com. On a self-managed instance, Snowplow can be enabled by navigating to: - -- **Admin Area > Settings > General** in the UI. -- `admin/application_settings/integrations` in your browser. - -The following configuration is required: - -| Name | Value | -|---------------|---------------------------| -| Collector | `snowplow.trx.gitlab.net` | -| Site ID | `gitlab` | -| Cookie domain | `.gitlab.com` | - -## Snowplow request flow - -The following example shows a basic request/response flow between the following components: - -- Snowplow JS / Ruby Trackers on GitLab.com -- [GitLab.com Snowplow Collector](https://gitlab.com/gitlab-com/gl-infra/readiness/-/blob/master/library/snowplow/index.md) -- The GitLab S3 Bucket -- The GitLab Snowflake Data Warehouse -- Sisense: - -```mermaid -sequenceDiagram - participant Snowplow JS (Frontend) - participant Snowplow Ruby (Backend) - participant GitLab.com Snowplow Collector - participant S3 Bucket - participant Snowflake DW - participant Sisense Dashboards - Snowplow JS (Frontend) ->> GitLab.com Snowplow Collector: FE Tracking event - Snowplow Ruby (Backend) ->> GitLab.com Snowplow Collector: BE Tracking event - loop Process using Kinesis Stream - GitLab.com Snowplow Collector ->> GitLab.com Snowplow Collector: Log raw events - GitLab.com Snowplow Collector ->> GitLab.com Snowplow Collector: Enrich events - GitLab.com Snowplow Collector ->> GitLab.com Snowplow Collector: Write to disk - end - GitLab.com Snowplow Collector ->> S3 Bucket: Kinesis Firehose - S3 Bucket->>Snowflake DW: Import data - Snowflake DW->>Snowflake DW: Transform data using dbt - Snowflake DW->>Sisense Dashboards: Data available for querying -``` - -## Structured event taxonomy - -When adding new click events, we should add them in a way that's internally consistent. If we don't, it is very painful to perform analysis across features since each feature captures events differently. - -The current method provides several attributes that are sent on each click event. Please try to follow these guidelines when specifying events to capture: - -| attribute | type | required | description | -| --------- | ------- | -------- | ----------- | -| category | text | true | The page or backend area of the application. Unless infeasible, please use the Rails page attribute by default in the frontend, and namespace + class name on the backend. | -| action | text | true | The action the user is taking, or aspect that's being instrumented. The first word should always describe the action or aspect: clicks should be `click`, activations should be `activate`, creations should be `create`, etc. Use underscores to describe what was acted on; for example, activating a form field would be `activate_form_input`. An interface action like clicking on a dropdown would be `click_dropdown`, while a behavior like creating a project record from the backend would be `create_project` | -| label | text | false | The specific element, or object that's being acted on. This is either the label of the element (e.g. a tab labeled 'Create from template' may be `create_from_template`) or a unique identifier if no text is available (e.g. closing the Groups dropdown in the top navigation bar might be `groups_dropdown_close`), or it could be the name or title attribute of a record being created. | -| property | text | false | Any additional property of the element, or object being acted on. | -| value | decimal | false | Describes a numeric value or something directly related to the event. This could be the value of an input (e.g. `10` when clicking `internal` visibility). | - -### Web-specific parameters - -Snowplow JS adds many [web-specific parameters](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/snowplow-tracker-protocol/#Web-specific_parameters) to all web events by default. - -## Implementing Snowplow JS (Frontend) tracking - -GitLab provides `Tracking`, an interface that wraps the [Snowplow JavaScript Tracker](https://github.com/snowplow/snowplow/wiki/javascript-tracker) for tracking custom events. There are a few ways to use tracking, but each generally requires at minimum, a `category` and an `action`. Additional data can be provided that adheres to our [Structured event taxonomy](#structured-event-taxonomy). - -| field | type | default value | description | -|:-----------|:-------|:---------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `category` | string | `document.body.dataset.page` | Page or subsection of a page that events are being captured within. | -| `action` | string | 'generic' | Action the user is taking. Clicks should be `click` and activations should be `activate`, so for example, focusing a form field would be `activate_form_input`, and clicking a button would be `click_button`. | -| `data` | object | `{}` | Additional data such as `label`, `property`, `value`, and `context` as described in our [Structured event taxonomy](#structured-event-taxonomy). | - -### Tracking in HAML (or Vue Templates) - -When working within HAML (or Vue templates) we can add `data-track-*` attributes to elements of interest. All elements that have a `data-track-event` attribute automatically have event tracking bound on clicks. - -Below is an example of `data-track-*` attributes assigned to a button: - -```haml -%button.btn{ data: { track: { event: "click_button", label: "template_preview", property: "my-template" } } } -``` - -```html -<button class="btn" - data-track-event="click_button" - data-track-label="template_preview" - data-track-property="my-template" -/> -``` - -Event listeners are bound at the document level to handle click events on or within elements with these data attributes. This allows them to be properly handled on re-rendering and changes to the DOM. Note that because of the way these events are bound, click events should not be stopped from propagating up the DOM tree. If for any reason click events are being stopped from propagating, you need to implement your own listeners and follow the instructions in [Tracking in raw JavaScript](#tracking-in-raw-javascript). - -Below is a list of supported `data-track-*` attributes: - -| attribute | required | description | -|:----------------------|:---------|:------------| -| `data-track-event` | true | Action the user is taking. Clicks must be prepended with `click` and activations must be prepended with `activate`. For example, focusing a form field would be `activate_form_input` and clicking a button would be `click_button`. | -| `data-track-label` | false | The `label` as described in our [Structured event taxonomy](#structured-event-taxonomy). | -| `data-track-property` | false | The `property` as described in our [Structured event taxonomy](#structured-event-taxonomy). | -| `data-track-value` | false | The `value` as described in our [Structured event taxonomy](#structured-event-taxonomy). If omitted, this is the element's `value` property or an empty string. For checkboxes, the default value is the element's checked attribute or `false` when unchecked. | -| `data-track-context` | false | The `context` as described in our [Structured event taxonomy](#structured-event-taxonomy). | - -#### Caveats - -When using the GitLab helper method [`nav_link`](https://gitlab.com/gitlab-org/gitlab/-/blob/898b286de322e5df6a38d257b10c94974d580df8/app/helpers/tab_helper.rb#L69) be sure to wrap `html_options` under the `html_options` keyword argument. -Be careful, as this behavior can be confused with the `ActionView` helper method [`link_to`](https://api.rubyonrails.org/v5.2.3/classes/ActionView/Helpers/UrlHelper.html#method-i-link_to) that does not require additional wrapping of `html_options` - -`nav_link(controller: ['dashboard/groups', 'explore/groups'], html_options: { data: { track_label: "groups_dropdown", track_event: "click_dropdown" } })` - -vs - -`link_to assigned_issues_dashboard_path, title: _('Issues'), data: { track_label: 'main_navigation', track_event: 'click_issues_link' }` - -### Tracking within Vue components - -There's a tracking Vue mixin that can be used in components if more complex tracking is required. To use it, first import the `Tracking` library and request a mixin. - -```javascript -import Tracking from '~/tracking'; -const trackingMixin = Tracking.mixin({ label: 'right_sidebar' }); -``` - -You can provide default options that are passed along whenever an event is tracked from within your component. For instance, if all events within a component should be tracked with a given `label`, you can provide one at this time. Available defaults are `category`, `label`, `property`, and `value`. If no category is specified, `document.body.dataset.page` is used as the default. - -You can then use the mixin normally in your component with the `mixin` Vue declaration. The mixin also provides the ability to specify tracking options in `data` or `computed`. These override any defaults and allow the values to be dynamic from props, or based on state. - -```javascript -export default { - mixins: [trackingMixin], - // ...[component implementation]... - data() { - return { - expanded: false, - tracking: { - label: 'left_sidebar' - } - }; - }, -} -``` - -The mixin provides a `track` method that can be called within the template, or from component methods. An example of the whole implementation might look like the following. - -```javascript -export default { - mixins: [Tracking.mixin({ label: 'right_sidebar' })], - data() { - return { - expanded: false, - }; - }, - methods: { - toggle() { - this.expanded = !this.expanded; - this.track('click_toggle', { value: this.expanded }) - } - } -}; -``` - -And if needed within the template, you can use the `track` method directly as well. - -```html -<template> - <div> - <a class="toggle" @click.prevent="toggle">Toggle</a> - <div v-if="expanded"> - <p>Hello world!</p> - <a @click.prevent="track('click_action')">Track an event</a> - </div> - </div> -</template> -``` - -### Tracking in raw JavaScript - -Custom event tracking and instrumentation can be added by directly calling the `Tracking.event` static function. The following example demonstrates tracking a click on a button by calling `Tracking.event` manually. - -```javascript -import Tracking from '~/tracking'; - -const button = document.getElementById('create_from_template_button'); -button.addEventListener('click', () => { - Tracking.event('dashboard:projects:index', 'click_button', { - label: 'create_from_template', - property: 'template_preview', - value: 'rails', - }); -}) -``` - -### Tests and test helpers - -In Jest particularly in Vue tests, you can use the following: - -```javascript -import { mockTracking } from 'helpers/tracking_helper'; - -describe('MyTracking', () => { - let spy; - - beforeEach(() => { - spy = mockTracking('_category_', wrapper.element, jest.spyOn); - }); - - it('tracks an event when clicked on feedback', () => { - wrapper.find('.discover-feedback-icon').trigger('click'); - - expect(spy).toHaveBeenCalledWith('_category_', 'click_button', { - label: 'security-discover-feedback-cta', - property: '0', - }); - }); -}); -``` - -In obsolete Karma tests it's used as below: - -```javascript -import { mockTracking, triggerEvent } from 'spec/helpers/tracking_helper'; - -describe('my component', () => { - let trackingSpy; - - beforeEach(() => { - trackingSpy = mockTracking('_category_', vm.$el, spyOn); - }); - - const triggerEvent = () => { - // action which should trigger a event - }; - - it('tracks an event when toggled', () => { - expect(trackingSpy).not.toHaveBeenCalled(); - - triggerEvent('a.toggle'); - - expect(trackingSpy).toHaveBeenCalledWith('_category_', 'click_edit_button', { - label: 'right_sidebar', - property: 'confidentiality', - }); - }); -}); -``` - -## Implementing Snowplow Ruby (Backend) tracking - -GitLab provides `Gitlab::Tracking`, an interface that wraps the [Snowplow Ruby Tracker](https://github.com/snowplow/snowplow/wiki/ruby-tracker) for tracking custom events. - -Custom event tracking and instrumentation can be added by directly calling the `GitLab::Tracking.event` class method, which accepts the following arguments: - -| argument | type | default value | description | -|------------|---------------------------|---------------|-----------------------------------------------------------------------------------------------------------------------------------| -| `category` | String | | Area or aspect of the application. This could be `HealthCheckController` or `Lfs::FileTransformer` for instance. | -| `action` | String | | The action being taken, which can be anything from a controller action like `create` to something like an Active Record callback. | -| `label` | String | nil | As described in [Structured event taxonomy](#structured-event-taxonomy). | -| `property` | String | nil | As described in [Structured event taxonomy](#structured-event-taxonomy). | -| `value` | Numeric | nil | As described in [Structured event taxonomy](#structured-event-taxonomy). | -| `context` | Array\[SelfDescribingJSON\] | nil | An array of custom contexts to send with this event. Most events should not have any custom contexts. | -| `project` | Project | nil | The project associated with the event. | -| `user` | User | nil | The user associated with the event. | -| `namespace` | Namespace | nil | The namespace associated with the event. | - -Tracking can be viewed as either tracking user behavior, or can be used for instrumentation to monitor and visualize performance over time in an area or aspect of code. - -For example: - -```ruby -class Projects::CreateService < BaseService - def execute - project = Project.create(params) - - Gitlab::Tracking.event('Projects::CreateService', 'create_project', label: project.errors.full_messages.to_sentence, - property: project.valid?.to_s, project: project, user: current_user, namespace: namespace) - end -end -``` - -### Unit testing - -Use the `expect_snowplow_event` helper when testing backend Snowplow events. See [testing best practices]( -https://docs.gitlab.com/ee/development/testing_guide/best_practices.html#test-snowplow-events) for details. - -### Performance - -We use the [AsyncEmitter](https://github.com/snowplow/snowplow/wiki/Ruby-Tracker#52-the-asyncemitter-class) when tracking events, which allows for instrumentation calls to be run in a background thread. This is still an active area of development. - -## Developing and testing Snowplow - -There are several tools for developing and testing Snowplow Event - -| Testing Tool | Frontend Tracking | Backend Tracking | Local Development Environment | Production Environment | Production Environment | -|----------------------------------------------|--------------------|---------------------|-------------------------------|------------------------|------------------------| -| Snowplow Analytics Debugger Chrome Extension | **{check-circle}** | **{dotted-circle}** | **{check-circle}** | **{check-circle}** | **{check-circle}** | -| Snowplow Inspector Chrome Extension | **{check-circle}** | **{dotted-circle}** | **{check-circle}** | **{check-circle}** | **{check-circle}** | -| Snowplow Micro | **{check-circle}** | **{check-circle}** | **{check-circle}** | **{dotted-circle}** | **{dotted-circle}** | -| Snowplow Mini | **{check-circle}** | **{check-circle}** | **{dotted-circle}** | **{status_preparing}** | **{status_preparing}** | - -**Legend** - -**{check-circle}** Available, **{status_preparing}** In progress, **{dotted-circle}** Not Planned - -### Preparing your MR for Review - -1. For frontend events, in the MR description section, add a screenshot of the event's relevant section using the [Snowplow Analytics Debugger](https://chrome.google.com/webstore/detail/snowplow-analytics-debugg/jbnlcgeengmijcghameodeaenefieedm) Chrome browser extension. -1. For backend events, please use Snowplow Micro and add the output of the Snowplow Micro good events `GET http://localhost:9090/micro/good`. -1. Include a member of the Product Intelligence team as a reviewer of your MR. Mention `@gitlab-org/growth/product_intelligence/engineers` in your MR to request a review. - -### Snowplow Analytics Debugger Chrome Extension - -Snowplow Analytics Debugger is a browser extension for testing frontend events. This works on production, staging and local development environments. - -1. Install the [Snowplow Analytics Debugger](https://chrome.google.com/webstore/detail/snowplow-analytics-debugg/jbnlcgeengmijcghameodeaenefieedm) Chrome browser extension. -1. Open Chrome DevTools to the Snowplow Analytics Debugger tab. -1. Learn more at [Igloo Analytics](https://www.iglooanalytics.com/blog/snowplow-analytics-debugger-chrome-extension.html). - -### Snowplow Inspector Chrome Extension - -Snowplow Inspector Chrome Extension is a browser extension for testing frontend events. This works on production, staging and local development environments. - -1. Install [Snowplow Inspector](https://chrome.google.com/webstore/detail/snowplow-inspector/maplkdomeamdlngconidoefjpogkmljm?hl=en). -1. Open the Chrome extension by pressing the Snowplow Inspector icon beside the address bar. -1. Click around on a webpage with Snowplow and you should see JavaScript events firing in the inspector window. - -### Snowplow Micro - -Snowplow Micro is a very small version of a full Snowplow data collection pipeline: small enough that it can be launched by a test suite. Events can be recorded into Snowplow Micro just as they can a full Snowplow pipeline. Micro then exposes an API that can be queried. - -Snowplow Micro is a Docker-based solution for testing frontend and backend events in a local development environment. You need to modify GDK using the instructions below to set this up. - -- Read [Introducing Snowplow Micro](https://snowplowanalytics.com/blog/2019/07/17/introducing-snowplow-micro/) -- Look at the [Snowplow Micro repository](https://github.com/snowplow-incubator/snowplow-micro) -- Watch our <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [installation guide recording](https://www.youtube.com/watch?v=OX46fo_A0Ag) - -1. Ensure Docker is installed and running. - -1. Install [Snowplow Micro](https://github.com/snowplow-incubator/snowplow-micro) by cloning the settings in [this project](https://gitlab.com/gitlab-org/snowplow-micro-configuration): -1. Navigate to the directory with the cloned project, and start the appropriate Docker - container with the following script: - - ```shell - ./snowplow-micro.sh - ``` - -1. Update your instance's settings to enable Snowplow events and point to the Snowplow Micro collector: - - ```shell - gdk psql -d gitlabhq_development - update application_settings set snowplow_collector_hostname='localhost:9090', snowplow_enabled=true, snowplow_cookie_domain='.gitlab.com'; - ``` - -1. Update `DEFAULT_SNOWPLOW_OPTIONS` in `app/assets/javascripts/tracking.js` to remove `forceSecureTracker: true`: - - ```diff - diff --git a/app/assets/javascripts/tracking.js b/app/assets/javascripts/tracking.js - index 0a1211d0a76..3b98c8f28f2 100644 - --- a/app/assets/javascripts/tracking.js - +++ b/app/assets/javascripts/tracking.js - @@ -7,7 +7,6 @@ const DEFAULT_SNOWPLOW_OPTIONS = { - appId: '', - userFingerprint: false, - respectDoNotTrack: true, - - forceSecureTracker: true, - eventMethod: 'post', - contexts: { webPage: true, performanceTiming: true }, - formTracking: false, - - ``` - -1. Update `snowplow_options` in `lib/gitlab/tracking.rb` to add `protocol` and `port`: - - ```diff - diff --git a/lib/gitlab/tracking.rb b/lib/gitlab/tracking.rb - index 618e359211b..e9084623c43 100644 - --- a/lib/gitlab/tracking.rb - +++ b/lib/gitlab/tracking.rb - @@ -41,7 +41,9 @@ def snowplow_options(group) - cookie_domain: Gitlab::CurrentSettings.snowplow_cookie_domain, - app_id: Gitlab::CurrentSettings.snowplow_app_id, - form_tracking: additional_features, - - link_click_tracking: additional_features - + link_click_tracking: additional_features, - + protocol: 'http', - + port: 9090 - }.transform_keys! { |key| key.to_s.camelize(:lower).to_sym } - end - ``` - -1. Update `emitter` in `lib/gitlab/tracking/destinations/snowplow.rb` to change `protocol`: - - ```diff - diff --git a/lib/gitlab/tracking/destinations/snowplow.rb b/lib/gitlab/tracking/destinations/snowplow.rb - index 4fa844de325..5dd9d0eacfb 100644 - --- a/lib/gitlab/tracking/destinations/snowplow.rb - +++ b/lib/gitlab/tracking/destinations/snowplow.rb - @@ -40,7 +40,7 @@ def tracker - def emitter - SnowplowTracker::AsyncEmitter.new( - Gitlab::CurrentSettings.snowplow_collector_hostname, - - protocol: 'https' - + protocol: 'http' - ) - end - end - - ``` - -1. Restart GDK: - - ```shell - `gdk restart` - ``` - -1. Send a test Snowplow event from the Rails console: - - ```ruby - Gitlab::Tracking.self_describing_event('iglu:com.gitlab/pageview_context/jsonschema/1-0-0', data: { page_type: 'MY_TYPE' }, context: nil) - ``` - -1. Navigate to `localhost:9090/micro/good` to see the event. - -### Snowplow Mini - -[Snowplow Mini](https://github.com/snowplow/snowplow-mini) is an easily-deployable, single-instance version of Snowplow. - -Snowplow Mini can be used for testing frontend and backend events on a production, staging and local development environment. - -For GitLab.com, we're setting up a [QA and Testing environment](https://gitlab.com/gitlab-org/telemetry/-/issues/266) using Snowplow Mini. - -## Snowplow Schemas - -### `gitlab_standard` - -We are including the [`gitlab_standard` schema](https://gitlab.com/gitlab-org/iglu/-/blob/master/public/schemas/com.gitlab/gitlab_standard/jsonschema/) with every event. See [Standardize Snowplow Schema](https://gitlab.com/groups/gitlab-org/-/epics/5218) for details. - -The [`StandardContext`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/tracking/standard_context.rb) class represents this schema in the application. - -| Field Name | Required | Type | Description | -|----------------|---------------------|-----------------------|---------------------------------------------------------------------------------------------| -| `project_id` | **{dotted-circle}** | integer | | -| `namespace_id` | **{dotted-circle}** | integer | | -| `environment` | **{check-circle}** | string (max 32 chars) | Name of the source environment, such as `production` or `staging` | -| `source` | **{check-circle}** | string (max 32 chars) | Name of the source application, such as `gitlab-rails` or `gitlab-javascript` | - -### Default Schema - -| Field Name | Required | Type | Description | -|--------------------------|---------------------|-----------|----------------------------------------------------------------------------------------------------------------------------------| -| `app_id` | **{check-circle}** | string | Unique identifier for website / application | -| `base_currency` | **{dotted-circle}** | string | Reporting currency | -| `br_colordepth` | **{dotted-circle}** | integer | Browser color depth | -| `br_cookies` | **{dotted-circle}** | boolean | Does the browser permit cookies? | -| `br_family` | **{dotted-circle}** | string | Browser family | -| `br_features_director` | **{dotted-circle}** | boolean | Director plugin installed? | -| `br_features_flash` | **{dotted-circle}** | boolean | Flash plugin installed? | -| `br_features_gears` | **{dotted-circle}** | boolean | Google gears installed? | -| `br_features_java` | **{dotted-circle}** | boolean | Java plugin installed? | -| `br_features_pdf` | **{dotted-circle}** | boolean | Adobe PDF plugin installed? | -| `br_features_quicktime` | **{dotted-circle}** | boolean | Quicktime plugin installed? | -| `br_features_realplayer` | **{dotted-circle}** | boolean | RealPlayer plugin installed? | -| `br_features_silverlight` | **{dotted-circle}** | boolean | Silverlight plugin installed? | -| `br_features_windowsmedia` | **{dotted-circle}** | boolean | Windows media plugin installed? | -| `br_lang` | **{dotted-circle}** | string | Language the browser is set to | -| `br_name` | **{dotted-circle}** | string | Browser name | -| `br_renderengine` | **{dotted-circle}** | string | Browser rendering engine | -| `br_type` | **{dotted-circle}** | string | Browser type | -| `br_version` | **{dotted-circle}** | string | Browser version | -| `br_viewheight` | **{dotted-circle}** | string | Browser viewport height | -| `br_viewwidth` | **{dotted-circle}** | string | Browser viewport width | -| `collector_tstamp` | **{dotted-circle}** | timestamp | Time stamp for the event recorded by the collector | -| `contexts` | **{dotted-circle}** | | | -| `derived_contexts` | **{dotted-circle}** | | Contexts derived in the Enrich process | -| `derived_tstamp` | **{dotted-circle}** | timestamp | Timestamp making allowance for inaccurate device clock | -| `doc_charset` | **{dotted-circle}** | string | Web page’s character encoding | -| `doc_height` | **{dotted-circle}** | string | Web page height | -| `doc_width` | **{dotted-circle}** | string | Web page width | -| `domain_sessionid` | **{dotted-circle}** | string | Unique identifier (UUID) for this visit of this user_id to this domain | -| `domain_sessionidx` | **{dotted-circle}** | integer | Index of number of visits that this user_id has made to this domain (The first visit is `1`) | -| `domain_userid` | **{dotted-circle}** | string | Unique identifier for a user, based on a first party cookie (so domain specific) | -| `dvce_created_tstamp` | **{dotted-circle}** | timestamp | Timestamp when event occurred, as recorded by client device | -| `dvce_ismobile` | **{dotted-circle}** | boolean | Indicates whether device is mobile | -| `dvce_screenheight` | **{dotted-circle}** | string | Screen / monitor resolution | -| `dvce_screenwidth` | **{dotted-circle}** | string | Screen / monitor resolution | -| `dvce_sent_tstamp` | **{dotted-circle}** | timestamp | Timestamp when event was sent by client device to collector | -| `dvce_type` | **{dotted-circle}** | string | Type of device | -| `etl_tags` | **{dotted-circle}** | string | JSON of tags for this ETL run | -| `etl_tstamp` | **{dotted-circle}** | timestamp | Timestamp event began ETL | -| `event` | **{dotted-circle}** | string | Event type | -| `event_fingerprint` | **{dotted-circle}** | string | Hash client-set event fields | -| `event_format` | **{dotted-circle}** | string | Format for event | -| `event_id` | **{dotted-circle}** | string | Event UUID | -| `event_name` | **{dotted-circle}** | string | Event name | -| `event_vendor` | **{dotted-circle}** | string | The company who developed the event model | -| `event_version` | **{dotted-circle}** | string | Version of event schema | -| `geo_city` | **{dotted-circle}** | string | City of IP origin | -| `geo_country` | **{dotted-circle}** | string | Country of IP origin | -| `geo_latitude` | **{dotted-circle}** | string | An approximate latitude | -| `geo_longitude` | **{dotted-circle}** | string | An approximate longitude | -| `geo_region` | **{dotted-circle}** | string | Region of IP origin | -| `geo_region_name` | **{dotted-circle}** | string | Region of IP origin | -| `geo_timezone` | **{dotted-circle}** | string | Timezone of IP origin | -| `geo_zipcode` | **{dotted-circle}** | string | Zip (postal) code of IP origin | -| `ip_domain` | **{dotted-circle}** | string | Second level domain name associated with the visitor’s IP address | -| `ip_isp` | **{dotted-circle}** | string | Visitor’s ISP | -| `ip_netspeed` | **{dotted-circle}** | string | Visitor’s connection type | -| `ip_organization` | **{dotted-circle}** | string | Organization associated with the visitor’s IP address – defaults to ISP name if none is found | -| `mkt_campaign` | **{dotted-circle}** | string | The campaign ID | -| `mkt_clickid` | **{dotted-circle}** | string | The click ID | -| `mkt_content` | **{dotted-circle}** | string | The content or ID of the ad. | -| `mkt_medium` | **{dotted-circle}** | string | Type of traffic source | -| `mkt_network` | **{dotted-circle}** | string | The ad network to which the click ID belongs | -| `mkt_source` | **{dotted-circle}** | string | The company / website where the traffic came from | -| `mkt_term` | **{dotted-circle}** | string | Keywords associated with the referrer | -| `name_tracker` | **{dotted-circle}** | string | The tracker namespace | -| `network_userid` | **{dotted-circle}** | string | Unique identifier for a user, based on a cookie from the collector (so set at a network level and shouldn’t be set by a tracker) | -| `os_family` | **{dotted-circle}** | string | Operating system family | -| `os_manufacturer` | **{dotted-circle}** | string | Manufacturers of operating system | -| `os_name` | **{dotted-circle}** | string | Name of operating system | -| `os_timezone` | **{dotted-circle}** | string | Client operating system timezone | -| `page_referrer` | **{dotted-circle}** | string | Referrer URL | -| `page_title` | **{dotted-circle}** | string | Page title | -| `page_url` | **{dotted-circle}** | string | Page URL | -| `page_urlfragment` | **{dotted-circle}** | string | Fragment aka anchor | -| `page_urlhost` | **{dotted-circle}** | string | Host aka domain | -| `page_urlpath` | **{dotted-circle}** | string | Path to page | -| `page_urlport` | **{dotted-circle}** | integer | Port if specified, 80 if not | -| `page_urlquery` | **{dotted-circle}** | string | Query string | -| `page_urlscheme` | **{dotted-circle}** | string | Scheme (protocol name) | -| `platform` | **{dotted-circle}** | string | The platform the app runs on | -| `pp_xoffset_max` | **{dotted-circle}** | integer | Maximum page x offset seen in the last ping period | -| `pp_xoffset_min` | **{dotted-circle}** | integer | Minimum page x offset seen in the last ping period | -| `pp_yoffset_max` | **{dotted-circle}** | integer | Maximum page y offset seen in the last ping period | -| `pp_yoffset_min` | **{dotted-circle}** | integer | Minimum page y offset seen in the last ping period | -| `refr_domain_userid` | **{dotted-circle}** | string | The Snowplow `domain_userid` of the referring website | -| `refr_dvce_tstamp` | **{dotted-circle}** | timestamp | The time of attaching the `domain_userid` to the inbound link | -| `refr_medium` | **{dotted-circle}** | string | Type of referer | -| `refr_source` | **{dotted-circle}** | string | Name of referer if recognised | -| `refr_term` | **{dotted-circle}** | string | Keywords if source is a search engine | -| `refr_urlfragment` | **{dotted-circle}** | string | Referer URL fragment | -| `refr_urlhost` | **{dotted-circle}** | string | Referer host | -| `refr_urlpath` | **{dotted-circle}** | string | Referer page path | -| `refr_urlport` | **{dotted-circle}** | integer | Referer port | -| `refr_urlquery` | **{dotted-circle}** | string | Referer URL query string | -| `refr_urlscheme` | **{dotted-circle}** | string | Referer scheme | -| `se_action` | **{dotted-circle}** | string | The action / event itself | -| `se_category` | **{dotted-circle}** | string | The category of event | -| `se_label` | **{dotted-circle}** | string | A label often used to refer to the ‘object’ the action is performed on | -| `se_property` | **{dotted-circle}** | string | A property associated with either the action or the object | -| `se_value` | **{dotted-circle}** | decimal | A value associated with the user action | -| `ti_category` | **{dotted-circle}** | string | Item category | -| `ti_currency` | **{dotted-circle}** | string | Currency | -| `ti_name` | **{dotted-circle}** | string | Item name | -| `ti_orderid` | **{dotted-circle}** | string | Order ID | -| `ti_price` | **{dotted-circle}** | decimal | Item price | -| `ti_price_base` | **{dotted-circle}** | decimal | Item price in base currency | -| `ti_quantity` | **{dotted-circle}** | integer | Item quantity | -| `ti_sku` | **{dotted-circle}** | string | Item SKU | -| `tr_affiliation` | **{dotted-circle}** | string | Transaction affiliation (such as channel) | -| `tr_city` | **{dotted-circle}** | string | Delivery address: city | -| `tr_country` | **{dotted-circle}** | string | Delivery address: country | -| `tr_currency` | **{dotted-circle}** | string | Transaction Currency | -| `tr_orderid` | **{dotted-circle}** | string | Order ID | -| `tr_shipping` | **{dotted-circle}** | decimal | Delivery cost charged | -| `tr_shipping_base` | **{dotted-circle}** | decimal | Shipping cost in base currency | -| `tr_state` | **{dotted-circle}** | string | Delivery address: state | -| `tr_tax` | **{dotted-circle}** | decimal | Transaction tax value (such as amount of VAT included) | -| `tr_tax_base` | **{dotted-circle}** | decimal | Tax applied in base currency | -| `tr_total` | **{dotted-circle}** | decimal | Transaction total value | -| `tr_total_base` | **{dotted-circle}** | decimal | Total amount of transaction in base currency | -| `true_tstamp` | **{dotted-circle}** | timestamp | User-set exact timestamp | -| `txn_id` | **{dotted-circle}** | string | Transaction ID | -| `unstruct_event` | **{dotted-circle}** | JSON | The properties of the event | -| `uploaded_at` | **{dotted-circle}** | | | -| `user_fingerprint` | **{dotted-circle}** | integer | User identifier based on (hopefully unique) browser features | -| `user_id` | **{dotted-circle}** | string | Unique identifier for user, set by the business using setUserId | -| `user_ipaddress` | **{dotted-circle}** | string | IP address | -| `useragent` | **{dotted-circle}** | string | User agent (expressed as a browser string) | -| `v_collector` | **{dotted-circle}** | string | Collector version | -| `v_etl` | **{dotted-circle}** | string | ETL version | -| `v_tracker` | **{dotted-circle}** | string | Identifier for Snowplow tracker | +This document was moved to [another location](snowplow/index.md). +<!-- This redirect file can be deleted after 2021-06-31. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/snowplow/index.md b/doc/development/snowplow/index.md new file mode 100644 index 00000000000..c07291d61f2 --- /dev/null +++ b/doc/development/snowplow/index.md @@ -0,0 +1,719 @@ +--- +stage: Growth +group: Product Intelligence +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Snowplow Guide + +This guide provides an overview of how Snowplow works, and implementation details. + +For more information about Product Intelligence, see: + +- [Product Intelligence Guide](https://about.gitlab.com/handbook/product/product-intelligence-guide/) +- [Usage Ping Guide](../usage_ping/index.md) + +More useful links: + +- [Product Intelligence Direction](https://about.gitlab.com/direction/product-intelligence/) +- [Data Analysis Process](https://about.gitlab.com/handbook/business-ops/data-team/#data-analysis-process/) +- [Data for Product Managers](https://about.gitlab.com/handbook/business-ops/data-team/programs/data-for-product-managers/) +- [Data Infrastructure](https://about.gitlab.com/handbook/business-ops/data-team/platform/infrastructure/) + +## What is Snowplow + +Snowplow is an enterprise-grade marketing and Product Intelligence platform which helps track the way users engage with our website and application. + +[Snowplow](https://github.com/snowplow/snowplow) consists of the following loosely-coupled sub-systems: + +- **Trackers** fire Snowplow events. Snowplow has 12 trackers, covering web, mobile, desktop, server, and IoT. +- **Collectors** receive Snowplow events from trackers. We have three different event collectors, synchronizing events either to Amazon S3, Apache Kafka, or Amazon Kinesis. +- **Enrich** cleans up the raw Snowplow events, enriches them and puts them into storage. We have an Hadoop-based enrichment process, and a Kinesis-based or Kafka-based process. +- **Storage** is where the Snowplow events live. We store the Snowplow events in a flat file structure on S3, and in the Redshift and PostgreSQL databases. +- **Data modeling** is where event-level data is joined with other data sets and aggregated into smaller data sets, and business logic is applied. This produces a clean set of tables which make it easier to perform analysis on the data. We have data models for Redshift and Looker. +- **Analytics** are performed on the Snowplow events or on the aggregate tables. + + + +## Snowplow schema + +We have many definitions of Snowplow's schema. We have an active issue to [standardize this schema](https://gitlab.com/gitlab-org/gitlab/-/issues/207930) including the following definitions: + +- Frontend and backend taxonomy as listed below +- [Structured event taxonomy](#structured-event-taxonomy) +- [Self describing events](https://github.com/snowplow/snowplow/wiki/Custom-events#self-describing-events) +- [Iglu schema](https://gitlab.com/gitlab-org/iglu/) +- [Snowplow authored events](https://github.com/snowplow/snowplow/wiki/Snowplow-authored-events) + +## Enabling Snowplow + +Tracking can be enabled at: + +- The instance level, which enables tracking on both the frontend and backend layers. +- User level, though user tracking can be disabled on a per-user basis. GitLab tracking respects the [Do Not Track](https://www.eff.org/issues/do-not-track) standard, so any user who has enabled the Do Not Track option in their browser is not tracked at a user level. + +We use Snowplow for the majority of our tracking strategy and it is enabled on GitLab.com. On a self-managed instance, Snowplow can be enabled by navigating to: + +- **Admin Area > Settings > General** in the UI. +- `admin/application_settings/integrations` in your browser. + +Example configuration: + +| Name | Value | +|---------------|-------------------------------| +| Collector | `your-snowplow-collector.net` | +| Site ID | `gitlab` | +| Cookie domain | `.your-gitlab-instance.com` | + +## Snowplow request flow + +The following example shows a basic request/response flow between the following components: + +- Snowplow JS / Ruby Trackers on GitLab.com +- [GitLab.com Snowplow Collector](https://gitlab.com/gitlab-com/gl-infra/readiness/-/blob/master/library/snowplow/index.md) +- The GitLab S3 Bucket +- The GitLab Snowflake Data Warehouse +- Sisense: + +```mermaid +sequenceDiagram + participant Snowplow JS (Frontend) + participant Snowplow Ruby (Backend) + participant GitLab.com Snowplow Collector + participant S3 Bucket + participant Snowflake DW + participant Sisense Dashboards + Snowplow JS (Frontend) ->> GitLab.com Snowplow Collector: FE Tracking event + Snowplow Ruby (Backend) ->> GitLab.com Snowplow Collector: BE Tracking event + loop Process using Kinesis Stream + GitLab.com Snowplow Collector ->> GitLab.com Snowplow Collector: Log raw events + GitLab.com Snowplow Collector ->> GitLab.com Snowplow Collector: Enrich events + GitLab.com Snowplow Collector ->> GitLab.com Snowplow Collector: Write to disk + end + GitLab.com Snowplow Collector ->> S3 Bucket: Kinesis Firehose + S3 Bucket->>Snowflake DW: Import data + Snowflake DW->>Snowflake DW: Transform data using dbt + Snowflake DW->>Sisense Dashboards: Data available for querying +``` + +## Structured event taxonomy + +When adding new click events, we should add them in a way that's internally consistent. If we don't, it is very painful to perform analysis across features since each feature captures events differently. + +The current method provides several attributes that are sent on each click event. Please try to follow these guidelines when specifying events to capture: + +| attribute | type | required | description | +| --------- | ------- | -------- | ----------- | +| category | text | true | The page or backend area of the application. Unless infeasible, please use the Rails page attribute by default in the frontend, and namespace + class name on the backend. | +| action | text | true | The action the user is taking, or aspect that's being instrumented. The first word should always describe the action or aspect: clicks should be `click`, activations should be `activate`, creations should be `create`, etc. Use underscores to describe what was acted on; for example, activating a form field would be `activate_form_input`. An interface action like clicking on a dropdown would be `click_dropdown`, while a behavior like creating a project record from the backend would be `create_project` | +| label | text | false | The specific element, or object that's being acted on. This is either the label of the element (e.g. a tab labeled 'Create from template' may be `create_from_template`) or a unique identifier if no text is available (e.g. closing the Groups dropdown in the top navigation bar might be `groups_dropdown_close`), or it could be the name or title attribute of a record being created. | +| property | text | false | Any additional property of the element, or object being acted on. | +| value | decimal | false | Describes a numeric value or something directly related to the event. This could be the value of an input (e.g. `10` when clicking `internal` visibility). | + +### Examples + +| category* | label | action | property** | value | +|-------------|------------------|-----------------------|----------|:-----:| +| [root:index] | main_navigation | click_navigation_link | `[link_label]` | - | +| [groups:boards:show] | toggle_swimlanes | click_toggle_button | - | `[is_active]` | +| [projects:registry:index] | registry_delete | click_button | - | - | +| [projects:registry:index] | registry_delete | confirm_deletion | - | - | +| [projects:blob:show] | congratulate_first_pipeline | click_button | `[human_access]` | - | +| [projects:clusters:new] | chart_options | generate_link | `[chart_link]` | - | +| [projects:clusters:new] | chart_options | click_add_label_button | `[label_id]` | - | + +_* It's ok to omit the category, and use the default._<br> +_** Property is usually the best place for variable strings._ + +### Reference SQL + +#### Last 20 `reply_comment_button` events + +```sql +SELECT + event_id, + v_tracker, + event_label, + event_action, + event_property, + event_value, + event_category, + contexts +FROM legacy.snowplow_structured_events_all +WHERE + event_label = 'reply_comment_button' + AND event_action = 'click_button' + -- AND event_category = 'projects:issues:show' + -- AND event_value = 1 +ORDER BY collector_tstamp DESC +LIMIT 20 +``` + +### Web-specific parameters + +Snowplow JS adds many [web-specific parameters](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/snowplow-tracker-protocol/#Web-specific_parameters) to all web events by default. + +## Implementing Snowplow JS (Frontend) tracking + +GitLab provides `Tracking`, an interface that wraps the [Snowplow JavaScript Tracker](https://github.com/snowplow/snowplow/wiki/javascript-tracker) for tracking custom events. The simplest way to use it is to add `data-` attributes to clickable elements and dropdowns. There is also a Vue mixin (exposing a `track` method), and the static method `Tracking.event`. Each of these requires at minimum a `category` and an `action`. Additional data can be provided that adheres to our [Structured event taxonomy](#structured-event-taxonomy). + +| field | type | default value | description | +|:-----------|:-------|:---------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `category` | string | `document.body.dataset.page` | Page or subsection of a page that events are being captured within. | +| `action` | string | generic | Action the user is taking. Clicks should be `click` and activations should be `activate`, so for example, focusing a form field would be `activate_form_input`, and clicking a button would be `click_button`. | +| `data` | object | `{}` | Additional data such as `label`, `property`, `value`, and `context` as described in our [Structured event taxonomy](#structured-event-taxonomy). | + +### Usage recommendations + +- Use [data attributes](#tracking-with-data-attributes) on HTML elements that emits either the `click`, `show.bs.dropdown`, or `hide.bs.dropdown` events. +- Use the [Vue mixin](#tracking-within-vue-components) when tracking custom events, or if the supported events for data attributes are not propagating. +- Use the [Tracking class directly](#tracking-in-raw-javascript) when tracking on raw JS files. + +### Tracking with data attributes + +When working within HAML (or Vue templates) we can add `data-track-*` attributes to elements of interest. All elements that have a `data-track-action` attribute automatically have event tracking bound on clicks. + +Below is an example of `data-track-*` attributes assigned to a button: + +```haml +%button.btn{ data: { track: { action: "click_button", label: "template_preview", property: "my-template" } } } +``` + +```html +<button class="btn" + data-track-action="click_button" + data-track-label="template_preview" + data-track-property="my-template" +/> +``` + +Event listeners are bound at the document level to handle click events on or within elements with these data attributes. This allows them to be properly handled on re-rendering and changes to the DOM. Note that because of the way these events are bound, click events should not be stopped from propagating up the DOM tree. If for any reason click events are being stopped from propagating, you need to implement your own listeners and follow the instructions in [Tracking within Vue components](#tracking-within-vue-components) or [Tracking in raw JavaScript](#tracking-in-raw-javascript). + +Below is a list of supported `data-track-*` attributes: + +| attribute | required | description | +|:----------------------|:---------|:------------| +| `data-track-action` | true | Action the user is taking. Clicks must be prepended with `click` and activations must be prepended with `activate`. For example, focusing a form field would be `activate_form_input` and clicking a button would be `click_button`. Replaces `data-track-event`, which was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/290962) in GitLab 13.11. | +| `data-track-label` | false | The `label` as described in our [Structured event taxonomy](#structured-event-taxonomy). | +| `data-track-property` | false | The `property` as described in our [Structured event taxonomy](#structured-event-taxonomy). | +| `data-track-value` | false | The `value` as described in our [Structured event taxonomy](#structured-event-taxonomy). If omitted, this is the element's `value` property or an empty string. For checkboxes, the default value is the element's checked attribute or `false` when unchecked. | +| `data-track-context` | false | The `context` as described in our [Structured event taxonomy](#structured-event-taxonomy). | + +#### Available helpers + +```ruby +tracking_attrs(label, action, property) # { data: { track_label... } } + +%button{ **tracking_attrs('main_navigation', 'click_button', 'navigation') } +``` + +#### Caveats + +When using the GitLab helper method [`nav_link`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/helpers/tab_helper.rb#L76) be sure to wrap `html_options` under the `html_options` keyword argument. +Be careful, as this behavior can be confused with the `ActionView` helper method [`link_to`](https://api.rubyonrails.org/v5.2.3/classes/ActionView/Helpers/UrlHelper.html#method-i-link_to) that does not require additional wrapping of `html_options` + +```ruby +# Bad += nav_link(controller: ['dashboard/groups', 'explore/groups'], data: { track_label: "explore_groups", track_action: "click_button" }) + +# Good += nav_link(controller: ['dashboard/groups', 'explore/groups'], html_options: { data: { track_label: "explore_groups", track_action: "click_button" } }) + +# Good (other helpers) += link_to explore_groups_path, title: _("Explore"), data: { track_label: "explore_groups", track_action: "click_button" } +``` + +### Tracking within Vue components + +There's a tracking Vue mixin that can be used in components if more complex tracking is required. To use it, first import the `Tracking` library and request a mixin. + +```javascript +import Tracking from '~/tracking'; +const trackingMixin = Tracking.mixin({ label: 'right_sidebar' }); +``` + +You can provide default options that are passed along whenever an event is tracked from within your component. For instance, if all events within a component should be tracked with a given `label`, you can provide one at this time. Available defaults are `category`, `label`, `property`, and `value`. If no category is specified, `document.body.dataset.page` is used as the default. + +You can then use the mixin normally in your component with the `mixin` Vue declaration. The mixin also provides the ability to specify tracking options in `data` or `computed`. These override any defaults and allow the values to be dynamic from props, or based on state. + +```javascript +export default { + mixins: [trackingMixin], + // ...[component implementation]... + data() { + return { + expanded: false, + tracking: { + label: 'left_sidebar', + }, + }; + }, +}; +``` + +The mixin provides a `track` method that can be called within the template, +or from component methods. An example of the whole implementation might look like this: + +```javascript +export default { + name: 'RightSidebar', + mixins: [Tracking.mixin({ label: 'right_sidebar' })], + data() { + return { + expanded: false, + }; + }, + methods: { + toggle() { + this.expanded = !this.expanded; + // Additional data will be merged, like `value` below + this.track('click_toggle', { value: Number(this.expanded) }); + } + } +}; +``` + +The event data can be provided with a `tracking` object, declared in the `data` function, +or as a `computed property`. + +```javascript +export default { + name: 'RightSidebar', + mixins: [Tracking.mixin()], + data() { + return { + tracking: { + label: 'right_sidebar', + // category: '', + // property: '', + // value: '', + }, + }; + }, +}; +``` + +The event data can be provided directly in the `track` function as well. +This object will merge with any previously provided options. + +```javascript +this.track('click_button', { + label: 'right_sidebar', +}); +``` + +Lastly, if needed within the template, you can use the `track` method directly as well. + +```html +<template> + <div> + <button data-testid="toggle" @click="toggle">Toggle</button> + + <div v-if="expanded"> + <p>Hello world!</p> + <button @click="track('click_action')">Track another event</button> + </div> + </div> +</template> +``` + +#### Testing example + +```javascript +import { mockTracking } from 'helpers/tracking_helper'; +// mockTracking(category, documentOverride, spyMethod) + +describe('RightSidebar.vue', () => { + let trackingSpy; + let wrapper; + + beforeEach(() => { + trackingSpy = mockTracking(undefined, wrapper.element, jest.spyOn); + }); + + const findToggle = () => wrapper.find('[data-testid="toggle"]'); + + it('tracks turning off toggle', () => { + findToggle().trigger('click'); + + expect(trackingSpy).toHaveBeenCalledWith(undefined, 'click_toggle', { + label: 'right_sidebar', + value: 0, + }); + }); +}); +``` + +### Tracking in raw JavaScript + +Custom event tracking and instrumentation can be added by directly calling the `Tracking.event` static function. The following example demonstrates tracking a click on a button by calling `Tracking.event` manually. + +```javascript +import Tracking from '~/tracking'; + +const button = document.getElementById('create_from_template_button'); + +button.addEventListener('click', () => { + Tracking.event('dashboard:projects:index', 'click_button', { + label: 'create_from_template', + property: 'template_preview', + }); +}); +``` + +#### Testing example + +```javascript +import Tracking from '~/tracking'; + +describe('MyTracking', () => { + let wrapper; + + beforeEach(() => { + jest.spyOn(Tracking, 'event'); + }); + + const findButton = () => wrapper.find('[data-testid="create_from_template"]'); + + it('tracks event', () => { + findButton().trigger('click'); + + expect(Tracking.event).toHaveBeenCalledWith(undefined, 'click_button', { + label: 'create_from_template', + property: 'template_preview', + }); + }); +}); +``` + +## Implementing Snowplow Ruby (Backend) tracking + +GitLab provides `Gitlab::Tracking`, an interface that wraps the [Snowplow Ruby Tracker](https://github.com/snowplow/snowplow/wiki/ruby-tracker) for tracking custom events. + +Custom event tracking and instrumentation can be added by directly calling the `GitLab::Tracking.event` class method, which accepts the following arguments: + +| argument | type | default value | description | +|------------|---------------------------|---------------|-----------------------------------------------------------------------------------------------------------------------------------| +| `category` | String | | Area or aspect of the application. This could be `HealthCheckController` or `Lfs::FileTransformer` for instance. | +| `action` | String | | The action being taken, which can be anything from a controller action like `create` to something like an Active Record callback. | +| `label` | String | nil | As described in [Structured event taxonomy](#structured-event-taxonomy). | +| `property` | String | nil | As described in [Structured event taxonomy](#structured-event-taxonomy). | +| `value` | Numeric | nil | As described in [Structured event taxonomy](#structured-event-taxonomy). | +| `context` | Array\[SelfDescribingJSON\] | nil | An array of custom contexts to send with this event. Most events should not have any custom contexts. | +| `project` | Project | nil | The project associated with the event. | +| `user` | User | nil | The user associated with the event. | +| `namespace` | Namespace | nil | The namespace associated with the event. | +| `extra` | Hash | `{}` | Additional keyword arguments are collected into a hash and sent with the event. | + +Tracking can be viewed as either tracking user behavior, or can be used for instrumentation to monitor and visualize performance over time in an area or aspect of code. + +For example: + +```ruby +class Projects::CreateService < BaseService + def execute + project = Project.create(params) + + Gitlab::Tracking.event('Projects::CreateService', 'create_project', label: project.errors.full_messages.to_sentence, + property: project.valid?.to_s, project: project, user: current_user, namespace: namespace) + end +end +``` + +### Unit testing + +Use the `expect_snowplow_event` helper when testing backend Snowplow events. See [testing best practices]( +https://docs.gitlab.com/ee/development/testing_guide/best_practices.html#test-snowplow-events) for details. + +### Performance + +We use the [AsyncEmitter](https://github.com/snowplow/snowplow/wiki/Ruby-Tracker#52-the-asyncemitter-class) when tracking events, which allows for instrumentation calls to be run in a background thread. This is still an active area of development. + +## Developing and testing Snowplow + +There are several tools for developing and testing Snowplow Event + +| Testing Tool | Frontend Tracking | Backend Tracking | Local Development Environment | Production Environment | Production Environment | +|----------------------------------------------|--------------------|---------------------|-------------------------------|------------------------|------------------------| +| Snowplow Analytics Debugger Chrome Extension | **{check-circle}** | **{dotted-circle}** | **{check-circle}** | **{check-circle}** | **{check-circle}** | +| Snowplow Inspector Chrome Extension | **{check-circle}** | **{dotted-circle}** | **{check-circle}** | **{check-circle}** | **{check-circle}** | +| Snowplow Micro | **{check-circle}** | **{check-circle}** | **{check-circle}** | **{dotted-circle}** | **{dotted-circle}** | +| Snowplow Mini | **{check-circle}** | **{check-circle}** | **{dotted-circle}** | **{status_preparing}** | **{status_preparing}** | + +**Legend** + +**{check-circle}** Available, **{status_preparing}** In progress, **{dotted-circle}** Not Planned + +### Snowplow Analytics Debugger Chrome Extension + +Snowplow Analytics Debugger is a browser extension for testing frontend events. This works on production, staging and local development environments. + +1. Install the [Snowplow Analytics Debugger](https://chrome.google.com/webstore/detail/snowplow-analytics-debugg/jbnlcgeengmijcghameodeaenefieedm) Chrome browser extension. +1. Open Chrome DevTools to the Snowplow Analytics Debugger tab. +1. Learn more at [Igloo Analytics](https://www.iglooanalytics.com/blog/snowplow-analytics-debugger-chrome-extension.html). + +### Snowplow Inspector Chrome Extension + +Snowplow Inspector Chrome Extension is a browser extension for testing frontend events. This works on production, staging and local development environments. + +1. Install [Snowplow Inspector](https://chrome.google.com/webstore/detail/snowplow-inspector/maplkdomeamdlngconidoefjpogkmljm?hl=en). +1. Open the Chrome extension by pressing the Snowplow Inspector icon beside the address bar. +1. Click around on a webpage with Snowplow and you should see JavaScript events firing in the inspector window. + +### Snowplow Micro + +Snowplow Micro is a very small version of a full Snowplow data collection pipeline: small enough that it can be launched by a test suite. Events can be recorded into Snowplow Micro just as they can a full Snowplow pipeline. Micro then exposes an API that can be queried. + +Snowplow Micro is a Docker-based solution for testing frontend and backend events in a local development environment. You need to modify GDK using the instructions below to set this up. + +- Read [Introducing Snowplow Micro](https://snowplowanalytics.com/blog/2019/07/17/introducing-snowplow-micro/) +- Look at the [Snowplow Micro repository](https://github.com/snowplow-incubator/snowplow-micro) +- Watch our <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [installation guide recording](https://www.youtube.com/watch?v=OX46fo_A0Ag) + +1. Ensure Docker is installed and running. + +1. Install [Snowplow Micro](https://github.com/snowplow-incubator/snowplow-micro) by cloning the settings in [this project](https://gitlab.com/gitlab-org/snowplow-micro-configuration): +1. Navigate to the directory with the cloned project, and start the appropriate Docker + container with the following script: + + ```shell + ./snowplow-micro.sh + ``` + +1. Update your instance's settings to enable Snowplow events and point to the Snowplow Micro collector: + + ```shell + gdk psql -d gitlabhq_development + update application_settings set snowplow_collector_hostname='localhost:9090', snowplow_enabled=true, snowplow_cookie_domain='.gitlab.com'; + ``` + +1. Update `DEFAULT_SNOWPLOW_OPTIONS` in `app/assets/javascripts/tracking.js` to remove `forceSecureTracker: true`: + + ```diff + diff --git a/app/assets/javascripts/tracking.js b/app/assets/javascripts/tracking.js + index 0a1211d0a76..3b98c8f28f2 100644 + --- a/app/assets/javascripts/tracking.js + +++ b/app/assets/javascripts/tracking.js + @@ -7,7 +7,6 @@ const DEFAULT_SNOWPLOW_OPTIONS = { + appId: '', + userFingerprint: false, + respectDoNotTrack: true, + - forceSecureTracker: true, + eventMethod: 'post', + contexts: { webPage: true, performanceTiming: true }, + formTracking: false, + + ``` + +1. Update `snowplow_options` in `lib/gitlab/tracking.rb` to add `protocol` and `port`: + + ```diff + diff --git a/lib/gitlab/tracking.rb b/lib/gitlab/tracking.rb + index 618e359211b..e9084623c43 100644 + --- a/lib/gitlab/tracking.rb + +++ b/lib/gitlab/tracking.rb + @@ -41,7 +41,9 @@ def snowplow_options(group) + cookie_domain: Gitlab::CurrentSettings.snowplow_cookie_domain, + app_id: Gitlab::CurrentSettings.snowplow_app_id, + form_tracking: additional_features, + - link_click_tracking: additional_features + + link_click_tracking: additional_features, + + protocol: 'http', + + port: 9090 + }.transform_keys! { |key| key.to_s.camelize(:lower).to_sym } + end + ``` + +1. Update `emitter` in `lib/gitlab/tracking/destinations/snowplow.rb` to change `protocol`: + + ```diff + diff --git a/lib/gitlab/tracking/destinations/snowplow.rb b/lib/gitlab/tracking/destinations/snowplow.rb + index 4fa844de325..5dd9d0eacfb 100644 + --- a/lib/gitlab/tracking/destinations/snowplow.rb + +++ b/lib/gitlab/tracking/destinations/snowplow.rb + @@ -40,7 +40,7 @@ def tracker + def emitter + SnowplowTracker::AsyncEmitter.new( + Gitlab::CurrentSettings.snowplow_collector_hostname, + - protocol: 'https' + + protocol: 'http' + ) + end + end + + ``` + +1. Restart GDK: + + ```shell + `gdk restart` + ``` + +1. Send a test Snowplow event from the Rails console: + + ```ruby + Gitlab::Tracking.event('category', 'action') + ``` + +1. Navigate to `localhost:9090/micro/good` to see the event. + +### Snowplow Mini + +[Snowplow Mini](https://github.com/snowplow/snowplow-mini) is an easily-deployable, single-instance version of Snowplow. + +Snowplow Mini can be used for testing frontend and backend events on a production, staging and local development environment. + +For GitLab.com, we're setting up a [QA and Testing environment](https://gitlab.com/gitlab-org/telemetry/-/issues/266) using Snowplow Mini. + +## Snowplow Schemas + +### `gitlab_standard` + +We are including the [`gitlab_standard` schema](https://gitlab.com/gitlab-org/iglu/-/blob/master/public/schemas/com.gitlab/gitlab_standard/jsonschema/) with every event. See [Standardize Snowplow Schema](https://gitlab.com/groups/gitlab-org/-/epics/5218) for details. + +The [`StandardContext`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/tracking/standard_context.rb) class represents this schema in the application. + +| Field Name | Required | Type | Description | +|----------------|---------------------|-----------------------|---------------------------------------------------------------------------------------------| +| `project_id` | **{dotted-circle}** | integer | | +| `namespace_id` | **{dotted-circle}** | integer | | +| `environment` | **{check-circle}** | string (max 32 chars) | Name of the source environment, such as `production` or `staging` | +| `source` | **{check-circle}** | string (max 32 chars) | Name of the source application, such as `gitlab-rails` or `gitlab-javascript` | +| `extra` | **{dotted-circle}** | JSON | Any additional data associated with the event, in the form of key-value pairs | + +### Default Schema + +| Field Name | Required | Type | Description | +|--------------------------|---------------------|-----------|----------------------------------------------------------------------------------------------------------------------------------| +| `app_id` | **{check-circle}** | string | Unique identifier for website / application | +| `base_currency` | **{dotted-circle}** | string | Reporting currency | +| `br_colordepth` | **{dotted-circle}** | integer | Browser color depth | +| `br_cookies` | **{dotted-circle}** | boolean | Does the browser permit cookies? | +| `br_family` | **{dotted-circle}** | string | Browser family | +| `br_features_director` | **{dotted-circle}** | boolean | Director plugin installed? | +| `br_features_flash` | **{dotted-circle}** | boolean | Flash plugin installed? | +| `br_features_gears` | **{dotted-circle}** | boolean | Google gears installed? | +| `br_features_java` | **{dotted-circle}** | boolean | Java plugin installed? | +| `br_features_pdf` | **{dotted-circle}** | boolean | Adobe PDF plugin installed? | +| `br_features_quicktime` | **{dotted-circle}** | boolean | Quicktime plugin installed? | +| `br_features_realplayer` | **{dotted-circle}** | boolean | RealPlayer plugin installed? | +| `br_features_silverlight` | **{dotted-circle}** | boolean | Silverlight plugin installed? | +| `br_features_windowsmedia` | **{dotted-circle}** | boolean | Windows media plugin installed? | +| `br_lang` | **{dotted-circle}** | string | Language the browser is set to | +| `br_name` | **{dotted-circle}** | string | Browser name | +| `br_renderengine` | **{dotted-circle}** | string | Browser rendering engine | +| `br_type` | **{dotted-circle}** | string | Browser type | +| `br_version` | **{dotted-circle}** | string | Browser version | +| `br_viewheight` | **{dotted-circle}** | string | Browser viewport height | +| `br_viewwidth` | **{dotted-circle}** | string | Browser viewport width | +| `collector_tstamp` | **{dotted-circle}** | timestamp | Time stamp for the event recorded by the collector | +| `contexts` | **{dotted-circle}** | | | +| `derived_contexts` | **{dotted-circle}** | | Contexts derived in the Enrich process | +| `derived_tstamp` | **{dotted-circle}** | timestamp | Timestamp making allowance for inaccurate device clock | +| `doc_charset` | **{dotted-circle}** | string | Web page's character encoding | +| `doc_height` | **{dotted-circle}** | string | Web page height | +| `doc_width` | **{dotted-circle}** | string | Web page width | +| `domain_sessionid` | **{dotted-circle}** | string | Unique identifier (UUID) for this visit of this user_id to this domain | +| `domain_sessionidx` | **{dotted-circle}** | integer | Index of number of visits that this user_id has made to this domain (The first visit is `1`) | +| `domain_userid` | **{dotted-circle}** | string | Unique identifier for a user, based on a first party cookie (so domain specific) | +| `dvce_created_tstamp` | **{dotted-circle}** | timestamp | Timestamp when event occurred, as recorded by client device | +| `dvce_ismobile` | **{dotted-circle}** | boolean | Indicates whether device is mobile | +| `dvce_screenheight` | **{dotted-circle}** | string | Screen / monitor resolution | +| `dvce_screenwidth` | **{dotted-circle}** | string | Screen / monitor resolution | +| `dvce_sent_tstamp` | **{dotted-circle}** | timestamp | Timestamp when event was sent by client device to collector | +| `dvce_type` | **{dotted-circle}** | string | Type of device | +| `etl_tags` | **{dotted-circle}** | string | JSON of tags for this ETL run | +| `etl_tstamp` | **{dotted-circle}** | timestamp | Timestamp event began ETL | +| `event` | **{dotted-circle}** | string | Event type | +| `event_fingerprint` | **{dotted-circle}** | string | Hash client-set event fields | +| `event_format` | **{dotted-circle}** | string | Format for event | +| `event_id` | **{dotted-circle}** | string | Event UUID | +| `event_name` | **{dotted-circle}** | string | Event name | +| `event_vendor` | **{dotted-circle}** | string | The company who developed the event model | +| `event_version` | **{dotted-circle}** | string | Version of event schema | +| `geo_city` | **{dotted-circle}** | string | City of IP origin | +| `geo_country` | **{dotted-circle}** | string | Country of IP origin | +| `geo_latitude` | **{dotted-circle}** | string | An approximate latitude | +| `geo_longitude` | **{dotted-circle}** | string | An approximate longitude | +| `geo_region` | **{dotted-circle}** | string | Region of IP origin | +| `geo_region_name` | **{dotted-circle}** | string | Region of IP origin | +| `geo_timezone` | **{dotted-circle}** | string | Timezone of IP origin | +| `geo_zipcode` | **{dotted-circle}** | string | Zip (postal) code of IP origin | +| `ip_domain` | **{dotted-circle}** | string | Second level domain name associated with the visitor's IP address | +| `ip_isp` | **{dotted-circle}** | string | Visitor's ISP | +| `ip_netspeed` | **{dotted-circle}** | string | Visitor's connection type | +| `ip_organization` | **{dotted-circle}** | string | Organization associated with the visitor's IP address – defaults to ISP name if none is found | +| `mkt_campaign` | **{dotted-circle}** | string | The campaign ID | +| `mkt_clickid` | **{dotted-circle}** | string | The click ID | +| `mkt_content` | **{dotted-circle}** | string | The content or ID of the ad. | +| `mkt_medium` | **{dotted-circle}** | string | Type of traffic source | +| `mkt_network` | **{dotted-circle}** | string | The ad network to which the click ID belongs | +| `mkt_source` | **{dotted-circle}** | string | The company / website where the traffic came from | +| `mkt_term` | **{dotted-circle}** | string | Keywords associated with the referrer | +| `name_tracker` | **{dotted-circle}** | string | The tracker namespace | +| `network_userid` | **{dotted-circle}** | string | Unique identifier for a user, based on a cookie from the collector (so set at a network level and shouldn't be set by a tracker) | +| `os_family` | **{dotted-circle}** | string | Operating system family | +| `os_manufacturer` | **{dotted-circle}** | string | Manufacturers of operating system | +| `os_name` | **{dotted-circle}** | string | Name of operating system | +| `os_timezone` | **{dotted-circle}** | string | Client operating system timezone | +| `page_referrer` | **{dotted-circle}** | string | Referrer URL | +| `page_title` | **{dotted-circle}** | string | Page title | +| `page_url` | **{dotted-circle}** | string | Page URL | +| `page_urlfragment` | **{dotted-circle}** | string | Fragment aka anchor | +| `page_urlhost` | **{dotted-circle}** | string | Host aka domain | +| `page_urlpath` | **{dotted-circle}** | string | Path to page | +| `page_urlport` | **{dotted-circle}** | integer | Port if specified, 80 if not | +| `page_urlquery` | **{dotted-circle}** | string | Query string | +| `page_urlscheme` | **{dotted-circle}** | string | Scheme (protocol name) | +| `platform` | **{dotted-circle}** | string | The platform the app runs on | +| `pp_xoffset_max` | **{dotted-circle}** | integer | Maximum page x offset seen in the last ping period | +| `pp_xoffset_min` | **{dotted-circle}** | integer | Minimum page x offset seen in the last ping period | +| `pp_yoffset_max` | **{dotted-circle}** | integer | Maximum page y offset seen in the last ping period | +| `pp_yoffset_min` | **{dotted-circle}** | integer | Minimum page y offset seen in the last ping period | +| `refr_domain_userid` | **{dotted-circle}** | string | The Snowplow `domain_userid` of the referring website | +| `refr_dvce_tstamp` | **{dotted-circle}** | timestamp | The time of attaching the `domain_userid` to the inbound link | +| `refr_medium` | **{dotted-circle}** | string | Type of referer | +| `refr_source` | **{dotted-circle}** | string | Name of referer if recognised | +| `refr_term` | **{dotted-circle}** | string | Keywords if source is a search engine | +| `refr_urlfragment` | **{dotted-circle}** | string | Referer URL fragment | +| `refr_urlhost` | **{dotted-circle}** | string | Referer host | +| `refr_urlpath` | **{dotted-circle}** | string | Referer page path | +| `refr_urlport` | **{dotted-circle}** | integer | Referer port | +| `refr_urlquery` | **{dotted-circle}** | string | Referer URL query string | +| `refr_urlscheme` | **{dotted-circle}** | string | Referer scheme | +| `se_action` | **{dotted-circle}** | string | The action / event itself | +| `se_category` | **{dotted-circle}** | string | The category of event | +| `se_label` | **{dotted-circle}** | string | A label often used to refer to the 'object' the action is performed on | +| `se_property` | **{dotted-circle}** | string | A property associated with either the action or the object | +| `se_value` | **{dotted-circle}** | decimal | A value associated with the user action | +| `ti_category` | **{dotted-circle}** | string | Item category | +| `ti_currency` | **{dotted-circle}** | string | Currency | +| `ti_name` | **{dotted-circle}** | string | Item name | +| `ti_orderid` | **{dotted-circle}** | string | Order ID | +| `ti_price` | **{dotted-circle}** | decimal | Item price | +| `ti_price_base` | **{dotted-circle}** | decimal | Item price in base currency | +| `ti_quantity` | **{dotted-circle}** | integer | Item quantity | +| `ti_sku` | **{dotted-circle}** | string | Item SKU | +| `tr_affiliation` | **{dotted-circle}** | string | Transaction affiliation (such as channel) | +| `tr_city` | **{dotted-circle}** | string | Delivery address: city | +| `tr_country` | **{dotted-circle}** | string | Delivery address: country | +| `tr_currency` | **{dotted-circle}** | string | Transaction Currency | +| `tr_orderid` | **{dotted-circle}** | string | Order ID | +| `tr_shipping` | **{dotted-circle}** | decimal | Delivery cost charged | +| `tr_shipping_base` | **{dotted-circle}** | decimal | Shipping cost in base currency | +| `tr_state` | **{dotted-circle}** | string | Delivery address: state | +| `tr_tax` | **{dotted-circle}** | decimal | Transaction tax value (such as amount of VAT included) | +| `tr_tax_base` | **{dotted-circle}** | decimal | Tax applied in base currency | +| `tr_total` | **{dotted-circle}** | decimal | Transaction total value | +| `tr_total_base` | **{dotted-circle}** | decimal | Total amount of transaction in base currency | +| `true_tstamp` | **{dotted-circle}** | timestamp | User-set exact timestamp | +| `txn_id` | **{dotted-circle}** | string | Transaction ID | +| `unstruct_event` | **{dotted-circle}** | JSON | The properties of the event | +| `uploaded_at` | **{dotted-circle}** | | | +| `user_fingerprint` | **{dotted-circle}** | integer | User identifier based on (hopefully unique) browser features | +| `user_id` | **{dotted-circle}** | string | Unique identifier for user, set by the business using setUserId | +| `user_ipaddress` | **{dotted-circle}** | string | IP address | +| `useragent` | **{dotted-circle}** | string | User agent (expressed as a browser string) | +| `v_collector` | **{dotted-circle}** | string | Collector version | +| `v_etl` | **{dotted-circle}** | string | ETL version | +| `v_tracker` | **{dotted-circle}** | string | Identifier for Snowplow tracker | diff --git a/doc/development/sql.md b/doc/development/sql.md index 8726c1331e8..a98645cfcae 100644 --- a/doc/development/sql.md +++ b/doc/development/sql.md @@ -313,9 +313,9 @@ Project.from("(#{union.to_sql}) projects") ## Ordering by Creation Date -When ordering records based on the time they were created you can simply order +When ordering records based on the time they were created, you can order by the `id` column instead of ordering by `created_at`. Because IDs are always -unique and incremented in the order that rows are created this will produce the +unique and incremented in the order that rows are created, doing so will produce the exact same results. This also means there's no need to add an index on `created_at` to ensure consistent performance as `id` is already indexed by default. @@ -367,3 +367,12 @@ retries if it were to fail because of an To be able to use this method, make sure the model you want to use this on inherits from `ApplicationRecord`. + +## Monitor SQL queries in production + +GitLab team members can monitor slow or canceled queries on GitLab.com +using the PostgreSQL logs, which are indexed in Elasticsearch and +searchable using Kibana. + +See [the runbook](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/patroni/pg_collect_query_data.md#searching-postgresql-logs-with-kibanaelasticsearch) +for more details. diff --git a/doc/development/stage_group_dashboards.md b/doc/development/stage_group_dashboards.md index e75237869ba..58e998e46a8 100644 --- a/doc/development/stage_group_dashboards.md +++ b/doc/development/stage_group_dashboards.md @@ -18,6 +18,55 @@ The list of dashboards for each stage group is accessible at <https://dashboards The dashboards for stage groups are at a very early stage. All contributions are welcome. If you have any questions or suggestions, please submit an issue in the [Scalability Team issues tracker](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/new). +## Dashboard content + +### Error budget + +Read more about how we are using error budgets overall in our +[handbook](https://about.gitlab.com/handbook/engineering/error-budgets/). + +By default, the first row of panels on the dashbhoard will show the [error +budget for the stage +group](https://about.gitlab.com/handbook/engineering/error-budgets/#budget-spend-by-stage-group). This +row shows how the features owned by +the group are contributing to our [overall +availability](https://about.gitlab.com/handbook/engineering/infrastructure/performance-indicators/#gitlabcom-availability). + +The budget is always aggregated over the 28 days before the [time +selected on the dashboard](#time-range-controls). + +We're currently displaying the information in 2 formats: + +1. Availability: This number can be compared to GitLab.com's overall + availability target of 99.95% uptime. +1. Budget Spent: This shows the time over the past 28 days that + features owned by the group have not been performing adequately. + +We're still discussing which of these is more understandable, please +contribute in +[Scalability issue #946](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/946) +if you have thoughts on this topic. + +The budget is calculated based on indicators per component. Each +component has 2 indicators: + +1. [Apdex](https://en.wikipedia.org/wiki/Apdex): The rate of + operations that performed adequately. +1. Error rate: The rate of operations that had errors. + +The calculation to a ratio then happens as follows: + +```math +\frac {operations\_meeting\_apdex + (total\_operations - operations\_with_\errors)} {total\_apdex\_measurements + total\_operations} +``` + +*Caveat:* Not all components are included, causing the +calculation to be less accurate for some groups. We're working on +adding all components in +[&437](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/437). This +could cause the dashboard to display "No Data" for features with lower +traffic. + ## Usage Inside a stage group dashboard, there are some notable components. Let's take the [Source Code group's dashboard](https://dashboards.gitlab.net/d/stage-groups-source_code/stage-groups-group-dashboard-create-source-code?orgId=1) as an example. diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index ee8401e08d4..828e9925d46 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -86,7 +86,7 @@ a parent context. Examples of these are: - `:clean_gitlab_redis_cache` which provides a clean Redis cache to the examples. - `:request_store` which provides a request store to the examples. -Obviously we should reduce test dependencies, and avoiding +We should reduce test dependencies, and avoiding capabilities also reduces the amount of set-up needed. `:js` is particularly important to avoid. This must only be used if the feature @@ -350,12 +350,140 @@ writing one](testing_levels.md#consider-not-writing-a-system-test)! - It's ok to look for DOM elements, but don't abuse it, because it makes the tests more brittle -#### Debugging Capybara +#### UI testing -Sometimes you may need to debug Capybara tests by observing browser behavior. +When testing the UI, write tests that simulate what a user sees and how they interact with the UI. +This means preferring Capybara's semantic methods and avoiding querying by IDs, classes, or attributes. + +The benefits of testing in this way are that: + +- It ensures all interactive elements have an [accessible name](../fe_guide/accessibility.md#provide-accessible-names-for-screen-readers). +- It is more readable, as it uses more natural language. +- It is less brittle, as it avoids querying by IDs, classes, and attributes, which are not visible to the user. + +We strongly recommend that you query by the element's text label instead of by ID, class name, or `data-testid`. + +If needed, you can scope interactions within a specific area of the page by using `within`. +As you will likely be scoping to an element such as a `div`, which typically does not have a label, +you may use a `data-testid` selector in this case. + +##### Actions + +Where possible, use more specific [actions](https://rubydoc.info/github/teamcapybara/capybara/master/Capybara/Node/Actions), such as the ones below. + +```ruby +# good +click_button 'Submit review' + +click_link 'UI testing docs' + +fill_in 'Search projects', with: 'gitlab' # fill in text input with text + +select 'Last updated', from: 'Sort by' # select an option from a select input + +check 'Checkbox label' +uncheck 'Checkbox label' + +choose 'Radio input label' + +attach_file('Attach a file', '/path/to/file.png') + +# bad - interactive elements must have accessible names, so +# we should be able to use one of the specific actions above +find('.group-name', text: group.name).click +find('.js-show-diff-settings').click +find('[data-testid="submit-review"]').click +find('input[type="checkbox"]').click +find('.search').native.send_keys('gitlab') +``` + +##### Finders + +Where possible, use more specific [finders](https://rubydoc.info/github/teamcapybara/capybara/master/Capybara/Node/Finders), such as the ones below. + +```ruby +# good +find_button 'Submit review' +find_button 'Submit review', disabled: true + +find_link 'UI testing docs' +find_link 'UI testing docs', href: docs_url + +find_field 'Search projects' +find_field 'Search projects', with: 'gitlab' # find the input field with text +find_field 'Search projects', disabled: true +find_field 'Checkbox label', checked: true +find_field 'Checkbox label', unchecked: true + +# acceptable when finding a element that is not a button, link, or field +find('[data-testid="element"]') +``` + +##### Matchers + +Where possible, use more specific [matchers](https://rubydoc.info/github/teamcapybara/capybara/master/Capybara/RSpecMatchers), such as the ones below. + +```ruby +# good +expect(page).to have_button 'Submit review' +expect(page).to have_button 'Submit review', disabled: true +expect(page).to have_button 'Notifications', class: 'is-checked' # assert the "Notifications" GlToggle is checked + +expect(page).to have_link 'UI testing docs' +expect(page).to have_link 'UI testing docs', href: docs_url # assert the link has an href + +expect(page).to have_field 'Search projects' +expect(page).to have_field 'Search projects', disabled: true +expect(page).to have_field 'Search projects', with: 'gitlab' # assert the input field has text + +expect(page).to have_checked_field 'Checkbox label' +expect(page).to have_unchecked_field 'Radio input label' + +expect(page).to have_select 'Sort by' +expect(page).to have_select 'Sort by', selected: 'Last updated' # assert the option is selected +expect(page).to have_select 'Sort by', options: ['Last updated', 'Created date', 'Due date'] # assert an exact list of options +expect(page).to have_select 'Sort by', with_options: ['Created date', 'Due date'] # assert a partial list of options + +expect(page).to have_text 'Some paragraph text.' +expect(page).to have_text 'Some paragraph text.', exact: true # assert exact match + +expect(page).to have_current_path 'gitlab/gitlab-test/-/issues' + +expect(page).to have_title 'Not Found' + +# acceptable when a more specific matcher above is not possible +expect(page).to have_css 'h2', text: 'Issue title' +expect(page).to have_css 'p', text: 'Issue description', exact: true +expect(page).to have_css '[data-testid="weight"]', text: 2 +expect(page).to have_css '.atwho-view ul', visible: true +``` + +##### Other useful methods + +After you retrieve an element using a [finder method](#finders), you can invoke a number of +[element methods](https://rubydoc.info/github/teamcapybara/capybara/master/Capybara/Node/Element) +on it, such as `hover`. + +Capybara tests also have a number of [session methods](https://rubydoc.info/github/teamcapybara/capybara/master/Capybara/Session) available, such as `accept_confirm`. + +Some other useful methods are shown below: + +```ruby +refresh # refresh the page + +send_keys([:shift, 'i']) # press Shift+I keys to go to the Issues dashboard page + +current_window.resize_to(1000, 1000) # resize the window + +scroll_to(find_field('Comment')) # scroll to an element +``` + +You can also find a number of GitLab custom helpers in the `spec/support/helpers/` directory. #### Live debug +Sometimes you may need to debug Capybara tests by observing browser behavior. + You can pause Capybara and view the website on the browser by using the `live_debug` method in your spec. The current page is automatically opened in your default browser. diff --git a/doc/development/testing_guide/end_to_end/beginners_guide.md b/doc/development/testing_guide/end_to_end/beginners_guide.md index d60b780eeea..29f6c93d65a 100644 --- a/doc/development/testing_guide/end_to_end/beginners_guide.md +++ b/doc/development/testing_guide/end_to_end/beginners_guide.md @@ -332,12 +332,13 @@ can see it. ## Run the spec -Before running the spec, confirm: +Before running the spec, make sure that: -- The GDK is installed. -- The GDK is running on port 3000 locally. +- GDK is installed. +- GDK is running locally on port 3000. - No additional [RSpec metadata tags](rspec_metadata_tests.md) have been applied. - Your working directory is `qa/` within your GDK GitLab installation. +- Your GitLab instance-level settings are default. If you changed the default settings, some tests might have unexpected results. To run the spec, run the following command: diff --git a/doc/development/testing_guide/end_to_end/best_practices.md b/doc/development/testing_guide/end_to_end/best_practices.md index 2b4212a0172..15520d8a6b1 100644 --- a/doc/development/testing_guide/end_to_end/best_practices.md +++ b/doc/development/testing_guide/end_to_end/best_practices.md @@ -223,6 +223,15 @@ In summary: - **Do**: Split tests across separate files, unless the tests share expensive setup. - **Don't**: Put new tests in an existing file without considering the impact on parallelization. +## `let` variables vs instance variables + +By default, follow the [testing best practices](../best_practices.md#subject-and-let-variables) when using `let` +or instance variables. However, in end-to-end tests, set-ups such as creating resources are expensive. +If you use `let` to store a resource, it will be created for each example separately. +If the resource can be shared among multiple examples, use an instance variable in the `before(:all)` +block instead of `let` to save run time. +When the variable cannot be shared by multiple examples, use `let`. + ## Limit the use of the UI in `before(:context)` and `after` hooks Limit the use of `before(:context)` hooks to perform setup tasks with only API calls, diff --git a/doc/development/testing_guide/end_to_end/index.md b/doc/development/testing_guide/end_to_end/index.md index d981f9bcbba..e6da4771e55 100644 --- a/doc/development/testing_guide/end_to_end/index.md +++ b/doc/development/testing_guide/end_to_end/index.md @@ -22,13 +22,13 @@ a black-box testing framework for the API and the UI. ### Testing nightly builds We run scheduled pipelines each night to test nightly builds created by Omnibus. -You can find these nightly pipelines at `https://gitlab.com/gitlab-org/quality/nightly/pipelines` +You can find these pipelines at <https://gitlab.com/gitlab-org/quality/nightly/pipelines> (need Developer access permissions). Results are reported in the `#qa-nightly` Slack channel. ### Testing staging We run scheduled pipelines each night to test staging. -You can find these nightly pipelines at `https://gitlab.com/gitlab-org/quality/staging/pipelines` +You can find these pipelines at <https://gitlab.com/gitlab-org/quality/staging/pipelines> (need Developer access permissions). Results are reported in the `#qa-staging` Slack channel. ### Testing code in merge requests @@ -36,64 +36,76 @@ You can find these nightly pipelines at `https://gitlab.com/gitlab-org/quality/s #### Using the `package-and-qa` job It is possible to run end-to-end tests for a merge request, eventually being run in -a pipeline in the [`gitlab-qa-mirror`](https://gitlab.com/gitlab-org/gitlab-qa-mirror/) project, -by triggering the `package-and-qa` manual action in the `test` stage (not +a pipeline in the [`gitlab-org/gitlab-qa-mirror`](https://gitlab.com/gitlab-org/gitlab-qa-mirror) project, +by triggering the `package-and-qa` manual action in the `qa` stage (not available for forks). -**This runs end-to-end tests against a custom CE and EE (with an Ultimate license) -Omnibus package built from your merge request's changes.** +**This runs end-to-end tests against a custom EE (with an Ultimate license) +Docker image built from your merge request's changes.** -Manual action that starts end-to-end tests is also available in merge requests -in [Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab). - -Below you can read more about how to use it and how does it work. +Manual action that starts end-to-end tests is also available +in [`gitlab-org/omnibus-gitlab` merge requests](https://docs.gitlab.com/omnibus/build/team_member_docs.html#i-have-an-mr-in-the-omnibus-gitlab-project-and-want-a-package-or-docker-image-to-test-it). #### How does it work? -Currently, we are using _multi-project pipeline_-like approach to run QA +Currently, we are using _multi-project pipeline_-like approach to run end-to-end pipelines. ```mermaid -graph LR - A1 -.->|1. Triggers an omnibus-gitlab-mirror pipeline and wait for it to be done| A2 - B2[`Trigger-qa` stage<br>`Trigger:qa-test` job] -.->|2. Triggers a gitlab-qa-mirror pipeline and wait for it to be done| A3 - -subgraph "gitlab-foss/gitlab pipeline" - A1[`test` stage<br>`package-and-qa` job] +graph TB + A1 -.->|once done, can be triggered| A2 + A2 -.->|1. Triggers an `omnibus-gitlab-mirror` pipeline<br>and wait for it to be done| B1 + B2[`Trigger-qa` stage<br>`Trigger:qa-test` job] -.->|2. Triggers a `gitlab-qa-mirror` pipeline<br>and wait for it to be done| C1 + +subgraph "`gitlab-org/gitlab` pipeline" + A1[`build-images` stage<br>`build-qa-image` and `build-assets-image` jobs] + A2[`qa` stage<br>`package-and-qa` job] end -subgraph "omnibus-gitlab pipeline" - A2[`Trigger-docker` stage<br>`Trigger:gitlab-docker` job] -->|once done| B2 +subgraph "`gitlab-org/build/omnibus-gitlab-mirror` pipeline" + B1[`Trigger-docker` stage<br>`Trigger:gitlab-docker` job] -->|once done| B2 end -subgraph "gitlab-qa-mirror pipeline" - A3>QA jobs run] -.->|3. Reports back the pipeline result to the `package-and-qa` job<br>and post the result on the original commit tested| A1 +subgraph "`gitlab-org/gitlab-qa-mirror` pipeline" + C1>End-to-end jobs run] end ``` -1. Developer triggers a manual action, that can be found in GitLab merge - requests. This starts a chain of pipelines in multiple projects. - -1. The script being executed triggers a pipeline in - [Omnibus GitLab Mirror](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror) - and waits for the resulting status. We call this a _status attribution_. - -1. GitLab packages are being built in the [Omnibus GitLab Mirror](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror) - pipeline. Packages are then pushed to its Container Registry. +1. In the [`gitlab-org/gitlab` pipeline](https://gitlab.com/gitlab-org/gitlab): + 1. Developer triggers the `package-and-qa` manual action (available once the `build-qa-image` and + `build-assets-image` jobs are done), that can be found in GitLab merge + requests. This starts a chain of pipelines in multiple projects. + 1. The script being executed triggers a pipeline in + [`gitlab-org/build/omnibus-gitlab-mirror`](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror) + and polls for the resulting status. We call this a _status attribution_. -1. When packages are ready, and available in the registry, a final step in the - [Omnibus GitLab Mirror](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror) pipeline, triggers a new - GitLab QA pipeline (those with access can view them at `https://gitlab.com/gitlab-org/gitlab-qa-mirror/pipelines`). It also waits for a resulting status. +1. In the [`gitlab-org/build/omnibus-gitlab-mirror` pipeline](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror): + 1. Docker image is being built and pushed to its Container Registry. + 1. Finally, the `Trigger:qa-test` job triggers a new end-to-end pipeline in + [`gitlab-org/gitlab-qa-mirror`](https://gitlab.com/gitlab-org/gitlab-qa-mirror/pipelines) and polls for the resulting status. -1. GitLab QA pulls images from the registry, spins-up containers and runs tests - against a test environment that has been just orchestrated by the `gitlab-qa` - tool. +1. In the [`gitlab-org/gitlab-qa-mirror` pipeline](https://gitlab.com/gitlab-org/gitlab-qa-mirror): + 1. Container for the Docker image stored in the [`gitlab-org/build/omnibus-gitlab-mirror`](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror) registry is spun-up. + 1. End-to-end tests are run with the `gitlab-qa` executable, which spin up a container for the end-to-end image from the [`gitlab-org/gitlab`](https://gitlab.com/gitlab-org/gitlab) registry. -1. The result of the GitLab QA pipeline is being - propagated upstream, through Omnibus, back to the GitLab merge request. +1. The result of the [`gitlab-org/gitlab-qa-mirror` pipeline](https://gitlab.com/gitlab-org/gitlab-qa-mirror) is being + propagated upstream (through polling from upstream pipelines), through [`gitlab-org/build/omnibus-gitlab-mirror`](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror), back to the [`gitlab-org/gitlab`](https://gitlab.com/gitlab-org/gitlab) merge request. Please note, we plan to [add more specific information](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/156) -about the tests included in each job/scenario that runs in `gitlab-qa-mirror`. +about the tests included in each job/scenario that runs in `gitlab-org/gitlab-qa-mirror`. + +NOTE: +You may have noticed that we use `gitlab-org/build/omnibus-gitlab-mirror` instead of +`gitlab-org/omnibus-gitlab`, and `gitlab-org/gitlab-qa-mirror` instead of `gitlab-org/gitlab-qa`. +This is due to technical limitations in the GitLab permission model: the ability to run a pipeline +against a protected branch is controlled by the ability to push/merge to this branch. +This means that for developers to be able to trigger a pipeline for the default branch in +`gitlab-org/omnibus-gitlab`/`gitlab-org/gitlab-qa`, they would need to have Maintainer permission in those projects. +For security reasons and implications, we couldn't open up the default branch to all the Developers. +Hence we created these mirrors where Developers and Maintainers are allowed to push/merge to the default branch. +This problem was discovered in <https://gitlab.com/gitlab-org/gitlab-qa/-/issues/63#note_107175160> and the "mirror" +work-around was suggested in <https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4717>. +A feature proposal to segregate access control regarding running pipelines from ability to push/merge was also created at <https://gitlab.com/gitlab-org/gitlab/-/issues/24585>. #### With Pipeline for Merged Results @@ -160,9 +172,9 @@ See [Review Apps](../review_apps.md) for more details about Review Apps. ## How do I run the tests? If you are not [testing code in a merge request](#testing-code-in-merge-requests), -there are two main options for running the tests. If you simply want to run -the existing tests against a live GitLab instance or against a pre-built Docker image -you can use the [GitLab QA orchestrator](https://gitlab.com/gitlab-org/gitlab-qa/tree/master/README.md). See also [examples +there are two main options for running the tests. If you want to run +the existing tests against a live GitLab instance or against a pre-built Docker image, +use the [GitLab QA orchestrator](https://gitlab.com/gitlab-org/gitlab-qa/tree/master/README.md). See also [examples of the test scenarios you can run via the orchestrator](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#examples). On the other hand, if you would like to run against a local development GitLab diff --git a/doc/development/testing_guide/end_to_end/style_guide.md b/doc/development/testing_guide/end_to_end/style_guide.md index f9c13d5dd67..22ddae6e836 100644 --- a/doc/development/testing_guide/end_to_end/style_guide.md +++ b/doc/development/testing_guide/end_to_end/style_guide.md @@ -122,7 +122,7 @@ avoid confusion or make the code more readable. For example, if a page object is named `New`, it could be confusing to name the block argument `new` because that is used to instantiate objects, so `new_page` would be acceptable. -We chose not to simply use `page` because that would shadow the +We chose not to use `page` because that would shadow the Capybara DSL, potentially leading to confusion and bugs. ### Examples diff --git a/doc/development/testing_guide/flaky_tests.md b/doc/development/testing_guide/flaky_tests.md index 126d8725c21..a9af8f03d63 100644 --- a/doc/development/testing_guide/flaky_tests.md +++ b/doc/development/testing_guide/flaky_tests.md @@ -108,7 +108,7 @@ For instance `RETRIES=1 bin/rspec ...` would retry the failing examples once. #### PhantomJS / WebKit related issues -- Memory is through the roof! (TL;DR: Load images but block images requests!): <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12003> +- Memory is through the roof! (Load images but block images requests!): <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12003> #### Capybara expectation times out diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md index 9facca10142..7289e66a045 100644 --- a/doc/development/testing_guide/frontend_testing.md +++ b/doc/development/testing_guide/frontend_testing.md @@ -55,8 +55,8 @@ which have to be stubbed. - Because Jest runs in a Node.js environment, it uses [jsdom](https://github.com/jsdom/jsdom) by default. See also its [limitations](#limitations-of-jsdom) below. - Jest does not have access to Webpack loaders or aliases. The aliases used by Jest are defined in its [own configuration](https://gitlab.com/gitlab-org/gitlab/blob/master/jest.config.js). -- All calls to `setTimeout` and `setInterval` are mocked away. See also [Jest Timer Mocks](https://jestjs.io/docs/en/timer-mocks). -- `rewire` is not required because Jest supports mocking modules. See also [Manual Mocks](https://jestjs.io/docs/en/manual-mocks). +- All calls to `setTimeout` and `setInterval` are mocked away. See also [Jest Timer Mocks](https://jestjs.io/docs/timer-mocks). +- `rewire` is not required because Jest supports mocking modules. See also [Manual Mocks](https://jestjs.io/docs/manual-mocks). - No [context object](https://jasmine.github.io/tutorials/your_first_suite#section-The_%3Ccode%3Ethis%3C/code%3E_keyword) is passed to tests in Jest. This means sharing `this.something` between `beforeEach()` and `it()` for example does not work. Instead you should declare shared variables in the context that they are needed (via `const` / `let`). @@ -78,7 +78,7 @@ See also the issue for [support running Jest tests in browsers](https://gitlab.c ### Debugging Jest tests -Running `yarn jest-debug` runs Jest in debug mode, allowing you to debug/inspect as described in the [Jest docs](https://jestjs.io/docs/en/troubleshooting#tests-are-failing-and-you-don-t-know-why). +Running `yarn jest-debug` runs Jest in debug mode, allowing you to debug/inspect as described in the [Jest docs](https://jestjs.io/docs/troubleshooting#tests-are-failing-and-you-don-t-know-why). ### Timeout error @@ -104,6 +104,15 @@ describe('Component', () => { Remember that the performance of each test depends on the environment. +### Test-specific stylesheets + +To help facilitate RSpec integration tests we have two test-specific stylesheets. These can be used to do things like disable animations to improve test speed, or to make elements visible when they need to be targeted by Capybara click events: + +- `app/assets/stylesheets/disable_animations.scss` +- `app/assets/stylesheets/test_environment.scss` + +Because the test environment should match the production environment as much as possible, use these minimally and only add to them when necessary. + ## What and how to test Before jumping into more gritty details about Jest-specific workflows like mocks and spies, we should briefly cover what to test with Jest. @@ -212,8 +221,8 @@ When it comes to querying DOM elements in your tests, it is best to uniquely and the element. Preferentially, this is done by targeting what the user actually sees using [DOM Testing Library](https://testing-library.com/docs/dom-testing-library/intro/). -When selecting by text it is best to use [`getByRole` or `findByRole`](https://testing-library.com/docs/queries/byrole/) -as these enforce accessibility best practices as well. The examples below demonstrate the order of preference. +When selecting by text it is best to use the [`byRole`](https://testing-library.com/docs/queries/byrole) query +as it helps enforce accessibility best practices. `findByRole` and the other [DOM Testing Library queries](https://testing-library.com/docs/queries/about) are available when using [`shallowMountExtended` or `mountExtended`](#shallowmountextended-and-mountextended). When writing Vue component unit tests, it can be wise to query children by component, so that the unit test can focus on comprehensive value coverage rather than dealing with the complexity of a child component's behavior. @@ -223,25 +232,27 @@ possible selectors include: - A semantic attribute like `name` (also verifies that `name` was setup properly) - A `data-testid` attribute ([recommended by maintainers of `@vue/test-utils`](https://github.com/vuejs/vue-test-utils/issues/1498#issuecomment-610133465)) - optionally combined with [`findByTestId`](#extendedwrapper-and-findbytestid) + optionally combined with [`shallowMountExtended` or `mountExtended`](#shallowmountextended-and-mountextended) - a Vue `ref` (if using `@vue/test-utils`) ```javascript -import { getByRole, getByText } from '@testing-library/dom' +import { shallowMountExtended } from 'helpers/vue_test_utils_helper' + +const wrapper = shallowMountExtended(ExampleComponent); // In this example, `wrapper` is a `@vue/test-utils` wrapper returned from `mount` or `shallowMount`. it('exists', () => { // Best (especially for integration tests) - getByRole(wrapper.element, 'link', { name: /Click Me/i }) - getByRole(wrapper.element, 'link', { name: 'Click Me' }) - getByText(wrapper.element, 'Click Me') - getByText(wrapper.element, /Click Me/i) + wrapper.findByRole('link', { name: /Click Me/i }) + wrapper.findByRole('link', { name: 'Click Me' }) + wrapper.findByText('Click Me') + wrapper.findByText(/Click Me/i) // Good (especially for unit tests) wrapper.find(FooComponent); wrapper.find('input[name=foo]'); wrapper.find('[data-testid="my-foo-id"]'); - wrapper.findByTestId('my-foo-id'); // with the extendedWrapper utility – check below + wrapper.findByTestId('my-foo-id'); // with shallowMountExtended or mountExtended – check below wrapper.find({ ref: 'foo'}); // Bad @@ -376,7 +387,7 @@ Sometimes we have to test time-sensitive code. For example, recurring events tha If the application itself is waiting for some time, mock await the waiting. In Jest this is already [done by default](https://gitlab.com/gitlab-org/gitlab/blob/a2128edfee799e49a8732bfa235e2c5e14949c68/jest.config.js#L47) -(see also [Jest Timer Mocks](https://jestjs.io/docs/en/timer-mocks)). In Karma you can use the +(see also [Jest Timer Mocks](https://jestjs.io/docs/timer-mocks)). In Karma you can use the [Jasmine mock clock](https://jasmine.github.io/api/2.9/Clock.html). ```javascript @@ -564,9 +575,9 @@ This is however not entirely true as the `destroy` method does not remove everyt #### Prefer `toBe` over `toEqual` when comparing primitive values -Jest has [`toBe`](https://jestjs.io/docs/en/expect#tobevalue) and -[`toEqual`](https://jestjs.io/docs/en/expect#toequalvalue) matchers. -As [`toBe`](https://jestjs.io/docs/en/expect#tobevalue) uses +Jest has [`toBe`](https://jestjs.io/docs/expect#tobevalue) and +[`toEqual`](https://jestjs.io/docs/expect#toequalvalue) matchers. +As [`toBe`](https://jestjs.io/docs/expect#tobevalue) uses [`Object.is`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object/is) to compare values, it's faster (by default) than using `toEqual`. While the latter eventually falls back to leverage [`Object.is`](https://github.com/facebook/jest/blob/master/packages/expect/src/jasmineUtils.ts#L91), @@ -588,7 +599,7 @@ expect(foo).toBe(1); Jest provides useful matchers like `toHaveLength` or `toBeUndefined` to make your tests more readable and to produce more understandable error messages. Check their docs for the -[full list of matchers](https://jestjs.io/docs/en/expect#methods). +[full list of matchers](https://jestjs.io/docs/expect#methods). Examples: @@ -719,7 +730,7 @@ TBU Jasmine provides stubbing and mocking capabilities. There are some subtle differences in how to use it within Karma and Jest. Stubs or spies are often used synonymously. In Jest it's quite easy thanks to the `.spyOn` method. -[Official docs](https://jestjs.io/docs/en/jest-object#jestspyonobject-methodname) +[Official docs](https://jestjs.io/docs/jest-object#jestspyonobject-methodname) The more challenging part are mocks, which can be used for functions or even dependencies. ### Manual module mocks @@ -728,12 +739,12 @@ Manual mocks are used to mock modules across the entire Jest environment. This i unit testing by mocking out modules which cannot be easily consumed in our test environment. > **WARNING:** Do not use manual mocks if a mock should not be consistently applied in every spec (i.e. it's only needed by a few specs). -> Instead, consider using [`jest.mock(..)`](https://jestjs.io/docs/en/jest-object#jestmockmodulename-factory-options) +> Instead, consider using [`jest.mock(..)`](https://jestjs.io/docs/jest-object#jestmockmodulename-factory-options) > (or a similar mocking function) in the relevant spec file. #### Where should I put manual mocks? -Jest supports [manual module mocks](https://jestjs.io/docs/en/manual-mocks) by placing a mock in a `__mocks__/` directory next to the source module +Jest supports [manual module mocks](https://jestjs.io/docs/manual-mocks) by placing a mock in a `__mocks__/` directory next to the source module (e.g. `app/assets/javascripts/ide/__mocks__`). **Don't do this.** We want to keep all of our test-related code in one place (the `spec/` folder). If a manual mock is needed for a `node_modules` package, use the `spec/frontend/__mocks__` folder. Here's an example of @@ -755,7 +766,7 @@ If a manual mock is needed for a CE module, place it in `spec/frontend/mocks/ce` any behavior, only provides a nice es6 compatible wrapper. - [`__mocks__/monaco-editor/index.js`](https://gitlab.com/gitlab-org/gitlab/blob/b7f914cddec9fc5971238cdf12766e79fa1629d7/spec/frontend/__mocks__/monaco-editor/index.js) - This mock is helpful because the Monaco package is completely incompatible in a Jest environment. In fact, webpack requires a special loader to make it work. This mock - simply makes this package consumable by Jest. + makes this package consumable by Jest. ### Keep mocks light @@ -766,7 +777,7 @@ Global mocks introduce magic and technically can reduce test coverage. When mock ### Additional mocking techniques -Consult the [official Jest docs](https://jestjs.io/docs/en/jest-object#mock-modules) for a full overview of the available mocking features. +Consult the [official Jest docs](https://jestjs.io/docs/jest-object#mock-modules) for a full overview of the available mocking features. ## Running Frontend Tests @@ -939,8 +950,8 @@ it('uses some HTML element', () => { Similar to [RSpec's parameterized tests](best_practices.md#table-based--parameterized-tests), Jest supports data-driven tests for: -- Individual tests using [`test.each`](https://jestjs.io/docs/en/api#testeachtable-name-fn-timeout) (aliased to `it.each`). -- Groups of tests using [`describe.each`](https://jestjs.io/docs/en/api#describeeachtable-name-fn-timeout). +- Individual tests using [`test.each`](https://jestjs.io/docs/api#testeachtable-name-fn-timeout) (aliased to `it.each`). +- Groups of tests using [`describe.each`](https://jestjs.io/docs/api#describeeachtable-name-fn-timeout). These can be useful for reducing repetition within tests. Each option can take an array of data values or a tagged template literal. @@ -1138,23 +1149,40 @@ These are very useful if you don't have a handle to the request's Promise, for e Both functions run `callback` on the next tick after the requests finish (using `setImmediate()`), to allow any `.then()` or `.catch()` handlers to run. -### `extendedWrapper` and `findByTestId` +### `shallowMountExtended` and `mountExtended` -Using `data-testid` is one of the [recommended ways to query DOM elements](#how-to-query-dom-elements). -You can use the `extendedWrapper` utility on the `wrapper` returned by `shalowMount`/`mount`. -By doing so, the `wrapper` provides you with the ability to perform a `findByTestId`, -which is a shortcut to the more verbose `wrapper.find('[data-testid="my-test-id"]');` +The `shallowMountExtended` and `mountExtended` utilities provide you with the ability to perform +any of the available [DOM Testing Library queries](https://testing-library.com/docs/queries/about) +by prefixing them with `find` or `findAll`. ```javascript -import { extendedWrapper } from 'helpers/vue_test_utils_helper'; +import { shallowMountExtended } from 'helpers/vue_test_utils_helper'; describe('FooComponent', () => { - const wrapper = extendedWrapper(shallowMount({ - template: `<div data-testid="my-test-id"></div>`, - })); + const wrapper = shallowMountExtended({ + template: ` + <div data-testid="gitlab-frontend-stack"> + <p>GitLab frontend stack</p> + <div role="tablist"> + <button role="tab" aria-selected="true">Vue.js</button> + <button role="tab" aria-selected="false">GraphQL</button> + <button role="tab" aria-selected="false">SCSS</button> + </div> + </div> + `, + }); + + it('finds elements with `findByTestId`', () => { + expect(wrapper.findByTestId('gitlab-frontend-stack').exists()).toBe(true); + }); + + it('finds elements with `findByText`', () => { + expect(wrapper.findByText('GitLab frontend stack').exists()).toBe(true); + expect(wrapper.findByText('TypeScript').exists()).toBe(false); + }); - it('exists', () => { - expect(wrapper.findByTestId('my-test-id').exists()).toBe(true); + it('finds elements with `findAllByRole`', () => { + expect(wrapper.findAllByRole('tab').length).toBe(3); }); }); ``` @@ -1189,7 +1217,7 @@ You can download any older version of Firefox from the releases FTP server, <htt ## Snapshots -By now you've probably heard of [Jest snapshot tests](https://jestjs.io/docs/en/snapshot-testing) and why they are useful for various reasons. +By now you've probably heard of [Jest snapshot tests](https://jestjs.io/docs/snapshot-testing) and why they are useful for various reasons. To use them within GitLab, there are a few guidelines that should be highlighted: - Treat snapshots as code @@ -1199,7 +1227,7 @@ To use them within GitLab, there are a few guidelines that should be highlighted Think of a snapshot test as a simple way to store a raw `String` representation of what you've put into the item being tested. This can be used to evaluate changes in a component, a store, a complex piece of generated output, etc. You can see more in the list below for some recommended `Do's and Don'ts`. While snapshot tests can be a very powerful tool. They are meant to supplement, not to replace unit tests. -Jest provides a great set of docs on [best practices](https://jestjs.io/docs/en/snapshot-testing#best-practices) that we should keep in mind when creating snapshots. +Jest provides a great set of docs on [best practices](https://jestjs.io/docs/snapshot-testing#best-practices) that we should keep in mind when creating snapshots. ### How does a snapshot work? @@ -1207,7 +1235,7 @@ A snapshot is purely a stringified version of what you ask to be tested on the l Should the outcome of your spec be different from what is in the generated snapshot file, you'll be notified about it by a failing test in your test suite. -Find all the details in Jests official documentation [https://jestjs.io/docs/en/snapshot-testing](https://jestjs.io/docs/en/snapshot-testing) +Find all the details in Jests official documentation [https://jestjs.io/docs/snapshot-testing](https://jestjs.io/docs/snapshot-testing) ### How to take a snapshot diff --git a/doc/development/transient/prevention-patterns.md b/doc/development/transient/prevention-patterns.md index 2bfd188fd43..472b5756805 100644 --- a/doc/development/transient/prevention-patterns.md +++ b/doc/development/transient/prevention-patterns.md @@ -55,12 +55,12 @@ Including when that expanded content is: ### Using assertions to detect transient bugs caused by unmet conditions Transient bugs happen in the context of code that executes under the assumption -that the application’s state meets one or more conditions. We may write a feature +that the application's state meets one or more conditions. We may write a feature that assumes a server-side API response always include a group of attributes or that an operation only executes when the application has successfully transitioned to a new state. -Transient bugs are difficult to debug because there isn’t any mechanism that alerts +Transient bugs are difficult to debug because there isn't any mechanism that alerts the user or the developer about unsatisfied conditions. These conditions are usually not expressed explicitly in the code. A useful debugging technique for such situations is placing assertions to make any assumption explicit. They can help detect the cause diff --git a/doc/development/understanding_explain_plans.md b/doc/development/understanding_explain_plans.md index 3f596e89e29..e0176c190d6 100644 --- a/doc/development/understanding_explain_plans.md +++ b/doc/development/understanding_explain_plans.md @@ -280,7 +280,7 @@ FROM users WHERE twitter != ''; ``` -This query simply counts the number of users that have a Twitter profile set. +This query counts the number of users that have a Twitter profile set. Let's run this using `EXPLAIN (ANALYZE, BUFFERS)`: ```sql @@ -388,7 +388,7 @@ we created the index: CREATE INDEX CONCURRENTLY twitter_test ON users (twitter); ``` -We simply told PostgreSQL to index all possible values of the `twitter` column, +We told PostgreSQL to index all possible values of the `twitter` column, even empty strings. Our query in turn uses `WHERE twitter != ''`. This means that the index does improve things, as we don't need to do a sequential scan, but we may still encounter empty strings. This means PostgreSQL _has_ to apply a diff --git a/doc/development/usage_ping/dictionary.md b/doc/development/usage_ping/dictionary.md index c0e9aa82247..cb53d088907 100644 --- a/doc/development/usage_ping/dictionary.md +++ b/doc/development/usage_ping/dictionary.md @@ -430,15 +430,15 @@ Tiers: `free` ### `counts.assignee_lists` -Missing description +Count of assignee lists created on Boards [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216181100_assignee_lists.yml) -Group: `group::plan` +Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `counts.auto_devops_disabled` @@ -466,7 +466,7 @@ Tiers: `free` ### `counts.boards` -Missing description +Count of total Boards created [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181252_boards.yml) @@ -474,7 +474,7 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.ci_builds` @@ -790,15 +790,15 @@ Tiers: `free` ### `counts.confidential_epics` -Missing description +Count of confidential epics -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181205_confidential_epics.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216181205_confidential_epics.yml) -Group: `group::portfolio management` +Group: `group::product planning` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `counts.container_scanning_jobs` @@ -962,7 +962,7 @@ Count of issues that are assigned to an epic [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216181208_epic_issues.yml) -Group: `group::portfolio management` +Group: `group::product planning` Status: `data_available` @@ -970,27 +970,27 @@ Tiers: `premium`, `ultimate` ### `counts.epics` -Missing description +Count of all epics -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181206_epics.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216181206_epics.yml) -Group: `group::portfolio management` +Group: `group::product planning` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `counts.epics_deepest_relationship_level` -Missing description +Count of the deepest relationship level for epics [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216181212_epics_deepest_relationship_level.yml) -Group: `group::portfolio management` +Group: `group::product planning` Status: `data_available` -Tiers: +Tiers: `ultimate` ### `counts.failed_deployments` @@ -1028,18 +1028,6 @@ Status: `data_available` Tiers: `premium`, `ultimate` -### `counts.geo_node_usage.git_fetch_event_count_weekly` - -Number of Git fetch events from Prometheus on the Geo secondary - -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210309194425_git_fetch_event_count_weekly.yml) - -Group: `group::geo` - -Status: `implemented` - -Tiers: `premium`, `ultimate` - ### `counts.geo_nodes` Total number of sites in a Geo deployment @@ -2056,7 +2044,7 @@ Whether or not ModSecurity is set to blocking mode Group: `group::container security` -Status: `data_available` +Status: `deprecated` Tiers: `free`, `premium`, `ultimate` @@ -2068,7 +2056,7 @@ Whether or not ModSecurity is disabled within Ingress Group: `group::container security` -Status: `data_available` +Status: `deprecated` Tiers: `free`, `premium`, `ultimate` @@ -2080,7 +2068,7 @@ Whether or not ModSecurity is set to logging mode Group: `group::container security` -Status: `data_available` +Status: `deprecated` Tiers: `free`, `premium`, `ultimate` @@ -2092,7 +2080,7 @@ Whether or not ModSecurity has not been installed into the cluster Group: `group::container security` -Status: `data_available` +Status: `deprecated` Tiers: `free`, `premium`, `ultimate` @@ -2104,7 +2092,7 @@ Cumulative count of packets identified as anomalous by ModSecurity since Usage P Group: `group::container security` -Status: `data_available` +Status: `deprecated` Tiers: `free`, `premium`, `ultimate` @@ -2116,7 +2104,7 @@ Cumulative count of packets processed by ModSecurity since Usage Ping was last r Group: `group::container security` -Status: `data_available` +Status: `deprecated` Tiers: `free`, `premium`, `ultimate` @@ -2128,7 +2116,7 @@ Whether or not ModSecurity statistics are unavailable Group: `group::container security` -Status: `data_available` +Status: `deprecated` Tiers: `ultimate` @@ -2710,15 +2698,15 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.issues_with_health_status` -Missing description +Count of issues with health status -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181210_issues_with_health_status.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216181210_issues_with_health_status.yml) -Group: `group::portfolio management` +Group: `group::product planning` Status: `data_available` -Tiers: `free` +Tiers: `ultimate` ### `counts.jira_imports_projects_count` @@ -2726,7 +2714,7 @@ Count of Projects that imported Issues from Jira [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181259_jira_imports_projects_count.yml) -Group: `group::project management` +Group: `group::ecosystem` Status: `data_available` @@ -2734,11 +2722,11 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.jira_imports_total_imported_count` -Count of Issues imported from Jira +Count of Jira imports completed [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181258_jira_imports_total_imported_count.yml) -Group: `group::project management` +Group: `group::ecosystem` Status: `data_available` @@ -2746,11 +2734,11 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.jira_imports_total_imported_issues_count` -Count of Jira imports run +Count of total issues imported via the Jira Importer [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181301_jira_imports_total_imported_issues_count.yml) -Group: `group::project management` +Group: `group::ecosystem` Status: `data_available` @@ -2806,15 +2794,15 @@ Tiers: `premium`, `ultimate` ### `counts.label_lists` -Missing description +Count of label lists created on Boards [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181104_label_lists.yml) -Group: `group::plan` +Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.labels` @@ -2822,7 +2810,7 @@ Count of Labels [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181111_labels.yml) -Group: `group::plan` +Group: `group::project management` Status: `data_available` @@ -2950,27 +2938,27 @@ Tiers: `free` ### `counts.milestone_lists` -Missing description +Count of milestone lists created on Boards -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181106_milestone_lists.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216181106_milestone_lists.yml) -Group: `group::plan` +Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `counts.milestones` -Missing description +Count of milestones created [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181108_milestones.yml) -Group: `group::plan` +Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.navbar_searches` @@ -3014,7 +3002,7 @@ Count of Notes across all objects that use them [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181113_notes.yml) -Group: `group::plan` +Group: `group::project management` Status: `data_available` @@ -3730,7 +3718,7 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.projects` -Count of Projects +Count of Projects created [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181254_projects.yml) @@ -3738,7 +3726,7 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.projects_asana_active` @@ -5914,7 +5902,7 @@ Tiers: `free` ### `counts.todos` -Count of ToDos +Count of todos created [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181256_todos.yml) @@ -5930,7 +5918,7 @@ Count of Uploads via Notes and Descriptions [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181109_uploads.yml) -Group: `group::plan` +Group: `group::project management` Status: `data_available` @@ -5940,7 +5928,7 @@ Tiers: `free`, `premium`, `ultimate` Count of users who set personal preference to see Details on Group overview page -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182203_user_preferences_group_overview_details.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216182203_user_preferences_group_overview_details.yml) Group: `group::threat insights` @@ -5952,7 +5940,7 @@ Tiers: `ultimate` Count of users who set personal preference to see Security Dashboard on Group overview page -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182205_user_preferences_group_overview_security_dashboard.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216182205_user_preferences_group_overview_security_dashboard.yml) Group: `group::threat insights` @@ -5998,7 +5986,7 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.web_ide_merge_requests` -Count of Merge Requests created from Web IDE +Count of merge requests created from Web IDE [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216180246_web_ide_merge_requests.yml) @@ -6118,15 +6106,15 @@ Tiers: `free` ### `counts_monthly.aggregated_metrics.i_testing_paid_monthly_active_user_total` -Missing description +Aggregated count of users who have engaged with a Premium or Ultimate tier testing feature per month. -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216183209_i_testing_paid_monthly_active_user_total.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216183209_i_testing_paid_monthly_active_user_total.yml) -Group: `` +Group: `group::testing` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `counts_monthly.aggregated_metrics.incident_management_alerts_total_unique_counts` @@ -6154,27 +6142,27 @@ Tiers: `free` ### `counts_monthly.aggregated_metrics.product_analytics_test_metrics_intersection` -Missing description +This was test metric used for purpose of assuring correct implementation of aggregated metrics feature [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216183205_product_analytics_test_metrics_intersection.yml) -Group: `` +Group: `group::product intelligence` -Status: `data_available` +Status: `removed` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts_monthly.aggregated_metrics.product_analytics_test_metrics_union` -Missing description +This was test metric used for purpose of assuring correct implementation of aggregated metrics feature [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216183203_product_analytics_test_metrics_union.yml) -Group: `` +Group: `group::product intelligence` -Status: `data_available` +Status: `removed` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts_monthly.deployments` @@ -6286,15 +6274,15 @@ Tiers: ### `counts_weekly.aggregated_metrics.i_testing_paid_monthly_active_user_total` -Missing description +Aggregated count of users who have engaged with a Premium or Ultimate tier testing feature per week. [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216183219_i_testing_paid_monthly_active_user_total.yml) -Group: `` +Group: `group::testing` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `counts_weekly.aggregated_metrics.incident_management_alerts_total_unique_counts` @@ -6322,27 +6310,27 @@ Tiers: ### `counts_weekly.aggregated_metrics.product_analytics_test_metrics_intersection` -Missing description +This was test metric used for purpose of assuring correct implementation of aggregated metrics feature -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216183215_product_analytics_test_metrics_intersection.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216183215_product_analytics_test_metrics_intersection.yml) -Group: `` +Group: `group::product intelligence` -Status: `data_available` +Status: `removed` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `counts_weekly.aggregated_metrics.product_analytics_test_metrics_union` -Missing description +This was test metric used for purpose of assuring correct implementation of aggregated metrics feature -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216183213_product_analytics_test_metrics_union.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216183213_product_analytics_test_metrics_union.yml) -Group: `` +Group: `group::product intelligence` -Status: `data_available` +Status: `removed` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `database.adapter` @@ -6416,30 +6404,6 @@ Status: `data_available` Tiers: `premium`, `ultimate` -### `g_project_management_epic_created_monthly` - -Count of MAU creating epics - -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210305144719_g_product_planning_epic_created_monthly.yml) - -Group: `group::product planning` - -Status: `implemented` - -Tiers: `premium`, `ultimate` - -### `g_project_management_epic_created_weekly` - -Count of WAU creating epics - -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210305145820_g_product_planning_epic_created_weekly.yml) - -Group: `group::product planning` - -Status: `implemented` - -Tiers: `premium`, `ultimate` - ### `geo_enabled` Is Geo enabled? @@ -6616,7 +6580,7 @@ Whether or not ModSecurity is enabled within Ingress Group: `group::container security` -Status: `data_available` +Status: `deprecated` Tiers: `free`, `premium`, `ultimate` @@ -6804,9 +6768,9 @@ Tiers: `premium`, `ultimate` The value of the SMTP server that is used -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216174829_smtp_server.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/settings/20210216174829_smtp_server.yml) -Group: `group::acquisition` +Group: `group::activation` Status: `data_available` @@ -7182,7 +7146,7 @@ Group: `group::product intelligence` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `recording_ee_finished_at` @@ -7388,6 +7352,30 @@ Status: `data_available` Tiers: +### `redis_hll_counters.analytics.i_analytics_dev_ops_adoption_monthly` + +Counts visits to DevOps Adoption page per month + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210401092244_i_analytics_dev_ops_adoption_monthly.yml) + +Group: `group::optimize` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.analytics.i_analytics_dev_ops_adoption_weekly` + +Counts visits to DevOps Adoption page per week + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210401092244_i_analytics_dev_ops_adoption_weekly.yml) + +Group: `group::optimize` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + ### `redis_hll_counters.analytics.i_analytics_dev_ops_score_monthly` Missing description @@ -8308,7 +8296,7 @@ Count of unique users per month who changed assignees of a MR Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8320,7 +8308,7 @@ Count of unique users per week who changed assignees of a MR Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8524,7 +8512,7 @@ Count of unique users per month who changed labels of a MR Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8536,7 +8524,7 @@ Count of unique users per week who changed labels of a MR Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8596,7 +8584,7 @@ Count of unique users per month who changed milestone of a MR Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8608,7 +8596,7 @@ Count of unique users per week who changed milestone of a MR Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8620,7 +8608,7 @@ Count of unique users per month who locked a MR Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8632,7 +8620,7 @@ Count of unique users per week who locked a MR Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8644,7 +8632,7 @@ Count of unique users per month who unlocked a MR Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8656,7 +8644,7 @@ Count of unique users per week who unlocked a MR Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8812,7 +8800,7 @@ Count of unique users per month who changed reviewers of a MR Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8824,7 +8812,7 @@ Count of unique users per week who changed reviewers of a MR Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8860,7 +8848,7 @@ Count of unique users per month who changed time estimate of a MR Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8872,7 +8860,7 @@ Count of unique users per week who changed time estimate of a MR Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8884,7 +8872,7 @@ Count of unique users per month who changed time spent on a MR Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8896,7 +8884,7 @@ Count of unique users per week who changed time spent on a MR Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -9604,7 +9592,7 @@ Calculated unique users to trigger a Slack message by performing an action on a Group: `group::ecosystem` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -9616,7 +9604,7 @@ Calculated unique users to trigger a Slack message by performing an action on a Group: `group::ecosystem` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -9628,7 +9616,7 @@ Calculated unique users to trigger a Slack message by creating a confidential no Group: `group::ecosystem` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -9640,7 +9628,7 @@ Calculated unique users to trigger a Slack message by creating a confidential no Group: `group::ecosystem` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -9652,7 +9640,7 @@ Calculated unique users to trigger a Slack message by performing a deployment by Group: `group::ecosystem` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -9664,7 +9652,7 @@ Calculated unique users to trigger a Slack message by performing a deployment by Group: `group::ecosystem` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -9676,7 +9664,7 @@ Calculated unique users to trigger a Slack message by performing an action on an Group: `group::ecosystem` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -9688,7 +9676,7 @@ Calculated unique users to trigger a Slack message by performing an action on an Group: `group::ecosystem` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -9700,7 +9688,7 @@ Calculated unique users to trigger a Slack message by performing an action on a Group: `group::ecosystem` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -9712,7 +9700,7 @@ Calculated unique users to trigger a Slack message by performing an action on a Group: `group::ecosystem` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -9724,7 +9712,7 @@ Calculated unique users to trigger a Slack message by creating a note by month Group: `group::ecosystem` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -9736,7 +9724,7 @@ Calculated unique users to trigger a Slack message by creating a note by week Group: `group::ecosystem` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -9748,7 +9736,7 @@ Calculated unique users to trigger a Slack message by performing a Git push by m Group: `group::ecosystem` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -9760,7 +9748,7 @@ Calculated unique users to trigger a Slack message by performing a Git push by w Group: `group::ecosystem` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -9772,7 +9760,7 @@ Calculated unique users to trigger a Slack message by performing a tag push by m Group: `group::ecosystem` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -9784,7 +9772,7 @@ Calculated unique users to trigger a Slack message by performing a tag push by w Group: `group::ecosystem` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -9796,7 +9784,7 @@ Calculated unique users to trigger a Slack message by performing an action on a Group: `group::ecosystem` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -9808,10 +9796,562 @@ Calculated unique users to trigger a Slack message by performing an action on a Group: `group::ecosystem` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` +### `redis_hll_counters.epics_usage.epics_usage_total_unique_counts_monthly` + +Total monthly users count for epics_usage + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210318183733_epics_usage_total_unique_counts_monthly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.epics_usage_total_unique_counts_weekly` + +Total weekly users count for epics_usage + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210318183220_epics_usage_total_unique_counts_weekly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_epic_closed_monthly` + +Counts of MAU closing epics + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210310163213_g_project_management_epic_closed_monthly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_epic_closed_weekly` + +Counts of WAU closing epics + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210310162703_g_project_management_epic_closed_weekly.yml) + +Group: `group::product planning` + +Status: `data_available` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_epic_created_monthly` + +Count of MAU creating epics + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210305144719_g_product_planning_epic_created_monthly.yml) + +Group: `group::product planning` + +Status: `data_available` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_epic_created_weekly` + +Count of WAU creating epics + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210305145820_g_product_planning_epic_created_weekly.yml) + +Group: `group::product planning` + +Status: `data_available` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_epic_destroyed_monthly` + +Count of MAU destroying epics + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210413174710_g_project_management_epic_destroyed_monthly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_epic_destroyed_weekly` + +Count of WAU destroying epics + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210413174449_g_project_management_epic_destroyed_weekly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_epic_issue_added_monthly` + +Count of MAU adding issues to epics + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210312144719_g_product_planning_epic_issue_added_monthly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_epic_issue_added_weekly` + +Count of WAU adding issues to epics + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210312181918_g_product_planning_epic_issue_added_weekly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_epic_issue_moved_from_project_monthly` + +Counts of MAU moving epic issues between projects + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210405190240_g_project_management_epic_issue_moved_from_project_monthly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_epic_issue_moved_from_project_weekly` + +Counts of WAU moving epic issues between projects + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210405185814_g_project_management_epic_issue_moved_from_project_weekly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_epic_issue_removed_monthly` + +Count of MAU removing issues from epics + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210401183230_g_project_management_epic_issue_removed_monthly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_epic_issue_removed_weekly` + +Counts of WAU removing issues from epics + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210401182457_g_project_management_g_project_management_epic_issue_removed_weekly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_epic_reopened_monthly` + +Counts of MAU closing epics + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210310164247_g_project_management_epic_reopened_monthly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_epic_reopened_weekly` + +Counts of WAU re-opening epics + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210310164112_g_project_management_epic_reopened_weekly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_epic_users_changing_labels_monthly` + +Count of MAU chaging the epic lables + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210312195730_g_project_management_epic_labels_monthly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_epic_users_changing_labels_weekly` + +Count of WAU chaging the epic lables + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210312195849_g_project_management_epic_labels_weekly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_issue_promoted_to_epic_monthly` + +Count of MAU promoting issues to epics + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210331193236_g_project_management_issue_promoted_to_epic_monthly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_issue_promoted_to_epic_weekly` + +Counts of WAU promoting issues to epics + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210331192332_g_project_management_issue_promoted_to_epic_weekly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_creating_epic_notes_monthly` + +Counts of MAU adding epic notes + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210314215451_g_project_management_users_creating_epic_notes_monthly.yml) + +Group: `group::product planning` + +Status: `data_available` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_creating_epic_notes_weekly` + +Counts of WAU adding epic notes + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210314231518_g_project_management_users_creating_epic_notes_weekly.yml) + +Group: `group::product planning` + +Status: `data_available` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_destroying_epic_notes_monthly` + +Counts of MAU destroying epic notes + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210315034808_g_project_management_users_destroying_epic_notes_monthly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_destroying_epic_notes_weekly` + +Counts of WAU destroying epic notes + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210315034846_g_project_management_users_destroying_epic_notes_weekly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_setting_epic_confidential_monthly` + +Count of MAU making epics confidential + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210310203049_g_project_management_epic_confidential_monthly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_setting_epic_confidential_weekly` + +Count of WAU making epics confidential + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210310203225_g_project_management_epic_confidential_weekly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_setting_epic_due_date_as_fixed_monthly` + +Counts of MAU setting epic due date as inherited + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210325060507_g_project_management_users_setting_epic_due_date_as_fixed_monthly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_setting_epic_due_date_as_fixed_weekly` + +Counts of WAU setting epic due date as fixed + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210325060623_g_project_management_users_setting_epic_due_date_as_fixed_weekly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_setting_epic_due_date_as_inherited_monthly` + +Counts of MAU setting epic due date as inherited + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210325060315_g_project_management_users_setting_epic_due_date_as_inherited_monthly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_setting_epic_due_date_as_inherited_weekly` + +Counts of WAU setting epic due date as inherited + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210325060903_g_project_management_users_setting_epic_due_date_as_inherited_weekly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_setting_epic_start_date_as_fixed_monthly` + +Counts of MAU setting epic start date as fixed + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210315055624_g_project_management_users_setting_epic_start_date_as_fixed_monthly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_setting_epic_start_date_as_fixed_weekly` + +Counts of WAU setting epic start date as fixed + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210315054905_g_project_management_users_setting_epic_start_date_as_fixed_weekly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_setting_epic_start_date_as_inherited_monthly` + +Counts of MAU setting epic start date as inherited + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210315055439_g_project_management_users_setting_epic_start_date_as_inherited_monthly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_setting_epic_start_date_as_inherited_weekly` + +Counts of WAU setting epic start date as inherited + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210315055342_g_project_management_users_setting_epic_start_date_as_inherited_weekly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_setting_epic_visible_monthly` + +Count of MAU making epics visible + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210312093611_g_project_management_epic_visible_monthly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_setting_epic_visible_weekly` + +Count of WAU making epics visible + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210312093243_g_poject_management_epic_visible_weekly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_updating_epic_descriptions_monthly` + +Counts of MAU changing epic descriptions + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210312102051_g_project_management_users_updating_epic_descriptions_monthly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_updating_epic_descriptions_weekly` + +Counts of WAU changing epic descriptions + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210312101753_g_project_management_users_updating_epic_descriptions_weekly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_updating_epic_notes_monthly` + +Counts of MAU updating epic notes + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210314234202_g_project_management_users_updating_epic_notes_monthly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_updating_epic_notes_weekly` + +Counts of WAU updating epic notes + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210314234041_g_project_management_users_updating_epic_notes_weekly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_updating_epic_titles_monthly` + +Counts of MAU changing epic titles + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210312101935_g_project_management_users_updating_epic_titles_monthly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_updating_epic_titles_weekly` + +Counts of WAU changing epic titles + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210312101826_g_project_management_users_updating_epic_titles_weekly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_updating_fixed_epic_due_date_monthly` + +Counts of MAU manually updating fixed due date + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210329043548_g_project_management_users_updating_fixed_epic_due_date_monthly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_updating_fixed_epic_due_date_weekly` + +Counts of WAU manually updating fixed due date + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210329042536_g_project_management_users_updating_fixed_epic_due_date_weekly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_updating_fixed_epic_start_date_monthly` + +Counts of MAU manually updating fixed start date + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210329043509_g_project_management_users_updating_fixed_epic_start_date_monthly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.g_project_management_users_updating_fixed_epic_start_date_weekly` + +Counts of WAU manually updating fixed start date + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210329043402_g_project_management_users_updating_fixed_epic_start_date_weekly.yml) + +Group: `group::product planning` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + ### `redis_hll_counters.ide_edit.g_edit_by_sfe_monthly` Missing description @@ -9932,6 +10472,30 @@ Status: `data_available` Tiers: +### `redis_hll_counters.incident_management.i_incident_management_oncall_notification_sent_monthly` + +Count of unique users to receive a notification while on-call + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210405222005_i_incident_management_oncall_notification_sent_monthly.yml) + +Group: `group::monitor` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.incident_management.i_incident_management_oncall_notification_sent_weekly` + +Count of unique users to receive a notification while on-call + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210405220139_i_incident_management_oncall_notification_sent_weekly.yml) + +Group: `group::monitor` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + ### `redis_hll_counters.incident_management.incident_management_alert_assigned_monthly` Missing description @@ -10320,13 +10884,13 @@ Tiers: Count of MAU adding an issue to an epic -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181414_g_project_management_issue_added_to_epic_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181414_g_project_management_issue_added_to_epic_monthly.yml) Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_added_to_epic_weekly` @@ -10338,43 +10902,43 @@ Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_assignee_changed_monthly` Count of MAU changing issue assignees -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181311_g_project_management_issue_assignee_changed_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181311_g_project_management_issue_assignee_changed_monthly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_assignee_changed_weekly` Count of WAU changing issue assignees -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181310_g_project_management_issue_assignee_changed_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181310_g_project_management_issue_assignee_changed_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_changed_epic_monthly` Count of MAU changing the epic on an issue -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181420_g_project_management_issue_changed_epic_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181420_g_project_management_issue_changed_epic_monthly.yml) Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_changed_epic_weekly` @@ -10386,11 +10950,11 @@ Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_cloned_monthly` -Missing description +Count of MAU cloning an issue [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181501_g_project_management_issue_cloned_monthly.yml) @@ -10398,19 +10962,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_cloned_weekly` -Missing description +Count of WAU cloning an issue -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181459_g_project_management_issue_cloned_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181459_g_project_management_issue_cloned_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_closed_monthly` @@ -10422,19 +10986,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_closed_weekly` Count of WAU closing an issue -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181324_g_project_management_issue_closed_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181324_g_project_management_issue_closed_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_comment_added_monthly` @@ -10446,19 +11010,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_comment_added_weekly` Count of WAU commenting on an issue -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181444_g_project_management_issue_comment_added_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181444_g_project_management_issue_comment_added_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_comment_edited_monthly` @@ -10470,19 +11034,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_comment_edited_weekly` Count of WAU editing a comment on an issue -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181448_g_project_management_issue_comment_edited_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181448_g_project_management_issue_comment_edited_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_comment_removed_monthly` @@ -10494,19 +11058,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_comment_removed_weekly` Count of WAU deleting a comment from an issue -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181451_g_project_management_issue_comment_removed_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181451_g_project_management_issue_comment_removed_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_created_monthly` @@ -10518,19 +11082,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_created_weekly` Count of WAU creating issues -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181321_g_project_management_issue_created_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181321_g_project_management_issue_created_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_cross_referenced_monthly` @@ -10542,19 +11106,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_cross_referenced_weekly` -Count of WAU referncing an issue from somewhere else +Count of WAU referencing an issue from somewhere else -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181347_g_project_management_issue_cross_referenced_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181347_g_project_management_issue_cross_referenced_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_description_changed_monthly` @@ -10566,19 +11130,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_description_changed_weekly` Count of WAU editing an issue description -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181306_g_project_management_issue_description_changed_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181306_g_project_management_issue_description_changed_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_designs_added_monthly` @@ -10590,19 +11154,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_designs_added_weekly` Count of WAU adding a design to an issue -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181422_g_project_management_issue_designs_added_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181422_g_project_management_issue_designs_added_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_designs_modified_monthly` @@ -10614,19 +11178,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_designs_modified_weekly` Count of WAU modifying a design on an issue -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181425_g_project_management_issue_designs_modified_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181425_g_project_management_issue_designs_modified_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_designs_removed_monthly` @@ -10638,19 +11202,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_designs_removed_weekly` Count of WAU removing a design from an issue -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181429_g_project_management_issue_designs_removed_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181429_g_project_management_issue_designs_removed_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_due_date_changed_monthly` @@ -10662,31 +11226,31 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_due_date_changed_weekly` Count of WAU changing an issue due date -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181433_g_project_management_issue_due_date_changed_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181433_g_project_management_issue_due_date_changed_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_health_status_changed_monthly` Count of MAU changing the health status on an issue -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181457_g_project_management_issue_health_status_changed_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181457_g_project_management_issue_health_status_changed_monthly.yml) -Group: `group::project management` +Group: `group::product planning` Status: `data_available` -Tiers: `free` +Tiers: `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_health_status_changed_weekly` @@ -10694,23 +11258,23 @@ Count of WAU changing the health status on an issue [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181455_g_project_management_issue_health_status_changed_weekly.yml) -Group: `group::project management` +Group: `group::product planning` Status: `data_available` -Tiers: +Tiers: `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_iteration_changed_monthly` Count of MAU changing an issue's iteration -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181341_g_project_management_issue_iteration_changed_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181341_g_project_management_issue_iteration_changed_monthly.yml) Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_iteration_changed_weekly` @@ -10722,7 +11286,7 @@ Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_label_changed_monthly` @@ -10734,19 +11298,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_label_changed_weekly` Count of WAU changing an issue's label -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181332_g_project_management_issue_label_changed_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181332_g_project_management_issue_label_changed_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_locked_monthly` @@ -10758,19 +11322,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_locked_weekly` Count of WAU locking an issue -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181405_g_project_management_issue_locked_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181405_g_project_management_issue_locked_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_made_confidential_monthly` @@ -10782,19 +11346,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_made_confidential_weekly` Count of WAU making an issue confidential -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181313_g_project_management_issue_made_confidential_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181313_g_project_management_issue_made_confidential_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_made_visible_monthly` @@ -10806,19 +11370,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_made_visible_weekly` Count of WAU making an issue not confidential -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181317_g_project_management_issue_made_visible_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181317_g_project_management_issue_made_visible_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_marked_as_duplicate_monthly` @@ -10830,19 +11394,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_marked_as_duplicate_weekly` Count of WAU marking an issue as a duplicate -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181401_g_project_management_issue_marked_as_duplicate_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181401_g_project_management_issue_marked_as_duplicate_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_milestone_changed_monthly` @@ -10854,19 +11418,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_milestone_changed_weekly` Count of WAU changing an issue's milestone -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181336_g_project_management_issue_milestone_changed_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181336_g_project_management_issue_milestone_changed_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_moved_monthly` @@ -10878,19 +11442,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_moved_weekly` Count of WAU moving an issue to another project -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181350_g_project_management_issue_moved_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181350_g_project_management_issue_moved_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_related_monthly` @@ -10902,31 +11466,31 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_related_weekly` Count of WAU relating an issue to another issue -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181354_g_project_management_issue_related_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181354_g_project_management_issue_related_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_removed_from_epic_monthly` Count of MAU removing an issue from an epic -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181416_g_project_management_issue_removed_from_epic_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181416_g_project_management_issue_removed_from_epic_monthly.yml) Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_removed_from_epic_weekly` @@ -10950,19 +11514,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_reopened_weekly` Count of WAU re-opening a closed issue -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181328_g_project_management_issue_reopened_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181328_g_project_management_issue_reopened_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_time_estimate_changed_monthly` @@ -10974,19 +11538,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_time_estimate_changed_weekly` Count of WAU changing an issue time estimate -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181437_g_project_management_issue_time_estimate_changed_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181437_g_project_management_issue_time_estimate_changed_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_time_spent_changed_monthly` @@ -10998,19 +11562,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_time_spent_changed_weekly` Count of WAU recording time spent on an issue -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181440_g_project_management_issue_time_spent_changed_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181440_g_project_management_issue_time_spent_changed_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_title_changed_monthly` @@ -11022,11 +11586,11 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_title_changed_weekly` -Distinct users count that changed issue title in a group for last recent week +Count of WAU editing an issue title [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210201124931_g_project_management_issue_title_changed_weekly.yml) @@ -11038,7 +11602,7 @@ Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_unlocked_monthly` -Count of MAU marking an issue as blocked or blocked by +Count of MAU unlocking an issue [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181411_g_project_management_issue_unlocked_monthly.yml) @@ -11046,19 +11610,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_unlocked_weekly` -Count of WAU marking an issue as blocked or blocked by +Count of WAU unlocking an issue -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181409_g_project_management_issue_unlocked_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181409_g_project_management_issue_unlocked_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_unrelated_monthly` @@ -11070,31 +11634,31 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_unrelated_weekly` Count of WAU unrelating an issue to another issue -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181358_g_project_management_issue_unrelated_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181358_g_project_management_issue_unrelated_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_weight_changed_monthly` Count of MAU changing an issue's weight -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181345_g_project_management_issue_weight_changed_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181345_g_project_management_issue_weight_changed_monthly.yml) Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `redis_hll_counters.issues_edit.g_project_management_issue_weight_changed_weekly` @@ -11106,11 +11670,11 @@ Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `redis_hll_counters.issues_edit.issues_edit_total_unique_counts_monthly` -Count of MAU taking an action related to an issue +Aggregate count of MAU taking an action related to an issue [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181504_issues_edit_total_unique_counts_monthly.yml) @@ -11118,19 +11682,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.issues_edit.issues_edit_total_unique_counts_weekly` -Count of WAU taking an action related to an issue +Aggregate count of WAU taking an action related to an issue -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181503_issues_edit_total_unique_counts_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181503_issues_edit_total_unique_counts_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.pipeline_authoring.o_pipeline_authoring_unique_users_committing_ciconfigfile_monthly` @@ -11164,7 +11728,7 @@ Monthly unique user count having merge requests which contains the CI config fil Group: `group::pipeline authoring` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -11176,49 +11740,49 @@ Weekly unique user count having merge requests which contains the CI config file Group: `group::pipeline authoring` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_approve_monthly` -Missing description +Count of MAU using the `/approve` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181508_i_quickactions_approve_monthly.yml) -Group: `group::project management` +Group: `group::code review` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_approve_weekly` -Missing description +Count of WAU using the `/approve` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181506_i_quickactions_approve_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181506_i_quickactions_approve_weekly.yml) -Group: `group::project management` +Group: `group::code review` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_assign_multiple_monthly` -Missing description +Count of MAU using the `/assign @user1 @user2` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181516_i_quickactions_assign_multiple_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181516_i_quickactions_assign_multiple_monthly.yml) Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_assign_multiple_weekly` -Missing description +Count of WAU using the `/assign @user1 @user2` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181514_i_quickactions_assign_multiple_weekly.yml) @@ -11226,35 +11790,35 @@ Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_assign_reviewer_monthly` -Missing description +Count of MAU using the `/assign_reviewer` or `request_reviewer` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181523_i_quickactions_assign_reviewer_monthly.yml) -Group: `group::project management` +Group: `group::code review` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_assign_reviewer_weekly` -Missing description +Count of WAU using the `/assign_reviewer` or `request_reviewer` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181521_i_quickactions_assign_reviewer_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181521_i_quickactions_assign_reviewer_weekly.yml) -Group: `group::project management` +Group: `group::code review` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_assign_self_monthly` -Missing description +Count of MAU using the `/assign me` quick action to assign self to an issuable [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181519_i_quickactions_assign_self_monthly.yml) @@ -11262,23 +11826,23 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_assign_self_weekly` -Missing description +Count of WAU using the `/assign me` quick action to assign self to an issuable -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181517_i_quickactions_assign_self_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181517_i_quickactions_assign_self_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_assign_single_monthly` -Missing description +Count of MAU using the `/assign @user1` quick action to assign a single individual to an issuable [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181512_i_quickactions_assign_single_monthly.yml) @@ -11286,23 +11850,23 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_assign_single_weekly` -Missing description +Count of WAU using the `/assign @user1` quick action to assign a single individual to an issuable -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181510_i_quickactions_assign_single_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181510_i_quickactions_assign_single_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_award_monthly` -Missing description +Count of MAU using the `/award` quick action to set an award emoji on an issuable [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181527_i_quickactions_award_monthly.yml) @@ -11310,23 +11874,23 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_award_weekly` -Missing description +Count of WAU using the `/award` quick action to set an award emoji on an issuable -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181525_i_quickactions_award_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181525_i_quickactions_award_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_board_move_monthly` -Missing description +Count of MAU using the `/board_move` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181530_i_quickactions_board_move_monthly.yml) @@ -11334,59 +11898,59 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_board_move_weekly` -Missing description +Count of WAU using the `/board_move` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181529_i_quickactions_board_move_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181529_i_quickactions_board_move_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_child_epic_monthly` -Missing description +Count of MAU using the `/child_epic` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181534_i_quickactions_child_epic_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181534_i_quickactions_child_epic_monthly.yml) -Group: `group::project management` +Group: `group::product planning` Status: `data_available` -Tiers: `free` +Tiers: `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_child_epic_weekly` -Missing description +Count of WAU using the `/child_epic` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181532_i_quickactions_child_epic_weekly.yml) -Group: `group::project management` +Group: `group::product planning` Status: `data_available` -Tiers: +Tiers: `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_clear_weight_monthly` -Missing description +Count of MAU using the `/clear_weight` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181538_i_quickactions_clear_weight_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181538_i_quickactions_clear_weight_monthly.yml) Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_clear_weight_weekly` -Missing description +Count of WAU using the `/clear_weight` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181536_i_quickactions_clear_weight_weekly.yml) @@ -11394,11 +11958,11 @@ Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_clone_monthly` -Missing description +Count of MAU using the `/clone` quick action to clone an issue. [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181541_i_quickactions_clone_monthly.yml) @@ -11406,23 +11970,23 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_clone_weekly` -Missing description +Count of WAU using the `/clone` quick action to clone an issue. -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181540_i_quickactions_clone_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181540_i_quickactions_clone_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_close_monthly` -Missing description +Count of MAU using the `/close` quick action to close an issuable [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181545_i_quickactions_close_monthly.yml) @@ -11430,23 +11994,23 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_close_weekly` -Missing description +Count of WAU using the `/close` quick action to close an issuable -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181543_i_quickactions_close_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181543_i_quickactions_close_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_confidential_monthly` -Missing description +Count of MAU using the `/confidential` quick action to set an issue as confidential [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181549_i_quickactions_confidential_monthly.yml) @@ -11454,23 +12018,23 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_confidential_weekly` -Missing description +Count of WAU using the `/confidential` quick action to set an issue as confidential -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181547_i_quickactions_confidential_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181547_i_quickactions_confidential_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_copy_metadata_issue_monthly` -Missing description +Count of MAU using the `/copy_metadata` quick action on an issue [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181556_i_quickactions_copy_metadata_issue_monthly.yml) @@ -11478,47 +12042,47 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_copy_metadata_issue_weekly` -Missing description +Count of WAU using the `/copy_metadata` quick action on an issue -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181554_i_quickactions_copy_metadata_issue_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181554_i_quickactions_copy_metadata_issue_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_copy_metadata_merge_request_monthly` -Missing description +Count of MAU using the `/copy_metadata` quick action on a Merge Request [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181553_i_quickactions_copy_metadata_merge_request_monthly.yml) -Group: `group::project management` +Group: `group::source code` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_copy_metadata_merge_request_weekly` -Missing description +Count of WAU using the `/copy_metadata` quick action on a Merge Request -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181551_i_quickactions_copy_metadata_merge_request_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181551_i_quickactions_copy_metadata_merge_request_weekly.yml) -Group: `group::project management` +Group: `group::source code` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_create_merge_request_monthly` -Missing description +Count of MAU using the `/create_merge_request` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181600_i_quickactions_create_merge_request_monthly.yml) @@ -11526,23 +12090,23 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_create_merge_request_weekly` -Missing description +Count of WAU using the `/create_merge_request` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181558_i_quickactions_create_merge_request_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181558_i_quickactions_create_merge_request_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_done_monthly` -Missing description +Count of MAU using the `/done` quick action to mark a todo as done [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181604_i_quickactions_done_monthly.yml) @@ -11550,47 +12114,47 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_done_weekly` -Missing description +Count of WAU using the `/done` quick action to mark a todo as done -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181602_i_quickactions_done_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181602_i_quickactions_done_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_draft_monthly` -Missing description +Count of MAU using the `/draft` quick action on a Merge Request [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181607_i_quickactions_draft_monthly.yml) -Group: `group::project management` +Group: `group::source code` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_draft_weekly` -Missing description +Count of WAU using the `/draft` quick action on a Merge Request -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181605_i_quickactions_draft_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181605_i_quickactions_draft_weekly.yml) -Group: `group::project management` +Group: `group::source code` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_due_monthly` -Missing description +Count of MAU using the `/due` quick action to change the due date on an issuable [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181611_i_quickactions_due_monthly.yml) @@ -11598,23 +12162,23 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_due_weekly` -Missing description +Count of WAU using the `/due` quick action to change the due date on an issuable -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181609_i_quickactions_due_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181609_i_quickactions_due_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_duplicate_monthly` -Missing description +Count of MAU using the `/duplicate` quick action to mark an issue as a duplicate of another [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181615_i_quickactions_duplicate_monthly.yml) @@ -11622,35 +12186,35 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_duplicate_weekly` -Missing description +Count of WAU using the `/duplicate` quick action to mark an issue as a duplicate of another -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181613_i_quickactions_duplicate_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181613_i_quickactions_duplicate_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_epic_monthly` -Missing description +Count of MAU using the `/epic` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181618_i_quickactions_epic_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181618_i_quickactions_epic_monthly.yml) Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_epic_weekly` -Missing description +Count of WAU using the `/epic` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181617_i_quickactions_epic_weekly.yml) @@ -11658,11 +12222,11 @@ Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_estimate_monthly` -Missing description +Count of MAU using the `/estimate` quick action to set a time estimate on an issue [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181622_i_quickactions_estimate_monthly.yml) @@ -11670,19 +12234,19 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_estimate_weekly` -Missing description +Count of WAU using the `/estimate` quick action to set a time estimate on an issue -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181620_i_quickactions_estimate_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181620_i_quickactions_estimate_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_invite_email_multiple_monthly` @@ -11692,7 +12256,7 @@ Unique users using the /invite_email quick action to add a multiple email partic Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -11716,7 +12280,7 @@ Unique users using the /invite_email quick action to add a single email particip Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -11734,19 +12298,19 @@ Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_iteration_monthly` -Missing description +Count of MAU using the `/iteration` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181626_i_quickactions_iteration_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181626_i_quickactions_iteration_monthly.yml) Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_iteration_weekly` -Missing description +Count of WAU using the `/iteration` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181624_i_quickactions_iteration_weekly.yml) @@ -11754,11 +12318,11 @@ Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_label_monthly` -Missing description +Count of MAU using the `/label` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181629_i_quickactions_label_monthly.yml) @@ -11766,23 +12330,23 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_label_weekly` -Missing description +Count of WAU using the `/label` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181628_i_quickactions_label_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181628_i_quickactions_label_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_lock_monthly` -Missing description +Count of MAU using the `/lock` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181633_i_quickactions_lock_monthly.yml) @@ -11790,47 +12354,47 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_lock_weekly` -Missing description +Count of WAU using the `/lock` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181631_i_quickactions_lock_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181631_i_quickactions_lock_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_merge_monthly` -Missing description +Count of MAU using the `/merge` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181637_i_quickactions_merge_monthly.yml) -Group: `group::project management` +Group: `group::source code` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_merge_weekly` -Missing description +Count of WAU using the `/merge` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181635_i_quickactions_merge_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181635_i_quickactions_merge_weekly.yml) -Group: `group::project management` +Group: `group::source code` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_milestone_monthly` -Missing description +Count of MAU using the `/milestone` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181641_i_quickactions_milestone_monthly.yml) @@ -11838,23 +12402,23 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_milestone_weekly` -Missing description +Count of WAU using the `/milestone` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181639_i_quickactions_milestone_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181639_i_quickactions_milestone_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_move_monthly` -Missing description +Count of MAU using the `/move` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181644_i_quickactions_move_monthly.yml) @@ -11862,59 +12426,59 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_move_weekly` -Missing description +Count of WAU using the `/move` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181642_i_quickactions_move_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181642_i_quickactions_move_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_parent_epic_monthly` -Missing description +Count of MAU using the `/parent_epic` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181648_i_quickactions_parent_epic_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181648_i_quickactions_parent_epic_monthly.yml) -Group: `group::project management` +Group: `group::product planning` Status: `data_available` -Tiers: `free` +Tiers: `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_parent_epic_weekly` -Missing description +Count of WAU using the `/parent_epic` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181646_i_quickactions_parent_epic_weekly.yml) -Group: `group::project management` +Group: `group::product planning` Status: `data_available` -Tiers: +Tiers: `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_promote_monthly` -Missing description +Count of MAU using the `/promote` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181652_i_quickactions_promote_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181652_i_quickactions_promote_monthly.yml) Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_promote_weekly` -Missing description +Count of WAU using the `/promote` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181650_i_quickactions_promote_weekly.yml) @@ -11922,35 +12486,35 @@ Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_publish_monthly` -Missing description +Count of MAU using the `/publish` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181655_i_quickactions_publish_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181655_i_quickactions_publish_monthly.yml) -Group: `group::project management` +Group: `group::monitor` Status: `data_available` -Tiers: `free` +Tiers: `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_publish_weekly` -Missing description +Count of WAU using the `/publish` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181654_i_quickactions_publish_weekly.yml) -Group: `group::project management` +Group: `group::monitor` Status: `data_available` -Tiers: +Tiers: `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_reassign_monthly` -Missing description +Count of MAU using the `/reassign @user1` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181659_i_quickactions_reassign_monthly.yml) @@ -11958,71 +12522,71 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_reassign_reviewer_monthly` -Missing description +Count of MAU using the `/reassign_reviewer` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181703_i_quickactions_reassign_reviewer_monthly.yml) -Group: `group::project management` +Group: `group::code review` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_reassign_reviewer_weekly` -Missing description +Count of WAU using the `/reassign_reviewer` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181701_i_quickactions_reassign_reviewer_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181701_i_quickactions_reassign_reviewer_weekly.yml) -Group: `group::project management` +Group: `group::code review` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_reassign_weekly` -Missing description +Count of WAU using the `/reassign @user1` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181657_i_quickactions_reassign_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181657_i_quickactions_reassign_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_rebase_monthly` -Missing description +Count of MAU using the `/rebase` quick action on a Merge Request [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181707_i_quickactions_rebase_monthly.yml) -Group: `group::project management` +Group: `group::source code` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_rebase_weekly` -Missing description +Count of WAU using the `/rebase` quick action on a Merge Request -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181705_i_quickactions_rebase_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181705_i_quickactions_rebase_weekly.yml) -Group: `group::project management` +Group: `group::source code` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_relabel_monthly` -Missing description +Count of MAU using the `/relabel` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181710_i_quickactions_relabel_monthly.yml) @@ -12030,23 +12594,23 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_relabel_weekly` -Missing description +Count of WAU using the `/relabel` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181708_i_quickactions_relabel_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181708_i_quickactions_relabel_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_relate_monthly` -Missing description +Count of MAU using the `/relate` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181714_i_quickactions_relate_monthly.yml) @@ -12054,47 +12618,47 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_relate_weekly` -Missing description +Count of WAU using the `/relate` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181712_i_quickactions_relate_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181712_i_quickactions_relate_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_remove_child_epic_monthly` -Missing description +Count of MAU using the `/remove_child_epic` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181718_i_quickactions_remove_child_epic_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181718_i_quickactions_remove_child_epic_monthly.yml) -Group: `group::project management` +Group: `group::product planning` Status: `data_available` -Tiers: `free` +Tiers: `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_remove_child_epic_weekly` -Missing description +Count of WAU using the `/remove_child_epic` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181716_i_quickactions_remove_child_epic_weekly.yml) -Group: `group::project management` +Group: `group::product planning` Status: `data_available` -Tiers: +Tiers: `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_remove_due_date_monthly` -Missing description +Count of MAU using the `/remove_due_date` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181721_i_quickactions_remove_due_date_monthly.yml) @@ -12102,35 +12666,35 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_remove_due_date_weekly` -Missing description +Count of WAU using the `/remove_due_date` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181719_i_quickactions_remove_due_date_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181719_i_quickactions_remove_due_date_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_remove_epic_monthly` -Missing description +Count of MAU using the `/remove_epic` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181725_i_quickactions_remove_epic_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181725_i_quickactions_remove_epic_monthly.yml) Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_remove_epic_weekly` -Missing description +Count of WAU using the `/remove_epic` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181723_i_quickactions_remove_epic_weekly.yml) @@ -12138,11 +12702,11 @@ Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_remove_estimate_monthly` -Missing description +Count of MAU using the `/remove_estimate` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181729_i_quickactions_remove_estimate_monthly.yml) @@ -12150,35 +12714,35 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_remove_estimate_weekly` -Missing description +Count of WAU using the `/remove_estimate` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181727_i_quickactions_remove_estimate_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181727_i_quickactions_remove_estimate_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_remove_iteration_monthly` -Missing description +Count of MAU using the `/remove_iteration` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181732_i_quickactions_remove_iteration_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181732_i_quickactions_remove_iteration_monthly.yml) Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_remove_iteration_weekly` -Missing description +Count of WAU using the `/remove_iteration` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181731_i_quickactions_remove_iteration_weekly.yml) @@ -12186,11 +12750,11 @@ Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_remove_milestone_monthly` -Missing description +Count of MAU using the `/remove_milestone` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181736_i_quickactions_remove_milestone_monthly.yml) @@ -12198,47 +12762,47 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_remove_milestone_weekly` -Missing description +Count of WAU using the `/remove_milestone` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181734_i_quickactions_remove_milestone_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181734_i_quickactions_remove_milestone_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_remove_parent_epic_monthly` -Missing description +Count of MAU using the `/remove_parent_epic` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181740_i_quickactions_remove_parent_epic_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181740_i_quickactions_remove_parent_epic_monthly.yml) -Group: `group::project management` +Group: `group::product planning` Status: `data_available` -Tiers: `free` +Tiers: `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_remove_parent_epic_weekly` -Missing description +Count of WAU using the `/remove_parent_epic` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181738_i_quickactions_remove_parent_epic_weekly.yml) -Group: `group::project management` +Group: `group::product planning` Status: `data_available` -Tiers: +Tiers: `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_remove_time_spent_monthly` -Missing description +Count of MAU using the `/remove_time_spent` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181744_i_quickactions_remove_time_spent_monthly.yml) @@ -12246,23 +12810,23 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_remove_time_spent_weekly` -Missing description +Count of WAU using the `/remove_time_spent` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181742_i_quickactions_remove_time_spent_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181742_i_quickactions_remove_time_spent_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_remove_zoom_monthly` -Missing description +Count of MAU using the `/remove_zoom` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181747_i_quickactions_remove_zoom_monthly.yml) @@ -12270,23 +12834,23 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_remove_zoom_weekly` -Missing description +Count of WAU using the `/remove_zoom` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181745_i_quickactions_remove_zoom_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181745_i_quickactions_remove_zoom_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_reopen_monthly` -Missing description +Count of MAU using the `/reopen` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181751_i_quickactions_reopen_monthly.yml) @@ -12294,23 +12858,23 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_reopen_weekly` -Missing description +Count of WAU using the `/reopen` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181749_i_quickactions_reopen_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181749_i_quickactions_reopen_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_shrug_monthly` -Missing description +Count of MAU using the `/shrug` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181755_i_quickactions_shrug_monthly.yml) @@ -12318,23 +12882,23 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_shrug_weekly` -Missing description +Count of WAU using the `/shrug` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181753_i_quickactions_shrug_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181753_i_quickactions_shrug_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_spend_add_monthly` -Missing description +Count of MAU using the `/spend` quick action to add time spent [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181802_i_quickactions_spend_add_monthly.yml) @@ -12342,23 +12906,23 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_spend_add_weekly` -Missing description +Count of WAU using the `/spend` quick action to add time spent -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181800_i_quickactions_spend_add_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181800_i_quickactions_spend_add_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_spend_subtract_monthly` -Missing description +Count of MAU using the `/spend` quick action to subtract time spent [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181758_i_quickactions_spend_subtract_monthly.yml) @@ -12366,47 +12930,47 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_spend_subtract_weekly` -Missing description +Count of WAU using the `/spend` quick action to subtract time spent -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181756_i_quickactions_spend_subtract_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181756_i_quickactions_spend_subtract_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_submit_review_monthly` -Missing description +Count of MAU using the `/submit_review` quick action on Merge Requests [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181806_i_quickactions_submit_review_monthly.yml) -Group: `group::project management` +Group: `group::code review` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_submit_review_weekly` -Missing description +Count of WAU using the `/submit_review` quick action on Merge Requests -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181804_i_quickactions_submit_review_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181804_i_quickactions_submit_review_weekly.yml) -Group: `group::project management` +Group: `group::code review` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_subscribe_monthly` -Missing description +Count of MAU using the `/subscribe` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181809_i_quickactions_subscribe_monthly.yml) @@ -12414,23 +12978,23 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_subscribe_weekly` -Missing description +Count of WAU using the `/subscribe` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181808_i_quickactions_subscribe_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181808_i_quickactions_subscribe_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_tableflip_monthly` -Missing description +Count of MAU using the `/tableflip` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181813_i_quickactions_tableflip_monthly.yml) @@ -12438,71 +13002,71 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_tableflip_weekly` -Missing description +Count of WAU using the `/tableflip` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181811_i_quickactions_tableflip_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181811_i_quickactions_tableflip_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_tag_monthly` -Missing description +Count of MAU using the `/tag` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181817_i_quickactions_tag_monthly.yml) -Group: `group::project management` +Group: `group::source code` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_tag_weekly` -Missing description +Count of WAU using the `/tag` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181815_i_quickactions_tag_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181815_i_quickactions_tag_weekly.yml) -Group: `group::project management` +Group: `group::source code` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_target_branch_monthly` -Missing description +Count of MAU using the `/target_branch` quick action on Merge Requests [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181821_i_quickactions_target_branch_monthly.yml) -Group: `group::project management` +Group: `group::source code` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_target_branch_weekly` -Missing description +Count of WAU using the `/target_branch` quick action on Merge Requests -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181819_i_quickactions_target_branch_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181819_i_quickactions_target_branch_weekly.yml) -Group: `group::project management` +Group: `group::source code` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_title_monthly` -Missing description +Count of MAU using the `/title` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181824_i_quickactions_title_monthly.yml) @@ -12510,23 +13074,23 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_title_weekly` -Missing description +Count of WAU using the `/title` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181822_i_quickactions_title_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181822_i_quickactions_title_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_todo_monthly` -Missing description +Count of MAU using the `/todo` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181828_i_quickactions_todo_monthly.yml) @@ -12534,95 +13098,95 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_todo_weekly` -Missing description +Count of WAU using the `/todo` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181826_i_quickactions_todo_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181826_i_quickactions_todo_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_unassign_all_monthly` -Missing description +Count of MAU using the `/unassign` quick action on Merge Requests [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181835_i_quickactions_unassign_all_monthly.yml) -Group: `group::project management` +Group: `group::source code` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_unassign_all_weekly` -Missing description +Count of WAU using the `/unassign` quick action on Merge Requests -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181833_i_quickactions_unassign_all_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181833_i_quickactions_unassign_all_weekly.yml) -Group: `group::project management` +Group: `group::source code` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_unassign_reviewer_monthly` -Missing description +Count of MAU using the `/unassign_reviewer` or `/remove_reviewer` quick action on Merge Requests [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181839_i_quickactions_unassign_reviewer_monthly.yml) -Group: `group::project management` +Group: `group::code review` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_unassign_reviewer_weekly` -Missing description +Count of WAU using the `/unassign_reviewer` or `/remove_reviewer` quick action on Merge Requests -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181837_i_quickactions_unassign_reviewer_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181837_i_quickactions_unassign_reviewer_weekly.yml) -Group: `group::project management` +Group: `group::code review` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_unassign_specific_monthly` -Missing description +Count of MAU using the `/unassign @user1` quick action on Merge Requests [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181832_i_quickactions_unassign_specific_monthly.yml) -Group: `group::project management` +Group: `group::source code` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_unassign_specific_weekly` -Missing description +Count of WAU using the `/unassign @user1` quick action on Merge Requests -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181830_i_quickactions_unassign_specific_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181830_i_quickactions_unassign_specific_weekly.yml) -Group: `group::project management` +Group: `group::source code` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_unlabel_all_monthly` -Missing description +Count of MAU using the `/unlabel` quick action to remove all labels [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181846_i_quickactions_unlabel_all_monthly.yml) @@ -12630,23 +13194,23 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_unlabel_all_weekly` -Missing description +Count of WAU using the `/unlabel` quick action to remove all labels -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181845_i_quickactions_unlabel_all_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181845_i_quickactions_unlabel_all_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_unlabel_specific_monthly` -Missing description +Count of MAU using the `/unlabel` or `/remove_label` quick action to remove one or more specific labels [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181843_i_quickactions_unlabel_specific_monthly.yml) @@ -12654,23 +13218,23 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_unlabel_specific_weekly` -Missing description +Count of WAU using the `/unlabel` or `/remove_label` quick action to remove one or more specific labels -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181841_i_quickactions_unlabel_specific_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181841_i_quickactions_unlabel_specific_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_unlock_monthly` -Missing description +Count of MAU using the `/unlock` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181850_i_quickactions_unlock_monthly.yml) @@ -12678,23 +13242,23 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_unlock_weekly` -Missing description +Count of WAU using the `/unlock` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181848_i_quickactions_unlock_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181848_i_quickactions_unlock_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_unsubscribe_monthly` -Missing description +Count of MAU using the `/unsubscribe` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181854_i_quickactions_unsubscribe_monthly.yml) @@ -12702,35 +13266,35 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_unsubscribe_weekly` -Missing description +Count of WAU using the `/unsubscribe` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181852_i_quickactions_unsubscribe_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181852_i_quickactions_unsubscribe_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_weight_monthly` -Missing description +Count of MAU using the `/weight` quick action -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181857_i_quickactions_weight_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181857_i_quickactions_weight_monthly.yml) Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_weight_weekly` -Missing description +Count of WAU using the `/weight` quick action [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181856_i_quickactions_weight_weekly.yml) @@ -12738,35 +13302,35 @@ Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_wip_monthly` -Missing description +Count of MAU using the `/wip` quick action on Merge Requests [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181901_i_quickactions_wip_monthly.yml) -Group: `group::project management` +Group: `group::source code` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_wip_weekly` -Missing description +Count of WAU using the `/wip` quick action on Merge Requests -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181859_i_quickactions_wip_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181859_i_quickactions_wip_weekly.yml) -Group: `group::project management` +Group: `group::source code` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_zoom_monthly` -Missing description +Count of MAU using the `/zoom` quick action on Issues [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181905_i_quickactions_zoom_monthly.yml) @@ -12774,31 +13338,31 @@ Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.i_quickactions_zoom_weekly` -Missing description +Count of WAU using the `/zoom` quick action on Issues -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216181903_i_quickactions_zoom_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216181903_i_quickactions_zoom_weekly.yml) Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.quickactions_total_unique_counts_monthly` -Missing description +Count of MAU using one or more quick actions [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216184803_quickactions_total_unique_counts_monthly.yml) -Group: `` +Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.quickactions.quickactions_total_unique_counts_weekly` @@ -12908,6 +13472,30 @@ Status: `data_available` Tiers: `free`, `premium`, `ultimate` +### `redis_hll_counters.secure.users_expanding_secure_security_report_monthly` + +Count of expanding the security report widget + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210409095855_users_expanding_secure_security_report_monthly.yml) + +Group: `group::static analysis` + +Status: `implemented` + +Tiers: `free`, `premium`, `ultimate` + +### `redis_hll_counters.secure.users_expanding_secure_security_report_weekly` + +Count of expanding the security report widget + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210409095855_users_expanding_secure_security_report_weekly.yml) + +Group: `group::static analysis` + +Status: `implemented` + +Tiers: `free`, `premium`, `ultimate` + ### `redis_hll_counters.snippets.i_snippets_show_monthly` Missing description @@ -13114,7 +13702,7 @@ Tiers: `premium`, `ultimate` ### `redis_hll_counters.testing.i_testing_full_code_quality_report_total_weekly` -Count of unique users per week|month who visit the full code quality report +Count of unique users per week who visit the full code quality report [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216182145_i_testing_full_code_quality_report_total_weekly.yml) @@ -13126,7 +13714,7 @@ Tiers: `premium`, `ultimate` ### `redis_hll_counters.testing.i_testing_group_code_coverage_project_click_total_monthly` -Count of unique users per week|month who click on a project link in the group code coverage table +Aggregated count of unique users who have clicked from group code coverage page to an individual project page each month. [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216182153_i_testing_group_code_coverage_project_click_total_monthly.yml) @@ -13138,19 +13726,19 @@ Tiers: `premium`, `ultimate` ### `redis_hll_counters.testing.i_testing_group_code_coverage_project_click_total_weekly` -Missing description +Aggregated count of unique users who have clicked from group code coverage page to an individual project page each week. [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216184132_i_testing_group_code_coverage_project_click_total_weekly.yml) -Group: `` +Group: `group::testing` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `redis_hll_counters.testing.i_testing_group_code_coverage_visit_total_monthly` -Count of unique users per week|month who visited the group code coverage page +Count of unique users per month who visited the group code coverage page [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216182143_i_testing_group_code_coverage_visit_total_monthly.yml) @@ -13162,7 +13750,7 @@ Tiers: `premium`, `ultimate` ### `redis_hll_counters.testing.i_testing_group_code_coverage_visit_total_weekly` -Count of unique users per week|month who visited the group code coverage page +Count of unique users per week who visited the group code coverage page [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216182141_i_testing_group_code_coverage_visit_total_weekly.yml) @@ -13174,7 +13762,7 @@ Tiers: `premium`, `ultimate` ### `redis_hll_counters.testing.i_testing_load_performance_widget_total_monthly` -Count of unique users per week|month who expanded the load performance report MR widget +Count of unique users per month who expanded the load performance report MR widget [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216182156_i_testing_load_performance_widget_total_monthly.yml) @@ -13186,7 +13774,7 @@ Tiers: `premium`, `ultimate` ### `redis_hll_counters.testing.i_testing_load_performance_widget_total_weekly` -Count of unique users per week|month who expanded the load performance report MR widget +Count of unique users per week who expanded the load performance report MR widget [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216182154_i_testing_load_performance_widget_total_weekly.yml) @@ -13198,31 +13786,31 @@ Tiers: `premium`, `ultimate` ### `redis_hll_counters.testing.i_testing_metrics_report_artifact_uploaders_monthly` -Internal Tracking to count number of unit tests parsed for planning of future code testing features. Data available [here](https://app.periscopedata.com/app/gitlab/788674/Verify:Testing-Group-Metrics?widget=10454394&udv=0) +Tracks number of metrics reports uploaded monthly. -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216182200_i_testing_metrics_report_artifact_uploaders_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216182200_i_testing_metrics_report_artifact_uploaders_monthly.yml) Group: `group::testing` Status: `data_available` -Tiers: `free`, `premium`, `ultimate` +Tiers: `premium`, `ultimate` ### `redis_hll_counters.testing.i_testing_metrics_report_artifact_uploaders_weekly` -Internal Tracking to count number of unit tests parsed for planning of future code testing features. Data available [here](https://app.periscopedata.com/app/gitlab/788674/Verify:Testing-Group-Metrics?widget=10454394&udv=0) +Count of unique users per week who trigger a pipeline that uploads a metrics report. -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216182158_i_testing_metrics_report_artifact_uploaders_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216182158_i_testing_metrics_report_artifact_uploaders_weekly.yml) Group: `group::testing` Status: `data_available` -Tiers: `free`, `premium`, `ultimate` +Tiers: `premium`, `ultimate` ### `redis_hll_counters.testing.i_testing_metrics_report_widget_total_monthly` -Count of unique users per week|month who expanded the metrics report MR widget +Count of unique users per month who expanded the metrics report MR widget [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216182139_i_testing_metrics_report_widget_total_monthly.yml) @@ -13234,7 +13822,7 @@ Tiers: `premium`, `ultimate` ### `redis_hll_counters.testing.i_testing_metrics_report_widget_total_weekly` -Count of unique users per week|month who expanded the metrics report MR widget +Count of unique users per week who expanded the metrics report MR widget [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216182138_i_testing_metrics_report_widget_total_weekly.yml) @@ -13244,6 +13832,30 @@ Status: `data_available` Tiers: `premium`, `ultimate` +### `redis_hll_counters.testing.i_testing_summary_widget_total_monthly` + +Unique users that expand the test summary merge request widget by month + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210413205507_i_testing_summary_widget_total_monthly.yml) + +Group: `group::testing` + +Status: `implemented` + +Tiers: `free`, `premium`, `ultimate` + +### `redis_hll_counters.testing.i_testing_summary_widget_total_weekly` + +Unique users that expand the test summary merge request widget by week + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210413205507_i_testing_summary_widget_total_weekly.yml) + +Group: `group::testing` + +Status: `implemented` + +Tiers: `free`, `premium`, `ultimate` + ### `redis_hll_counters.testing.i_testing_test_case_parsed_monthly` Internal Tracking to count number of unit tests parsed for planning of future code testing features. Data available [here](https://app.periscopedata.com/app/gitlab/788674/Verify:Testing-Group-Metrics?widget=10454394&udv=0) @@ -13270,7 +13882,7 @@ Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.testing.i_testing_web_performance_widget_total_monthly` -Count of unique users per week|month who expanded the browser performance report MR widget +Count of unique users per month who expanded the browser performance report MR widget [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216182151_i_testing_web_performance_widget_total_monthly.yml) @@ -13282,7 +13894,7 @@ Tiers: `premium`, `ultimate` ### `redis_hll_counters.testing.i_testing_web_performance_widget_total_weekly` -Count of unique users per week|month who expanded the browser performance report MR widget +Count of unique users per week who expanded the browser performance report MR widget [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216182149_i_testing_web_performance_widget_total_weekly.yml) @@ -13300,7 +13912,7 @@ Missing description Group: `` -Status: `data_available` +Status: `removed` Tiers: `free` @@ -13312,9 +13924,57 @@ Missing description Group: `` -Status: `data_available` +Status: `removed` -Tiers: +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.testing.users_expanding_testing_accessibility_report_monthly` + +Count of expanding the accessibility report widget + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210409100628_users_expanding_testing_accessibility_report_monthly.yml) + +Group: `group::testing` + +Status: `implemented` + +Tiers: `free`, `premium`, `ultimate` + +### `redis_hll_counters.testing.users_expanding_testing_accessibility_report_weekly` + +Count of expanding the accessibility report widget + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210409100628_users_expanding_testing_accessibility_report_weekly.yml) + +Group: `group::testing` + +Status: `implemented` + +Tiers: `free`, `premium`, `ultimate` + +### `redis_hll_counters.testing.users_expanding_testing_code_quality_report_monthly` + +Count of expanding the code quality widget + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210409100451_users_expanding_testing_code_quality_report_monthly.yml) + +Group: `group::testing` + +Status: `implemented` + +Tiers: `free`, `premium`, `ultimate` + +### `redis_hll_counters.testing.users_expanding_testing_code_quality_report_weekly` + +Count of expanding the code quality widget + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210409100451_users_expanding_testing_code_quality_report_weekly.yml) + +Group: `group::testing` + +Status: `implemented` + +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.user_packages.i_package_composer_user_monthly` @@ -13720,7 +14380,7 @@ Information about the operating system running GitLab Group: `group::distribution` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -13736,23 +14396,13 @@ Status: `data_available` Tiers: `free`, `premium`, `ultimate` -### `topology.duration_s` - -Time it took to collect topology data - -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216180922_duration_s.yml) - -Group: `group::memory` - -Status: `data_available` - -Tiers: `free`, `premium`, `ultimate` +### `topology` -### `topology.failures` +Topology data -Contains information about failed queries +[Object JSON schema](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/objects_schemas/topology_schema.json) -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216180924_failures.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/settings/20210323120839_topology.yml) Group: `group::memory` @@ -14086,7 +14736,7 @@ Tiers: `free` ### `usage_activity_by_stage.create.merge_requests_with_added_rules` -Merge Requests with added rules +Merge requests with added rules [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216175047_merge_requests_with_added_rules.yml) @@ -14300,6 +14950,18 @@ Status: `data_available` Tiers: `free` +### `usage_activity_by_stage.enablement.counts.geo_node_usage.git_fetch_event_count_weekly` + +Number of Git fetch events from Prometheus on the Geo secondary + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210309194425_git_fetch_event_count_weekly.yml) + +Group: `group::geo` + +Status: `data_available` + +Tiers: `premium`, `ultimate` + ### `usage_activity_by_stage.manage.bulk_imports.gitlab` Distinct count of users that triggered an import using the Group Migration tool @@ -15022,15 +15684,15 @@ Tiers: `free` ### `usage_activity_by_stage.plan.assignee_lists` -Missing description +Count of users creating assignee lists on Boards [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216181132_assignee_lists.yml) -Group: `group::plan` +Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `usage_activity_by_stage.plan.epics` @@ -15046,63 +15708,63 @@ Tiers: `free` ### `usage_activity_by_stage.plan.issues` -Missing description +Count of users creating Issues [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181115_issues.yml) -Group: `group::plan` +Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage.plan.label_lists` -Missing description +Count of users creating label lists on Boards [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181135_label_lists.yml) -Group: `group::plan` +Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage.plan.milestone_lists` -Missing description +Count of users creating milestone lists on Boards -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181137_milestone_lists.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216181137_milestone_lists.yml) -Group: `group::plan` +Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `usage_activity_by_stage.plan.notes` -Missing description +Count of users creating Notes on Issues [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181117_notes.yml) -Group: `group::plan` +Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage.plan.projects` -Missing description +Count of users creating projects [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181119_projects.yml) -Group: `group::plan` +Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage.plan.projects_jira_active` @@ -15166,15 +15828,15 @@ Tiers: `free` ### `usage_activity_by_stage.plan.todos` -Missing description +Count of users todos created [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181121_todos.yml) -Group: `group::plan` +Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage.release.deployments` @@ -15346,7 +16008,7 @@ Tiers: `free` ### `usage_activity_by_stage.secure.user_container_scanning_jobs` -no idea, Count of Container Scanning jobs run, it implies user but AFAIK we don't track per user +Distinct count per user of Container Scanning jobs run [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216175501_user_container_scanning_jobs.yml) @@ -15408,7 +16070,7 @@ Tiers: `ultimate` Users who set personal preference to see Details on Group overview page -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182207_user_preferences_group_overview_security_dashboard.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216182207_user_preferences_group_overview_security_dashboard.yml) Group: `group::threat insights` @@ -16018,7 +16680,7 @@ Tiers: `free` ### `usage_activity_by_stage_monthly.create.merge_requests_with_added_rules` -Merge Requests with added rules +Merge requests with added rules [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216175103_merge_requests_with_added_rules.yml) @@ -16942,15 +17604,15 @@ Tiers: `free` ### `usage_activity_by_stage_monthly.plan.assignee_lists` -Missing description +Count of MAU creating assignee lists on Boards [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181156_assignee_lists.yml) -Group: `group::plan` +Group: `group::project management` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `usage_activity_by_stage_monthly.plan.epics` @@ -16966,63 +17628,63 @@ Tiers: `free` ### `usage_activity_by_stage_monthly.plan.issues` -Missing description +Count of MAU creating issues [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181139_issues.yml) -Group: `group::plan` +Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.plan.label_lists` -Missing description +Count of MAU creating label lists on Boards [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181200_label_lists.yml) -Group: `group::plan` +Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.plan.milestone_lists` -Missing description +Count of MAU created milestone lists on Boards -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181201_milestone_lists.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181201_milestone_lists.yml) -Group: `group::plan` +Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `usage_activity_by_stage_monthly.plan.notes` -Missing description +Count of MAU commenting on an issuable [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181141_notes.yml) -Group: `group::plan` +Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.plan.projects` -Missing description +Count of MAU creating projects [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181143_projects.yml) -Group: `group::plan` +Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.plan.projects_jira_active` @@ -17086,15 +17748,15 @@ Tiers: `free` ### `usage_activity_by_stage_monthly.plan.todos` -Missing description +Count of MAU creating todos [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181145_todos.yml) -Group: `group::plan` +Group: `group::project management` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.release.deployments` @@ -17182,7 +17844,7 @@ Tiers: `free` ### `usage_activity_by_stage_monthly.secure.container_scanning_pipeline` -no idea, what is this when did it get added? guess pipelines containing a CS job +Pipelines containing a Container Scanning job [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216175507_container_scanning_pipeline.yml) @@ -17350,7 +18012,7 @@ Tiers: `free` ### `usage_activity_by_stage_monthly.secure.user_container_scanning_jobs` -no idea, Count of Container Scanning jobs run, it implies user and monthly, but AFAIK we don't track per user +Distinct count per user of Container Scanning jobs run monthly [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216175505_user_container_scanning_jobs.yml) @@ -17412,7 +18074,7 @@ Tiers: `ultimate` Users who set personal preference to see Security Dashboard on Group overview page -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216182209_user_preferences_group_overview_security_dashboard.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216182209_user_preferences_group_overview_security_dashboard.yml) Group: `group::threat insights` diff --git a/doc/development/usage_ping/index.md b/doc/development/usage_ping/index.md index 604d59f42e1..bf423d68700 100644 --- a/doc/development/usage_ping/index.md +++ b/doc/development/usage_ping/index.md @@ -13,7 +13,7 @@ This guide describes Usage Ping's purpose and how it's implemented. For more information about Product Intelligence, see: - [Product Intelligence Guide](https://about.gitlab.com/handbook/product/product-intelligence-guide/) -- [Snowplow Guide](../snowplow.md) +- [Snowplow Guide](../snowplow/index.md) More links: @@ -25,7 +25,7 @@ More links: ## What is Usage Ping? - GitLab sends a weekly payload containing usage data to GitLab Inc. Usage Ping provides high-level data to help our product, support, and sales teams. It does not send any project names, usernames, or any other specific data. The information from the usage ping is not anonymous, it is linked to the hostname of the instance. Sending usage ping is optional, and any instance can disable analytics. -- The usage data is primarily composed of row counts for different tables in the instance’s database. By comparing these counts month over month (or week over week), we can get a rough sense for how an instance is using the different features in the product. In addition to counts, other facts +- The usage data is primarily composed of row counts for different tables in the instance's database. By comparing these counts month over month (or week over week), we can get a rough sense for how an instance is using the different features in the product. In addition to counts, other facts that help us classify and understand GitLab installations are collected. - Usage ping is important to GitLab as we use it to calculate our Stage Monthly Active Users (SMAU) which helps us measure the success of our stages and features. - While usage ping is enabled, GitLab gathers data from the other instances and can show usage statistics of your instance to your users. @@ -33,8 +33,8 @@ More links: ### Why should we enable Usage Ping? - The main purpose of Usage Ping is to build a better GitLab. Data about how GitLab is used is collected to better understand feature/stage adoption and usage, which helps us understand how GitLab is adding value and helps our team better understand the reasons why people use GitLab and with this knowledge we're able to make better product decisions. -- As a benefit of having the usage ping active, GitLab lets you analyze the users’ activities over time of your GitLab installation. -- As a benefit of having the usage ping active, GitLab provides you with The DevOps Report,which gives you an overview of your entire instance’s adoption of Concurrent DevOps from planning to monitoring. +- As a benefit of having the usage ping active, GitLab lets you analyze the users' activities over time of your GitLab installation. +- As a benefit of having the usage ping active, GitLab provides you with The DevOps Report,which gives you an overview of your entire instance's adoption of Concurrent DevOps from planning to monitoring. - You get better, more proactive support. (assuming that our TAMs and support organization used the data to deliver more value) - You get insight and advice into how to get the most value out of your investment in GitLab. Wouldn't you want to know that a number of features or values are not being adopted in your organization? - You get a report that illustrates how you compare against other similar organizations (anonymized), with specific advice and recommendations on how to improve your DevOps processes. @@ -49,7 +49,9 @@ More links: You can view the exact JSON payload sent to GitLab Inc. in the administration panel. To view the payload: -1. Navigate to **Admin Area > Settings > Metrics and profiling**. +1. Sign in as a user with [Administrator](../../user/permissions.md) permissions. +1. In the top navigation bar, click **(admin)** **Admin Area**. +1. In the left sidebar, go to **Settings > Metrics and profiling**. 1. Expand the **Usage statistics** section. 1. Click the **Preview payload** button. @@ -57,9 +59,17 @@ For an example payload, see [Example Usage Ping payload](#example-usage-ping-pay ## Disable Usage Ping -To disable Usage Ping in the GitLab UI, go to the **Settings** page of your administration panel and uncheck the **Usage Ping** checkbox. +To disable Usage Ping in the GitLab UI: -To disable Usage Ping and prevent it from being configured in the future through the administration panel, Omnibus installs can set the following in [`gitlab.rb`](https://docs.gitlab.com/omnibus/settings/configuration.html#configuration-options): +1. Sign in as a user with [Administrator](../../user/permissions.md) permissions. +1. In the top navigation bar, click **(admin)** **Admin Area**. +1. In the left sidebar, go to **Settings > Metrics and profiling**. +1. Expand the **Usage statistics** section. +1. Clear the **Usage Ping** checkbox and click **Save changes**. + +To disable Usage Ping and prevent it from being configured in the future through +the administration panel, Omnibus installs can set the following in +[`gitlab.rb`](https://docs.gitlab.com/omnibus/settings/configuration.html#configuration-options): ```ruby gitlab_rails['usage_ping_enabled'] = false @@ -204,11 +214,12 @@ For GitLab.com, there are extremely large tables with 15 second query timeouts, | `merge_request_diff_files` | 1082 | | `events` | 514 | -We have several batch counting methods available: +The following operation methods are available for your use: - [Ordinary Batch Counters](#ordinary-batch-counters) - [Distinct Batch Counters](#distinct-batch-counters) -- [Sum Batch Counters](#sum-batch-counters) +- [Sum Batch Operation](#sum-batch-operation) +- [Add Operation](#add-operation) - [Estimated Batch Counters](#estimated-batch-counters) Batch counting requires indexes on columns to calculate max, min, and range queries. In some cases, @@ -266,7 +277,7 @@ distinct_count(::Note.with_suggestions.where(time_period), :author_id, start: :: distinct_count(::Clusters::Applications::CertManager.where(time_period).available.joins(:cluster), 'clusters.user_id') ``` -### Sum Batch Counters +### Sum Batch Operation Handles `ActiveRecord::StatementInvalid` error @@ -307,6 +318,25 @@ sum(Issue.group(:state_id), :weight)) # returns => {1=>3542, 2=>6820} ``` +### Add Operation + +Handles `StandardError`. + +Returns `-1` if any of the arguments are `-1`. + +Sum the values given as parameters. + +Method: `add(*args)` + +Examples + +```ruby +project_imports = distinct_count(::Project.where.not(import_type: nil), :creator_id) +bulk_imports = distinct_count(::BulkImport, :user_id) + + add(project_imports, bulk_imports) +``` + ### Estimated Batch Counters > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/48233) in GitLab 13.7. @@ -467,6 +497,7 @@ Implemented using Redis methods [PFADD](https://redis.io/commands/pfadd) and [PF redis_slot: compliance expiry: 42 # 6 weeks aggregation: weekly + feature_flag: usage_data_i_compliance_credential_inventory ``` Keys: @@ -498,17 +529,18 @@ Implemented using Redis methods [PFADD](https://redis.io/commands/pfadd) and [PF aggregation. - `aggregation`: may be set to a `:daily` or `:weekly` key. Defines how counting data is stored in Redis. Aggregation on a `daily` basis does not pull more fine grained data. - - `feature_flag`: optional `default_enabled: :yaml`. If no feature flag is set then the tracking is enabled. For details, see our [GitLab internal Feature flags](../feature_flags/index.md) documentation. The feature flags are owned by the group adding the event tracking. + - `feature_flag`: optional `default_enabled: :yaml`. If no feature flag is set then the tracking is enabled. One feature flag can be used for multiple events. For details, see our [GitLab internal Feature flags](../feature_flags/index.md) documentation. The feature flags are owned by the group adding the event tracking. Use one of the following methods to track events: -1. Track event in controller using `RedisTracking` module with `track_redis_hll_event(*controller_actions, name:, if: nil)`. +1. Track event in controller using `RedisTracking` module with `track_redis_hll_event(*controller_actions, name:, if: nil, &block)`. Arguments: - `controller_actions`: controller actions we want to track. - `name`: event name. - `if`: optional custom conditions, using the same format as with Rails callbacks. + - `&block`: optional block that computes and returns the `custom_id` that we want to track. This will override the `visitor_id`. Example usage: @@ -536,8 +568,6 @@ Use one of the following methods to track events: 1. Track event in API using `increment_unique_values(event_name, values)` helper method. - To be able to track the event, Usage Ping must be enabled and the event feature `usage_data_<event_name>` must be enabled. - Arguments: - `event_name`: event name. @@ -581,10 +611,6 @@ Use one of the following methods to track events: API requests are protected by checking for a valid CSRF token. - To increment the values, the related feature `usage_data_<event_name>` should be - set to `default_enabled: true`. For more information, see - [Feature flags in development of GitLab](../feature_flags/index.md). - ```plaintext POST /usage_data/increment_unique_users ``` @@ -609,8 +635,6 @@ Use one of the following methods to track events: Usage Data API is behind `usage_data_api` feature flag which, as of GitLab 13.7, is now set to `default_enabled: true`. - Each event tracked using Usage Data API is behind a feature flag `usage_data_#{event_name}` which should be `default_enabled: true` - ```javascript import api from '~/api'; @@ -784,6 +808,16 @@ end Please refer to [the `PrometheusClient` definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/prometheus_client.rb) for how to use its API to query for data. +### Fallback values for UsagePing + +We return fallback values in these cases: + +| Case | Value | +|-----------------------------|-------| +| Deprecated Metric | -1000 | +| Timeouts, general failures | -1 | +| Standard errors in counters | -2 | + ## Developing and testing Usage Ping ### 1. Naming and placing the metrics @@ -827,7 +861,7 @@ pry(main)> Gitlab::UsageData.count(User.active) Paste the SQL query into `#database-lab` to see how the query performs at scale. - `#database-lab` is a Slack channel which uses a production-sized environment to test your queries. -- GitLab.com’s production database has a 15 second timeout. +- GitLab.com's production database has a 15 second timeout. - Any single query must stay below [1 second execution time](../query_performance.md#timing-guidelines-for-queries) with cold caches. - Add a specialized index on columns involved to reduce the execution time. @@ -875,18 +909,56 @@ On GitLab.com, we have DangerBot setup to monitor Product Intelligence related f On GitLab.com, the Product Intelligence team regularly monitors Usage Ping. They may alert you that your metrics need further optimization to run quicker and with greater success. You may also use the [Usage Ping QA dashboard](https://app.periscopedata.com/app/gitlab/632033/Usage-Ping-QA) to check how well your metric performs. The dashboard allows filtering by GitLab version, by "Self-managed" & "SaaS" and shows you how many failures have occurred for each metric. Whenever you notice a high failure rate, you may re-optimize your metric. -### Optional: Test Prometheus based Usage Ping +### Usage Ping local setup + +To set up Usage Ping locally, you must: + +1. [Set up local repositories]#(set-up-local-repositories) +1. [Test local setup](#test-local-setup) +1. (Optional) [Test Prometheus-based usage ping](#test-prometheus-based-usage-ping) -If the data submitted includes metrics [queried from Prometheus](#prometheus-queries) that you would like to inspect and verify, -then you need to ensure that a Prometheus server is running locally, and that furthermore the respective GitLab components -are exporting metrics to it. If you do not need to test data coming from Prometheus, no further action +#### Set up local repositories + +1. Clone and start [GitLab](https://gitlab.com/gitlab-org/gitlab-development-kit). +1. Clone and start [Versions Application](https://gitlab.com/gitlab-services/version-gitlab-com). + Make sure to run `docker-compose up` to start a PostgreSQL and Redis instance. +1. Point GitLab to the Versions Application endpoint instead of the default endpoint: + 1. Open [submit_usage_ping_service.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/submit_usage_ping_service.rb#L4) in your local and modified `PRODUCTION_URL`. + 1. Set it to the local Versions Application URL `http://localhost:3000/usage_data`. + +#### Test local setup + +1. Using the `gitlab` Rails console, manually trigger a usage ping: + + ```ruby + SubmitUsagePingService.new.execute + ``` + +1. Use the `versions` Rails console to check the usage ping was successfully received, + parsed, and stored in the Versions database: + + ```ruby + UsageData.last + ``` + +### Test Prometheus-based usage ping + +If the data submitted includes metrics [queried from Prometheus](#prometheus-queries) +you want to inspect and verify, you must: + +- Ensure that a Prometheus server is running locally. +- Ensure the respective GitLab components are exporting metrics to the Prometheus server. + +If you do not need to test data coming from Prometheus, no further action is necessary. Usage Ping should degrade gracefully in the absence of a running Prometheus server. -There are three kinds of components that may export data to Prometheus, and which are included in Usage Ping: +Three kinds of components may export data to Prometheus, and are included in Usage Ping: -- [`node_exporter`](https://github.com/prometheus/node_exporter) - Exports node metrics from the host machine -- [`gitlab-exporter`](https://gitlab.com/gitlab-org/gitlab-exporter) - Exports process metrics from various GitLab components -- various GitLab services such as Sidekiq and the Rails server that export their own metrics +- [`node_exporter`](https://github.com/prometheus/node_exporter): Exports node metrics + from the host machine. +- [`gitlab-exporter`](https://gitlab.com/gitlab-org/gitlab-exporter): Exports process metrics + from various GitLab components. +- Other various GitLab services, such as Sidekiq and the Rails server, which export their own metrics. #### Test with an Omnibus container @@ -928,8 +1000,8 @@ appear to be associated to any of the services running, because they all appear WARNING: This feature is intended solely for internal GitLab use. -To add data for aggregated metrics into Usage Ping payload you should add corresponding definition at [`lib/gitlab/usage_data_counters/aggregated_metrics/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/aggregated_metrics/) for metrics available at Community Edition and at [`ee/lib/gitlab/usage_data_counters/aggregated_metrics/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/gitlab/usage_data_counters/aggregated_metrics/) for Enterprise Edition ones. - +To add data for aggregated metrics into Usage Ping payload you should add corresponding definition at [`config/metrics/aggregates/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/aggregates/) for metrics available at Community Edition and at [`ee/config/metrics/aggregates/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/aggregates/) for Enterprise Edition ones. + Each aggregate definition includes following parts: - `name`: Unique name under which the aggregate metric is added to the Usage Ping payload. @@ -957,7 +1029,10 @@ Example aggregated metric entries: ```yaml - name: example_metrics_union operator: OR - events: ['i_search_total', 'i_search_advanced', 'i_search_paid'] + events: + - 'i_search_total' + - 'i_search_advanced' + - 'i_search_paid' source: redis time_frame: - 7d @@ -968,7 +1043,9 @@ Example aggregated metric entries: time_frame: - 28d - all - events: ['dependency_scanning_pipeline_all_time', 'container_scanning_pipeline_all_time'] + events: + - 'dependency_scanning_pipeline_all_time' + - 'container_scanning_pipeline_all_time' feature_flag: example_aggregated_metric ``` @@ -993,7 +1070,7 @@ Aggregated metrics collected in `7d` and `28d` time frames are added into Usage } ``` -Aggregated metrics for `all` time frame are present in the `count` top level key, with the `aggregate_` prefix added to their name. +Aggregated metrics for `all` time frame are present in the `count` top level key, with the `aggregate_` prefix added to their name. For example: @@ -1001,7 +1078,7 @@ For example: Becomes: -`counts.aggregate_example_metrics_intersection` +`counts.aggregate_example_metrics_intersection` ```ruby { @@ -1085,7 +1162,7 @@ end #### Add new aggregated metric definition After all metrics are persisted, you can add an aggregated metric definition at -[`aggregated_metrics/`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/aggregated_metrics/). +[`aggregated_metrics/`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/aggregates/). To declare the aggregate of metrics collected with [Estimated Batch Counters](#estimated-batch-counters), you must fulfill the following requirements: @@ -1099,7 +1176,9 @@ Example definition: - name: example_metrics_intersection_database_sourced operator: AND source: database - events: ['dependency_scanning_pipeline', 'container_scanning_pipeline'] + events: + - 'dependency_scanning_pipeline' + - 'container_scanning_pipeline' time_frame: - 28d - all @@ -1334,4 +1413,24 @@ bin/rake gitlab:usage_data:dump_sql_in_yaml > ~/Desktop/usage-metrics-2020-09-02 ## Generating and troubleshooting usage ping -To get a usage ping, or to troubleshoot caching issues on your GitLab instance, please follow [instructions to generate usage ping](../../administration/troubleshooting/gitlab_rails_cheat_sheet.md#generate-usage-ping). +This activity is to be done via a detached screen session on a remote server. + +Before you begin these steps, make sure the key is added to the SSH agent locally +with the `ssh-add` command. + +### Triggering + +1. Connect to bastion with agent forwarding: `$ ssh -A lb-bastion.gprd.gitlab.com` +1. Create named screen: `$ screen -S <username>_usage_ping_<date>` +1. Connect to console host: `$ ssh $USER-rails@console-01-sv-gprd.c.gitlab-production.internal` +1. Run `SubmitUsagePingService.new.execute` +1. Detach from screen: `ctrl + a, ctrl + d` +1. Exit from bastion: `$ exit` + +### Verification (After approx 30 hours) + +1. Reconnect to bastion: `$ ssh -A lb-bastion.gprd.gitlab.com` +1. Find your screen session: `$ screen -ls` +1. Attach to your screen session: `$ screen -x 14226.mwawrzyniak_usage_ping_2021_01_22` +1. Check the last payload in `raw_usage_data` table: `RawUsageData.last.payload` +1. Check the when the payload was sent: `RawUsageData.last.sent_at` diff --git a/doc/development/usage_ping/metrics_dictionary.md b/doc/development/usage_ping/metrics_dictionary.md index 3c08fb0cc87..b55d4714580 100644 --- a/doc/development/usage_ping/metrics_dictionary.md +++ b/doc/development/usage_ping/metrics_dictionary.md @@ -27,13 +27,14 @@ Each metric is defined in a separate YAML file consisting of a number of fields: | Field | Required | Additional information | |---------------------|----------|----------------------------------------------------------------| | `key_path` | yes | JSON key path for the metric, location in Usage Ping payload. | +| `name` | no | Metric name suggestion. Can replace the last part of `key_path`. | | `description` | yes | | | `product_section` | yes | The [section](https://gitlab.com/gitlab-com/www-gitlab-com/-/blob/master/data/sections.yml). | | `product_stage` | no | The [stage](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/stages.yml) for the metric. | | `product_group` | yes | The [group](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/stages.yml) that owns the metric. | | `product_category` | no | The [product category](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/categories.yml) for the metric. | -| `value_type` | yes | `string`; one of `string`, `number`, `boolean`, `object`. | -| `status` | yes | `string`; status of the metric, may be set to `data_available`, `planned`, `in_progress`, `implemented`, `not_used`, `deprecated` | +| `value_type` | yes | `string`; one of [`string`, `number`, `boolean`, `object`](https://json-schema.org/understanding-json-schema/reference/type.html). | +| `status` | yes | `string`; [status](#metric-statuses) of the metric, may be set to `data_available`, `implemented`, `not_used`, `deprecated`, `removed`. | | `time_frame` | yes | `string`; may be set to a value like `7d`, `28d`, `all`, `none`. | | `data_source` | yes | `string`; may be set to a value like `database`, `redis`, `redis_hll`, `prometheus`, `ruby`. | | `distribution` | yes | `array`; may be set to one of `ce, ee` or `ee`. The [distribution](https://about.gitlab.com/handbook/marketing/strategic-marketing/tiers/#definitions) where the tracked feature is available. | @@ -43,6 +44,91 @@ Each metric is defined in a separate YAML file consisting of a number of fields: | `introduced_by_url` | no | The URL to the Merge Request that introduced the metric. | | `skip_validation` | no | This should **not** be set. [Used for imported metrics until we review, update and make them valid](https://gitlab.com/groups/gitlab-org/-/epics/5425). | +### Metric statuses + +Metric definitions can have one of the following statuses: + +- `data_available`: Metric data is available and used in a Sisense dashboard. +- `implemented`: Metric is implemented but data is not yet available. This is a temporary + status for newly added metrics awaiting inclusion in a new release. +- `not_used`: Metric is not used in any dashboard. +- `deprecated`: Metric is deprecated and possibly planned to be removed. +- `removed`: Metric was removed, but it may appear in Usage Ping payloads sent from instances running on older versions of GitLab. + +### Metric name + +To improve metric discoverability by a wider audience, each metric with +instrumentation added at an appointed `key_path` receives a `name` attribute +filled with the name suggestion, corresponding to the metric `data_source` and instrumentation. +Metric name suggestions can contain two types of elements: + +1. **User input prompts**: Enclosed by `<>`, these pieces should be replaced or + removed when you create a metrics YAML file. +1. **Fixed suggestion**: Plaintext parts generated according to well-defined algorithms. + They are based on underlying instrumentation, and should not be changed. + +For a metric name to be valid, it must not include any prompt, and no fixed suggestions +should be changed. + +### Metric name suggestion examples + +#### Metric with `data_source: database` + +For a metric instrumented with SQL: + +```sql +SELECT COUNT(DISTINCT user_id) FROM clusters WHERE clusters.management_project_id IS NOT NULL +``` + +- **Suggested name**: `count_distinct_user_id_from_<adjective describing: '(clusters.management_project_id IS NOT NULL)'>_clusters` +- **Prompt**: `<adjective describing: '(clusters.management_project_id IS NOT NULL)'>` + should be replaced with an adjective that best represents filter conditions, such as `project_management` +- **Final metric name**: For example, `count_distinct_user_id_from_project_management_clusters` + +For metric instrumented with SQL: + +```sql +SELECT COUNT(DISTINCT clusters.user_id) +FROM clusters_applications_helm +INNER JOIN clusters ON clusters.id = clusters_applications_helm.cluster_id +WHERE clusters_applications_helm.status IN (3, 5) +``` + +- **Suggested name**: `count_distinct_user_id_from_<adjective describing: '(clusters_applications_helm.status IN (3, 5))'>_clusters_<with>_<adjective describing: '(clusters_applications_helm.status IN (3, 5))'>_clusters_applications_helm` +- **Prompt**: `<adjective describing: '(clusters_applications_helm.status IN (3, 5))'>` + should be replaced with an adjective that best represents filter conditions +- **Final metric name**: `count_distinct_user_id_from_clusters_with_available_clusters_applications_helm` + +In the previous example, the prompt is irrelevant, and user can remove it. The second +occurrence corresponds with the `available` scope defined in `Clusters::Concerns::ApplicationStatus`. +It can be used as the right adjective to replace prompt. + +The `<with>` represents a suggested conjunction for the suggested name of the joined relation. +The person documenting the metric can use it by either: + +- Removing the surrounding `<>`. +- Using a different conjunction, such as `having` or `including`. + +#### Metric with `data_source: redis` or `redis_hll` + +For metrics instrumented with a Redis-based counter, the suggested name includes +only the single prompt to be replaced by the person working with metrics YAML. + +- **Prompt**: `<please fill metric name, suggested format is: {subject}_{verb}{ing|ed}_{object} eg: users_creating_epics or merge_requests_viewed_in_single_file_mode>` +- **Final metric name**: We suggest the metric name should follow the format of + `{subject}_{verb}{ing|ed}_{object}`, such as `user_creating_epics`, `users_triggering_security_scans`, + or `merge_requests_viewed_in_single_file_mode` + +#### Metric with `data_source: prometheus` or `ruby` + +For metrics instrumented with Prometheus or Ruby, the suggested name includes only +the single prompt by person working with metrics YAML. + +- **Prompt**: `<please fill metric name>` +- **Final metric name**: Due to the variety of cases that can apply to this kind of metric, + no naming convention exists. Each person instrumenting a metric should use their + best judgment to come up with a descriptive name. + ### Example YAML metric definition The linked [`uuid`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/license/uuid.yml) @@ -99,4 +185,12 @@ create ee/config/metrics/counts_7d/issues.yml The [Redis HLL metrics](index.md#known-events-are-added-automatically-in-usage-data-payload) are added automatically to Usage Ping payload. -A YAML metric definition is required for each metric. +A YAML metric definition is required for each metric. A dedicated generator is provided to create metric definitions for Redis HLL events. + +The generator takes `category` and `event` arguments, as the root key will be `redis_hll_counters`, and creates two metric definitions for weekly and monthly timeframes: + +```shell +bundle exec rails generate gitlab:usage_metric_definition:redis_hll issues i_closed +create config/metrics/counts_7d/i_closed_weekly.yml +create config/metrics/counts_28d/i_closed_monthly.yml +``` diff --git a/doc/development/usage_ping/product_intelligence_review.md b/doc/development/usage_ping/product_intelligence_review.md index c667bc8354c..3a8f9143b70 100644 --- a/doc/development/usage_ping/product_intelligence_review.md +++ b/doc/development/usage_ping/product_intelligence_review.md @@ -14,7 +14,7 @@ general best practices for code reviews, refer to our [code review guide](../cod ## Resources for Product Intelligence reviewers - [Usage Ping Guide](index.md) -- [Snowplow Guide](../snowplow.md) +- [Snowplow Guide](../snowplow/index.md) - [Metrics Dictionary](metrics_dictionary.md) ## Review process @@ -34,7 +34,7 @@ Product Intelligence files. ### Roles and process -The merge request **author** should: +#### The merge request **author** should - Decide whether a Product Intelligence review is needed. - If a Product Intelligence review is needed, add the labels @@ -48,7 +48,15 @@ The merge request **author** should: [Metrics Dictionary](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/usage_ping/dictionary.md) if it is needed. - Add a changelog [according to guidelines](../changelog.md). -The Product Intelligence **reviewer** should: +##### When adding or modifiying Snowplow events + +- For frontend events, when relevant, add a screenshot of the event in + the [testing tool](../snowplow/index.md#developing-and-testing-snowplow) used. +- For backend events, when relevant, add the output of the Snowplow Micro + good events `GET http://localhost:9090/micro/good` (it might be a good idea + to reset with `GET http://localhost:9090/micro/reset` first). + +#### The Product Intelligence **reviewer** should - Perform a first-pass review on the merge request and suggest improvements to the author. - Approve the MR, and relabel the MR with `~"product intelligence::approved"`. @@ -71,6 +79,9 @@ Any of the Product Intelligence engineers can be assigned for the Product Intell - For tracking using Redis HLL (HyperLogLog): - Check the Redis slot. - Check if a [feature flag is needed](index.md#recommendations). +- For tracking with Snowplow: + - Check that the [event taxonomy](../snowplow/index.md#structured-event-taxonomy) is correct. + - Check the [usage recomendations](../snowplow/index.md#usage-recommendations). - Metrics YAML definitions: - Check the metric `description`. - Check the metrics `key_path`. diff --git a/doc/development/what_requires_downtime.md b/doc/development/what_requires_downtime.md index 63a7ea4dfbd..7d20382973a 100644 --- a/doc/development/what_requires_downtime.md +++ b/doc/development/what_requires_downtime.md @@ -1,414 +1,8 @@ --- -stage: Enablement -group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: 'avoiding_downtime_in_migrations.md' --- -# What requires downtime? +This document was moved to [another location](avoiding_downtime_in_migrations.md). -When working with a database certain operations can be performed without taking -GitLab offline, others do require a downtime period. This guide describes -various operations, their impact, and how to perform them without requiring -downtime. - -## Dropping Columns - -Removing columns is tricky because running GitLab processes may still be using -the columns. To work around this safely, you will need three steps in three releases: - -1. Ignoring the column (release M) -1. Dropping the column (release M+1) -1. Removing the ignore rule (release M+2) - -The reason we spread this out across three releases is that dropping a column is -a destructive operation that can't be rolled back easily. - -Following this procedure helps us to make sure there are no deployments to GitLab.com -and upgrade processes for self-managed installations that lump together any of these steps. - -### Step 1: Ignoring the column (release M) - -The first step is to ignore the column in the application code. This is -necessary because Rails caches the columns and re-uses this cache in various -places. This can be done by defining the columns to ignore. For example, to ignore -`updated_at` in the User model you'd use the following: - -```ruby -class User < ApplicationRecord - include IgnorableColumns - ignore_column :updated_at, remove_with: '12.7', remove_after: '2020-01-22' -end -``` - -Multiple columns can be ignored, too: - -```ruby -ignore_columns %i[updated_at created_at], remove_with: '12.7', remove_after: '2020-01-22' -``` - -We require indication of when it is safe to remove the column ignore with: - -- `remove_with`: set to a GitLab release typically two releases (M+2) after adding the - column ignore. -- `remove_after`: set to a date after which we consider it safe to remove the column - ignore, typically after the M+1 release date, during the M+2 development cycle. - -This information allows us to reason better about column ignores and makes sure we -don't remove column ignores too early for both regular releases and deployments to GitLab.com. For -example, this avoids a situation where we deploy a bulk of changes that include both changes -to ignore the column and subsequently remove the column ignore (which would result in a downtime). - -In this example, the change to ignore the column went into release 12.5. - -### Step 2: Dropping the column (release M+1) - -Continuing our example, dropping the column goes into a _post-deployment_ migration in release 12.6: - -```ruby - remove_column :user, :updated_at -``` - -### Step 3: Removing the ignore rule (release M+2) - -With the next release, in this example 12.7, we set up another merge request to remove the ignore rule. -This removes the `ignore_column` line and - if not needed anymore - also the inclusion of `IgnoreableColumns`. - -This should only get merged with the release indicated with `remove_with` and once -the `remove_after` date has passed. - -## Renaming Columns - -Renaming columns the normal way requires downtime as an application may continue -using the old column name during/after a database migration. To rename a column -without requiring downtime we need two migrations: a regular migration, and a -post-deployment migration. Both these migration can go in the same release. - -### Step 1: Add The Regular Migration - -First we need to create the regular migration. This migration should use -`Gitlab::Database::MigrationHelpers#rename_column_concurrently` to perform the -renaming. For example - -```ruby -# A regular migration in db/migrate -class RenameUsersUpdatedAtToUpdatedAtTimestamp < ActiveRecord::Migration[4.2] - include Gitlab::Database::MigrationHelpers - - DOWNTIME = false - - disable_ddl_transaction! - - def up - rename_column_concurrently :users, :updated_at, :updated_at_timestamp - end - - def down - undo_rename_column_concurrently :users, :updated_at, :updated_at_timestamp - end -end -``` - -This will take care of renaming the column, ensuring data stays in sync, and -copying over indexes and foreign keys. - -If a column contains one or more indexes that don't contain the name of the -original column, the previously described procedure will fail. In that case, -you'll first need to rename these indexes. - -### Step 2: Add A Post-Deployment Migration - -The renaming procedure requires some cleaning up in a post-deployment migration. -We can perform this cleanup using -`Gitlab::Database::MigrationHelpers#cleanup_concurrent_column_rename`: - -```ruby -# A post-deployment migration in db/post_migrate -class CleanupUsersUpdatedAtRename < ActiveRecord::Migration[4.2] - include Gitlab::Database::MigrationHelpers - - disable_ddl_transaction! - - def up - cleanup_concurrent_column_rename :users, :updated_at, :updated_at_timestamp - end - - def down - undo_cleanup_concurrent_column_rename :users, :updated_at, :updated_at_timestamp - end -end -``` - -If you're renaming a [large table](https://gitlab.com/gitlab-org/gitlab/-/blob/master/rubocop/rubocop-migrations.yml#L3), please carefully consider the state when the first migration has run but the second cleanup migration hasn't been run yet. -With [Canary](https://gitlab.com/gitlab-com/gl-infra/readiness/-/tree/master/library/canary/) it is possible that the system runs in this state for a significant amount of time. - -## Changing Column Constraints - -Adding or removing a `NOT NULL` clause (or another constraint) can typically be -done without requiring downtime. However, this does require that any application -changes are deployed _first_. Thus, changing the constraints of a column should -happen in a post-deployment migration. - -Avoid using `change_column` as it produces an inefficient query because it re-defines -the whole column type. - -You can check the following guides for each specific use case: - -- [Adding foreign-key constraints](migration_style_guide.md#adding-foreign-key-constraints) -- [Adding `NOT NULL` constraints](database/not_null_constraints.md) -- [Adding limits to text columns](database/strings_and_the_text_data_type.md) - -## Changing Column Types - -Changing the type of a column can be done using -`Gitlab::Database::MigrationHelpers#change_column_type_concurrently`. This -method works similarly to `rename_column_concurrently`. For example, let's say -we want to change the type of `users.username` from `string` to `text`. - -### Step 1: Create A Regular Migration - -A regular migration is used to create a new column with a temporary name along -with setting up some triggers to keep data in sync. Such a migration would look -as follows: - -```ruby -# A regular migration in db/migrate -class ChangeUsersUsernameStringToText < ActiveRecord::Migration[4.2] - include Gitlab::Database::MigrationHelpers - - disable_ddl_transaction! - - def up - change_column_type_concurrently :users, :username, :text - end - - def down - undo_change_column_type_concurrently :users, :username - end -end -``` - -### Step 2: Create A Post Deployment Migration - -Next we need to clean up our changes using a post-deployment migration: - -```ruby -# A post-deployment migration in db/post_migrate -class ChangeUsersUsernameStringToTextCleanup < ActiveRecord::Migration[4.2] - include Gitlab::Database::MigrationHelpers - - disable_ddl_transaction! - - def up - cleanup_concurrent_column_type_change :users, :username - end - - def down - undo_cleanup_concurrent_column_type_change :users, :username, :string - end -end -``` - -And that's it, we're done! - -### Casting data to a new type - -Some type changes require casting data to a new type. For example when changing from `text` to `jsonb`. -In this case, use the `type_cast_function` option. -Make sure there is no bad data and the cast will always succeed. You can also provide a custom function that handles -casting errors. - -Example migration: - -```ruby - def up - change_column_type_concurrently :users, :settings, :jsonb, type_cast_function: 'jsonb' - end -``` - -## Changing The Schema For Large Tables - -While `change_column_type_concurrently` and `rename_column_concurrently` can be -used for changing the schema of a table without downtime, it doesn't work very -well for large tables. Because all of the work happens in sequence the migration -can take a very long time to complete, preventing a deployment from proceeding. -They can also produce a lot of pressure on the database due to it rapidly -updating many rows in sequence. - -To reduce database pressure you should instead use -`change_column_type_using_background_migration` or `rename_column_using_background_migration` -when migrating a column in a large table (e.g. `issues`). These methods work -similarly to the concurrent counterparts but uses background migration to spread -the work / load over a longer time period, without slowing down deployments. - -For example, to change the column type using a background migration: - -```ruby -class ExampleMigration < ActiveRecord::Migration[4.2] - include Gitlab::Database::MigrationHelpers - - disable_ddl_transaction! - - class Issue < ActiveRecord::Base - self.table_name = 'issues' - - include EachBatch - - def self.to_migrate - where('closed_at IS NOT NULL') - end - end - - def up - change_column_type_using_background_migration( - Issue.to_migrate, - :closed_at, - :datetime_with_timezone - ) - end - - def down - change_column_type_using_background_migration( - Issue.to_migrate, - :closed_at, - :datetime - ) - end -end -``` - -This would change the type of `issues.closed_at` to `timestamp with time zone`. - -Keep in mind that the relation passed to -`change_column_type_using_background_migration` _must_ include `EachBatch`, -otherwise it will raise a `TypeError`. - -This migration then needs to be followed in a separate release (_not_ a patch -release) by a cleanup migration, which should steal from the queue and handle -any remaining rows. For example: - -```ruby -class MigrateRemainingIssuesClosedAt < ActiveRecord::Migration[4.2] - include Gitlab::Database::MigrationHelpers - - DOWNTIME = false - - disable_ddl_transaction! - - class Issue < ActiveRecord::Base - self.table_name = 'issues' - include EachBatch - end - - def up - Gitlab::BackgroundMigration.steal('CopyColumn') - Gitlab::BackgroundMigration.steal('CleanupConcurrentTypeChange') - - migrate_remaining_rows if migrate_column_type? - end - - def down - # Previous migrations already revert the changes made here. - end - - def migrate_remaining_rows - Issue.where('closed_at_for_type_change IS NULL AND closed_at IS NOT NULL').each_batch do |batch| - batch.update_all('closed_at_for_type_change = closed_at') - end - - cleanup_concurrent_column_type_change(:issues, :closed_at) - end - - def migrate_column_type? - # Some environments may have already executed the previous version of this - # migration, thus we don't need to migrate those environments again. - column_for('issues', 'closed_at').type == :datetime # rubocop:disable Migration/Datetime - end -end -``` - -The same applies to `rename_column_using_background_migration`: - -1. Create a migration using the helper, which will schedule background - migrations to spread the writes over a longer period of time. -1. In the next monthly release, create a clean-up migration to steal from the - Sidekiq queues, migrate any missing rows, and cleanup the rename. This - migration should skip the steps after stealing from the Sidekiq queues if the - column has already been renamed. - -For more information, see [the documentation on cleaning up background -migrations](background_migrations.md#cleaning-up). - -## Adding Indexes - -Adding indexes does not require downtime when `add_concurrent_index` -is used. - -See also [Migration Style Guide](migration_style_guide.md#adding-indexes) -for more information. - -## Dropping Indexes - -Dropping an index does not require downtime. - -## Adding Tables - -This operation is safe as there's no code using the table just yet. - -## Dropping Tables - -Dropping tables can be done safely using a post-deployment migration, but only -if the application no longer uses the table. - -## Renaming Tables - -Renaming tables requires downtime as an application may continue -using the old table name during/after a database migration. - -## Adding Foreign Keys - -Adding foreign keys usually works in 3 steps: - -1. Start a transaction -1. Run `ALTER TABLE` to add the constraint(s) -1. Check all existing data - -Because `ALTER TABLE` typically acquires an exclusive lock until the end of a -transaction this means this approach would require downtime. - -GitLab allows you to work around this by using -`Gitlab::Database::MigrationHelpers#add_concurrent_foreign_key`. This method -ensures that no downtime is needed. - -## Removing Foreign Keys - -This operation does not require downtime. - -## Data Migrations - -Data migrations can be tricky. The usual approach to migrate data is to take a 3 -step approach: - -1. Migrate the initial batch of data -1. Deploy the application code -1. Migrate any remaining data - -Usually this works, but not always. For example, if a field's format is to be -changed from JSON to something else we have a bit of a problem. If we were to -change existing data before deploying application code we'll most likely run -into errors. On the other hand, if we were to migrate after deploying the -application code we could run into the same problems. - -If you merely need to correct some invalid data, then a post-deployment -migration is usually enough. If you need to change the format of data (e.g. from -JSON to something else) it's typically best to add a new column for the new data -format, and have the application use that. In such a case the procedure would -be: - -1. Add a new column in the new format -1. Copy over existing data to this new column -1. Deploy the application code -1. In a post-deployment migration, copy over any remaining data - -In general there is no one-size-fits-all solution, therefore it's best to -discuss these kind of migrations in a merge request to make sure they are -implemented in the best way possible. +<!-- This redirect file can be deleted after <2021-07-01>. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/wikis.md b/doc/development/wikis.md index f47c87137ae..9998e29b596 100644 --- a/doc/development/wikis.md +++ b/doc/development/wikis.md @@ -19,7 +19,7 @@ Wikis use Git repositories as storage backend, and can be accessed through: - The [Web UI](../user/project/wiki/index.md) - The [REST API](../api/wikis.md) -- [Git itself](../user/project/wiki/#adding-and-editing-wiki-pages-locally) +- [Git itself](../user/project/wiki/index.md#create-or-edit-wiki-pages-locally) [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2214) in GitLab 13.5, wikis are also available for groups, in addition to projects. @@ -92,3 +92,7 @@ Only some data is persisted in the database: The web UI uploads attachments through the REST API, which stores the files as commits in the wiki repository. Prior to GitLab 11.3 attachments were stored outside of the repository, [see this issue](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/33475). + +## Related topics + +- [Gollum installation instructions](https://github.com/gollum/gollum/wiki/Installation) |
