diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2021-09-20 13:18:24 +0000 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2021-09-20 13:18:24 +0000 |
| commit | 0653e08efd039a5905f3fa4f6e9cef9f5d2f799c (patch) | |
| tree | 4dcc884cf6d81db44adae4aa99f8ec1233a41f55 /doc/development | |
| parent | 744144d28e3e7fddc117924fef88de5d9674fe4c (diff) | |
| download | gitlab-ce-0653e08efd039a5905f3fa4f6e9cef9f5d2f799c.tar.gz | |
Add latest changes from gitlab-org/gitlab@14-3-stable-eev14.3.0-rc42
Diffstat (limited to 'doc/development')
101 files changed, 4075 insertions, 2097 deletions
diff --git a/doc/development/adding_service_component.md b/doc/development/adding_service_component.md index 503d1b7e55b..f5acf0d26eb 100644 --- a/doc/development/adding_service_component.md +++ b/doc/development/adding_service_component.md @@ -47,8 +47,7 @@ Adding a new service follows the same [merge request workflow](contributing/merg The first iteration should be to add the ability to connect and use the service as an externally installed component. Often this involves providing settings in GitLab to connect to the service, or allow connections from it. And then shipping documentation on how to install and configure the service with GitLab. -NOTE: -[Elasticsearch](../integration/elasticsearch.md#installing-elasticsearch) is an example of a service that has been integrated this way. And many of the other services, including internal projects like Gitaly, started off as separately installed alternatives. +[Elasticsearch](../integration/elasticsearch.md#install-elasticsearch) is an example of a service that has been integrated this way. Many of the other services, including internal projects like Gitaly, started off as separately installed alternatives. **For services that depend on the existing GitLab codebase:** diff --git a/doc/development/api_graphql_styleguide.md b/doc/development/api_graphql_styleguide.md index 40cc8f5ec45..9a17ac4c813 100644 --- a/doc/development/api_graphql_styleguide.md +++ b/doc/development/api_graphql_styleguide.md @@ -532,7 +532,7 @@ Example: ```ruby field :foo, GraphQL::Types::String, null: true, - description: 'Some test field. Will always return `null`' \ + description: 'Some test field. Returns `null`' \ 'if `my_feature_flag` feature flag is disabled.' def foo @@ -940,7 +940,9 @@ class PostResolver < BaseResolver end ``` -You should never re-use resolvers directly. Resolvers have a complex life-cycle, with +While you can use the same resolver class in two different places, +such as in two different fields where the same object is exposed, +you should never re-use resolver objects directly. Resolvers have a complex life-cycle, with authorization, readiness and resolution orchestrated by the framework, and at each stage [lazy values](#laziness) can be returned to take advantage of batching opportunities. Never instantiate a resolver or a mutation in application code. diff --git a/doc/development/application_limits.md b/doc/development/application_limits.md index b606cda1124..2075e7cda3c 100644 --- a/doc/development/application_limits.md +++ b/doc/development/application_limits.md @@ -40,9 +40,7 @@ It's recommended to create two separate migration script files. desired limit using `create_or_update_plan_limit` migration helper, such as: ```ruby - class InsertProjectHooksPlanLimits < ActiveRecord::Migration[5.2] - include Gitlab::Database::MigrationHelpers - + class InsertProjectHooksPlanLimits < Gitlab::Database::Migration[1.0] def up create_or_update_plan_limit('project_hooks', 'default', 0) create_or_update_plan_limit('project_hooks', 'free', 10) diff --git a/doc/development/architecture.md b/doc/development/architecture.md index a487e84d090..fe2b621da29 100644 --- a/doc/development/architecture.md +++ b/doc/development/architecture.md @@ -603,14 +603,14 @@ For monitoring deployed apps, see [Jaeger tracing documentation](../operations/t - Layer: Core Service - Process: `logrotate` -GitLab is comprised of a large number of services that all log. We started bundling our own Logrotate -as of GitLab 7.4 to make sure we were logging responsibly. This is just a packaged version of the common open source offering. +GitLab is comprised of a large number of services that all log. We bundle our own Logrotate +to make sure we were logging responsibly. This is just a packaged version of the common open source offering. #### Mattermost - [Project page](https://github.com/mattermost/mattermost-server/blob/master/README.md) - Configuration: - - [Omnibus](https://docs.gitlab.com/omnibus/gitlab-mattermost/) + - [Omnibus](../integration/mattermost/index.md) - [Charts](https://docs.mattermost.com/install/install-mmte-helm-gitlab-helm.html) - Layer: Core Service (Processor) - GitLab.com: [Mattermost](../user/project/integrations/mattermost.md) diff --git a/doc/development/avoiding_downtime_in_migrations.md b/doc/development/avoiding_downtime_in_migrations.md index b844415c94e..9418eafa487 100644 --- a/doc/development/avoiding_downtime_in_migrations.md +++ b/doc/development/avoiding_downtime_in_migrations.md @@ -95,9 +95,7 @@ renaming. For example ```ruby # A regular migration in db/migrate -class RenameUsersUpdatedAtToUpdatedAtTimestamp < ActiveRecord::Migration[4.2] - include Gitlab::Database::MigrationHelpers - +class RenameUsersUpdatedAtToUpdatedAtTimestamp < Gitlab::Database::Migration[1.0] disable_ddl_transaction! def up @@ -125,9 +123,7 @@ We can perform this cleanup using ```ruby # A post-deployment migration in db/post_migrate -class CleanupUsersUpdatedAtRename < ActiveRecord::Migration[4.2] - include Gitlab::Database::MigrationHelpers - +class CleanupUsersUpdatedAtRename < Gitlab::Database::Migration[1.0] disable_ddl_transaction! def up @@ -174,9 +170,7 @@ as follows: ```ruby # A regular migration in db/migrate -class ChangeUsersUsernameStringToText < ActiveRecord::Migration[4.2] - include Gitlab::Database::MigrationHelpers - +class ChangeUsersUsernameStringToText < Gitlab::Database::Migration[1.0] disable_ddl_transaction! def up @@ -195,9 +189,7 @@ Next we need to clean up our changes using a post-deployment migration: ```ruby # A post-deployment migration in db/post_migrate -class ChangeUsersUsernameStringToTextCleanup < ActiveRecord::Migration[4.2] - include Gitlab::Database::MigrationHelpers - +class ChangeUsersUsernameStringToTextCleanup < Gitlab::Database::Migration[1.0] disable_ddl_transaction! def up @@ -245,9 +237,7 @@ the work / load over a longer time period, without slowing down deployments. For example, to change the column type using a background migration: ```ruby -class ExampleMigration < ActiveRecord::Migration[4.2] - include Gitlab::Database::MigrationHelpers - +class ExampleMigration < Gitlab::Database::Migration[1.0] disable_ddl_transaction! class Issue < ActiveRecord::Base @@ -289,9 +279,7 @@ release) by a cleanup migration, which should steal from the queue and handle any remaining rows. For example: ```ruby -class MigrateRemainingIssuesClosedAt < ActiveRecord::Migration[4.2] - include Gitlab::Database::MigrationHelpers - +class MigrateRemainingIssuesClosedAt < Gitlab::Database::Migration[1.0] disable_ddl_transaction! class Issue < ActiveRecord::Base diff --git a/doc/development/background_migrations.md b/doc/development/background_migrations.md index 695c565ca83..c93b5b448f0 100644 --- a/doc/development/background_migrations.md +++ b/doc/development/background_migrations.md @@ -254,7 +254,7 @@ existing data. Since we're dealing with a lot of rows we'll schedule jobs in batches instead of doing this one by one: ```ruby -class ScheduleExtractServicesUrl < ActiveRecord::Migration[4.2] +class ScheduleExtractServicesUrl < Gitlab::Database::Migration[1.0] disable_ddl_transaction! def up @@ -281,7 +281,7 @@ jobs and manually run on any un-migrated rows. Such a migration would look like this: ```ruby -class ConsumeRemainingExtractServicesUrlJobs < ActiveRecord::Migration[4.2] +class ConsumeRemainingExtractServicesUrlJobs < Gitlab::Database::Migration[1.0] disable_ddl_transaction! def up @@ -463,8 +463,6 @@ end ```ruby # Post deployment migration -include Gitlab::Database::MigrationHelpers - MIGRATION = 'YourBackgroundMigrationName' DELAY_INTERVAL = 2.minutes.to_i # can be different BATCH_SIZE = 10_000 # can be different @@ -494,8 +492,6 @@ You can reschedule pending migrations from the `background_migration_jobs` table ```ruby # Post deployment migration -include Gitlab::Database::MigrationHelpers - MIGRATION = 'YourBackgroundMigrationName' DELAY_INTERVAL = 2.minutes @@ -511,3 +507,16 @@ end ``` See [`db/post_migrate/20210604070207_retry_backfill_traversal_ids.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/db/post_migrate/20210604070207_retry_backfill_traversal_ids.rb) for a full example. + +### Viewing failure error logs + +After running a background migration, if any jobs have failed, you can view the logs in [Kibana](https://log.gprd.gitlab.net/goto/3afc1393447c401d7602c1874793e2f6). +View the production Sidekiq log and filter for: + +- `json.class: BackgroundMigrationWorker` +- `json.job_status: fail` +- `json.args: <MyBackgroundMigrationClassName>` + +Looking at the `json.error_class`, `json.error_message` and `json.error_backtrace` values may be helpful in understanding why the jobs failed. + +Depending on when and how the failure occurred, you may find other helpful information by filtering with `json.class: <MyBackgroundMigrationClassName>`. diff --git a/doc/development/cascading_settings.md b/doc/development/cascading_settings.md index d1c5756fa2c..0fa0e220ba9 100644 --- a/doc/development/cascading_settings.md +++ b/doc/development/cascading_settings.md @@ -38,9 +38,11 @@ Settings are not cascading by default. To define a cascading setting, take the f `application_settings`. ```ruby - class AddDelayedProjectRemovalCascadingSetting < ActiveRecord::Migration[6.0] + class AddDelayedProjectRemovalCascadingSetting < Gitlab::Database::Migration[1.0] include Gitlab::Database::MigrationHelpers::CascadingNamespaceSettings + enable_lock_retries! + def up add_cascading_namespace_setting :delayed_project_removal, :boolean, default: false, null: false end diff --git a/doc/development/changelog.md b/doc/development/changelog.md index c96fe2c18c1..be46d61eb4c 100644 --- a/doc/development/changelog.md +++ b/doc/development/changelog.md @@ -98,6 +98,7 @@ EE: true database records created during Cycle Analytics model spec." - _Any_ contribution from a community member, no matter how small, **may** have a changelog entry regardless of these guidelines if the contributor wants one. +- Any [GLEX experiment](experiment_guide/gitlab_experiment.md) changes **should not** have a changelog entry. - [Removing](feature_flags/#changelog) a feature flag, when the new code is retained. ## Writing good changelog entries diff --git a/doc/development/cicd/cicd_reference_documentation_guide.md b/doc/development/cicd/cicd_reference_documentation_guide.md index 33bc416d8bc..aa3888cd866 100644 --- a/doc/development/cicd/cicd_reference_documentation_guide.md +++ b/doc/development/cicd/cicd_reference_documentation_guide.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Pipeline Execution +group: Pipeline Authoring info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/cicd/index.md b/doc/development/cicd/index.md index 76c756b0e95..b4e32066ba8 100644 --- a/doc/development/cicd/index.md +++ b/doc/development/cicd/index.md @@ -16,7 +16,7 @@ to learn how to update the [reference page](../../ci/yaml/index.md). ## CI Architecture overview -The following is a simplified diagram of the CI architecture. Some details are left out in order to focus on +The following is a simplified diagram of the CI architecture. Some details are left out to focus on the main components.  @@ -73,7 +73,7 @@ which picks the next job and assigns it to the runner. At this point the job tra For more details read [Job scheduling](#job-scheduling)). While a job is being executed, the runner sends logs back to the server as well any possible artifacts -that need to be stored. Also, a job may depend on artifacts from previous jobs in order to run. In this +that must be stored. Also, a job may depend on artifacts from previous jobs to run. In this case the runner downloads them using a dedicated API endpoint. Artifacts are stored in object storage, while metadata is kept in the database. An important example of artifacts @@ -111,7 +111,7 @@ Once all jobs are completed for the current stage, the server "unlocks" all the ### Communication between runner and GitLab server -Once the runner is [registered](https://docs.gitlab.com/runner/register/) using the registration token, the server knows what type of jobs it can execute. This depends on: +After the runner is [registered](https://docs.gitlab.com/runner/register/) using the registration token, the server knows what type of jobs it can execute. This depends on: - The type of runner it is registered as: - a shared runner @@ -119,7 +119,7 @@ Once the runner is [registered](https://docs.gitlab.com/runner/register/) using - a project specific runner - Any associated tags. -The runner initiates the communication by requesting jobs to execute with `POST /api/v4/jobs/request`. Although this polling generally happens every few seconds we leverage caching via HTTP headers to reduce the server-side work load if the job queue doesn't change. +The runner initiates the communication by requesting jobs to execute with `POST /api/v4/jobs/request`. Although polling happens every few seconds, we leverage caching through HTTP headers to reduce the server-side work load if the job queue doesn't change. This API endpoint runs [`Ci::RegisterJobService`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/ci/register_job_service.rb), which: diff --git a/doc/development/cicd/templates.md b/doc/development/cicd/templates.md index 03823a4b712..3fc464e661f 100644 --- a/doc/development/cicd/templates.md +++ b/doc/development/cicd/templates.md @@ -178,10 +178,10 @@ This includes: - Repository/project requirements. - Expected behavior. -- Any places that need to be edited by users before using the template. +- Any places that must be edited by users before using the template. - If the template should be used by copy pasting it into a configuration file, or by using it with the `include` keyword in an existing pipeline. -- If any variables need to be saved in the project's CI/CD settings. +- If any variables must be saved in the project's CI/CD settings. ```yaml # Use this template to publish an application that uses the ABC server. @@ -289,9 +289,9 @@ If the `latest` template does not exist yet, you can copy [the stable template]( ### How to include an older stable template Users may want to use an older [stable template](#stable-version) that is not bundled -in the current GitLab package. For example, the stable templates in GitLab v13.0 and -GitLab v14.0 could be so different that a user wants to continue using the v13.0 template even -after upgrading to GitLab 14.0. +in the current GitLab package. For example, the stable templates in GitLab 13.0 and +GitLab 14.0 could be so different that a user wants to continue using the GitLab 13.0 +template even after upgrading to GitLab 14.0. You can add a note in the template or in documentation explaining how to use `include:remote` to include older template versions. If other templates are included with `include: template`, @@ -335,7 +335,7 @@ follow the progress. ## Testing -Each CI/CD template must be tested in order to make sure that it's safe to be published. +Each CI/CD template must be tested to make sure that it's safe to be published. ### Manual QA @@ -380,7 +380,7 @@ is updated in a major version GitLab release. A template could contain malicious code. For example, a template that contains the `export` shell command in a job might accidentally expose secret project CI/CD variables in a job log. -If you're unsure if it's secure or not, you need to ask security experts for cross-validation. +If you're unsure if it's secure or not, you must ask security experts for cross-validation. ## Contribute CI/CD template merge requests diff --git a/doc/development/code_intelligence/index.md b/doc/development/code_intelligence/index.md index 790ba1539b7..e1e2105298c 100644 --- a/doc/development/code_intelligence/index.md +++ b/doc/development/code_intelligence/index.md @@ -37,7 +37,7 @@ sequenceDiagram 1. The CI/CD job generates a document in an LSIF format (usually `dump.lsif`) using [an indexer](https://lsif.dev) for the language of a project. The format - [describes](https://github.com/sourcegraph/sourcegraph/blob/master/doc/user/code_intelligence/writing_an_indexer.md) + [describes](https://github.com/sourcegraph/sourcegraph/blob/main/doc/code_intelligence/explanations/writing_an_indexer.md) interactions between a method or function and its definition(s) or references. The document is marked to be stored as an LSIF report artifact. diff --git a/doc/development/code_review.md b/doc/development/code_review.md index d66f246ac8c..12cc63ef56d 100644 --- a/doc/development/code_review.md +++ b/doc/development/code_review.md @@ -10,7 +10,7 @@ This guide contains advice and best practices for performing code review, and having your code reviewed. All merge requests for GitLab CE and EE, whether written by a GitLab team member -or a volunteer contributor, must go through a code review process to ensure the +or a wider community member, must go through a code review process to ensure the code is effective, understandable, maintainable, and secure. ## Getting your merge request reviewed, approved, and merged @@ -35,7 +35,7 @@ If you need assistance with security scans or comments, feel free to include the Application Security Team (`@gitlab-com/gl-security/appsec`) in the review. Depending on the areas your merge request touches, it must be **approved** by one -or more [maintainers](https://about.gitlab.com/handbook/engineering/workflow/code-review/#maintainer): +or more [maintainers](https://about.gitlab.com/handbook/engineering/workflow/code-review/#maintainer). For approvals, we use the approval functionality found in the merge request widget. For reviewers, we use the [reviewer functionality](../user/project/merge_requests/getting_started.md#reviewer) in the sidebar. @@ -46,9 +46,11 @@ more than one approval, the last maintainer to review and approve merges it. ### Domain experts -Domain experts are team members who have substantial experience with a specific technology, product feature or area of the codebase. Team members are encouraged to self-identify as domain experts and add it to their [team profile](https://gitlab.com/gitlab-com/www-gitlab-com/-/blob/master/data/team.yml). +Domain experts are team members who have substantial experience with a specific technology, +product feature, or area of the codebase. Team members are encouraged to self-identify as +domain experts and add it to their [team profiles](https://gitlab.com/gitlab-com/www-gitlab-com/-/blob/master/data/team_members/person/README.md). -When self-identifying as a domain expert, it is recommended to assign the MR changing the `team.yml` to be merged by an already established Domain Expert or a corresponding Engineering Manager. +When self-identifying as a domain expert, it is recommended to assign the MR changing the `.yml` file to be merged by an already established Domain Expert or a corresponding Engineering Manager. We make the following assumption with regards to automatically being considered a domain expert: @@ -107,7 +109,7 @@ with [domain expertise](#domain-experts). 1. If your merge request includes adding a new JavaScript library (*1*)... - If the library significantly increases the [bundle size](https://gitlab.com/gitlab-org/frontend/playground/webpack-memory-metrics/-/blob/master/doc/report.md), it must - be **approved by a [frontend foundations member](https://about.gitlab.com/direction/create/ecosystem/frontend-ux-foundations/)**. + be **approved by a [frontend foundations member](https://about.gitlab.com/direction/ecosystem/foundations/)**. - If the license used by the new library hasn't been approved for use in GitLab, the license must be **approved by a [legal department member](https://about.gitlab.com/handbook/legal/)**. More information about license compatibility can be found in our @@ -117,11 +119,12 @@ with [domain expertise](#domain-experts). 1. If your merge request includes documentation changes, it must be **approved by a [Technical writer](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments)**, based on assignments in the appropriate [DevOps stage group](https://about.gitlab.com/handbook/product/categories/#devops-stages). +1. If your merge request includes changes to development guidelines, follow the [review process](index.md#development-guidelines-review) and get the approvals accordingly. 1. If your merge request includes end-to-end **and** non-end-to-end changes (*4*), it must be **approved by a [Software Engineer in Test](https://about.gitlab.com/handbook/engineering/quality/#individual-contributors)**. 1. If your merge request only includes end-to-end changes (*4*) **or** if the MR author is a [Software Engineer in Test](https://about.gitlab.com/handbook/engineering/quality/#individual-contributors), it must be **approved by a [Quality maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_maintainers_qa)** 1. If your merge request includes a new or updated [application limit](https://about.gitlab.com/handbook/product/product-processes/#introducing-application-limits), it must be **approved by a [product manager](https://about.gitlab.com/company/team/)**. -1. If your merge request includes Product Intelligence (telemetry or analytics) changes, it should be reviewed and approved by a [Product Intelligence engineer](https://gitlab.com/gitlab-org/growth/product_intelligence/engineers). +1. If your merge request includes Product Intelligence (telemetry or analytics) changes, it should be reviewed and approved by a [Product Intelligence engineer](https://gitlab.com/gitlab-org/growth/product-intelligence/engineers). 1. If your merge request includes an addition of, or changes to a [Feature spec](testing_guide/testing_levels.md#frontend-feature-tests), it must be **approved by a [Quality maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_maintainers_qa) or [Quality reviewer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_reviewers_qa)**. 1. If your merge request introduces a new service to GitLab (Puma, Sidekiq, Gitaly are examples), it must be **approved by a [product manager](https://about.gitlab.com/company/team/)**. See the [process for adding a service component to GitLab](adding_service_component.md) for details. @@ -134,15 +137,60 @@ with [domain expertise](#domain-experts). the content. - (*4*): End-to-end changes include all files within the `qa` directory. -#### Security requirements +#### Acceptance checklist -View the updated documentation regarding [internal application security reviews](https://about.gitlab.com/handbook/engineering/security/#internal-application-security-reviews) for **when** and **how** to request a security review. +This checklist encourages the authors, reviewers, and maintainers of merge requests (MRs) to confirm changes were analyzed for high-impact risks to quality, performance, reliability, security, and maintainability. + +Using checklists improves quality in software engineering. This checklist is a straightforward tool to support and bolster the skills of contributors to the GitLab codebase. + +##### Quality + +See the [test engineering process](https://about.gitlab.com/handbook/engineering/quality/test-engineering/) for further quality guidelines. + +1. I have self-reviewed this MR per [code review guidelines](code_review.md). +1. For the code that this change impacts, I believe that the automated tests ([Testing Guide](testing_guide/index.md)) validate functionality that is highly important to users (including consideration of [all test levels](testing_guide/testing_levels.md)). +1. If the existing automated tests do not cover the above functionality, I have added the necessary additional tests or added an issue to describe the automation testing gap and linked it to this MR. +1. I have considered the technical aspects of this change's impact on GitLab.com hosted customers and self-managed customers. +1. I have considered the impact of this change on the frontend, backend, and database portions of the system where appropriate and applied the `~frontend`, `~backend`, and `~database` labels accordingly. +1. I have tested this MR in [all supported browsers](../install/requirements.md#supported-web-browsers), or determined that this testing is not needed. +1. I have confirmed that this change is [backwards compatible across updates](multi_version_compatibility.md), or I have decided that this does not apply. +1. I have properly separated EE content from FOSS, or this MR is FOSS only. + - [Where should EE code go?](ee_features.md#separation-of-ee-code) +1. If I am introducing a new expectation for existing data, I have confirmed that existing data meets this expectation or I have made this expectation optional rather than required. + +##### Performance, reliability, and availability + +1. I am confident that this MR does not harm performance, or I have asked a reviewer to help assess the performance impact. ([Merge request performance guidelines](merge_request_performance_guidelines.md)) +1. I have added [information for database reviewers in the MR description](database_review.md#required), or I have decided that it is unnecessary. + - [Does this MR have database-related changes?](database_review.md) +1. I have considered the availability and reliability risks of this change. +1. I have considered the scalability risk based on future predicted growth. +1. I have considered the performance, reliability, and availability impacts of this change on large customers who may have significantly more data than the average customer. + +##### Documentation + +1. I have included changelog trailers, or I have decided that they are not needed. + - [Does this MR need a changelog?](changelog.md#what-warrants-a-changelog-entry) +1. I have added/updated documentation or decided that documentation changes are unnecessary for this MR. + - [Is documentation required?](https://about.gitlab.com/handbook/engineering/ux/technical-writing/workflow/#when-documentation-is-required) + +##### Security + +1. I have confirmed that if this MR contains changes to processing or storing of credentials or tokens, authorization, and authentication methods, or other items described in [the security review guidelines](https://about.gitlab.com/handbook/engineering/security/#when-to-request-a-security-review), I have added the `~security` label and I have `@`-mentioned `@gitlab-com/gl-security/appsec`. +1. I have reviewed the documentation regarding [internal application security reviews](https://about.gitlab.com/handbook/engineering/security/#internal-application-security-reviews) for **when** and **how** to request a security review and requested a security review if this is warranted for this change. + +##### Deployment + +1. I have considered using a feature flag for this change because the change may be high risk. +1. If I am using a feature flag, I plan to test the change in staging before I test it in production, and I have considered rolling it out to a subset of production customers before rolling it out to all customers. + - [When to use a feature flag](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#when-to-use-feature-flags) +1. I have informed the Infrastructure department of a default setting or new setting change per [definition of done](contributing/merge_request_workflow.md#definition-of-done), or decided that this is unnecessary. ### The responsibility of the merge request author The responsibility to find the best solution and implement it lies with the merge request author. The author or [directly responsible individual](https://about.gitlab.com/handbook/people-group/directly-responsible-individuals/) -will stay assigned to the merge request as the assignee throughout +(DRI) stays assigned to the merge request as the assignee throughout the code review lifecycle. If you are unable to set yourself as an assignee, ask a [reviewer](https://about.gitlab.com/handbook/engineering/workflow/code-review/#reviewer) to do this for you. Before requesting a review from a maintainer to approve and merge, they @@ -169,7 +217,7 @@ database specialists to get input on the data model or specific queries, or to any other developer to get an in-depth review of the solution. If an author is unsure if a merge request needs a [domain expert's](#domain-experts) opinion, -that indicates it does. Without it it's unlikely they have the required level of confidence in their +then that indicates it does. Without it, it's unlikely they have the required level of confidence in their solution. Before the review, the author is requested to submit comments on the merge @@ -249,7 +297,7 @@ without duly verifying them. Note that certain Merge Requests may target a stable branch. These are rare events. These types of Merge Requests cannot be merged by the Maintainer. -Instead these should be sent to the [Release Manager](https://about.gitlab.com/community/release-managers/). +Instead, these should be sent to the [Release Manager](https://about.gitlab.com/community/release-managers/). After merging, a maintainer should stay as the reviewer listed on the merge request. @@ -305,8 +353,8 @@ first time. codebase. Thorough descriptions help all reviewers understand your request and test effectively. - If you know your change depends on another being merged first, note it in the - description and set an [merge request dependency](../user/project/merge_requests/merge_request_dependencies.md). -- Be grateful for the reviewer's suggestions. (`Good call. I'll make that change.`) + description and set a [merge request dependency](../user/project/merge_requests/merge_request_dependencies.md). +- Be grateful for the reviewer's suggestions. ("Good call. I'll make that change.") - Don't take it personally. The review is of the code, not of you. - Explain why the code exists. ("It's like that because of these reasons. Would it be more clear if I rename this class/file/method/variable?") @@ -425,20 +473,20 @@ WARNING: - Start a new merge request pipeline with the `Run pipeline` button in the merge request's "Pipelines" tab, and enable "Merge When Pipeline Succeeds" (MWPS). Note that: - - If **[main is broken](https://about.gitlab.com/handbook/engineering/workflow/#broken-master), + - If **[the default branch is broken](https://about.gitlab.com/handbook/engineering/workflow/#broken-master), do not merge the merge request** except for [very specific cases](https://about.gitlab.com/handbook/engineering/workflow/#criteria-for-merging-during-broken-master). For other cases, follow these [handbook instructions](https://about.gitlab.com/handbook/engineering/workflow/#merging-during-broken-master). - If the latest pipeline was created before the merge request was approved, start a new pipeline to ensure that full RSpec suite has been run. You may skip this step only if the merge request does not contain any backend change. - If the **latest [Pipeline for Merged Results](../ci/pipelines/pipelines_for_merged_results.md)** finished less than 2 hours ago, you - might merge without starting a new pipeline as the merge request is close + may merge without starting a new pipeline as the merge request is close enough to `main`. - When you set the MR to "Merge When Pipeline Succeeds", you should take over subsequent revisions for anything that would be spotted after that. - For merge requests that have had [Squash and merge](../user/project/merge_requests/squash_and_merge.md#squash-and-merge) set, - the squashed commit’s default commit message is taken from the merge request title. - You're encouraged to [select a commit with a more informative commit message](../user/project/merge_requests/squash_and_merge.md#overview) before merging. + the squashed commit's default commit message is taken from the merge request title. + You're encouraged to [select a commit with a more informative commit message](../user/project/merge_requests/squash_and_merge.md) before merging. Thanks to **Pipeline for Merged Results**, authors no longer have to rebase their branch as frequently anymore (only when there are conflicts) because the Merge @@ -526,7 +574,7 @@ author. GitLab is used in a lot of places. Many users use our [Omnibus packages](https://about.gitlab.com/install/), but some use -the [Docker images](https://docs.gitlab.com/omnibus/docker/), some are +the [Docker images](../install/docker.md), some are [installed from source](../install/installation.md), and there are other installation methods available. GitLab.com itself is a large Enterprise Edition instance. This has some implications: @@ -566,7 +614,7 @@ Enterprise Edition instance. This has some implications: If you're adding a new setting in `gitlab.yml`: 1. Try to avoid that, and add to `ApplicationSetting` instead. 1. Ensure that it is also - [added to Omnibus](https://docs.gitlab.com/omnibus/settings/gitlab.yml.html#adding-a-new-setting-to-gitlab-yml). + [added to Omnibus](https://docs.gitlab.com/omnibus/settings/gitlab.yml#adding-a-new-setting-to-gitlabyml). 1. **File system access** is not possible in a [cloud-native architecture](architecture.md#adapting-existing-and-introducing-new-components). Ensure that we support object storage for any file storage we need to perform. For more information, see the [uploads documentation](uploads.md). @@ -613,7 +661,7 @@ A merge request may benefit from being considered a customer critical priority b Properties of customer critical merge requests: -- The [VP of Development](https://about.gitlab.com/job-families/engineering/development/management/vp) ([@clefelhocz1](https://gitlab.com/clefelhocz1)) is the DRI for deciding if a merge request qualifies as customer critical. +- The [VP of Development](https://about.gitlab.com/job-families/engineering/development/management/vp/) ([@clefelhocz1](https://gitlab.com/clefelhocz1)) is the DRI for deciding if a merge request qualifies as customer critical. - The DRI applies the `customer-critical-merge-request` label to the merge request. - It is required that the reviewer(s) and maintainer(s) involved with a customer critical merge request are engaged as soon as this decision is made. - It is required to prioritize work for those involved on a customer critical merge request so that they have the time available necessary to focus on it. @@ -650,7 +698,3 @@ A good example of collaboration on an MR touching multiple parts of the codebase ### Credits Largely based on the [`thoughtbot` code review guide](https://github.com/thoughtbot/guides/tree/master/code-review). - ---- - -[Return to Development documentation](index.md) diff --git a/doc/development/contributing/community_roles.md b/doc/development/contributing/community_roles.md index 3804aa7f8a8..37c3c24a7d1 100644 --- a/doc/development/contributing/community_roles.md +++ b/doc/development/contributing/community_roles.md @@ -16,7 +16,3 @@ GitLab community members and their privileges/responsibilities. | Contributor | Can make contributions to all GitLab public projects | Have a GitLab.com account | [List of current reviewers/maintainers](https://about.gitlab.com/handbook/engineering/projects/#gitlab-ce). - ---- - -[Return to Contributing documentation](index.md) diff --git a/doc/development/contributing/design.md b/doc/development/contributing/design.md index c1dd5ff4c0b..9e8375fcbdd 100644 --- a/doc/development/contributing/design.md +++ b/doc/development/contributing/design.md @@ -11,32 +11,28 @@ For guidance on UX implementation at GitLab, please refer to our [Design System] The UX team uses labels to manage their workflow. -The ~"UX" label on an issue is a signal to the UX team that it will need UX attention. +The `~UX` label on an issue is a signal to the UX team that it will need UX attention. To better understand the priority by which UX tackles issues, see the [UX section](https://about.gitlab.com/handbook/engineering/ux/) of the handbook. -Once an issue has been worked on and is ready for development, a UXer removes the ~"UX" label and applies the ~"UX ready" label to that issue. +Once an issue has been worked on and is ready for development, a UXer removes the `~UX` label and applies the `~"UX ready"` label to that issue. -There is a special type label called ~"product discovery" intended for UX, -PM, FE, and BE. It represents a discovery issue to discuss the problem and +There is a special type label called `~"product discovery"` intended for UX (user experience), +PM (product manager), FE (frontend), and BE (backend). It represents a discovery issue to discuss the problem and potential solutions. The final output for this issue could be a doc of requirements, a design artifact, or even a prototype. The solution will be developed in a subsequent milestone. -~"product discovery" issues are like any other issue and should contain a milestone label, ~"Deliverable" or ~"Stretch", when scheduled in the current milestone. +`~"product discovery"` issues are like any other issue and should contain a milestone label, `~Deliverable` or `~Stretch`, when scheduled in the current milestone. The initial issue should be about the problem we are solving. If a separate [product discovery issue](https://about.gitlab.com/handbook/engineering/ux/ux-department-workflow/#how-we-use-labels) is needed for additional research and design work, it will be created by a PM or UX person. -Assign the ~UX, ~"product discovery" and ~"Deliverable" labels, add a milestone and +Assign the `~UX`, `~"product discovery"` and `~Deliverable` labels, add a milestone and use a title that makes it clear that the scheduled issue is product discovery (for example, `Product discovery for XYZ`). In order to complete a product discovery issue in a release, you must complete the following: -1. UXer removes the ~UX label, adds the ~"UX ready" label. +1. UXer removes the `~UX` label, adds the `~"UX ready"` label. 1. Modify the issue description in the product discovery issue to contain the final design. If it makes sense, the original information indicating the need for the design can be moved to a lower "Original Information" section. 1. Copy the design to the description of the delivery issue for which the product discovery issue was created. Do not simply refer to the product discovery issue as a separate source of truth. 1. In some cases, a product discovery issue also identifies future enhancements that will not go into the issue that originated the product discovery issue. For these items, create new issues containing the designs to ensure they are not lost. Put the issues in the backlog if they are agreed upon as good ideas. Otherwise leave them for triage. - ---- - -[Return to Contributing documentation](index.md) diff --git a/doc/development/contributing/issue_workflow.md b/doc/development/contributing/issue_workflow.md index 1dfe560d68d..29f6eb57160 100644 --- a/doc/development/contributing/issue_workflow.md +++ b/doc/development/contributing/issue_workflow.md @@ -15,9 +15,7 @@ feature proposal. Show your support with an award emoji and/or join the discussion. Please submit bugs using the ['Bug' issue template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/Bug.md) provided on the issue tracker. -The text in the parenthesis is there to help you with what to include. Omit it -when submitting the actual issue. You can copy-paste it and then edit as you -see fit. +The text in the comments (`<!-- ... -->`) is there to help you with what to include. ## Issue triaging @@ -30,12 +28,12 @@ The most important thing is making sure valid issues receive feedback from the development team. Therefore the priority is mentioning developers that can help on those issues. Please select someone with relevant experience from the [GitLab team](https://about.gitlab.com/company/team/). -If there is nobody mentioned with that expertise look in the commit history for +If there is nobody mentioned with that expertise, look in the commit history for the affected files to find someone. We also use [GitLab Triage](https://gitlab.com/gitlab-org/gitlab-triage) to automate some triaging policies. This is currently set up as a scheduled pipeline -(`https://gitlab.com/gitlab-org/quality/triage-ops/pipeline_schedules/10512/editpipeline_schedules/10512/edit`, +(`https://gitlab.com/gitlab-org/quality/triage-ops/-/pipeline_schedules/10512/edit`, must have at least the Developer role in the project) running on [quality/triage-ops](https://gitlab.com/gitlab-org/quality/triage-ops) project. @@ -132,9 +130,9 @@ their color is `#A8D695`. <https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/stages.yml>, with `_` replaced with a space. -For instance, the "Continuous Integration" group is represented by the -~"group::continuous integration" label in the `gitlab-org` group since its key -under `stages.manage.groups` is `continuous_integration`. +For instance, the "Pipeline Execution" group is represented by the +~"group::pipeline execution" label in the `gitlab-org` group since its key +under `stages.manage.groups` is `pipeline_execution`. The current group labels can be found by [searching the labels list for `group::`](https://gitlab.com/groups/gitlab-org/-/labels?search=group::). @@ -146,17 +144,6 @@ You can find the groups listed in the [Product Stages, Groups, and Categories](h We use the term group to map down product requirements from our product stages. As a team needs some way to collect the work their members are planning to be assigned to, we use the `~group::` labels to do so. -Normally there is a 1:1 relationship between Stage labels and Group labels. In -the spirit of "Everyone can contribute", any issue can be picked up by any group, -depending on current priorities. When picking up an issue belonging to a different -group, it should be relabeled. For example, if an issue labeled `~"devops::create"` -and `~"group::knowledge"` is picked up by someone in the Access group of the Plan stage, -the issue should be relabeled as `~"group::access"` while keeping the original -`~"devops::create"` unchanged. - -We also use stage and group labels to help measure our [merge request rates](https://about.gitlab.com/handbook/engineering/metrics/#merge-request-rate). -Please read [Stage and Group labels](https://about.gitlab.com/handbook/engineering/metrics/#stage-and-group-labels) for more information on how the labels are used in this context. - ### Category labels From the handbook's @@ -384,7 +371,7 @@ below will make it easy to manage this, without unnecessary overhead. 1. If you don't agree with a set weight, discuss with other developers until consensus is reached about the weight 1. Issue weights are an abstract measurement of complexity of the issue. Do not - relate issue weight directly to time. This is called [anchoring](https://en.wikipedia.org/wiki/Anchoring) + relate issue weight directly to time. This is called [anchoring](https://en.wikipedia.org/wiki/Anchoring_(cognitive_bias)) and something you want to avoid. 1. Something that has a weight of 1 (or no weight) is really small and simple. Something that is 9 is rewriting a large fundamental part of GitLab, @@ -476,7 +463,3 @@ should be of the same quality as those created [in the usual manner](#technical-and-ux-debt) - in particular, the issue title **must not** begin with `Follow-up`! The creating maintainer should also expect to be involved in some capacity when work begins on the follow-up issue. - ---- - -[Return to Contributing documentation](index.md) diff --git a/doc/development/contributing/merge_request_workflow.md b/doc/development/contributing/merge_request_workflow.md index 534150e4d37..25561764bd6 100644 --- a/doc/development/contributing/merge_request_workflow.md +++ b/doc/development/contributing/merge_request_workflow.md @@ -223,11 +223,19 @@ the contribution acceptance criteria below: ## Definition of done -If you contribute to GitLab please know that changes involve more than just +If you contribute to GitLab, please know that changes involve more than just code. We use the following [definition of done](https://www.agilealliance.org/glossary/definition-of-done). -Your contribution is not *done* until you have made sure it meets all of these +To reach the definition of done, the merge request must create no regressions and meet all these criteria: + +- Verified as working in production on GitLab.com. +- Verified as working for self-managed instances. + +If a regression occurs, we prefer you revert the change. We break the definition of done into two phases: [MR Merge](#mr-merge) and [Production use](#production-use). +Your contribution is *incomplete* until you have made sure it meets all of these requirements. +### MR Merge + 1. Clear description explaining the relevancy of the contribution. 1. Working and clean code that is commented where needed. 1. [Unit, integration, and system tests](../testing_guide/index.md) that all pass @@ -238,17 +246,30 @@ requirements. 1. [Secure coding guidelines](https://gitlab.com/gitlab-com/gl-security/security-guidelines) have been followed. 1. [Documented](../documentation/index.md) in the `/doc` directory. 1. [Changelog entry added](../changelog.md), if necessary. -1. Reviewed by relevant reviewers and all concerns are addressed for Availability, Regressions, Security. Documentation reviews should take place as soon as possible, but they should not block a merge request. -1. Merged by a project maintainer. +1. Reviewed by relevant reviewers, and all concerns are addressed for Availability, Regressions, and Security. Documentation reviews should take place as soon as possible, but they should not block a merge request. +1. The [MR acceptance checklist](../code_review.md#acceptance-checklist) has been checked as confirmed in the MR. 1. Create an issue in the [infrastructure issue tracker](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues) to inform the Infrastructure department when your contribution is changing default settings or introduces a new setting, if relevant. -1. Confirmed to be working in the [Canary stage](https://about.gitlab.com/handbook/engineering/#canary-testing) with no new [Sentry](https://about.gitlab.com/handbook/engineering/#sentry) errors or on GitLab.com once the contribution is deployed. -1. Added to the [release post](https://about.gitlab.com/handbook/marketing/blog/release-posts/), - if relevant. -1. Added to [the website](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/features.yml), if relevant. 1. [Black-box tests/end-to-end tests](../testing_guide/testing_levels.md#black-box-tests-at-the-system-level-aka-end-to-end-tests) added if required. Please contact [the quality team](https://about.gitlab.com/handbook/engineering/quality/#teams) with any questions. +1. The change is tested in a review app where possible and if appropriate. 1. The new feature does not degrade the user experience of the product. +1. The change is evaluated to [limit the impact of far-reaching work](https://about.gitlab.com/handbook/engineering/development/#reducing-the-impact-of-far-reaching-work). +1. An agreed-upon rollout plan. +1. Merged by a project maintainer. + +### Production use + +1. Confirmed to be working in staging before implementing the change in production, where possible. +1. Confirmed to be working in the production with no new [Sentry](https://about.gitlab.com/handbook/engineering/#sentry) errors after the contribution is deployed. +1. Confirmed that the rollout plan has been completed. +1. If there is a performance risk in the change, I have analyzed the performance of the system before and after the change. +1. *If the merge request uses feature flags, per-project or per-group enablement, and a staged rollout:* + - Confirmed to be working on GitLab projects. + - Confirmed to be working at each stage for all projects added. +1. Added to the [release post](https://about.gitlab.com/handbook/marketing/blog/release-posts/), + if relevant. +1. Added to [the website](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/features.yml), if relevant. Contributions do not require approval from the [Product team](https://about.gitlab.com/handbook/product/product-processes/#gitlab-pms-arent-the-arbiters-of-community-contributions). diff --git a/doc/development/contributing/style_guides.md b/doc/development/contributing/style_guides.md index 1b339b7f252..754e6c7aec6 100644 --- a/doc/development/contributing/style_guides.md +++ b/doc/development/contributing/style_guides.md @@ -177,11 +177,10 @@ This ensures that our list isn't mistakenly removed by another auto generation o the `.rubocop_todo.yml`. This also allows us greater visibility into the exceptions which are currently being resolved. -One way to generate the initial list is to run the `todo` auto generation, -with `exclude limit` set to a high number. +One way to generate the initial list is to run the Rake task `rubocop:todo:generate`: ```shell -bundle exec rubocop --auto-gen-config --auto-gen-only-exclude --exclude-limit=100000 +bundle exec rake rubocop:todo:generate ``` You can then move the list from the freshly generated `.rubocop_todo.yml` for the Cop being actively @@ -242,7 +241,3 @@ See the dedicated [Python Development Guidelines](../python_guide/index.md). ## Misc Code should be written in [US English](https://en.wikipedia.org/wiki/American_English). - ---- - -[Return to Contributing documentation](index.md) diff --git a/doc/development/database/add_foreign_key_to_existing_column.md b/doc/development/database/add_foreign_key_to_existing_column.md index f83dc35b4a6..d74f826cc14 100644 --- a/doc/development/database/add_foreign_key_to_existing_column.md +++ b/doc/development/database/add_foreign_key_to_existing_column.md @@ -4,11 +4,17 @@ group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Adding foreign key constraint to an existing column +# Add a foreign key constraint to an existing column -Foreign keys help ensure consistency between related database tables. The current database review process **always** encourages you to add [foreign keys](../foreign_keys.md) when creating tables that reference records from other tables. +Foreign keys ensure consistency between related database tables. The current database review process **always** encourages you to add [foreign keys](../foreign_keys.md) when creating tables that reference records from other tables. -Starting with Rails version 4, Rails includes migration helpers to add foreign key constraints to database tables. Before Rails 4, the only way for ensuring some level of consistency was the [`dependent`](https://guides.rubyonrails.org/association_basics.html#options-for-belongs-to-dependent) option within the association definition. Ensuring data consistency on the application level could fail in some unfortunate cases, so we might end up with inconsistent data in the table. This is mostly affecting older tables, where we simply didn't have the framework support to ensure consistency on the database level. These data inconsistencies can easily cause unexpected application behavior or bugs. +Starting with Rails version 4, Rails includes migration helpers to add foreign key constraints +to database tables. Before Rails 4, the only way for ensuring some level of consistency was the +[`dependent`](https://guides.rubyonrails.org/association_basics.html#options-for-belongs-to-dependent) +option in the association definition. Ensuring data consistency on the application level could fail +in some unfortunate cases, so we might end up with inconsistent data in the table. This mostly affects +older tables, where we didn't have the framework support to ensure consistency on the database level. +These data inconsistencies can cause unexpected application behavior or bugs. Adding a foreign key to an existing database column requires database structure changes and potential data changes. In case the table is in use, we should always assume that there is inconsistent data. @@ -45,7 +51,7 @@ class Email < ActiveRecord::Base end ``` -Problem: when the user is removed, the email records related to the removed user will stay in the `emails` table: +Problem: when the user is removed, the email records related to the removed user stays in the `emails` table: ```ruby user = User.find(1) @@ -66,9 +72,7 @@ In the example above, you'd be still able to update records in the `emails` tabl Migration file for adding `NOT VALID` foreign key: ```ruby -class AddNotValidForeignKeyToEmailsUser < ActiveRecord::Migration[5.2] - include Gitlab::Database::MigrationHelpers - +class AddNotValidForeignKeyToEmailsUser < Gitlab::Database::Migration[1.0] def up # safe to use: it requires short lock on the table since we don't validate the foreign key add_foreign_key :emails, :users, on_delete: :cascade, validate: false @@ -85,16 +89,16 @@ Avoid using the `add_foreign_key` constraint more than once per migration file, #### Data migration to fix existing records -The approach here depends on the data volume and the cleanup strategy. If we can easily find "invalid" records by doing a simple database query and the record count is not that high, then the data migration can be executed within a Rails migration. +The approach here depends on the data volume and the cleanup strategy. If we can find "invalid" +records by doing a database query and the record count is not high, then the data migration can +be executed in a Rails migration. In case the data volume is higher (>1000 records), it's better to create a background migration. If unsure, please contact the database team for advice. -Example for cleaning up records in the `emails` table within a database migration: +Example for cleaning up records in the `emails` table in a database migration: ```ruby -class RemoveRecordsWithoutUserFromEmailsTable < ActiveRecord::Migration[5.2] - include Gitlab::Database::MigrationHelpers - +class RemoveRecordsWithoutUserFromEmailsTable < Gitlab::Database::Migration[1.0] disable_ddl_transaction! class Email < ActiveRecord::Base @@ -116,7 +120,7 @@ end ### Validate the foreign key -Validating the foreign key will scan the whole table and make sure that each relation is correct. +Validating the foreign key scans the whole table and makes sure that each relation is correct. NOTE: When using [background migrations](../background_migrations.md), foreign key validation should happen in the next GitLab release. @@ -126,9 +130,7 @@ Migration file for validating the foreign key: ```ruby # frozen_string_literal: true -class ValidateForeignKeyOnEmailUsers < ActiveRecord::Migration[5.2] - include Gitlab::Database::MigrationHelpers - +class ValidateForeignKeyOnEmailUsers < Gitlab::Database::Migration[1.0] def up validate_foreign_key :emails, :user_id end diff --git a/doc/development/database/constraint_naming_convention.md b/doc/development/database/constraint_naming_convention.md index 3faef8aee09..a22ddc1551c 100644 --- a/doc/development/database/constraint_naming_convention.md +++ b/doc/development/database/constraint_naming_convention.md @@ -6,7 +6,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Constraints naming conventions -The most common option is to let Rails pick the name for database constraints and indexes or let PostgreSQL use the defaults (when applicable). However, when needing to define custom names in Rails or working in Go applications where no ORM is used, it is important to follow strict naming conventions to improve consistency and discoverability. +The most common option is to let Rails pick the name for database constraints and indexes or let +PostgreSQL use the defaults (when applicable). However, when defining custom names in Rails, or +working in Go applications where no ORM is used, it is important to follow strict naming conventions +to improve consistency and discoverability. The table below describes the naming conventions for custom PostgreSQL constraints. The intent is not to retroactively change names in existing databases but rather ensure consistency of future changes. diff --git a/doc/development/database/database_reviewer_guidelines.md b/doc/development/database/database_reviewer_guidelines.md index 7a9c08d9d49..59653c6dde3 100644 --- a/doc/development/database/database_reviewer_guidelines.md +++ b/doc/development/database/database_reviewer_guidelines.md @@ -19,7 +19,7 @@ Database reviewers are domain experts who have substantial experience with datab A database review is required whenever an application update [touches the database](../database_review.md#general-process). The database reviewer is tasked with reviewing the database specific updates and -making sure that any queries or modifications will perform without issues +making sure that any queries or modifications perform without issues at the scale of GitLab.com. For more information on the database review process, check the [database review guidelines](../database_review.md). @@ -72,7 +72,7 @@ topics and use cases. The most frequently required during database reviewing are - [Avoiding downtime in migrations](../avoiding_downtime_in_migrations.md). - [SQL guidelines](../sql.md) for working with SQL queries. -## How to apply for becoming a database maintainer +## How to apply to become a database maintainer Once a database reviewer feels confident on switching to a database maintainer, they can update their [team profile](https://gitlab.com/gitlab-com/www-gitlab-com/-/blob/master/data/team.yml) diff --git a/doc/development/database/efficient_in_operator_queries.md b/doc/development/database/efficient_in_operator_queries.md new file mode 100644 index 00000000000..bc72bce30bf --- /dev/null +++ b/doc/development/database/efficient_in_operator_queries.md @@ -0,0 +1,949 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Efficient `IN` operator queries + +This document describes a technique for building efficient ordered database queries with the `IN` +SQL operator and the usage of a GitLab utility module to help apply the technique. + +NOTE: +The described technique makes heavy use of +[keyset pagination](pagination_guidelines.md#keyset-pagination). +It's advised to get familiar with the topic first. + +## Motivation + +In GitLab, many domain objects like `Issue` live under nested hierarchies of projects and groups. +To fetch nested database records for domain objects at the group-level, +we often perform queries with the `IN` SQL operator. +We are usually interested in ordering the records by some attributes +and limiting the number of records using `ORDER BY` and `LIMIT` clauses for performance. +Pagination may be used to fetch subsequent records. + +Example tasks requiring querying nested domain objects from the group level: + +- Show first 20 issues by creation date or due date from the group `gitlab-org`. +- Show first 20 merge_requests by merged at date from the group `gitlab-com`. + +Unfortunately, ordered group-level queries typically perform badly +as their executions require heavy I/O, memory, and computations. +Let's do an in-depth examination of executing one such query. + +### Performance problems with `IN` queries + +Consider the task of fetching the twenty oldest created issues +from the group `gitlab-org` with the following query: + +```sql +SELECT "issues".* +FROM "issues" +WHERE "issues"."project_id" IN + (SELECT "projects"."id" + FROM "projects" + WHERE "projects"."namespace_id" IN + (SELECT traversal_ids[array_length(traversal_ids, 1)] AS id + FROM "namespaces" + WHERE (traversal_ids @> ('{9970}')))) +ORDER BY "issues"."created_at" ASC, + "issues"."id" ASC +LIMIT 20 +``` + +NOTE: +For pagination, ordering by the `created_at` column is not enough, +we must add the `id` column as a +[tie-breaker](pagination_performance_guidelines.md#tie-breaker-column). + +The execution of the query can be largely broken down into three steps: + +1. The database accesses both `namespaces` and `projects` tables + to find all projects from all groups in the group hierarchy. +1. The database retrieves `issues` records for each project causing heavy disk I/O. + Ideally, an appropriate index configuration should optimize this process. +1. The database sorts the `issues` rows in memory by `created_at` and returns `LIMIT 20` rows to + the end-user. For large groups, this final step requires both large memory and CPU resources. + +<details> +<summary>Expand this sentence to see the execution plan for this DB query.</summary> +<pre><code> + Limit (cost=90170.07..90170.12 rows=20 width=1329) (actual time=967.597..967.607 rows=20 loops=1) + Buffers: shared hit=239127 read=3060 + I/O Timings: read=336.879 + -> Sort (cost=90170.07..90224.02 rows=21578 width=1329) (actual time=967.596..967.603 rows=20 loops=1) + Sort Key: issues.created_at, issues.id + Sort Method: top-N heapsort Memory: 74kB + Buffers: shared hit=239127 read=3060 + I/O Timings: read=336.879 + -> Nested Loop (cost=1305.66..89595.89 rows=21578 width=1329) (actual time=4.709..797.659 rows=241534 loops=1) + Buffers: shared hit=239121 read=3060 + I/O Timings: read=336.879 + -> HashAggregate (cost=1305.10..1360.22 rows=5512 width=4) (actual time=4.657..5.370 rows=1528 loops=1) + Group Key: projects.id + Buffers: shared hit=2597 + -> Nested Loop (cost=576.76..1291.32 rows=5512 width=4) (actual time=2.427..4.244 rows=1528 loops=1) + Buffers: shared hit=2597 + -> HashAggregate (cost=576.32..579.06 rows=274 width=25) (actual time=2.406..2.447 rows=265 loops=1) + Group Key: namespaces.traversal_ids[array_length(namespaces.traversal_ids, 1)] + Buffers: shared hit=334 + -> Bitmap Heap Scan on namespaces (cost=141.62..575.63 rows=274 width=25) (actual time=1.933..2.330 rows=265 loops=1) + Recheck Cond: (traversal_ids @> '{9970}'::integer[]) + Heap Blocks: exact=243 + Buffers: shared hit=334 + -> Bitmap Index Scan on index_namespaces_on_traversal_ids (cost=0.00..141.55 rows=274 width=0) (actual time=1.897..1.898 rows=265 loops=1) + Index Cond: (traversal_ids @> '{9970}'::integer[]) + Buffers: shared hit=91 + -> Index Only Scan using index_projects_on_namespace_id_and_id on projects (cost=0.44..2.40 rows=20 width=8) (actual time=0.004..0.006 rows=6 loops=265) + Index Cond: (namespace_id = (namespaces.traversal_ids)[array_length(namespaces.traversal_ids, 1)]) + Heap Fetches: 51 + Buffers: shared hit=2263 + -> Index Scan using index_issues_on_project_id_and_iid on issues (cost=0.57..10.57 rows=544 width=1329) (actual time=0.114..0.484 rows=158 loops=1528) + Index Cond: (project_id = projects.id) + Buffers: shared hit=236524 read=3060 + I/O Timings: read=336.879 + Planning Time: 7.750 ms + Execution Time: 967.973 ms +(36 rows) +</code></pre> +</details> + +The performance of the query depends on the number of rows in the database. +On average, we can say the following: + +- Number of groups in the group-hierarchy: less than 1 000 +- Number of projects: less than 5 000 +- Number of issues: less than 100 000 + +From the list, it's apparent that the number of `issues` records has +the largest impact on the performance. +As per normal usage, we can say that the number of issue records grows +at a faster rate than the `namespaces` and the `projects` records. + +This problem affects most of our group-level features where records are listed +in a specific order, such as group-level issues, merge requests pages, and APIs. +For very large groups the database queries can easily time out, causing HTTP 500 errors. + +## Optimizing ordered `IN` queries + +In the talk +["How to teach an elephant to dance rock'n'roll"](https://www.youtube.com/watch?v=Ha38lcjVyhQ), +Maxim Boguk demonstrated a technique to optimize a special class of ordered `IN` queries, +such as our ordered group-level queries. + +A typical ordered `IN` query may look like this: + +```sql +SELECT t.* FROM t +WHERE t.fkey IN (value_set) +ORDER BY t.pkey +LIMIT N; +``` + +Here's the key insight used in the technique: we need at most `|value_set| + N` record lookups, +rather than retrieving all records satisfying the condition `t.fkey IN value_set` (`|value_set|` +is the number of values in `value_set`). + +We adopted and generalized the technique for use in GitLab by implementing utilities in the +`Gitlab::Pagination::Keyset::InOperatorOptimization` class to facilitate building efficient `IN` +queries. + +### Requirements + +The technique is not a drop-in replacement for the existing group-level queries using `IN` operator. +The technique can only optimize `IN` queries that satisfy the following requirements: + +- `LIMIT` is present, which usually means that the query is paginated + (offset or keyset pagination). +- The column used with the `IN` query and the columns in the `ORDER BY` + clause are covered with a database index. The columns in the index must be + in the following order: `column_for_the_in_query`, `order by column 1`, and + `order by column 2`. +- The columns in the `ORDER BY` clause are distinct + (the combination of the columns uniquely identifies one particular column in the table). + +WARNING: +This technique will not improve the performance of the `COUNT(*)` queries. + +## The `InOperatorOptimization` module + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67352) in GitLab 14.3. + +The `Gitlab::Pagination::Keyset::InOperatorOptimization` module implements utilities for applying a generalized version of +the efficient `IN` query technique described in the previous section. + +To build optimized, ordered `IN` queries that meet [the requirements](#requirements), +use the utility class `QueryBuilder` from the module. + +NOTE: +The generic keyset pagination module introduced in the merge request +[51481](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/51481) +plays a fundamental role in the generalized implementation of the technique +in `Gitlab::Pagination::Keyset::InOperatorOptimization`. + +### Basic usage of `QueryBuilder` + +To illustrate a basic usage, we will build a query that +fetches 20 issues with the oldest `created_at` from the group `gitlab-org`. + +The following ActiveRecord query would produce a query similar to +[the unoptimized query](#performance-problems-with-in-queries) that we examined earlier: + +```ruby +scope = Issue + .where(project_id: Group.find(9970).all_projects.select(:id)) # `gitlab-org` group and its subgroups + .order(:created_at, :id) + .limit(20) +``` + +Instead, use the query builder `InOperatorOptimization::QueryBuilder` to produce an optimized +version: + +```ruby +scope = Issue.order(:created_at, :id) +array_scope = Group.find(9970).all_projects.select(:id) +array_mapping_scope = -> (id_expression) { Issue.where(Issue.arel_table[:project_id].eq(id_expression)) } +finder_query = -> (created_at_expression, id_expression) { Issue.where(Issue.arel_table[:id].eq(id_expression)) } + +Gitlab::Pagination::Keyset::InOperatorOptimization::QueryBuilder.new( + scope: scope, + array_scope: array_scope, + array_mapping_scope: array_mapping_scope, + finder_query: finder_query +).execute.limit(20) +``` + +- `scope` represents the original `ActiveRecord::Relation` object without the `IN` query. The + relation should define an order which must be supported by the + [keyset pagination library](keyset_pagination.md#usage). +- `array_scope` contains the `ActiveRecord::Relation` object, which represents the original + `IN (subquery)`. The select values must contain the columns by which the subquery is "connected" + to the main query: the `id` of the project record. +- `array_mapping_scope` defines a lambda returning an `ActiveRecord::Relation` object. The lambda + matches (`=`) single select values from the `array_scope`. The lambda yields as many + arguments as the select values defined in the `array_scope`. The arguments are Arel SQL expressions. +- `finder_query` loads the actual record row from the database. It must also be a lambda, where + the order by column expressions is available for locating the record. In this example, the + yielded values are `created_at` and `id` SQL expressions. Finding a record is very fast via the + primary key, so we don't use the `created_at` value. + +The following database index on the `issues` table must be present +to make the query execute efficiently: + +```sql +"idx_issues_on_project_id_and_created_at_and_id" btree (project_id, created_at, id) +``` + +<details> +<summary>Expand this sentence to see the SQL query.</summary> +<pre><code> +SELECT "issues".* +FROM + (WITH RECURSIVE "array_cte" AS MATERIALIZED + (SELECT "projects"."id" + FROM "projects" + WHERE "projects"."namespace_id" IN + (SELECT traversal_ids[array_length(traversal_ids, 1)] AS id + FROM "namespaces" + WHERE (traversal_ids @> ('{9970}')))), + "recursive_keyset_cte" AS ( -- initializer row start + (SELECT NULL::issues AS records, + array_cte_id_array, + issues_created_at_array, + issues_id_array, + 0::bigint AS COUNT + FROM + (SELECT ARRAY_AGG("array_cte"."id") AS array_cte_id_array, + ARRAY_AGG("issues"."created_at") AS issues_created_at_array, + ARRAY_AGG("issues"."id") AS issues_id_array + FROM + (SELECT "array_cte"."id" + FROM array_cte) array_cte + LEFT JOIN LATERAL + (SELECT "issues"."created_at", + "issues"."id" + FROM "issues" + WHERE "issues"."project_id" = "array_cte"."id" + ORDER BY "issues"."created_at" ASC, "issues"."id" ASC + LIMIT 1) issues ON TRUE + WHERE "issues"."created_at" IS NOT NULL + AND "issues"."id" IS NOT NULL) array_scope_lateral_query + LIMIT 1) + -- initializer row finished + UNION ALL + (SELECT + -- result row start + (SELECT issues -- record finder query as the first column + FROM "issues" + WHERE "issues"."id" = recursive_keyset_cte.issues_id_array[position] + LIMIT 1), + array_cte_id_array, + recursive_keyset_cte.issues_created_at_array[:position_query.position-1]||next_cursor_values.created_at||recursive_keyset_cte.issues_created_at_array[position_query.position+1:], + recursive_keyset_cte.issues_id_array[:position_query.position-1]||next_cursor_values.id||recursive_keyset_cte.issues_id_array[position_query.position+1:], + recursive_keyset_cte.count + 1 + -- result row finished + FROM recursive_keyset_cte, + LATERAL + -- finding the cursor values of the next record start + (SELECT created_at, + id, + position + FROM UNNEST(issues_created_at_array, issues_id_array) WITH + ORDINALITY AS u(created_at, id, position) + WHERE created_at IS NOT NULL + AND id IS NOT NULL + ORDER BY "created_at" ASC, "id" ASC + LIMIT 1) AS position_query, + -- finding the cursor values of the next record end + -- finding the next cursor values (next_cursor_values_query) start + LATERAL + (SELECT "record"."created_at", + "record"."id" + FROM ( + VALUES (NULL, + NULL)) AS nulls + LEFT JOIN + (SELECT "issues"."created_at", + "issues"."id" + FROM ( + (SELECT "issues"."created_at", + "issues"."id" + FROM "issues" + WHERE "issues"."project_id" = recursive_keyset_cte.array_cte_id_array[position] + AND recursive_keyset_cte.issues_created_at_array[position] IS NULL + AND "issues"."created_at" IS NULL + AND "issues"."id" > recursive_keyset_cte.issues_id_array[position] + ORDER BY "issues"."created_at" ASC, "issues"."id" ASC) + UNION ALL + (SELECT "issues"."created_at", + "issues"."id" + FROM "issues" + WHERE "issues"."project_id" = recursive_keyset_cte.array_cte_id_array[position] + AND recursive_keyset_cte.issues_created_at_array[position] IS NOT NULL + AND "issues"."created_at" IS NULL + ORDER BY "issues"."created_at" ASC, "issues"."id" ASC) + UNION ALL + (SELECT "issues"."created_at", + "issues"."id" + FROM "issues" + WHERE "issues"."project_id" = recursive_keyset_cte.array_cte_id_array[position] + AND recursive_keyset_cte.issues_created_at_array[position] IS NOT NULL + AND "issues"."created_at" > recursive_keyset_cte.issues_created_at_array[position] + ORDER BY "issues"."created_at" ASC, "issues"."id" ASC) + UNION ALL + (SELECT "issues"."created_at", + "issues"."id" + FROM "issues" + WHERE "issues"."project_id" = recursive_keyset_cte.array_cte_id_array[position] + AND recursive_keyset_cte.issues_created_at_array[position] IS NOT NULL + AND "issues"."created_at" = recursive_keyset_cte.issues_created_at_array[position] + AND "issues"."id" > recursive_keyset_cte.issues_id_array[position] + ORDER BY "issues"."created_at" ASC, "issues"."id" ASC)) issues + ORDER BY "issues"."created_at" ASC, "issues"."id" ASC + LIMIT 1) record ON TRUE + LIMIT 1) AS next_cursor_values)) + -- finding the next cursor values (next_cursor_values_query) END +SELECT (records).* + FROM "recursive_keyset_cte" AS "issues" + WHERE (COUNT <> 0)) issues -- filtering out the initializer row +LIMIT 20 +</code></pre> +</details> + +### Using the `IN` query optimization + +#### Adding more filters + +In this example, let's add an extra filter by `milestone_id`. + +Be careful when adding extra filters to the query. If the column is not covered by the same index, +then the query might perform worse than the non-optimized query. The `milestone_id` column in the +`issues` table is currently covered by a different index: + +```sql +"index_issues_on_milestone_id" btree (milestone_id) +``` + +Adding the `miletone_id = X` filter to the `scope` argument or to the optimized scope causes bad performance. + +Example (bad): + +```ruby +Gitlab::Pagination::Keyset::InOperatorOptimization::QueryBuilder.new( + scope: scope, + array_scope: array_scope, + array_mapping_scope: array_mapping_scope, + finder_query: finder_query +).execute + .where(milestone_id: 5) + .limit(20) +``` + +To address this concern, we could define another index: + +```sql +"idx_issues_on_project_id_and_milestone_id_and_created_at_and_id" btree (project_id, milestone_id, created_at, id) +``` + +Adding more indexes to the `issues` table could significantly affect the performance of +the `UPDATE` queries. In this case, it's better to rely on the original query. It means that if we +want to use the optimization for the unfiltered page we need to add extra logic in the application code: + +```ruby +if optimization_possible? # no extra params or params covered with the same index as the ORDER BY clause + run_optimized_query +else + run_normal_in_query +end +``` + +#### Multiple `IN` queries + +Let's assume that we want to extend the group-level queries to include only incident and test case +issue types. + +The original ActiveRecord query would look like this: + +```ruby +scope = Issue + .where(project_id: Group.find(9970).all_projects.select(:id)) # `gitlab-org` group and its subgroups + .where(issue_type: [:incident, :test_case]) # 1, 2 + .order(:created_at, :id) + .limit(20) +``` + +To construct the array scope, we'll need to take the Cartesian product of the `project_id IN` and +the `issue_type IN` queries. `issue_type` is an ActiveRecord enum, so we need to +construct the following table: + +| `project_id` | `issue_type_value` | +| ------------ | ------------------ | +| 2 | 1 | +| 2 | 2 | +| 5 | 1 | +| 5 | 2 | +| 10 | 1 | +| 10 | 2 | +| 9 | 1 | +| 9 | 2 | + +For the `issue_types` query we can construct a value list without querying a table: + +```ruby +value_list = Arel::Nodes::ValuesList.new([[Issue.issue_types[:incident]],[Issue.issue_types[:test_case]]]) +issue_type_values = Arel::Nodes::Grouping.new(value_list).as('issue_type_values (value)').to_sql + +array_scope = Group + .find(9970) + .all_projects + .from("#{Project.table_name}, #{issue_type_values}") + .select(:id, :value) +``` + +Building the `array_mapping_scope` query requires two arguments: `id` and `issue_type_value`: + +```ruby +array_mapping_scope = -> (id_expression, issue_type_value_expression) { Issue.where(Issue.arel_table[:project_id].eq(id_expression)).where(Issue.arel_table[:issue_type].eq(issue_type_value_expression)) } +``` + +The `scope` and the `finder` queries don't change: + +```ruby +scope = Issue.order(:created_at, :id) +finder_query = -> (created_at_expression, id_expression) { Issue.where(Issue.arel_table[:id].eq(id_expression)) } + +Gitlab::Pagination::Keyset::InOperatorOptimization::QueryBuilder.new( + scope: scope, + array_scope: array_scope, + array_mapping_scope: array_mapping_scope, + finder_query: finder_query +).execute.limit(20) +``` + +<details> +<summary>Expand this sentence to see the SQL query.</summary> +<pre><code> +SELECT "issues".* +FROM + (WITH RECURSIVE "array_cte" AS MATERIALIZED + (SELECT "projects"."id", "value" + FROM projects, ( + VALUES (1), (2)) AS issue_type_values (value) + WHERE "projects"."namespace_id" IN + (WITH RECURSIVE "base_and_descendants" AS ( + (SELECT "namespaces".* + FROM "namespaces" + WHERE "namespaces"."type" = 'Group' + AND "namespaces"."id" = 9970) + UNION + (SELECT "namespaces".* + FROM "namespaces", "base_and_descendants" + WHERE "namespaces"."type" = 'Group' + AND "namespaces"."parent_id" = "base_and_descendants"."id")) SELECT "id" + FROM "base_and_descendants" AS "namespaces")), + "recursive_keyset_cte" AS ( + (SELECT NULL::issues AS records, + array_cte_id_array, + array_cte_value_array, + issues_created_at_array, + issues_id_array, + 0::bigint AS COUNT + FROM + (SELECT ARRAY_AGG("array_cte"."id") AS array_cte_id_array, + ARRAY_AGG("array_cte"."value") AS array_cte_value_array, + ARRAY_AGG("issues"."created_at") AS issues_created_at_array, + ARRAY_AGG("issues"."id") AS issues_id_array + FROM + (SELECT "array_cte"."id", + "array_cte"."value" + FROM array_cte) array_cte + LEFT JOIN LATERAL + (SELECT "issues"."created_at", + "issues"."id" + FROM "issues" + WHERE "issues"."project_id" = "array_cte"."id" + AND "issues"."issue_type" = "array_cte"."value" + ORDER BY "issues"."created_at" ASC, "issues"."id" ASC + LIMIT 1) issues ON TRUE + WHERE "issues"."created_at" IS NOT NULL + AND "issues"."id" IS NOT NULL) array_scope_lateral_query + LIMIT 1) + UNION ALL + (SELECT + (SELECT issues + FROM "issues" + WHERE "issues"."id" = recursive_keyset_cte.issues_id_array[POSITION] + LIMIT 1), array_cte_id_array, + array_cte_value_array, + recursive_keyset_cte.issues_created_at_array[:position_query.position-1]||next_cursor_values.created_at||recursive_keyset_cte.issues_created_at_array[position_query.position+1:], recursive_keyset_cte.issues_id_array[:position_query.position-1]||next_cursor_values.id||recursive_keyset_cte.issues_id_array[position_query.position+1:], recursive_keyset_cte.count + 1 + FROM recursive_keyset_cte, + LATERAL + (SELECT created_at, + id, + POSITION + FROM UNNEST(issues_created_at_array, issues_id_array) WITH + ORDINALITY AS u(created_at, id, POSITION) + WHERE created_at IS NOT NULL + AND id IS NOT NULL + ORDER BY "created_at" ASC, "id" ASC + LIMIT 1) AS position_query, + LATERAL + (SELECT "record"."created_at", + "record"."id" + FROM ( + VALUES (NULL, + NULL)) AS nulls + LEFT JOIN + (SELECT "issues"."created_at", + "issues"."id" + FROM ( + (SELECT "issues"."created_at", + "issues"."id" + FROM "issues" + WHERE "issues"."project_id" = recursive_keyset_cte.array_cte_id_array[POSITION] + AND "issues"."issue_type" = recursive_keyset_cte.array_cte_value_array[POSITION] + AND recursive_keyset_cte.issues_created_at_array[POSITION] IS NULL + AND "issues"."created_at" IS NULL + AND "issues"."id" > recursive_keyset_cte.issues_id_array[POSITION] + ORDER BY "issues"."created_at" ASC, "issues"."id" ASC) + UNION ALL + (SELECT "issues"."created_at", + "issues"."id" + FROM "issues" + WHERE "issues"."project_id" = recursive_keyset_cte.array_cte_id_array[POSITION] + AND "issues"."issue_type" = recursive_keyset_cte.array_cte_value_array[POSITION] + AND recursive_keyset_cte.issues_created_at_array[POSITION] IS NOT NULL + AND "issues"."created_at" IS NULL + ORDER BY "issues"."created_at" ASC, "issues"."id" ASC) + UNION ALL + (SELECT "issues"."created_at", + "issues"."id" + FROM "issues" + WHERE "issues"."project_id" = recursive_keyset_cte.array_cte_id_array[POSITION] + AND "issues"."issue_type" = recursive_keyset_cte.array_cte_value_array[POSITION] + AND recursive_keyset_cte.issues_created_at_array[POSITION] IS NOT NULL + AND "issues"."created_at" > recursive_keyset_cte.issues_created_at_array[POSITION] + ORDER BY "issues"."created_at" ASC, "issues"."id" ASC) + UNION ALL + (SELECT "issues"."created_at", + "issues"."id" + FROM "issues" + WHERE "issues"."project_id" = recursive_keyset_cte.array_cte_id_array[POSITION] + AND "issues"."issue_type" = recursive_keyset_cte.array_cte_value_array[POSITION] + AND recursive_keyset_cte.issues_created_at_array[POSITION] IS NOT NULL + AND "issues"."created_at" = recursive_keyset_cte.issues_created_at_array[POSITION] + AND "issues"."id" > recursive_keyset_cte.issues_id_array[POSITION] + ORDER BY "issues"."created_at" ASC, "issues"."id" ASC)) issues + ORDER BY "issues"."created_at" ASC, "issues"."id" ASC + LIMIT 1) record ON TRUE + LIMIT 1) AS next_cursor_values)) SELECT (records).* + FROM "recursive_keyset_cte" AS "issues" + WHERE (COUNT <> 0)) issues +LIMIT 20 +</code> +</pre> +</details> + +NOTE: +To make the query efficient, the following columns need to be covered with an index: `project_id`, `issue_type`, `created_at`, and `id`. + +#### Batch iteration + +Batch iteration over the records is possible via the keyset `Iterator` class. + +```ruby +scope = Issue.order(:created_at, :id) +array_scope = Group.find(9970).all_projects.select(:id) +array_mapping_scope = -> (id_expression) { Issue.where(Issue.arel_table[:project_id].eq(id_expression)) } +finder_query = -> (created_at_expression, id_expression) { Issue.where(Issue.arel_table[:id].eq(id_expression)) } + +opts = { + in_operator_optimization_options: { + array_scope: array_scope, + array_mapping_scope: array_mapping_scope, + finder_query: finder_query + } +} + +Gitlab::Pagination::Keyset::Iterator.new(scope: scope, **opts).each_batch(of: 100) do |records| + puts records.select(:id).map { |r| [r.id] } +end +``` + +#### Keyset pagination + +The optimization works out of the box with GraphQL and the `keyset_paginate` helper method. +Read more about [keyset pagination](keyset_pagination.md). + +```ruby +array_scope = Group.find(9970).all_projects.select(:id) +array_mapping_scope = -> (id_expression) { Issue.where(Issue.arel_table[:project_id].eq(id_expression)) } +finder_query = -> (created_at_expression, id_expression) { Issue.where(Issue.arel_table[:id].eq(id_expression)) } + +opts = { + in_operator_optimization_options: { + array_scope: array_scope, + array_mapping_scope: array_mapping_scope, + finder_query: finder_query + } +} + +issues = Issue + .order(:created_at, :id) + .keyset_paginate(per_page: 20, keyset_order_options: opts) + .records +``` + +#### Offset pagination with Kaminari + +The `ActiveRecord` scope produced by the `InOperatorOptimization` class can be used in +[offset-paginated](pagination_guidelines.md#offset-pagination) +queries. + +```ruby +Gitlab::Pagination::Keyset::InOperatorOptimization::QueryBuilder + .new(...) + .execute + .page(1) + .per(20) + .without_count +``` + +## Generalized `IN` optimization technique + +Let's dive into how `QueryBuilder` builds the optimized query +to fetch the twenty oldest created issues from the group `gitlab-org` +using the generalized `IN` optimization technique. + +### Array CTE + +As the first step, we use a common table expression (CTE) for collecting the `projects.id` values. +This is done by wrapping the incoming `array_scope` ActiveRecord relation parameter with a CTE. + +```sql +WITH array_cte AS MATERIALIZED ( + SELECT "projects"."id" + FROM "projects" + WHERE "projects"."namespace_id" IN + (SELECT traversal_ids[array_length(traversal_ids, 1)] AS id + FROM "namespaces" + WHERE (traversal_ids @> ('{9970}'))) +) +``` + +This query produces the following result set with only one column (`projects.id`): + +| ID | +| --- | +| 9 | +| 2 | +| 5 | +| 10 | + +### Array mapping + +For each project (that is, each record storing a project ID in `array_cte`), +we will fetch the cursor value identifying the first issue respecting the `ORDER BY` clause. + +As an example, let's pick the first record `ID=9` from `array_cte`. +The following query should fetch the cursor value `(created_at, id)` identifying +the first issue record respecting the `ORDER BY` clause for the project with `ID=9`: + +```sql +SELECT "issues"."created_at", "issues"."id" +FROM "issues"."project_id"=9 +ORDER BY "issues"."created_at" ASC, "issues"."id" ASC +LIMIT 1; +``` + +We will use `LATERAL JOIN` to loop over the records in the `array_cte` and find the +cursor value for each project. The query would be built using the `array_mapping_scope` lambda +function. + +```sql +SELECT ARRAY_AGG("array_cte"."id") AS array_cte_id_array, + ARRAY_AGG("issues"."created_at") AS issues_created_at_array, + ARRAY_AGG("issues"."id") AS issues_id_array +FROM ( + SELECT "array_cte"."id" FROM array_cte +) array_cte +LEFT JOIN LATERAL +( + SELECT "issues"."created_at", "issues"."id" + FROM "issues" + WHERE "issues"."project_id" = "array_cte"."id" + ORDER BY "issues"."created_at" ASC, "issues"."id" ASC + LIMIT 1 +) issues ON TRUE +``` + +Since we have an index on `project_id`, `created_at`, and `id`, +index-only scans should quickly locate all the cursor values. + +This is how the query could be translated to Ruby: + +```ruby +created_at_values = [] +id_values = [] +project_ids.map do |project_id| + created_at, id = Issue.select(:created_at, :id).where(project_id: project_id).order(:created_at, :id).limit(1).first # N+1 but it's fast + created_at_values << created_at + id_values << id +end +``` + +This is what the result set would look like: + +| `project_ids` | `created_at_values` | `id_values` | +| ------------- | ------------------- | ----------- | +| 2 | 2020-01-10 | 5 | +| 5 | 2020-01-05 | 4 | +| 10 | 2020-01-15 | 7 | +| 9 | 2020-01-05 | 3 | + +The table shows the cursor values (`created_at, id`) of the first record for each project +respecting the `ORDER BY` clause. + +At this point, we have the initial data. To start collecting the actual records from the database, +we'll use a recursive CTE query where each recursion locates one row until +the `LIMIT` is reached or no more data can be found. + +Here's an outline of the steps we will take in the recursive CTE query +(expressing the steps in SQL is non-trivial but will be explained next): + +1. Sort the initial resultset according to the `ORDER BY` clause. +1. Pick the top cursor to fetch the record, this is our first record. In the example, +this cursor would be (`2020-01-05`, `3`) for `project_id=9`. +1. We can use (`2020-01-05`, `3`) to fetch the next issue respecting the `ORDER BY` clause +`project_id=9` filter. This produces an updated resultset. + + | `project_ids` | `created_at_values` | `id_values` | + | ------------- | ------------------- | ----------- | + | 2 | 2020-01-10 | 5 | + | 5 | 2020-01-05 | 4 | + | 10 | 2020-01-15 | 7 | + | **9** | **2020-01-06** | **6** | + +1. Repeat 1 to 3 with the updated resultset until we have fetched `N=20` records. + +### Initializing the recursive CTE query + +For the initial recursive query, we'll need to produce exactly one row, we call this the +initializer query (`initializer_query`). + +Use `ARRAY_AGG` function to compact the initial result set into a single row +and use the row as the initial value for the recursive CTE query: + +Example initializer row: + +| `records` | `project_ids` | `created_at_values` | `id_values` | `Count` | `Position` | +| -------------- | --------------- | ------------------- | ----------- | ------- | ---------- | +| `NULL::issues` | `[9, 2, 5, 10]` | `[...]` | `[...]` | `0` | `NULL` | + +- The `records` column contains our sorted database records, and the initializer query sets the +first value to `NULL`, which is filtered out later. +- The `count` column tracks the number of records found. We use this column to filter out the +initializer row from the result set. + +### Recursive portion of the CTE query + +The result row is produced with the following steps: + +1. [Order the keyset arrays.](#order-the-keyset-arrays) +1. [Find the next cursor.](#find-the-next-cursor) +1. [Produce a new row.](#produce-a-new-row) + +#### Order the keyset arrays + +Order the keyset arrays according to the original `ORDER BY` clause with `LIMIT 1` using the +`UNNEST [] WITH ORDINALITY` table function. The function locates the "lowest" keyset cursor +values and gives us the array position. These cursor values are used to locate the record. + +NOTE: +At this point, we haven't read anything from the database tables, because we relied on +fast index-only scans. + +| `project_ids` | `created_at_values` | `id_values` | +| ------------- | ------------------- | ----------- | +| 2 | 2020-01-10 | 5 | +| 5 | 2020-01-05 | 4 | +| 10 | 2020-01-15 | 7 | +| 9 | 2020-01-05 | 3 | + +The first row is the 4th one (`position = 4`), because it has the lowest `created_at` and +`id` values. The `UNNEST` function also exposes the position using an extra column (note: +PostgreSQL uses 1-based index). + +Demonstration of the `UNNEST [] WITH ORDINALITY` table function: + +```sql +SELECT position FROM unnest('{2020-01-10, 2020-01-05, 2020-01-15, 2020-01-05}'::timestamp[], '{5, 4, 7, 3}'::int[]) + WITH ORDINALITY AS t(created_at, id, position) ORDER BY created_at ASC, id ASC LIMIT 1; +``` + +Result: + +```sql +position +---------- + 4 +(1 row) +``` + +#### Find the next cursor + +Now, let's find the next cursor values (`next_cursor_values_query`) for the project with `id = 9`. +To do that, we build a keyset pagination SQL query. Find the next row after +`created_at = 2020-01-05` and `id = 3`. Because we order by two database columns, there can be two +cases: + +- There are rows with `created_at = 2020-01-05` and `id > 3`. +- There are rows with `created_at > 2020-01-05`. + +Generating this query is done by the generic keyset pagination library. After the query is done, +we have a temporary table with the next cursor values: + +| `created_at` | ID | +| ------------ | --- | +| 2020-01-06 | 6 | + +#### Produce a new row + +As the final step, we need to produce a new row by manipulating the initializer row +(`data_collector_query` method). Two things happen here: + +- Read the full row from the DB and return it in the `records` columns. (`result_collector_columns` +method) +- Replace the cursor values at the current position with the results from the keyset query. + +Reading the full row from the database is only one index scan by the primary key. We use the +ActiveRecord query passed as the `finder_query`: + +```sql +(SELECT "issues".* FROM issues WHERE id = id_values[position] LIMIT 1) +``` + +By adding parentheses, the result row can be put into the `records` column. + +Replacing the cursor values at `position` can be done via standard PostgreSQL array operators: + +```sql +-- created_at_values column value +created_at_values[:position-1]||next_cursor_values.created_at||created_at_values[position+1:] + +-- id_values column value +id_values[:position-1]||next_cursor_values.id||id_values[position+1:] +``` + +The Ruby equivalent would be the following: + +```ruby +id_values[0..(position - 1)] + [next_cursor_values.id] + id_values[(position + 1)..-1] +``` + +After this, the recursion starts again by finding the next lowest cursor value. + +### Finalizing the query + +For producing the final `issues` rows, we're going to wrap the query with another `SELECT` statement: + +```sql +SELECT "issues".* +FROM ( + SELECT (records).* -- similar to ruby splat operator + FROM recursive_keyset_cte + WHERE recursive_keyset_cte.count <> 0 -- filter out the initializer row +) AS issues +``` + +### Performance comparison + +Assuming that we have the correct database index in place, we can compare the query performance by +looking at the number of database rows accessed by the query. + +- Number of groups: 100 +- Number of projects: 500 +- Number of issues (in the group hierarchy): 50 000 + +Standard `IN` query: + +| Query | Entries read from index | Rows read from the table | Rows sorted in memory | +| ------------------------ | ----------------------- | ------------------------ | --------------------- | +| group hierarchy subquery | 100 | 0 | 0 | +| project lookup query | 500 | 0 | 0 | +| issue lookup query | 50 000 | 20 | 50 000 | + +Optimized `IN` query: + +| Query | Entries read from index | Rows read from the table | Rows sorted in memory | +| ------------------------ | ----------------------- | ------------------------ | --------------------- | +| group hierarchy subquery | 100 | 0 | 0 | +| project lookup query | 500 | 0 | 0 | +| issue lookup query | 519 | 20 | 10 000 | + +The group and project queries are not using sorting, the necessary columns are read from database +indexes. These values are accessed frequently so it's very likely that most of the data will be +in the PostgreSQL's buffer cache. + +The optimized `IN` query will read maximum 519 entries (cursor values) from the index: + +- 500 index-only scans for populating the arrays for each project. The cursor values of the first +record will be here. +- Maximum 19 additional index-only scans for the consecutive records. + +The optimized `IN` query will sort the array (cursor values per project array) 20 times, which +means we'll sort 20 x 500 rows. However, this might be a less memory-intensive task than +sorting 10 000 rows at once. + +Performance comparison for the `gitlab-org` group: + +| Query | Number of 8K Buffers involved | Uncached execution time | Cached execution time | +| -------------------- | ----------------------------- | ----------------------- | --------------------- | +| `IN` query | 240833 | 1.2s | 660ms | +| Optimized `IN` query | 9783 | 450ms | 22ms | + +NOTE: +Before taking measurements, the group lookup query was executed separately in order to make +the group data available in the buffer cache. Since it's a frequently called query, it's going to +hit many shared buffers during the query execution in the production environment. diff --git a/doc/development/database/index.md b/doc/development/database/index.md index b61a71ffb8e..a7b752e14ef 100644 --- a/doc/development/database/index.md +++ b/doc/development/database/index.md @@ -62,6 +62,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w - [Query performance guidelines](../query_performance.md) - [Pagination guidelines](pagination_guidelines.md) - [Pagination performance guidelines](pagination_performance_guidelines.md) +- [Efficient `IN` operator queries](efficient_in_operator_queries.md) ## Case studies diff --git a/doc/development/database/keyset_pagination.md b/doc/development/database/keyset_pagination.md index e30c3cc8832..fd62c36b753 100644 --- a/doc/development/database/keyset_pagination.md +++ b/doc/development/database/keyset_pagination.md @@ -36,7 +36,8 @@ Keyset pagination works without any configuration for simple ActiveRecord querie - Order by one column. - Order by two columns, where the last column is the primary key. -The library can detect nullable and non-distinct columns and based on these, it will add extra ordering using the primary key. This is necessary because keyset pagination expects distinct order by values: +The library detects nullable and non-distinct columns and based on these, adds extra ordering +using the primary key. This is necessary because keyset pagination expects distinct order by values: ```ruby Project.order(:created_at).keyset_paginate.records # ORDER BY created_at, id @@ -79,7 +80,7 @@ cursor = paginator.cursor_for_next_page # encoded column attributes for the next paginator = Project.order(:name).keyset_paginate(cursor: cursor).records # loading the next page ``` -Since keyset pagination does not support page numbers, we are restricted to go to the following pages: +Because keyset pagination does not support page numbers, we are restricted to go to the following pages: - Next page - Previous page @@ -111,7 +112,8 @@ In the HAML file, we can render the records: The performance of the keyset pagination depends on the database index configuration and the number of columns we use in the `ORDER BY` clause. -In case we order by the primary key (`id`), then the generated queries will be efficient since the primary key is covered by a database index. +In case we order by the primary key (`id`), then the generated queries are efficient because +the primary key is covered by a database index. When two or more columns are used in the `ORDER BY` clause, it's advised to check the generated database query and make sure that the correct index configuration is used. More information can be found on the [pagination guideline page](pagination_guidelines.md#index-coverage). @@ -149,7 +151,9 @@ puts paginator2.records.to_a # UNION query ## Complex order configuration -Common `ORDER BY` configurations will be handled by the `keyset_paginate` method automatically so no manual configuration is needed. There are a few edge cases where order object configuration is necessary: +Common `ORDER BY` configurations are handled by the `keyset_paginate` method automatically +so no manual configuration is needed. There are a few edge cases where order object +configuration is necessary: - `NULLS LAST` ordering. - Function-based ordering. @@ -170,12 +174,13 @@ scope.keyset_paginate # raises: Gitlab::Pagination::Keyset::Paginator::Unsupport The `keyset_paginate` method raises an error because the order value on the query is a custom SQL string and not an [`Arel`](https://www.rubydoc.info/gems/arel) AST node. The keyset library cannot automatically infer configuration values from these kinds of queries. -To make keyset pagination work, we need to configure custom order objects, to do so, we need to collect information about the order columns: +To make keyset pagination work, we must configure custom order objects, to do so, we must +collect information about the order columns: -- `relative_position` can have duplicated values since no unique index is present. -- `relative_position` can have null values because we don't have a not null constraint on the column. For this, we need to determine where will we see NULL values, at the beginning of the resultset or the end (`NULLS LAST`). -- Keyset pagination requires distinct order columns, so we'll need to add the primary key (`id`) to make the order distinct. -- Jumping to the last page and paginating backwards actually reverses the `ORDER BY` clause. For this, we'll need to provide the reversed `ORDER BY` clause. +- `relative_position` can have duplicated values because no unique index is present. +- `relative_position` can have null values because we don't have a not null constraint on the column. For this, we must determine where we see NULL values, at the beginning of the result set, or the end (`NULLS LAST`). +- Keyset pagination requires distinct order columns, so we must add the primary key (`id`) to make the order distinct. +- Jumping to the last page and paginating backwards actually reverses the `ORDER BY` clause. For this, we must provide the reversed `ORDER BY` clause. Example: @@ -206,7 +211,8 @@ scope.keyset_paginate.records # works ### Function-based ordering -In the following example, we multiply the `id` by 10 and ordering by that value. Since the `id` column is unique, we need to define only one column: +In the following example, we multiply the `id` by 10 and order by that value. Because the `id` +column is unique, we define only one column: ```ruby order = Gitlab::Pagination::Keyset::Order.build([ @@ -233,7 +239,8 @@ The `add_to_projections` flag tells the paginator to expose the column expressio ### `iid` based ordering -When ordering issues, the database ensures that we'll have distinct `iid` values within a project. Ordering by one column is enough to make the pagination work if the `project_id` filter is present: +When ordering issues, the database ensures that we have distinct `iid` values in a project. +Ordering by one column is enough to make the pagination work if the `project_id` filter is present: ```ruby order = Gitlab::Pagination::Keyset::Order.build([ diff --git a/doc/development/database/multiple_databases.md b/doc/development/database/multiple_databases.md index 71dcc5bb866..0fd9f821fab 100644 --- a/doc/development/database/multiple_databases.md +++ b/doc/development/database/multiple_databases.md @@ -24,24 +24,26 @@ configurations. For example, given a `config/database.yml` like below: ```yaml development: - adapter: postgresql - encoding: unicode - database: gitlabhq_development - host: /path/to/gdk/postgresql - pool: 10 - prepared_statements: false - variables: - statement_timeout: 120s + main: + adapter: postgresql + encoding: unicode + database: gitlabhq_development + host: /path/to/gdk/postgresql + pool: 10 + prepared_statements: false + variables: + statement_timeout: 120s test: &test - adapter: postgresql - encoding: unicode - database: gitlabhq_test - host: /path/to/gdk/postgresql - pool: 10 - prepared_statements: false - variables: - statement_timeout: 120s + main: + adapter: postgresql + encoding: unicode + database: gitlabhq_test + host: /path/to/gdk/postgresql + pool: 10 + prepared_statements: false + variables: + statement_timeout: 120s ``` Edit the `config/database.yml` to look like this: @@ -98,18 +100,45 @@ and their tables must be placed in two directories for now: We aim to keep the schema for both tables the same across both databases. +<!-- +NOTE: The `validate_cross_joins!` method in `spec/support/database/prevent_cross_joins.rb` references + the following heading in the code, so if you make a change to this heading, make sure to update + the corresponding documentation URL used in `spec/support/database/prevent_cross_joins.rb`. +--> + ### Removing joins between `ci_*` and non `ci_*` tables -We are planning on moving all the `ci_*` tables to a separate database so +Queries that join across databases raise an error. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68620) +in GitLab 14.3, for new queries only. Pre-existing queries do not raise an error. + +We are planning on moving all the `ci_*` tables to a separate database, so referencing `ci_*` tables with other tables will not be possible. This means, that using any kind of `JOIN` in SQL queries will not work. We have identified already many such examples that need to be fixed in <https://gitlab.com/groups/gitlab-org/-/epics/6289> . -The following are some real examples that have resulted from this and these -patterns may apply to future cases. +#### Path to removing cross-database joins + +The following steps are the process to remove cross-database joins between +`ci_*` and non `ci_*` tables: + +1. **{check-circle}** Add all failing specs to the [`cross-join-allowlist.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/f5de89daeb468fc45e1e95a76d1b5297aa53da11/spec/support/database/cross-join-allowlist.yml) + file. +1. **{dotted-circle}** Find the code that caused the spec failure and wrap the isolated code + in [`allow_cross_joins_across_databases`](#allowlist-for-existing-cross-joins). + Link to a new issue assigned to the correct team to remove the specs from the + `cross-join-allowlist.yml` file. +1. **{dotted-circle}** Remove the `cross-join-allowlist.yml` file and stop allowing + whole test files. +1. **{dotted-circle}** Fix the problem and remove the `allow_cross_joins_across_databases` call. +1. **{dotted-circle}** Fix all the cross-joins and remove the `allow_cross_joins_across_databases` method. + +#### Suggestions for removing cross-database joins -#### Remove the code +The following sections are some real examples that were identified as joining across databases, +along with possible suggestions on how to fix them. + +##### Remove the code The simplest solution we've seen several times now has been an existing scope that is unused. This is the easiest example to fix. So the first step is to @@ -131,7 +160,7 @@ to evaluate, because `UsageData` is not critical to users and it may be possible to get a similarly useful metric with a simpler approach. Alternatively we may find that nobody is using these metrics, so we can remove them. -#### Use `preload` instead of `includes` +##### Use `preload` instead of `includes` The `includes` and `preload` methods in Rails are both ways to avoid an N+1 query. The `includes` method in Rails uses a heuristic approach to determine @@ -145,7 +174,7 @@ allows you to avoid the join, while still avoiding the N+1 query. You can see a real example of this solution being used in <https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67655>. -#### De-normalize some foreign key to the table +##### De-normalize some foreign key to the table De-normalization refers to adding redundant precomputed (duplicated) data to a table to simplify certain queries or to improve performance. In this @@ -198,7 +227,7 @@ You can see this approach implemented in <https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66963> . This MR also de-normalizes `pipeline_id` to fix a similar query. -#### De-normalize into an extra table +##### De-normalize into an extra table Sometimes the previous de-normalization (adding an extra column) doesn't work for your specific case. This may be due to the fact that your data is not 1:1, or @@ -245,18 +274,88 @@ logic to delete these rows if or whenever necessary in your domain. Finally, this de-normalization and new query also improves performance because it does less joins and needs less filtering. -#### Summary of cross-join removal patterns +##### Use `disable_joins` for `has_one` or `has_many` `through:` relations + +Sometimes a join query is caused by using `has_one ... through:` or `has_many +... through:` across tables that span the different databases. These joins +sometimes can be solved by adding +[`disable_joins:true`](https://edgeguides.rubyonrails.org/active_record_multiple_databases.html#handling-associations-with-joins-across-databases). +This is a Rails feature which we +[backported](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66400). We +also extended the feature to allow a lambda syntax for enabling `disable_joins` +with a feature flag. If you use this feature we encourage using a feature flag +as it mitigates risk if there is some serious performance regression. + +You can see an example where this was used in +<https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66709/diffs>. + +With any change to DB queries it is important to analyze and compare the SQL +before and after the change. `disable_joins` can introduce very poorly performing +code depending on the actual logic of the `has_many` or `has_one` relationship. +The key thing to look for is whether any of the intermediate result sets +used to construct the final result set have an unbounded amount of data loaded. +The best way to tell is by looking at the SQL generated and confirming that +each one is limited in some way. You can tell by either a `LIMIT 1` clause or +by `WHERE` clause that is limiting based on a unique column. Any unbounded +intermediate dataset could lead to loading too many IDs into memory. + +An example where you may see very poor performance is the following +hypothetical code: + +```ruby +class Project + has_many :pipelines + has_many :builds, through: :pipelines +end + +class Pipeline + has_many :builds +end + +class Build + belongs_to :pipeline +end + +def some_action + @builds = Project.find(5).builds.order(created_at: :desc).limit(10) +end +``` + +In the above case `some_action` will generate a query like: + +```sql +select * from builds +inner join pipelines on builds.pipeline_id = pipelines.id +where pipelines.project_id = 5 +order by builds.created_at desc +limit 10 +``` + +However, if you changed the relation to be: + +```ruby +class Project + has_many :pipelines + has_many :builds, through: :pipelines, disable_joins: true +end +``` -A quick checklist for fixing a specific join query would be: +Then you would get the following 2 queries: -1. Is the code even used? If not just remove it -1. If the code is used, then is this feature even used or can we implement the - feature in a simpler way and still meet the requirements. Always prefer the - simplest option. -1. Can we remove the join if we de-normalize the data you are joining to by - adding a new column -1. Can we remove the join by adding a new table in the correct database that - replicates the minimum data needed to do the join +```sql +select id from pipelines where project_id = 5; + +select * from builds where pipeline_id in (...) +order by created_at desc +limit 10; +``` + +Because the first query does not limit by any unique column or +have a `LIMIT` clause, it can load an unlimited number of +pipeline IDs into memory, which are then sent in the following query. +This can lead to very poor performance in the Rails application and the +database. In cases like this, you might need to re-write the +query or look at other patterns described above for removing cross-joins. #### How to validate you have correctly removed a cross-join @@ -291,3 +390,74 @@ end You can see a real example of using this method for fixing a cross-join in <https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67655>. + +#### Allowlist for existing cross-joins + +A cross-join across databases can be explicitly allowed by wrapping the code in the +`::Gitlab::Database.allow_cross_joins_across_databases` helper method. + +This method should only be used: + +- For existing code. +- If the code is required to help migrate away from a cross-join. For example, + in a migration that backfills data for future use to remove a cross-join. + +The `allow_cross_joins_across_databases` helper method can be used as follows: + +```ruby +::Gitlab::Database.allow_cross_joins_across_databases(url: 'https://gitlab.com/gitlab-org/gitlab/-/issues/336590') do + subject.perform(1, 4) +end +``` + +The `url` parameter should point to an issue with a milestone for when we intend +to fix the cross-join. If the cross-join is being used in a migration, we do not +need to fix the code. See <https://gitlab.com/gitlab-org/gitlab/-/issues/340017> +for more details. + +## `config/database.yml` + +GitLab will support running multiple databases in the future, for example to [separate tables for the continuous integration features](https://gitlab.com/groups/gitlab-org/-/epics/6167) from the main database. In order to prepare for this change, we [validate the structure of the configuration](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67877) in `database.yml` to ensure that only known databases are used. + +Previously, the `config/database.yml` would look like this: + +```yaml +production: + adapter: postgresql + encoding: unicode + database: gitlabhq_production + ... +``` + +With the support for many databases the support for this +syntax is deprecated and will be removed in [15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/338182). + +The new `config/database.yml` needs to include a database name +to define a database configuration. Only `main:` and `ci:` database +names are supported today. The `main:` needs to always be a first +entry in a hash. This change applies to decomposed and non-decomposed +change. If an invalidate or deprecated syntax is used the error +or warning will be printed during application start. + +```yaml +# Non-decomposed database +production: + main: + adapter: postgresql + encoding: unicode + database: gitlabhq_production + ... + +# Decomposed database +production: + main: + adapter: postgresql + encoding: unicode + database: gitlabhq_production + ... + ci: + adapter: postgresql + encoding: unicode + database: gitlabhq_production_ci + ... +``` diff --git a/doc/development/database/not_null_constraints.md b/doc/development/database/not_null_constraints.md index 178a207dab5..de070f7e434 100644 --- a/doc/development/database/not_null_constraints.md +++ b/doc/development/database/not_null_constraints.md @@ -25,7 +25,7 @@ For example, consider a migration that creates a table with two `NOT NULL` colum `db/migrate/20200401000001_create_db_guides.rb`: ```ruby -class CreateDbGuides < ActiveRecord::Migration[6.0] +class CreateDbGuides < Gitlab::Database::Migration[1.0] def change create_table :db_guides do |t| t.bigint :stars, default: 0, null: false @@ -44,7 +44,7 @@ For example, consider a migration that adds a new `NOT NULL` column `active` to `db/migrate/20200501000001_add_active_to_db_guides.rb`: ```ruby -class AddExtendedTitleToSprints < ActiveRecord::Migration[6.0] +class AddExtendedTitleToSprints < Gitlab::Database::Migration[1.0] def change add_column :db_guides, :active, :boolean, default: true, null: false end @@ -111,9 +111,7 @@ with `validate: false` in a post-deployment migration, `db/post_migrate/20200501000001_add_not_null_constraint_to_epics_description.rb`: ```ruby -class AddNotNullConstraintToEpicsDescription < ActiveRecord::Migration[6.0] - include Gitlab::Database::MigrationHelpers - +class AddNotNullConstraintToEpicsDescription < Gitlab::Database::Migration[1.0] disable_ddl_transaction! def up @@ -144,9 +142,7 @@ so we are going to add a post-deployment migration for the 13.0 milestone (curre `db/post_migrate/20200501000002_cleanup_epics_with_null_description.rb`: ```ruby -class CleanupEpicsWithNullDescription < ActiveRecord::Migration[6.0] - include Gitlab::Database::MigrationHelpers - +class CleanupEpicsWithNullDescription < Gitlab::Database::Migration[1.0] # With BATCH_SIZE=1000 and epics.count=29500 on GitLab.com # - 30 iterations will be run # - each requires on average ~150ms @@ -184,9 +180,7 @@ migration helper in a final post-deployment migration, `db/post_migrate/20200601000001_validate_not_null_constraint_on_epics_description.rb`: ```ruby -class ValidateNotNullConstraintOnEpicsDescription < ActiveRecord::Migration[6.0] - include Gitlab::Database::MigrationHelpers - +class ValidateNotNullConstraintOnEpicsDescription < Gitlab::Database::Migration[1.0] disable_ddl_transaction! def up diff --git a/doc/development/database/pagination_guidelines.md b/doc/development/database/pagination_guidelines.md index b7209b4ca30..3a772b10a6d 100644 --- a/doc/development/database/pagination_guidelines.md +++ b/doc/development/database/pagination_guidelines.md @@ -302,7 +302,7 @@ LIMIT 20 ##### Tooling -A generic keyset pagination library is available within the GitLab project which can most of the cases easly replace the existing, kaminari based pagination with significant performance improvements when dealing with large datasets. +A generic keyset pagination library is available within the GitLab project which can most of the cases easily replace the existing, kaminari based pagination with significant performance improvements when dealing with large datasets. Example: diff --git a/doc/development/database/rename_database_tables.md b/doc/development/database/rename_database_tables.md index 8ac50d2c0a0..881adf00ad0 100644 --- a/doc/development/database/rename_database_tables.md +++ b/doc/development/database/rename_database_tables.md @@ -60,7 +60,7 @@ Consider the next release as "Release N.M". Execute a standard migration (not a post-migration): ```ruby - include Gitlab::Database::MigrationHelpers + enable_lock_retries! def up rename_table_safely(:issues, :tickets) @@ -96,8 +96,6 @@ At this point, we don't have applications using the old database table name in t 1. Remove the database view through a post-migration: ```ruby - include Gitlab::Database::MigrationHelpers - def up finalize_table_rename(:issues, :tickets) end diff --git a/doc/development/database/strings_and_the_text_data_type.md b/doc/development/database/strings_and_the_text_data_type.md index 688d811b897..a0dda42fdc7 100644 --- a/doc/development/database/strings_and_the_text_data_type.md +++ b/doc/development/database/strings_and_the_text_data_type.md @@ -11,11 +11,13 @@ info: To determine the technical writer assigned to the Stage/Group associated w When adding new columns that will be used to store strings or other textual information: 1. We always use the `text` data type instead of the `string` data type. -1. `text` columns should always have a limit set, either by using the `create_table_with_constraints` helper -when creating a table, or by using the `add_text_limit` when altering an existing table. +1. `text` columns should always have a limit set, either by using the `create_table` with +the `#text ... limit: 100` helper (see below) when creating a table, or by using the `add_text_limit` +when altering an existing table. -The `text` data type can not be defined with a limit, so `create_table_with_constraints` and `add_text_limit` enforce -that by adding a [check constraint](https://www.postgresql.org/docs/11/ddl-constraints.html) on the column. +The standard Rails `text` column type can not be defined with a limit, but we extend `create_table` to +add a `limit: 255` option. Outside of `create_table`, `add_text_limit` can be used to add a [check constraint](https://www.postgresql.org/docs/11/ddl-constraints.html) +to an already existing column. ## Background information @@ -41,36 +43,24 @@ Don't use text columns for `attr_encrypted` attributes. Use a ## Create a new table with text columns When adding a new table, the limits for all text columns should be added in the same migration as -the table creation. +the table creation. We add a `limit:` attribute to Rails' `#text` method, which allows adding a limit +for this column. For example, consider a migration that creates a table with two text columns, `db/migrate/20200401000001_create_db_guides.rb`: ```ruby -class CreateDbGuides < ActiveRecord::Migration[6.0] - include Gitlab::Database::MigrationHelpers - - def up - create_table_with_constraints :db_guides do |t| +class CreateDbGuides < Gitlab::Database::Migration[1.0] + def change + create_table :db_guides do |t| t.bigint :stars, default: 0, null: false - t.text :title - t.text :notes - - t.text_limit :title, 128 - t.text_limit :notes, 1024 + t.text :title, limit: 128 + t.text :notes, limit: 1024 end end - - def down - # No need to drop the constraints, drop_table takes care of everything - drop_table :db_guides - end end ``` -Note that the `create_table_with_constraints` helper uses the `with_lock_retries` helper -internally, so we don't need to manually wrap the method call in the migration. - ## Add a text column to an existing table Adding a column to an existing table requires an exclusive lock for that table. Even though that lock @@ -84,7 +74,7 @@ For example, consider a migration that adds a new text column `extended_title` t `db/migrate/20200501000001_add_extended_title_to_sprints.rb`: ```ruby -class AddExtendedTitleToSprints < ActiveRecord::Migration[6.0] +class AddExtendedTitleToSprints < Gitlab::Database::Migration[1.0] # rubocop:disable Migration/AddLimitToTextColumns # limit is added in 20200501000002_add_text_limit_to_sprints_extended_title @@ -99,8 +89,7 @@ A second migration should follow the first one with a limit added to `extended_t `db/migrate/20200501000002_add_text_limit_to_sprints_extended_title.rb`: ```ruby -class AddTextLimitToSprintsExtendedTitle < ActiveRecord::Migration[6.0] - include Gitlab::Database::MigrationHelpers +class AddTextLimitToSprintsExtendedTitle < Gitlab::Database::Migration[1.0] disable_ddl_transaction! def up @@ -175,9 +164,7 @@ in a post-deployment migration, `db/post_migrate/20200501000001_add_text_limit_migration.rb`: ```ruby -class AddTextLimitMigration < ActiveRecord::Migration[6.0] - include Gitlab::Database::MigrationHelpers - +class AddTextLimitMigration < Gitlab::Database::Migration[1.0] disable_ddl_transaction! def up @@ -208,9 +195,7 @@ to add a background migration for the 13.0 milestone (current), `db/post_migrate/20200501000002_schedule_cap_title_length_on_issues.rb`: ```ruby -class ScheduleCapTitleLengthOnIssues < ActiveRecord::Migration[6.0] - include Gitlab::Database::MigrationHelpers - +class ScheduleCapTitleLengthOnIssues < Gitlab::Database::Migration[1.0] # Info on how many records will be affected on GitLab.com # time each batch needs to run on average, etc ... BATCH_SIZE = 5000 @@ -255,9 +240,7 @@ helper in a final post-deployment migration, `db/post_migrate/20200601000001_validate_text_limit_migration.rb`: ```ruby -class ValidateTextLimitMigration < ActiveRecord::Migration[6.0] - include Gitlab::Database::MigrationHelpers - +class ValidateTextLimitMigration < Gitlab::Database::Migration[1.0] disable_ddl_transaction! def up diff --git a/doc/development/database/table_partitioning.md b/doc/development/database/table_partitioning.md index 207d5a733ce..5319c73aad0 100644 --- a/doc/development/database/table_partitioning.md +++ b/doc/development/database/table_partitioning.md @@ -173,7 +173,7 @@ An example migration of partitioning the `audit_events` table by its `created_at` column would look like: ```ruby -class PartitionAuditEvents < ActiveRecord::Migration[6.0] +class PartitionAuditEvents < Gitlab::Database::Migration[1.0] include Gitlab::Database::PartitioningMigrationHelpers def up @@ -200,7 +200,7 @@ into the partitioned copy. Continuing the above example, the migration would look like: ```ruby -class BackfillPartitionAuditEvents < ActiveRecord::Migration[6.0] +class BackfillPartitionAuditEvents < Gitlab::Database::Migration[1.0] include Gitlab::Database::PartitioningMigrationHelpers def up @@ -233,7 +233,7 @@ failed jobs. Once again, continuing the example, this migration would look like: ```ruby -class CleanupPartitionedAuditEventsBackfill < ActiveRecord::Migration[6.0] +class CleanupPartitionedAuditEventsBackfill < Gitlab::Database::Migration[1.0] include Gitlab::Database::PartitioningMigrationHelpers def up diff --git a/doc/development/database/transaction_guidelines.md b/doc/development/database/transaction_guidelines.md index 1c25496b153..4c586135015 100644 --- a/doc/development/database/transaction_guidelines.md +++ b/doc/development/database/transaction_guidelines.md @@ -12,7 +12,7 @@ For further reference please check PostgreSQL documentation about [transactions] ## Database decomposition and sharding -The [sharding group](https://about.gitlab.com/handbook/engineering/development/enablement/sharding) plans to split the main GitLab database and move some of the database tables to other database servers. +The [sharding group](https://about.gitlab.com/handbook/engineering/development/enablement/sharding/) plans to split the main GitLab database and move some of the database tables to other database servers. The group will start decomposing the `ci_*` related database tables first. To maintain the current application development experience, tooling and static analyzers will be added to the codebase to ensure correct data access and data modification methods. By using the correct form for defining database transactions, we can save significant refactoring work in the future. diff --git a/doc/development/database_debugging.md b/doc/development/database_debugging.md index 67ec1b3c4f1..b1c8508c884 100644 --- a/doc/development/database_debugging.md +++ b/doc/development/database_debugging.md @@ -25,6 +25,12 @@ If you just want to delete everything and start over with an empty DB (approxima bundle exec rake db:reset RAILS_ENV=development ``` +If you want to seed the empty DB with sample data (approximately 4 minutes): + +```shell +bundle exec rake dev:setup +``` + If you just want to delete everything and start over with sample data (approximately 4 minutes). This also does `db:reset` and runs DB-specific migrations: @@ -64,6 +70,36 @@ bundle exec rails db -e development - `SELECT * FROM schema_migrations WHERE version = '20170926203418';`: Check if a migration was run - `DELETE FROM schema_migrations WHERE version = '20170926203418';`: Manually remove a migration +## Access the database with a GUI + +Most GUIs (DataGrid, RubyMine, DBeaver) require a TCP connection to the database, but by default +the database runs on a UNIX socket. To be able to access the database from these tools, some steps +are needed: + +1. On the GDK root directory, run: + + ```shell + gdk config set postgresql.host localhost + ``` + +1. Open your `gdk.yml`, and confirm that it has the following lines: + + ```yaml + postgresql: + host: localhost + ``` + +1. Reconfigure GDK: + + ```shell + gdk reconfigure + ``` + +1. On your database GUI, select `localhost` as host, `5432` as port and `gitlabhq_development` as database. + Alternatively, you can use the connection string `postgresql://localhost:5432/gitlabhq_development`. + +The new connection should be working now. + ## Access the GDK database with Visual Studio Code Use these instructions for exploring the GitLab database while developing with the GDK: diff --git a/doc/development/database_review.md b/doc/development/database_review.md index 2746d9f6582..42bfa656a61 100644 --- a/doc/development/database_review.md +++ b/doc/development/database_review.md @@ -108,7 +108,7 @@ the following preparations into account. - Ensure the down method reverts the changes in `db/structure.sql`. - Update the migration output whenever you modify the migrations during the review process. - Add tests for the migration in `spec/migrations` if necessary. See [Testing Rails migrations at GitLab](testing_guide/testing_migrations_guide.md) for more details. -- When [high-traffic](https://gitlab.com/gitlab-org/gitlab/-/blob/master/rubocop/rubocop-migrations.yml#L3) tables are involved in the migration, use the [`with_lock_retries`](migration_style_guide.md#retry-mechanism-when-acquiring-database-locks) helper method. Review the relevant [examples in our documentation](migration_style_guide.md#examples) for use cases and solutions. +- When [high-traffic](https://gitlab.com/gitlab-org/gitlab/-/blob/master/rubocop/rubocop-migrations.yml#L3) tables are involved in the migration, use the [`enable_lock_retries`](migration_style_guide.md#retry-mechanism-when-acquiring-database-locks) method to enable lock-retries. Review the relevant [examples in our documentation](migration_style_guide.md#usage-with-transactional-migrations) for use cases and solutions. - Ensure RuboCop checks are not disabled unless there's a valid reason to. - When adding an index to a [large table](https://gitlab.com/gitlab-org/gitlab/-/blob/master/rubocop/rubocop-migrations.yml#L3), test its execution using `CREATE INDEX CONCURRENTLY` in the `#database-lab` Slack channel and add the execution time to the MR description: @@ -128,7 +128,9 @@ test its execution using `CREATE INDEX CONCURRENTLY` in the `#database-lab` Slac - Write the raw SQL in the MR description. Preferably formatted nicely with [pgFormatter](https://sqlformat.darold.net) or [paste.depesz.com](https://paste.depesz.com) and using regular quotes + <!-- vale off --> (for example, `"projects"."id"`) and avoiding smart quotes (for example, `“projects”.“id”`). + <!-- vale on --> - In case of queries generated dynamically by using parameters, there should be one raw SQL query for each variation. For example, a finder for issues that may take as a parameter an optional filter on projects, diff --git a/doc/development/deprecation_guidelines/index.md b/doc/development/deprecation_guidelines/index.md index 3543345aa34..f8ee29e6904 100644 --- a/doc/development/deprecation_guidelines/index.md +++ b/doc/development/deprecation_guidelines/index.md @@ -32,6 +32,6 @@ It also should be [deprecated in advance](https://about.gitlab.com/handbook/mark For API removals, see the [GraphQL](../../api/graphql/index.md#deprecation-and-removal-process) and [GitLab API](../../api/index.md#compatibility-guidelines) guidelines. -For configuration removals, see the [Omnibus deprecation policy](https://docs.gitlab.com/omnibus/package-information/deprecation_policy.html). +For configuration removals, see the [Omnibus deprecation policy](../../administration/package_information/deprecation_policy.md). For versioning and upgrade details, see our [Release and Maintenance policy](../../policy/maintenance.md). diff --git a/doc/development/documentation/feature_flags.md b/doc/development/documentation/feature_flags.md index b0fa6c3428c..5a4d365ed20 100644 --- a/doc/development/documentation/feature_flags.md +++ b/doc/development/documentation/feature_flags.md @@ -67,10 +67,12 @@ When the state of a flag changes (for example, disabled by default to enabled by Possible version history entries are: ```markdown -> - [Enabled on GitLab.com](issue-link) in GitLab X.X and is ready for production use. -> - [Enabled on GitLab.com](issue-link) in GitLab X.X and is ready for production use. Available to GitLab.com administrators only. -> - [Enabled with <flag name> flag](issue-link) for self-managed GitLab in GitLab X.X and is ready for production use. -> - [Feature flag <flag name> removed](issue-line) in GitLab X.X. +> - [Introduced](issue-link) in GitLab X.X. [Deployed behind the <flag name> flag](../../administration/feature_flags.md), disabled by default. +> - [Enabled on GitLab.com](issue-link) in GitLab X.X. +> - [Enabled on GitLab.com](issue-link) in GitLab X.X. Available to GitLab.com administrators only. +> - [Enabled on self-managed](issue-link) in GitLab X.X. +> - [Feature flag <flag name> removed](issue-link) in GitLab X.X. +> - [Generally available](issue-link) in GitLab X.X. ``` ## Feature flag documentation examples @@ -78,7 +80,7 @@ Possible version history entries are: The following examples show the progression of a feature flag. ```markdown -> Introduced in GitLab 13.7. +> Introduced in GitLab 13.7. [Deployed behind the `forti_token_cloud` flag](../../administration/feature_flags.md), disabled by default. FLAG: On self-managed GitLab, by default this feature is not available. To make it available, @@ -86,11 +88,11 @@ ask an administrator to [enable the `forti_token_cloud` flag](../administration/ The feature is not ready for production use. ``` -If it were to be updated in the future to enable its use in production, you can update the version history: +When the feature is enabled in production, you can update the version history: ```markdown -> - Introduced in GitLab 13.7. -> - [Enabled with `forti_token_cloud` flag](https://gitlab.com/issue/etc) for self-managed GitLab in GitLab X.X and ready for production use. +> - Introduced in GitLab 13.7. [Deployed behind the `forti_token_cloud` flag](../../administration/feature_flags.md), disabled by default. +> - [Enabled on self-managed](https://gitlab.com/issue/etc) GitLab 13.8. FLAG: On self-managed GitLab, by default this feature is available. To hide the feature per user, @@ -100,8 +102,9 @@ ask an administrator to [disable the `forti_token_cloud` flag](../administration And, when the feature is done and fully available to all users: ```markdown -> - Introduced in GitLab 13.7. -> - [Enabled on GitLab.com](https://gitlab.com/issue/etc) in GitLab X.X and is ready for production use. -> - [Enabled with `forti_token_cloud` flag](https://gitlab.com/issue/etc) for self-managed GitLab in GitLab X.X and is ready for production use. -> - [Feature flag `forti_token_cloud`](https://gitlab.com/issue/etc) removed in GitLab X.X. +> - Introduced in GitLab 13.7. [Deployed behind the `forti_token_cloud` flag](../../administration/feature_flags.md), disabled by default. +> - [Enabled on self-managed](https://gitlab.com/issue/etc) GitLab 13.8. +> - [Enabled on GitLab.com](https://gitlab.com/issue/etc) in GitLab 13.9. +> - [Feature flag `forti_token_cloud`](https://gitlab.com/issue/etc) removed in GitLab 14.0. +> - [Generally available](issue-link) in GitLab 14.0. ``` diff --git a/doc/development/documentation/index.md b/doc/development/documentation/index.md index 59a1b8c7b99..a597ea512c6 100644 --- a/doc/development/documentation/index.md +++ b/doc/development/documentation/index.md @@ -131,10 +131,10 @@ The following metadata should be added when a page is moved to another location: - `redirect_to`: The relative path and filename (with an `.md` extension) of the location to which visitors should be redirected for a moved page. - [Learn more](#move-or-rename-a-page). + [Learn more](redirects.md). - `disqus_identifier`: Identifier for Disqus commenting system. Used to keep comments with a page that's been moved to a new URL. - [Learn more](#redirections-for-pages-with-disqus-comments). + [Learn more](redirects.md#redirections-for-pages-with-disqus-comments). ### Comments metadata @@ -156,133 +156,7 @@ Nanoc layout), which is displayed at the top of the page if defined. ## Move or rename a page -Moving or renaming a document is the same as changing its location. Be sure to -assign a technical writer to any merge request that renames or moves a page. -Technical Writers can help with any questions and can review your change. - -When moving or renaming a page, you must redirect browsers to the new page. -This ensures users find the new page, and have the opportunity to update their -bookmarks. - -There are two types of redirects: - -- Redirect codes added into the documentation files themselves, for users who - view the docs in `/help` on self-managed instances. For example, - [`/help` on GitLab.com](https://gitlab.com/help). -- [GitLab Pages redirects](../../user/project/pages/redirects.md), - for users who view the docs on [`docs.gitlab.com`](https://docs.gitlab.com). - -The Technical Writing team manages the [process](https://gitlab.com/gitlab-org/technical-writing/-/blob/main/.gitlab/issue_templates/tw-monthly-tasks.md) -to regularly update the [`redirects.yaml`](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/content/_data/redirects.yaml) -file. - -To add a redirect: - -1. In the repository (`gitlab`, `gitlab-runner`, `omnibus-gitlab`, or `charts`), - create a new documentation file. Don't delete the old one. The easiest - way is to copy it. For example: - - ```shell - cp doc/user/search/old_file.md doc/api/new_file.md - ``` - -1. Add the redirect code to the old documentation file by running the - following Rake task. The first argument is the path of the old file, - and the second argument is the path of the new file: - - - To redirect to a page in the same project, use relative paths and - the `.md` extension. Both old and new paths start from the same location. - In the following example, both paths are relative to `doc/`: - - ```shell - bundle exec rake "gitlab:docs:redirect[doc/user/search/old_file.md, doc/api/new_file.md]" - ``` - - - To redirect to a page in a different project or site, use the full URL (with `https://`) : - - ```shell - bundle exec rake "gitlab:docs:redirect[doc/user/search/old_file.md, https://example.com]" - ``` - - Alternatively, you can omit the arguments and be asked to enter their values: - - ```shell - bundle exec rake gitlab:docs:redirect - ``` - - If you don't want to use the Rake task, you can use the following template. - However, the file paths must be relative to the `doc` or `docs` directory. - - Replace the value of `redirect_to` with the new file path and `YYYY-MM-DD` - with the date the file should be removed. - - Redirect files that link to docs in internal documentation projects - are removed after three months. Redirect files that link to external sites are - removed after one year: - - ```markdown - --- - redirect_to: '../newpath/to/file/index.md' - remove_date: 'YYYY-MM-DD' - --- - - This document was moved to [another location](../path/to/file/index.md). - - <!-- This redirect file can be deleted after <YYYY-MM-DD>. --> - <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> - ``` - -1. If the documentation page being moved has any Disqus comments, follow the steps - described in [Redirections for pages with Disqus comments](#redirections-for-pages-with-disqus-comments). -1. Open a merge request with your changes. If a documentation page - you're removing includes images that aren't used - with any other documentation pages, be sure to use your merge request to delete - those images from the repository. -1. Assign the merge request to a technical writer for review and merge. -1. Search for links to the old documentation file. You must find and update all - links that point to the old documentation file: - - - In <https://gitlab.com/gitlab-com/www-gitlab-com>, search for full URLs: - `grep -r "docs.gitlab.com/ee/path/to/file.html" .` - - In <https://gitlab.com/gitlab-org/gitlab-docs/-/tree/master/content/_data>, - search the navigation bar configuration files for the path with `.html`: - `grep -r "path/to/file.html" .` - - In any of the four internal projects, search for links in the docs - and codebase. Search for all variations, including full URL and just the path. - For example, go to the root directory of the `gitlab` project and run: - - ```shell - grep -r "docs.gitlab.com/ee/path/to/file.html" . - grep -r "path/to/file.html" . - grep -r "path/to/file.md" . - grep -r "path/to/file" . - ``` - - You may need to try variations of relative links, such as `../path/to/file` or - `../file` to find every case. - -### Redirections for pages with Disqus comments - -If the documentation page being relocated already has Disqus comments, -we need to preserve the Disqus thread. - -Disqus uses an identifier per page, and for <https://docs.gitlab.com>, the page identifier -is configured to be the page URL. Therefore, when we change the document location, -we need to preserve the old URL as the same Disqus identifier. - -To do that, add to the front matter the variable `disqus_identifier`, -using the old URL as value. For example, let's say we moved the document -available under `https://docs.gitlab.com/my-old-location/README.html` to a new location, -`https://docs.gitlab.com/my-new-location/index.html`. - -Into the **new document** front matter, we add the following information. You must -include the filename in the `disqus_identifier` URL, even if it's `index.html` or `README.html`. - -```yaml ---- -disqus_identifier: 'https://docs.gitlab.com/my-old-location/README.html' ---- -``` +See [redirects](redirects.md). ## Merge requests for GitLab documentation @@ -405,76 +279,7 @@ on how the left-side navigation menu is built and updated. ## Previewing the changes live -NOTE: -To preview your changes to documentation locally, follow this -[development guide](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/README.md#development-when-contributing-to-gitlab-documentation) or [these instructions for GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/main/doc/howto/gitlab_docs.md). - -The live preview is currently enabled for the following projects: - -- [`gitlab`](https://gitlab.com/gitlab-org/gitlab) -- [`omnibus-gitlab`](https://gitlab.com/gitlab-org/omnibus-gitlab) -- [`gitlab-runner`](https://gitlab.com/gitlab-org/gitlab-runner) - -If your merge request has docs changes, you can use the manual `review-docs-deploy` job -to deploy the docs review app for your merge request. - - - -You must push a branch to those repositories, as it doesn't work for forks. - -The `review-docs-deploy*` job: - -1. Triggers a cross project pipeline and build the docs site with your changes. - -In case the review app URL returns 404, this means that either the site is not -yet deployed, or something went wrong with the remote pipeline. Give it a few -minutes and it should appear online, otherwise you can check the status of the -remote pipeline from the link in the merge request's job output. -If the pipeline failed or got stuck, drop a line in the `#docs` chat channel. - -NOTE: -Someone with no merge rights to the GitLab projects (think of forks from -contributors) cannot run the manual job. In that case, you can -ask someone from the GitLab team who has the permissions to do that for you. - -### Troubleshooting review apps - -In case the review app URL returns 404, follow these steps to debug: - -1. **Did you follow the URL from the merge request widget?** If yes, then check if - the link is the same as the one in the job output. -1. **Did you follow the URL from the job output?** If yes, then it means that - either the site is not yet deployed or something went wrong with the remote - pipeline. Give it a few minutes and it should appear online, otherwise you - can check the status of the remote pipeline from the link in the job output. - If the pipeline failed or got stuck, drop a line in the `#docs` chat channel. - -### Technical aspects - -If you want to know the in-depth details, here's what's really happening: - -1. You manually run the `review-docs-deploy` job in a merge request. -1. The job runs the [`scripts/trigger-build`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/scripts/trigger-build) - script with the `docs deploy` flag, which triggers the "Triggered from `gitlab-org/gitlab` 'review-docs-deploy' job" - pipeline trigger in the `gitlab-org/gitlab-docs` project for the `$DOCS_BRANCH` (defaults to `main`). -1. The preview URL is shown both at the job output and in the merge request - widget. You also get the link to the remote pipeline. -1. In the `gitlab-org/gitlab-docs` project, the pipeline is created and it - [skips the test jobs](https://gitlab.com/gitlab-org/gitlab-docs/blob/8d5d5c750c602a835614b02f9db42ead1c4b2f5e/.gitlab-ci.yml#L50-55) - to lower the build time. -1. Once the docs site is built, the HTML files are uploaded as artifacts. -1. A specific runner tied only to the docs project, runs the Review App job - that downloads the artifacts and uses `rsync` to transfer the files over - to a location where NGINX serves them. - -The following GitLab features are used among others: - -- [Manual jobs](../../ci/jobs/job_control.md#create-a-job-that-must-be-run-manually) -- [Multi project pipelines](../../ci/pipelines/multi_project_pipelines.md) -- [Review Apps](../../ci/review_apps/index.md) -- [Artifacts](../../ci/yaml/index.md#artifacts) -- [Specific runner](../../ci/runners/runners_scope.md#prevent-a-specific-runner-from-being-enabled-for-other-projects) -- [Pipelines for merge requests](../../ci/pipelines/merge_request_pipelines.md) +See how you can use review apps to [preview your changes live](review_apps.md). ## Testing diff --git a/doc/development/documentation/redirects.md b/doc/development/documentation/redirects.md new file mode 100644 index 00000000000..eb6878f5870 --- /dev/null +++ b/doc/development/documentation/redirects.md @@ -0,0 +1,155 @@ +--- +stage: none +group: Documentation Guidelines +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +description: Learn how to contribute to GitLab Documentation. +--- + +<!--- + The clean_redirects Rake task in the gitlab-docs repository manually + excludes this file. If the line containing remove_date is moved to a new + document, update the Rake task with the new location. + + https://gitlab.com/gitlab-org/gitlab-docs/-/blob/1979f985708d64558bb487fbe9ed5273729c01b7/Rakefile#L306 +---> + +# Redirects in GitLab documentation + +Moving or renaming a document is the same as changing its location. Be sure +to assign a technical writer to any merge request that renames or moves a page. +Technical Writers can help with any questions and can review your change. + +When moving or renaming a page, you must redirect browsers to the new page. +This ensures users find the new page, and have the opportunity to update their +bookmarks. + +There are two types of redirects: + +- Redirect added into the documentation files themselves, for users who + view the docs in `/help` on self-managed instances. For example, + [`/help` on GitLab.com](https://gitlab.com/help). +- [GitLab Pages redirects](../../user/project/pages/redirects.md), + for users who view the docs on [`docs.gitlab.com`](https://docs.gitlab.com). + + The Technical Writing team manages the [process](https://gitlab.com/gitlab-org/technical-writing/-/blob/main/.gitlab/issue_templates/tw-monthly-tasks.md) + to regularly update and [clean up the redirects](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/doc/raketasks.md#clean-up-redirects). + If you're a contributor, you may add a new redirect, but you don't need to delete + the old ones. This process is automatic and handled by the Technical + Writing team. + +NOTE: +If the old page you're renaming doesn't exist in a stable branch, skip the +following steps and ask a Technical Writer to add the redirect in +[`redirects.yaml`](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/content/_data/redirects.yaml). +For example, if you add a new page on the 3rd of the month and then rename it before it gets +added in the stable branch on the 18th, the old page will never be part of the internal `/help`. +In that case, you can jump straight to the +[Pages redirect](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/doc/maintenance.md#pages-redirects). + +To add a redirect: + +1. In the repository (`gitlab`, `gitlab-runner`, `omnibus-gitlab`, or `charts`), + create a new documentation file. Don't delete the old one. The easiest + way is to copy it. For example: + + ```shell + cp doc/user/search/old_file.md doc/api/new_file.md + ``` + +1. Add the redirect code to the old documentation file by running the + following Rake task. The first argument is the path of the old file, + and the second argument is the path of the new file: + + - To redirect to a page in the same project, use relative paths and + the `.md` extension. Both old and new paths start from the same location. + In the following example, both paths are relative to `doc/`: + + ```shell + bundle exec rake "gitlab:docs:redirect[doc/user/search/old_file.md, doc/api/new_file.md]" + ``` + + - To redirect to a page in a different project or site, use the full URL (with `https://`) : + + ```shell + bundle exec rake "gitlab:docs:redirect[doc/user/search/old_file.md, https://example.com]" + ``` + + Alternatively, you can omit the arguments and be asked to enter their values: + + ```shell + bundle exec rake gitlab:docs:redirect + ``` + + If you don't want to use the Rake task, you can use the following template. + However, the file paths must be relative to the `doc` or `docs` directory. + + Replace the value of `redirect_to` with the new file path and `YYYY-MM-DD` + with the date the file should be removed. + + Redirect files that link to docs in internal documentation projects + are removed after three months. Redirect files that link to external sites are + removed after one year: + + ```markdown + --- + redirect_to: '../newpath/to/file/index.md' + remove_date: 'YYYY-MM-DD' + --- + + This document was moved to [another location](../path/to/file/index.md). + + <!-- This redirect file can be deleted after <YYYY-MM-DD>. --> + <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> + ``` + +1. If the documentation page being moved has any Disqus comments, follow the steps + described in [Redirections for pages with Disqus comments](#redirections-for-pages-with-disqus-comments). +1. Open a merge request with your changes. If a documentation page + you're removing includes images that aren't used + with any other documentation pages, be sure to use your merge request to delete + those images from the repository. +1. Assign the merge request to a technical writer for review and merge. +1. Search for links to the old documentation file. You must find and update all + links that point to the old documentation file: + + - In <https://gitlab.com/gitlab-com/www-gitlab-com>, search for full URLs: + `grep -r "docs.gitlab.com/ee/path/to/file.html" .` + - In <https://gitlab.com/gitlab-org/gitlab-docs/-/tree/master/content/_data>, + search the navigation bar configuration files for the path with `.html`: + `grep -r "path/to/file.html" .` + - In any of the four internal projects, search for links in the docs + and codebase. Search for all variations, including full URL and just the path. + For example, go to the root directory of the `gitlab` project and run: + + ```shell + grep -r "docs.gitlab.com/ee/path/to/file.html" . + grep -r "path/to/file.html" . + grep -r "path/to/file.md" . + grep -r "path/to/file" . + ``` + + You may need to try variations of relative links, such as `../path/to/file` or + `../file` to find every case. + +## Redirections for pages with Disqus comments + +If the documentation page being relocated already has Disqus comments, +we need to preserve the Disqus thread. + +Disqus uses an identifier per page, and for <https://docs.gitlab.com>, the page identifier +is configured to be the page URL. Therefore, when we change the document location, +we need to preserve the old URL as the same Disqus identifier. + +To do that, add to the front matter the variable `disqus_identifier`, +using the old URL as value. For example, let's say we moved the document +available under `https://docs.gitlab.com/my-old-location/README.html` to a new location, +`https://docs.gitlab.com/my-new-location/index.html`. + +Into the **new document** front matter, we add the following information. You must +include the filename in the `disqus_identifier` URL, even if it's `index.html` or `README.html`. + +```yaml +--- +disqus_identifier: 'https://docs.gitlab.com/my-old-location/README.html' +--- +``` diff --git a/doc/development/documentation/review_apps.md b/doc/development/documentation/review_apps.md new file mode 100644 index 00000000000..2b8c412f165 --- /dev/null +++ b/doc/development/documentation/review_apps.md @@ -0,0 +1,101 @@ +--- +stage: none +group: Documentation Guidelines +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +description: Learn how documentation review apps work. +--- + +# Documentation review apps + +If you're a GitLab team member and your merge request contains documentation changes, you can use a review app to preview +how they would look if they were deployed to the [GitLab Docs site](https://docs.gitlab.com). + +Review apps are enabled for the following projects: + +- [GitLab](https://gitlab.com/gitlab-org/gitlab) +- [Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab) +- [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) +- [GitLab Charts](https://gitlab.com/gitlab-org/charts/gitlab) + +Alternatively, check the [`gitlab-docs` development guide](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/README.md#development-when-contributing-to-gitlab-documentation) +or [the GDK documentation](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/main/doc/howto/gitlab_docs.md) +to render and preview the documentation locally. + +## How to trigger a review app + +If a merge request has documentation changes, use the `review-docs-deploy` manual job +to deploy the documentation review app for your merge request. + + + +The `review-docs-deploy*` job triggers a cross project pipeline and builds the +docs site with your changes. When the pipeline finishes, the review app URL +appears in the merge request widget. Use it to navigate to your changes. + +You must have the Developer role in the project. Users without the Developer role, such +as external contributors, cannot run the manual job. In that case, ask someone from +the GitLab team to run the job. + +## Technical aspects + +If you want to know the in-depth details, here's what's really happening: + +1. You manually run the `review-docs-deploy` job in a merge request. +1. The job runs the [`scripts/trigger-build`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/scripts/trigger-build) + script with the `docs deploy` flag, which triggers the "Triggered from `gitlab-org/gitlab` 'review-docs-deploy' job" + pipeline trigger in the `gitlab-org/gitlab-docs` project for the `$DOCS_BRANCH` (defaults to `main`). +1. The preview URL is shown both at the job output and in the merge request + widget. You also get the link to the remote pipeline. +1. In the `gitlab-org/gitlab-docs` project, the pipeline is created and it + [skips the test jobs](https://gitlab.com/gitlab-org/gitlab-docs/blob/8d5d5c750c602a835614b02f9db42ead1c4b2f5e/.gitlab-ci.yml#L50-55) + to lower the build time. +1. Once the docs site is built, the HTML files are uploaded as artifacts. +1. A specific runner tied only to the docs project, runs the Review App job + that downloads the artifacts and uses `rsync` to transfer the files over + to a location where NGINX serves them. + +The following GitLab features are used among others: + +- [Manual jobs](../../ci/jobs/job_control.md#create-a-job-that-must-be-run-manually) +- [Multi project pipelines](../../ci/pipelines/multi_project_pipelines.md) +- [Review Apps](../../ci/review_apps/index.md) +- [Artifacts](../../ci/yaml/index.md#artifacts) +- [Specific runner](../../ci/runners/runners_scope.md#prevent-a-specific-runner-from-being-enabled-for-other-projects) +- [Pipelines for merge requests](../../ci/pipelines/merge_request_pipelines.md) + +## Troubleshooting review apps + +### Review app returns a 404 error + +If the review app URL returns a 404 error, either the site is not +yet deployed, or something went wrong with the remote pipeline. You can: + +- Wait a few minutes and it should appear online. +- Check the manual job's log and verify the URL. If the URL is different, try the + one from the job log. +- Check the status of the remote pipeline from the link in the merge request's job output. + If the pipeline failed or got stuck, GitLab team members can ask for help in the `#docs` + chat channel. Contributors can ping a technical writer in the merge request. + +### Not enough disk space + +Sometimes the review app server is full and there is no more disk space. Each review +app takes about 570MB of disk space. + +A cron job to remove review apps older than 20 days runs hourly, +but the disk space still occasionally fills up. To manually free up more space, +a GitLab technical writing team member can: + +1. Navigate to the [`gitlab-docs` schedules page](https://gitlab.com/gitlab-org/gitlab-docs/-/pipeline_schedules). +1. Select the play button for the `Remove old review apps from review app server` + schedule. By default, this cleans up review apps older than 14 days. +1. Navigate to the [pipelines page](https://gitlab.com/gitlab-org/gitlab-docs/-/pipelines) + and start the manual job called `clean-pages`. + +If the job says no review apps were found in that period, edit the `CLEAN_REVIEW_APPS_DAYS` +variable in the schedule, and repeat the process above. Gradually decrease the variable +until the free disk space reaches an acceptable amount (for example, 3GB). +Remember to set it to 14 again when you're done. + +There's an issue to [migrate from the DigitalOcean server to GCP buckets](https://gitlab.com/gitlab-org/gitlab-docs/-/issues/735)), +which should solve the disk space problem. diff --git a/doc/development/documentation/site_architecture/index.md b/doc/development/documentation/site_architecture/index.md index 046de5c6d86..cd69154217c 100644 --- a/doc/development/documentation/site_architecture/index.md +++ b/doc/development/documentation/site_architecture/index.md @@ -33,7 +33,6 @@ from where content is sourced, the `gitlab-docs` project, and the published outp D --> E E -- Build pipeline --> F F[docs.gitlab.com] - G[/ce/] H[/ee/] I[/runner/] J[/omnibus/] @@ -42,7 +41,6 @@ from where content is sourced, the `gitlab-docs` project, and the published outp F --> I F --> J F --> K - H -- symlink --> G ``` GitLab docs content isn't kept in the `gitlab-docs` repository. @@ -54,15 +52,6 @@ product, and all together are pulled to generate the docs website: - [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner/-/tree/main/docs) - [GitLab Chart](https://gitlab.com/charts/gitlab/tree/master/doc) -NOTE: -In September 2019, we [moved towards a single codebase](https://gitlab.com/gitlab-org/gitlab/-/issues/2952), -as such the docs for CE and EE are now identical. For historical reasons and -in order not to break any existing links throughout the internet, we still -maintain the CE docs (`https://docs.gitlab.com/ce/`), although it is hidden -from the website, and is now a symlink to the EE docs. When -[Support wildcard redirects](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/500) is resolved, -we can remove this completely. - ## Assets To provide an optimized site structure, design, and a search-engine friendly diff --git a/doc/development/documentation/styleguide/index.md b/doc/development/documentation/styleguide/index.md index 4e548179b9e..2cbecc91b20 100644 --- a/doc/development/documentation/styleguide/index.md +++ b/doc/development/documentation/styleguide/index.md @@ -420,6 +420,11 @@ Some contractions, however, should be avoided: | Requests to localhost are not allowed. | Requests to localhost aren't allowed. | | Specified URL cannot be used. | Specified URL can't be used. | +### Acronyms + +If you use an acronym, spell it out on first use on a page. You do not need to spell it out more than once on a page. +When possible, try to avoid acronyms in headings. + ## Text - [Write in Markdown](#markdown). @@ -438,8 +443,21 @@ Some contractions, however, should be avoided: - List item 2 ``` +### Comments + +To embed comments within Markdown, use standard HTML comments that are not rendered +when published. Example: + +```html +<!-- This is a comment that is not rendered --> +``` + ### Emphasis +Use **bold** rather than italic to provide emphasis. GitLab uses a sans-serif font and italic text does not stand out as much as it would in a serif font. For details, see [Butterick's Practical Typography guide on bold or italic](https://practicaltypography.com/bold-or-italic.html). + +You can use italics when you are introducing a term for the first time. Otherwise, use bold. + - Use double asterisks (`**`) to mark a word or text in bold (`**bold**`). - Use underscore (`_`) for text in italics (`_italic_`). - Use greater than (`>`) for blockquotes. @@ -460,6 +478,7 @@ Follow these guidelines for punctuation: | Use serial commas (Oxford commas) before the final **and** or **or** in a list of three or more items. (Tested in [`OxfordComma.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/OxfordComma.yml).) | You can create new issues, merge requests, and milestones. | | Always add a space before and after dashes when using it in a sentence (for replacing a comma, for example). | You should try this - or not. | | When a colon is part of a sentence, always use lowercase after the colon. | Linked issues: a way to create a relationship between issues. | +| Do not use typographer's quotes. Use straight quotes instead. (Tested in [`NonStandardQuotes.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/NonStandardQuotes.yml).) | "It's the questions we can't answer that teach us the most"---Patrick Rothfuss | <!-- vale gitlab.Repetition = YES --> @@ -751,6 +770,7 @@ Valid for Markdown content only, not for front matter entries: For other punctuation rules, refer to the [Pajamas Design System Punctuation section](https://design.gitlab.com/content/punctuation/). +This is overridden by the [documentation-specific punctuation rules](#punctuation). ## Headings @@ -1003,9 +1023,23 @@ document to ensure it links to the most recent version of the file. ## Navigation -When documenting navigation through the user interface, use these terms and styles. +When documenting how to navigate through the GitLab UI: + +- Always use location, then action. + - `From the **Visibility** list,` (location) `select **Public**.` (action) +- Be brief and specific. For example: + - Avoid: `Select **Save** for the changes to take effect.` + - Use instead: `Select **Save**.` +- When selecting from high-level UI elements, use the word **on**. + - Avoid: `From the left sidebar...` or `In the left sidebar...` + - Use instead: `On the left sidebar...` +- If a step must include a reason, start the step with it. + - Avoid: `Select the link in the merge request to view the changes.` + - Use instead: `To view the changes, select the link in the merge request.` +- If a step is optional, start the step with the word `Optional` followed by a period. + - `1. Optional. Enter a name for the dog.` -### What to call the menus +### Names for menus Use these terms when referring to the main GitLab user interface elements: @@ -1017,9 +1051,14 @@ elements: - **Right sidebar**: This is the navigation sidebar on the right of the user interface, specific to the open issue, merge request, or epic. -### How to document the menus +### Names for UI elements -To be consistent, use this format when you write about UI navigation. +UI elements, like button and checkbox names, should be **bold**. +Guidance for each individual UI element is in [the word list](word_list.md). + +### How to write navigation task steps + +To be consistent, use this format when you write navigation steps in a task topic. 1. On the top bar, select **Menu > Projects** and find your project. 1. On the left sidebar, select **Settings > CI/CD**. @@ -1034,20 +1073,27 @@ Another example: An Admin Area example: ```markdown -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. ``` -This text renders this output: +To select your avatar: -1. On the top bar, select **Menu >** **{admin}** **Admin**. +```markdown +1. On the top bar, in the top right corner, select your avatar. +``` ## Images Images, including screenshots, can help a reader better understand a concept. -However, they can be hard to maintain, and should be used sparingly. +However, they should be used sparingly because: -Before including an image in the documentation, ensure it provides value to the -reader. +- They tend to become out-of-date. +- They are difficult and expensive to localize. +- They cannot be read by screen readers. + +If you do include an image in the documentation, ensure it provides value. +Don't use `lorem ipsum` text. Try to replicate how the feature would be +used in a real-world scenario, and [use realistic text](#fake-user-information). ### Capture the image @@ -1106,7 +1152,7 @@ known tool is [`pngquant`](https://pngquant.org/), which is cross-platform and open source. Install it by visiting the official website and following the instructions for your OS. -GitLab has a [Rake task](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/tasks/pngquant.rake) +GitLab has a [Ruby script](https://gitlab.com/gitlab-org/gitlab/-/blob/master/bin/pngquant) that you can use to automate the process. In the root directory of your local copy of `https://gitlab.com/gitlab-org/gitlab`, run in a terminal: @@ -1114,19 +1160,26 @@ copy of `https://gitlab.com/gitlab-org/gitlab`, run in a terminal: been compressed: ```shell - bundle exec rake pngquant:lint + bin/pngquant lint ``` - Compress all documentation PNG images using `pngquant`: ```shell - bundle exec rake pngquant:compress + bin/pngquant compress + ``` + +- Compress specific files: + + ```shell + bin/pngquant compress doc/user/img/award_emoji_select.png doc/user/img/markdown_logo.png ``` -The only caveat is that the task runs on all images under `doc/`, not only the -ones you might have included in a merge request. In that case, you can run the -compress task and only commit the images that are relevant to your merge -request. +- Compress all PNG files in a specific directory: + + ```shell + bin/pngquant compress doc/user/img + ``` ## Videos @@ -1288,64 +1341,22 @@ For a complete reference on code blocks, see the [Kramdown guide](https://about. > [Introduced](https://gitlab.com/gitlab-org/gitlab-docs/-/issues/384) in GitLab 12.7. You can use icons from the [GitLab SVG library](https://gitlab-org.gitlab.io/gitlab-svgs/) -directly in the documentation. - -This way, you can achieve a consistent look when writing about interacting with -GitLab user interface elements. - -Usage examples: - -- Icon with default size (16px): `**{icon-name}**` +directly in the documentation. For example, `**{tanuki}**` renders as: **{tanuki}**. - Example: `**{tanuki}**` renders as: **{tanuki}**. -- Icon with custom size: `**{icon-name, size}**` +In most cases, you should avoid using the icons in text. +However, you can use an icon when hover text is the only +available way to describe a UI element. For example, **Delete** or **Edit** buttons +often have hover text only. - Available sizes (in pixels): 8, 10, 12, 14, 16, 18, 24, 32, 48, and 72 +When you do use an icon, start with the hover text and follow it with the SVG reference in parentheses. - Example: `**{tanuki, 24}**` renders as: **{tanuki, 24}**. -- Icon with custom size and class: `**{icon-name, size, class-name}**`. +- Avoid: `Select **{pencil}** **Edit**.` This generates as: Select **{pencil}** **Edit**. +- Use instead: `Select **Edit** (**{pencil}**).` This generates as: Select **Edit** (**{pencil}**). - You can access any class available to this element in GitLab documentation CSS. +Do not use words to describe the icon: - Example with `float-right`, a - [Bootstrap utility class](https://getbootstrap.com/docs/4.4/utilities/float/): - `**{tanuki, 32, float-right}**` renders as: **{tanuki, 32, float-right}** - -### When to use icons - -Icons should be used sparingly, and only in ways that aid and do not hinder the -readability of the text. - -For example, this Markdown adds little to the accompanying text: - -```markdown -1. Go to **{home}** **Project information > Details**. -``` - -1. Go to **{home}** **Project information > Details**. - -However, these tables might help the reader connect the text to the user -interface: - -```markdown -| Section | Description | -|:-------------------------|:----------------------------------------------------------------------------------------------------------------------------| -| **{overview}** Overview | View your GitLab Dashboard, and administer projects, users, groups, jobs, runners, and Gitaly servers. | -| **{monitor}** Monitoring | View GitLab system information, and information on background jobs, logs, health checks, requests profiles, and audit events. | -| **{messages}** Messages | Send and manage broadcast messages for your users. | -``` - -| Section | Description | -|:-------------------------|:----------------------------------------------------------------------------------------------------------------------------| -| **{overview}** Overview | View your GitLab Dashboard, and administer projects, users, groups, jobs, runners, and Gitaly servers. | -| **{monitor}** Monitoring | View GitLab system information, and information on background jobs, logs, health checks, requests profiles, and audit events. | -| **{messages}** Messages | Send and manage broadcast messages for your users. | - -Use an icon when you find yourself having to describe an interface element. For -example: - -- Do: Select the Admin Area icon ( **{admin}** ). -- Don't: Select the Admin Area icon (the wrench icon). +- Avoid: `Select **Erase job log** (the trash icon).` +- Use instead: `Select **Erase job log** (**{remove}**).` This generates as: Select **Erase job log** (**{remove}**). ## Alert boxes @@ -1456,27 +1467,9 @@ Follow these styles when you're describing user interface elements in an application: - For elements with a visible label, use that label in bold with matching case. - For example, `the **Cancel** button`. + For example, `Select **Cancel**`. - For elements with a tooltip or hover label, use that label in bold with - matching case. For example, `the **Add status emoji** button`. - -### Verbs for UI elements - -Use these verbs for specific uses with user interface -elements: - -| Recommended | Used for | Replaces | -|:--------------------|:--------------------------------------|:----------------------| -| select | buttons, links, menu items, dropdowns | click, press, hit | -| select or clear | checkboxes | enable, click, press | -| expand | expandable sections | open | -| turn on or turn off | toggles | flip, enable, disable | - -### Other Verbs - -| Recommended | Used for | Replaces | -|:------------|:--------------------------------|:----------------------| -| go to | making a browser go to location | navigate to, open | + matching case. For example, `Select **Add status emoji**`. ## GitLab versions @@ -1504,10 +1497,6 @@ tagged and released set of documentation for your installed version: When a feature is added or updated, you can include its version information either as a **Version history** item or as an inline text reference. -Version text shouldn't include information about the tier in which the feature -is available. This information is provided by the [product badge](#product-tier-badges) -displayed for the page or feature. - #### Version text in the **Version History** If all content in a section is related, add version text after the header for @@ -1523,6 +1512,10 @@ the section. The version information must: - Whenever possible, include a link to the completed issue, merge request, or epic that introduced the feature. An issue is preferred over a merge request, and a merge request is preferred over an epic. +- Do not include information about the tier, unless documenting a tier change + (for example, `Feature X [moved](issue-link) to Premium in GitLab 19.2`). +- Do not link to the pricing page. + The tier is provided by the [product badge](#product-tier-badges) on the heading. ```markdown ## Feature name @@ -1647,37 +1640,24 @@ When names change, it is more complicated to search or grep text that has line b ### Product tier badges -Tier badges are displayed as orange text next to a heading. For example: +Tier badges are displayed as orange text next to a heading. These badges link to the GitLab +pricing page. For example:  You must assign a tier badge: -- To [all H1 topic headings](#product-tier-badges-on-headings). +- To all H1 topic headings. - To topic headings that don't apply to the same tier as the H1. -- To [sections of a topic](#product-tier-badges-on-other-content), - if they apply to a tier other than what applies to the H1. - -#### Product tier badges on headings -To add a tier badge to a heading, add the relevant [tier badge](#available-product-tier-badges) +To add a tier badge to a heading, add the relevant tier badge after the heading text. For example: ```markdown # Heading title **(FREE)** ``` -#### Product tier badges on other content - -In paragraphs, list names, and table cells, an information icon displays when you -add a tier badge. More verbose information displays when a user points to the icon: - -- `**(FREE)**` displays as **(FREE)** -- `**(FREE SELF)**` displays as **(FREE SELF)** -- `**(FREE SAAS)**` displays as **(FREE SAAS)** - -The `**(FREE)**` generates a `span` element to trigger the -badges and tooltips (`<span class="badge-trigger free">`). +Do not add tier badges inline with other text. The single source of truth for a feature should be the heading where the functionality is described. #### Available product tier badges diff --git a/doc/development/documentation/styleguide/word_list.md b/doc/development/documentation/styleguide/word_list.md index 9e921bb30f0..eafe0e7a1c2 100644 --- a/doc/development/documentation/styleguide/word_list.md +++ b/doc/development/documentation/styleguide/word_list.md @@ -17,15 +17,17 @@ For guidance not on this page, we defer to these style guides: <!-- vale off --> <!-- markdownlint-disable --> -## @mention +## `@mention` -Try to avoid. Say "mention" instead, and consider linking to the +Try to avoid **`@mention`**. Say **mention** instead, and consider linking to the [mentions topic](../../../user/project/issues/issue_data_and_actions.md#mentions). -Don't use `code formatting`. +Don't use backticks. ## above -Try to avoid extra words when referring to an example or table in a documentation page, but if required, use **previously** instead. +Try to avoid using **above** when referring to an example or table in a documentation page. If required, use **previous** instead. For example: + +- In the previous example, the dog had fleas. ## admin, admin area @@ -33,55 +35,111 @@ Use **administration**, **administrator**, **administer**, or **Admin Area** ins ## allow, enable -Try to avoid, unless you are talking about security-related features. For example: +Try to avoid **allow** and **enable**, unless you are talking about security-related features. For example: -- Avoid: This feature allows you to create a pipeline. -- Use instead: Use this feature to create a pipeline. +- Do: Use this feature to create a pipeline. +- Do not: This feature allows you to create a pipeline. This phrasing is more active and is from the user perspective, rather than the person who implemented the feature. [View details in the Microsoft style guide](https://docs.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/a/allow-allows). ## Alpha -Uppercase. For example: **The XYZ feature is in Alpha.** or **This Alpha release is ready to test.** +Use uppercase for **Alpha**. For example: **The XYZ feature is in Alpha.** or **This Alpha release is ready to test.** You might also want to link to [this section](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha-beta-ga) in the handbook when writing about Alpha features. ## and/or -Instead of **and/or**, use or or rewrite the sentence to spell out both options. +Instead of **and/or**, use **or** or rewrite the sentence to spell out both options. + +## area + +Use [**section**](#section) instead of **area**. The only exception is [the Admin Area](#admin-admin-area). ## below -Try to avoid extra words when referring to an example or table in a documentation page, but if required, use **following** instead. +Try to avoid **below** when referring to an example or table in a documentation page. If required, use **following** instead. For example: + +- In the following example, the dog has fleas. ## Beta -Uppercase. For example: **The XYZ feature is in Beta.** or **This Beta release is ready to test.** +Use uppercase for **Beta**. For example: **The XYZ feature is in Beta.** or **This Beta release is ready to test.** You might also want to link to [this section](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha-beta-ga) in the handbook when writing about Beta features. ## blacklist -Do not use. Another option is **denylist**. ([Vale](../testing.md#vale) rule: [`InclusionCultural.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/InclusionCultural.yml)) +Do not use **blacklist**. Another option is **denylist**. ([Vale](../testing.md#vale) rule: [`InclusionCultural.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/InclusionCultural.yml)) ## board Use lowercase for **boards**, **issue boards**, and **epic boards**. +## box + +Use **text box** to refer to the UI field. Do not use **field** or **box**. For example: + +- In the **Variable name** text box, enter `my text`. + +## button + +Don't use a descriptor with **button**. + +- Do: Select **Run pipelines**. +- Do not: Select the **Run pipelines** button. + +## cannot, can not + +Use **cannot** instead of **can not**. You can also use **can't**. + +See also [contractions](index.md#contractions). + ## checkbox -One word, **checkbox**. Do not use **check box**. +Use one word for **checkbox**. Do not use **check box**. + +You **select** (not **check** or **enable**) and **clear** (not **deselect** or **disable**) checkboxes. +For example: + +- Select the **Protect environment** checkbox. +- Clear the **Protect environment** checkbox. + +If you must refer to the checkbox, you can say it is selected or cleared. For example: + +- Ensure the **Protect environment** checkbox is cleared. +- Ensure the **Protect environment** checkbox is selected. ## CI/CD -Always uppercase. No need to spell out on first use. +CI/CD is always uppercase. No need to spell it out on first use. + +## click + +Do not use **click**. Instead, use **select** with buttons, links, menu items, and lists. +**Select** applies to more devices, while **click** is more specific to a mouse. + +## collapse + +Use **collapse** instead of **close** when you are talking about expanding or collapsing a section in the UI. + +## confirmation dialog + +Use **confirmation dialog** to describe the dialog box that asks you to confirm your action. For example: + +- On the confirmation dialog, select **OK**. ## currently -Do not use when talking about the product or its features. The documentation describes the product as it is today. ([Vale](../testing.md#vale) rule: [`CurrentStatus.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/CurrentStatus.yml)) +Do not use **currently** when talking about the product or its features. The documentation describes the product as it is today. +([Vale](../testing.md#vale) rule: [`CurrentStatus.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/CurrentStatus.yml)) + +## deploy board + +Use lowercase for **deploy board**. ## Developer @@ -92,26 +150,35 @@ When writing about the Developer role: - Do not use the phrase, **if you are a developer** to mean someone who is assigned the Developer role. Instead, write it out. For example, **if you are assigned the Developer role**. - To describe a situation where the Developer role is the minimum required: - - Avoid: **the Developer role or higher** - - Use instead: **at least the Developer role** + - Avoid: the Developer role or higher + - Use instead: at least the Developer role Do not use **Developer permissions**. A user who is assigned the Developer role has a set of associated permissions. ## disable -See [the Microsoft style guide](https://docs.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/d/disable-disabled) for guidance. +See [the Microsoft style guide](https://docs.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/d/disable-disabled) for guidance on **disable**. Use **inactive** or **off** instead. ([Vale](../testing.md#vale) rule: [`InclusionAbleism.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/InclusionAbleism.yml)) +## dropdown list + +Use **dropdown list** to refer to the UI element. Do not use **dropdown** without **list** after it. +Do not use **drop-down** (hyphenated), **dropdown menu**, or other variants. + +For example: + +- From the **Visibility** dropdown list, select **Public**. + ## earlier -Use when talking about version numbers. +Use **earlier** when talking about version numbers. -- Avoid: In GitLab 14.1 and lower. -- Use instead: In GitLab 14.1 and earlier. +- Do: In GitLab 14.1 and earlier. +- Do not: In GitLab 14.1 and lower. ## easily -Do not use. If the user doesn't find the process to be easy, we lose their trust. +Do not use **easily**. If the user doesn't find the process to be easy, we lose their trust. ## e.g. @@ -119,60 +186,75 @@ Do not use Latin abbreviations. Use **for example**, **such as**, **for instance ## email -Do not use **e-mail** with a hyphen. When plural, use **emails** or **email messages**. +Do not use **e-mail** with a hyphen. When plural, use **emails** or **email messages**. ([Vale](../testing.md#vale) rule: [`SubstitutionSuggestions.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/SubstitutionSuggestions.yml)) ## enable -See [the Microsoft style guide](https://docs.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/e/enable-enables) for guidance. +See [the Microsoft style guide](https://docs.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/e/enable-enables) for guidance on **enable**. Use **active** or **on** instead. ([Vale](../testing.md#vale) rule: [`InclusionAbleism.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/InclusionAbleism.yml)) +## enter + +Use **enter** instead of **type** when talking about putting values into text boxes. + ## epic -Lowercase. +Use lowercase for **epic**. ## epic board -Lowercase. +Use lowercase for **epic board**. ## etc. -Try to avoid. Be as specific as you can. Do not use **and so on** as a replacement. +Try to avoid **etc.**. Be as specific as you can. Do not use **and so on** as a replacement. -- Avoid: You can update objects, like merge requests, issues, etc. -- Use instead: You can update objects, like merge requests and issues. +- Do: You can update objects, like merge requests and issues. +- Do not: You can update objects, like merge requests, issues, etc. + +## expand + +Use **expand** instead of **open** when you are talking about expanding or collapsing a section in the UI. + +## field + +Use **box** instead of **field** or **text box**. + +- Avoid: In the **Variable name** field, enter `my text`. +- Use instead: In the **Variable name** box, enter `my text`. ## foo -Do not use in product documentation. You can use it in our API and contributor documentation, but try to use a clearer and more meaningful example instead. +Do not use **foo** in product documentation. You can use it in our API and contributor documentation, but try to use a clearer and more meaningful example instead. ## future tense -When possible, use present tense instead. For example, use `after you execute this command, GitLab displays the result` instead of `after you execute this command, GitLab will display the result`. ([Vale](../testing.md#vale) rule: [`FutureTense.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/FutureTense.yml)) +When possible, use present tense instead of future tense. For example, use **after you execute this command, GitLab displays the result** instead of **after you execute this command, GitLab will display the result**. ([Vale](../testing.md#vale) rule: [`FutureTense.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/FutureTense.yml)) ## Geo -Title case. +Use title case for **Geo**. ## GitLab -Do not make possessive (GitLab's). This guidance follows [GitLab Trademark Guidelines](https://about.gitlab.com/handbook/marketing/corporate-marketing/brand-activation/trademark-guidelines/). +Do not make **GitLab** possessive (GitLab's). This guidance follows [GitLab Trademark Guidelines](https://about.gitlab.com/handbook/marketing/corporate-marketing/brand-activation/trademark-guidelines/). ## GitLab.com -Refers to the GitLab instance managed by GitLab itself. +**GitLab.com** refers to the GitLab instance managed by GitLab itself. ## GitLab SaaS -Refers to the product license that provides access to GitLab.com. Does not refer to the +**GitLab SaaS** refers to the product license that provides access to GitLab.com. It does not refer to the GitLab instance managed by GitLab itself. ## GitLab Runner -Title case. This is the product you install. See also [runners](#runner-runners) and [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/233529). +Use title case for **GitLab Runner**. This is the product you install. See also [runners](#runner-runners) and [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/233529). ## GitLab self-managed -Refers to the product license for GitLab instances managed by customers themselves. +Use **GitLab self-managed** to refer to the product license for GitLab instances managed by customers themselves. ## Guest @@ -183,25 +265,32 @@ When writing about the Guest role: - Do not use the phrase, **if you are a guest** to mean someone who is assigned the Guest role. Instead, write it out. For example, **if you are assigned the Guest role**. - To describe a situation where the Guest role is the minimum required: - - Avoid: **the Guest role or higher** - - Use instead: **at least the Guest role** + - Avoid: the Guest role or higher + - Use instead: at least the Guest role Do not use **Guest permissions**. A user who is assigned the Guest role has a set of associated permissions. ## handy -Do not use. If the user doesn't find the feature or process to be handy, we lose their trust. +Do not use **handy**. If the user doesn't find the feature or process to be handy, we lose their trust. ([Vale](../testing.md#vale) rule: [`Simplicity.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/Simplicity.yml)) ## high availability, HA -Do not use. Instead, direct readers to the GitLab [reference architectures](../../../administration/reference_architectures/index.md) for information about configuring GitLab for handling greater amounts of users. +Do not use **high availability** or **HA**. Instead, direct readers to the GitLab [reference architectures](../../../administration/reference_architectures/index.md) for information about configuring GitLab for handling greater amounts of users. ## higher -Do not use when talking about version numbers. +Do not use **higher** when talking about version numbers. -- Avoid: In GitLab 14.1 and higher. -- Use instead: In GitLab 14.1 and later. +- Do: In GitLab 14.1 and later. +- Do not: In GitLab 14.1 and higher. + +## hit + +Don't use **hit** to mean **press**. + +- Avoid: Hit the **ENTER** button. +- Use instead: Press **ENTER**. ## I @@ -213,19 +302,19 @@ Do not use Latin abbreviations. Use **that is** instead. ([Vale](../testing.md#v ## in order to -Do not use. Use **to** instead. ([Vale](../testing.md#vale) rule: [`Wordy.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/Wordy.yml)) +Do not use **in order to**. Use **to** instead. ([Vale](../testing.md#vale) rule: [`Wordy.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/Wordy.yml)) ## issue -Lowercase. +Use lowercase for **issue**. ## issue board -Lowercase. +Use lowercase for **issue board**. ## issue weights -Lowercase. +Use lowercase for **issue weights**. ## job @@ -235,21 +324,26 @@ If you want to use **CI** with the word **job**, use **CI/CD job** rather than * ## later -Use when talking about version numbers. +Use **later** when talking about version numbers. - Avoid: In GitLab 14.1 and higher. - Use instead: In GitLab 14.1 and later. +## list + +Do not use **list** when referring to a [**dropdown list**](#dropdown-list). +Use the full phrase **dropdown list** instead. + ## log in, log on -Do not use. Use [sign in](#sign-in) instead. If the user interface has **Log in**, you can use it. +Do not use **log in** or **log on**. Use [sign in](#sign-in) instead. If the user interface has **Log in**, you can use it. ## lower -Do not use when talking about version numbers. +Do not use **lower** when talking about version numbers. -- Avoid: In GitLab 14.1 and lower. -- Use instead: In GitLab 14.1 and earlier. +- Do: In GitLab 14.1 and earlier. +- Do not: In GitLab 14.1 and lower. ## Maintainer @@ -260,22 +354,22 @@ When writing about the Maintainer role: - Do not use the phrase, **if you are a maintainer** to mean someone who is assigned the Maintainer role. Instead, write it out. For example, **if you are assigned the Maintainer role**. - To describe a situation where the Maintainer role is the minimum required: - - Avoid: **the Maintainer role or higher** - - Use instead: **at least the Maintainer role** + - Avoid: the Maintainer role or higher + - Use instead: at least the Maintainer role Do not use **Maintainer permissions**. A user who is assigned the Maintainer role has a set of associated permissions. ## mankind -Do not use. Use **people** or **humanity** instead. ([Vale](../testing.md#vale) rule: [`InclusionGender.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/InclusionGender.yml)) +Do not use **mankind**. Use **people** or **humanity** instead. ([Vale](../testing.md#vale) rule: [`InclusionGender.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/InclusionGender.yml)) ## manpower -Do not use. Use words like **workforce** or **GitLab team members**. ([Vale](../testing.md#vale) rule: [`InclusionGender.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/InclusionGender.yml)) +Do not use **manpower**. Use words like **workforce** or **GitLab team members**. ([Vale](../testing.md#vale) rule: [`InclusionGender.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/InclusionGender.yml)) ## master -Do not use. Options are **primary** or **main**. ([Vale](../testing.md#vale) rule: [`InclusionCultural.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/InclusionCultural.yml)) +Do not use **master**. Options are **primary** or **main**. ([Vale](../testing.md#vale) rule: [`InclusionCultural.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/InclusionCultural.yml)) ## may, might @@ -287,18 +381,25 @@ Do not use first-person singular. Use **you**, **we**, or **us** instead. ([Vale ## merge requests -Lowercase. If you use **MR** as the acronym, spell it out on first use. +Use lowercase for **merge requests**. If you use **MR** as the acronym, spell it out on first use. ## milestones -Lowercase. +Use lowercase for **milestones**. + +## navigate + +Do not use **navigate**. Use **go** instead. For example: + +- Go to this webpage. +- Open a terminal and go to the `runner` directory. ## need to, should -Try to avoid. If something is required, use **must**. +Try to avoid **needs to**, because it's wordy. Avoid **should** when you can be more specific. If something is required, use **must**. -- Avoid: You need to set the variable. -- Use instead: You must set the variable. Or: Set the variable. +- Do: You must set the variable. Or: Set the variable. +- Do not: You need to set the variable. **Should** is acceptable for recommended actions or items, or in cases where an event may not happen. For example: @@ -310,10 +411,10 @@ happen. For example: ## note that -Do not use. +Do not use **note that** because it's wordy. -- Avoid: Note that you can change the settings. -- Use instead: You can change the settings. +- Do: You can change the settings. +- Do not: Note that you can change the settings. ## Owner @@ -328,15 +429,21 @@ Do not use **Owner permissions**. A user who is assigned the Owner role has a se ## permissions -Do not use roles and permissions interchangeably. Each user is assigned a role. Each role includes a set of permissions. +Do not use **roles** and **permissions** interchangeably. Each user is assigned a role. Each role includes a set of permissions. ## please -Do not use. For details, see the [Microsoft style guide](https://docs.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/p/please). +Do not use **please**. For details, see the [Microsoft style guide](https://docs.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/p/please). + +## press + +Use **press** when talking about keyboard keys. For example: + +- To stop the command, press <kbd>Control</kbd>+<kbd>C</kbd>. ## profanity -Do not use. Doing so may negatively affect other users and contributors, which is contrary to the GitLab value of [Diversity, Inclusion, and Belonging](https://about.gitlab.com/handbook/values/#diversity-inclusion). +Do not use profanity. Doing so may negatively affect other users and contributors, which is contrary to the GitLab value of [Diversity, Inclusion, and Belonging](https://about.gitlab.com/handbook/values/#diversity-inclusion). ## Reporter @@ -347,30 +454,48 @@ When writing about the Reporter role: - Do not use the phrase, **if you are a reporter** to mean someone who is assigned the Reporter role. Instead, write it out. For example, **if you are assigned the Reporter role**. - To describe a situation where the Reporter role is the minimum required: - - Avoid: **the Reporter role or higher** - - Use instead: **at least the Reporter role** + - Avoid: the Reporter role or higher + - Use instead: at least the Reporter role Do not use **Reporter permissions**. A user who is assigned the Reporter role has a set of associated permissions. ## Repository Mirroring -Title case. +Use title case for **Repository Mirroring**. ## roles -Do not use roles and permissions interchangeably. Each user is assigned a role. Each role includes a set of permissions. +Do not use **roles** and **permissions** interchangeably. Each user is assigned a role. Each role includes a set of permissions. ## runner, runners -Lowercase. These are the agents that run CI/CD jobs. See also [GitLab Runner](#gitlab-runner) and [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/233529). +Use lowercase for **runners**. These are the agents that run CI/CD jobs. See also [GitLab Runner](#gitlab-runner) and [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/233529). ## sanity check -Do not use. Use **check for completeness** instead. ([Vale](../testing.md#vale) rule: [`InclusionAbleism.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/InclusionAbleism.yml)) +Do not use **sanity check**. Use **check for completeness** instead. ([Vale](../testing.md#vale) rule: [`InclusionAbleism.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/InclusionAbleism.yml)) ## scalability -Do not use when talking about increasing GitLab performance for additional users. The words scale or scaling are sometimes acceptable, but references to increasing GitLab performance for additional users should direct readers to the GitLab [reference architectures](../../../administration/reference_architectures/index.md) page. +Do not use **scalability** when talking about increasing GitLab performance for additional users. The words scale or scaling +are sometimes acceptable, but references to increasing GitLab performance for additional users should direct readers +to the GitLab [reference architectures](../../../administration/reference_architectures/index.md) page. + +## section + +Use **section** to describe an area on a page. For example, if a page has lines that separate the UI +into separate areas, refer to these areas as sections. + +We often think of expandable/collapsible areas as **sections**. When you refer to expanding +or collapsing a section, don't include the word **section**. + +- Do: Expand **Auto DevOps**. +- Do not: Expand the **Auto DevOps** section. + +## select + +Use **select** with buttons, links, menu items, and lists. **Select** applies to more devices, +while **click** is more specific to a mouse. ## setup, set up @@ -381,13 +506,13 @@ Use **setup** as a noun, and **set up** as a verb. For example: ## sign in -Use instead of **sign on** or **log on** or **log in**. If the user interface has different words, use those. +Use **sign in** instead of **sign on** or **log on** or **log in**. If the user interface has different words, use those. You can use **single sign-on**. ## simply, simple -Do not use. If the user doesn't find the process to be simple, we lose their trust. +Do not use **simply** or **simple**. If the user doesn't find the process to be simple, we lose their trust. ([Vale](../testing.md#vale) rule: [`Simplicity.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/Simplicity.yml)) ## slashes @@ -395,34 +520,38 @@ Instead of **and/or**, use **or** or re-write the sentence. This rule also appli ## slave -Do not use. Another option is **secondary**. ([Vale](../testing.md#vale) rule: [`InclusionCultural.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/InclusionCultural.yml)) +Do not use **slave**. Another option is **secondary**. ([Vale](../testing.md#vale) rule: [`InclusionCultural.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/InclusionCultural.yml)) ## subgroup -Use instead of **sub-group**. +Use **subgroup** (no hyphen) instead of **sub-group**. ([Vale](../testing.md#vale) rule: [`SubstitutionSuggestions.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/SubstitutionSuggestions.yml)) ## that -Do not use when describing a noun. For example: +Do not use **that** when describing a noun. For example: -- Avoid: The file **that** you save... -- Use instead: The file you save... +- Do: The file you save... +- Do not: The file **that** you save... See also [this, these, that, those](#this-these-that-those). ## terminal -Lowercase. For example: +Use lowercase for **terminal**. For example: - Open a terminal. - From a terminal, run the `docker login` command. +## text box + +Use **text box** instead of **field** or **box** when referring to the UI element. + ## there is, there are -Try to avoid. These phrases hide the subject. +Try to avoid **there is** and **there are**. These phrases hide the subject. -- Avoid: There are holes in the bucket. -- Use instead: The bucket has holes. +- Do: The bucket has holes. +- Do not: There are holes in the bucket. ## they @@ -434,37 +563,48 @@ a gender-neutral pronoun. Always follow these words with a noun. For example: -- Avoid: **This** improves performance. -- Use instead: **This setting** improves performance. +- Do: **This setting** improves performance. +- Do not: **This** improves performance. -- Avoid: **These** are the best. -- Use instead: **These pants** are the best. +- Do: **These pants** are the best. +- Do not: **These** are the best. -- Avoid: **That** is the one you are looking for. -- Use instead: **That Jedi** is the one you are looking for. +- Do: **That droid** is the one you are looking for. +- Do not: **That** is the one you are looking for. -- Avoid: **Those** need to be configured. -- Use instead: **Those settings** need to be configured. (Or even better, **Configure those settings.**) +- Do: **Those settings** need to be configured. (Or even better, **Configure those settings.**) +- Do not: **Those** need to be configured. ## to-do item -Use lowercase. ([Vale](../testing.md#vale) rule: [`ToDo.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/ToDo.yml)) +Use lowercase and hyphenate **to-do** item. ([Vale](../testing.md#vale) rule: [`ToDo.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/ToDo.yml)) ## To-Do List -Use title case. ([Vale](../testing.md#vale) rule: [`ToDo.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/ToDo.yml)) +Use title case for **To-Do List**. ([Vale](../testing.md#vale) rule: [`ToDo.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/ToDo.yml)) + +## toggle + +You **turn on** or **turn off** a toggle. For example: + +- Turn on the **blah** toggle. + +## type + +Do not use **type** if you can avoid it. Use **enter** instead. ## useful -Do not use. If the user doesn't find the process to be useful, we lose their trust. +Do not use **useful**. If the user doesn't find the process to be useful, we lose their trust. ([Vale](../testing.md#vale) rule: [`Simplicity.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/Simplicity.yml)) ## utilize -Do not use. Use **use** instead. It's more succinct and easier for non-native English speakers to understand. +Do not use **utilize**. Use **use** instead. It's more succinct and easier for non-native English speakers to understand. +([Vale](../testing.md#vale) rule: [`SubstitutionSuggestions.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/SubstitutionSuggestions.yml)) ## Value Stream Analytics -Title case. +Use title case for **Value Stream Analytics**. ## via @@ -474,14 +614,14 @@ Do not use Latin abbreviations. Use **with**, **through**, or **by using** inste Try to avoid **we** and focus instead on how the user can accomplish something in GitLab. -- Avoid: We created a feature for you to add widgets. -- Instead, use: Use widgets when you have work you want to organize. +- Do: Use widgets when you have work you want to organize. +- Do not: We created a feature for you to add widgets. -One exception: You can use **we recommend** instead of **it is recommended** or **GitLab recommends**. +One exception: You can use **we recommend** instead of **it is recommended** or **GitLab recommends**. ([Vale](../testing.md#vale) rule: [`SubstitutionSuggestions.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/SubstitutionSuggestions.yml)) ## whitelist -Do not use. Another option is **allowlist**. ([Vale](../testing.md#vale) rule: [`InclusionCultural.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/InclusionCultural.yml)) +Do not use **whitelist**. Another option is **allowlist**. ([Vale](../testing.md#vale) rule: [`InclusionCultural.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/InclusionCultural.yml)) <!-- vale on --> <!-- markdownlint-enable --> diff --git a/doc/development/documentation/testing.md b/doc/development/documentation/testing.md index 2ade6c1e71d..dfa2f3ed55a 100644 --- a/doc/development/documentation/testing.md +++ b/doc/development/documentation/testing.md @@ -228,7 +228,7 @@ guidelines: In [`ReadingLevel.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/ReadingLevel.yml), we have implemented -[the Flesch-Kincaid grade level test](https://readable.com/blog/the-flesch-reading-ease-and-flesch-kincaid-grade-level/) +[the Flesch-Kincaid grade level test](https://readable.com/readability/flesch-reading-ease-flesch-kincaid-grade-level/) to determine the readability of our documentation. As a general guideline, the lower the score, the more readable the documentation. @@ -319,6 +319,38 @@ To configure Vale in your editor, install one of the following as appropriate: cases, `vale` should work. To find the location, run `which vale` in a terminal. - Vim [ALE plugin](https://github.com/dense-analysis/ale). +- Emacs [Flycheck extension](https://github.com/flycheck/flycheck). + This requires some configuration: + + - `Flycheck` supports `markdownlint-cli` out of the box, but you must point it + to the `.markdownlint.yml` at the base of the project directory. A `.dir-locals.el` + file can accomplish this: + + ```lisp + ;; Place this code in a file called `.dir-locals.el` at the root of the gitlab project. + ((markdown-mode . ((flycheck-markdown-markdownlint-cli-config . ".markdownlint.yml")))) + + ``` + + - A minimal configuration for Flycheck to work with Vale could look like this: + + ```lisp + (flycheck-define-checker vale + "A checker for prose" + :command ("vale" "--output" "line" "--no-wrap" + source) + :standard-input nil + :error-patterns + ((error line-start (file-name) ":" line ":" column ":" (id (one-or-more (not (any ":")))) ":" (message) line-end)) + :modes (markdown-mode org-mode text-mode) + :next-checkers ((t . markdown-markdownlint-cli)) + ) + + (add-to-list 'flycheck-checkers 'vale) + ``` + + In this setup the `markdownlint` checker is set as a "next" checker from the defined `vale` checker. + Enabling this custom Vale checker provides error linting from both Vale and markdownlint. We don't use [Vale Server](https://errata-ai.github.io/vale/#using-vale-with-a-text-editor-or-another-third-party-application). diff --git a/doc/development/elasticsearch.md b/doc/development/elasticsearch.md index 4b87f1c28f1..bba4e1cda23 100644 --- a/doc/development/elasticsearch.md +++ b/doc/development/elasticsearch.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w This area is to maintain a compendium of useful information when working with Elasticsearch. Information on how to enable Elasticsearch and perform the initial indexing is in -the [Elasticsearch integration documentation](../integration/elasticsearch.md#enabling-advanced-search). +the [Elasticsearch integration documentation](../integration/elasticsearch.md#enable-advanced-search). ## Deep Dive @@ -233,6 +233,11 @@ Any data or index cleanup needed to support migration retries should be handled will re-enqueue itself with a delay which is set using the `throttle_delay` option described below. The batching must be handled within the `migrate` method, this setting controls the re-enqueuing only. +- `batch_size` - Sets the number of documents modified during a `batched!` migration run. This size should be set to a value which allows the updates + enough time to finish. This can be tuned in combination with the `throttle_delay` option described below. The batching + must be handled within a custom `migrate` method or by using the [`Elastic::MigrationBackfillHelper`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/concerns/elastic/migration_backfill_helper.rb) + `migrate` method which uses this setting. Default value is 1000 documents. + - `throttle_delay` - Sets the wait time in between batch runs. This time should be set high enough to allow each migration batch enough time to finish. Additionally, the time should be less than 30 minutes since that is how often the [`Elastic::MigrationWorker`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/elastic/migration_worker.rb) diff --git a/doc/development/experiment_guide/experimentation.md b/doc/development/experiment_guide/experimentation.md index ee0f63342f1..b242646c549 100644 --- a/doc/development/experiment_guide/experimentation.md +++ b/doc/development/experiment_guide/experimentation.md @@ -106,7 +106,7 @@ class SomeWorker # Since we cannot access cookies in a worker, we need to bucket models # based on a unique, unchanging attribute instead. - # It is therefore necessery to always provide the same subject. + # It is therefore necessary to always provide the same subject. if Gitlab::Experimentation.in_experiment_group?(:experiment_key, subject: user) # execute experimental code else diff --git a/doc/development/experiment_guide/index.md b/doc/development/experiment_guide/index.md index e4a97091a81..4de272fec20 100644 --- a/doc/development/experiment_guide/index.md +++ b/doc/development/experiment_guide/index.md @@ -8,11 +8,13 @@ info: To determine the technical writer assigned to the Stage/Group associated w Experiments can be conducted by any GitLab team, most often the teams from the [Growth Sub-department](https://about.gitlab.com/handbook/engineering/development/growth/). Experiments are not tied to releases because they primarily target GitLab.com. -Experiments are run as an A/B/n test, and are behind a feature flag to turn the test on or off. Based on the data the experiment generates, the team decides if the experiment had a positive impact and should be made the new default, or rolled back. +Experiments are run as an A/B/n test, and are behind an [experiment feature flag](../feature_flags/#experiment-type) to turn the test on or off. Based on the data the experiment generates, the team decides if the experiment had a positive impact and should be made the new default, or rolled back. -## Experiment tracking issue +## Experiment rollout issue -Each experiment should have an [Experiment tracking](https://gitlab.com/groups/gitlab-org/-/issues?scope=all&state=opened&label_name[]=growth%20experiment&search=%22Experiment+tracking%22) issue to track the experiment from roll-out through to cleanup/removal. The tracking issue is similar to a feature flag rollout issue, and is also used to track the status of an experiment. Immediately after an experiment is deployed, the due date of the issue should be set (this depends on the experiment but can be up to a few weeks in the future). +Each experiment should have an [experiment rollout](https://gitlab.com/groups/gitlab-org/-/boards/1352542) issue to track the experiment from rollout through to cleanup and removal. +The rollout issue is similar to a feature flag rollout issue, and is also used to track the status of an experiment. +When an experiment is deployed, the due date of the issue should be set (this depends on the experiment but can be up to a few weeks in the future). After the deadline, the issue needs to be resolved and either: - It was successful and the experiment becomes the new default. @@ -29,6 +31,10 @@ run) shouldn't impact GitLab availability. To avoid or identify issues, experiments are initially deployed to a small number of users. Regardless, experiments still need tests. +Experiments must have corresponding [frontend or feature tests](../testing_guide/index.md) to ensure they +exist in the application. These tests should help prevent the experiment code from +being removed before the [experiment cleanup process](https://about.gitlab.com/handbook/engineering/development/growth/experimentation/#experiment-cleanup-issue) starts. + If, as a reviewer or maintainer, you find code that would usually fail review but is acceptable for now, mention your concerns with a note that there's no need to change the code. The author can then add a comment to this piece of code @@ -38,22 +44,14 @@ addressed. ## Implementing an experiment -There are currently two options when implementing an experiment. - -One is built into GitLab directly and has been around for a while (this is called -`Exerimentation Module`), and the other is provided by -[`gitlab-experiment`](https://gitlab.com/gitlab-org/gitlab-experiment) and is referred -to as `Gitlab::Experiment` -- GLEX for short. +[`GLEX`](https://gitlab.com/gitlab-org/gitlab-experiment) - or `Gitlab::Experiment`, the `gitlab-experiment` gem - is the preferred option for implementing an experiment in GitLab. -Both approaches use [experiment](../feature_flags/index.md#experiment-type) -feature flags. We recommend using GLEX rather than `Experimentation Module` for new experiments. +For more information, see [Implementing an A/B/n experiment using GLEX](gitlab_experiment.md). -- [Implementing an A/B/n experiment using GLEX](gitlab_experiment.md) -- [Implementing an A/B experiment using `Experimentation Module`](experimentation.md) +There are still some longer running experiments using the [`Exerimentation Module`](experimentation.md). -Historical Context: `Experimentation Module` was built iteratively with the needs that -appeared while implementing Growth sub-department experiments, while GLEX was built -with the findings of the team and an easier to use API. +Both approaches use [experiment](../feature_flags/index.md#experiment-type) feature flags. +`GLEX` is the preferred option for new experiments. ### Add new icons and illustrations for experiments diff --git a/doc/development/fe_guide/accessibility.md b/doc/development/fe_guide/accessibility.md index 0cd7cf58b58..7c870de9a6c 100644 --- a/doc/development/fe_guide/accessibility.md +++ b/doc/development/fe_guide/accessibility.md @@ -334,7 +334,7 @@ Keep in mind that: - When you add `:hover` styles, in most cases you should add `:focus` styles too so that the styling is applied for both mouse **and** keyboard users. - If you remove an interactive element's `outline`, make sure you maintain visual focus state in another way such as with `box-shadow`. -See the [Pajamas Keyboard-only page](https://design.gitlab.com/accessibility-audits/2-keyboard-only/) for more detail. +See the [Pajamas Keyboard-only page](https://design.gitlab.com/accessibility-audits/keyboard-only) for more detail. ## Tabindex @@ -510,7 +510,7 @@ Proper research and testing should be done to ensure compliance with [WCAG](http ### Viewing the browser accessibility tree - [Firefox DevTools guide](https://developer.mozilla.org/en-US/docs/Tools/Accessibility_inspector#accessing_the_accessibility_inspector) -- [Chrome DevTools guide](https://developer.chrome.com/docs/devtools/accessibility/reference#pane) +- [Chrome DevTools guide](https://developer.chrome.com/docs/devtools/accessibility/reference/#pane) ### Browser extensions diff --git a/doc/development/fe_guide/axios.md b/doc/development/fe_guide/axios.md index 2d699b305ce..b42a17d7870 100644 --- a/doc/development/fe_guide/axios.md +++ b/doc/development/fe_guide/axios.md @@ -75,7 +75,7 @@ We have also decided against using [Axios interceptors](https://github.com/axios ### Mock poll requests in tests with Axios -Because polling function requires a header object, we need to always include an object as the third argument: +Because a polling function requires a header object, we need to always include an object as the third argument: ```javascript mock.onGet('/users').reply(200, { foo: 'bar' }, {}); diff --git a/doc/development/fe_guide/content_editor.md b/doc/development/fe_guide/content_editor.md index 6cf4076bf83..956e7d0d56e 100644 --- a/doc/development/fe_guide/content_editor.md +++ b/doc/development/fe_guide/content_editor.md @@ -4,10 +4,10 @@ group: Editor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Content Editor **(FREE)** +# Content Editor development guidelines **(FREE)** The Content Editor is a UI component that provides a WYSIWYG editing -experience for [GitLab Flavored Markdown](../../user/markdown.md) (GFM) in the GitLab application. +experience for [GitLab Flavored Markdown](../../user/markdown.md) in the GitLab application. It also serves as the foundation for implementing Markdown-focused editors that target other engines, like static site generators. @@ -16,103 +16,339 @@ to build the Content Editor. These frameworks provide a level of abstraction on the native [`contenteditable`](https://developer.mozilla.org/en-US/docs/Web/Guide/HTML/Editable_content) web technology. -## Architecture remarks +## Usage guide -At a high level, the Content Editor: +Follow these instructions to include the Content Editor in a feature. -- Imports arbitrary Markdown. -- Renders it in a HTML editing area. -- Exports it back to Markdown with changes introduced by the user. +1. [Include the Content Editor component](#include-the-content-editor-component). +1. [Set and get Markdown](#set-and-get-markdown). +1. [Listen for changes](#listen-for-changes). -The Content Editor relies on the -[Markdown API endpoint](../../api/markdown.md) to transform Markdown -into HTML. It sends the Markdown input to the REST API and displays the API's -HTML output in the editing area. The editor exports the content back to Markdown -using a client-side library that serializes editable documents into Markdown. +### Include the Content Editor component - +Import the `ContentEditor` Vue component. We recommend using asynchronous named imports to +take advantage of caching, as the ContentEditor is a big dependency. -Check the [Content Editor technical design document](https://docs.google.com/document/d/1fKOiWpdHned4KOLVOOFYVvX1euEjMP5rTntUhpapdBg) -for more information about the design decisions that drive the development of the editor. - -**NOTE**: We also designed the Content Editor to be extensible. We intend to provide -more information about extension development for supporting new types of content in upcoming -milestones. - -## GitLab Flavored Markdown support +```html +<script> +export default { + components: { + ContentEditor: () => + import( + /* webpackChunkName: 'content_editor' */ '~/content_editor/components/content_editor.vue' + ), + }, + // rest of the component definition +} +</script> +``` -The [GitLab Flavored Markdown](../../user/markdown.md) extends -the [CommonMark specification](https://spec.commonmark.org/0.29/) with support for a -variety of content types like diagrams, math expressions, and tables. Supporting -all GitLab Flavored Markdown content types in the Content Editor is a work in progress. For -the status of the ongoing development for CommonMark and GitLab Flavored Markdown support, read: +The Content Editor requires two properties: -- [Basic Markdown formatting extensions](https://gitlab.com/groups/gitlab-org/-/epics/5404) epic. -- [GitLab Flavored Markdown extensions](https://gitlab.com/groups/gitlab-org/-/epics/5438) epic. +- `renderMarkdown` is an asynchronous function that returns the response (String) of invoking the +[Markdown API](../../api/markdown.md). +- `uploadsPath` is a URL that points to a [GitLab upload service](../uploads.md#upload-encodings) + with `multipart/form-data` support. -## Usage +See the [`WikiForm.vue`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/javascripts/pages/shared/wikis/components/wiki_form.vue#L207) +component for a production example of these two properties. -To include the Content Editor in your feature, import the `createContentEditor` factory -function and the `ContentEditor` Vue component. `createContentEditor` sets up an instance -of [tiptap's Editor class](https://www.tiptap.dev/api/editor/) with all the necessary -extensions to support editing GitLab Flavored Markdown content. It also creates -a Markdown serializer that allows exporting tiptap's document format to Markdown. +### Set and get Markdown -`createContentEditor` requires a `renderMarkdown` parameter invoked -by the editor every time it needs to convert Markdown to HTML. The Content Editor -does not provide a default value for this function yet. +The `ContentEditor` Vue component doesn't implement Vue data binding flow (`v-model`) +because setting and getting Markdown are expensive operations. Data binding would +trigger these operations every time the user interacts with the component. -**NOTE**: The Content Editor is in an early development stage. Usage and development -guidelines are subject to breaking changes in the upcoming months. +Instead, you should obtain an instance of the `ContentEditor` class by listening to the +`initialized` event: ```html <script> -import { GlButton } from '@gitlab/ui'; -import { createContentEditor, ContentEditor } from '~/content_editor'; -import { __ } from '~/locale'; import createFlash from '~/flash'; +import { __ } from '~/locale'; export default { - components: { - ContentEditor, - GlButton, + methods: { + async loadInitialContent(contentEditor) { + this.contentEditor = contentEditor; + + try { + await this.contentEditor.setSerializedContent(this.content); + } catch (e) { + createFlash(__('Could not load initial document')); + } + }, + submitChanges() { + const markdown = this.contentEditor.getSerializedContent(); + }, }, +}; +</script> +<template> + <content-editor + :render-markdown="renderMarkdown" + :uploads-path="pageInfo.uploadsPath" + @initialized="loadInitialContent" + /> +</template> +``` + +### Listen for changes + +You can still react to changes in the Content Editor. Reacting to changes helps +you know if the document is empty or dirty. Use the `@change` event handler for +this purpose. + +```html +<script> +export default { data() { return { - contentEditor: null, - } + empty: false, + }; }, - created() { - this.contentEditor = createContentEditor({ - renderMarkdown: (markdown) => Api.markdown({ text: markdown }), - }); - - try { - await this.contentEditor.setSerializedContent(this.content); - } catch (e) { - createFlash({ - message: __('There was an error loading content in the editor'), error: e - }); + methods: { + handleContentEditorChange({ empty }) { + this.empty = empty; } }, +}; +</script> +<template> + <div> + <content-editor + :render-markdown="renderMarkdown" + :uploads-path="pageInfo.uploadsPath" + @initialized="loadInitialContent" + @change="handleContentEditorChange" + /> + <gl-button :disabled="empty" @click="submitChanges"> + {{ __('Submit changes') }} + </gl-button> + </div> +</template> +``` + +## Implementation guide + +The Content Editor is composed of three main layers: + +- **The editing tools UI**, like the toolbar and the table structure editor. They + display the editor's state and mutate it by dispatching commands. +- **The Tiptap Editor object** manages the editor's state, + and exposes business logic as commands executed by the editing tools UI. +- **The Markdown serializer** transforms a Markdown source string into a ProseMirror + document and vice versa. + +### Editing tools UI + +The editing tools UI are Vue components that display the editor's state and +dispatch [commands](https://www.tiptap.dev/api/commands/#commands) to mutate it. +They are located in the `~/content_editor/components` directory. For example, +the **Bold** toolbar button displays the editor's state by becoming active when +the user selects bold text. This button also dispatches the `toggleBold` command +to format text as bold: + +```mermaid +sequenceDiagram + participant A as Editing tools UI + participant B as Tiptap object + A->>B: queries state/dispatches commands + B--)A: notifies state changes +``` + +#### Node views + +We implement [node views](https://www.tiptap.dev/guide/node-views/vue/#node-views-with-vue) +to provide inline editing tools for some content types, like tables and images. Node views +allow separating the presentation of a content type from its +[model](https://prosemirror.net/docs/guide/#doc.data_structures). Using a Vue component in +the presentation layer enables sophisticated editing experiences in the Content Editor. +Node views are located in `~/content_editor/components/wrappers`. + +#### Dispatch commands + +You can inject the Tiptap Editor object to Vue components to dispatch +commands. + +NOTE: +Do not implement logic that changes the editor's +state in Vue components. Encapsulate this logic in commands, and dispatch +the command from the component's methods. + +```html +<script> +export default { + inject: ['tiptapEditor'], methods: { - async save() { - await Api.updateContent({ - content: this.contentEditor.getSerializedContent(), - }); + execute() { + //Incorrect + const { state, view } = this.tiptapEditor.state; + const { tr, schema } = state; + tr.addMark(state.selection.from, state.selection.to, null, null, schema.mark('bold')); + + // Correct + this.tiptapEditor.chain().toggleBold().focus().run(); + }, + } +}; +</script> +<template> +``` + +#### Query editor's state + +Use the `EditorStateObserver` renderless component to react to changes in the +editor's state, such as when the document or the selection changes. You can listen to +the following events: + +- `docUpdate` +- `selectionUpdate` +- `transaction` +- `focus` +- `blur` +- `error`. + +Learn more about these events in [Tiptap's event guide](https://www.tiptap.dev/api/events/). + +```html +<script> +// Parts of the code has been hidden for efficiency +import EditorStateObserver from './editor_state_observer.vue'; + +export default { + components: { + EditorStateObserver, + }, + data() { + return { + error: null, + }; + }, + methods: { + displayError({ message }) { + this.error = message; + }, + dismissError() { + this.error = null; }, }, }; </script> <template> - <div> - <content-editor :content-editor="contentEditor" /> - <gl-button @click="save()">Save</gl-button> - </div> + <editor-state-observer @error="displayError"> + <gl-alert v-if="error" class="gl-mb-6" variant="danger" @dismiss="dismissError"> + {{ error }} + </gl-alert> + </editor-state-observer> </template> ``` -Call `setSerializedContent` to set initial Markdown in the Editor. This method is -asynchronous because it makes an API request to render the Markdown input. -`getSerializedContent` returns a Markdown string that represents the serialized -version of the editable document. +### The Tiptap editor object + +The Tiptap [Editor](https://www.tiptap.dev/api/editor) class manages +the editor's state and encapsulates all the business logic that powers +the Content Editor. The Content Editor constructs a new instance of this class and +provides all the necessary extensions to support +[GitLab Flavored Markdown](../../user/markdown.md). + +#### Implement new extensions + +Extensions are the building blocks of the Content Editor. You can learn how to implement +new ones by reading [Tiptap's guide](https://www.tiptap.dev/guide/custom-extensions). +We recommend checking the list of built-in [nodes](https://www.tiptap.dev/api/nodes) and +[marks](https://www.tiptap.dev/api/marks) before implementing a new extension +from scratch. + +Store the Content Editor extensions in the `~/content_editor/extensions` directory. +When using a Tiptap's built-in extension, wrap it in a ES6 module inside this directory: + +```javascript +export { Bold as default } from '@tiptap/extension-bold'; +``` + +Use the `extend` method to customize the Extension's behavior: + +```javascript +import { HardBreak } from '@tiptap/extension-hard-break'; + +export default HardBreak.extend({ + addKeyboardShortcuts() { + return { + 'Shift-Enter': () => this.editor.commands.setHardBreak(), + }; + }, +}); +``` + +#### Register extensions + +Register the new extension in `~/content_editor/services/create_content_editor.js`. Import +the extension module and add it to the `builtInContentEditorExtensions` array: + +```javascript +import Emoji from '../extensions/emoji'; + +const builtInContentEditorExtensions = [ + Code, + CodeBlockHighlight, + Document, + Dropcursor, + Emoji, + // Other extensions +``` + +### The Markdown serializer + +The Markdown Serializer transforms a Markdown String to a +[ProseMirror document](https://prosemirror.net/docs/guide/#doc) and vice versa. + +#### Deserialization + +Deserialization is the process of converting Markdown to a ProseMirror document. +We take advantage of ProseMirror's +[HTML parsing and serialization capabilities](https://prosemirror.net/docs/guide/#schema.serialization_and_parsing) +by first rendering the Markdown as HTML using the [Markdown API endpoint](../../api/markdown.md): + +```mermaid +sequenceDiagram + participant A as Content Editor + participant E as Tiptap Object + participant B as Markdown Serializer + participant C as Markdown API + participant D as ProseMirror Parser + A->>B: deserialize(markdown) + B->>C: render(markdown) + C-->>B: html + B->>D: to document(html) + D-->>A: document + A->>E: setContent(document) +``` + +Deserializers live in the extension modules. Read Tiptap's +[parseHTML](https://www.tiptap.dev/guide/custom-extensions#parse-html) and +[addAttributes](https://www.tiptap.dev/guide/custom-extensions#attributes) documentation to +learn how to implement them. Titap's API is a wrapper around ProseMirror's +[schema spec API](https://prosemirror.net/docs/ref/#model.SchemaSpec). + +#### Serialization + +Serialization is the process of converting a ProseMirror document to Markdown. The Content +Editor uses [`prosemirror-markdown`](https://github.com/ProseMirror/prosemirror-markdown) +to serialize documents. We recommend reading the +[MarkdownSerializer](https://github.com/ProseMirror/prosemirror-markdown#class-markdownserializer) +and [MarkdownSerializerState](https://github.com/ProseMirror/prosemirror-markdown#class-markdownserializerstate) +classes documentation before implementing a serializer: + +```mermaid +sequenceDiagram + participant A as Content Editor + participant B as Markdown Serializer + participant C as ProseMirror Markdown + A->>B: serialize(document) + B->>C: serialize(document, serializers) + C-->>A: markdown string +``` + +`prosemirror-markdown` requires implementing a serializer function for each content type supported +by the Content Editor. We implement serializers in `~/content_editor/services/markdown_serializer.js`. diff --git a/doc/development/fe_guide/development_process.md b/doc/development/fe_guide/development_process.md index b85ed4da442..4e50621add4 100644 --- a/doc/development/fe_guide/development_process.md +++ b/doc/development/fe_guide/development_process.md @@ -88,7 +88,7 @@ With the purpose of being [respectful of others' time](https://about.gitlab.com/ GitLab architecture. 1. Add a diagram to the issue and ask a frontend maintainer in the Slack channel `#frontend_maintainers` about it. -  +  1. Don't take more than one week between starting work on a feature and sharing a Merge Request with a reviewer or a maintainer. diff --git a/doc/development/fe_guide/graphql.md b/doc/development/fe_guide/graphql.md index 3b49601f027..0229aa0123a 100644 --- a/doc/development/fe_guide/graphql.md +++ b/doc/development/fe_guide/graphql.md @@ -421,14 +421,16 @@ is still validated. Again, make sure that those overrides are as short-lived as possible by tracking their removal in the appropriate issue. -#### Feature flags in queries +#### Feature-flagged queries -Sometimes it may be helpful to have an entity in the GraphQL query behind a feature flag. -One example is working on a feature where the backend has already been merged but the frontend -has not. In this case, you may consider putting the GraphQL entity behind a feature flag to allow smaller -merge requests to be created and merged. +In cases where the backend is complete and the frontend is being implemented behind a feature flag, +a couple options are available to leverage the feature flag in the GraphQL queries. -To do this we can use the `@include` directive to exclude an entity if the `if` statement passes. +##### The `@include` directive + +The `@include` (or its opposite, `@skip`) can be used to control whether an entity should be +included in the query. If the `@include` directive evaluates to `false`, the entity's resolver is +not hit and the entity is excluded from the response. For example: ```graphql query getAuthorData($authorNameEnabled: Boolean = false) { @@ -456,6 +458,34 @@ export default { }; ``` +Note that, even if the directive evaluates to `false`, the guarded entity is sent to the backend and +matched against the GraphQL schema. So this approach requires that the feature-flagged entity +exists in the schema, even if the feature flag is disabled. When the feature flag is turned off, it +is recommended that the resolver returns `null` at the very least. + +##### Different versions of a query + +There's another approach that involves duplicating the standard query, and it should be avoided. The copy includes the new entities +while the original remains unchanged. It is up to the production code to trigger the right query +based on the feature flag's status. For example: + +```javascript +export default { + apollo: { + user: { + query() { + return this.glFeatures.authorNameEnabled ? NEW_QUERY : ORIGINAL_QUERY, + } + } + }, +}; +``` + +This approach is not recommended as it results in bigger merge requests and requires maintaining +two similar queries for as long as the feature flag exists. This can be used in cases where the new +GraphQL entities are not yet part of the schema, or if they are feature-flagged at the schema level +(`new_entity: :feature_flag`). + ### Manually triggering queries Queries on a component's `apollo` property are made automatically when the component is created. @@ -1310,7 +1340,7 @@ describe('when query times out', () => { expect(getAlert().exists()).toBe(false); expect(getGraph().exists()).toBe(true); - /* fails again, alert retuns but data persists */ + /* fails again, alert returns but data persists */ await advanceApolloTimers(); expect(getAlert().exists()).toBe(true); expect(getGraph().exists()).toBe(true); diff --git a/doc/development/fe_guide/img/content_editor_highlevel_diagram.png b/doc/development/fe_guide/img/content_editor_highlevel_diagram.png Binary files differdeleted file mode 100644 index 73a71cf5843..00000000000 --- a/doc/development/fe_guide/img/content_editor_highlevel_diagram.png +++ /dev/null diff --git a/doc/development/fe_guide/index.md b/doc/development/fe_guide/index.md index 549fa3261b1..a6b49394733 100644 --- a/doc/development/fe_guide/index.md +++ b/doc/development/fe_guide/index.md @@ -22,7 +22,7 @@ Be wary of [the limitations that come with using Hamlit](https://github.com/k0ku We also use [SCSS](https://sass-lang.com) and plain JavaScript with modern ECMAScript standards supported through [Babel](https://babeljs.io/) and ES module support through [webpack](https://webpack.js.org/). -Working with our frontend assets requires Node (v10.13.0 or greater) and Yarn +Working with our frontend assets requires Node (v12.22.1 or greater) and Yarn (v1.10.0 or greater). You can find information on how to install these on our [installation guide](../../install/installation.md#4-node). diff --git a/doc/development/fe_guide/source_editor.md b/doc/development/fe_guide/source_editor.md index fc128c0ecb1..2ff0bacfc3a 100644 --- a/doc/development/fe_guide/source_editor.md +++ b/doc/development/fe_guide/source_editor.md @@ -15,7 +15,7 @@ GitLab features use it, including: - [CI Linter](../../ci/lint.md) - [Snippets](../../user/snippets.md) - [Web Editor](../../user/project/repository/web_editor.md) -- [Security Policies](../../user/application_security/threat_monitoring/index.md) +- [Security Policies](../../user/application_security/policies/index.md) ## How to use Source Editor diff --git a/doc/development/github_importer.md b/doc/development/github_importer.md index 84c10e0c005..57cb74a6159 100644 --- a/doc/development/github_importer.md +++ b/doc/development/github_importer.md @@ -272,3 +272,16 @@ The last log entry reports the number of objects fetched and imported: "import_stage": "Gitlab::GithubImport::Stage::FinishImportWorker" } ``` + +## Errors when importing large projects + +The GitHub importer may encounter errors when importing large projects. For help with this, see the +documentation for the following use cases: + +- [Alternative way to import notes and diff notes](../user/project/import/github.md#alternative-way-to-import-notes-and-diff-notes) +- [Reduce GitHub API request objects per page](../user/project/import/github.md#reduce-github-api-request-objects-per-page) + +## Metrics dashboards + +To assess the GitHub importer health, the [GitHub importer dashboard](https://dashboards.gitlab.net/d/importers-github-importer/importers-github-importer) +provides information about the total number of objects fetched vs. imported over time. diff --git a/doc/development/go_guide/dependencies.md b/doc/development/go_guide/dependencies.md index c5af21d0772..8aa8f286edc 100644 --- a/doc/development/go_guide/dependencies.md +++ b/doc/development/go_guide/dependencies.md @@ -45,7 +45,7 @@ end with a timestamp and the first 12 characters of the commit identifier: If a VCS tag matches one of these patterns, it is ignored. For a complete understanding of Go modules and versioning, see [this series of -blog posts](https://blog.golang.org/using-go-modules) on the official Go +blog posts](https://go.dev/blog/using-go-modules) on the official Go website. ## 'Module' vs 'Package' diff --git a/doc/development/go_guide/index.md b/doc/development/go_guide/index.md index 224d8a0a0f5..0ee73da48db 100644 --- a/doc/development/go_guide/index.md +++ b/doc/development/go_guide/index.md @@ -65,7 +65,7 @@ Remember to run [SAST](../../user/application_security/sast/index.md) and [Dependency Scanning](../../user/application_security/dependency_scanning/index.md) **(ULTIMATE)** on your project (or at least the [`gosec` analyzer](https://gitlab.com/gitlab-org/security-products/analyzers/gosec)), -and to follow our [Security requirements](../code_review.md#security-requirements). +and to follow our [Security requirements](../code_review.md#security). Web servers can take advantages of middlewares like [Secure](https://github.com/unrolled/secure). @@ -196,7 +196,7 @@ library or framework: ### Subtests -Use [subtests](https://blog.golang.org/subtests) whenever possible to improve +Use [subtests](https://go.dev/blog/subtests) whenever possible to improve code readability and test output. ### Better output in tests @@ -319,7 +319,7 @@ A few things to keep in mind when adding context: ### References for working with errors -- [Go 1.13 errors](https://blog.golang.org/go1.13-errors). +- [Go 1.13 errors](https://go.dev/blog/go1.13-errors). - [Programing with errors](https://peter.bourgon.org/blog/2019/09/11/programming-with-errors.html). - [Don't just check errors, handle them diff --git a/doc/development/gotchas.md b/doc/development/gotchas.md index 40598eaff95..fbb6b0219aa 100644 --- a/doc/development/gotchas.md +++ b/doc/development/gotchas.md @@ -203,76 +203,6 @@ in an initializer. - Stack Overflow: [Why you should not write inline JavaScript](https://softwareengineering.stackexchange.com/questions/86589/why-should-i-avoid-inline-scripting) -## Auto loading - -Rails auto-loading on `development` differs from the load policy in the `production` environment. -In development mode, `config.eager_load` is set to `false`, which means classes -are loaded as needed. With the classic Rails autoloader, it is known that this can lead to -[Rails resolving the wrong class](https://guides.rubyonrails.org/v5.2/autoloading_and_reloading_constants.html#when-constants-aren-t-missed-relative-references) -if the class name is ambiguous. This can be fixed by specifying the complete namespace to the class. - -### Error prone example - -```ruby -# app/controllers/application_controller.rb -class ApplicationController < ActionController::Base - ... -end - -# app/controllers/projects/application_controller.rb -class Projects::ApplicationController < ApplicationController - ... - private - - def project - ... - end -end - -# app/controllers/projects/submodule/some_controller.rb -module Projects - module Submodule - class SomeController < ApplicationController - def index - @some_id = project.id - end - end - end -end -``` - -In this case, if for any reason the top level `ApplicationController` -is loaded but `Projects::ApplicationController` is not, `ApplicationController` -would be resolved to `::ApplicationController` and then the `project` method is -undefined, causing an error. - -#### Solution - -```ruby -# app/controllers/projects/submodule/some_controller.rb -module Projects - module Submodule - class SomeController < Projects::ApplicationController - def index - @some_id = project.id - end - end - end -end -``` - -By specifying `Projects::`, we tell Rails exactly what class we are referring -to and we would avoid the issue. - -NOTE: -This problem disappears as soon as we upgrade to Rails 6 and use the Zeitwerk autoloader. - -### Further reading - -- Rails Guides: [Autoloading and Reloading Constants (Classic Mode)](https://guides.rubyonrails.org/autoloading_and_reloading_constants_classic_mode.html) -- Ruby Constant lookup: [Everything you ever wanted to know about constant lookup in Ruby](https://cirw.in/blog/constant-lookup) -- Rails 6 and Zeitwerk autoloader: [Understanding Zeitwerk in Rails 6](https://medium.com/cedarcode/understanding-zeitwerk-in-rails-6-f168a9f09a1f) - ## Storing assets that do not require pre-compiling Assets that need to be served to the user are stored under the `app/assets` directory, which is later pre-compiled and placed in the `public/` directory. diff --git a/doc/development/graphql_guide/graphql_pro.md b/doc/development/graphql_guide/graphql_pro.md index ca20d66dd87..3170f0cfdc2 100644 --- a/doc/development/graphql_guide/graphql_pro.md +++ b/doc/development/graphql_guide/graphql_pro.md @@ -1,6 +1,6 @@ --- -stage: Plan -group: Project Management +stage: Ecosystem +group: Integrations info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/graphql_guide/index.md b/doc/development/graphql_guide/index.md index b8d4b53992e..cc97e41df05 100644 --- a/doc/development/graphql_guide/index.md +++ b/doc/development/graphql_guide/index.md @@ -1,6 +1,6 @@ --- -stage: Plan -group: Project Management +stage: Ecosystem +group: Integrations info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/graphql_guide/pagination.md b/doc/development/graphql_guide/pagination.md index 5fd2179ea9b..a37c47f1b11 100644 --- a/doc/development/graphql_guide/pagination.md +++ b/doc/development/graphql_guide/pagination.md @@ -1,6 +1,6 @@ --- -stage: Plan -group: Project Management +stage: Ecosystem +group: Integrations info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/i18n/externalization.md b/doc/development/i18n/externalization.md index 53825f0904a..52a7f839286 100644 --- a/doc/development/i18n/externalization.md +++ b/doc/development/i18n/externalization.md @@ -137,7 +137,7 @@ The `~/locale` module exports the following key functions for externalization: - `s__()` (namespaced double underscore parenthesis) - `__()` Mark content for translation (note the double underscore). - `s__()` Mark namespaced content for translation -- `n__()` Mark pluralized content for translation +- `n__()` Mark pluralized content for translation ```javascript import { __, s__, n__ } from '~/locale'; @@ -171,6 +171,45 @@ If you need to translate strings in the Vue component's JavaScript, you can impo To test Vue translations, learn about [manually testing translations from the UI](#manually-test-translations-from-the-ui). +### Test files + +Test expectations against externalized contents should not be hard coded, +because we may need to run the tests with non-default locale, and tests with +hard coded contents will fail. + +This means any expectations against externalized contents should call the +same externalizing method to match the translation. + +Bad: + +```ruby +click_button 'Submit review' + +expect(rendered).to have_content('Thank you for your feedback!') +``` + +Good: + +```ruby +click_button _('Submit review') + +expect(rendered).to have_content(_('Thank you for your feedback!')) +``` + +This includes JavaScript tests: + +Bad: + +```javascript +expect(findUpdateIgnoreStatusButton().text()).toBe('Ignore'); +``` + +Good: + +```javascript +expect(findUpdateIgnoreStatusButton().text()).toBe(__('Ignore')); +``` + #### Recommendations If strings are reused throughout a component, it can be useful to define these strings as variables. We recommend defining an `i18n` property on the component's `$options` object. If there is a mixture of many-use and single-use strings in the component, consider using this approach to create a local [Single Source of Truth](https://about.gitlab.com/handbook/values/#single-source-of-truth) for externalized strings. @@ -751,6 +790,28 @@ translate correctly if you extract individual words from the sentence. When in doubt, try to follow the best practices described in this [Mozilla Developer documentation](https://developer.mozilla.org/en-US/docs/Mozilla/Localization/Localization_content_best_practices#Splitting). +### Always pass string literals to the translation helpers + +The `bin/rake gettext:regenerate` script parses the codebase and extracts all the strings from the +[translation helpers](#preparing-a-page-for-translation) ready to be translated. + +The script cannot resolve the strings if they are passed as variables or function calls. Therefore, +make sure to always pass string literals to the helpers. + +```javascript +// Good +__('Some label'); +s__('Namespace', 'Label'); +s__('Namespace|Label'); +n__('%d apple', '%d apples', appleCount); + +// Bad +__(LABEL); +s__(getLabel()); +s__(NAMESPACE, LABEL); +n__(LABEL_SINGULAR, LABEL_PLURAL, appleCount); +``` + ## Updating the PO files with the new content Now that the new content is marked for translation, run this command to update the diff --git a/doc/development/i18n/proofreader.md b/doc/development/i18n/proofreader.md index 462c3fde7d6..76ab00eebfb 100644 --- a/doc/development/i18n/proofreader.md +++ b/doc/development/i18n/proofreader.md @@ -43,6 +43,7 @@ are very appreciative of the work done by translators and proofreaders! - Jan Urbanec - [GitLab](https://gitlab.com/TatranskyMedved), [Crowdin](https://crowdin.com/profile/Tatranskymedved) - Danish - Saederup92 - [GitLab](https://gitlab.com/Saederup92), [Crowdin](https://crowdin.com/profile/Saederup92) + - scootergrisen - [GitLab](https://gitlab.com/scootergrisen), [Crowdin](https://crowdin.com/profile/scootergrisen) - Dutch - Emily Hendle - [GitLab](https://gitlab.com/pundachan), [Crowdin](https://crowdin.com/profile/pandachan) - Esperanto @@ -98,7 +99,8 @@ are very appreciative of the work done by translators and proofreaders! - André Gama - [GitLab](https://gitlab.com/andregamma), [Crowdin](https://crowdin.com/profile/ToeOficial) - Eduardo Addad de Oliveira - [GitLab](https://gitlab.com/eduardoaddad), [Crowdin](https://crowdin.com/profile/eduardoaddad) - Romanian - - Proofreaders needed. + - Mircea Pop - [GitLab](https://gitlab.com/eeex), [Crowdin](https://crowdin.com/profile/eex) + - Rareș Pița - [GitLab](https://gitlab.com/dlphin), [Crowdin](https://crowdin.com/profile/dlphin) - Russian - Nikita Grylov - [GitLab](https://gitlab.com/nixel2007), [Crowdin](https://crowdin.com/profile/nixel2007) - Alexy Lustin - [GitLab](https://gitlab.com/allustin), [Crowdin](https://crowdin.com/profile/lustin) @@ -117,6 +119,7 @@ are very appreciative of the work done by translators and proofreaders! - Turkish - Ali Demirtaş - [GitLab](https://gitlab.com/alidemirtas), [Crowdin](https://crowdin.com/profile/alidemirtas) - Rıfat Ünalmış (Rifat Unalmis) - [GitLab](https://gitlab.com/runalmis), [Crowdin](https://crowdin.com/profile/runalmis) + - İsmail Arılık - [GitLab](https://gitlab.com/ismailarilik), [Crowdin](https://crowdin.com/profile/ismailarilik) - Ukrainian - Volodymyr Sobotovych - [GitLab](https://gitlab.com/wheleph), [Crowdin](https://crowdin.com/profile/wheleph) - Andrew Vityuk - [GitLab](https://gitlab.com/3_1_3_u), [Crowdin](https://crowdin.com/profile/andruwa13) diff --git a/doc/development/img/elasticsearch_architecture.svg b/doc/development/img/elasticsearch_architecture.svg index 2f38f9b04ee..516214c8b8e 100644 --- a/doc/development/img/elasticsearch_architecture.svg +++ b/doc/development/img/elasticsearch_architecture.svg @@ -1 +1 @@ -<svg version="1.2" width="210mm" height="297mm" viewBox="0 0 21000 29700" preserveAspectRatio="xMidYMid" fill-rule="evenodd" stroke-width="28.222" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><defs class="ClipPathGroup"><clipPath id="a" clipPathUnits="userSpaceOnUse"><path d="M0 0h21000v29700H0z"/></clipPath></defs><g class="SlideGroup"><g class="Slide" clip-path="url(#a)"><g class="Page"><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M1975 5575h3051v1651H1975z"/><path fill="#FFF" d="M3500 7200H2000V5600h3000v1600H3500z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M3500 7200H2000V5600h3000v1600H3500z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="2778" y="6311"><tspan>Snippet</tspan></tspan><tspan class="TextPosition" x="2099" y="6785"><tspan>(ActiveRecord)</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M1475 3975h4051v3551H1475z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M3500 7500H1500V4000h4000v3500H3500z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="1788" y="5048"><tspan>ApplicationSearch</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M5975 4675h8051v701H5975z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M6000 5350h4000v-650h4000"/></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M5975 5325h8051v1101H5975z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M6000 5350h4000v1050h4000"/></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M1075 2875h4951v4951H1075z"/><path fill="none" stroke="#F33" stroke-width="50" d="M3550 7800H1100V2900h4900v4900H3550z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="700"><tspan class="TextPosition" x="1946" y="3514"><tspan fill="#C9211E">SnippetsSearch</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M1975 12175h3051v1651H1975z"/><path fill="#FFF" d="M3500 13800H2000v-1600h3000v1600H3500z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M3500 13800H2000v-1600h3000v1600H3500z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="2778" y="12911"><tspan>Snippet</tspan></tspan><tspan class="TextPosition" x="2099" y="13385"><tspan>(ActiveRecord)</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M1075 10775h4951v3251H1075z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M3550 14000H1100v-3200h4900v3200H3550z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="2511" y="11461"><tspan>Application</tspan></tspan><tspan class="TextPosition" x="1933" y="11935"><tspan>VersionedSearch</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M3525 13975h4501v7451H3525z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M3550 14000v7400h4450"/></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M14008 14075h4985v851h-4985z"/><path fill="none" stroke="#999" stroke-width="50" d="M16500 14900h-2467v-800h4934v800h-2467z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="14720" y="14648"><tspan fill="gray">ClassMethodProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M13375 13075h6251v2151h-6251z"/><path fill="none" stroke="#F33" stroke-width="50" d="M16500 15200h-3100v-2100h6200v2100h-3100z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="700"><tspan class="TextPosition" x="13799" y="13731"><tspan fill="#C9211E">V12p1::SnippetClassProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M7975 14575h3051v1851H7975z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M9500 16400H8000v-1800h3000v1800H9500z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="8277" y="15411"><tspan>MultiVersion-</tspan></tspan><tspan class="TextPosition" x="8429" y="15885"><tspan>ClassProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M14008 16875h4985v851h-4985z"/><path fill="none" stroke="#999" stroke-width="50" d="M16500 17700h-2467v-800h4934v800h-2467z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="14720" y="17448"><tspan fill="gray">ClassMethodProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M13375 15875h6251v2151h-6251z"/><path fill="none" stroke="#F33" stroke-width="50" d="M16500 18000h-3100v-2100h6200v2100h-3100z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="700"><tspan class="TextPosition" x="13799" y="16531"><tspan fill="#C9211E">V12p2::SnippetClassProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M10975 14125h2451v1401h-2451z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M11000 15500h1463v-1350h937"/></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M10975 15475h2451v1501h-2451z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M11000 15500h1463v1450h937"/></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M3525 13975h4501v1551H3525z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M3550 14000v1500h4450"/></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M14008 19975h4985v851h-4985z"/><path fill="none" stroke="#999" stroke-width="50" d="M16500 20800h-2467v-800h4934v800h-2467z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="14445" y="20548"><tspan fill="gray">InstanceMethodProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M13375 18975h6251v2151h-6251z"/><path fill="none" stroke="#F33" stroke-width="50" d="M16500 21100h-3100v-2100h6200v2100h-3100z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="700"><tspan class="TextPosition" x="13505" y="19631"><tspan fill="#C9211E">V12p1::SnippetInstanceProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M7975 20275h3051v2251H7975z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M9500 22500H8000v-2200h3000v2200H9500z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="8277" y="21311"><tspan>MultiVersion-</tspan></tspan><tspan class="TextPosition" x="8154" y="21785"><tspan>InstanceProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M14008 22775h4985v851h-4985z"/><path fill="none" stroke="#999" stroke-width="50" d="M16500 23600h-2467v-800h4934v800h-2467z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="14445" y="23348"><tspan fill="gray">InstanceMethodProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M13375 21775h6251v2151h-6251z"/><path fill="none" stroke="#F33" stroke-width="50" d="M16500 23900h-3100v-2100h6200v2100h-3100z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="700"><tspan class="TextPosition" x="13505" y="22431"><tspan fill="#C9211E">V12p2::SnippetInstanceProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M10975 20025h2451v1401h-2451z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M11000 21400h1463v-1350h937"/></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M10975 21375h2451v1501h-2451z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M11000 21400h1463v1450h937"/></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M900 1600h10697v879H900z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="564" font-weight="400"><tspan class="TextPosition" x="1150" y="2233"><tspan>Standard elasticsearch-rails setup</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M900 9300h7683v879H900z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="564" font-weight="400"><tspan class="TextPosition" x="1150" y="9933"><tspan>GitLab multi-indices setup</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M3400 21300h4821v1197H3400z"/><text class="TextShape"><tspan class="TextParagraph" font-size="388" font-weight="400"><tspan class="TextPosition" x="4250" y="21840"><tspan fill="gray">(instance method)</tspan></tspan><tspan class="TextPosition" x="3651" y="22264"><tspan font-family="Courier" font-size="423">__elasticsearch__</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M3380 15400h4821v1197H3380z"/><text class="TextShape"><tspan class="TextParagraph" font-size="388" font-weight="400"><tspan class="TextPosition" x="4512" y="15940"><tspan fill="gray">(class method)</tspan></tspan><tspan class="TextPosition" x="3631" y="16364"><tspan font-family="Courier" font-size="423">__elasticsearch__</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M9000 3500h4821v1197H9000z"/><text class="TextShape"><tspan class="TextParagraph" font-size="388" font-weight="400"><tspan class="TextPosition" x="10132" y="4040"><tspan fill="gray">(class method)</tspan></tspan><tspan class="TextPosition" x="9251" y="4464"><tspan font-family="Courier" font-size="423">__elasticsearch__</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M9000 6400h4821v1197H9000z"/><text class="TextShape"><tspan class="TextParagraph" font-size="388" font-weight="400"><tspan class="TextPosition" x="9850" y="6940"><tspan fill="gray">(instance method)</tspan></tspan><tspan class="TextPosition" x="9251" y="7364"><tspan font-family="Courier" font-size="423">__elasticsearch__</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M1975 25175h2051v851H1975z"/><path fill="none" stroke="#999" stroke-width="50" d="M3000 26000H2000v-800h2000v800H3000z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="2634" y="25748"><tspan fill="gray">Foo</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M4400 25200h7101v726H4400z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="4650" y="25710"><tspan>elasticsearch-rails’ internal class</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M4400 26400h8601v1200H4400z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="4650" y="26910"><tspan>where model-specific logic is</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M1975 26275h2051v851H1975z"/><path fill="none" stroke="#F33" stroke-width="50" d="M3000 27100H2000v-800h2000v800H3000z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="700"><tspan class="TextPosition" x="2613" y="26848"><tspan fill="#C9211E">Foo</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M4900 17289h5901v2312H4900z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="370" font-weight="400"><tspan class="TextPosition" x="7236" y="17748"><tspan fill="gray">Write operations like </tspan></tspan><tspan class="TextPosition" x="5323" y="18159"><tspan fill="gray">indexing/updating are forwarded </tspan></tspan><tspan class="TextPosition" x="8024" y="18570"><tspan fill="gray">to all instances.</tspan></tspan></tspan><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="370" font-weight="400"><tspan class="TextPosition" x="5501" y="18981"><tspan fill="gray">Read operations are forwarded </tspan></tspan><tspan class="TextPosition" x="7126" y="19392"><tspan fill="gray">to specified instance.</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M10785 15769h1422v2691h-1422z"/><path fill="none" stroke="#999" stroke-width="30" d="M10800 18444c1429 0 934-1618 1119-2337"/><path fill="#999" d="M12206 15769l-460 293 267 217 193-510z"/></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M10785 18429h1528v2862h-1528z"/><path fill="none" stroke="#999" stroke-width="30" d="M10800 18444c1509 0 970 1782 1200 2526"/><path fill="#999" d="M12312 21290l-227-496-252 235 479 261z"/></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M1800 24000h7101v807H1800z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="494" font-weight="700"><tspan class="TextPosition" x="2050" y="24574"><tspan>Legend</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M13975 4275h5085v851h-5085z"/><path fill="none" stroke="#999" stroke-width="50" d="M16517 5100h-2517v-800h5034v800h-2517z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="14737" y="4848"><tspan fill="gray">ClassMethodProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M13975 5975h5085v851h-5085z"/><path fill="none" stroke="#999" stroke-width="50" d="M16517 6800h-2517v-800h5034v800h-2517z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="14462" y="6548"><tspan fill="gray">InstanceMethodProxy</tspan></tspan></tspan></text></g></g></g></g></svg>
\ No newline at end of file +<svg version="1.2" width="210mm" height="297mm" viewBox="0 0 21000 29700" preserveAspectRatio="xMidYMid" fill-rule="evenodd" stroke-width="28.222" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><defs class="ClipPathGroup"><clipPath id="a" clipPathUnits="userSpaceOnUse"><path d="M0 0h21000v29700H0z"/></clipPath></defs><g class="SlideGroup"><g class="Slide" clip-path="url(#a)"><g class="Page"><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M1975 5575h3051v1651H1975z"/><path fill="#FFF" d="M3500 7200H2000V5600h3000v1600H3500z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M3500 7200H2000V5600h3000v1600H3500z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="2778" y="6311"><tspan>Snippet</tspan></tspan><tspan class="TextPosition" x="2099" y="6785"><tspan>(ActiveRecord)</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M1475 3975h4051v3551H1475z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M3500 7500H1500V4000h4000v3500H3500z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="1788" y="5048"><tspan>ApplicationSearch</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M5975 4675h8051v701H5975z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M6000 5350h4000v-650h4000"/></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M5975 5325h8051v1101H5975z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M6000 5350h4000v1050h4000"/></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M1075 2875h4951v4951H1075z"/><path fill="none" stroke="#F33" stroke-width="50" d="M3550 7800H1100V2900h4900v4900H3550z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="700"><tspan class="TextPosition" x="1946" y="3514"><tspan fill="#C9211E">SnippetsSearch</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M1975 12175h3051v1651H1975z"/><path fill="#FFF" d="M3500 13800H2000v-1600h3000v1600H3500z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M3500 13800H2000v-1600h3000v1600H3500z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="2778" y="12911"><tspan>Snippet</tspan></tspan><tspan class="TextPosition" x="2099" y="13385"><tspan>(ActiveRecord)</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M1075 10775h4951v3251H1075z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M3550 14000H1100v-3200h4900v3200H3550z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="2511" y="11461"><tspan>Application</tspan></tspan><tspan class="TextPosition" x="1933" y="11935"><tspan>VersionedSearch</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M3525 13975h4501v7451H3525z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M3550 14000v7400h4450"/></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M14008 14075h4985v851h-4985z"/><path fill="none" stroke="#999" stroke-width="50" d="M16500 14900h-2467v-800h4934v800h-2467z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="14720" y="14648"><tspan fill="gray">ClassMethodProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M13375 13075h6251v2151h-6251z"/><path fill="none" stroke="#F33" stroke-width="50" d="M16500 15200h-3100v-2100h6200v2100h-3100z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="700"><tspan class="TextPosition" x="13799" y="13731"><tspan fill="#C9211E">V12p1::SnippetClassProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M7975 14575h3051v1851H7975z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M9500 16400H8000v-1800h3000v1800H9500z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="8277" y="15411"><tspan>MultiVersion-</tspan></tspan><tspan class="TextPosition" x="8429" y="15885"><tspan>ClassProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M14008 16875h4985v851h-4985z"/><path fill="none" stroke="#999" stroke-width="50" d="M16500 17700h-2467v-800h4934v800h-2467z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="14720" y="17448"><tspan fill="gray">ClassMethodProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M13375 15875h6251v2151h-6251z"/><path fill="none" stroke="#F33" stroke-width="50" d="M16500 18000h-3100v-2100h6200v2100h-3100z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="700"><tspan class="TextPosition" x="13799" y="16531"><tspan fill="#C9211E">V12p2::SnippetClassProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M10975 14125h2451v1401h-2451z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M11000 15500h1463v-1350h937"/></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M10975 15475h2451v1501h-2451z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M11000 15500h1463v1450h937"/></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M3525 13975h4501v1551H3525z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M3550 14000v1500h4450"/></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M14008 19975h4985v851h-4985z"/><path fill="none" stroke="#999" stroke-width="50" d="M16500 20800h-2467v-800h4934v800h-2467z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="14445" y="20548"><tspan fill="gray">InstanceMethodProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M13375 18975h6251v2151h-6251z"/><path fill="none" stroke="#F33" stroke-width="50" d="M16500 21100h-3100v-2100h6200v2100h-3100z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="700"><tspan class="TextPosition" x="13505" y="19631"><tspan fill="#C9211E">V12p1::SnippetInstanceProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M7975 20275h3051v2251H7975z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M9500 22500H8000v-2200h3000v2200H9500z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="8277" y="21311"><tspan>MultiVersion-</tspan></tspan><tspan class="TextPosition" x="8154" y="21785"><tspan>InstanceProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M14008 22775h4985v851h-4985z"/><path fill="none" stroke="#999" stroke-width="50" d="M16500 23600h-2467v-800h4934v800h-2467z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="14445" y="23348"><tspan fill="gray">InstanceMethodProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M13375 21775h6251v2151h-6251z"/><path fill="none" stroke="#F33" stroke-width="50" d="M16500 23900h-3100v-2100h6200v2100h-3100z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="700"><tspan class="TextPosition" x="13505" y="22431"><tspan fill="#C9211E">V12p2::SnippetInstanceProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M10975 20025h2451v1401h-2451z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M11000 21400h1463v-1350h937"/></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M10975 21375h2451v1501h-2451z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M11000 21400h1463v1450h937"/></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M900 1600h10697v879H900z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="564" font-weight="400"><tspan class="TextPosition" x="1150" y="2233"><tspan>Standard elasticsearch-rails setup</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M900 9300h7683v879H900z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="564" font-weight="400"><tspan class="TextPosition" x="1150" y="9933"><tspan>GitLab multi-indices setup</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M3400 21300h4821v1197H3400z"/><text class="TextShape"><tspan class="TextParagraph" font-size="388" font-weight="400"><tspan class="TextPosition" x="4250" y="21840"><tspan fill="gray">(instance method)</tspan></tspan><tspan class="TextPosition" x="3651" y="22264"><tspan font-family="Courier" font-size="423">__elasticsearch__</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M3380 15400h4821v1197H3380z"/><text class="TextShape"><tspan class="TextParagraph" font-size="388" font-weight="400"><tspan class="TextPosition" x="4512" y="15940"><tspan fill="gray">(class method)</tspan></tspan><tspan class="TextPosition" x="3631" y="16364"><tspan font-family="Courier" font-size="423">__elasticsearch__</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M9000 3500h4821v1197H9000z"/><text class="TextShape"><tspan class="TextParagraph" font-size="388" font-weight="400"><tspan class="TextPosition" x="10132" y="4040"><tspan fill="gray">(class method)</tspan></tspan><tspan class="TextPosition" x="9251" y="4464"><tspan font-family="Courier" font-size="423">__elasticsearch__</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M9000 6400h4821v1197H9000z"/><text class="TextShape"><tspan class="TextParagraph" font-size="388" font-weight="400"><tspan class="TextPosition" x="9850" y="6940"><tspan fill="gray">(instance method)</tspan></tspan><tspan class="TextPosition" x="9251" y="7364"><tspan font-family="Courier" font-size="423">__elasticsearch__</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M1975 25175h2051v851H1975z"/><path fill="none" stroke="#999" stroke-width="50" d="M3000 26000H2000v-800h2000v800H3000z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="2634" y="25748"><tspan fill="gray">Foo</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M4400 25200h7101v726H4400z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="4650" y="25710"><tspan>elasticsearch-rails' internal class</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M4400 26400h8601v1200H4400z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="4650" y="26910"><tspan>where model-specific logic is</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M1975 26275h2051v851H1975z"/><path fill="none" stroke="#F33" stroke-width="50" d="M3000 27100H2000v-800h2000v800H3000z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="700"><tspan class="TextPosition" x="2613" y="26848"><tspan fill="#C9211E">Foo</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M4900 17289h5901v2312H4900z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="370" font-weight="400"><tspan class="TextPosition" x="7236" y="17748"><tspan fill="gray">Write operations like </tspan></tspan><tspan class="TextPosition" x="5323" y="18159"><tspan fill="gray">indexing/updating are forwarded </tspan></tspan><tspan class="TextPosition" x="8024" y="18570"><tspan fill="gray">to all instances.</tspan></tspan></tspan><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="370" font-weight="400"><tspan class="TextPosition" x="5501" y="18981"><tspan fill="gray">Read operations are forwarded </tspan></tspan><tspan class="TextPosition" x="7126" y="19392"><tspan fill="gray">to specified instance.</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M10785 15769h1422v2691h-1422z"/><path fill="none" stroke="#999" stroke-width="30" d="M10800 18444c1429 0 934-1618 1119-2337"/><path fill="#999" d="M12206 15769l-460 293 267 217 193-510z"/></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M10785 18429h1528v2862h-1528z"/><path fill="none" stroke="#999" stroke-width="30" d="M10800 18444c1509 0 970 1782 1200 2526"/><path fill="#999" d="M12312 21290l-227-496-252 235 479 261z"/></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M1800 24000h7101v807H1800z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="494" font-weight="700"><tspan class="TextPosition" x="2050" y="24574"><tspan>Legend</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M13975 4275h5085v851h-5085z"/><path fill="none" stroke="#999" stroke-width="50" d="M16517 5100h-2517v-800h5034v800h-2517z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="14737" y="4848"><tspan fill="gray">ClassMethodProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M13975 5975h5085v851h-5085z"/><path fill="none" stroke="#999" stroke-width="50" d="M16517 6800h-2517v-800h5034v800h-2517z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="14462" y="6548"><tspan fill="gray">InstanceMethodProxy</tspan></tspan></tspan></text></g></g></g></g></svg>
\ No newline at end of file diff --git a/doc/development/import_project.md b/doc/development/import_project.md index 69e5873cd87..d021126c8eb 100644 --- a/doc/development/import_project.md +++ b/doc/development/import_project.md @@ -195,7 +195,7 @@ You can use this snippet: `https://gitlab.com/gitlab-org/gitlab/snippets/1924954 You can execute the script from the `gdk/gitlab` directory like this: ```shell -bundle exec rails r /path_to_sript/script.rb project_name /path_to_extracted_project request_store_enabled +bundle exec rails r /path_to_script/script.rb project_name /path_to_extracted_project request_store_enabled ``` ## Troubleshooting diff --git a/doc/development/integrations/jenkins.md b/doc/development/integrations/jenkins.md index a1ad259319d..3987c6658c3 100644 --- a/doc/development/integrations/jenkins.md +++ b/doc/development/integrations/jenkins.md @@ -24,8 +24,8 @@ brew services start jenkins GitLab does not allow requests to localhost or the local network by default. When running Jenkins on your local machine, you need to enable local access. 1. Log into your GitLab instance as an administrator. -1. On the top bar, select **Menu >** **{admin}** **Admin**. -1. In the left sidebar, select **Settings > Network**. +1. On the top bar, select **Menu > Admin**. +1. On the left sidebar, select **Settings > Network**. 1. Expand **Outbound requests** and check the following checkboxes: - **Allow requests to the local network from web hooks and services** diff --git a/doc/development/integrations/jira_connect.md b/doc/development/integrations/jira_connect.md index 9772f7504cf..ca3dc3660ee 100644 --- a/doc/development/integrations/jira_connect.md +++ b/doc/development/integrations/jira_connect.md @@ -54,7 +54,7 @@ To install the app in Jira: 1. Click **Upload**. - If the install was successful, you should see the **GitLab for Jira** app under **Manage apps**. + If the install was successful, you should see the **GitLab.com for Jira Cloud** app under **Manage apps**. You can also click **Getting Started** to open the configuration page rendered from your GitLab instance. _Note that any changes to the app descriptor requires you to uninstall then reinstall the app._ diff --git a/doc/development/integrations/secure.md b/doc/development/integrations/secure.md index 42a57e7f4fb..d37ce29e353 100644 --- a/doc/development/integrations/secure.md +++ b/doc/development/integrations/secure.md @@ -444,6 +444,10 @@ the system saves only the first 20 of them. Note that vulnerabilities in the [Pi Security](../../user/application_security/security_dashboard/#pipeline-security) tab do not enforce this limit and all identifiers present in the report artifact are displayed. +### Details + +The `details` field is an object that supports many different content elements that are displayed when viewing vulnerability information. An example of the various data elements can be seen in the [security-reports repository](https://gitlab.com/gitlab-examples/security/security-reports/-/tree/master/samples/details-example). + ### Location The `location` indicates where the vulnerability has been detected. @@ -454,10 +458,6 @@ which is used to track vulnerabilities as new commits are pushed to the repository. The attributes used to generate the location fingerprint also depend on the type of scanning. -### Details - -The `details` field is an object that supports many different content elements that are displayed when viewing vulnerability information. An example of the various data elements can be seen in the [security-reports repository](https://gitlab.com/gitlab-examples/security/security-reports/-/tree/master/samples/details-example). - #### Dependency Scanning The `location` of a Dependency Scanning vulnerability is composed of a `dependency` and a `file`. diff --git a/doc/development/internal_api.md b/doc/development/internal_api.md index c7fc4bed38c..660d8c60ba8 100644 --- a/doc/development/internal_api.md +++ b/doc/development/internal_api.md @@ -501,6 +501,56 @@ curl --request POST --header "Gitlab-Kas-Api-Request: <JWT token>" \ "http://localhost:3000/api/v4/internal/kubernetes/modules/cilium_alert" ``` +### Create Starboard vulnerability + +Called from the GitLab Kubernetes Agent Server (`kas`) to create a security vulnerability +from a Starboard vulnerability report. This request is idempotent. Multiple requests with the same data +create a single vulnerability. + +| Attribute | Type | Required | Description | +|:----------------|:-------|:---------|:------------| +| `vulnerability` | Hash | yes | Vulnerability data matching the security report schema [`vulnerability` field](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/src/security-report-format.json). | +| `scanner` | Hash | yes | Scanner data matching the security report schema [`scanner` field](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/src/security-report-format.json). | + +```plaintext +PUT internal/kubernetes/modules/starboard_vulnerability +``` + +Example Request: + +```shell +curl --request PUT --header "Gitlab-Kas-Api-Request: <JWT token>" \ + --header "Authorization: Bearer <agent token>" --header "Content-Type: application/json" \ + --url "http://localhost:3000/api/v4/internal/kubernetes/modules/starboard_vulnerability" \ + --data '{ + "vulnerability": { + "name": "CVE-123-4567 in libc", + "severity": "high", + "confidence": "unknown", + "location": { + "kubernetes_resource": { + "namespace": "production", + "kind": "deployment", + "name": "nginx", + "container": "nginx" + } + }, + "identifiers": [ + { + "type": "cve", + "name": "CVE-123-4567", + "value": "CVE-123-4567" + } + ] + }, + "scanner": { + "id": "starboard_trivy", + "name": "Trivy (via Starboard Operator)", + "vendor": "GitLab" + } +}' +``` + ## Subscriptions The subscriptions endpoint is used by [CustomersDot](https://gitlab.com/gitlab-org/customers-gitlab-com) (`customers.gitlab.com`) @@ -675,7 +725,7 @@ Example request: ```shell curl --request POST \ - --url http://localhost:3000/api/v4/namespaces/123/minutes \ + --url "http://localhost:3000/api/v4/namespaces/123/minutes" \ --header 'Content-Type: application/json' \ --header 'PRIVATE-TOKEN: <admin access token>' \ --data '{ @@ -719,7 +769,7 @@ Example request: ```shell curl --request PATCH \ - --url http://localhost:3000/api/v4/namespaces/123/minutes/move/321 \ + --url "http://localhost:3000/api/v4/namespaces/123/minutes/move/321" \ --header 'PRIVATE-TOKEN: <admin access token>' ``` diff --git a/doc/development/issue_types.md b/doc/development/issue_types.md index d02ff590352..31fa50e1d97 100644 --- a/doc/development/issue_types.md +++ b/doc/development/issue_types.md @@ -4,7 +4,10 @@ group: Project Management info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Issue Types +# Issue Types (DEPRECATED) + +WARNING: +We are deprecating Issue Types as of GitLab 14.2 in favor of [Work Items and Work Item Types](work_items.md). Sometimes when a new resource type is added it's not clear if it should be only an "extension" of Issue (Issue Type) or if it should be a new first-class resource type diff --git a/doc/development/migration_style_guide.md b/doc/development/migration_style_guide.md index bbaa6527e84..ce564551fbf 100644 --- a/doc/development/migration_style_guide.md +++ b/doc/development/migration_style_guide.md @@ -70,7 +70,7 @@ graph LR H -->|Yes| E[Regular migration] H -->|No| I[Post-deploy migration<br/>+ feature flag] - + D -->|Yes| F[Post-deploy migration] D -->|No| G[Background migration] ``` @@ -217,6 +217,40 @@ In case you need to insert, update, or delete a significant amount of data, you: - Must disable the single transaction with `disable_ddl_transaction!`. - Should consider doing it in a [Background Migration](background_migrations.md). +## Migration helpers and versioning + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339115) in GitLab 14.3. + +Various helper methods are available for many common patterns in database migrations. Those +helpers can be found in `Gitlab::Database::MigrationHelpers` and related modules. + +In order to allow changing a helper's behavior over time, we implement a versioning scheme +for migration helpers. This allows us to maintain the behavior of a helper for already +existing migrations but change the behavior for any new migrations. + +For that purpose, all database migrations should inherit from `Gitlab::Database::Migration`, +which is a "versioned" class. For new migrations, the latest version should be used (which +can be looked up in `Gitlab::Database::Migration::MIGRATION_CLASSES`) to use the latest version +of migration helpers. + +In this example, we use version 1.0 of the migration class: + +```ruby +class TestMigration < Gitlab::Database::Migration[1.0] + def change + end +end +``` + +Do not include `Gitlab::Database::MigrationHelpers` directly into a +migration. Instead, use the latest version of `Gitlab::Database::Migration`, which exposes the latest +version of migration helpers automatically. + +Migration helpers and versioning were [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68986) +in GitLab 14.3. +For merge requests targeting previous stable branches, use the old format and still inherit from +`ActiveRecord::Migration[6.1]` instead of `Gitlab::Database::Migration[1.0]`. + ## Retry mechanism when acquiring database locks When changing the database schema, we use helper methods to invoke DDL (Data Definition @@ -247,87 +281,91 @@ This problem could cause failed application upgrade processes and even applicati stability issues, since the table may be inaccessible for a short period of time. To increase the reliability and stability of database migrations, the GitLab codebase -offers a helper method to retry the operations with different `lock_timeout` settings -and wait time between the attempts. Multiple smaller attempts to acquire the necessary +offers a method to retry the operations with different `lock_timeout` settings +and wait time between the attempts. Multiple shorter attempts to acquire the necessary lock allow the database to process other statements. -### Examples +There are two distinct ways to use lock retries: + +1. Inside a transactional migration: use `enable_lock_retries!`. +1. Inside a non-transactional migration: use `with_lock_retries`. + +If possible, enable lock-retries for any migration that touches a [high-traffic table](#high-traffic-tables). + +### Usage with transactional migrations + +Regular migrations execute the full migration in a transaction. We can enable the +lock-retry methodology by calling `enable_lock_retries!` at the migration level. + +This leads to the lock timeout being controlled for this migration. Also, it can lead to retrying the full +migration if the lock could not be granted within the timeout. + +Note that, while this is currently an opt-in setting, we prefer to use lock-retries for all migrations and +plan to make this the default going forward. + +Occasionally a migration may need to acquire multiple locks on different objects. +To prevent catalog bloat, ask for all those locks explicitly before performing any DDL. +A better strategy is to split the migration, so that we only need to acquire one lock at the time. **Removing a column:** ```ruby -include Gitlab::Database::MigrationHelpers +enable_lock_retries! def up - with_lock_retries do - remove_column :users, :full_name - end + remove_column :users, :full_name end def down - with_lock_retries do - add_column :users, :full_name, :string - end + add_column :users, :full_name, :string end ``` **Multiple changes on the same table:** -The helper `with_lock_retries` wraps all operations into a single transaction. When you have the lock, +With the lock-retry methodology enabled, all operations wrap into a single transaction. When you have the lock, you should do as much as possible inside the transaction rather than trying to get another lock later. Be careful about running long database statements within the block. The acquired locks are kept until the transaction (block) finishes and depending on the lock type, it might block other database operations. ```ruby -include Gitlab::Database::MigrationHelpers +enable_lock_retries! def up - with_lock_retries do - add_column :users, :full_name, :string - add_column :users, :bio, :string - end + add_column :users, :full_name, :string + add_column :users, :bio, :string end def down - with_lock_retries do - remove_column :users, :full_name - remove_column :users, :bio - end + remove_column :users, :full_name + remove_column :users, :bio end ``` **Removing a foreign key:** ```ruby -include Gitlab::Database::MigrationHelpers +enable_lock_retries! def up - with_lock_retries do - remove_foreign_key :issues, :projects - end + remove_foreign_key :issues, :projects end def down - with_lock_retries do - add_foreign_key :issues, :projects - end + add_foreign_key :issues, :projects end ``` **Changing default value for a column:** ```ruby -include Gitlab::Database::MigrationHelpers +enable_lock_retries! def up - with_lock_retries do - change_column_default :merge_requests, :lock_version, from: nil, to: 0 - end + change_column_default :merge_requests, :lock_version, from: nil, to: 0 end def down - with_lock_retries do - change_column_default :merge_requests, :lock_version, from: 0, to: nil - end + change_column_default :merge_requests, :lock_version, from: 0, to: nil end ``` @@ -336,25 +374,23 @@ end We can wrap the `create_table` method with `with_lock_retries`: ```ruby +enable_lock_retries! + def up - with_lock_retries do - create_table :issues do |t| - t.references :project, index: true, null: false, foreign_key: { on_delete: :cascade } - t.string :title, limit: 255 - end + create_table :issues do |t| + t.references :project, index: true, null: false, foreign_key: { on_delete: :cascade } + t.string :title, limit: 255 end end def down - with_lock_retries do - drop_table :issues - end + drop_table :issues end ``` **Creating a new table when we have two foreign keys:** -Only one foreign key should be created per migration. This is because [the addition of a foreign key constraint requires a `SHARE ROW EXCLUSIVE` lock on the referenced table](https://www.postgresql.org/docs/12/sql-createtable.html#:~:text=The%20addition%20of%20a%20foreign%20key%20constraint%20requires%20a%20SHARE%20ROW%20EXCLUSIVE%20lock%20on%20the%20referenced%20table), and locking multiple tables in the same transaction should be avoided. +Only one foreign key should be created per transaction. This is because [the addition of a foreign key constraint requires a `SHARE ROW EXCLUSIVE` lock on the referenced table](https://www.postgresql.org/docs/12/sql-createtable.html#:~:text=The%20addition%20of%20a%20foreign%20key%20constraint%20requires%20a%20SHARE%20ROW%20EXCLUSIVE%20lock%20on%20the%20referenced%20table), and locking multiple tables in the same transaction should be avoided. For this, we need three migrations: @@ -387,8 +423,6 @@ We can use the `add_concurrent_foreign_key` method in this case, as this helper has the lock retries built into it. ```ruby -include Gitlab::Database::MigrationHelpers - disable_ddl_transaction! def up @@ -405,8 +439,6 @@ end Adding foreign key to `users`: ```ruby -include Gitlab::Database::MigrationHelpers - disable_ddl_transaction! def up @@ -420,16 +452,20 @@ def down end ``` -**Usage with `disable_ddl_transaction!`** +### Usage with non-transactional migrations (`disable_ddl_transaction!`) -Generally the `with_lock_retries` helper should work with `disable_ddl_transaction!`. A custom RuboCop rule ensures that only allowed methods can be placed within the lock retries block. +Only when we disable transactional migrations using `disable_ddl_transaction!`, we can use +the `with_lock_retries` helper to guard an individual sequence of steps. It opens a transaction +to execute the given block. + +A custom RuboCop rule ensures that only allowed methods can be placed within the lock retries block. ```ruby disable_ddl_transaction! def up with_lock_retries do - add_column :users, :name, :text + add_column :users, :name, :text unless column_exists?(:users, :name) end add_text_limit :users, :name, 255 # Includes constraint validation (full table scan) @@ -450,7 +486,8 @@ end ### When to use the helper method -The `with_lock_retries` helper method can be used when you normally use +You can **only** use the `with_lock_retries` helper method when the execution is not already inside +an open transaction (using Postgres subtransactions is discouraged). It can be used with standard Rails migration helper methods. Calling more than one migration helper is not a problem if they're executed on the same table. @@ -498,11 +535,11 @@ by calling the method `disable_ddl_transaction!` in the body of your migration class like so: ```ruby -class MyMigration < ActiveRecord::Migration[6.0] - include Gitlab::Database::MigrationHelpers +class MyMigration < Gitlab::Database::Migration[1.0] disable_ddl_transaction! INDEX_NAME = 'index_name' + def up remove_concurrent_index :table_name, :column_name, name: INDEX_NAME end @@ -549,7 +586,7 @@ by calling the method `disable_ddl_transaction!` in the body of your migration class like so: ```ruby -class MyMigration < ActiveRecord::Migration[6.0] +class MyMigration < Gitlab::Database::Migration[1.0] include Gitlab::Database::MigrationHelpers disable_ddl_transaction! @@ -594,7 +631,7 @@ The easiest way to test for existence of an index by name is to use the be used with a name option. For example: ```ruby -class MyMigration < ActiveRecord::Migration[6.0] +class MyMigration < Gitlab::Database::Migration[1.0] include Gitlab::Database::MigrationHelpers INDEX_NAME = 'index_name' @@ -631,7 +668,7 @@ Here's an example where we add a new column with a foreign key constraint. Note it includes `index: true` to create an index for it. ```ruby -class Migration < ActiveRecord::Migration[6.0] +class Migration < Gitlab::Database::Migration[1.0] def change add_reference :model, :other_model, index: true, foreign_key: { on_delete: :cascade } @@ -677,7 +714,7 @@ expensive and disruptive operation for larger tables, but in reality it's not. Take the following migration as an example: ```ruby -class DefaultRequestAccessGroups < ActiveRecord::Migration[5.2] +class DefaultRequestAccessGroups < Gitlab::Database::Migration[1.0] def change change_column_default(:namespaces, :request_access_enabled, from: false, to: true) end @@ -884,7 +921,7 @@ The Rails 5 natively supports `JSONB` (binary JSON) column type. Example migration adding this column: ```ruby -class AddOptionsToBuildMetadata < ActiveRecord::Migration[5.0] +class AddOptionsToBuildMetadata < Gitlab::Database::Migration[1.0] def change add_column :ci_builds_metadata, :config_options, :jsonb end @@ -916,7 +953,7 @@ Do not store `attr_encrypted` attributes as `:text` in the database; use efficient: ```ruby -class AddSecretToSomething < ActiveRecord::Migration[5.0] +class AddSecretToSomething < Gitlab::Database::Migration[1.0] def change add_column :something, :encrypted_secret, :binary add_column :something, :encrypted_secret_iv, :binary @@ -974,7 +1011,7 @@ If you need more complex logic, you can define and use models local to a migration. For example: ```ruby -class MyMigration < ActiveRecord::Migration[6.0] +class MyMigration < Gitlab::Database::Migration[1.0] class Project < ActiveRecord::Base self.table_name = 'projects' end @@ -1073,7 +1110,7 @@ in a previous migration. It is important not to leave out the `User.reset_column_information` command, in order to ensure that the old schema is dropped from the cache and ActiveRecord loads the updated schema information. ```ruby -class AddAndSeedMyColumn < ActiveRecord::Migration[6.0] +class AddAndSeedMyColumn < Gitlab::Database::Migration[1.0] class User < ActiveRecord::Base self.table_name = 'users' end diff --git a/doc/development/multi_version_compatibility.md b/doc/development/multi_version_compatibility.md index 3314b5e7ddc..f834f4f4ee3 100644 --- a/doc/development/multi_version_compatibility.md +++ b/doc/development/multi_version_compatibility.md @@ -124,7 +124,7 @@ GitLab.com, the feature can be enabled in ChatOps and validated on GitLab.com. **However, it is not necessarily safe to enable the feature by default.** If the feature flag is removed, or the default is flipped to enabled, in the same release -where the code was merged, then customers performing [zero-downtime updates](https://docs.gitlab.com/omnibus/update/#zero-downtime-updates) +where the code was merged, then customers performing [zero-downtime updates](../update/zero_downtime.md) will end up running the new frontend code against the previous release's API. If you're not sure whether it's safe to enable all the changes at once, then one @@ -201,7 +201,7 @@ gantt section Database Schema A :done, schemaA, 00:00 , 1h Schema B :crit, schemaB, after migr, 58m - Schema C. : schmeaC, after postmigr, 1h + Schema C. : schemaC, after postmigr, 1h section Machine A Version N :done, mavn, 00:00 , 75m diff --git a/doc/development/packages.md b/doc/development/packages.md index 94882cefc30..869a1755d8f 100644 --- a/doc/development/packages.md +++ b/doc/development/packages.md @@ -133,7 +133,7 @@ During this phase, the idea is to collect as much information as possible about - **Authentication**: What authentication mechanisms are available (OAuth, Basic Authorization, other). Keep in mind that GitLab users often want to use their [Personal Access Tokens](../user/profile/personal_access_tokens.md). - Although not needed for the MVC first iteration, the [CI/CD job tokens](../api/index.md#gitlab-cicd-job-token) + Although not needed for the MVC first iteration, the [CI/CD job tokens](../ci/jobs/ci_job_token.md) have to be supported at some point in the future. - **Requests**: Which requests are needed to have a working MVC. Ideally, produce a list of all the requests needed for the MVC (including required actions). Further diff --git a/doc/development/pipelines.md b/doc/development/pipelines.md index 820299a426b..dd45091a31b 100644 --- a/doc/development/pipelines.md +++ b/doc/development/pipelines.md @@ -64,7 +64,7 @@ graph LR click 1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8100542&udv=0" 1-2["docs-lint markdown (1.5 minutes)"]; click 1-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=10224335&udv=0" - 1-3["docs-lint links (6 minutes)"]; + 1-3["docs-lint links (5 minutes)"]; click 1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356757&udv=0" 1-4["ui-docs-links lint (2.5 minutes)"]; click 1-4 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=10823717&udv=1020379" @@ -104,7 +104,7 @@ graph RL; 1-18["kubesec-sast"]; 1-19["nodejs-scan-sast"]; 1-20["secrets-sast"]; - 1-21["static-analysis (30 minutes)"]; + 1-21["static-analysis (14 minutes)"]; click 1-21 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914471&udv=0" class 1-3 criticalPath; @@ -123,7 +123,7 @@ graph RL; 2_1-1 & 2_1-2 & 2_1-3 & 2_1-4 --> 1-6; end - 2_2-2["rspec frontend_fixture/rspec-ee frontend_fixture (11 minutes)"]; + 2_2-2["rspec frontend_fixture/rspec-ee frontend_fixture (7 minutes)"]; class 2_2-2 criticalPath; click 2_2-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7910143&udv=0" 2_2-4["memory-on-boot (3.5 minutes)"]; @@ -152,16 +152,14 @@ graph RL; click 2_5-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations" end - 3_1-1["jest (16 minutes)"]; + 3_1-1["jest (14.5 minutes)"]; class 3_1-1 criticalPath; click 3_1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914204&udv=0" - 3_1-2["karma (2 minutes)"]; - click 3_1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914200&udv=0" subgraph "Needs `rspec frontend_fixture/rspec-ee frontend_fixture`"; - 3_1-1 & 3_1-2 --> 2_2-2; + 3_1-1 --> 2_2-2; end - 3_2-1["rspec:coverage (5.3 minutes)"]; + 3_2-1["rspec:coverage (4 minutes)"]; subgraph "Depends on `rspec` jobs"; 3_2-1 -.->|"(don't use needs because of limitations)"| 2_5-1; click 3_2-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7248745&udv=0" @@ -208,7 +206,7 @@ graph RL; 1-18["kubesec-sast"]; 1-19["nodejs-scan-sast"]; 1-20["secrets-sast"]; - 1-21["static-analysis (30 minutes)"]; + 1-21["static-analysis (14 minutes)"]; click 1-21 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914471&udv=0" class 1-3 criticalPath; @@ -228,7 +226,7 @@ graph RL; 2_1-1 & 2_1-2 & 2_1-3 & 2_1-4 --> 1-6; end - 2_2-2["rspec frontend_fixture/rspec-ee frontend_fixture (11 minutes)"]; + 2_2-2["rspec frontend_fixture/rspec-ee frontend_fixture (7 minutes)"]; class 2_2-2 criticalPath; click 2_2-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7910143&udv=0" 2_2-4["memory-on-boot (3.5 minutes)"]; @@ -265,16 +263,14 @@ graph RL; click 2_6-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914314&udv=0" end - 3_1-1["jest (16 minutes)"]; + 3_1-1["jest (14.5 minutes)"]; class 3_1-1 criticalPath; click 3_1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914204&udv=0" - 3_1-2["karma (2 minutes)"]; - click 3_1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914200&udv=0" subgraph "Needs `rspec frontend_fixture/rspec-ee frontend_fixture`"; - 3_1-1 & 3_1-2 --> 2_2-2; + 3_1-1 --> 2_2-2; end - 3_2-1["rspec:coverage (5.3 minutes)"]; + 3_2-1["rspec:coverage (4 minutes)"]; subgraph "Depends on `rspec` jobs"; 3_2-1 -.->|"(don't use needs because of limitations)"| 2_5-1; click 3_2-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7248745&udv=0" @@ -287,7 +283,7 @@ graph RL; click 4_1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7910777&udv=0" end - 3_3-1["review-deploy (10.5 minutes)"]; + 3_3-1["review-deploy (9 minutes)"]; subgraph "Played by `review-build-cng`"; 3_3-1 --> 2_6-1; class 3_3-1 criticalPath; @@ -336,7 +332,7 @@ graph RL; 1-18["kubesec-sast"]; 1-19["nodejs-scan-sast"]; 1-20["secrets-sast"]; - 1-21["static-analysis (30 minutes)"]; + 1-21["static-analysis (14 minutes)"]; click 1-21 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914471&udv=0" class 1-5 criticalPath; @@ -354,7 +350,7 @@ graph RL; class 2_3-1 criticalPath; end - 2_4-1["package-and-qa (140 minutes)"]; + 2_4-1["package-and-qa (113 minutes)"]; subgraph "Needs `build-qa-image` & `build-assets-image`"; 2_4-1 --> 1-2 & 2_3-1; class 2_4-1 criticalPath; @@ -434,10 +430,27 @@ In the `detect-tests` job, we use this mapping to identify the minimal tests nee After a merge request has been approved, the pipeline would contain the full RSpec tests. This will ensure that all tests have been run before a merge request is merged. +### Jest minimal jobs + +Before a merge request is approved, the pipeline will run a minimal set of Jest tests that are related to the merge request changes. +This is to reduce the pipeline cost and shorten the job duration. + +To identify the minimal set of tests needed, we pass a list of all the changed files into `jest` using the [`--findRelatedTests`](https://jestjs.io/docs/cli#--findrelatedtests-spaceseparatedlistofsourcefiles) option. +In this mode, `jest` would resolve all the dependencies of related to the changed files, which include test files that have these files in the dependency chain. + +After a merge request has been approved, the pipeline would contain the full Jest tests. This will ensure that all tests +have been run before a merge request is merged. + +In addition, there are a few circumstances where we would always run the full Jest tests: + +- when `package.json`, `yarn.lock`, `jest` config changes +- when vendored JavaScript is changed +- when `.graphql` files are changed + ### PostgreSQL versions testing Our test suite runs against PG12 as GitLab.com runs on PG12 and -[Omnibus defaults to PG12 for new installs and upgrades](https://docs.gitlab.com/omnibus/package-information/postgresql_versions.html), +[Omnibus defaults to PG12 for new installs and upgrades](../administration/package_information/postgresql_versions.md), Our test suite is currently running against PG11, since GitLab.com still runs on PG11. We do run our test suite against PG11 on nightly scheduled pipelines as well as upon specific @@ -454,7 +467,7 @@ database library changes in MRs and `main` pipelines (with the `rspec db-library #### Long-term plan -We follow the [PostgreSQL versions shipped with Omnibus GitLab](https://docs.gitlab.com/omnibus/package-information/postgresql_versions.html): +We follow the [PostgreSQL versions shipped with Omnibus GitLab](../administration/package_information/postgresql_versions.md): | PostgreSQL version | 13.11 (April 2021) | 13.12 (May 2021) | 14.0 (June 2021?) | | -------------------| ---------------------- | ---------------------- | ---------------------- | @@ -627,7 +640,6 @@ that is deployed in stage `review`. the `qa` stage's jobs (for example, Review App performance report). - `pages`: This stage includes a job that deploys the various reports as GitLab Pages (for example, [`coverage-ruby`](https://gitlab-org.gitlab.io/gitlab/coverage-ruby/), - [`coverage-javascript`](https://gitlab-org.gitlab.io/gitlab/coverage-javascript/), and `webpack-report` (found at `https://gitlab-org.gitlab.io/gitlab/webpack-report/`, but there is [an issue with the deployment](https://gitlab.com/gitlab-org/gitlab/-/issues/233458)). - `notify`: This stage includes jobs that notify various failures to Slack. diff --git a/doc/development/prometheus_metrics.md b/doc/development/prometheus_metrics.md index 66e980978bf..da6ba14cdd8 100644 --- a/doc/development/prometheus_metrics.md +++ b/doc/development/prometheus_metrics.md @@ -36,9 +36,7 @@ After you add or change an existing common metric, you must [re-run the import s Or, you can create a database migration: ```ruby -class ImportCommonMetrics < ActiveRecord::Migration[4.2] - include Gitlab::Database::MigrationHelpers - +class ImportCommonMetrics < Gitlab::Database::Migration[1.0] def up ::Gitlab::DatabaseImporters::CommonMetrics::Importer.new.execute end diff --git a/doc/development/query_recorder.md b/doc/development/query_recorder.md index 8759bd09538..424c089f88e 100644 --- a/doc/development/query_recorder.md +++ b/doc/development/query_recorder.md @@ -18,9 +18,9 @@ This style of test works by counting the number of SQL queries executed by Activ ```ruby it "avoids N+1 database queries" do - control_count = ActiveRecord::QueryRecorder.new { visit_some_page }.count + control = ActiveRecord::QueryRecorder.new { visit_some_page } create_list(:issue, 5) - expect { visit_some_page }.not_to exceed_query_limit(control_count) + expect { visit_some_page }.not_to exceed_query_limit(control) end ``` @@ -37,9 +37,9 @@ You should pass the `skip_cached` variable to `QueryRecorder` and use the `excee ```ruby it "avoids N+1 database queries", :use_sql_query_cache do - control_count = ActiveRecord::QueryRecorder.new(skip_cached: false) { visit_some_page }.count + control = ActiveRecord::QueryRecorder.new(skip_cached: false) { visit_some_page } create_list(:issue, 5) - expect { visit_some_page }.not_to exceed_all_query_limit(control_count) + expect { visit_some_page }.not_to exceed_all_query_limit(control) end ``` diff --git a/doc/development/rake_tasks.md b/doc/development/rake_tasks.md index 6b2b941a0c1..5c8e2d5fc55 100644 --- a/doc/development/rake_tasks.md +++ b/doc/development/rake_tasks.md @@ -128,7 +128,6 @@ In order to run the test you can use the following commands: - `bin/rake spec:unit` to run only the unit tests - `bin/rake spec:integration` to run only the integration tests - `bin/rake spec:system` to run only the system tests -- `bin/rake karma` to run the Karma test suite `bin/rake spec` takes significant time to pass. Instead of running the full test suite locally, you can save a lot of time by running @@ -189,6 +188,17 @@ Alternatively you can use the following on each spec run, bundle exec spring rspec some_spec.rb ``` +## Generate initial RuboCop TODO list + +One way to generate the initial list is to run the Rake task `rubocop:todo:generate`: + +```shell +bundle exec rake rubocop:todo:generate +``` + +See [Resolving RuboCop exceptions](contributing/style_guides.md#resolving-rubocop-exceptions) +on how to proceed from here. + ## Compile Frontend Assets You shouldn't ever need to compile frontend assets manually in development, but diff --git a/doc/development/secure_coding_guidelines.md b/doc/development/secure_coding_guidelines.md index fc60c1d7d7f..0f13b8ecae9 100644 --- a/doc/development/secure_coding_guidelines.md +++ b/doc/development/secure_coding_guidelines.md @@ -295,6 +295,8 @@ The injected client-side code is executed on the victim's browser in the context Much of the impact is contingent upon the function of the application and the capabilities of the victim's session. For further impact possibilities, please check out [the beef project](https://beefproject.com/). +For a demonstration of the impact on GitLab with a realistic attack scenario, see [this video on the GitLab Unfiltered channel](https://www.youtube.com/watch?v=t4PzHNycoKo) (internal, it requires being logged in with the GitLab Unfiltered account). + ### When to consider? When user submitted data is included in responses to end users, which is just about anywhere. diff --git a/doc/development/service_ping/implement.md b/doc/development/service_ping/implement.md index 629128af0c6..d86b06a6965 100644 --- a/doc/development/service_ping/implement.md +++ b/doc/development/service_ping/implement.md @@ -4,20 +4,680 @@ group: Product Intelligence info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Develop and test Service Ping +# Implement Service Ping -To add a new metric and test Service Ping: +Service Ping consists of two kinds of data: +- **Counters**: Track how often a certain event happened over time, such as how many CI/CD pipelines have run. + They are monotonic and always trend up. +- **Observations**: Facts collected from one or more GitLab instances and can carry arbitrary data. + There are no general guidelines for how to collect those, due to the individual nature of that data. + +To implement a new metric in Service Ping, follow these steps: + +1. [Implement the required counter](#types-of-counters) 1. [Name and place the metric](#name-and-place-the-metric) 1. [Test counters manually using your Rails console](#test-counters-manually-using-your-rails-console) 1. [Generate the SQL query](#generate-the-sql-query) 1. [Optimize queries with `#database-lab`](#optimize-queries-with-database-lab) -1. [Add the metric definition](#add-the-metric-definition) +1. [Add the metric definition to the Metrics Dictionary](#add-the-metric-definition) 1. [Add the metric to the Versions Application](#add-the-metric-to-the-versions-application) 1. [Create a merge request](#create-a-merge-request) 1. [Verify your metric](#verify-your-metric) 1. [Set up and test Service Ping locally](#set-up-and-test-service-ping-locally) +## Instrumentation classes + +We recommend you use [instrumentation classes](metrics_instrumentation.md) in `usage_data.rb` where possible. + +For example, we have the following instrumentation class: +`lib/gitlab/usage/metrics/instrumentations/count_boards_metric.rb`. + +You should add it to `usage_data.rb` as follows: + +```ruby +boards: add_metric('CountBoardsMetric', time_frame: 'all'), +``` + +## Types of counters + +There are several types of counters in `usage_data.rb`: + +- **[Batch counters](#batch-counters)**: Used for counts and sums. +- **[Redis counters](#redis-counters):** Used for in-memory counts. +- **[Alternative counters](#alternative-counters):** Used for settings and configurations. + +NOTE: +Only use the provided counter methods. Each counter method contains a built-in fail-safe mechanism that isolates each counter to avoid breaking the entire Service Ping process. + +### Batch counters + +For large tables, PostgreSQL can take a long time to count rows due to MVCC [(Multi-version Concurrency Control)](https://en.wikipedia.org/wiki/Multiversion_concurrency_control). Batch counting is a counting method where a single large query is broken into multiple smaller queries. For example, instead of a single query querying 1,000,000 records, with batch counting, you can execute 100 queries of 10,000 records each. Batch counting is useful for avoiding database timeouts as each batch query is significantly shorter than one single long running query. + +For GitLab.com, there are extremely large tables with 15 second query timeouts, so we use batch counting to avoid encountering timeouts. Here are the sizes of some GitLab.com tables: + +| Table | Row counts in millions | +|------------------------------|------------------------| +| `merge_request_diff_commits` | 2280 | +| `ci_build_trace_sections` | 1764 | +| `merge_request_diff_files` | 1082 | +| `events` | 514 | + +Batch counting requires indexes on columns to calculate max, min, and range queries. In some cases, +you must add a specialized index on the columns involved in a counter. + +#### Ordinary batch counters + +Simple count of a given `ActiveRecord_Relation`, does a non-distinct batch count, smartly reduces `batch_size`, and handles errors. +Handles the `ActiveRecord::StatementInvalid` error. + +Method: + +```ruby +count(relation, column = nil, batch: true, start: nil, finish: nil) +``` + +Arguments: + +- `relation` the ActiveRecord_Relation to perform the count +- `column` the column to perform the count on, by default is the primary key +- `batch`: default `true` to use batch counting +- `start`: custom start of the batch counting to avoid complex min calculations +- `end`: custom end of the batch counting to avoid complex min calculations + +Examples: + +```ruby +count(User.active) +count(::Clusters::Cluster.aws_installed.enabled, :cluster_id) +count(::Clusters::Cluster.aws_installed.enabled, :cluster_id, start: ::Clusters::Cluster.minimum(:id), finish: ::Clusters::Cluster.maximum(:id)) +``` + +#### Distinct batch counters + +Distinct count of a given `ActiveRecord_Relation` on given column, a distinct batch count, smartly reduces `batch_size`, and handles errors. +Handles the `ActiveRecord::StatementInvalid` error. + +Method: + +```ruby +distinct_count(relation, column = nil, batch: true, batch_size: nil, start: nil, finish: nil) +``` + +Arguments: + +- `relation`: the ActiveRecord_Relation to perform the count +- `column`: the column to perform the distinct count, by default is the primary key +- `batch`: default `true` to use batch counting +- `batch_size`: if none set it uses default value 10000 from `Gitlab::Database::BatchCounter` +- `start`: custom start of the batch counting to avoid complex min calculations +- `end`: custom end of the batch counting to avoid complex min calculations + +WARNING: +Counting over non-unique columns can lead to performance issues. For more information, see the [iterating tables in batches](../iterating_tables_in_batches.md) guide. + +Examples: + +```ruby +distinct_count(::Project, :creator_id) +distinct_count(::Note.with_suggestions.where(time_period), :author_id, start: ::User.minimum(:id), finish: ::User.maximum(:id)) +distinct_count(::Clusters::Applications::CertManager.where(time_period).available.joins(:cluster), 'clusters.user_id') +``` + +#### Sum batch operation + +Sum the values of a given ActiveRecord_Relation on given column and handles errors. +Handles the `ActiveRecord::StatementInvalid` error + +Method: + +```ruby +sum(relation, column, batch_size: nil, start: nil, finish: nil) +``` + +Arguments: + +- `relation`: the ActiveRecord_Relation to perform the operation +- `column`: the column to sum on +- `batch_size`: if none set it uses default value 1000 from `Gitlab::Database::BatchCounter` +- `start`: custom start of the batch counting to avoid complex min calculations +- `end`: custom end of the batch counting to avoid complex min calculations + +Examples: + +```ruby +sum(JiraImportState.finished, :imported_issues_count) +``` + +#### Grouping and batch operations + +The `count`, `distinct_count`, and `sum` batch counters can accept an `ActiveRecord::Relation` +object, which groups by a specified column. With a grouped relation, the methods do batch counting, +handle errors, and returns a hash table of key-value pairs. + +Examples: + +```ruby +count(Namespace.group(:type)) +# returns => {nil=>179, "Group"=>54} + +distinct_count(Project.group(:visibility_level), :creator_id) +# returns => {0=>1, 10=>1, 20=>11} + +sum(Issue.group(:state_id), :weight)) +# returns => {1=>3542, 2=>6820} +``` + +#### Add operation + +Sum the values given as parameters. Handles the `StandardError`. +Returns `-1` if any of the arguments are `-1`. + +Method: + +```ruby +add(*args) +``` + +Examples: + +```ruby +project_imports = distinct_count(::Project.where.not(import_type: nil), :creator_id) +bulk_imports = distinct_count(::BulkImport, :user_id) + + add(project_imports, bulk_imports) +``` + +#### Estimated batch counters + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/48233) in GitLab 13.7. + +Estimated batch counter functionality handles `ActiveRecord::StatementInvalid` errors +when used through the provided `estimate_batch_distinct_count` method. +Errors return a value of `-1`. + +WARNING: +This functionality estimates a distinct count of a specific ActiveRecord_Relation in a given column, +which uses the [HyperLogLog](http://algo.inria.fr/flajolet/Publications/FlFuGaMe07.pdf) algorithm. +As the HyperLogLog algorithm is probabilistic, the **results always include error**. +The highest encountered error rate is 4.9%. + +When correctly used, the `estimate_batch_distinct_count` method enables efficient counting over +columns that contain non-unique values, which can not be assured by other counters. + +##### estimate_batch_distinct_count method + +Method: + +```ruby +estimate_batch_distinct_count(relation, column = nil, batch_size: nil, start: nil, finish: nil) +``` + +The [method](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/utils/usage_data.rb#L63) +includes the following arguments: + +- `relation`: The ActiveRecord_Relation to perform the count. +- `column`: The column to perform the distinct count. The default is the primary key. +- `batch_size`: From `Gitlab::Database::PostgresHll::BatchDistinctCounter::DEFAULT_BATCH_SIZE`. Default value: 10,000. +- `start`: The custom start of the batch count, to avoid complex minimum calculations. +- `finish`: The custom end of the batch count to avoid complex maximum calculations. + +The method includes the following prerequisites: + +- The supplied `relation` must include the primary key defined as the numeric column. + For example: `id bigint NOT NULL`. +- The `estimate_batch_distinct_count` can handle a joined relation. To use its ability to + count non-unique columns, the joined relation **must not** have a one-to-many relationship, + such as `has_many :boards`. +- Both `start` and `finish` arguments should always represent primary key relationship values, + even if the estimated count refers to another column, for example: + + ```ruby + estimate_batch_distinct_count(::Note, :author_id, start: ::Note.minimum(:id), finish: ::Note.maximum(:id)) + ``` + +Examples: + +1. Simple execution of estimated batch counter, with only relation provided, + returned value represents estimated number of unique values in `id` column + (which is the primary key) of `Project` relation: + + ```ruby + estimate_batch_distinct_count(::Project) + ``` + +1. Execution of estimated batch counter, where provided relation has applied + additional filter (`.where(time_period)`), number of unique values estimated + in custom column (`:author_id`), and parameters: `start` and `finish` together + apply boundaries that defines range of provided relation to analyze: + + ```ruby + estimate_batch_distinct_count(::Note.with_suggestions.where(time_period), :author_id, start: ::Note.minimum(:id), finish: ::Note.maximum(:id)) + ``` + +1. Execution of estimated batch counter with joined relation (`joins(:cluster)`), + for a custom column (`'clusters.user_id'`): + + ```ruby + estimate_batch_distinct_count(::Clusters::Applications::CertManager.where(time_period).available.joins(:cluster), 'clusters.user_id') + ``` + +When instrumenting metric with usage of estimated batch counter please add +`_estimated` suffix to its name, for example: + +```ruby + "counts": { + "ci_builds_estimated": estimate_batch_distinct_count(Ci::Build), + ... +``` + +### Redis counters + +Handles `::Redis::CommandError` and `Gitlab::UsageDataCounters::BaseCounter::UnknownEvent`. +Returns -1 when a block is sent or hash with all values and -1 when a `counter(Gitlab::UsageDataCounters)` is sent. +The different behavior is due to 2 different implementations of the Redis counter. + +Method: + +```ruby +redis_usage_data(counter, &block) +``` + +Arguments: + +- `counter`: a counter from `Gitlab::UsageDataCounters`, that has `fallback_totals` method implemented +- or a `block`: which is evaluated + +#### Ordinary Redis counters + +Examples of implementation: + +- Using Redis methods [`INCR`](https://redis.io/commands/incr), [`GET`](https://redis.io/commands/get), and [`Gitlab::UsageDataCounters::WikiPageCounter`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/wiki_page_counter.rb) +- Using Redis methods [`HINCRBY`](https://redis.io/commands/hincrby), [`HGETALL`](https://redis.io/commands/hgetall), and [`Gitlab::UsageCounters::PodLogs`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_counters/pod_logs.rb) + +##### UsageData API tracking + +<!-- There's nearly identical content in `##### Adding new events`. If you fix errors here, you may need to fix the same errors in the other location. --> + +1. Track event using `UsageData` API + + Increment event count using ordinary Redis counter, for given event name. + + Tracking events using the `UsageData` API requires the `usage_data_api` feature flag to be enabled, which is enabled by default. + + API requests are protected by checking for a valid CSRF token. + + To be able to increment the values, the related feature `usage_data_<event_name>` should be enabled. + + ```plaintext + POST /usage_data/increment_counter + ``` + + | Attribute | Type | Required | Description | + | :-------- | :--- | :------- | :---------- | + | `event` | string | yes | The event name it should be tracked | + + Response: + + - `200` if event was tracked + - `400 Bad request` if event parameter is missing + - `401 Unauthorized` if user is not authenticated + - `403 Forbidden` for invalid CSRF token provided + +1. Track events using JavaScript/Vue API helper which calls the API above + + Note that `usage_data_api` and `usage_data_#{event_name}` should be enabled to be able to track events + + ```javascript + import api from '~/api'; + + api.trackRedisCounterEvent('my_already_defined_event_name'), + ``` + +#### Redis HLL counters + +WARNING: +HyperLogLog (HLL) is a probabilistic algorithm and its **results always includes some small error**. According to [Redis documentation](https://redis.io/commands/pfcount), data from +used HLL implementation is "approximated with a standard error of 0.81%". + +With `Gitlab::UsageDataCounters::HLLRedisCounter` we have available data structures used to count unique values. + +Implemented using Redis methods [PFADD](https://redis.io/commands/pfadd) and [PFCOUNT](https://redis.io/commands/pfcount). + +##### Add new events + +1. Define events in [`known_events`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events/). + + Example event: + + ```yaml + - name: users_creating_epics + category: epics_usage + redis_slot: users + aggregation: weekly + feature_flag: track_epics_activity + ``` + + Keys: + + - `name`: unique event name. + + Name format for Redis HLL events `<name>_<redis_slot>`. + + [See Metric name](metrics_dictionary.md#metric-name) for a complete guide on metric naming suggestion. + + Consider including in the event's name the Redis slot to be able to count totals for a specific category. + + Example names: `users_creating_epics`, `users_triggering_security_scans`. + + - `category`: event category. Used for getting total counts for events in a category, for easier + access to a group of events. + - `redis_slot`: optional Redis slot. Default value: event name. Only event data that is stored in the same slot + can be aggregated. Ensure keys are in the same slot. For example: + `users_creating_epics` with `redis_slot: 'users'` builds Redis key + `{users}_creating_epics-2020-34`. If `redis_slot` is not defined the Redis key will + be `{users_creating_epics}-2020-34`. + Recommended slots to use are: `users`, `projects`. This is the value we count. + - `expiry`: expiry time in days. Default: 29 days for daily aggregation and 6 weeks for weekly + aggregation. + - `aggregation`: may be set to a `:daily` or `:weekly` key. Defines how counting data is stored in Redis. + Aggregation on a `daily` basis does not pull more fine grained data. + - `feature_flag`: optional `default_enabled: :yaml`. If no feature flag is set then the tracking is enabled. One feature flag can be used for multiple events. For details, see our [GitLab internal Feature flags](../feature_flags/index.md) documentation. The feature flags are owned by the group adding the event tracking. + +1. Use one of the following methods to track the event: + + - In the controller using the `RedisTracking` module and the following format: + + ```ruby + track_redis_hll_event(*controller_actions, name:, if: nil, &block) + ``` + + Arguments: + + - `controller_actions`: the controller actions to track. + - `name`: the event name. + - `if`: optional custom conditions. Uses the same format as Rails callbacks. + - `&block`: optional block that computes and returns the `custom_id` that we want to track. This overrides the `visitor_id`. + + Example: + + ```ruby + # controller + class ProjectsController < Projects::ApplicationController + include RedisTracking + + skip_before_action :authenticate_user!, only: :show + track_redis_hll_event :index, :show, name: 'users_visiting_projects' + + def index + render html: 'index' + end + + def new + render html: 'new' + end + + def show + render html: 'show' + end + end + ``` + + - In the API using the `increment_unique_values(event_name, values)` helper method. + + Arguments: + + - `event_name`: the event name. + - `values`: the values counted. Can be one value or an array of values. + + Example: + + ```ruby + get ':id/registry/repositories' do + repositories = ContainerRepositoriesFinder.new( + user: current_user, subject: user_group + ).execute + + increment_unique_values('users_listing_repositories', current_user.id) + + present paginate(repositories), with: Entities::ContainerRegistry::Repository, tags: params[:tags], tags_count: params[:tags_count] + end + ``` + + - Using `track_usage_event(event_name, values)` in services and GraphQL. + + Increment unique values count using Redis HLL, for a given event name. + + Examples: + + - [Track usage event for an incident in a service](https://gitlab.com/gitlab-org/gitlab/-/blob/v13.8.3-ee/app/services/issues/update_service.rb#L66) + - [Track usage event for an incident in GraphQL](https://gitlab.com/gitlab-org/gitlab/-/blob/v13.8.3-ee/app/graphql/mutations/alert_management/update_alert_status.rb#L16) + + ```ruby + track_usage_event(:incident_management_incident_created, current_user.id) + ``` + + - Using the `UsageData` API. + <!-- There's nearly identical content in `##### UsageData API Tracking`. If you find / fix errors here, you may need to fix errors in that section too. --> + + Increment unique users count using Redis HLL, for a given event name. + + To track events using the `UsageData` API, ensure the `usage_data_api` feature flag + is set to `default_enabled: true`. Enabled by default in GitLab 13.7 and later. + + API requests are protected by checking for a valid CSRF token. + + ```plaintext + POST /usage_data/increment_unique_users + ``` + + | Attribute | Type | Required | Description | + | :-------- | :--- | :------- | :---------- | + | `event` | string | yes | The event name to track | + + Response: + + - `200` if the event was tracked, or if tracking failed for any reason. + - `400 Bad request` if an event parameter is missing. + - `401 Unauthorized` if the user is not authenticated. + - `403 Forbidden` if an invalid CSRF token is provided. + + - Using the JavaScript/Vue API helper, which calls the `UsageData` API. + + To track events using the `UsageData` API, ensure the `usage_data_api` feature flag + is set to `default_enabled: true`. Enabled by default in GitLab 13.7 and later. + + Example for an existing event already defined in [known events](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events/): + + ```javascript + import api from '~/api'; + + api.trackRedisHllUserEvent('my_already_defined_event_name'), + ``` + +1. Get event data using `Gitlab::UsageDataCounters::HLLRedisCounter.unique_events(event_names:, start_date:, end_date:, context: '')`. + + Arguments: + + - `event_names`: the list of event names. + - `start_date`: start date of the period for which we want to get event data. + - `end_date`: end date of the period for which we want to get event data. + - `context`: context of the event. Allowed values are `default`, `free`, `bronze`, `silver`, `gold`, `starter`, `premium`, `ultimate`. + +1. Testing tracking and getting unique events + +Trigger events in rails console by using `track_event` method + + ```ruby + Gitlab::UsageDataCounters::HLLRedisCounter.track_event('users_viewing_compliance_audit_events', values: 1) + Gitlab::UsageDataCounters::HLLRedisCounter.track_event('users_viewing_compliance_audit_events', values: [2, 3]) + ``` + +Next, get the unique events for the current week. + + ```ruby + # Get unique events for metric for current_week + Gitlab::UsageDataCounters::HLLRedisCounter.unique_events(event_names: 'users_viewing_compliance_audit_events', + start_date: Date.current.beginning_of_week, end_date: Date.current.next_week) + ``` + +##### Recommendations + +We have the following recommendations for [adding new events](#add-new-events): + +- Event aggregation: weekly. +- Key expiry time: + - Daily: 29 days. + - Weekly: 42 days. +- When adding new metrics, use a [feature flag](../../operations/feature_flags.md) to control the impact. +- For feature flags triggered by another service, set `default_enabled: false`, + - Events can be triggered using the `UsageData` API, which helps when there are > 10 events per change + +##### Enable or disable Redis HLL tracking + +Events are tracked behind optional [feature flags](../feature_flags/index.md) due to concerns for Redis performance and scalability. + +For a full list of events and corresponding feature flags see, [known_events](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events/) files. + +To enable or disable tracking for specific event in <https://gitlab.com> or <https://about.staging.gitlab.com>, run commands such as the following to +[enable or disable the corresponding feature](../feature_flags/index.md). + +```shell +/chatops run feature set <feature_name> true +/chatops run feature set <feature_name> false +``` + +We can also disable tracking completely by using the global flag: + +```shell +/chatops run feature set redis_hll_tracking true +/chatops run feature set redis_hll_tracking false +``` + +##### Known events are added automatically in Service Data payload + +All events added in [`known_events/common.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events/common.yml) are automatically added to Service Data generation under the `redis_hll_counters` key. This column is stored in [version-app as a JSON](https://gitlab.com/gitlab-services/version-gitlab-com/-/blob/master/db/schema.rb#L209). +For each event we add metrics for the weekly and monthly time frames, and totals for each where applicable: + +- `#{event_name}_weekly`: Data for 7 days for daily [aggregation](#add-new-events) events and data for the last complete week for weekly [aggregation](#add-new-events) events. +- `#{event_name}_monthly`: Data for 28 days for daily [aggregation](#add-new-events) events and data for the last 4 complete weeks for weekly [aggregation](#add-new-events) events. + +Redis HLL implementation calculates automatic total metrics, if there are more than one metric for the same category, aggregation, and Redis slot. + +- `#{category}_total_unique_counts_weekly`: Total unique counts for events in the same category for the last 7 days or the last complete week, if events are in the same Redis slot and we have more than one metric. +- `#{category}_total_unique_counts_monthly`: Total unique counts for events in same category for the last 28 days or the last 4 complete weeks, if events are in the same Redis slot and we have more than one metric. + +Example of `redis_hll_counters` data: + +```ruby +{:redis_hll_counters=> + {"compliance"=> + {"users_viewing_compliance_dashboard_weekly"=>0, + "users_viewing_compliance_dashboard_monthly"=>0, + "users_viewing_compliance_audit_events_weekly"=>0, + "users_viewing_audit_events_monthly"=>0, + "compliance_total_unique_counts_weekly"=>0, + "compliance_total_unique_counts_monthly"=>0}, + "analytics"=> + {"users_viewing_analytics_group_devops_adoption_weekly"=>0, + "users_viewing_analytics_group_devops_adoption_monthly"=>0, + "analytics_total_unique_counts_weekly"=>0, + "analytics_total_unique_counts_monthly"=>0}, + "ide_edit"=> + {"users_editing_by_web_ide_weekly"=>0, + "users_editing_by_web_ide_monthly"=>0, + "users_editing_by_sfe_weekly"=>0, + "users_editing_by_sfe_monthly"=>0, + "ide_edit_total_unique_counts_weekly"=>0, + "ide_edit_total_unique_counts_monthly"=>0} + } +``` + +Example: + +```ruby +# Redis Counters +redis_usage_data(Gitlab::UsageDataCounters::WikiPageCounter) +redis_usage_data { ::Gitlab::UsageCounters::PodLogs.usage_totals[:total] } + +# Define events in common.yml https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events/common.yml + +# Tracking events +Gitlab::UsageDataCounters::HLLRedisCounter.track_event('users_expanding_vulnerabilities', values: visitor_id) + +# Get unique events for metric +redis_usage_data { Gitlab::UsageDataCounters::HLLRedisCounter.unique_events(event_names: 'users_expanding_vulnerabilities', start_date: 28.days.ago, end_date: Date.current) } +``` + +### Alternative counters + +Handles `StandardError` and fallbacks into -1 this way not all measures fail if we encounter one exception. +Mainly used for settings and configurations. + +Method: + +```ruby +alt_usage_data(value = nil, fallback: -1, &block) +``` + +Arguments: + +- `value`: a simple static value in which case the value is simply returned. +- or a `block`: which is evaluated +- `fallback: -1`: the common value used for any metrics that are failing. + +Example: + +```ruby +alt_usage_data { Gitlab::VERSION } +alt_usage_data { Gitlab::CurrentSettings.uuid } +alt_usage_data(999) +``` + +### Add counters to build new metrics + +When adding the results of two counters, use the `add` Service Data method that +handles fallback values and exceptions. It also generates a valid [SQL export](index.md#export-service-ping-sql-queries-and-definitions). + +Example: + +```ruby +add(User.active, User.bot) +``` + +### Prometheus queries + +In those cases where operational metrics should be part of Service Ping, a database or Redis query is unlikely +to provide useful data. Instead, Prometheus might be more appropriate, because most GitLab architectural +components publish metrics to it that can be queried back, aggregated, and included as Service Data. + +NOTE: +Prometheus as a data source for Service Ping is only available for single-node Omnibus installations +that are running the [bundled Prometheus](../../administration/monitoring/prometheus/index.md) instance. + +To query Prometheus for metrics, a helper method is available to `yield` a fully configured +`PrometheusClient`, given it is available as per the note above: + +```ruby +with_prometheus_client do |client| + response = client.query('<your query>') + ... +end +``` + +Refer to [the `PrometheusClient` definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/prometheus_client.rb) +for how to use its API to query for data. + +### Fallback values for Service Ping + +We return fallback values in these cases: + +| Case | Value | +|-----------------------------|-------| +| Deprecated Metric | -1000 | +| Timeouts, general failures | -1 | +| Standard errors in counters | -2 | + ## Name and place the metric Add the metric in one of the top-level keys: @@ -159,7 +819,7 @@ To set up Service Ping locally, you must: ## Test Prometheus-based Service Ping -If the data submitted includes metrics [queried from Prometheus](index.md#prometheus-queries) +If the data submitted includes metrics [queried from Prometheus](#prometheus-queries) you want to inspect and verify, you must: - Ensure that a Prometheus server is running locally. @@ -208,3 +868,197 @@ However, it has the following limitations: with any of the other running services. That is not how node metrics are reported in a production setup, where `node_exporter` always runs as a process alongside other GitLab components on any given node. For Service Ping, none of the node data would therefore appear to be associated to any of the services running, because they all appear to be running on different hosts. To alleviate this problem, the `node_exporter` in GCK was arbitrarily "assigned" to the `web` service, meaning only for this service `node_*` metrics appears in Service Ping. + +## Aggregated metrics + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/45979) in GitLab 13.6. + +WARNING: +This feature is intended solely for internal GitLab use. + +To add data for aggregated metrics to the Service Ping payload, add a corresponding definition to: + +- [`config/metrics/aggregates/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/aggregates/) for metrics available in the Community Edition. +- [`ee/config/metrics/aggregates/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/aggregates/) for metrics available in the Enterprise Edition. + +Each aggregate definition includes following parts: + +- `name`: Unique name under which the aggregate metric is added to the Service Ping payload. +- `operator`: Operator that defines how the aggregated metric data is counted. Available operators are: + - `OR`: Removes duplicates and counts all entries that triggered any of listed events. + - `AND`: Removes duplicates and counts all elements that were observed triggering all of following events. +- `time_frame`: One or more valid time frames. Use these to limit the data included in aggregated metric to events within a specific date-range. Valid time frames are: + - `7d`: Last seven days of data. + - `28d`: Last twenty eight days of data. + - `all`: All historical data, only available for `database` sourced aggregated metrics. +- `source`: Data source used to collect all events data included in aggregated metric. Valid data sources are: + - [`database`](#database-sourced-aggregated-metrics) + - [`redis`](#redis-sourced-aggregated-metrics) +- `events`: list of events names to aggregate into metric. All events in this list must + relay on the same data source. Additional data source requirements are described in the + [Database sourced aggregated metrics](#database-sourced-aggregated-metrics) and + [Redis sourced aggregated metrics](#redis-sourced-aggregated-metrics) sections. +- `feature_flag`: Name of [development feature flag](../feature_flags/index.md#development-type) + that is checked before metrics aggregation is performed. Corresponding feature flag + should have `default_enabled` attribute set to `false`. The `feature_flag` attribute + is optional and can be omitted. When `feature_flag` is missing, no feature flag is checked. + +Example aggregated metric entries: + +```yaml +- name: example_metrics_union + operator: OR + events: + - 'users_expanding_secure_security_report' + - 'users_expanding_testing_code_quality_report' + - 'users_expanding_testing_accessibility_report' + source: redis + time_frame: + - 7d + - 28d +- name: example_metrics_intersection + operator: AND + source: database + time_frame: + - 28d + - all + events: + - 'dependency_scanning_pipeline_all_time' + - 'container_scanning_pipeline_all_time' + feature_flag: example_aggregated_metric +``` + +Aggregated metrics collected in `7d` and `28d` time frames are added into Service Ping payload under the `aggregated_metrics` sub-key in the `counts_weekly` and `counts_monthly` top level keys. + +```ruby +{ + :counts_monthly => { + :deployments => 1003, + :successful_deployments => 78, + :failed_deployments => 275, + :packages => 155, + :personal_snippets => 2106, + :project_snippets => 407, + :promoted_issues => 719, + :aggregated_metrics => { + :example_metrics_union => 7, + :example_metrics_intersection => 2 + }, + :snippets => 2513 + } +} +``` + +Aggregated metrics for `all` time frame are present in the `count` top level key, with the `aggregate_` prefix added to their name. + +For example: + +`example_metrics_intersection` + +Becomes: + +`counts.aggregate_example_metrics_intersection` + +```ruby +{ + :counts => { + :deployments => 11003, + :successful_deployments => 178, + :failed_deployments => 1275, + :aggregate_example_metrics_intersection => 12 + } +} +``` + +### Redis sourced aggregated metrics + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/45979) in GitLab 13.6. + +To declare the aggregate of events collected with [Redis HLL Counters](#redis-hll-counters), +you must fulfill the following requirements: + +1. All events listed at `events` attribute must come from + [`known_events/*.yml`](#known-events-are-added-automatically-in-service-data-payload) files. +1. All events listed at `events` attribute must have the same `redis_slot` attribute. +1. All events listed at `events` attribute must have the same `aggregation` attribute. +1. `time_frame` does not include `all` value, which is unavailable for Redis sourced aggregated metrics. + +### Database sourced aggregated metrics + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/52784) in GitLab 13.9. +> - It's [deployed behind a feature flag](../../user/feature_flags.md), disabled by default. +> - It's enabled on GitLab.com. + +To declare an aggregate of metrics based on events collected from database, follow +these steps: + +1. [Persist the metrics for aggregation](#persist-metrics-for-aggregation). +1. [Add new aggregated metric definition](#add-new-aggregated-metric-definition). + +#### Persist metrics for aggregation + +Only metrics calculated with [Estimated Batch Counters](#estimated-batch-counters) +can be persisted for database sourced aggregated metrics. To persist a metric, +inject a Ruby block into the +[estimate_batch_distinct_count](#estimate_batch_distinct_count-method) method. +This block should invoke the +`Gitlab::Usage::Metrics::Aggregates::Sources::PostgresHll.save_aggregated_metrics` +[method](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage/metrics/aggregates/sources/postgres_hll.rb#L21), +which stores `estimate_batch_distinct_count` results for future use in aggregated metrics. + +The `Gitlab::Usage::Metrics::Aggregates::Sources::PostgresHll.save_aggregated_metrics` +method accepts the following arguments: + +- `metric_name`: The name of metric to use for aggregations. Should be the same + as the key under which the metric is added into Service Ping. +- `recorded_at_timestamp`: The timestamp representing the moment when a given + Service Ping payload was collected. You should use the convenience method `recorded_at` + to fill `recorded_at_timestamp` argument, like this: `recorded_at_timestamp: recorded_at` +- `time_period`: The time period used to build the `relation` argument passed into + `estimate_batch_distinct_count`. To collect the metric with all available historical + data, set a `nil` value as time period: `time_period: nil`. +- `data`: HyperLogLog buckets structure representing unique entries in `relation`. + The `estimate_batch_distinct_count` method always passes the correct argument + into the block, so `data` argument must always have a value equal to block argument, + like this: `data: result` + +Example metrics persistence: + +```ruby +class UsageData + def count_secure_pipelines(time_period) + ... + relation = ::Security::Scan.latest_successful_by_build.by_scan_types(scan_type).where(security_scans: time_period) + + pipelines_with_secure_jobs['dependency_scanning_pipeline'] = estimate_batch_distinct_count(relation, :commit_id, batch_size: 1000, start: start_id, finish: finish_id) do |result| + ::Gitlab::Usage::Metrics::Aggregates::Sources::PostgresHll + .save_aggregated_metrics(metric_name: 'dependency_scanning_pipeline', recorded_at_timestamp: recorded_at, time_period: time_period, data: result) + end + end +end +``` + +#### Add new aggregated metric definition + +After all metrics are persisted, you can add an aggregated metric definition at +[`aggregated_metrics/`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/aggregates/). + +To declare the aggregate of metrics collected with [Estimated Batch Counters](#estimated-batch-counters), +you must fulfill the following requirements: + +- Metrics names listed in the `events:` attribute, have to use the same names you passed in the `metric_name` argument while persisting metrics in previous step. +- Every metric listed in the `events:` attribute, has to be persisted for **every** selected `time_frame:` value. + +Example definition: + +```yaml +- name: example_metrics_intersection_database_sourced + operator: AND + source: database + events: + - 'dependency_scanning_pipeline' + - 'container_scanning_pipeline' + time_frame: + - 28d + - all +``` diff --git a/doc/development/service_ping/index.md b/doc/development/service_ping/index.md index 816743a3e97..0a94fa2ff6c 100644 --- a/doc/development/service_ping/index.md +++ b/doc/development/service_ping/index.md @@ -66,7 +66,7 @@ We use the following terminology to describe the Service Ping components: > Introduced in GitLab 14.1. -Starting with GitLab version 14.1, free self-managed users running [GitLab EE](../ee_features.md) can receive paid features by registering with GitLab and sending us activity data via [Service Ping](#what-is-service-ping). +Starting with GitLab version 14.1, free self-managed users running [GitLab EE](../ee_features.md) can receive paid features by registering with GitLab and sending us activity data via [Service Ping](#what-is-service-ping). Features introduced here do not remove the feature from its paid tier. Users can continue to access the features in a paid tier without sharing usage data. The paid feature available in this offering is [Email from GitLab](../../tools/email.md). Administrators can use this [Premium](https://about.gitlab.com/pricing/premium/) feature to streamline @@ -85,7 +85,7 @@ Registration is not yet required for participation, but will be added in a futur You can view the exact JSON payload sent to GitLab Inc. in the Admin Area. To view the payload: 1. Sign in as a user with the [Administrator](../../user/permissions.md) role. -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > Metrics and profiling**. 1. Expand the **Usage statistics** section. 1. Select **Preview payload**. @@ -107,7 +107,7 @@ configuration file. To disable Service Ping in the GitLab UI: 1. Sign in as a user with the [Administrator](../../user/permissions.md) role. -1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > Metrics and profiling**. 1. Expand the **Usage statistics** section. 1. Clear the **Enable service ping** checkbox. @@ -222,844 +222,7 @@ We also collect metrics specific to [Geo](../../administration/geo/index.md) sec ## Implementing Service Ping -Service Ping consists of two kinds of data, counters and observations. Counters track how often a certain event -happened over time, such as how many CI pipelines have run. They are monotonic and always trend up. -Observations are facts collected from one or more GitLab instances and can carry arbitrary data. There are no -general guidelines around how to collect those, due to the individual nature of that data. - -### Types of counters - -There are several types of counters in `usage_data.rb`: - -- **Ordinary Batch Counters:** Simple count of a given ActiveRecord_Relation -- **Distinct Batch Counters:** Distinct count of a given ActiveRecord_Relation in a given column -- **Sum Batch Counters:** Sum the values of a given ActiveRecord_Relation in a given column -- **Alternative Counters:** Used for settings and configurations -- **Redis Counters:** Used for in-memory counts. - -NOTE: -Only use the provided counter methods. Each counter method contains a built-in fail-safe mechanism that isolates each counter to avoid breaking the entire Service Ping process. - -### Instrumentation classes - -We recommend you use [instrumentation classes](metrics_instrumentation.md) in `usage_data.rb` where possible. - -For example, we have the following instrumentation class: -`lib/gitlab/usage/metrics/instrumentations/count_boards_metric.rb`. - -You should add it to `usage_data.rb` as follows: - -```ruby -boards: add_metric('CountBoardsMetric', time_frame: 'all'), -``` - -### Batch counting - -For large tables, PostgreSQL can take a long time to count rows due to MVCC [(Multi-version Concurrency Control)](https://en.wikipedia.org/wiki/Multiversion_concurrency_control). Batch counting is a counting method where a single large query is broken into multiple smaller queries. For example, instead of a single query querying 1,000,000 records, with batch counting, you can execute 100 queries of 10,000 records each. Batch counting is useful for avoiding database timeouts as each batch query is significantly shorter than one single long running query. - -For GitLab.com, there are extremely large tables with 15 second query timeouts, so we use batch counting to avoid encountering timeouts. Here are the sizes of some GitLab.com tables: - -| Table | Row counts in millions | -|------------------------------|------------------------| -| `merge_request_diff_commits` | 2280 | -| `ci_build_trace_sections` | 1764 | -| `merge_request_diff_files` | 1082 | -| `events` | 514 | - -The following operation methods are available: - -- [Ordinary batch counters](#ordinary-batch-counters) -- [Distinct batch counters](#distinct-batch-counters) -- [Sum batch operation](#sum-batch-operation) -- [Add operation](#add-operation) -- [Estimated batch counters](#estimated-batch-counters) - -Batch counting requires indexes on columns to calculate max, min, and range queries. In some cases, -you may need to add a specialized index on the columns involved in a counter. - -### Ordinary batch counters - -Handles `ActiveRecord::StatementInvalid` error - -Simple count of a given `ActiveRecord_Relation`, does a non-distinct batch count, smartly reduces `batch_size`, and handles errors. - -Method: `count(relation, column = nil, batch: true, start: nil, finish: nil)` - -Arguments: - -- `relation` the ActiveRecord_Relation to perform the count -- `column` the column to perform the count on, by default is the primary key -- `batch`: default `true` to use batch counting -- `start`: custom start of the batch counting to avoid complex min calculations -- `end`: custom end of the batch counting to avoid complex min calculations - -Examples: - -```ruby -count(User.active) -count(::Clusters::Cluster.aws_installed.enabled, :cluster_id) -count(::Clusters::Cluster.aws_installed.enabled, :cluster_id, start: ::Clusters::Cluster.minimum(:id), finish: ::Clusters::Cluster.maximum(:id)) -``` - -### Distinct batch counters - -Handles `ActiveRecord::StatementInvalid` error - -Distinct count of a given `ActiveRecord_Relation` on given column, a distinct batch count, smartly reduces `batch_size`, and handles errors. - -Method: `distinct_count(relation, column = nil, batch: true, batch_size: nil, start: nil, finish: nil)` - -Arguments: - -- `relation` the ActiveRecord_Relation to perform the count -- `column` the column to perform the distinct count, by default is the primary key -- `batch`: default `true` to use batch counting -- `batch_size`: if none set it uses default value 10000 from `Gitlab::Database::BatchCounter` -- `start`: custom start of the batch counting to avoid complex min calculations -- `end`: custom end of the batch counting to avoid complex min calculations - -WARNING: -Counting over non-unique columns can lead to performance issues. For more information, see the [iterating tables in batches](../iterating_tables_in_batches.md) guide. - -Examples: - -```ruby -distinct_count(::Project, :creator_id) -distinct_count(::Note.with_suggestions.where(time_period), :author_id, start: ::User.minimum(:id), finish: ::User.maximum(:id)) -distinct_count(::Clusters::Applications::CertManager.where(time_period).available.joins(:cluster), 'clusters.user_id') -``` - -### Sum batch operation - -Handles `ActiveRecord::StatementInvalid` error - -Sum the values of a given ActiveRecord_Relation on given column and handles errors. - -Method: `sum(relation, column, batch_size: nil, start: nil, finish: nil)` - -Arguments: - -- `relation` the ActiveRecord_Relation to perform the operation -- `column` the column to sum on -- `batch_size`: if none set it uses default value 1000 from `Gitlab::Database::BatchCounter` -- `start`: custom start of the batch counting to avoid complex min calculations -- `end`: custom end of the batch counting to avoid complex min calculations - -Examples: - -```ruby -sum(JiraImportState.finished, :imported_issues_count) -``` - -### Grouping and batch operations - -The `count`, `distinct_count`, and `sum` batch counters can accept an `ActiveRecord::Relation` -object, which groups by a specified column. With a grouped relation, the methods do batch counting, -handle errors, and returns a hash table of key-value pairs. - -Examples: - -```ruby -count(Namespace.group(:type)) -# returns => {nil=>179, "Group"=>54} - -distinct_count(Project.group(:visibility_level), :creator_id) -# returns => {0=>1, 10=>1, 20=>11} - -sum(Issue.group(:state_id), :weight)) -# returns => {1=>3542, 2=>6820} -``` - -### Add operation - -Handles `StandardError`. - -Returns `-1` if any of the arguments are `-1`. - -Sum the values given as parameters. - -Method: `add(*args)` - -Examples: - -```ruby -project_imports = distinct_count(::Project.where.not(import_type: nil), :creator_id) -bulk_imports = distinct_count(::BulkImport, :user_id) - - add(project_imports, bulk_imports) -``` - -### Estimated batch counters - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/48233) in GitLab 13.7. - -Estimated batch counter functionality handles `ActiveRecord::StatementInvalid` errors -when used through the provided `estimate_batch_distinct_count` method. -Errors return a value of `-1`. - -WARNING: -This functionality estimates a distinct count of a specific ActiveRecord_Relation in a given column, -which uses the [HyperLogLog](http://algo.inria.fr/flajolet/Publications/FlFuGaMe07.pdf) algorithm. -As the HyperLogLog algorithm is probabilistic, the **results always include error**. -The highest encountered error rate is 4.9%. - -When correctly used, the `estimate_batch_distinct_count` method enables efficient counting over -columns that contain non-unique values, which can not be assured by other counters. - -#### estimate_batch_distinct_count method - -Method: `estimate_batch_distinct_count(relation, column = nil, batch_size: nil, start: nil, finish: nil)` - -The [method](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/utils/usage_data.rb#L63) -includes the following arguments: - -- `relation`: The ActiveRecord_Relation to perform the count. -- `column`: The column to perform the distinct count. The default is the primary key. -- `batch_size`: From `Gitlab::Database::PostgresHll::BatchDistinctCounter::DEFAULT_BATCH_SIZE`. Default value: 10,000. -- `start`: The custom start of the batch count, to avoid complex minimum calculations. -- `finish`: The custom end of the batch count to avoid complex maximum calculations. - -The method includes the following prerequisites: - -1. The supplied `relation` must include the primary key defined as the numeric column. - For example: `id bigint NOT NULL`. -1. The `estimate_batch_distinct_count` can handle a joined relation. To use its ability to - count non-unique columns, the joined relation **must not** have a one-to-many relationship, - such as `has_many :boards`. -1. Both `start` and `finish` arguments should always represent primary key relationship values, - even if the estimated count refers to another column, for example: - - ```ruby - estimate_batch_distinct_count(::Note, :author_id, start: ::Note.minimum(:id), finish: ::Note.maximum(:id)) - ``` - -Examples: - -1. Simple execution of estimated batch counter, with only relation provided, - returned value represents estimated number of unique values in `id` column - (which is the primary key) of `Project` relation: - - ```ruby - estimate_batch_distinct_count(::Project) - ``` - -1. Execution of estimated batch counter, where provided relation has applied - additional filter (`.where(time_period)`), number of unique values estimated - in custom column (`:author_id`), and parameters: `start` and `finish` together - apply boundaries that defines range of provided relation to analyze: - - ```ruby - estimate_batch_distinct_count(::Note.with_suggestions.where(time_period), :author_id, start: ::Note.minimum(:id), finish: ::Note.maximum(:id)) - ``` - -1. Execution of estimated batch counter with joined relation (`joins(:cluster)`), - for a custom column (`'clusters.user_id'`): - - ```ruby - estimate_batch_distinct_count(::Clusters::Applications::CertManager.where(time_period).available.joins(:cluster), 'clusters.user_id') - ``` - -When instrumenting metric with usage of estimated batch counter please add -`_estimated` suffix to its name, for example: - -```ruby - "counts": { - "ci_builds_estimated": estimate_batch_distinct_count(Ci::Build), - ... -``` - -### Redis counters - -Handles `::Redis::CommandError` and `Gitlab::UsageDataCounters::BaseCounter::UnknownEvent` -returns -1 when a block is sent or hash with all values -1 when a `counter(Gitlab::UsageDataCounters)` is sent -different behavior due to 2 different implementations of Redis counter - -Method: `redis_usage_data(counter, &block)` - -Arguments: - -- `counter`: a counter from `Gitlab::UsageDataCounters`, that has `fallback_totals` method implemented -- or a `block`: which is evaluated - -#### Ordinary Redis counters - -Examples of implementation: - -- Using Redis methods [`INCR`](https://redis.io/commands/incr), [`GET`](https://redis.io/commands/get), and [`Gitlab::UsageDataCounters::WikiPageCounter`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/wiki_page_counter.rb) -- Using Redis methods [`HINCRBY`](https://redis.io/commands/hincrby), [`HGETALL`](https://redis.io/commands/hgetall), and [`Gitlab::UsageCounters::PodLogs`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_counters/pod_logs.rb) - -##### UsageData API tracking - -<!-- There's nearly identical content in `##### Adding new events`. If you fix errors here, you may need to fix the same errors in the other location. --> - -1. Track event using `UsageData` API - - Increment event count using ordinary Redis counter, for given event name. - - Tracking events using the `UsageData` API requires the `usage_data_api` feature flag to be enabled, which is enabled by default. - - API requests are protected by checking for a valid CSRF token. - - To be able to increment the values, the related feature `usage_data_<event_name>` should be enabled. - - ```plaintext - POST /usage_data/increment_counter - ``` - - | Attribute | Type | Required | Description | - | :-------- | :--- | :------- | :---------- | - | `event` | string | yes | The event name it should be tracked | - - Response: - - - `200` if event was tracked - - `400 Bad request` if event parameter is missing - - `401 Unauthorized` if user is not authenticated - - `403 Forbidden` for invalid CSRF token provided - -1. Track events using JavaScript/Vue API helper which calls the API above - - Note that `usage_data_api` and `usage_data_#{event_name}` should be enabled to be able to track events - - ```javascript - import api from '~/api'; - - api.trackRedisCounterEvent('my_already_defined_event_name'), - ``` - -#### Redis HLL counters - -WARNING: -HyperLogLog (HLL) is a probabilistic algorithm and its **results always includes some small error**. According to [Redis documentation](https://redis.io/commands/pfcount), data from -used HLL implementation is "approximated with a standard error of 0.81%". - -With `Gitlab::UsageDataCounters::HLLRedisCounter` we have available data structures used to count unique values. - -Implemented using Redis methods [PFADD](https://redis.io/commands/pfadd) and [PFCOUNT](https://redis.io/commands/pfcount). - -##### Add new events - -1. Define events in [`known_events`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events/). - - Example event: - - ```yaml - - name: users_creating_epics - category: epics_usage - redis_slot: users - aggregation: weekly - feature_flag: track_epics_activity - ``` - - Keys: - - - `name`: unique event name. - - Name format for Redis HLL events `<name>_<redis_slot>`. - - [See Metric name](metrics_dictionary.md#metric-name) for a complete guide on metric naming suggestion. - - Consider including in the event's name the Redis slot to be able to count totals for a specific category. - - Example names: `users_creating_epics`, `users_triggering_security_scans`. - - - `category`: event category. Used for getting total counts for events in a category, for easier - access to a group of events. - - `redis_slot`: optional Redis slot. Default value: event name. Only event data that is stored in the same slot - can be aggregated. Ensure keys are in the same slot. For example: - `users_creating_epics` with `redis_slot: 'users'` builds Redis key - `{users}_creating_epics-2020-34`. If `redis_slot` is not defined the Redis key will - be `{users_creating_epics}-2020-34`. - Recommended slots to use are: `users`, `projects`. This is the value we count. - - `expiry`: expiry time in days. Default: 29 days for daily aggregation and 6 weeks for weekly - aggregation. - - `aggregation`: may be set to a `:daily` or `:weekly` key. Defines how counting data is stored in Redis. - Aggregation on a `daily` basis does not pull more fine grained data. - - `feature_flag`: optional `default_enabled: :yaml`. If no feature flag is set then the tracking is enabled. One feature flag can be used for multiple events. For details, see our [GitLab internal Feature flags](../feature_flags/index.md) documentation. The feature flags are owned by the group adding the event tracking. - -1. Use one of the following methods to track the event: - - - In the controller using the `RedisTracking` module and the following format: - - ```ruby - track_redis_hll_event(*controller_actions, name:, if: nil, &block) - ``` - - Arguments: - - - `controller_actions`: the controller actions to track. - - `name`: the event name. - - `if`: optional custom conditions. Uses the same format as Rails callbacks. - - `&block`: optional block that computes and returns the `custom_id` that we want to track. This overrides the `visitor_id`. - - Example: - - ```ruby - # controller - class ProjectsController < Projects::ApplicationController - include RedisTracking - - skip_before_action :authenticate_user!, only: :show - track_redis_hll_event :index, :show, name: 'users_visiting_projects' - - def index - render html: 'index' - end - - def new - render html: 'new' - end - - def show - render html: 'show' - end - end - ``` - - - In the API using the `increment_unique_values(event_name, values)` helper method. - - Arguments: - - - `event_name`: the event name. - - `values`: the values counted. Can be one value or an array of values. - - Example: - - ```ruby - get ':id/registry/repositories' do - repositories = ContainerRepositoriesFinder.new( - user: current_user, subject: user_group - ).execute - - increment_unique_values('users_listing_repositories', current_user.id) - - present paginate(repositories), with: Entities::ContainerRegistry::Repository, tags: params[:tags], tags_count: params[:tags_count] - end - ``` - - - Using `track_usage_event(event_name, values)` in services and GraphQL. - - Increment unique values count using Redis HLL, for a given event name. - - Examples: - - - [Track usage event for an incident in a service](https://gitlab.com/gitlab-org/gitlab/-/blob/v13.8.3-ee/app/services/issues/update_service.rb#L66) - - [Track usage event for an incident in GraphQL](https://gitlab.com/gitlab-org/gitlab/-/blob/v13.8.3-ee/app/graphql/mutations/alert_management/update_alert_status.rb#L16) - - ```ruby - track_usage_event(:incident_management_incident_created, current_user.id) - ``` - - - Using the `UsageData` API. - <!-- There's nearly identical content in `##### UsageData API Tracking`. If you find / fix errors here, you may need to fix errors in that section too. --> - - Increment unique users count using Redis HLL, for a given event name. - - To track events using the `UsageData` API, ensure the `usage_data_api` feature flag - is set to `default_enabled: true`. Enabled by default in GitLab 13.7 and later. - - API requests are protected by checking for a valid CSRF token. - - ```plaintext - POST /usage_data/increment_unique_users - ``` - - | Attribute | Type | Required | Description | - | :-------- | :--- | :------- | :---------- | - | `event` | string | yes | The event name to track | - - Response: - - - `200` if the event was tracked, or if tracking failed for any reason. - - `400 Bad request` if an event parameter is missing. - - `401 Unauthorized` if the user is not authenticated. - - `403 Forbidden` if an invalid CSRF token is provided. - - - Using the JavaScript/Vue API helper, which calls the `UsageData` API. - - To track events using the `UsageData` API, ensure the `usage_data_api` feature flag - is set to `default_enabled: true`. Enabled by default in GitLab 13.7 and later. - - Example for an existing event already defined in [known events](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events/): - - ```javascript - import api from '~/api'; - - api.trackRedisHllUserEvent('my_already_defined_event_name'), - ``` - -1. Get event data using `Gitlab::UsageDataCounters::HLLRedisCounter.unique_events(event_names:, start_date:, end_date:, context: '')`. - - Arguments: - - - `event_names`: the list of event names. - - `start_date`: start date of the period for which we want to get event data. - - `end_date`: end date of the period for which we want to get event data. - - `context`: context of the event. Allowed values are `default`, `free`, `bronze`, `silver`, `gold`, `starter`, `premium`, `ultimate`. - -1. Testing tracking and getting unique events - -Trigger events in rails console by using `track_event` method - - ```ruby - Gitlab::UsageDataCounters::HLLRedisCounter.track_event('users_viewing_compliance_audit_events', values: 1) - Gitlab::UsageDataCounters::HLLRedisCounter.track_event('users_viewing_compliance_audit_events', values: [2, 3]) - ``` - -Next, get the unique events for the current week. - - ```ruby - # Get unique events for metric for current_week - Gitlab::UsageDataCounters::HLLRedisCounter.unique_events(event_names: 'users_viewing_compliance_audit_events', - start_date: Date.current.beginning_of_week, end_date: Date.current.next_week) - ``` - -##### Recommendations - -We have the following recommendations for [adding new events](#add-new-events): - -- Event aggregation: weekly. -- Key expiry time: - - Daily: 29 days. - - Weekly: 42 days. -- When adding new metrics, use a [feature flag](../../operations/feature_flags.md) to control the impact. -- For feature flags triggered by another service, set `default_enabled: false`, - - Events can be triggered using the `UsageData` API, which helps when there are > 10 events per change - -##### Enable or disable Redis HLL tracking - -Events are tracked behind optional [feature flags](../feature_flags/index.md) due to concerns for Redis performance and scalability. - -For a full list of events and corresponding feature flags see, [known_events](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events/) files. - -To enable or disable tracking for specific event in <https://gitlab.com> or <https://about.staging.gitlab.com>, run commands such as the following to -[enable or disable the corresponding feature](../feature_flags/index.md). - -```shell -/chatops run feature set <feature_name> true -/chatops run feature set <feature_name> false -``` - -We can also disable tracking completely by using the global flag: - -```shell -/chatops run feature set redis_hll_tracking true -/chatops run feature set redis_hll_tracking false -``` - -##### Known events are added automatically in Service Data payload - -All events added in [`known_events/common.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events/common.yml) are automatically added to Service Data generation under the `redis_hll_counters` key. This column is stored in [version-app as a JSON](https://gitlab.com/gitlab-services/version-gitlab-com/-/blob/master/db/schema.rb#L209). -For each event we add metrics for the weekly and monthly time frames, and totals for each where applicable: - -- `#{event_name}_weekly`: Data for 7 days for daily [aggregation](#add-new-events) events and data for the last complete week for weekly [aggregation](#add-new-events) events. -- `#{event_name}_monthly`: Data for 28 days for daily [aggregation](#add-new-events) events and data for the last 4 complete weeks for weekly [aggregation](#add-new-events) events. - -Redis HLL implementation calculates automatic total metrics, if there are more than one metric for the same category, aggregation, and Redis slot. - -- `#{category}_total_unique_counts_weekly`: Total unique counts for events in the same category for the last 7 days or the last complete week, if events are in the same Redis slot and we have more than one metric. -- `#{category}_total_unique_counts_monthly`: Total unique counts for events in same category for the last 28 days or the last 4 complete weeks, if events are in the same Redis slot and we have more than one metric. - -Example of `redis_hll_counters` data: - -```ruby -{:redis_hll_counters=> - {"compliance"=> - {"users_viewing_compliance_dashboard_weekly"=>0, - "users_viewing_compliance_dashboard_monthly"=>0, - "users_viewing_compliance_audit_events_weekly"=>0, - "users_viewing_audit_events_monthly"=>0, - "compliance_total_unique_counts_weekly"=>0, - "compliance_total_unique_counts_monthly"=>0}, - "analytics"=> - {"users_viewing_analytics_group_devops_adoption_weekly"=>0, - "users_viewing_analytics_group_devops_adoption_monthly"=>0, - "analytics_total_unique_counts_weekly"=>0, - "analytics_total_unique_counts_monthly"=>0}, - "ide_edit"=> - {"users_editing_by_web_ide_weekly"=>0, - "users_editing_by_web_ide_monthly"=>0, - "users_editing_by_sfe_weekly"=>0, - "users_editing_by_sfe_monthly"=>0, - "ide_edit_total_unique_counts_weekly"=>0, - "ide_edit_total_unique_counts_monthly"=>0} - } -``` - -Example: - -```ruby -# Redis Counters -redis_usage_data(Gitlab::UsageDataCounters::WikiPageCounter) -redis_usage_data { ::Gitlab::UsageCounters::PodLogs.usage_totals[:total] } - -# Define events in common.yml https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events/common.yml - -# Tracking events -Gitlab::UsageDataCounters::HLLRedisCounter.track_event('users_expanding_vulnerabilities', values: visitor_id) - -# Get unique events for metric -redis_usage_data { Gitlab::UsageDataCounters::HLLRedisCounter.unique_events(event_names: 'users_expanding_vulnerabilities', start_date: 28.days.ago, end_date: Date.current) } -``` - -### Alternative counters - -Handles `StandardError` and fallbacks into -1 this way not all measures fail if we encounter one exception. -Mainly used for settings and configurations. - -Method: `alt_usage_data(value = nil, fallback: -1, &block)` - -Arguments: - -- `value`: a simple static value in which case the value is simply returned. -- or a `block`: which is evaluated -- `fallback: -1`: the common value used for any metrics that are failing. - -Example: - -```ruby -alt_usage_data { Gitlab::VERSION } -alt_usage_data { Gitlab::CurrentSettings.uuid } -alt_usage_data(999) -``` - -### Add counters to build new metrics - -When adding the results of two counters, use the `add` Service Data method that -handles fallback values and exceptions. It also generates a valid [SQL export](#export-service-ping-sql-queries-and-definitions). - -Example: - -```ruby -add(User.active, User.bot) -``` - -### Prometheus queries - -In those cases where operational metrics should be part of Service Ping, a database or Redis query is unlikely -to provide useful data. Instead, Prometheus might be more appropriate, because most GitLab architectural -components publish metrics to it that can be queried back, aggregated, and included as Service Data. - -NOTE: -Prometheus as a data source for Service Ping is only available for single-node Omnibus installations -that are running the [bundled Prometheus](../../administration/monitoring/prometheus/index.md) instance. - -To query Prometheus for metrics, a helper method is available to `yield` a fully configured -`PrometheusClient`, given it is available as per the note above: - -```ruby -with_prometheus_client do |client| - response = client.query('<your query>') - ... -end -``` - -Refer to [the `PrometheusClient` definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/prometheus_client.rb) -for how to use its API to query for data. - -### Fallback values for Service Ping - -We return fallback values in these cases: - -| Case | Value | -|-----------------------------|-------| -| Deprecated Metric | -1000 | -| Timeouts, general failures | -1 | -| Standard errors in counters | -2 | - -## Aggregated metrics - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/45979) in GitLab 13.6. - -WARNING: -This feature is intended solely for internal GitLab use. - -To add data for aggregated metrics to the Service Ping payload, add a corresponding definition to: - -- [`config/metrics/aggregates/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/aggregates/) for metrics available in the Community Edition. -- [`ee/config/metrics/aggregates/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/aggregates/) for metrics available in the Enterprise Edition. - -Each aggregate definition includes following parts: - -- `name`: Unique name under which the aggregate metric is added to the Service Ping payload. -- `operator`: Operator that defines how the aggregated metric data is counted. Available operators are: - - `OR`: Removes duplicates and counts all entries that triggered any of listed events. - - `AND`: Removes duplicates and counts all elements that were observed triggering all of following events. -- `time_frame`: One or more valid time frames. Use these to limit the data included in aggregated metric to events within a specific date-range. Valid time frames are: - - `7d`: Last seven days of data. - - `28d`: Last twenty eight days of data. - - `all`: All historical data, only available for `database` sourced aggregated metrics. -- `source`: Data source used to collect all events data included in aggregated metric. Valid data sources are: - - [`database`](#database-sourced-aggregated-metrics) - - [`redis`](#redis-sourced-aggregated-metrics) -- `events`: list of events names to aggregate into metric. All events in this list must - relay on the same data source. Additional data source requirements are described in the - [Database sourced aggregated metrics](#database-sourced-aggregated-metrics) and - [Redis sourced aggregated metrics](#redis-sourced-aggregated-metrics) sections. -- `feature_flag`: Name of [development feature flag](../feature_flags/index.md#development-type) - that is checked before metrics aggregation is performed. Corresponding feature flag - should have `default_enabled` attribute set to `false`. The `feature_flag` attribute - is optional and can be omitted. When `feature_flag` is missing, no feature flag is checked. - -Example aggregated metric entries: - -```yaml -- name: example_metrics_union - operator: OR - events: - - 'users_expanding_secure_security_report' - - 'users_expanding_testing_code_quality_report' - - 'users_expanding_testing_accessibility_report' - source: redis - time_frame: - - 7d - - 28d -- name: example_metrics_intersection - operator: AND - source: database - time_frame: - - 28d - - all - events: - - 'dependency_scanning_pipeline_all_time' - - 'container_scanning_pipeline_all_time' - feature_flag: example_aggregated_metric -``` - -Aggregated metrics collected in `7d` and `28d` time frames are added into Service Ping payload under the `aggregated_metrics` sub-key in the `counts_weekly` and `counts_monthly` top level keys. - -```ruby -{ - :counts_monthly => { - :deployments => 1003, - :successful_deployments => 78, - :failed_deployments => 275, - :packages => 155, - :personal_snippets => 2106, - :project_snippets => 407, - :promoted_issues => 719, - :aggregated_metrics => { - :example_metrics_union => 7, - :example_metrics_intersection => 2 - }, - :snippets => 2513 - } -} -``` - -Aggregated metrics for `all` time frame are present in the `count` top level key, with the `aggregate_` prefix added to their name. - -For example: - -`example_metrics_intersection` - -Becomes: - -`counts.aggregate_example_metrics_intersection` - -```ruby -{ - :counts => { - :deployments => 11003, - :successful_deployments => 178, - :failed_deployments => 1275, - :aggregate_example_metrics_intersection => 12 - } -} -``` - -### Redis sourced aggregated metrics - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/45979) in GitLab 13.6. - -To declare the aggregate of events collected with [Redis HLL Counters](#redis-hll-counters), -you must fulfill the following requirements: - -1. All events listed at `events` attribute must come from - [`known_events/*.yml`](#known-events-are-added-automatically-in-service-data-payload) files. -1. All events listed at `events` attribute must have the same `redis_slot` attribute. -1. All events listed at `events` attribute must have the same `aggregation` attribute. -1. `time_frame` does not include `all` value, which is unavailable for Redis sourced aggregated metrics. - -### Database sourced aggregated metrics - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/52784) in GitLab 13.9. -> - It's [deployed behind a feature flag](../../user/feature_flags.md), disabled by default. -> - It's enabled on GitLab.com. - -To declare an aggregate of metrics based on events collected from database, follow -these steps: - -1. [Persist the metrics for aggregation](#persist-metrics-for-aggregation). -1. [Add new aggregated metric definition](#add-new-aggregated-metric-definition). - -#### Persist metrics for aggregation - -Only metrics calculated with [Estimated Batch Counters](#estimated-batch-counters) -can be persisted for database sourced aggregated metrics. To persist a metric, -inject a Ruby block into the -[estimate_batch_distinct_count](#estimate_batch_distinct_count-method) method. -This block should invoke the -`Gitlab::Usage::Metrics::Aggregates::Sources::PostgresHll.save_aggregated_metrics` -[method](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage/metrics/aggregates/sources/postgres_hll.rb#L21), -which stores `estimate_batch_distinct_count` results for future use in aggregated metrics. - -The `Gitlab::Usage::Metrics::Aggregates::Sources::PostgresHll.save_aggregated_metrics` -method accepts the following arguments: - -- `metric_name`: The name of metric to use for aggregations. Should be the same - as the key under which the metric is added into Service Ping. -- `recorded_at_timestamp`: The timestamp representing the moment when a given - Service Ping payload was collected. You should use the convenience method `recorded_at` - to fill `recorded_at_timestamp` argument, like this: `recorded_at_timestamp: recorded_at` -- `time_period`: The time period used to build the `relation` argument passed into - `estimate_batch_distinct_count`. To collect the metric with all available historical - data, set a `nil` value as time period: `time_period: nil`. -- `data`: HyperLogLog buckets structure representing unique entries in `relation`. - The `estimate_batch_distinct_count` method always passes the correct argument - into the block, so `data` argument must always have a value equal to block argument, - like this: `data: result` - -Example metrics persistence: - -```ruby -class UsageData - def count_secure_pipelines(time_period) - ... - relation = ::Security::Scan.latest_successful_by_build.by_scan_types(scan_type).where(security_scans: time_period) - - pipelines_with_secure_jobs['dependency_scanning_pipeline'] = estimate_batch_distinct_count(relation, :commit_id, batch_size: 1000, start: start_id, finish: finish_id) do |result| - ::Gitlab::Usage::Metrics::Aggregates::Sources::PostgresHll - .save_aggregated_metrics(metric_name: 'dependency_scanning_pipeline', recorded_at_timestamp: recorded_at, time_period: time_period, data: result) - end - end -end -``` - -#### Add new aggregated metric definition - -After all metrics are persisted, you can add an aggregated metric definition at -[`aggregated_metrics/`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/aggregates/). - -To declare the aggregate of metrics collected with [Estimated Batch Counters](#estimated-batch-counters), -you must fulfill the following requirements: - -- Metrics names listed in the `events:` attribute, have to use the same names you passed in the `metric_name` argument while persisting metrics in previous step. -- Every metric listed in the `events:` attribute, has to be persisted for **every** selected `time_frame:` value. - -Example definition: - -```yaml -- name: example_metrics_intersection_database_sourced - operator: AND - source: database - events: - - 'dependency_scanning_pipeline' - - 'container_scanning_pipeline' - time_frame: - - 28d - - all -``` +See the [implement Service Ping](implement.md) guide. ## Example Service Ping payload @@ -1331,7 +494,7 @@ checking the configuration file of your GitLab instance: - Using the Admin Area: - 1. On the top bar, select **Menu >** **{admin}** **Admin**. + 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > Metrics and profiling**. 1. Expand **Usage Statistics**. 1. Are you able to check or uncheck the checkbox to disable Service Ping? @@ -1388,7 +551,7 @@ To work around this bug, you have two options: sudo gitlab-ctl reconfigure ``` - 1. In GitLab, on the top bar, select **Menu >** **{admin}** **Admin**. + 1. In GitLab, on the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > Metrics and profiling**. 1. Expand **Usage Statistics**. 1. Clear the **Enable service ping** checkbox. diff --git a/doc/development/service_ping/metrics_dictionary.md b/doc/development/service_ping/metrics_dictionary.md index b3c5f078a4a..8dc2d2255d1 100644 --- a/doc/development/service_ping/metrics_dictionary.md +++ b/doc/development/service_ping/metrics_dictionary.md @@ -22,6 +22,9 @@ All metrics are stored in YAML files: - [`config/metrics`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/config/metrics) +WARNING: +Only metrics with a metric definition YAML are added to the Service Ping JSON payload. + Each metric is defined in a separate YAML file consisting of a number of fields: | Field | Required | Additional information | @@ -34,14 +37,14 @@ Each metric is defined in a separate YAML file consisting of a number of fields: | `product_group` | yes | The [group](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/stages.yml) that owns the metric. | | `product_category` | no | The [product category](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/categories.yml) for the metric. | | `value_type` | yes | `string`; one of [`string`, `number`, `boolean`, `object`](https://json-schema.org/understanding-json-schema/reference/type.html). | -| `status` | yes | `string`; [status](#metric-statuses) of the metric, may be set to `data_available`, `implemented`, `not_used`, `deprecated`, `removed`, `broken`. | +| `status` | yes | `string`; [status](#metric-statuses) of the metric, may be set to `active`, `deprecated`, `removed`, `broken`. | | `time_frame` | yes | `string`; may be set to a value like `7d`, `28d`, `all`, `none`. | | `data_source` | yes | `string`; may be set to a value like `database`, `redis`, `redis_hll`, `prometheus`, `system`. | | `data_category` | yes | `string`; [categories](#data-category) of the metric, may be set to `operational`, `optional`, `subscription`, `standard`. The default value is `optional`.| | `instrumentation_class` | no | `string`; [the class that implements the metric](metrics_instrumentation.md). | | `distribution` | yes | `array`; may be set to one of `ce, ee` or `ee`. The [distribution](https://about.gitlab.com/handbook/marketing/strategic-marketing/tiers/#definitions) where the tracked feature is available. | | `performance_indicator_type` | no | `array`; may be set to one of [`gmau`, `smau`, `paid_gmau`, or `umau`](https://about.gitlab.com/handbook/business-technology/data-team/data-catalog/xmau-analysis/). | -| `tier` | yes | `array`; may contain one or a combination of `free`, `premium` or `ultimate`. The [tier]( https://about.gitlab.com/handbook/marketing/strategic-marketing/tiers/) where the tracked feature is available. | +| `tier` | yes | `array`; may contain one or a combination of `free`, `premium` or `ultimate`. The [tier]( https://about.gitlab.com/handbook/marketing/strategic-marketing/tiers/) where the tracked feature is available. This should be verbose and contain all tiers where a metric is available. | | `milestone` | no | The milestone when the metric is introduced. | | `milestone_removed` | no | The milestone when the metric is removed. | | `introduced_by_url` | no | The URL to the Merge Request that introduced the metric. | @@ -53,11 +56,8 @@ Each metric is defined in a separate YAML file consisting of a number of fields: Metric definitions can have one of the following statuses: -- `data_available`: Metric data is available and used in a Sisense dashboard. -- `implemented`: Metric is implemented but data is not yet available. This is a temporary - status for newly added metrics awaiting inclusion in a new release. +- `active`: Metric is used and reports data. - `broken`: Metric reports broken data (for example, -1 fallback), or does not report data at all. A metric marked as `broken` must also have the `repair_issue_url` attribute. -- `not_used`: Metric is not used in any dashboard. - `deprecated`: Metric is deprecated and possibly planned to be removed. - `removed`: Metric was removed, but it may appear in Service Ping payloads sent from instances running on older versions of GitLab. @@ -177,7 +177,7 @@ product_section: growth product_stage: growth product_group: group::product intelligence value_type: string -status: data_available +status: active milestone: 9.1 introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/1521 time_frame: none @@ -217,11 +217,11 @@ create ee/config/metrics/counts_7d/issues.yml ## Metrics added dynamic to Service Ping payload -The [Redis HLL metrics](index.md#known-events-are-added-automatically-in-service-data-payload) are added automatically to Service Ping payload. +The [Redis HLL metrics](implement.md#known-events-are-added-automatically-in-service-data-payload) are added automatically to Service Ping payload. A YAML metric definition is required for each metric. A dedicated generator is provided to create metric definitions for Redis HLL events. -The generator takes `category` and `event` arguments, as the root key will be `redis_hll_counters`, and creates two metric definitions for weekly and monthly timeframes: +The generator takes `category` and `event` arguments, as the root key is `redis_hll_counters`, and creates two metric definitions for weekly and monthly time frames: ```shell bundle exec rails generate gitlab:usage_metric_definition:redis_hll issues i_closed diff --git a/doc/development/service_ping/metrics_instrumentation.md b/doc/development/service_ping/metrics_instrumentation.md index 8ad1ae9cce2..6fdbd1eea31 100644 --- a/doc/development/service_ping/metrics_instrumentation.md +++ b/doc/development/service_ping/metrics_instrumentation.md @@ -115,13 +115,13 @@ There is support for: - [Redis HLL metrics](#redis-hyperloglog-metrics). - [Generic metrics](#generic-metrics), which are metrics based on settings or configurations. -Currently, there is no support for: +There is no support for: - `add`, `sum`, `histogram` for database metrics. You can [track the progress to support these](https://gitlab.com/groups/gitlab-org/-/epics/6118). -## Creating a new metric instrumentation class +## Create a new metric instrumentation class To create a stub instrumentation for a Service Ping metric, you can use a dedicated [generator](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/generators/gitlab/usage_metric_generator.rb): diff --git a/doc/development/service_ping/metrics_lifecycle.md b/doc/development/service_ping/metrics_lifecycle.md index 68c5ddecc1f..c0446aece8b 100644 --- a/doc/development/service_ping/metrics_lifecycle.md +++ b/doc/development/service_ping/metrics_lifecycle.md @@ -10,7 +10,7 @@ The following guidelines explain the steps to follow at each stage of a metric's ## Add a new metric -Please follow the [Implementing Service Ping](index.md#implementing-service-ping) guide. +Follow the [Implement Service Ping](implement.md) guide. ## Change an existing metric @@ -39,7 +39,7 @@ For GitLab 12.6, the metric was changed to filter out archived projects: } ``` -In this scenario all instances running up to GitLab 12.5 continue to report `example_metric`, +In this scenario, all instances running up to GitLab 12.5 continue to report `example_metric`, including all archived projects, while all instances running GitLab 12.6 and higher filters out such projects. As Service Ping data is collected from all reporting instances, the resulting dataset includes mixed data, which distorts any following business analysis. diff --git a/doc/development/service_ping/review_guidelines.md b/doc/development/service_ping/review_guidelines.md index f961fd376bd..048b705636f 100644 --- a/doc/development/service_ping/review_guidelines.md +++ b/doc/development/service_ping/review_guidelines.md @@ -59,7 +59,7 @@ are regular backend changes. metrics that are based on Database. - For tracking using Redis HLL (HyperLogLog): - Check the Redis slot. - - Check if a [feature flag is needed](index.md#recommendations). + - Check if a [feature flag is needed](implement.md#recommendations). - For a metric's YAML definition: - Check the metric's `description`. - Check the metric's `key_path`. diff --git a/doc/development/sidekiq_style_guide.md b/doc/development/sidekiq_style_guide.md index c1733a974e4..04b7e2f5c45 100644 --- a/doc/development/sidekiq_style_guide.md +++ b/doc/development/sidekiq_style_guide.md @@ -713,7 +713,7 @@ default weight, which is 1. ## Worker context -> - [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/9) in GitLab 12.8. +> [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/9) in GitLab 12.8. To have some more information about workers in the logs, we add [metadata to the jobs in the form of an @@ -1002,9 +1002,7 @@ When renaming queues, use the `sidekiq_queue_migrate` helper migration method in a **post-deployment migration**: ```ruby -class MigrateTheRenamedSidekiqQueue < ActiveRecord::Migration[5.0] - include Gitlab::Database::MigrationHelpers - +class MigrateTheRenamedSidekiqQueue < Gitlab::Database::Migration[1.0] def up sidekiq_queue_migrate 'old_queue_name', to: 'new_queue_name' end diff --git a/doc/development/single_table_inheritance.md b/doc/development/single_table_inheritance.md index aa4fe540b0d..eb406b02a91 100644 --- a/doc/development/single_table_inheritance.md +++ b/doc/development/single_table_inheritance.md @@ -31,7 +31,7 @@ could result in loading unexpected code or associations which may cause unintend side effects or failures during upgrades. ```ruby -class SomeMigration < ActiveRecord::Migration[6.0] +class SomeMigration < Gitlab::Database::Migration[1.0] class Services < ActiveRecord::Base self.table_name = 'services' self.inheritance_column = :_type_disabled @@ -47,7 +47,7 @@ This ensures that the migration loads the columns for the migration in isolation and the helper disables STI by default. ```ruby -class EnqueueSomeBackgroundMigration < ActiveRecord::Migration[6.0] +class EnqueueSomeBackgroundMigration < Gitlab::Database::Migration[1.0] disable_ddl_transaction! def up diff --git a/doc/development/snowplow/index.md b/doc/development/snowplow/index.md index 527b4292b23..e8b7d871b77 100644 --- a/doc/development/snowplow/index.md +++ b/doc/development/snowplow/index.md @@ -53,7 +53,7 @@ Snowplow tracking is enabled on GitLab.com, and we use it for most of our tracki To enable Snowplow tracking on a self-managed instance: -1. On the top bar, select **Menu >** **{admin}** **Admin**, then select **Settings > General**. +1. On the top bar, select **Menu > Admin**, then select **Settings > General**. Alternatively, go to `admin/application_settings/general` in your browser. 1. Expand **Snowplow**. @@ -101,7 +101,7 @@ sequenceDiagram ## Structured event taxonomy -When adding new click events, we should add them in a way that's internally consistent. If we don't, it is very painful to perform analysis across features since each feature captures events differently. +When adding new click events, we should add them in a way that's internally consistent. If we don't, it is difficult to perform analysis across features because each feature captures events differently. The current method provides several attributes that are sent on each click event. Please try to follow these guidelines when specifying events to capture: @@ -109,9 +109,9 @@ The current method provides several attributes that are sent on each click event | --------- | ------- | -------- | ----------- | | category | text | true | The page or backend area of the application. Unless infeasible, please use the Rails page attribute by default in the frontend, and namespace + class name on the backend. | | action | text | true | The action the user is taking, or aspect that's being instrumented. The first word should always describe the action or aspect: clicks should be `click`, activations should be `activate`, creations should be `create`, etc. Use underscores to describe what was acted on; for example, activating a form field would be `activate_form_input`. An interface action like clicking on a dropdown would be `click_dropdown`, while a behavior like creating a project record from the backend would be `create_project` | -| label | text | false | The specific element, or object that's being acted on. This is either the label of the element (e.g. a tab labeled 'Create from template' may be `create_from_template`) or a unique identifier if no text is available (e.g. closing the Groups dropdown in the top navigation bar might be `groups_dropdown_close`), or it could be the name or title attribute of a record being created. | +| label | text | false | The specific element or object to act on. This can be one of the following: the label of the element (for example, a tab labeled 'Create from template' for `create_from_template`), a unique identifier if no text is available (for example, `groups_dropdown_close` for closing the Groups dropdown in the top bar), or the name or title attribute of a record being created. | | property | text | false | Any additional property of the element, or object being acted on. | -| value | decimal | false | Describes a numeric value or something directly related to the event. This could be the value of an input (e.g. `10` when clicking `internal` visibility). | +| value | decimal | false | Describes a numeric value or something directly related to the event. This could be the value of an input. For example, `10` when clicking `internal` visibility. | ### Examples @@ -156,7 +156,7 @@ LIMIT 20 Snowplow JS adds many [web-specific parameters](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/snowplow-tracker-protocol/#Web-specific_parameters) to all web events by default. -## Implementing Snowplow JS (Frontend) tracking +## Implement Snowplow JS (Frontend) tracking GitLab provides `Tracking`, an interface that wraps the [Snowplow JavaScript Tracker](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/javascript-trackers/) for tracking custom events. The simplest way to use it is to add `data-` attributes to clickable elements and dropdowns. There is also a Vue mixin (exposing a `track` method), and the static method `Tracking.event`. Each of these requires at minimum a `category` and an `action`. You can provide additional [Structured event taxonomy](#structured-event-taxonomy) properties along with an `extra` object that accepts key-value pairs. @@ -174,7 +174,7 @@ GitLab provides `Tracking`, an interface that wraps the [Snowplow JavaScript Tra ### Tracking with data attributes -When working within HAML (or Vue templates) we can add `data-track-*` attributes to elements of interest. All elements that have a `data-track-action` attribute automatically have event tracking bound on clicks. You can provide extra data as a valid JSON string using `data-track-extra`. +When working in HAML (or Vue templates) we can add `data-track-*` attributes to elements of interest. All elements that have a `data-track-action` attribute automatically have event tracking bound on clicks. You can provide extra data as a valid JSON string using `data-track-extra`. Below is an example of `data-track-*` attributes assigned to a button: @@ -191,7 +191,7 @@ Below is an example of `data-track-*` attributes assigned to a button: /> ``` -Event listeners are bound at the document level to handle click events on or within elements with these data attributes. This allows them to be properly handled on re-rendering and changes to the DOM. Note that because of the way these events are bound, click events should not be stopped from propagating up the DOM tree. If for any reason click events are being stopped from propagating, you need to implement your own listeners and follow the instructions in [Tracking within Vue components](#tracking-within-vue-components) or [Tracking in raw JavaScript](#tracking-in-raw-javascript). +Event listeners are bound at the document level to handle click events on or within elements with these data attributes. This allows them to be properly handled on re-rendering and changes to the DOM. Note that because of the way these events are bound, click events should not be stopped from propagating up the DOM tree. If click events are being stopped from propagating, you must implement your own listeners and follow the instructions in [Tracking within Vue components](#tracking-within-vue-components) or [Tracking in raw JavaScript](#tracking-in-raw-javascript). Below is a list of supported `data-track-*` attributes: @@ -237,7 +237,7 @@ import Tracking from '~/tracking'; const trackingMixin = Tracking.mixin({ label: 'right_sidebar' }); ``` -You can provide default options that are passed along whenever an event is tracked from within your component. For instance, if all events within a component should be tracked with a given `label`, you can provide one at this time. Available defaults are `category`, `label`, `property`, and `value`. If no category is specified, `document.body.dataset.page` is used as the default. +You can provide default options that are passed along whenever an event is tracked from within your component. For example, if all events in a component should be tracked with a given `label`, you can provide one at this time. Available defaults are `category`, `label`, `property`, and `value`. If no category is specified, `document.body.dataset.page` is used as the default. You can then use the mixin normally in your component with the `mixin` Vue declaration. The mixin also provides the ability to specify tracking options in `data` or `computed`. These override any defaults and allow the values to be dynamic from props, or based on state. @@ -256,7 +256,7 @@ export default { }; ``` -The mixin provides a `track` method that can be called within the template, +The mixin provides a `track` method that can be called from within the template, or from component methods. An example of the whole implementation might look like this: ```javascript @@ -302,7 +302,7 @@ export default { ``` The event data can be provided directly in the `track` function as well. -This object will merge with any previously provided options. +This object merges with any previously provided options. ```javascript this.track('click_button', { @@ -404,7 +404,7 @@ describe('MyTracking', () => { ### Form tracking -You can enable Snowplow automatic [form tracking](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/javascript-trackers/javascript-tracker/javascript-tracker-v2/tracking-specific-events/#form-tracking) by calling `Tracking.enableFormTracking` (after the DOM is ready) and providing a `config` object that includes at least one of the following elements: +Enable Snowplow automatic [form tracking](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/javascript-trackers/javascript-tracker/javascript-tracker-v2/tracking-specific-events/#form-tracking) by calling `Tracking.enableFormTracking` (after the DOM is ready) and providing a `config` object that includes at least one of the following elements: - `forms`: determines which forms are tracked, and are identified by the CSS class name. - `fields`: determines which fields inside the tracked forms are tracked, and are identified by the field `name`. @@ -442,7 +442,7 @@ describe('MyFormTracking', () => { }); ``` -## Implementing Snowplow Ruby (Backend) tracking +## Implement Snowplow Ruby (Backend) tracking GitLab provides `Gitlab::Tracking`, an interface that wraps the [Snowplow Ruby Tracker](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/ruby-tracker/) for tracking custom events. @@ -483,11 +483,11 @@ https://docs.gitlab.com/ee/development/testing_guide/best_practices.html#test-sn ### Performance -We use the [AsyncEmitter](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/ruby-tracker//emitters/#the-asyncemitter-class) when tracking events, which allows for instrumentation calls to be run in a background thread. This is still an active area of development. +We use the [AsyncEmitter](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/ruby-tracker/emitters/#the-asyncemitter-class) when tracking events, which allows for instrumentation calls to be run in a background thread. This is still an active area of development. -## Developing and testing Snowplow +## Develop and test Snowplow -There are several tools for developing and testing Snowplow Event +There are several tools for developing and testing a Snowplow event. | Testing Tool | Frontend Tracking | Backend Tracking | Local Development Environment | Production Environment | Production Environment | |----------------------------------------------|--------------------|---------------------|-------------------------------|------------------------|------------------------| @@ -510,7 +510,7 @@ To test frontend events in development: #### Snowplow Analytics Debugger Chrome Extension -Snowplow Analytics Debugger is a browser extension for testing frontend events. This works on production, staging and local development environments. +Snowplow Analytics Debugger is a browser extension for testing frontend events. This works on production, staging, and local development environments. 1. Install the [Snowplow Analytics Debugger](https://chrome.google.com/webstore/detail/snowplow-analytics-debugg/jbnlcgeengmijcghameodeaenefieedm) Chrome browser extension. 1. Open Chrome DevTools to the Snowplow Analytics Debugger tab. @@ -528,7 +528,7 @@ Snowplow Inspector Chrome Extension is a browser extension for testing frontend Snowplow Micro is a very small version of a full Snowplow data collection pipeline: small enough that it can be launched by a test suite. Events can be recorded into Snowplow Micro just as they can a full Snowplow pipeline. Micro then exposes an API that can be queried. -Snowplow Micro is a Docker-based solution for testing frontend and backend events in a local development environment. You need to modify GDK using the instructions below to set this up. +Snowplow Micro is a Docker-based solution for testing frontend and backend events in a local development environment. You must modify GDK using the instructions below to set this up. - Read [Introducing Snowplow Micro](https://snowplowanalytics.com/blog/2019/07/17/introducing-snowplow-micro/) - Look at the [Snowplow Micro repository](https://github.com/snowplow-incubator/snowplow-micro) @@ -544,10 +544,15 @@ Snowplow Micro is a Docker-based solution for testing frontend and backend event ./snowplow-micro.sh ``` -1. Update your instance's settings to enable Snowplow events and point to the Snowplow Micro collector: +1. Use GDK to start the PostgreSQL terminal and connect to the `gitlabhq_development` database: ```shell gdk psql -d gitlabhq_development + ``` + +1. Update your instance's settings to enable Snowplow events and point to the Snowplow Micro collector: + + ```shell update application_settings set snowplow_collector_hostname='localhost:9090', snowplow_enabled=true, snowplow_cookie_domain='.gitlab.com'; ``` @@ -568,14 +573,14 @@ Snowplow Micro is a Docker-based solution for testing frontend and backend event formTracking: false, ``` -1. Update `snowplow_options` in `lib/gitlab/tracking.rb` to add `protocol` and `port`: +1. Update `options` in `lib/gitlab/tracking.rb` to add `protocol` and `port`: ```diff diff --git a/lib/gitlab/tracking.rb b/lib/gitlab/tracking.rb index 618e359211b..e9084623c43 100644 --- a/lib/gitlab/tracking.rb +++ b/lib/gitlab/tracking.rb - @@ -41,7 +41,9 @@ def snowplow_options(group) + @@ -41,7 +41,9 @@ def options(group) cookie_domain: Gitlab::CurrentSettings.snowplow_cookie_domain, app_id: Gitlab::CurrentSettings.snowplow_app_id, form_tracking: additional_features, @@ -609,7 +614,7 @@ Snowplow Micro is a Docker-based solution for testing frontend and backend event 1. Restart GDK: ```shell - `gdk restart` + gdk restart ``` 1. Send a test Snowplow event from the Rails console: diff --git a/doc/development/snowplow/review_guidelines.md b/doc/development/snowplow/review_guidelines.md index 285fbc3b44b..8edcbf06a0e 100644 --- a/doc/development/snowplow/review_guidelines.md +++ b/doc/development/snowplow/review_guidelines.md @@ -26,7 +26,7 @@ events or touches Snowplow related files. #### The merge request **author** should - For frontend events, when relevant, add a screenshot of the event in - the [testing tool](../snowplow/index.md#developing-and-testing-snowplow) used. + the [testing tool](../snowplow/index.md#develop-and-test-snowplow) used. - For backend events, when relevant, add the output of the [Snowplow Micro](index.md#snowplow-mini) good events `GET http://localhost:9090/micro/good` (it might be a good idea @@ -39,5 +39,5 @@ events or touches Snowplow related files. - Check the [usage recommendations](../snowplow/index.md#usage-recommendations). - Check that the [Event Dictionary](event_dictionary_guide.md) is up-to-date. - If needed, check that the events are firing locally using one of the -[testing tools](../snowplow/index.md#developing-and-testing-snowplow) available. +[testing tools](../snowplow/index.md#develop-and-test-snowplow) available. - Approve the MR, and relabel the MR with `~"product intelligence::approved"`. diff --git a/doc/development/sql.md b/doc/development/sql.md index 40ee19c0b9e..3483305c113 100644 --- a/doc/development/sql.md +++ b/doc/development/sql.md @@ -102,7 +102,7 @@ transaction. Transactions for migrations can be disabled using the following pattern: ```ruby -class MigrationName < ActiveRecord::Migration[4.2] +class MigrationName < Gitlab::Database::Migration[1.0] disable_ddl_transaction! end ``` @@ -110,7 +110,7 @@ end For example: ```ruby -class AddUsersLowerUsernameEmailIndexes < ActiveRecord::Migration[4.2] +class AddUsersLowerUsernameEmailIndexes < Gitlab::Database::Migration[1.0] disable_ddl_transaction! def up diff --git a/doc/development/stage_group_dashboards.md b/doc/development/stage_group_dashboards.md index 61a98ece892..7c518e9b6ca 100644 --- a/doc/development/stage_group_dashboards.md +++ b/doc/development/stage_group_dashboards.md @@ -56,7 +56,7 @@ component can have 2 indicators: [Git](https://gitlab.com/gitlab-com/runbooks/-/blob/f22f40b2c2eab37d85e23ccac45e658b2c914445/metrics-catalog/services/git.jsonnet#L216), and [Web](https://gitlab.com/gitlab-com/runbooks/-/blob/f22f40b2c2eab37d85e23ccac45e658b2c914445/metrics-catalog/services/web.jsonnet#L154) - services, that threshold is **1 second**. + services, that threshold is **5 seconds**. For Sidekiq job execution, the threshold depends on the [job urgency](sidekiq_style_guide.md#job-urgency). It is diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index ba7312b760f..79664490368 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -54,7 +54,7 @@ When using spring and guard together, use `SPRING=1 bundle exec guard` instead t > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/47767) in GitLab 13.7. -We've enabled [deprecation warnings](https://ruby-doc.org/core-2.7.2/Warning.html) +We've enabled [deprecation warnings](https://ruby-doc.org/core-2.7.4/Warning.html) by default when running specs. Making these warnings more visible to developers helps upgrading to newer Ruby versions. @@ -367,26 +367,34 @@ If needed, you can scope interactions within a specific area of the page by usin As you will likely be scoping to an element such as a `div`, which typically does not have a label, you may use a `data-testid` selector in this case. +##### Externalized contents + +Test expectations against externalized contents should call the same +externalizing method to match the translation. For example, you should use the `_` +method in Ruby and `__` method in JavaScript. + +See [Internationalization for GitLab - Test files](../i18n/externalization.md#test-files) for details. + ##### Actions Where possible, use more specific [actions](https://rubydoc.info/github/teamcapybara/capybara/master/Capybara/Node/Actions), such as the ones below. ```ruby # good -click_button 'Submit review' +click_button _('Submit review') -click_link 'UI testing docs' +click_link _('UI testing docs') -fill_in 'Search projects', with: 'gitlab' # fill in text input with text +fill_in _('Search projects'), with: 'gitlab' # fill in text input with text -select 'Last updated', from: 'Sort by' # select an option from a select input +select _('Last updated'), from: 'Sort by' # select an option from a select input -check 'Checkbox label' -uncheck 'Checkbox label' +check _('Checkbox label') +uncheck _('Checkbox label') -choose 'Radio input label' +choose _('Radio input label') -attach_file('Attach a file', '/path/to/file.png') +attach_file(_('Attach a file'), '/path/to/file.png') # bad - interactive elements must have accessible names, so # we should be able to use one of the specific actions above @@ -403,17 +411,17 @@ Where possible, use more specific [finders](https://rubydoc.info/github/teamcapy ```ruby # good -find_button 'Submit review' -find_button 'Submit review', disabled: true +find_button _('Submit review') +find_button _('Submit review'), disabled: true -find_link 'UI testing docs' -find_link 'UI testing docs', href: docs_url +find_link _('UI testing docs') +find_link _('UI testing docs'), href: docs_url -find_field 'Search projects' -find_field 'Search projects', with: 'gitlab' # find the input field with text -find_field 'Search projects', disabled: true -find_field 'Checkbox label', checked: true -find_field 'Checkbox label', unchecked: true +find_field _('Search projects') +find_field _('Search projects'), with: 'gitlab' # find the input field with text +find_field _('Search projects'), disabled: true +find_field _('Checkbox label'), checked: true +find_field _('Checkbox label'), unchecked: true # acceptable when finding a element that is not a button, link, or field find('[data-testid="element"]') @@ -425,31 +433,31 @@ Where possible, use more specific [matchers](https://rubydoc.info/github/teamcap ```ruby # good -expect(page).to have_button 'Submit review' -expect(page).to have_button 'Submit review', disabled: true -expect(page).to have_button 'Notifications', class: 'is-checked' # assert the "Notifications" GlToggle is checked +expect(page).to have_button _('Submit review') +expect(page).to have_button _('Submit review'), disabled: true +expect(page).to have_button _('Notifications'), class: 'is-checked' # assert the "Notifications" GlToggle is checked -expect(page).to have_link 'UI testing docs' -expect(page).to have_link 'UI testing docs', href: docs_url # assert the link has an href +expect(page).to have_link _('UI testing docs') +expect(page).to have_link _('UI testing docs'), href: docs_url # assert the link has an href -expect(page).to have_field 'Search projects' -expect(page).to have_field 'Search projects', disabled: true -expect(page).to have_field 'Search projects', with: 'gitlab' # assert the input field has text +expect(page).to have_field _('Search projects') +expect(page).to have_field _('Search projects'), disabled: true +expect(page).to have_field _('Search projects'), with: 'gitlab' # assert the input field has text -expect(page).to have_checked_field 'Checkbox label' -expect(page).to have_unchecked_field 'Radio input label' +expect(page).to have_checked_field _('Checkbox label') +expect(page).to have_unchecked_field _('Radio input label') -expect(page).to have_select 'Sort by' -expect(page).to have_select 'Sort by', selected: 'Last updated' # assert the option is selected -expect(page).to have_select 'Sort by', options: ['Last updated', 'Created date', 'Due date'] # assert an exact list of options -expect(page).to have_select 'Sort by', with_options: ['Created date', 'Due date'] # assert a partial list of options +expect(page).to have_select _('Sort by') +expect(page).to have_select _('Sort by'), selected: 'Last updated' # assert the option is selected +expect(page).to have_select _('Sort by'), options: ['Last updated', 'Created date', 'Due date'] # assert an exact list of options +expect(page).to have_select _('Sort by'), with_options: ['Created date', 'Due date'] # assert a partial list of options -expect(page).to have_text 'Some paragraph text.' -expect(page).to have_text 'Some paragraph text.', exact: true # assert exact match +expect(page).to have_text _('Some paragraph text.') +expect(page).to have_text _('Some paragraph text.'), exact: true # assert exact match expect(page).to have_current_path 'gitlab/gitlab-test/-/issues' -expect(page).to have_title 'Not Found' +expect(page).to have_title _('Not Found') # acceptable when a more specific matcher above is not possible expect(page).to have_css 'h2', text: 'Issue title' @@ -653,6 +661,12 @@ let_it_be_with_refind(:project) { create(:project) } let_it_be(:project, refind: true) { create(:project) } ``` +Note that `let_it_be` cannot be used with factories that has stubs, such as `allow`. +The reason is that `let_it_be` happens in a `before(:all)` block, and RSpec does not +allow stubs in `before(:all)`. +See this [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/340487) for more details. +To resolve, use `let`, or change the factory to not use stubs. + ### Time-sensitive tests [`ActiveSupport::Testing::TimeHelpers`](https://api.rubyonrails.org/v6.0.3.1/classes/ActiveSupport/Testing/TimeHelpers.html) @@ -1197,6 +1211,8 @@ GitLab uses [factory_bot](https://github.com/thoughtbot/factory_bot) as a test f - Factories don't have to be limited to `ActiveRecord` objects. [See example](https://gitlab.com/gitlab-org/gitlab-foss/commit/0b8cefd3b2385a21cfed779bd659978c0402766d). - Factories and their traits should produce valid objects that are [verified by specs](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/factories_spec.rb). +- Avoid the use of [`skip_callback`](https://api.rubyonrails.org/classes/ActiveSupport/Callbacks/ClassMethods.html#method-i-skip_callback) in factories. + See [issue #247865](https://gitlab.com/gitlab-org/gitlab/-/issues/247865) for details. ### Fixtures diff --git a/doc/development/testing_guide/end_to_end/beginners_guide.md b/doc/development/testing_guide/end_to_end/beginners_guide.md index e0f0e9e7089..7370cc5771b 100644 --- a/doc/development/testing_guide/end_to_end/beginners_guide.md +++ b/doc/development/testing_guide/end_to_end/beginners_guide.md @@ -349,8 +349,8 @@ GITLAB_PASSWORD=<GDK root password> bundle exec bin/qa Test::Instance::All http: Where `<test_file>` is: -- `qa/specs/features/browser_ui/1_manage/login/login_spec.rb` when running the Login example. -- `qa/specs/features/browser_ui/2_plan/issues/issue_spec.rb` when running the Issue example. +- `qa/specs/features/browser_ui/1_manage/login/log_in_spec.rb` when running the Login example. +- `qa/specs/features/browser_ui/2_plan/issues/create_issue_spec.rb` when running the Issue example. ## End-to-end test merge request template diff --git a/doc/development/testing_guide/end_to_end/best_practices.md b/doc/development/testing_guide/end_to_end/best_practices.md index 74c02d19d0a..a3caa8bf2b3 100644 --- a/doc/development/testing_guide/end_to_end/best_practices.md +++ b/doc/development/testing_guide/end_to_end/best_practices.md @@ -8,27 +8,33 @@ info: To determine the technical writer assigned to the Stage/Group associated w This is a tailored extension of the Best Practices [found in the testing guide](../best_practices.md). -## Link a test to its test-case issue +## Class and module naming -Every test should have a corresponding issue in the [Quality Test Cases project](https://gitlab.com/gitlab-org/quality/testcases/). -It's recommended that you reuse the issue created to plan the test. If one does not already exist you -can create the issue yourself. Alternatively, you can run the test in a pipeline that has reporting -enabled and the test-case issue reporter will automatically create a new issue. +The QA framework uses [Zeitwerk](https://github.com/fxn/zeitwerk) for class and module autoloading. The default Zeitwerk [inflector](https://github.com/fxn/zeitwerk#zeitwerkinflector) simply converts snake_cased file names to PascalCased module or class names. It is advised to stick to this pattern to avoid manual maintenance of inflections. -Whether you create a new test-case issue or one is created automatically, you will need to manually add -a `testcase` RSpec metadata tag. In most cases, a single test will be associated with a single test-case -issue ([see below for exceptions](#exceptions)). +In case custom inflection logic is needed, custom inflectors are added in the [qa.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa.rb) file in the `loader.inflector.inflect` method invocation. + +## Link a test to its test case + +Every test should have a corresponding test case as well as a results issue in the [Quality Test Cases project](https://gitlab.com/gitlab-org/quality/testcases/). +It's recommended that you reuse the issue created to plan the test as the results issue. If a test case or results issue does not already exist you +can create them yourself. Alternatively, you can run the test in a pipeline that has reporting +enabled and the test-case reporter will automatically create a new test case and/or results issue and link the results issue to it's corresponding test case. + +Whether you create a new test case or one is created automatically, you will need to manually add +a `testcase` RSpec metadata tag. In most cases, a single test will be associated with a single test case + ([see below for exceptions](#exceptions)). For example: ```ruby RSpec.describe 'Stage' do describe 'General description of the feature under test' do - it 'test name', testcase: 'https://gitlab.com/gitlab-org/quality/testcases/-/issues/:issue_id' do + it 'test name', testcase: 'https://gitlab.com/gitlab-org/quality/testcases/-/quality/test_cases/:test_case_id' do ... end - it 'another test', testcase: 'https://gitlab.com/gitlab-org/quality/testcases/-/issues/:another_issue_id' do + it 'another test', testcase: 'https://gitlab.com/gitlab-org/quality/testcases/-/quality/test_cases/:another_test_case_id' do ... end end @@ -38,10 +44,10 @@ end ### Exceptions Most tests are defined by a single line of a `spec` file, which is why those tests can be linked to a -single test-case issue via the `testcase` tag. +single test case via the `testcase` tag. -However, some tests don't have a one-to-one relationship between a line of a `spec` file and a test-case -issue. This is because some tests are defined in a way that means a single line is associated with +However, some tests don't have a one-to-one relationship between a line of a `spec` file and a test case. +This is because some tests are defined in a way that means a single line is associated with multiple tests, including: - Parallelized tests. @@ -49,13 +55,13 @@ multiple tests, including: - Tests in shared examples that include more than one example. In those and similar cases we can't assign a single `testcase` tag and so we rely on the test-case -reporter to programmatically determine the correct test-case issue based on the name and description of -the test. In such cases, the test-case reporter will automatically create a test-case issue the first time -the test runs, if no issue exists already. +reporter to programmatically determine the correct test case based on the name and description of +the test. In such cases, the test-case reporter will automatically create a test case and/or results issue +the first time the test runs, if none exist already. -In such a case, if you create the issue yourself or want to reuse an existing issue, +In such a case, if you create the test case or results issue yourself or want to reuse an existing issue, you must use this [end-to-end test issue template](https://gitlab.com/gitlab-org/quality/testcases/-/blob/master/.gitlab/issue_templates/End-to-end%20Test.md) -to format the issue description. +to format the issue description. (Note you must copy/paste this for test cases as templates aren't currently available.) To illustrate, there are two tests in the shared examples in [`qa/specs/features/ee/browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/47b17db82c38ab704a23b5ba5d296ea0c6a732c8/qa/qa/specs/features/ee/browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb): @@ -84,9 +90,9 @@ RSpec.describe 'Create' do end ``` -There would be two associated test-case issues, one for each shared example, with the following content: +There would be two associated test cases, one for each shared example, with the following content: -[Test 1](https://gitlab.com/gitlab-org/quality/testcases/-/issues/600): +[Test 1 Test Case](https://gitlab.com/gitlab-org/quality/testcases/-/quality/test_cases/1491): ````markdown ```markdown @@ -105,10 +111,66 @@ pushes and merges ./qa/specs/features/ee/browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb +### DO NOT EDIT BELOW THIS LINE + +Active and historical test results: + +https://gitlab.com/gitlab-org/quality/testcases/-/issues/600 + +``` +```` + +[Test 1 Results Issue](https://gitlab.com/gitlab-org/quality/testcases/-/issues/600): + +````markdown +```markdown +Title: browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb | Create Restricted +protected branch push and merge when only one user is allowed to merge and push to a protected +branch behaves like only user with access pushes and merges selecte... + +Description: +### Full description + +Create Restricted protected branch push and merge when only one user is allowed to merge and push +to a protected branch behaves like only user with access pushes and merges selected developer user +pushes and merges + +### File path + +./qa/specs/features/ee/browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb + +``` +```` + +[Test 2 Test Case](https://gitlab.com/gitlab-org/quality/testcases/-/quality/test_cases/602): + +````markdown +```markdown +Title: browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb | Create Restricted +protected branch push and merge when only one user is allowed to merge and push to a protected +branch behaves like only user with access pushes and merges unselec... + +Description: +### Full description + +Create Restricted protected branch push and merge when only one user is allowed to merge and push +to a protected branch behaves like only user with access pushes and merges unselected maintainer +user fails to push + +### File path + +./qa/specs/features/ee/browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb + +### DO NOT EDIT BELOW THIS LINE + +Active and historical test results: + +https://gitlab.com/gitlab-org/quality/testcases/-/issues/602 + ``` ```` -[Test 2](https://gitlab.com/gitlab-org/quality/testcases/-/issues/602): +[Test 2 Results Issue](https://gitlab.com/gitlab-org/quality/testcases/-/issues/602): ````markdown ```markdown @@ -342,7 +404,7 @@ end When something requires waiting to be matched, use `eventually_` matchers with clear wait duration definition. -`Eventually` matchers use the following naming pattern: `eventually_${rspec_matcher_name}`. They are defined in [eventually_matcher.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/spec/support/matchers/eventually_matcher.rb). +`Eventually` matchers use the following naming pattern: `eventually_${rspec_matcher_name}`. They are defined in [eventually_matcher.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/support/matchers/eventually_matcher.rb). ```ruby expect { async_value }.to eventually_eq(value).within(max_duration: 120, max_attempts: 60, reload_page: page) diff --git a/doc/development/testing_guide/end_to_end/index.md b/doc/development/testing_guide/end_to_end/index.md index f4b01c64385..36c0f5adf00 100644 --- a/doc/development/testing_guide/end_to_end/index.md +++ b/doc/development/testing_guide/end_to_end/index.md @@ -170,6 +170,45 @@ Helm chart](https://gitlab.com/gitlab-org/charts/gitlab/), itself deployed with See [Review Apps](../review_apps.md) for more details about Review Apps. +## Test reports + +### Allure report + +For additional test results visibility, tests that run on pipelines generate +and host [Allure](https://github.com/allure-framework/allure2) test reports. + +The `QA` framework is using the [Allure RSpec](https://github.com/allure-framework/allure-ruby/blob/master/allure-rspec/README.md) +gem to generate source files for the `Allure` test report. An additional job +in the pipeline: + +- Fetches these source files from all test jobs. +- Generates and uploads the report to the `GCS` bucket `gitlab-qa-allure-report` under the project `gitlab-qa-resources`. + +A common CI template for report uploading is stored in +[`allure-report.yml`](https://gitlab.com/gitlab-org/quality/pipeline-common/-/blob/master/ci/allure-report.yml). + +#### Merge requests + +When these tests are executed in the scope of merge requests, the `Allure` report is +uploaded to the `GCS` bucket and comment is added linking to their respective reports. + +#### Scheduled pipelines + +Scheduled pipelines for these tests contain a `generate-allure-report` job under the `Report` stage. They also output +a link to the current test report. + +#### Static report links + +Each type of scheduled pipeline generates a static link for the latest test report according to its stage: + +- [`master`](https://storage.googleapis.com/gitlab-qa-allure-reports/package-and-qa/master/index.html) +- [`staging-full`](https://storage.googleapis.com/gitlab-qa-allure-reports/staging-full/master/index.html) +- [`staging-sanity`](https://storage.googleapis.com/gitlab-qa-allure-reports/staging-sanity/master/index.html) +- [`staging-sanity-no-admin`](https://storage.googleapis.com/gitlab-qa-allure-reports/staging-sanity-no-admin/master/index.html) +- [`canary-sanity`](https://storage.googleapis.com/gitlab-qa-allure-reports/canary-sanity/master/index.html) +- [`production`](https://storage.googleapis.com/gitlab-qa-allure-reports/production/master/index.html) +- [`production-sanity`](https://storage.googleapis.com/gitlab-qa-allure-reports/production-sanity/master/index.html) + ## How do I run the tests? If you are not [testing code in a merge request](#testing-code-in-merge-requests), diff --git a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md index 3a016c0e95c..b6e92367f89 100644 --- a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md +++ b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md @@ -44,3 +44,4 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec | `:smtp` | The test requires a GitLab instance to be configured to use an SMTP server. Tests SMTP notification email delivery from GitLab by using MailHog. | | `:testcase` | The link to the test case issue in the [Quality Test Cases project](https://gitlab.com/gitlab-org/quality/testcases/). | | `:transient` | The test tests transient bugs. It is excluded by default. | +| `:issue`, `:issue_${num}` | Optional links to issues which might be related to the spec. Helps keeping track of related issues and can also be used by tools that create test reports. Currently added automatically to `Allure` test report. Multiple tags can be used by adding optional number postfix like `issue_1`, `issue_2` etc. | diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md index 3af806d8f57..76687db3a3f 100644 --- a/doc/development/testing_guide/frontend_testing.md +++ b/doc/development/testing_guide/frontend_testing.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Frontend testing standards and style guidelines There are two types of test suites encountered while developing frontend code -at GitLab. We use Karma with Jasmine and Jest for JavaScript unit and integration testing, +at GitLab. We use Jest for JavaScript unit and integration testing, and RSpec feature tests with Capybara for e2e (end-to-end) integration testing. Unit and feature tests need to be written for all new features. @@ -28,46 +28,9 @@ If you are looking for a guide on Vue component testing, you can jump right away We use Jest to write frontend unit and integration tests. Jest tests can be found in `/spec/frontend` and `/ee/spec/frontend` in EE. -## Karma test suite - -While GitLab has switched over to [Jest](https://jestjs.io), Karma tests still exist in our -application because some of our specs require a browser and can't be easily migrated to Jest. -Those specs intend to eventually drop Karma in favor of either Jest or RSpec. You can track this migration -in the [related epic](https://gitlab.com/groups/gitlab-org/-/epics/4900). - -[Karma](http://karma-runner.github.io/) is a test runner which uses -[Jasmine](https://jasmine.github.io/) as its test framework. Jest also uses Jasmine as foundation, -that's why it's looking quite similar. - -Karma tests live in `spec/javascripts/` and `/ee/spec/javascripts` in EE. - -`app/assets/javascripts/behaviors/autosize.js` -might have a corresponding `spec/javascripts/behaviors/autosize_spec.js` file. - -Keep in mind that in a CI environment, these tests are run in a headless -browser and you don't have access to certain APIs, such as -[`Notification`](https://developer.mozilla.org/en-US/docs/Web/API/notification), -which have to be stubbed. - -### Differences to Karma - -- Jest runs in a Node.js environment, not in a browser. [An issue exists](https://gitlab.com/gitlab-org/gitlab/-/issues/26982) for running Jest tests in a browser. -- Because Jest runs in a Node.js environment, it uses [jsdom](https://github.com/jsdom/jsdom) by default. See also its [limitations](#limitations-of-jsdom) below. -- Jest does not have access to Webpack loaders or aliases. - The aliases used by Jest are defined in its [own configuration](https://gitlab.com/gitlab-org/gitlab/-/blob/master/jest.config.js). -- All calls to `setTimeout` and `setInterval` are mocked away. See also [Jest Timer Mocks](https://jestjs.io/docs/timer-mocks). -- `rewire` is not required because Jest supports mocking modules. See also [Manual Mocks](https://jestjs.io/docs/manual-mocks). -- No [context object](https://jasmine.github.io/tutorials/your_first_suite#section-The_%3Ccode%3Ethis%3C/code%3E_keyword) is passed to tests in Jest. - This means sharing `this.something` between `beforeEach()` and `it()` for example does not work. - Instead you should declare shared variables in the context that they are needed (via `const` / `let`). -- The following cause tests to fail in Jest: - - Unmocked requests. - - Unhandled Promise rejections. - - Calls to `console.warn`, including warnings from libraries like Vue. - ### Limitations of jsdom -As mentioned [above](#differences-to-karma), Jest uses jsdom instead of a browser for running tests. +Jest uses jsdom instead of a browser for running tests. This comes with a number of limitations, namely: - [No scrolling support](https://github.com/jsdom/jsdom/blob/15.1.1/lib/jsdom/browser/Window.js#L623-L625) @@ -387,8 +350,7 @@ Sometimes we have to test time-sensitive code. For example, recurring events tha If the application itself is waiting for some time, mock await the waiting. In Jest this is already [done by default](https://gitlab.com/gitlab-org/gitlab/-/blob/a2128edfee799e49a8732bfa235e2c5e14949c68/jest.config.js#L47) -(see also [Jest Timer Mocks](https://jestjs.io/docs/timer-mocks)). In Karma you can use the -[Jasmine mock clock](https://jasmine.github.io/api/2.9/Clock.html). +(see also [Jest Timer Mocks](https://jestjs.io/docs/timer-mocks)). ```javascript const doSomethingLater = () => { @@ -409,20 +371,6 @@ it('does something', () => { }); ``` -**in Karma:** - -```javascript -it('does something', () => { - jasmine.clock().install(); - - doSomethingLater(); - jasmine.clock().tick(4000); - - expect(something).toBe('done'); - jasmine.clock().uninstall(); -}); -``` - ### Mocking the current location in Jest NOTE: @@ -476,8 +424,7 @@ it('passes', () => { Sometimes a test needs to wait for something to happen in the application before it continues. Avoid using [`setTimeout`](https://developer.mozilla.org/en-US/docs/Web/API/WindowOrWorkerGlobalScope/setTimeout) -because it makes the reason for waiting unclear and if used within Karma with a time larger than zero it slows down our test suite. -Instead use one of the following approaches. +because it makes the reason for waiting unclear. Instead use one of the following approaches. #### Promises and Ajax calls @@ -505,19 +452,6 @@ it('waits for an Ajax call', async () => { }); ``` -**in Karma:** - -```javascript -it('waits for an Ajax call', done => { - askTheServer() - .then(() => { - expect(something).toBe('done'); - }) - .then(done) - .catch(done.fail); -}); -``` - If you are not able to register handlers to the `Promise`, for example because it is executed in a synchronous Vue life cycle hook, take a look at the [waitFor](#wait-until-axios-requests-finish) helpers or you can flush all pending `Promise`s: **in Jest:** @@ -548,22 +482,6 @@ it('renders something', () => { }); ``` -**in Karma:** - -```javascript -it('renders something', done => { - wrapper.setProps({ value: 'new value' }); - - wrapper.vm - .$nextTick() - .then(() => { - expect(wrapper.text()).toBe('new value'); - }) - .then(done) - .catch(done.fail); -}); -``` - #### Events If the application triggers an event that you need to wait for in your test, register an event handler which contains @@ -776,8 +694,6 @@ TBU ### Stubbing and Mocking -Jasmine provides stubbing and mocking capabilities. There are some subtle differences in how to use it within Karma and Jest. - Stubs or spies are often used synonymously. In Jest it's quite easy thanks to the `.spyOn` method. [Official docs](https://jestjs.io/docs/jest-object#jestspyonobject-methodname) The more challenging part are mocks, which can be used for functions or even dependencies. @@ -835,7 +751,6 @@ For running the frontend tests, you need the following commands: - `rake frontend:fixtures` (re-)generates [fixtures](#frontend-test-fixtures). Make sure that fixtures are up-to-date before running tests that require them. - `yarn jest` runs Jest tests. -- `yarn karma` runs Karma tests. ### Live testing and focused testing -- Jest @@ -860,49 +775,36 @@ yarn jest ./path/to/folder/ yarn jest term ``` -### Live testing and focused testing -- Karma +## Frontend test fixtures -Karma allows something similar, but it's way more costly. +Frontend fixtures are files containing responses from backend controllers. These responses can be either HTML +generated from HAML templates or JSON payloads. Frontend tests that rely on these responses are +often using fixtures to validate correct integration with the backend code. -Running Karma with `yarn run karma-start` compiles the JavaScript -assets and runs a server at `http://localhost:9876/` where it automatically -runs the tests on any browser which connects to it. You can enter that URL on -multiple browsers at once to have it run the tests on each in parallel. +### Use fixtures -While Karma is running, any changes you make instantly trigger a recompile -and retest of the **entire test suite**, so you can see instantly if you've broken -a test with your changes. You can use [Jasmine focused](https://jasmine.github.io/2.5/focused_specs.html) or -excluded tests (with `fdescribe` or `xdescribe`) to get Karma to run only the -tests you want while you're working on a specific feature, but make sure to -remove these directives when you commit your code. +Jest uses `spec/frontend/__helpers__/fixtures.js` to import fixtures in tests. -It is also possible to only run Karma on specific folders or files by filtering -the run tests via the argument `--filter-spec` or short `-f`: +The following are examples of tests that work for Jest: -```shell -# Run all files -yarn karma-start -# Run specific spec files -yarn karma-start --filter-spec profile/account/components/update_username_spec.js -# Run specific spec folder -yarn karma-start --filter-spec profile/account/components/ -# Run all specs which path contain vue_shared or vie -yarn karma-start -f vue_shared -f vue_mr_widget -``` +```javascript +it('makes a request', () => { + const responseBody = getJSONFixture('some/fixture.json'); // loads spec/frontend/fixtures/some/fixture.json + axiosMock.onGet(endpoint).reply(200, responseBody); -You can also use glob syntax to match files. Remember to put quotes around the -glob otherwise your shell may split it into multiple arguments: + myButton.click(); -```shell -# Run all specs named `file_spec` within the IDE subdirectory -yarn karma -f 'spec/javascripts/ide/**/file_spec.js' -``` + // ... +}); -## Frontend test fixtures +it('uses some HTML element', () => { + loadFixtures('some/page.html'); // loads spec/frontend/fixtures/some/page.html and adds it to the DOM -Frontend fixtures are files containing responses from backend controllers. These responses can be either HTML -generated from HAML templates or JSON payloads. Frontend tests that rely on these responses are -often using fixtures to validate correct integration with the backend code. + const element = document.getElementById('#my-id'); + + // ... +}); +``` ### Generate fixtures @@ -961,34 +863,6 @@ This will create a new fixture located at You can import the JSON fixture in a Jest test using the `getJSONFixture` method [as described below](#use-fixtures). -### Use fixtures - -Jest and Karma test suites import fixtures in different ways: - -- The Karma test suite are served by [jasmine-jquery](https://github.com/velesin/jasmine-jquery). -- Jest use `spec/frontend/__helpers__/fixtures.js`. - -The following are examples of tests that work for both Karma and Jest: - -```javascript -it('makes a request', () => { - const responseBody = getJSONFixture('some/fixture.json'); // loads spec/frontend/fixtures/some/fixture.json - axiosMock.onGet(endpoint).reply(200, responseBody); - - myButton.click(); - - // ... -}); - -it('uses some HTML element', () => { - loadFixtures('some/page.html'); // loads spec/frontend/fixtures/some/page.html and adds it to the DOM - - const element = document.getElementById('#my-id'); - - // ... -}); -``` - ## Data-driven tests Similar to [RSpec's parameterized tests](best_practices.md#table-based--parameterized-tests), @@ -1139,13 +1013,10 @@ Main information on frontend testing levels can be found in the [Testing Levels Tests relevant for frontend development can be found at the following places: -- `spec/javascripts/`, for Karma tests - `spec/frontend/`, for Jest tests - `spec/features/`, for RSpec tests -RSpec runs complete [feature tests](testing_levels.md#frontend-feature-tests), while the Jest and Karma directories contain [frontend unit tests](testing_levels.md#frontend-unit-tests), [frontend component tests](testing_levels.md#frontend-component-tests), and [frontend integration tests](testing_levels.md#frontend-integration-tests). - -All tests in `spec/javascripts/` are intended to be migrated to `spec/frontend/` (see also [#52483](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/52483)). +RSpec runs complete [feature tests](testing_levels.md#frontend-feature-tests), while the Jest directories contain [frontend unit tests](testing_levels.md#frontend-unit-tests), [frontend component tests](testing_levels.md#frontend-component-tests), and [frontend integration tests](testing_levels.md#frontend-integration-tests). Before May 2018, `features/` also contained feature tests run by Spinach. These tests were removed from the codebase in May 2018 ([#23036](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/23036)). @@ -1161,6 +1032,16 @@ If you introduce new helpers, place them in that directory. We have a helper available to make testing actions easier, as per [official documentation](https://vuex.vuejs.org/guide/testing.html): ```javascript +// prefer using like this, a single object argument so parameters are obvious from reading the test +await testAction({ + action: actions.actionName, + payload: { deleteListId: 1 }, + state: { lists: [1, 2, 3] }, + expectedMutations: [ { type: types.MUTATION} ], + expectedActions: [], +}); + +// old way, don't do this for new tests testAction( actions.actionName, // action { }, // params to be passed to action @@ -1177,8 +1058,6 @@ testAction( ); ``` -Check an example in [`spec/frontend/ide/stores/actions_spec.js`](https://gitlab.com/gitlab-org/gitlab/-/blob/fdc7197609dfa7caeb1d962042a26248e49f27da/spec/frontend/ide/stores/actions_spec.js#L392). - ### Wait until Axios requests finish <!-- vale gitlab.Spelling = NO --> diff --git a/doc/development/testing_guide/index.md b/doc/development/testing_guide/index.md index 889dc45d6e3..015d8a92a4d 100644 --- a/doc/development/testing_guide/index.md +++ b/doc/development/testing_guide/index.md @@ -19,7 +19,7 @@ importance. GitLab is built on top of [Ruby on Rails](https://rubyonrails.org/), and we're using [RSpec](https://github.com/rspec/rspec-rails#feature-specs) for all the backend tests, with [Capybara](https://github.com/teamcapybara/capybara) for end-to-end integration testing. -On the frontend side, we're using [Jest](https://jestjs.io/) and [Karma](http://karma-runner.github.io/)/[Jasmine](https://jasmine.github.io/) for JavaScript unit and +On the frontend side, we're using [Jest](https://jestjs.io/) for JavaScript unit and integration testing. Following are two great articles that everyone should read to understand what @@ -40,7 +40,7 @@ system tests, parameterized tests etc. ## [Frontend testing standards and style guidelines](frontend_testing.md) -Everything you should know about how to write good Frontend tests: Karma, +Everything you should know about how to write good Frontend tests: Jest, testing promises, stubbing etc. ## [Flaky tests](flaky_tests.md) diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md index cf757aad870..72d63fd8194 100644 --- a/doc/development/testing_guide/review_apps.md +++ b/doc/development/testing_guide/review_apps.md @@ -175,7 +175,7 @@ For GitLab Team Members only. If you want to sign in to the review app, review the GitLab handbook information for the [shared 1Password account](https://about.gitlab.com/handbook/security/#1password-for-teams). - The default username is `root`. -- The password can be found in the 1Password secure note named `gitlab-{ce,ee} Review App's root password`. +- The password can be found in the 1Password login item named `GitLab EE Review App`. ### Enable a feature flag for my Review App diff --git a/doc/development/testing_guide/testing_levels.md b/doc/development/testing_guide/testing_levels.md index 3a4a28702c7..29cdfab713e 100644 --- a/doc/development/testing_guide/testing_levels.md +++ b/doc/development/testing_guide/testing_levels.md @@ -31,7 +31,7 @@ records should use stubs/doubles as much as possible. | Code path | Tests path | Testing engine | Notes | | --------- | ---------- | -------------- | ----- | -| `app/assets/javascripts/` | `spec/javascripts/`, `spec/frontend/` | Karma & Jest | More details in the [Frontend Testing guide](frontend_testing.md) section. | +| `app/assets/javascripts/` | `spec/frontend/` | Jest | More details in the [Frontend Testing guide](frontend_testing.md) section. | | `app/finders/` | `spec/finders/` | RSpec | | | `app/graphql/` | `spec/graphql/` | RSpec | | | `app/helpers/` | `spec/helpers/` | RSpec | | @@ -233,7 +233,7 @@ They're useful to test permissions, redirections, what view is rendered etc. | `app/controllers/` | `spec/requests/`, `spec/controllers` | RSpec | Request specs are preferred over legacy controller specs. | | `app/mailers/` | `spec/mailers/` | RSpec | | | `lib/api/` | `spec/requests/api/` | RSpec | | -| `app/assets/javascripts/` | `spec/javascripts/`, `spec/frontend/` | Karma & Jest | [More details below](#frontend-integration-tests) | +| `app/assets/javascripts/` | `spec/frontend/` | Jest | [More details below](#frontend-integration-tests) | ### Frontend integration tests @@ -322,13 +322,6 @@ controller.instance_variable_set(:@user, user) and use methods [deprecated in Rails 5](https://gitlab.com/gitlab-org/gitlab/-/issues/16260). -### About Karma - -Karma is both in the Unit tests and the Integration tests category. Karma provides an environment to -run JavaScript tests, so you can either run unit tests (e.g. test a single -JavaScript method), or integration tests (e.g. test a component that is composed -of multiple components). - ## White-box tests at the system level (formerly known as System / Feature tests) Formal definitions: @@ -551,7 +544,7 @@ and the basic idea is that the cost of a test includes: There are cases where the behavior you are testing is not worth the time spent running the full application, for example, if you are testing styling, animation, edge cases or small actions that don't involve the backend, -you should write an integration test using Jasmine. +you should write an integration test using [Frontend integration tests](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/frontend_integration/README.md). --- diff --git a/doc/development/transient/prevention-patterns.md b/doc/development/transient/prevention-patterns.md index 472b5756805..c517a6bcd54 100644 --- a/doc/development/transient/prevention-patterns.md +++ b/doc/development/transient/prevention-patterns.md @@ -97,7 +97,7 @@ by the server-side endpoint satisfies the API contract. #### Related reading [Debug it!](https://pragprog.com/titles/pbdp/debug-it/) explores techniques to diagnose -and fix non-determinstic bugs and write software that is easier to debug. +and fix non-deterministic bugs and write software that is easier to debug. ## Backend diff --git a/doc/development/understanding_explain_plans.md b/doc/development/understanding_explain_plans.md index c3fefd40171..c3dfeaa6b92 100644 --- a/doc/development/understanding_explain_plans.md +++ b/doc/development/understanding_explain_plans.md @@ -826,4 +826,4 @@ A more extensive guide on understanding query plans can be found in the [presentation](https://public.dalibo.com/exports/conferences/_archives/_2012/201211_explain/understanding_explain.pdf) from [Dalibo.org](https://www.dalibo.com/en/). -Depesz's blog also has a good [section](https://www.depesz.com/tag/unexplainable) dedicated to query plans. +Depesz's blog also has a good [section](https://www.depesz.com/tag/unexplainable/) dedicated to query plans. diff --git a/doc/development/work_items.md b/doc/development/work_items.md new file mode 100644 index 00000000000..d4a1073461a --- /dev/null +++ b/doc/development/work_items.md @@ -0,0 +1,196 @@ +--- +stage: Plan +group: Project Management +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- +# Work items and work item types + +## Challenges + +Issues have the potential to be a centralized hub for collaboration. +We need to accept the +fact that different issue types require different fields and different context, depending +on what job they are being used to accomplish. For example: + +- A bug needs to list steps to reproduce. +- An incident needs references to stack traces and other contextual information relevant only + to that incident. + +Instead of each object type diverging into a separate model, we can standardize on an underlying +common model that we can customize with the widgets (one or more attributes) it contains. + +Here are some problems with current issues usage and why we are looking into work items: + +- Using labels to show issue types is cumbersome and makes reporting views more complex. +- Issue types are one of the top two use cases of labels, so it makes sense to provide first class + support for them. +- Issues are starting to become cluttered as we add more capabilities to them, and they are not + perfect: + + - There is no consistent pattern for how to surface relationships to other objects. + - There is not a coherent interaction model across different types of issues because we use + labels for this. + - The various implementations of issue types lack flexibility and extensibility. + +- Epics, issues, requirements, and others all have similar but just subtle enough + differences in common interactions that the user needs to hold a complicated mental + model of how they each behave. +- Issues are not extensible enough to support all of the emerging jobs they need to facilitate. +- Codebase maintainability and feature development become bigger challenges as we grow the Issue type + beyond its core role of issue tracking into supporting the different work item types and handling + logic and structure differences. +- New functionality is typically implemented with first class objects that import behavior from issues via + shared concerns. This leads to duplicated effort and ultimately small differences between common interactions. This + leads to inconsistent UX. +- Codebase maintainability and feature development becomes a bigger challenges as we grow issues + beyond its core role of issue tracking into supporting the different types and subtle differences between them. + +## Work item and work item type terms + +Using the terms "issue" or "issuable" to reference the types of collaboration objects +(for example, issue, bug, feature, or epic) often creates confusion. To avoid confusion, we will use the term +work item type (WIT) when referring to the type of a collaboration object. +An instance of a WIT is a work item (WI). For example, `issue#123`, `bug#456`, `requirement#789`. + +### Migration strategy + +WI model will be built on top of the existing `Issue` model and we'll gradually migrate `Issue` +model code to the WI model. + +One way to approach it is: + +```ruby +class WorkItems::WorkItem < ApplicationRecord + self.table_name = 'issues' + + # ... all the current issue.rb code +end + +class Issue < WorkItems::WorkItem + # Do not add code to this class add to WorkItems:WorkItem +end +``` + +We already use the concept of WITs within `issues` table through `issue_type` +column. There are `issue`, `incident`, and `test_case` issue types. To extend this +so that in future we can allow users to define custom WITs, we will move the +`issue_type` to a separate table: `work_item_types`. The migration process of `issue_type` +to `work_item_types` will involve creating the set of WITs for all root-level groups. + +NOTE: +At first, defining a WIT will only be possible at the root-level group, which would then be inherited by sub-groups. +We will investigate the possibility of defining new WITs at sub-group levels at a later iteration. + +### Introducing work_item_types table + +For example, suppose there are three root-level groups with IDs: `11`, `12`, and `13`. Also, +assume the following base types: `issue: 0`, `incident: 1`, `test_case: 2`. + +The respective `work_item_types` records: + +| `group_id` | `base_type` | `title` | +| -------------- | ----------- | --------- | +| 11 | 0 | Issue | +| 11 | 1 | Incident | +| 11 | 2 | Test Case | +| 12 | 0 | Issue | +| 12 | 1 | Incident | +| 12 | 2 | Test Case | +| 13 | 0 | Issue | +| 13 | 1 | Incident | +| 13 | 2 | Test Case | + +What we will do to achieve this: + +1. Add a `work_item_type_id` column to the `issues` table. +1. Ensure we write to both `issues#issue_type` and `issues#work_item_type_id` columns for + new or updated issues. +1. Backfill the `work_item_type_id` column to point to the `work_item_types#id` corresponding + to issue's project root groups. For example: + + ```ruby + issue.project.root_group.work_item_types.where(base_type: issue.issue_type).first.id. + ``` + +1. After `issues#work_item_type_id` is populated, we can switch our queries from + using `issue_type` to using `work_item_type_id`. + +To introduce a new WIT there are two options: + +- Follow the first step of the above process. We will still need to run a migration + that adds a new WIT for all root-level groups to make the WIT available to + all users. Besides a long-running migration, we'll need to + insert several million records to `work_item_types`. This might be unwanted for users + that do not want or need additional WITs in their workflow. +- Create an opt-in flow, so that the record in `work_item_types` for specific root-level group + is created only when a customer opts in. However, this implies a lower discoverability + of the newly introduced work item type. + +### Work item type widgets + +All WITs will share the same pool of predefined widgets and will be customized by +which widgets are active on a specific WIT. Every attribute (column or association) +will become a widget with self-encapsulated functionality regardless of the WIT it belongs to. +Because any WIT can have any widget, we only need to define which widget is active for a +specific WIT. So, after switching the type of a specific work item, we display a different set +of widgets. + +### Widgets metadata + +In order to customize each WIT with corresponding active widgets we will need a data +structure to map each WIT to specific widgets. + +NOTE: +The exact structure of the WITs widgets metadata is still to be defined. + +### Custom work item types + +With the WIT widget metadata and the workflow around mapping WIT to specific +widgets, we will be able to expose custom WITs to the users. Users will be able +to create their own WITs and customize them with widgets from the predefined pool. + +### Custom widgets + +The end goal is to allow users to define custom widgets and use these custom +widgets on any WIT. But this is a much further iteration and requires additional +investigation to determine both data and application architecture to be used. + +## Migrate requirements and epics to work item types + +We'll migrate requirements and epics into work item types, with their own set +of widgets. To achieve that, we'll migrate data to the `issues` table, +and we'll keep current `requirements` and `epics` tables to be used as proxies for old references to ensure +backward compatibility with already existing references. + +### Migrate requirements to work item types + +Currently `Requirement` attributes are a subset of `Issue` attributes, so the migration +consists mainly of: + +- Data migration. +- Keeping backwards compatibility at API levels. +- Ensuring that old references continue to work. + +The migration to a different underlying data structure should be seamless to the end user. + +### Migrate epics to work item types + +`Epic` has some extra functionality that the `Issue` WIT does not currently have. +So, migrating epics to a work item type requires providing feature parity between the current `Epic` object and WITs. + +The main missing features are: + +- Get WIs to the group level. This is dependent on [Consolidate Groups and Projects](https://gitlab.com/gitlab-org/architecture/tasks/-/issues/7) + initiative. +- A hierarchy widget: the ability to structure work items into hierarchies. +- Inherited date widget. + +To avoid disrupting workflows for users who are already using epics, we will introduce a new WIT +called `Feature` that will provide feature parity with epics at the project-level. Having that combined with progress +on [Consolidate Groups and Projects](https://gitlab.com/gitlab-org/architecture/tasks/-/issues/7) front will help us +provide a smooth migration path of epics to WIT with minimal disruption to user workflow. + +## Work item, work item type, and widgets roadmap + +We will move towards work items, work item types, and custom widgets (CW) in an iterative process. +For a rough outline of the work ahead of us, see [epic 6033](https://gitlab.com/groups/gitlab-org/-/epics/6033). |
