diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2022-11-17 11:33:21 +0000 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2022-11-17 11:33:21 +0000 |
| commit | 7021455bd1ed7b125c55eb1b33c5a01f2bc55ee0 (patch) | |
| tree | 5bdc2229f5198d516781f8d24eace62fc7e589e9 /doc/development | |
| parent | 185b095e93520f96e9cfc31d9c3e69b498cdab7c (diff) | |
| download | gitlab-ce-7021455bd1ed7b125c55eb1b33c5a01f2bc55ee0.tar.gz | |
Add latest changes from gitlab-org/gitlab@15-6-stable-eev15.6.0-rc42
Diffstat (limited to 'doc/development')
136 files changed, 3470 insertions, 1914 deletions
diff --git a/doc/development/api_graphql_styleguide.md b/doc/development/api_graphql_styleguide.md index d3053a1ca5f..c19341a1404 100644 --- a/doc/development/api_graphql_styleguide.md +++ b/doc/development/api_graphql_styleguide.md @@ -1294,15 +1294,25 @@ class MyThingResolver < BaseResolver end ``` -The final thing that is needed is that every field that uses this resolver needs -to advertise the need for lookahead: +The `LooksAhead` concern also provides basic support for preloading associations based on nested GraphQL field +definitions. The [WorkItemsResolver](https://gitlab.com/gitlab-org/gitlab/-/blob/e824a7e39e08a83fb162db6851de147cf0bfe14a/app/graphql/resolvers/work_items_resolver.rb#L46) +is a good example for this. `nested_preloads` is another method you can define to return a hash, but unlike the +`preloads` method, the value for each hash key is another hash and not the list of associations to preload. So in +the previous example, you could override `nested_preloads` like this: ```ruby - # in ParentType - field :my_things, MyThingType.connection_type, null: true, - extras: [:lookahead], # Necessary - resolver: MyThingResolver, - description: 'My things.' +class MyThingResolver < BaseResolver + # ... + + def nested_preloads + { + root_field: { + nested_field1: :association_to_preload, + nested_field2: [:association1, :association2] + } + } + end +end ``` For an example of real world use, please @@ -1733,15 +1743,18 @@ there are no problems we need to inform the user of. #### Failure (relevant to the user) -An error that affects the **user** occurred. We refer to these as _mutation errors_. In -this case there is typically no `thing` to return: +An error that affects the **user** occurred. We refer to these as _mutation errors_. + +In a _create_ mutation there is typically no `thing` to return. + +In an _update_ mutation we return the current true state of `thing`. Developers may need to call `#reset` on the `thing` instance to ensure this happens. ```javascript { data: { doTheThing: { errors: ["you cannot touch the thing"], - thing: null + thing: { .. } } } } @@ -2165,32 +2178,44 @@ end ```ruby NameError: uninitialized constant Resolvers::GroupIssuesResolver + + or + + GraphQL::Pagination::Connections::ImplementationMissingError ``` + though you might see something different. + To fix this, we must create a new file that encapsulates the connection type, and then reference it using double quotes. This gives a delayed resolution, and the proper connection type. For example: + [app/graphql/resolvers/base_issues_resolver.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/graphql/resolvers/base_issues_resolver.rb) + originally contained the line + ```ruby - module Types - # rubocop: disable Graphql/AuthorizeTypes - class IssueConnectionType < CountableConnectionType - end - end + type Types::IssueType.connection_type, null: true + ``` + + Running the specs locally for this file caused the + `NameError: uninitialized constant Resolvers::GroupIssuesResolver` error. + + The fix was to create a new file, [app/graphql/types/issue_connection.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/graphql/types/issue_connection.rb) with the + line: - Types::IssueConnectionType.prepend_mod_with('Types::IssueConnectionType') + ```ruby + Types::IssueConnection = Types::IssueType.connection_type ``` - in [types/issue_connection_type.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/graphql/types/issue_connection_type.rb) - defines a new `Types::IssueConnectionType`, and is then referenced in - [app/graphql/resolvers/base_issues_resolver.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/graphql/resolvers/base_issues_resolver.rb) + and in [app/graphql/resolvers/base_issues_resolver.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/graphql/resolvers/base_issues_resolver.rb) + we use the line ```ruby type "Types::IssueConnection", null: true ``` Only use this style if you are having spec failures. This is not intended to be a new - pattern that we use. This issue may disappear after we've upgraded to `2.x`. + pattern that we use. This issue should disappear after we've upgraded to `2.x`. - There can be instances where a spec fails because the class is not loaded correctly. It relates to the diff --git a/doc/development/application_limits.md b/doc/development/application_limits.md index edf159a116a..b64d25ccf64 100644 --- a/doc/development/application_limits.md +++ b/doc/development/application_limits.md @@ -38,7 +38,7 @@ It's recommended to create two separate migration script files. desired limit using `create_or_update_plan_limit` migration helper, such as: ```ruby - class InsertProjectHooksPlanLimits < Gitlab::Database::Migration[1.0] + class InsertProjectHooksPlanLimits < Gitlab::Database::Migration[2.0] def up create_or_update_plan_limit('project_hooks', 'default', 0) create_or_update_plan_limit('project_hooks', 'free', 10) diff --git a/doc/development/audit_event_guide/index.md b/doc/development/audit_event_guide/index.md index 5c8938aa46a..dfa6d56b3b5 100644 --- a/doc/development/audit_event_guide/index.md +++ b/doc/development/audit_event_guide/index.md @@ -21,9 +21,9 @@ While any events could trigger an Audit Event, not all events should. In general - Not attributable to one specific user. - Not of specific interest to an administrator or owner persona. - Are tracking information for product feature adoption. -- Are covered in the direction page's discussion on [what is not planned](https://about.gitlab.com/direction/manage/compliance/audit-events/#what-is-not-planned-right-now). +- Are covered in the direction page's discussion on [what is not planned](https://about.gitlab.com/direction/govern/compliance/audit-events/#what-is-not-planned-right-now). -If you have any questions, please reach out to `@gitlab-org/manage/compliance` to see if an Audit Event, or some other approach, may be best for your event. +If you have any questions, please reach out to `@gitlab-org/govern/compliance` to see if an Audit Event, or some other approach, may be best for your event. ## Audit Event Schemas @@ -120,7 +120,7 @@ end Because every audit event is persisted to the database, consider the amount of data we expect to generate, and the rate of generation, for new audit events. For new audit events that will produce a lot of data in the database, consider adding a [streaming-only audit event](#event-streaming) instead. If you have questions about this, feel free to ping -`@gitlab-org/manage/compliance/backend` in an issue or merge request. +`@gitlab-org/govern/compliance/backend` in an issue or merge request. ## Audit Event instrumentation flows @@ -205,8 +205,10 @@ All new audit events must have a type definition stored in `config/audit_events/ To add a new audit event type: -1. Create a new file in `config/audit_events/types/` with the filename matching the name of the event type. For example, a definition for the event type triggered when a - user is added to a project might be stored in `config/audit_events/types/project_add_user.yml`. +1. Create the YAML definition. You can either: + - Use the `bin/audit-event-type` CLI to create the YAML definition automatically. + - Perform manual steps to create a new file in `config/audit_events/types/` with the filename matching the name of the event type. For example, + a definition for the event type triggered when a user is added to a project might be stored in `config/audit_events/types/project_add_user.yml`. 1. Add contents to the file that conform to the [schema](#schema) defined in `config/audit_events/types/type_schema.json`. 1. Ensure that all calls to `Gitlab::Audit::Auditor` use the `name` defined in your file. diff --git a/doc/development/backend/ruby_style_guide.md b/doc/development/backend/ruby_style_guide.md index 9b5a68e4292..1a1c0db49f7 100644 --- a/doc/development/backend/ruby_style_guide.md +++ b/doc/development/backend/ruby_style_guide.md @@ -7,31 +7,32 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Ruby style guide -This is a GitLab-specific style guide for Ruby code. +This is a GitLab-specific style guide for Ruby code. Everything documented in this page can be [reopened for discussion](https://about.gitlab.com/handbook/values/#disagree-commit-and-disagree). -Generally, if a style is not covered by [existing RuboCop rules or style guides](../contributing/style_guides.md#ruby-rails-rspec), it shouldn't be a blocker. -Before adding a new cop to enforce a given style, make sure to discuss it with your team. -When the style is approved by a backend EM or by a BE staff eng, add a new section to this page to -document the new rule. For every new guideline, add it in a new section and link the discussion from the section's -[version history note](../documentation/versions.md#add-a-version-history-item) -to provide context and serve as a reference. +We use [RuboCop](../rubocop_development_guide.md) to enforce Ruby style guide rules. -See also [guidelines for reusing abstractions](../reusing_abstractions.md). +Where a RuboCop rule is absent, refer to the following style guides as general guidelines to write idiomatic Ruby: -Everything listed here can be [reopened for discussion](https://about.gitlab.com/handbook/values/#disagree-commit-and-disagree). +- [Ruby Style Guide](https://github.com/rubocop/ruby-style-guide). +- [Rails Style Guide](https://github.com/rubocop/rails-style-guide). +- [RSpec Style Guide](https://github.com/rubocop/rspec-style-guide). -## String literals quoting +Generally, if a style is not covered by existing RuboCop rules or the above style guides, it shouldn't be a blocker. -Due to the sheer amount of work to rectify, we do not care whether string -literals are single, or double quoted. +Some styles we have decided [no one should not have a strong opinion on](#styles-we-have-no-opinion-on). -Previous discussions include: +See also: -- <https://gitlab.com/gitlab-org/gitlab-foss/-/issues/44234> -- <https://gitlab.com/gitlab-org/gitlab-foss/-/issues/36076> -- <https://gitlab.com/gitlab-org/gitlab/-/issues/198046> +- [Guidelines for reusing abstractions](../reusing_abstractions.md). +- [Test-specific style guides and best practices](../testing_guide/index.md). + +## Styles we have no rule for -## Instance variable access using `attr_reader` +These styles are not backed by a RuboCop rule. + +For every style added to this section, link the discussion from the section's [version history note](../documentation/versions.md#add-a-version-history-item) to provide context and serve as a reference. + +### Instance variable access using `attr_reader` > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/52351) in GitLab 14.1. @@ -84,11 +85,11 @@ Public attributes should only be used if they are accessed outside of the class. There is not a strong opinion on what strategy is used when attributes are only accessed internally, as long as there is consistency in related code. -## Newlines style guide +### Newlines style guide -This style guide recommends best practices for newlines in Ruby code. +In addition to the RuboCops `Layout/EmptyLinesAroundMethodBody` and `Cop/LineBreakAroundConditionalBlock` that enforce some newline styles, we have the following guidelines that are not backed by RuboCop. -### Rule: separate code with newlines only to group together related logic +#### Rule: separate code with newlines only to group together related logic ```ruby # bad @@ -111,9 +112,7 @@ def method end ``` -### Rule: separate code and block with newlines - -#### Newline before block +#### Rule: newline before block ```ruby # bad @@ -136,35 +135,11 @@ def method end ``` -### Rule: Newline after block - -```ruby -# bad -def method - if issue.save - issue.send_email - end - render json: issue -end -``` - -```ruby -# good -def method - if issue.save - issue.send_email - end - - render json: issue -end -``` - -#### Exception: no need for newline when code block starts or ends right inside another code block +##### Exception: no need for a newline when code block starts or ends right inside another code block ```ruby # bad def method - if issue if issue.valid? @@ -172,7 +147,6 @@ def method end end - end ``` @@ -186,3 +160,18 @@ def method end end ``` + +## Styles we have no opinion on + +If a RuboCop rule is proposed and we choose not to add it, we should document that decision in this guide so it is more discoverable and link the relevant discussion as a reference. + +### Quoting string literals + +Due to the sheer amount of work to rectify, we do not care whether string +literals are single or double-quoted. + +Previous discussions include: + +- <https://gitlab.com/gitlab-org/gitlab-foss/-/issues/44234> +- <https://gitlab.com/gitlab-org/gitlab-foss/-/issues/36076> +- <https://gitlab.com/gitlab-org/gitlab/-/issues/198046> diff --git a/doc/development/cached_queries.md b/doc/development/cached_queries.md index fbb857106be..e4625a50d79 100644 --- a/doc/development/cached_queries.md +++ b/doc/development/cached_queries.md @@ -87,7 +87,7 @@ and the number of executed cached queries:  -The page included 55 cached queries. Clicking the number displays a modal window +The page included 55 cached queries. Selecting the number displays a modal window with more details about queries. Cached queries are marked with the `cached` label below the query. You can see multiple duplicate cached queries in this modal window: diff --git a/doc/development/cascading_settings.md b/doc/development/cascading_settings.md index 22f146c3f5a..1a0f0ec5b5f 100644 --- a/doc/development/cascading_settings.md +++ b/doc/development/cascading_settings.md @@ -38,7 +38,7 @@ Settings are not cascading by default. To define a cascading setting, take the f `application_settings`. ```ruby - class AddDelayedProjectRemovalCascadingSetting < Gitlab::Database::Migration[1.0] + class AddDelayedProjectRemovalCascadingSetting < Gitlab::Database::Migration[2.0] include Gitlab::Database::MigrationHelpers::CascadingNamespaceSettings enable_lock_retries! diff --git a/doc/development/changelog.md b/doc/development/changelog.md index 7dc3ae0a80b..27bcffe7560 100644 --- a/doc/development/changelog.md +++ b/doc/development/changelog.md @@ -41,6 +41,10 @@ vendor field to be `gitlab` to avoid cve matching old versions. Changelog: changed ``` +If your merge request has multiple commits, +[make sure to add the `Changelog` entry to the first commit](changelog.md#how-to-generate-a-changelog-entry). +This ensures that the correct entry is generated when commits are squashed. + ### Overriding the associated merge request GitLab automatically links the merge request to the commit when generating the diff --git a/doc/development/chatops_on_gitlabcom.md b/doc/development/chatops_on_gitlabcom.md index 16dc17dd229..c9cb2591e4e 100644 --- a/doc/development/chatops_on_gitlabcom.md +++ b/doc/development/chatops_on_gitlabcom.md @@ -23,7 +23,7 @@ To request access to ChatOps on GitLab.com: with one of the following methods (Okta is not supported): - The same username you use on GitLab.com. You may have to choose a different username later. - - Clicking the **Sign in with Google** button to sign in with your GitLab.com email address. + - Selecting the **Sign in with Google** button to sign in with your GitLab.com email address. 1. Confirm that your username in [Internal GitLab for Operations](https://ops.gitlab.net/) is the same as your username in [GitLab.com](https://gitlab.com/). If the usernames @@ -31,7 +31,7 @@ To request access to ChatOps on GitLab.com: 1. Comment in your onboarding issue, and tag your onboarding buddy and your manager. Request they add you to the `ops` ChatOps project by running this command - in the `#chat-ops-test` Slack channel, replacing `<username>` with your username: + in the `#chat-ops-test` Slack channel, replacing `<username>` with your GitLab.com username: `/chatops run member add <username> gitlab-com/chatops --ops` ```plaintext diff --git a/doc/development/cicd/index.md b/doc/development/cicd/index.md index cd31038ddd1..73ece709b8d 100644 --- a/doc/development/cicd/index.md +++ b/doc/development/cicd/index.md @@ -33,7 +33,7 @@ On the left side we have the events that can trigger a pipeline based on various - A `git push` is the most common event that triggers a pipeline. - The [Web API](../../api/pipelines.md#create-a-new-pipeline). -- A user clicking the "Run pipeline" button in the UI. +- A user selecting the "Run pipeline" button in the UI. - When a [merge request is created or updated](../../ci/pipelines/merge_request_pipelines.md). - When an MR is added to a [Merge Train](../../ci/pipelines/merge_trains.md#merge-trains). - A [scheduled pipeline](../../ci/pipelines/schedules.md). diff --git a/doc/development/cicd/templates.md b/doc/development/cicd/templates.md index 85ac58e749d..6bc6c57e809 100644 --- a/doc/development/cicd/templates.md +++ b/doc/development/cicd/templates.md @@ -418,7 +418,7 @@ is updated in a major version GitLab release. Every CI/CD template must also have metrics defined to track their use. The CI/CD template monthly usage report can be found in [Sisense (GitLab team members only)](https://app.periscopedata.com/app/gitlab/785953/Pipeline-Authoring-Dashboard?widget=13440051&udv=0). -Double click a template to see the graph for that single template. +Select a template to see the graph for that single template. To add a metric definition for a new template: diff --git a/doc/development/code_review.md b/doc/development/code_review.md index 5b745f06d22..90f33319365 100644 --- a/doc/development/code_review.md +++ b/doc/development/code_review.md @@ -28,6 +28,13 @@ The reviewer can: - Give you a second opinion on the chosen solution and implementation. - Help look for bugs, logic problems, or uncovered edge cases. +If the merge request is trivial (for example, fixing a typo or a tiny refactor that doesn't change the behavior or any data), +you can skip the reviewer step and directly ask a [maintainer](https://about.gitlab.com/handbook/engineering/workflow/code-review/#maintainer). +Otherwise, a merge request should always be first reviewed by a reviewer in each +[category (e.g. backend, database)](#approval-guidelines) +the MR touches, as maintainers may not have the relevant domain knowledge, and +also to spread the workload. + For assistance with security scans or comments, include the Application Security Team (`@gitlab-com/gl-security/appsec`). The reviewers use the [reviewer functionality](../user/project/merge_requests/getting_started.md#reviewer) in the sidebar. @@ -56,8 +63,8 @@ We make the following assumption with regards to automatically being considered - Team members working in a specific stage/group (for example, create: source code) are considered domain experts for that area of the app they work on. - Team members working on a specific feature (for example, search) are considered domain experts for that feature. -We default to assigning reviews to team members with domain expertise. -When a suitable [domain expert](#domain-experts) isn't available, you can choose any team member to review the MR, or follow the [Reviewer roulette](#reviewer-roulette) recommendation. +We default to assigning reviews to team members with domain expertise for code reviews. For UX reviews we default to the recommended designer from the Reviewer roulette. +When a suitable [domain expert](#domain-experts) isn't available, you can choose any team member to review the MR, or follow the [Reviewer roulette](#reviewer-roulette) recommendation (see above for UX reviews). To find a domain expert: @@ -77,7 +84,7 @@ Reviewer roulette is an internal tool for use on GitLab.com, and not available f The [Danger bot](dangerbot.md) randomly picks a reviewer and a maintainer for each area of the codebase that your merge request seems to touch. It makes **recommendations** for developer reviewers and you should override it if you think someone else is a better -fit. User-facing changes are required to have a UX review, too. Default to the recommended UX reviewer suggested. +fit. User-facing changes are also required to have a UX review, even if it's behind a feature flag. Default to the recommended UX reviewer suggested. It picks reviewers and maintainers from the list at the [engineering projects](https://about.gitlab.com/handbook/engineering/projects/) @@ -147,7 +154,7 @@ with [domain expertise](#domain-experts). | `~workhorse` changes | [Workhorse maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_maintainers_workhorse). | | `~frontend` changes (*1*) | [Frontend maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_maintainers_frontend). | | `~UX` user-facing changes (*3*) | [Product Designer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_reviewers_UX). Refer to the [design and user interface guidelines](contributing/design.md) for details. | -| Adding a new JavaScript library (*1*) | - [Frontend foundations member](https://about.gitlab.com/direction/ecosystem/foundations/) if the library significantly increases the [bundle size](https://gitlab.com/gitlab-org/frontend/playground/webpack-memory-metrics/-/blob/master/doc/report.md).<br/>- A [legal department member](https://about.gitlab.com/handbook/legal/) if the license used by the new library hasn't been approved for use in GitLab.<br/><br/>More information about license compatibility can be found in our [GitLab Licensing and Compatibility documentation](licensing.md). | +| Adding a new JavaScript library (*1*) | - [Frontend foundations member](https://about.gitlab.com/direction/manage/foundations/) if the library significantly increases the [bundle size](https://gitlab.com/gitlab-org/frontend/playground/webpack-memory-metrics/-/blob/master/doc/report.md).<br/>- A [legal department member](https://about.gitlab.com/handbook/legal/) if the license used by the new library hasn't been approved for use in GitLab.<br/><br/>More information about license compatibility can be found in our [GitLab Licensing and Compatibility documentation](licensing.md). | | A new dependency or a file system change | - [Distribution team member](https://about.gitlab.com/company/team/). See how to work with the [Distribution team](https://about.gitlab.com/handbook/engineering/development/enablement/systems/distribution/#how-to-work-with-distribution) for more details.<br/>- For Rubygems, request an [AppSec review](gemfile.md#request-an-appsec-review). | | `~documentation` or `~UI text` changes | [Technical writer](https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments) based on assignments in the appropriate [DevOps stage group](https://about.gitlab.com/handbook/product/categories/#devops-stages). | | Changes to development guidelines | Follow the [review process](development_processes.md#development-guidelines-review) and get the approvals accordingly. | @@ -212,8 +219,8 @@ See the [test engineering process](https://about.gitlab.com/handbook/engineering ##### Security -1. I have confirmed that if this MR contains changes to processing or storing of credentials or tokens, authorization, and authentication methods, or other items described in [the security review guidelines](https://about.gitlab.com/handbook/engineering/security/#when-to-request-a-security-review), I have added the `~security` label and I have `@`-mentioned `@gitlab-com/gl-security/appsec`. -1. I have reviewed the documentation regarding [internal application security reviews](https://about.gitlab.com/handbook/engineering/security/#internal-application-security-reviews) for **when** and **how** to request a security review and requested a security review if this is warranted for this change. +1. I have confirmed that if this MR contains changes to processing or storing of credentials or tokens, authorization, and authentication methods, or other items described in [the security review guidelines](https://about.gitlab.com/handbook/security/#when-to-request-a-security-review), I have added the `~security` label and I have `@`-mentioned `@gitlab-com/gl-security/appsec`. +1. I have reviewed the documentation regarding [internal application security reviews](https://about.gitlab.com/handbook/security/#internal-application-security-reviews) for **when** and **how** to request a security review and requested a security review if this is warranted for this change. ##### Deployment @@ -508,7 +515,7 @@ people who add commits to an MR are not authorized to approve the merge request, so they must seek a maintainer who has not contributed to the MR to approve the MR before it can be merged. This policy is in place to satisfy the CHG-04 control of the GitLab -[Change Management Controls](https://about.gitlab.com/handbook/engineering/security/security-assurance/security-compliance/guidance/change-management.html). +[Change Management Controls](https://about.gitlab.com/handbook/security/security-assurance/security-compliance/guidance/change-management.html). To implement this policy in `gitlab-org/gitlab`, we have enabled the following settings to ensure MRs get an approval from a top-level CODEOWNERS maintainer: @@ -518,6 +525,9 @@ settings to ensure MRs get an approval from a top-level CODEOWNERS maintainer: - [Prevent editing approval rules in merge requests](../user/project/merge_requests/approvals/settings.md#prevent-editing-approval-rules-in-merge-requests). - [Remove all approvals when commits are added to the source branch](../user/project/merge_requests/approvals/settings.md#remove-all-approvals-when-commits-are-added-to-the-source-branch) +To update the code owners in the `CODEOWNERS` file for `gitlab-org/gitlab`, follow +the process explained in the [code owners approvals handbook section](https://about.gitlab.com/handbook/engineering/workflow/code-review/#code-owner-approvals). + There are scenarios such as rebasing locally or applying suggestions that are considered the same as adding a commit and could reset existing approvals. Approvals are not removed when rebasing from the UI or with the [`/rebase` quick action](../user/project/quick_actions.md). diff --git a/doc/development/contributing/design.md b/doc/development/contributing/design.md index 9e54b92337a..e6b6b56cf73 100644 --- a/doc/development/contributing/design.md +++ b/doc/development/contributing/design.md @@ -24,7 +24,7 @@ screenshots (or videos) of your changes in the description, as explained in our [MR workflow](merge_request_workflow.md). These screenshots/videos are very helpful for all reviewers and can speed up the review process, especially if the changes are small. -- Attach the ~UX label to any merge request that impacts the user experience. This will enable Product Designers to [review](https://about.gitlab.com/handbook/engineering/ux/product-designer/mr-reviews/#stage-group-mrs/) any user facing changes. +- Attach the ~UX label to any merge request that impacts the user experience. This will enable Product Designers to [review](https://about.gitlab.com/handbook/product/ux/product-designer/mr-reviews/#stage-group-mrs/) any user facing changes. - Assign the Product Designer suggested by Reviewer Roulette as the reviewer of your merge request. The reviewer does not have to be the domain expert unless this is a community contribution. ## Checklist @@ -64,7 +64,7 @@ Check visual design properties using your browser's _elements inspector_ ([Chrom guidelines. - _Optionally_ consider [dark mode](../../user/profile/preferences.md#dark-mode). [^1] - [^1]: You're not required to design for [dark mode](../../user/profile/preferences.md#dark-mode) while the feature is in [alpha](../../policy/alpha-beta-support.md#alpha-features). The [UX Foundations team](https://about.gitlab.com/direction/ecosystem/foundations/) plans to improve the dark mode in the future. Until we integrate [Pajamas](https://design.gitlab.com/) components into the product and the underlying design strategy is in place to support dark mode, we cannot guarantee that we won't introduce bugs and debt to this mode. At your discretion, evaluate the need to create dark mode patches. + [^1]: You're not required to design for [dark mode](../../user/profile/preferences.md#dark-mode) while the feature is in [alpha](../../policy/alpha-beta-support.md#alpha-features). The [UX Foundations team](https://about.gitlab.com/direction/manage/foundations/) plans to improve the dark mode in the future. Until we integrate [Pajamas](https://design.gitlab.com/) components into the product and the underlying design strategy is in place to support dark mode, we cannot guarantee that we won't introduce bugs and debt to this mode. At your discretion, evaluate the need to create dark mode patches. ### States diff --git a/doc/development/contributing/index.md b/doc/development/contributing/index.md index cbccd832d78..8c0d18f877b 100644 --- a/doc/development/contributing/index.md +++ b/doc/development/contributing/index.md @@ -100,7 +100,7 @@ If you have any questions or need help, visit [Getting Help](https://about.gitla communicate with the GitLab community. GitLab prefers [asynchronous communication](https://about.gitlab.com/handbook/communication/#internal-communication) over real-time communication. We do encourage you to connect and hang out with us. GitLab has a Gitter room dedicated for [contributors](https://gitter.im/gitlab/contributors), which is bridged with our -internal Slack. We actively monitor this channel. There is also a community-run [Discord server](https://discord.gg/S4cwz9sR8u) where you can +internal Slack. We actively monitor this channel. There is also a community-run [Discord server](http://discord.gg/gitlab) where you can find other contributors in the `#contributors` channel. Thanks for your contribution! diff --git a/doc/development/contributing/merge_request_workflow.md b/doc/development/contributing/merge_request_workflow.md index b40df01cbe9..f2c06e289c9 100644 --- a/doc/development/contributing/merge_request_workflow.md +++ b/doc/development/contributing/merge_request_workflow.md @@ -38,6 +38,10 @@ and see the [Development section](../../index.md) for the required guidelines. If you find an issue, please submit a merge request with a fix or improvement, if you can, and include tests. +NOTE: +Consider placing your code behind a feature flag if you think it might affect production availability. +Not sure? Read [When to use feature flags](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#when-to-use-feature-flags). + If the change is non-trivial, we encourage you to start a discussion with [a product manager or a member of the team](https://about.gitlab.com/handbook/product/categories/). You can do diff --git a/doc/development/contributing/style_guides.md b/doc/development/contributing/style_guides.md index 9d04e1590d0..fe1aa8449c2 100644 --- a/doc/development/contributing/style_guides.md +++ b/doc/development/contributing/style_guides.md @@ -108,117 +108,6 @@ pre-push: For more information, check out [Lefthook documentation Skipping commands section](https://github.com/evilmartians/lefthook/blob/master/docs/full_guide.md#skipping-commands). -## Ruby, Rails, RSpec - -Our codebase style is defined and enforced by [RuboCop](https://github.com/rubocop-hq/rubocop). - -You can check for any offenses locally with `bundle exec rubocop --parallel`. -On the CI, this is automatically checked by the `static-analysis` jobs. - -In addition, you can [integrate RuboCop](../developing_with_solargraph.md) into -supported IDEs using the [Solargraph](https://github.com/castwide/solargraph) gem. - -For RuboCop rules that we have not taken a decision on yet, we follow the -[Ruby Style Guide](https://github.com/rubocop-hq/ruby-style-guide), -[Rails Style Guide](https://github.com/rubocop-hq/rails-style-guide), and -[RSpec Style Guide](https://github.com/rubocop-hq/rspec-style-guide) as general -guidelines to write idiomatic Ruby/Rails/RSpec, but reviewers/maintainers should -be tolerant and not too pedantic about style. - -Similarly, some RuboCop rules are currently disabled, and for those, -reviewers/maintainers must not ask authors to use one style or the other, as both -are accepted. This isn't an ideal situation since this leaves space for -[bike-shedding](https://en.wiktionary.org/wiki/bikeshedding), and ideally we -should enable all RuboCop rules to avoid style-related -discussions/nitpicking/back-and-forth in reviews. There are some styles that -commonly come up in reviews that are not enforced, the -[GitLab Ruby style guide](../backend/ruby_style_guide.md) includes a non-exhaustive -list of these topics. - -Additionally, we have a dedicated -[newlines style guide](../newlines_styleguide.md), as well as dedicated -[test-specific style guides and best practices](../testing_guide/index.md). - -### Creating new RuboCop cops - -Typically it is better for the linting rules to be enforced programmatically as it -reduces the aforementioned [bike-shedding](https://en.wiktionary.org/wiki/bikeshedding). - -To that end, we encourage creation of new RuboCop rules in the codebase. - -We maintain Cops across several Ruby code bases, and not all of them are -specific to the GitLab application. -When creating a new cop that could be applied to multiple applications, we encourage you -to add it to our [GitLab Styles](https://gitlab.com/gitlab-org/gitlab-styles) gem. -If the Cop targets rules that only apply to the main GitLab application, -it should be added to [GitLab](https://gitlab.com/gitlab-org/gitlab) instead. - -### Cop grace period - -A cop is in a "grace period" if it is enabled and has `Details: grace period` defined in its TODO YAML configuration. - -On the default branch, all of the offenses from cops in the ["grace period"](../rake_tasks.md#run-rubocop-in-graceful-mode) will not fail the RuboCop CI job. The job will notify Slack in the `#f_rubocop` channel when offenses have been silenced in the scheduled pipeline. However, on merge request pipelines, the RuboCop job will fail. - -A grace period can safely be lifted as soon as there are no warnings for 2 weeks in the `#f_rubocop` channel on Slack. - -### Enabling a new cop - -1. Enable the new cop in `.rubocop.yml` (if not already done via [`gitlab-styles`](https://gitlab.com/gitlab-org/ruby/gems/gitlab-styles)). -1. [Generate TODOs for the new cop](../rake_tasks.md#generate-initial-rubocop-todo-list). -1. [Set the new cop to "grace period"](#cop-grace-period). -1. Create an issue to fix TODOs and encourage Community contributions (via ~"good for new contributors" and/or ~"Seeking community contributions"). [See some examples](https://gitlab.com/gitlab-org/gitlab/-/issues/?sort=created_date&state=opened&label_name%5B%5D=good%20for%20new%20contributors&label_name%5B%5D=static%20code%20analysis&first_page_size=20). -1. Create an issue to remove "grace period" after 2 weeks silence in `#f_rubocop` Slack channel. ([See an example](https://gitlab.com/gitlab-org/gitlab/-/issues/374903).) - -### Silenced offenses - -When offenses are silenced for cops in ["grace period"](#cop-grace-period), -the `#f_rubocop` Slack channel receives a notification message every two hours. - -To fix this issue: - -1. Find cops with silenced offenses in the linked CI job. -1. [Generate TODOs](../rake_tasks.md#generate-initial-rubocop-todo-list) for these cops. - -#### RuboCop node pattern - -When creating [node patterns](https://docs.rubocop.org/rubocop-ast/node_pattern.html) to match -Ruby's AST, you can use [`scripts/rubocop-parse`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/scripts/rubocop-parse) -to display the AST of a Ruby expression, to help you create the matcher. -See also [!97024](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/97024). - -### Resolving RuboCop exceptions - -When the number of RuboCop exceptions exceed the default [`exclude-limit` of 15](https://docs.rubocop.org/rubocop/1.2/usage/basic_usage.html#command-line-flags), -we may want to resolve exceptions over multiple commits. To minimize confusion, -we should track our progress through the exception list. - -The preferred way to [generate the initial list or a list for specific RuboCop rules](../rake_tasks.md#generate-initial-rubocop-todo-list) -is to run the Rake task `rubocop:todo:generate`: - -```shell -# Initial list -bundle exec rake rubocop:todo:generate - -# List for specific RuboCop rules -bundle exec rake 'rubocop:todo:generate[Gitlab/NamespacedClass,Lint/Syntax]' -``` - -This Rake task creates or updates the exception list in `.rubocop_todo/`. For -example, the configuration for the RuboCop rule `Gitlab/NamespacedClass` is -located in `.rubocop_todo/gitlab/namespaced_class.yml`. - -Make sure to commit any changes in `.rubocop_todo/` after running the Rake task. - -### Reveal existing RuboCop exceptions - -To reveal existing RuboCop exceptions in the code that have been excluded via `.rubocop_todo.yml` and -`.rubocop_todo/**/*.yml`, set the environment variable `REVEAL_RUBOCOP_TODO` to `1`. - -This allows you to reveal existing RuboCop exceptions during your daily work cycle and fix them along the way. - -NOTE: -Define permanent `Exclude`s in `.rubocop.yml` instead of `.rubocop_todo/**/*.yml`. - ## Database migrations See the dedicated [Database Migrations Style Guide](../migration_style_guide.md). @@ -231,6 +120,10 @@ See the dedicated [JS Style Guide](../fe_guide/style/javascript.md). See the dedicated [SCSS Style Guide](../fe_guide/style/scss.md). +## Ruby + +See the dedicated [Ruby Style Guide](../backend/ruby_style_guide.md). + ## Go See the dedicated [Go standards and style guidelines](../go_guide/index.md). diff --git a/doc/development/database/adding_database_indexes.md b/doc/development/database/adding_database_indexes.md index 040c6780316..d4cd807ef22 100644 --- a/doc/development/database/adding_database_indexes.md +++ b/doc/development/database/adding_database_indexes.md @@ -215,6 +215,42 @@ def down end ``` +## Indexes for partitioned tables + +Indexes [cannot be created](https://www.postgresql.org/docs/15/ddl-partitioning.html#DDL-PARTITIONING-DECLARATIVE-MAINTENANCE) +**concurrently** on a partitioned table. You must use `CONCURRENTLY` to avoid service disruption in a hot system. + +To create an index on a partitioned table, use `add_concurrent_partitioned_index`, provided by the database team. + +Under the hood, `add_concurrent_partitioned_index`: + +1. Creates indexes on each partition using `CONCURRENTLY`. +1. Creates an index on the parent table. + +A Rails migration example: + +```ruby +# in db/post_migrate/ + +class AddIndexToPartitionedTable < Gitlab::Database::Migration[2.0] + include Gitlab::Database::PartitioningMigrationHelpers + + disable_ddl_transaction! + + TABLE_NAME = :table_name + COLUMN_NAMES = [:partition_id, :id] + INDEX_NAME = :index_name + + def up + add_concurrent_partitioned_index(TABLE_NAME, COLUMN_NAMES, name: INDEX_NAME) + end + + def down + remove_concurrent_partitioned_index_by_name(TABLE_NAME, INDEX_NAME) + end +end +``` + ## Create indexes asynchronously For very large tables, index creation can be a challenge to manage. diff --git a/doc/development/database/background_migrations.md b/doc/development/database/background_migrations.md index 8e6f29b9eb8..fe62bbc6b14 100644 --- a/doc/development/database/background_migrations.md +++ b/doc/development/database/background_migrations.md @@ -236,7 +236,7 @@ Next we need a post-deployment migration that schedules the migration for existing data. ```ruby -class ScheduleExtractIntegrationsUrl < Gitlab::Database::Migration[1.0] +class ScheduleExtractIntegrationsUrl < Gitlab::Database::Migration[2.0] disable_ddl_transaction! MIGRATION = 'ExtractIntegrationsUrl' @@ -263,7 +263,7 @@ jobs and manually run on any un-migrated rows. Such a migration would look like this: ```ruby -class ConsumeRemainingExtractIntegrationsUrlJobs < Gitlab::Database::Migration[1.0] +class ConsumeRemainingExtractIntegrationsUrlJobs < Gitlab::Database::Migration[2.0] disable_ddl_transaction! def up diff --git a/doc/development/database/batched_background_migrations.md b/doc/development/database/batched_background_migrations.md index a48a9c42e27..ca11e9c8dd3 100644 --- a/doc/development/database/batched_background_migrations.md +++ b/doc/development/database/batched_background_migrations.md @@ -219,6 +219,7 @@ In this example, `copy_from` returns `name`, and `copy_to` returns `name_convert ```ruby class CopyColumnUsingBackgroundMigrationJob < BatchedMigrationJob job_arguments :copy_from, :copy_to + operation_name :update_all def perform from_column = connection.quote_column_name(copy_from) @@ -226,7 +227,7 @@ class CopyColumnUsingBackgroundMigrationJob < BatchedMigrationJob assignment_clause = "#{to_column} = #{from_column}" - each_sub_batch(operation_name: :update_all) do |relation| + each_sub_batch do |relation| relation.update_all(assignment_clause) end end @@ -267,9 +268,10 @@ In the second (filtered) example, we know exactly 100 will be updated with each ```ruby class BackfillNamespaceType < BatchedMigrationJob scope_to ->(relation) { relation.where(type: nil) } + operation_name :update_all def perform - each_sub_batch(operation_name: :update_all) do |sub_batch| + each_sub_batch do |sub_batch| sub_batch.update_all(type: 'User') end end @@ -327,9 +329,10 @@ background migration. # self.table_name = 'routes' # end + operation_name :update_all + def perform each_sub_batch( - operation_name: :update_all, batching_scope: -> (relation) { relation.where("source_type <> 'UnusedType'") } ) do |sub_batch| sub_batch.update_all('namespace_id = source_id') @@ -483,8 +486,10 @@ module Gitlab class BatchProjectsWithIssues < Gitlab::BackgroundMigration::BatchedMigrationJob include Gitlab::Database::DynamicModelHelpers + operation_name :backfill_issues + def perform - distinct_each_batch(operation_name: :backfill_issues) do |batch| + distinct_each_batch do |batch| project_ids = batch.pluck(batch_column) # do something with the distinct project_ids end diff --git a/doc/development/database/database_debugging.md b/doc/development/database/database_debugging.md index 4dc6a3bdcfa..0d6e9955a19 100644 --- a/doc/development/database/database_debugging.md +++ b/doc/development/database/database_debugging.md @@ -49,9 +49,11 @@ bundle exec rake db:reset RAILS_ENV=test - `bundle exec rake db:migrate RAILS_ENV=development`: Execute any pending migrations that you may have picked up from a MR - `bundle exec rake db:migrate:status RAILS_ENV=development`: Check if all migrations are `up` or `down` -- `bundle exec rake db:migrate:down VERSION=20170926203418 RAILS_ENV=development`: Tear down a migration -- `bundle exec rake db:migrate:up VERSION=20170926203418 RAILS_ENV=development`: Set up a migration -- `bundle exec rake db:migrate:redo VERSION=20170926203418 RAILS_ENV=development`: Re-run a specific migration +- `bundle exec rake db:migrate:down:main VERSION=20170926203418 RAILS_ENV=development`: Tear down a migration +- `bundle exec rake db:migrate:up:main VERSION=20170926203418 RAILS_ENV=development`: Set up a migration +- `bundle exec rake db:migrate:redo:main VERSION=20170926203418 RAILS_ENV=development`: Re-run a specific migration + +Replace `main` in the above commands to execute agains the `ci` database instead of `main`. ## Manually access the database diff --git a/doc/development/database/database_migration_pipeline.md b/doc/development/database/database_migration_pipeline.md index 148dc1e94a0..06e16b4c7f1 100644 --- a/doc/development/database/database_migration_pipeline.md +++ b/doc/development/database/database_migration_pipeline.md @@ -22,34 +22,54 @@ For security reasons, access to the pipeline is restricted to database maintaine When the pipeline starts, a bot notifies you with a comment in the merge request. When it finishes, the comment gets updated with the test results. -There are three sections which are described below. + +The comment contains testing information for both the `main` and `ci` databases. +Each database tested has four sections which are described below. ## Summary The first section of the comment contains a summary of the test results, including: -| Result | Description | -|-------------------|---------------------------------------------------------------------------------------------------------------------| -| Warnings | Highlights critical issues such as exceptions or long-running queries. | -| Migrations | The time each migration took to complete, whether it was successful, and the increment in the size of the database. | -| Runtime histogram | Expand this section to see a histogram of query runtimes across all migrations. | +- **Warnings** - Highlights critical issues such as exceptions or long-running queries. +- **Migrations** - The time each migration took to complete, whether it was successful, + and the increment in the size of the database. +- **Runtime histogram** - Expand this section to see a histogram of query runtimes across all migrations. ## Migration details The next section of the comment contains detailed information for each migration, including: -| Result | Description | -|-------------------|-------------------------------------------------------------------------------------------------------------------------| -| Details | The type of migration, total duration, and database size change. | -| Queries | Every query executed during the migration, along with the number of calls, timings, and the number of the changed rows. | -| Runtime histogram | Indicates the distribution of query times for the migration. | +- **Details** - The type of migration, total duration, and database size change. +- **Queries** - Every query executed during the migration, along with the number of + calls, timings, and the number of the changed rows. +- **Runtime histogram** - Indicates the distribution of query times for the migration. + +## Background migration details + +The next section of the comment contains detailed information about each batched background migration, including: + +- **Sampling information** - The number of batches sampled during this test run. + Sampled batches are chosen uniformly across the table's ID range. Sampling runs + for 30 minutes, split evenly across each background migration to test. +- **Aggregated query information** - Aggregate data about each query executed across + all the sampled batches, along with the number of calls, timings, and the number of changed rows. +- **Batch runtime histogram** - A histogram of timings for each sampled batch + from the background migration. +- **Query runtime histogram** - A histogram of timings for all queries executed + in any batch of this background migration. ## Clone details and artifacts Some additional information is included at the bottom of the comment: -| Result | Description | -|----------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| Migrations pending on GitLab.com | A summary of migrations not deployed yet to GitLab.com. This information is useful when testing a migration that was merged but not deployed yet. | -| Clone details | A link to the `Postgres.ai` thin clone created for this testing pipeline, along with information about its expiry. This can be used to further explore the results of running the migration. Only accessible by database maintainers or with an access request. | -| Artifacts | A link to the pipeline's artifacts. Full query logs for each migration (ending in `.log`) are available there and only accessible by database maintainers or with an access request. | +- **Migrations pending on GitLab.com** - A summary of migrations not deployed yet + to GitLab.com. This information is useful when testing a migration that was merged + but not deployed yet. +- **Clone details** - A link to the `Postgres.ai` thin clone created for this + testing pipeline, along with information about its expiry. This can be used to + further explore the results of running the migration. Only accessible by + database maintainers or with an access request. +- **Artifacts** - A link to the pipeline's artifacts. Full query logs for each + migration (ending in `.log`) are available there, and only accessible by + database maintainers or with an access request. Details of the specific + batched background migration batches sampled are also available. diff --git a/doc/development/database/loose_foreign_keys.md b/doc/development/database/loose_foreign_keys.md index abf66368548..962cd2602bc 100644 --- a/doc/development/database/loose_foreign_keys.md +++ b/doc/development/database/loose_foreign_keys.md @@ -192,7 +192,7 @@ trigger needs to be configured only once. If the model already has at least one `loose_foreign_key` definition, then this step can be skipped: ```ruby -class TrackProjectRecordChanges < Gitlab::Database::Migration[1.0] +class TrackProjectRecordChanges < Gitlab::Database::Migration[2.0] include Gitlab::Database::MigrationHelpers::LooseForeignKeyHelpers enable_lock_retries! @@ -227,7 +227,7 @@ trigger. If the foreign key is deleted earlier, there is a good chance of introducing data inconsistency which needs manual cleanup: ```ruby -class RemoveProjectsCiPipelineFk < Gitlab::Database::Migration[1.0] +class RemoveProjectsCiPipelineFk < Gitlab::Database::Migration[2.0] disable_ddl_transaction! def up @@ -258,7 +258,7 @@ records in the database. Migration for removing the trigger: ```ruby -class UnTrackProjectRecordChanges < Gitlab::Database::Migration[1.0] +class UnTrackProjectRecordChanges < Gitlab::Database::Migration[2.0] include Gitlab::Database::MigrationHelpers::LooseForeignKeyHelpers enable_lock_retries! @@ -278,7 +278,7 @@ table however, there is still a chance for having leftover pending records in th must be removed with an inline data migration. ```ruby -class RemoveLeftoverProjectDeletions < Gitlab::Database::Migration[1.0] +class RemoveLeftoverProjectDeletions < Gitlab::Database::Migration[2.0] disable_ddl_transaction! def up diff --git a/doc/development/database/not_null_constraints.md b/doc/development/database/not_null_constraints.md index dccaff2df00..53ab9a83d60 100644 --- a/doc/development/database/not_null_constraints.md +++ b/doc/development/database/not_null_constraints.md @@ -25,7 +25,7 @@ For example, consider a migration that creates a table with two `NOT NULL` colum `db/migrate/20200401000001_create_db_guides.rb`: ```ruby -class CreateDbGuides < Gitlab::Database::Migration[1.0] +class CreateDbGuides < Gitlab::Database::Migration[2.0] def change create_table :db_guides do |t| t.bigint :stars, default: 0, null: false @@ -44,7 +44,7 @@ For example, consider a migration that adds a new `NOT NULL` column `active` to `db/migrate/20200501000001_add_active_to_db_guides.rb`: ```ruby -class AddExtendedTitleToSprints < Gitlab::Database::Migration[1.0] +class AddExtendedTitleToSprints < Gitlab::Database::Migration[2.0] def change add_column :db_guides, :active, :boolean, default: true, null: false end @@ -116,7 +116,7 @@ with `validate: false` in a post-deployment migration, `db/post_migrate/20200501000001_add_not_null_constraint_to_epics_description.rb`: ```ruby -class AddNotNullConstraintToEpicsDescription < Gitlab::Database::Migration[1.0] +class AddNotNullConstraintToEpicsDescription < Gitlab::Database::Migration[2.0] disable_ddl_transaction! def up @@ -147,7 +147,7 @@ so we add a post-deployment migration for the 13.0 milestone (current), `db/post_migrate/20200501000002_cleanup_epics_with_null_description.rb`: ```ruby -class CleanupEpicsWithNullDescription < Gitlab::Database::Migration[1.0] +class CleanupEpicsWithNullDescription < Gitlab::Database::Migration[2.0] # With BATCH_SIZE=1000 and epics.count=29500 on GitLab.com # - 30 iterations will be run # - each requires on average ~150ms @@ -185,7 +185,7 @@ migration helper in a final post-deployment migration, `db/post_migrate/20200601000001_validate_not_null_constraint_on_epics_description.rb`: ```ruby -class ValidateNotNullConstraintOnEpicsDescription < Gitlab::Database::Migration[1.0] +class ValidateNotNullConstraintOnEpicsDescription < Gitlab::Database::Migration[2.0] disable_ddl_transaction! def up diff --git a/doc/development/database/query_recorder.md b/doc/development/database/query_recorder.md index 3fc38c10d68..f1540e7e2ae 100644 --- a/doc/development/database/query_recorder.md +++ b/doc/development/database/query_recorder.md @@ -39,12 +39,17 @@ end As an example you might create 5 issues in between counts, which would cause the query count to increase by 5 if an N+1 problem exists. -In some cases the query count might change slightly between runs for unrelated reasons. In this case you might need to test `exceed_query_limit(control_count + acceptable_change)`, but this should be avoided if possible. +In some cases, the query count might change slightly between runs for unrelated reasons. In this case you might need to test `exceed_query_limit(control_count + acceptable_change)`, but this should be avoided if possible. If this test fails, and the control was passed as a `QueryRecorder`, then the failure message indicates where the extra queries are by matching queries on the longest common prefix, grouping similar queries together. +In some cases, N+1 specs have been written to include three requests: first one to +warm the cache, second one to establish a control, third one to validate that +ther are no N+1 queries. Rather than make an extra request to warm the cache, prefer two requests +(control and test) and configure your test to ignore [cached queries](#cached-queries) in N+1 specs. + ## Cached queries By default, QueryRecorder ignores [cached queries](../merge_request_performance_guidelines.md#cached-queries) in the count. However, it may be better to count @@ -62,7 +67,7 @@ end ## Use request specs instead of controller specs -Use a [request spec](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/spec/requests) when writing a N+1 test on the controller level. +Use a [request spec](https://gitlab.com/gitlab-org/gitlab/-/tree/master/spec/requests) when writing a N+1 test on the controller level. Controller specs should not be used to write N+1 tests as the controller is only initialized once per example. This could lead to false successes where subsequent "requests" could have queries reduced (for example, because of memoization). diff --git a/doc/development/database/single_table_inheritance.md b/doc/development/database/single_table_inheritance.md index ad0101e1594..32de1fdea35 100644 --- a/doc/development/database/single_table_inheritance.md +++ b/doc/development/database/single_table_inheritance.md @@ -47,7 +47,7 @@ This ensures that the migration loads the columns for the migration in isolation and the helper disables STI by default. ```ruby -class EnqueueSomeBackgroundMigration < Gitlab::Database::Migration[1.0] +class EnqueueSomeBackgroundMigration < Gitlab::Database::Migration[2.0] disable_ddl_transaction! def up diff --git a/doc/development/database/table_partitioning.md b/doc/development/database/table_partitioning.md index bf12329473d..ac715b871da 100644 --- a/doc/development/database/table_partitioning.md +++ b/doc/development/database/table_partitioning.md @@ -173,7 +173,7 @@ An example migration of partitioning the `audit_events` table by its `created_at` column would look like: ```ruby -class PartitionAuditEvents < Gitlab::Database::Migration[1.0] +class PartitionAuditEvents < Gitlab::Database::Migration[2.0] include Gitlab::Database::PartitioningMigrationHelpers def up @@ -200,7 +200,7 @@ into the partitioned copy. Continuing the above example, the migration would look like: ```ruby -class BackfillPartitionAuditEvents < Gitlab::Database::Migration[1.0] +class BackfillPartitionAuditEvents < Gitlab::Database::Migration[2.0] include Gitlab::Database::PartitioningMigrationHelpers def up @@ -233,7 +233,7 @@ failed jobs. Once again, continuing the example, this migration would look like: ```ruby -class CleanupPartitionedAuditEventsBackfill < Gitlab::Database::Migration[1.0] +class CleanupPartitionedAuditEventsBackfill < Gitlab::Database::Migration[2.0] include Gitlab::Database::PartitioningMigrationHelpers def up @@ -447,6 +447,7 @@ class ConvertTableToListPartitioning < Gitlab::Database::Migration[2.0] disable_ddl_transaction! TABLE_NAME = :table_name + TABLE_FK = :table_references_by_fk PARENT_TABLE_NAME = :p_table_name FIRST_PARTITION = 100 PARTITION_COLUMN = :partition_id @@ -457,6 +458,7 @@ class ConvertTableToListPartitioning < Gitlab::Database::Migration[2.0] partitioning_column: PARTITION_COLUMN, parent_table_name: PARENT_TABLE_NAME, initial_partitioning_value: FIRST_PARTITION + lock_tables: [TABLE_FK, TABLE_NAME] ) end @@ -470,3 +472,14 @@ class ConvertTableToListPartitioning < Gitlab::Database::Migration[2.0] end end ``` + +NOTE: +Do not forget to set the sequence name explicitly in your model because it will +be owned by the routing table and `ActiveRecord` can't determine it. This can +be cleaned up after the `table_name` is changed to the routing table. + +```ruby +class Model < ApplicationRecord + self.sequence_name = 'model_id_seq' +end +``` diff --git a/doc/development/database_review.md b/doc/development/database_review.md index 58776c5330c..7fbc48af91c 100644 --- a/doc/development/database_review.md +++ b/doc/development/database_review.md @@ -43,15 +43,14 @@ If your merge request description does not include these items, the review is re #### Migrations -If new migrations are introduced, in the MR **you are required to provide**: - -- The output of both migrating (`db:migrate`) and rolling back (`db:rollback`) for all migrations. +If new migrations are introduced, database reviewers must review the output of both migrating (`db:migrate`) +and rolling back (`db:rollback`) for all migrations. We have automated tooling for [GitLab](https://gitlab.com/gitlab-org/gitlab) (provided by the -[`db:check-migrations`](database/dbcheck-migrations-job.md) pipeline job) that provides this output for migrations on -~database merge requests. You do not need to provide this information manually -if the bot can do it for you. The bot also checks that migrations are correctly +[`db:check-migrations`](database/dbcheck-migrations-job.md) pipeline job) that provides this output in the CI job logs. +It is not required for the author to provide this output in the merge request description, +but doing so may be helpful for reviewers. The bot also checks that migrations are correctly reversible. #### Queries @@ -176,7 +175,7 @@ Include in the MR description: - The query plan for each raw SQL query included in the merge request along with the link to the query plan following each raw SQL snippet. - Provide a public link to the plan from either: - [postgres.ai](https://postgres.ai/): Follow the link in `#database-lab` and generate a shareable, public link - by clicking **Share** in the upper right corner. + by selecting **Share** in the upper right corner. - [explain.depesz.com](https://explain.depesz.com) or [explain.dalibo.com](https://explain.dalibo.com): Paste both the plan and the query used in the form. - When providing query plans, make sure it hits enough data: - You can use a GitLab production replica to test your queries on a large scale, diff --git a/doc/development/deprecation_guidelines/index.md b/doc/development/deprecation_guidelines/index.md index a940cd9404c..be4a3369dcb 100644 --- a/doc/development/deprecation_guidelines/index.md +++ b/doc/development/deprecation_guidelines/index.md @@ -83,7 +83,7 @@ For configuration removals, see the [Omnibus deprecation policy](../../administr For versioning and upgrade details, see our [Release and Maintenance policy](../../policy/maintenance.md). -## Update the deprecations and removals documentation +## Update the deprecations and removals documentation pages The [deprecations](../../update/deprecations.md) and [removals](../../update/removals.md) documentation is generated from the YAML files located in @@ -131,3 +131,7 @@ Related Handbook pages: - <https://about.gitlab.com/handbook/marketing/blog/release-posts/#deprecations-removals-and-breaking-changes> - <https://about.gitlab.com/handbook/marketing/blog/release-posts/#update-the-deprecations-and-removals-docs> + +## Update the related documentation + +When features are deprecated and removed, [update the related documentation](../documentation/versions.md#deprecations-and-removals). diff --git a/doc/development/development_processes.md b/doc/development/development_processes.md index 10818b749ab..e1df3b55d06 100644 --- a/doc/development/development_processes.md +++ b/doc/development/development_processes.md @@ -80,7 +80,7 @@ In these cases, use the following workflow: - [Backend](https://about.gitlab.com/handbook/engineering/) - [Database](https://about.gitlab.com/handbook/engineering/development/database/) - [User Experience (UX)](https://about.gitlab.com/handbook/product/ux/) - - [Security](https://about.gitlab.com/handbook/engineering/security/) + - [Security](https://about.gitlab.com/handbook/security/) - [Quality](https://about.gitlab.com/handbook/engineering/quality/) - [Engineering Productivity](https://about.gitlab.com/handbook/engineering/quality/engineering-productivity/) - [Infrastructure](https://about.gitlab.com/handbook/engineering/infrastructure/) diff --git a/doc/development/distributed_tracing.md b/doc/development/distributed_tracing.md index 0ca10bc93dd..9e62f019fbf 100644 --- a/doc/development/distributed_tracing.md +++ b/doc/development/distributed_tracing.md @@ -91,7 +91,7 @@ Once the performance bar is enabled, select **Trace** in the performance bar to the Jaeger UI. The Jaeger search UI returns a query for the `Correlation-ID` of the current request. Normally, -this search should return a single trace result. Clicking this result shows the detail of the +this search should return a single trace result. Selecting this result shows the detail of the trace in a hierarchical time-line.  diff --git a/doc/development/documentation/drawers.md b/doc/development/documentation/drawers.md new file mode 100644 index 00000000000..7ede1a0b652 --- /dev/null +++ b/doc/development/documentation/drawers.md @@ -0,0 +1,59 @@ +--- +stage: none +group: Documentation Guidelines +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Create content for drawers + +In the GitLab UI, you can display help content in +[a drawer component](https://design.gitlab.com/components/drawer/). +The component for Markdown is +[in the storybook](https://gitlab-org.gitlab.io/gitlab/storybook/?path=/story/vue-shared-markdown-drawer--default). + +The component points to a Markdown file. Any time you update the Markdown +file, the contents of the drawer are updated. + +Drawer content is displayed in drawers only, and not on `docs.gitlab.com`. +The content is rendered in GitLab Flavored Markdown. + +To create this content: + +1. In the [GitLab](https://gitlab.com/gitlab-org/gitlab) repository, + go to the `/doc/drawers` folder. +1. Create a Markdown file. Use a descriptive filename. + Do not create subfolders. +1. Add the standard page metadata. Also, include: + + ```markdown + type: drawer + ``` + +1. Author the content. +1. If the page includes content that is also on a page on `docs.gitlab.com`, + on the page's metadata, include a path to the other file. For example: + + ```markdown + source: /doc/user/search/global_search/advanced_search_syntax.md + ``` + +1. Work with the developer to view the content in the drawer and + verify that the content appears correctly. + +## Drawer content guidelines + +- The headings in the file are used as headings in the drawer. + The `H1` heading is the drawer title. +- Do not include any characters other than plain text in the `H1`. +- The drawer component is narrow and not resizable. + - If you include tables, the content within should be brief. + - While no technical limitation exists on the number of characters + you can use, you should preview the drawer content to + ensure it renders well. +- To link from the drawer to other content, use absolute URLs. +- Do not include: + - Tier badges + - Version history text + - Alert boxes + - Images + - SVG icons diff --git a/doc/development/documentation/feature_flags.md b/doc/development/documentation/feature_flags.md index dae62fea603..0fab693fdee 100644 --- a/doc/development/documentation/feature_flags.md +++ b/doc/development/documentation/feature_flags.md @@ -57,6 +57,13 @@ FLAG: <This feature is not ready for production use.> ``` +A `FLAG` note renders on the GitLab documentation site as: + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../../administration/feature_flags.md) named `example_flag`. +On GitLab.com, this feature is not available. +This feature is not ready for production use. + ### Self-managed GitLab availability information | If the feature is... | Use this text | diff --git a/doc/development/documentation/index.md b/doc/development/documentation/index.md index c52b3b5adae..d52db71b633 100644 --- a/doc/development/documentation/index.md +++ b/doc/development/documentation/index.md @@ -145,68 +145,41 @@ pages. You can safely remove the `type` metadata parameter and its values. ### Batch updates for TW metadata -NOTE: -This task is an MVC, and requires significant manual preparation of the output. -While the task can be time consuming, it is still faster than doing the work -entirely manually. +The [`CODEOWNERS`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/CODEOWNERS) +file contains a list of files and the associated technical writers. -It's important to keep the [`CODEOWNERS`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/CODEOWNERS) -file in the `gitlab` project up to date with the current Technical Writing team assignments. -This information is used in merge requests that contain documentation: +When a merge request contains documentation, the information in the `CODEOWNERS` file determines: -- To populate the eligible approvers section. -- By GitLab Bot to ping reviewers for community contributions. +- The list of users in the **Approvers** section. +- The technical writer that the GitLab Bot pings for community contributions. -GitLab cannot automatically associate the stage and group metadata in our documentation -pages with the technical writer assigned to that group, so we use a Rake task to -generate entries for the `CODEOWNERS` file. Declaring code owners for pages reduces -the number of times GitLab Bot pings the entire Technical Writing team. +You can use a Rake task to update the `CODEOWNERS` file. -The `tw:codeowners` Rake task, located in [`lib/tasks/gitlab/tw/codeowners.rake`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/tasks/gitlab/tw/codeowners.rake), -contains an array of groups and their assigned technical writer. This task: +#### Update the `CODEOWNERS` file -- Outputs a line for each doc with metadata that matches a group in `lib/tasks/gitlab/tw/codeowners.rake`. - Files not matching a group are skipped. -- Adds the full path to the page, and the assigned technical writer. +To update the `CODEOWNERS` file: -To prepare an update to the `CODEOWNERS` file: +1. Open a merge request to update + [the Rake task](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/tasks/gitlab/tw/codeowners.rake) + with the latest [TW team assignments](https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments). +1. Assign the merge request to a backend maintainer for review and merge. +1. After the MR is merged, go to the root of the `gitlab` repository. +1. Run the Rake task and save the output in a file: -1. Update `lib/tasks/gitlab/tw/codeowners.rake` with the latest [TW team assignments](https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments). - Make this update in a standalone merge request, as it runs a long pipeline and - requires backend maintainer review. Make sure this is merged before you update - `CODEOWNERS` in another merge request. -1. Run the task from the root directory of the `gitlab` repository, and save the output in a file: - - ```ruby + ```shell bundle exec rake tw:codeowners > ~/Desktop/updates.md ``` -1. Open the file you just created (`~/Desktop/updates.md` in this example), and prepare the output: - - Find and replace `./` with `/`. - - Sort the lines in alphabetical (ascending) order. If you use VS Code, you can - select everything, press <kbd>F1</kbd>, type `sort`, and select **Sort lines (ascending, case insensitive**. -1. Create a new branch for your `CODEOWNERS` updates. -1. Replace the documentation-related lines in the `^[Documentation Pages]` section - with the output you prepared. +1. Open the file (for example, `~/Desktop/updates.md`) and copy everything + except the errors at the bottom of the file. +1. Open the [`CODEOWNERS`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/CODEOWNERS) + file and paste the lines into the `^[Documentation Pages]` section. WARNING: The documentation section is not the last section of the `CODEOWNERS` file. Don't delete data that isn't ours! -1. Create a commit with the raw changes. -1. From the command line, run `git diff master`. -1. In the diff, look for directory-level assignments to manually restore to the - `CODEOWNERS` file. If all files in a single directory are assigned to the same - technical writer, we simplify these entries. Remove all the lines for the individual - files, and leave a single entry for the directory, for example: `/doc/directory/ @tech.writer`. -1. In the diff, look for changes that don't match your expectations: - - New pages, or newly moved pages, show up as added lines. - - Deleted pages, and pages that are now redirects, show up as deleted lines. - - If you see an unusual number of changes to pages that all seem related, - check the metadata for the pages. A group might have been renamed and the Rake task - must be updated to match. -1. Create another commit with your manual changes, and create a second merge request - with your changes to the `CODEOWNERS` file. Assign it to a technical writing manager for review. +1. Create a merge request and assign it to a technical writing manager for review. ## Move, rename, or delete a page diff --git a/doc/development/documentation/restful_api_styleguide.md b/doc/development/documentation/restful_api_styleguide.md index dc84f3a08dd..21c8c8543ab 100644 --- a/doc/development/documentation/restful_api_styleguide.md +++ b/doc/development/documentation/restful_api_styleguide.md @@ -129,7 +129,7 @@ To deprecate an attribute: ``` To widely announce a deprecation, or if it's a breaking change, -[update the deprecations and removals documentation](../deprecation_guidelines/index.md#update-the-deprecations-and-removals-documentation). +[update the deprecations and removals documentation pages](../deprecation_guidelines/index.md#update-the-deprecations-and-removals-documentation-pages). ## Method description @@ -289,7 +289,7 @@ contains spaces in its title. Observe how spaces are escaped using the `%20` ASCII code. ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/42/issues?title=Hello%20Dude" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/42/issues?title=Hello%20GitLab" ``` Use `%2F` for slashes (`/`). diff --git a/doc/development/documentation/review_apps.md b/doc/development/documentation/review_apps.md index cb04f0909c1..3cf77fda22b 100644 --- a/doc/development/documentation/review_apps.md +++ b/doc/development/documentation/review_apps.md @@ -47,12 +47,11 @@ If you want to know the in-depth details, here's what's really happening: 1. The preview URL is shown both at the job output and in the merge request widget. You also get the link to the remote pipeline. 1. In the `gitlab-org/gitlab-docs` project, the pipeline is created and it - [skips the test jobs](https://gitlab.com/gitlab-org/gitlab-docs/blob/8d5d5c750c602a835614b02f9db42ead1c4b2f5e/.gitlab-ci.yml#L50-55) + [skips most test jobs](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/d41ca9323f762132780d2d072f845d28817a5383/.gitlab/ci/rules.gitlab-ci.yml#L101-103) to lower the build time. -1. Once the docs site is built, the HTML files are uploaded as artifacts. -1. A specific runner tied only to the docs project, runs the Review App job - that downloads the artifacts and uses `rsync` to transfer the files over - to a location where NGINX serves them. +1. After the docs site is built, the HTML files are uploaded as artifacts to + a GCP bucket (see [issue `gitlab-com/gl-infra/reliability#11021`](https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/11021) + for the implementation details). The following GitLab features are used among others: @@ -60,42 +59,26 @@ The following GitLab features are used among others: - [Multi project pipelines](../../ci/pipelines/downstream_pipelines.md#multi-project-pipelines) - [Review Apps](../../ci/review_apps/index.md) - [Artifacts](../../ci/yaml/index.md#artifacts) -- [Specific runner](../../ci/runners/runners_scope.md#prevent-a-specific-runner-from-being-enabled-for-other-projects) - [Merge request pipelines](../../ci/pipelines/merge_request_pipelines.md) ## Troubleshooting review apps -### Review app returns a 404 error +### `NoSuchKey The specified key does not exist` -If the review app URL returns a 404 error, either the site is not -yet deployed, or something went wrong with the remote pipeline. You can: +If you see the following message in a review app, either the site is not +yet deployed, or something went wrong with the downstream pipeline in `gitlab-docs`. -- Wait a few minutes and it should appear online. -- Check the manual job's log and verify the URL. If the URL is different, try the - one from the job log. -- Check the status of the remote pipeline from the link in the merge request's job output. - If the pipeline failed or got stuck, GitLab team members can ask for help in the `#docs` - chat channel. Contributors can ping a technical writer in the merge request. - -### Not enough disk space - -Sometimes the review app server is full and there is no more disk space. Each review -app takes about 570MB of disk space. +```plaintext +NoSuchKeyThe specified key does not exist.No such object: <URL> +``` -A cron job to remove review apps older than 20 days runs hourly, -but the disk space still occasionally fills up. To manually free up more space, -a GitLab technical writing team member can: +In that case, you can: -1. Navigate to the [`gitlab-docs` schedules page](https://gitlab.com/gitlab-org/gitlab-docs/-/pipeline_schedules). -1. Select the play button for the `Remove old review apps from review app server` - schedule. By default, this cleans up review apps older than 14 days. -1. Navigate to the [pipelines page](https://gitlab.com/gitlab-org/gitlab-docs/-/pipelines) - and start the manual job called `clean-pages`. - -If the job says no review apps were found in that period, edit the `CLEAN_REVIEW_APPS_DAYS` -variable in the schedule, and repeat the process above. Gradually decrease the variable -until the free disk space reaches an acceptable amount (for example, 3GB). -Remember to set it to 14 again when you're done. - -There's an issue to [migrate from the DigitalOcean server to GCP buckets](https://gitlab.com/gitlab-org/gitlab-docs/-/issues/735)), -which should solve the disk space problem. +- Wait a few minutes and the review app should appear online. +- Check the `review-docs-deploy` job's log and verify the URL. If the URL shown in the merge + request UI is different than the job log, try the one from the job log. +- Check the status of the remote pipeline from the link in the merge request's job output. + If the pipeline failed or got stuck, GitLab team members can ask for help in the `#docs` + internal Slack channel. Contributors can ping a + [technical writer](https://about.gitlab.com/handbook/product/ux/technical-writing/#designated-technical-writers) + in the merge request. diff --git a/doc/development/documentation/site_architecture/deployment_process.md b/doc/development/documentation/site_architecture/deployment_process.md index b5c3a59b0eb..18cc27adaaa 100644 --- a/doc/development/documentation/site_architecture/deployment_process.md +++ b/doc/development/documentation/site_architecture/deployment_process.md @@ -6,6 +6,14 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Documentation deployments +## Deployment environments + +The [GitLab documentation site](https://docs.gitlab.com/) is a static site hosted by [GitLab Pages](../../../user/project/pages/index.md). The deployment is done by the [Pages deploy job](#pages-deploy-job). + +The website hosts documentation only for the [currently supported](../../../policy/maintenance.md) GitLab versions. Documentation for older versions is built and uploaded as Docker images to be downloaded from [GitLab Docs archives](https://docs.gitlab.com/archives/). + +## Parts of release process + The documentation [release process](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/doc/releases.md) involves: @@ -24,10 +32,8 @@ For general information on using Docker with CI/CD pipelines, see [Docker integr ## Stable branches -Stable branches for documentation include the relevant stable branches of all the projects required to publish the entire -documentation suite. For example, the stable version of documentation for version `14.4` includes: +Pipelines for stable branches in the documentation project pull the relevant stable branches of included projects. For example, the documentation for stable version `14.4` is built from the [`14.4`](https://gitlab.com/gitlab-org/gitlab-docs/-/tree/14.4) branch of the `gitlab-docs` project, which then includes: -- The [`14.4`](https://gitlab.com/gitlab-org/gitlab-docs/-/tree/14.4) branch of the `gitlab-docs` project. - The [`14-4-stable-ee`](https://gitlab.com/gitlab-org/gitlab/-/tree/14-4-stable-ee) branch of the `gitlab` project. - The [`14-4-stable`](https://gitlab.com/gitlab-org/gitlab-runner/-/tree/14-4-stable) branch of the `gitlab-runner` project. - The [`14-4-stable`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/tree/14-4-stable) branch of the `omnibus-gitlab` project. @@ -117,7 +123,7 @@ graph TD G--"Latest `gitlab-docs:latest` image<br>pushed up"-->H ``` -## Documentation Pages deployment +## Pages deploy job [GitLab Docs](https://docs.gitlab.com) is a [Pages site](../../../user/project/pages/index.md) and documentation updates for it must be deployed to become available. @@ -176,7 +182,7 @@ Dockerfiles to build and deploy <https://docs.gitlab.com>. It is heavily inspire Although build images are built automatically via GitLab CI/CD, you can build and tag all tooling images locally: -1. Make sure you have [Docker installed](https://docs.docker.com/install/). +1. Make sure you have [Docker installed](https://docs.docker.com/get-docker/). 1. Make sure you're in the `dockerfiles/` directory of the `gitlab-docs` repository. 1. Build the images: diff --git a/doc/development/documentation/site_architecture/global_nav.md b/doc/development/documentation/site_architecture/global_nav.md index 37d10b16fcd..d95ca720119 100644 --- a/doc/development/documentation/site_architecture/global_nav.md +++ b/doc/development/documentation/site_architecture/global_nav.md @@ -234,7 +234,7 @@ below the doc link: external_url: true ``` -All nav links are clickable. If the higher-level link does not have a link +All nav links are selectable. If the higher-level link does not have a link of its own, it must link to its first sub-item link, mimicking the navigation in GitLab. This must be avoided so that we don't have duplicated links nor two `.active` links at the same time. diff --git a/doc/development/documentation/styleguide/index.md b/doc/development/documentation/styleguide/index.md index d629bc0b87e..ef934186981 100644 --- a/doc/development/documentation/styleguide/index.md +++ b/doc/development/documentation/styleguide/index.md @@ -18,11 +18,22 @@ In addition to this page, the following resources can help you craft and contrib - [Recommended word list](word_list.md) - [Doc style and consistency testing](../testing.md) - [Guidelines for UI error messages](https://design.gitlab.com/content/error-messages/) +- [Documentation global navigation](../site_architecture/global_nav.md) - [GitLab Handbook style guidelines](https://about.gitlab.com/handbook/communication/#writing-style-guidelines) - [Microsoft Style Guide](https://learn.microsoft.com/en-us/style-guide/welcome/) - [Google Developer Documentation Style Guide](https://developers.google.com/style) - [Recent updates to this guide](https://gitlab.com/dashboard/merge_requests?scope=all&state=merged&label_name[]=tw-style¬[label_name][]=docs%3A%3Afix) +## The GitLab voice + +The GitLab brand guidelines define the +[voice used by the larger organization](https://design.gitlab.com/brand/overview/#tone-of-voice). + +Building on that guidance, the voice in the GitLab documentation strives to be concise, +direct, and precise. The goal is to provide information that's easy to search and scan. + +The voice in the documentation should be conversational but brief, friendly but succinct. + ## Documentation is the single source of truth (SSOT) The GitLab documentation is the SSOT for all @@ -125,6 +136,23 @@ Maintaining a knowledge base separate from the documentation would be against the documentation-first methodology, because the content would overlap with the documentation. +## Writing for localization + +The GitLab documentation is not localized, but we follow guidelines that +help benefit translation. For example, we: + +- Write in [active voice](word_list.md#active-voice). +- Write in [present tense](word_list.md#future-tense). +- Avoid words that can be translated incorrectly, like: + - [since and because](word_list.md#since) + - [once and after](word_list.md#once) + - [it](word_list.md#it) +- Avoid [ing](word_list.md#-ing-words) words. + +[The GitLab voice](#the-gitlab-voice) dictates that we write clearly and directly, +and with translation in mind. [The word list](word_list.md) and our Vale rules +also aid in consistency, which is important for localization. + ## Markdown All GitLab documentation is written using [Markdown](https://en.wikipedia.org/wiki/Markdown). @@ -143,26 +171,17 @@ Hard-coded HTML is valid, although it's discouraged from being used. HTML is per - Special styling is required. - Reviewed and approved by a technical writer. -### Headings in Markdown +### Heading levels in Markdown Each documentation page begins with a level 1 heading (`#`). This becomes the `h1` element when the page is rendered to HTML. There can be only **one** level 1 heading per page. - For each subsection, increment the heading level. In other words, increment the number of `#` characters - in front of the heading. -- Avoid headings greater than `H5` (`#####`). If you need more than five heading levels, move the topics to a new page instead. - Headings greater than `H5` do not display in the right sidebar navigation. + in front of the topic title. +- Avoid heading levels greater than `H5` (`#####`). If you need more than five heading levels, move the topics to a new page instead. + Heading levels greater than `H5` do not display in the right sidebar navigation. - Do not skip a level. For example: `##` > `####`. -- Leave one blank line before and after the heading. - -When you change heading text, the anchor link changes. To avoid broken links: - -- Do not use step numbers in headings. -- When possible, do not use words that might change in the future. - -Also, do not use links as part of heading text. - -See also [heading guidelines for specific topic types](../structure.md). +- Leave one blank line before and after the topic title. ### Backticks in Markdown @@ -221,9 +240,9 @@ GitLab documentation should be clear and easy to understand. As a company, we tend toward lowercase. -#### Headings +#### Topic titles -Use sentence case. For example: +Use sentence case for topic titles. For example: - `# Use variables to configure pipelines` - `## Use the To-Do List` @@ -329,7 +348,7 @@ Some contractions, however, should be avoided: ### Acronyms If you use an acronym, spell it out on first use on a page. You do not need to spell it out more than once on a page. -When possible, try to avoid acronyms in headings. +When possible, try to avoid acronyms in topic titles. ### Numbers @@ -374,22 +393,31 @@ You can use italics when you are introducing a term for the first time. Otherwis ### Punctuation -Follow these guidelines for punctuation: +Follow these guidelines for punctuation. <!-- vale gitlab.Repetition = NO --> - End full sentences with a period. -- Use one space between sentences. -- Do not use semicolons. Use two sentences instead. -- Do not use double spaces. (Tested in [`SentenceSpacing.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/SentenceSpacing.yml).) -- Do not use non-breaking spaces. Use standard spaces instead. (Tested in [`lint-doc.sh`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/scripts/lint-doc.sh).) -- Do not use tabs for indentation. Use spaces instead. You can configure your code editor to output spaces instead of tabs when pressing the tab key. - Use serial (Oxford) commas before the final **and** or **or** in a list of three or more items. (Tested in [`OxfordComma.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/OxfordComma.yml).) -- Avoid dashes. Use separate sentences, or commas, instead. -- Do not use typographer's ("curly") quotes. Use straight quotes instead. (Tested in [`NonStandardQuotes.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/NonStandardQuotes.yml).) <!-- vale gitlab.Repetition = YES --> +When spacing content: + +- Use one space between sentences. (Use of more than one space is tested in [`SentenceSpacing.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/SentenceSpacing.yml).) +- Do not use non-breaking spaces. Use standard spaces instead. (Tested in [`lint-doc.sh`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/scripts/lint-doc.sh).) +- Do not use tabs for indentation. Use spaces instead. You can configure your code editor to output spaces instead of tabs when pressing the <kbd>Tab</kbd> key. + +<!-- vale gitlab.NonStandardQuotes = NO --> + +Do not use these punctuation characters: + +- `;` (semicolon): Use two sentences instead. +- `–` (en dash) or `—` (em dash): Use separate sentences, or commas, instead. +- `“` `”` `‘` `’`: Double or single typographer's ("curly") quotation marks. Use straight quotes instead. (Tested in [`NonStandardQuotes.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/NonStandardQuotes.yml).) + +<!-- vale gitlab.NonStandardQuotes = YES --> + ### Placeholder text You might want to provide a command or configuration that @@ -677,22 +705,27 @@ use a URL like `https://docs.gitlab.com/charts/`. ### Anchor links -Each heading has an anchor link. For example, a topic with the title +Each topic title has an anchor link. For example, a topic with the title `## This is an example` has the anchor `#this-is-an-example`. -The first topic on a page (the `h1`) has an anchor link, +The first topic title on a page (the `h1`) has an anchor link, but do not use it. Link to the page instead. -If a heading has a [product tier badge](#product-tier-badges), -do not include it in the anchor link. For example, for the heading +If a topic title has a [product tier badge](#product-tier-badges), +do not include it in the anchor link. For example, for the topic `## This is an example **(FREE)**`, use the anchor `#this-is-an-example`. With Kramdown, you can add a custom ID to an HTML element, but these IDs don't work in `/help`, so you should not use them. +When you change topic title text, the anchor link changes. To avoid broken links: + +- Do not use step numbers in topic titles. +- When possible, do not use words that might change in the future. + #### Changing links and titles -When you change a heading, the anchor link changes. To ensure you update +When you change a topic title, the anchor link changes. To ensure you update any related links, search these directories: - `doc/*` @@ -823,7 +856,7 @@ To open either project or group settings: ```markdown 1. On the top bar, select **Main menu**, and: - - For a project, select ***Projects** and find your project. + - For a project, select **Projects** and find your project. - For a group, select **Groups** and find your group. 1. On the left sidebar, select **Settings > CI/CD**. 1. Expand **General pipelines**. @@ -1195,7 +1228,7 @@ Instead of adding a note: - Re-write the sentence as part of a paragraph. - Put the information into its own paragraph. -- Put the content under a new subheading. +- Put the content under a new topic title. If you must use a note, use this format: @@ -1311,6 +1344,7 @@ It renders on the GitLab documentation site as: > - Second item in the list ## Tabs +<!-- markdownlint-disable tabs-blank-lines --> On the docs site, you can format text so it's displayed as tabs. @@ -1329,6 +1363,7 @@ Here's some other content in tab two. ::EndTabs ``` +<!-- markdownlint-enable tabs-blank-lines --> This code renders on the GitLab documentation site as: @@ -1378,25 +1413,25 @@ When names change, it is more complicated to search or grep text that has line b ### Product tier badges -Tier badges are displayed as orange text next to a heading. These badges link to the GitLab +Tier badges are displayed as orange text next to a topic title. These badges link to the GitLab pricing page. For example:  You must assign a tier badge: -- To all H1 topic headings, except the pages under `doc/development/*`. -- To topic headings that don't apply to the same tier as the H1. +- To all H1 topic titles, except the pages under `doc/development/*`. +- To topic titles that don't apply to the same tier as the H1. -To add a tier badge to a heading, add the relevant tier badge -after the heading text. For example: +To add a tier badge to a topic title, add the relevant tier badge +after the title text. For example: ```markdown -# Heading title **(FREE)** +# Topic title **(FREE)** ``` Do not add tier badges inline with other text, except for [API attributes](../restful_api_styleguide.md). -The single source of truth for a feature should be the heading where the +The single source of truth for a feature should be the topic where the functionality is described. #### Available product tier badges diff --git a/doc/development/documentation/styleguide/word_list.md b/doc/development/documentation/styleguide/word_list.md index 65ad8dea688..d28972a644b 100644 --- a/doc/development/documentation/styleguide/word_list.md +++ b/doc/development/documentation/styleguide/word_list.md @@ -35,7 +35,7 @@ Don't use backticks. ## 2FA, two-factor authentication -Spell out **two-factor authentication** in sentence case for the first use and in section headings, and **2FA** +Spell out **two-factor authentication** in sentence case for the first use and in topic titles, and **2FA** thereafter. If the first word in a sentence, do not capitalize `factor` or `authentication`. For example: - Two-factor authentication (2FA) helps secure your account. Set up 2FA when you first log in. @@ -64,6 +64,23 @@ When you create a user, you choose an access level: **Regular**, **Auditor**, or Capitalize these words when you refer to the UI. Otherwise use lowercase. +## active voice + +Use active voice instead of passive. + +Use: + +- The contributor writes the documentation. + +Instead of: + +- The documentation is written by contributors. + +NOTE: +If you can add the phrase "by zombies" to the phrase, +the construction is passive. For example, `The button is selected by zombies` +is passive. `Zombies select the button` is active. + ## administrator Use **administrator access** instead of **admin** when talking about a user's access level. @@ -460,6 +477,10 @@ Do not use **foo** in product documentation. You can use it in our API and contr When possible, use present tense instead of future tense. For example, use **after you execute this command, GitLab displays the result** instead of **after you execute this command, GitLab will display the result**. ([Vale](../testing.md#vale) rule: [`FutureTense.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/FutureTense.yml)) +## GB, gigabytes + +For **GB** and **MB**, follow the [Microsoft guidance](https://learn.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/term-collections/bits-bytes-terms). + ## Geo Use title case for **Geo**. @@ -570,6 +591,15 @@ Do not use Latin abbreviations. Use **that is** instead. ([Vale](../testing.md#v Do not use **in order to**. Use **to** instead. ([Vale](../testing.md#vale) rule: [`Wordy.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/Wordy.yml)) +## -ing words + +Remove **-ing** words whenever possible. They can be difficult to translate, +and more precise terms are usually available. For example: + +- Instead of **The files using storage are deleted**, use **The files that use storage are deleted**. +- Instead of **Delete files using the Edit button**, use **Delete files by using the Edit button**. +- Instead of **Replicating your server is required**, use **You must replicate your server**. + ## issue Use lowercase for **issue**. @@ -582,6 +612,21 @@ Use lowercase for **issue board**. Use lowercase for **issue weights**. +## it + +When you use the word **it**, ensure the word it refers to is obvious. +If it's not obvious, repeat the word rather than using **it**. + +Use: + +- The field returns a connection. The field accepts four arguments. + +Instead of: + +- The field returns a connection. It accepts four arguments. + +See also [this, these, that, those](#this-these-that-those). + ## job Do not use **build** to be synonymous with **job**. A job is defined in the `.gitlab-ci.yml` file and runs as part of a pipeline. @@ -692,6 +737,10 @@ Do not use `master`. Use `main` when you need a sample [default branch name](#de **Might** means something has the probability of occurring. **May** gives permission to do something. Consider **can** instead of **may**. +## MB, megabytes + +For **MB** and **GB**, follow the [Microsoft guidance](https://learn.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/term-collections/bits-bytes-terms). + ## me, myself, mine Do not use first-person singular. Use **you**, **we**, or **us** instead. ([Vale](../testing.md#vale) rule: [`FirstPerson.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/FirstPerson.yml)) @@ -720,24 +769,18 @@ Do not use **navigate**. Use **go** instead. For example: ## need to, should -Try to avoid **needs to**, because it's wordy. Avoid **should** when you can be more specific. If something is required, use **must**. +Try to avoid **needs to**, because it's wordy. If something is recommended, use **should** instead. If something is required, use **must**. Use: -- You must set the variable. -- Set the variable. +- You should set the variable. (recommended) +- You must set the variable. (required) +- Set the variable. (required) Instead of: - You need to set the variable. - -**Should** is acceptable for recommended actions or items, or in cases where an event may not -happen. For example: - -- Although you can configure the installation manually, you should use the express configuration to - avoid complications. -- You should see a success message in the console. Contact support if an error message appears - instead. +- We recommend that you set the variable. ## note that @@ -811,7 +854,7 @@ When writing about the Owner role: - Instead of: if you are an owner Do not use bold. - + Do not use **Owner permissions**. A user who is assigned the Owner role has a set of associated permissions. An Owner is the highest role a user can have. @@ -893,6 +936,10 @@ Instead of: - Select **Create user** or **Save changes** if you created a new user or edited an existing one respectively. +## review app + +Use lowercase for **review app**. + ## roles Do not use **roles** and [**permissions**](#permissions) interchangeably. Each user is assigned a role. Each role includes a set of permissions. @@ -981,7 +1028,11 @@ Use **setup** as a noun, and **set up** as a verb. For example: ## sign in -Use **sign in** instead of **sign on** or **log on** or **log in**. If the user interface has different words, use those. +Use **sign in** or **sign in to**. + +Do not use **sign on** or **sign into**, or **log on**, **log in**, or **log into**. + +If the user interface has different words, use those. You can use **single sign-on**. @@ -1108,6 +1159,14 @@ For example: See also [**enter**](#enter). +## units of measurement + +Use a space between the number and the unit of measurement. For example, **128 GB**. +([Vale](../testing.md#vale) rule: [`Units.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/Units.yml)) + +For other guidance, follow +[the Microsoft style guidelines](https://learn.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/term-collections/bits-bytes-terms). + ## update Use **update** for installing a newer **patch** version of the software only. For example: @@ -1173,7 +1232,26 @@ Instead of: - We created a feature for you to add widgets. -One exception: You can use **we recommend** instead of **it is recommended** or **GitLab recommends**. ([Vale](../testing.md#vale) rule: [`SubstitutionSuggestions.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/SubstitutionSuggestions.yml)) +([Vale](../testing.md#vale) rule: [`SubstitutionSuggestions.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/SubstitutionSuggestions.yml)) + +## while + +Use **while** to refer only to something occurring in time. For example, +**Leave the window open while the process runs.** + +Do not use **while** for comparison. For example, use: + +- Job 1 can run quickly. However, job 2 is more precise. + +Instead of: + +- While job 1 can run quickly, job 2 is more precise. + +For details, see the [Microsoft style guide](https://learn.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/w/while). + +## whilst + +Do not use **whilst**. Use [while](#while) instead. **While** is more succinct and easier for non-native English speakers to understand. ## whitelist @@ -1203,5 +1281,19 @@ Instead of: - Users can configure a pipeline. +## you can + +When possible, start sentences with an active verb instead of **you can**. +For example: + +- Use code review analytics to view merge request data. +- Create a board to organize your team tasks. +- Configure variables to restrict pushes to a repository. + +Use **you can** for optional actions. For example: + +- Use code review analytics to view metrics per merge request. You can also use the API. +- Enter the name and value pairs. You can add up to 20 pairs per streaming destination. + <!-- vale on --> <!-- markdownlint-enable --> diff --git a/doc/development/documentation/testing.md b/doc/development/documentation/testing.md index c801bb9f877..8b8f281d7c1 100644 --- a/doc/development/documentation/testing.md +++ b/doc/development/documentation/testing.md @@ -86,7 +86,7 @@ job, which runs two types of link checks. In both cases, links with destinations that begin with `http` or `https` are considered external links, and skipped: - `bundle exec nanoc check internal_links`: Tests links to internal pages. -- `bundle exec nanoc check internal_anchors`: Tests links to subheadings (anchors) on internal pages. +- `bundle exec nanoc check internal_anchors`: Tests links to topic title anchors on internal pages. Failures from these tests are displayed at the end of the test results in the **Issues found!** area. For example, failures in the `internal_anchors` test follow this format: @@ -104,7 +104,7 @@ For example, failures in the `internal_anchors` test follow this format: - **Destination**: The full path to the file not found by the test. To find the file in the `gitlab` repository, replace `/tmp/gitlab-docs/public/ee` with `doc`, and `.html` with `.md`. - **Link**: The actual link the script attempted to find. -- **Anchor**: If present, the subheading (anchor) the script attempted to find. +- **Anchor**: If present, the topic title anchor the script attempted to find. Check for multiple instances of the same broken link on each page reporting an error. Even if a specific broken link appears multiple times on a page, the test reports it only once. diff --git a/doc/development/documentation/topic_types/concept.md b/doc/development/documentation/topic_types/concept.md index dfd003b642d..7be6bef4fad 100644 --- a/doc/development/documentation/topic_types/concept.md +++ b/doc/development/documentation/topic_types/concept.md @@ -32,13 +32,13 @@ Remember, if you start to describe about another concept, stop yourself. Each concept should be about one concept only. ``` -## Concept headings +## Concept topic titles -For the heading text, use a noun. For example, `Widgets` or `GDK dependency management`. +For the title text, use a noun. For example, `Widgets` or `GDK dependency management`. If a noun is ambiguous, you can add a gerund. For example, `Documenting versions` instead of `Versions`. -Avoid these heading titles: +Avoid these topic titles: - `Overview` or `Introduction`. Instead, use a more specific noun or phrase that someone would search for. diff --git a/doc/development/documentation/topic_types/index.md b/doc/development/documentation/topic_types/index.md index 8403fd26517..8e8c474ce3c 100644 --- a/doc/development/documentation/topic_types/index.md +++ b/doc/development/documentation/topic_types/index.md @@ -20,11 +20,10 @@ The acronym refers to the first letter of each topic type. In general, each page in the GitLab documentation contains multiple topics. Each topic on a page should be recognizable as a specific topic type. -## Other topic types +In addition to the four primary topic types, we also have a page type for +[Tutorials](tutorial.md) and [Get started](#get-started). -In addition to the four primary topic types, we have a few other types. - -### Related topics +## Related topics If inline links are not sufficient, you can create a topic called **Related topics** and include an unordered list of related topics. This topic should be above the Troubleshooting section. @@ -36,57 +35,7 @@ and include an unordered list of related topics. This topic should be above the - [Trigger a pipeline manually](link-to-topic). ``` -### Tutorials - -A tutorial is page that contains an end-to-end walkthrough of a complex workflow or scenario. -In general, you might consider using a tutorial when: - -- The workflow requires a number of sequential steps where each step consists - of sub-steps. -- The steps cover a variety of GitLab features or third-party tools. - -Tutorials are learning aids that complement our core documentation. -They do not introduce new features. -Always use the primary [topic types](#documentation-topic-types-ctrt) to document new features. - -Tutorials should be in this format: - -```markdown -# Title (starts with "Tutorial:" followed by an active verb, like "Tutorial: Create a website") - -A paragraph that explains what the tutorial does, and the expected outcome. - -To create a website: - -1. [Do the first task](#do-the-first-task) -1. [Do the second task](#do-the-second-task) - -Prerequisites (optional): - -- Thing 1 -- Thing 2 -- Thing 3 - -## Do the first task - -To do step 1: - -1. First step. -1. Another step. -1. Another step. - -## Do the second task - -Before you begin, make sure you have [done the first task](#do-the-first-task). - -To do step 2: - -1. First step. -1. Another step. -1. Another step. -``` - -### Get started +## Get started A get started page is a set of steps to help a user get set up quickly to use a single GitLab feature or tool. @@ -110,21 +59,21 @@ consider using subsections for each distinct task. In the left nav, use `Get started` as the text. On the page itself, spell out the full name. For example, `Get started with application security`. -### Topics and resources +## Topics and resources Some pages are solely a list of links to other documentation. We do not encourage this page type. Lists of links can get out-of-date quickly and offer little value to users, who prefer to search to find information. -## Heading text guidelines +## Topic text guidelines -In general, for heading text: +In general, for topic text: - Be clear and direct. Make every word count. - Use articles and prepositions. - Follow [capitalization](../styleguide/index.md#capitalization) guidelines. -- Do not repeat text from earlier headings. For example, if the page is about merge requests, +- Do not repeat text from earlier topic titles. For example, if the page is about merge requests, instead of `Troubleshooting merge requests`, use only `Troubleshooting`. -See also [guidelines for headings in Markdown](../styleguide/index.md#headings-in-markdown). +See also [guidelines for heading levels in Markdown](../styleguide/index.md#heading-levels-in-markdown). diff --git a/doc/development/documentation/topic_types/reference.md b/doc/development/documentation/topic_types/reference.md index e7ee8b20925..8bb89f4c210 100644 --- a/doc/development/documentation/topic_types/reference.md +++ b/doc/development/documentation/topic_types/reference.md @@ -19,11 +19,11 @@ Introductory sentence. | **Name** | Descriptive sentence about the setting. | ``` -## Reference headings +## Reference topic titles -Reference headings are usually nouns. +Reference topic titles are usually nouns. -Avoid these heading titles: +Avoid these topic titles: - `Important notes`. Instead, incorporate this information closer to where it belongs. For example, this information might be a prerequisite diff --git a/doc/development/documentation/topic_types/task.md b/doc/development/documentation/topic_types/task.md index 60508fbf6ee..78d670a16d6 100644 --- a/doc/development/documentation/topic_types/task.md +++ b/doc/development/documentation/topic_types/task.md @@ -52,14 +52,23 @@ To create an issue: The issue is created. You can view it by going to **Issues > List**. ``` -## Task headings +## Task topic titles -For the heading text, use the structure `active verb` + `noun`. +For the title text, use the structure `active verb` + `noun`. For example, `Create an issue`. If you have several tasks on a page that share prerequisites, you can use the title `Prerequisites` and link to it. +## Task introductions + +To start the task topic, use the structure `active verb` + `noun`, and +provide context about the action. +For example, `Create an issue when you want to track bugs or future work`. + +To start the task steps, use a succinct action followed by a colon. +For example, `To create an issue:` + ## Related topics - [View the format for writing task steps](../styleguide/index.md#navigation). diff --git a/doc/development/documentation/topic_types/troubleshooting.md b/doc/development/documentation/topic_types/troubleshooting.md index e2136de2e06..70475eb0ccf 100644 --- a/doc/development/documentation/topic_types/troubleshooting.md +++ b/doc/development/documentation/topic_types/troubleshooting.md @@ -6,10 +6,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Troubleshooting topic type -Troubleshooting topics should be the last topics on a page. +Troubleshooting topics should be the final topics on a page. If a page has more than five troubleshooting topics, put the content on a separate page that has troubleshooting information exclusively. Name the page `Troubleshooting <feature>` -and in the left nav, use the word `Troubleshoot` only. +and in the left nav, use the word `Troubleshooting` only. Troubleshooting can be one of three types. @@ -46,11 +46,22 @@ The workaround is... If multiple causes or workarounds exist, consider putting them into a table format. If you use the exact error message, surround it in backticks so it's styled as code. -## Troubleshooting headings +## Troubleshooting topic titles -For the heading of a **Troubleshooting reference** topic: +For the title of a **Troubleshooting reference** topic: - Consider including at least a partial error message. - Use fewer than 70 characters. +- Do not use links in the title. If you do not put the full error in the title, include it in the body text. + +## Rails console write functions + +If the troubleshooting suggestion includes a function that changes data on the GitLab instance, +add the following warning: + +```markdown +WARNING: +Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. +``` diff --git a/doc/development/documentation/topic_types/tutorial.md b/doc/development/documentation/topic_types/tutorial.md new file mode 100644 index 00000000000..1b1426a0465 --- /dev/null +++ b/doc/development/documentation/topic_types/tutorial.md @@ -0,0 +1,102 @@ +--- +stage: none +group: Style Guide +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Tutorial page type + +A tutorial is page that contains an end-to-end walkthrough of a complex workflow or scenario. +In general, you might consider using a tutorial when: + +- The workflow requires a number of sequential steps where each step consists + of sub-steps. +- The steps cover a variety of GitLab features or third-party tools. + +Tutorials are learning aids that complement our core documentation. +They do not introduce new features. +Always use the primary [topic types](index.md) to document new features. + +## Tutorial format + +Tutorials should be in this format: + +```markdown +# Title (starts with "Tutorial:" followed by an active verb, like "Tutorial: Create a website") + +A paragraph that explains what the tutorial does, and the expected outcome. + +To create a website: + +1. [Do the first task](#do-the-first-task) +1. [Do the second task](#do-the-second-task) + +Prerequisites (optional): + +- Thing 1 +- Thing 2 +- Thing 3 + +## Do the first task + +To do step 1: + +1. First step. +1. Another step. +1. Another step. + +## Do the second task + +Before you begin, make sure you have [done the first task](#do-the-first-task). + +To do step 2: + +1. First step. +1. Another step. +1. Another step. +``` + +An example of a tutorial that follows this format is +[Tutorial: Make your first Git commit](../../../tutorials/make_your_first_git_commit.md). + +## Tutorial page title + +Start the page title with `Tutorial:` followed by an active verb, like `Tutorial: Create a website`. + +In the left nav, use the full page title. Do not abbreviate it. +Put the text in quotes so the pipeline will pass. For example, +`"Tutorial: Make your first Git commit"`. + +On [the **Learn GitLab with tutorials** page](../../../tutorials/index.md), +do not use `Tutorial` in the title. + +## Screenshots + +You can include screenshots in a tutorial to illustrate important steps in the process. +In the core product documentation, you should [use screenshots sparingly](../styleguide/index.md#images). +However, in tutorials, screenshots can help users understand where they are in a complex process. + +Try to balance the number of screenshots in the tutorial so they don't disrupt +the narrative flow. For example, do not put one large screenshot in the middle of the tutorial. +Instead, put multiple, smaller screenshots throughout. + +## Tutorial voice + +Use a friendlier tone than you would for other topic types. For example, +you can: + +- Add encouraging or congratulatory phrases after tasks. +- Use future tense from time to time, especially when you're introducing + steps. For example, `Next, you will associate your issues with your epics`. +- Be more conversational. For example, `This task might take a while to complete`. + +## Metadata + +On pages that are tutorials, add the most appropriate `stage:` and `group:` metadata at the top of the file. +If the majority of the content does not align with a single group, specify `none` for the stage +and `Tutorials` for the group: + +```plaintext +stage: none +group: Tutorials +``` diff --git a/doc/development/documentation/versions.md b/doc/development/documentation/versions.md index 12381b84c2c..030bdec0361 100644 --- a/doc/development/documentation/versions.md +++ b/doc/development/documentation/versions.md @@ -31,7 +31,7 @@ You do not need to add version information on the pages in the `/development` di ### Add a **Version history** item -If all content in a topic is related, add a version history item after the topic heading. +If all content in a topic is related, add a version history item after the topic title. For example: ```markdown diff --git a/doc/development/documentation/workflow.md b/doc/development/documentation/workflow.md index 85d3d5e9cfc..9d8d25607c8 100644 --- a/doc/development/documentation/workflow.md +++ b/doc/development/documentation/workflow.md @@ -153,7 +153,7 @@ Ensure the following if skipping an initial Technical Writer review: - [Product badges](styleguide/index.md#product-tier-badges) are applied. - The GitLab [version](versions.md) that introduced the feature is included. -- Changes to headings don't affect in-app hyperlinks. +- Changes to topic titles don't affect in-app hyperlinks. - Specific [user permissions](../../user/permissions.md) are documented. - New documents are linked from higher-level indexes, for discoverability. - The style guide is followed: diff --git a/doc/development/ee_features.md b/doc/development/ee_features.md index 2516196d2e0..14df73b8779 100644 --- a/doc/development/ee_features.md +++ b/doc/development/ee_features.md @@ -120,24 +120,32 @@ To do so: ### Simulate a SaaS instance -If you're developing locally and need your instance to act like the SaaS version of the product, -you can simulate SaaS by exporting an environment variable: +If you're developing locally and need your instance to simulate the SaaS (GitLab.com) +version of the product: -```shell -export GITLAB_SIMULATE_SAAS=1 -``` +1. Export this environment variable: + + ```shell + export GITLAB_SIMULATE_SAAS=1 + ``` -There are many ways to pass an environment variable to your local GitLab instance. -For example, you can create a `env.runit` file in the root of your GDK with the above snippet. + There are many ways to pass an environment variable to your local GitLab instance. + For example, you can create an `env.runit` file in the root of your GDK with the above snippet. -#### Allow use of licensed EE feature +1. Enable **Allow use of licensed EE features** to make licensed EE features available to projects + only if the project namespace's plan includes the feature. -To enable plans per namespace turn on the `Allow use of licensed EE features` option from the settings page. -This will make licensed EE features available to projects only if the project namespace's plan includes the feature -or if the project is public. To enable it: + 1. Visit **Admin > Settings > General**. + 1. Expand **Account and limit**. + 1. Select the **Allow use of licensed EE features** checkbox. + 1. Click **Save changes**. -1. If you are developing locally, follow the steps in [Simulate a SaaS instance](#simulate-a-saas-instance) to make the option available. -1. Visit Admin > Settings > General > "Account and limit" and enable "Allow use of licensed EE features". +1. Ensure that the group for which you want to test the EE feature, is actually using an EE plan: + 1. On the top bar, select **Main menu > Admin**. + 1. On the left sidebar, select **Overview > Groups**. + 1. Identify the group you want to modify, and select **Edit**. + 1. Scroll to **Permissions and group features**. For **Plan**, select `Ultimate`. + 1. Select **Save changes**. ### Run CI pipelines in a FOSS context @@ -147,7 +155,7 @@ FOSS context as well. To run pipelines in both contexts, add the `~"pipeline:run-as-if-foss"` label to the merge request. -See the [As-if-FOSS jobs](pipelines.md#as-if-foss-jobs) pipelines documentation for more information. +See the [As-if-FOSS jobs](pipelines/index.md#as-if-foss-jobs) pipelines documentation for more information. ## Separation of EE code in the backend @@ -692,7 +700,7 @@ module EE prepended do params do - requires :id, type: String, desc: 'The ID of a project' + requires :id, types: [String, Integer], desc: 'The ID or URL-encoded path of the project' end resource :projects, requirements: ::API::API::NAMESPACE_OR_PROJECT_REQUIREMENTS do # ... diff --git a/doc/development/event_store.md b/doc/development/event_store.md index b9200d3be25..10dc0b1a7a9 100644 --- a/doc/development/event_store.md +++ b/doc/development/event_store.md @@ -351,6 +351,12 @@ RSpec.describe MergeRequests::UpdateHeadPipelineWorker do let(:event) { pipeline_created_event } end + # This shared example ensures that an published event is ignored. This might be useful for + # conditional dispatch testing. + it_behaves_like 'ignores the published event' do + let(:event) { pipeline_created_event } + end + it 'does something' do # This helper directly executes `perform` ensuring that `handle_event` is called correctly. consume_event(subscriber: described_class, event: pipeline_created_event) diff --git a/doc/development/fe_guide/graphql.md b/doc/development/fe_guide/graphql.md index 9ed3e551ff2..28ea84301a6 100644 --- a/doc/development/fe_guide/graphql.md +++ b/doc/development/fe_guide/graphql.md @@ -907,7 +907,7 @@ For example, we have a query like this: query searchGroupsWhereUserCanTransfer { currentUser { id - groups { + groups(after: 'somecursor') { nodes { id fullName @@ -920,9 +920,7 @@ query searchGroupsWhereUserCanTransfer { } ``` -Here, the `groups` field doesn't have a good candidate for `keyArgs`: both -`nodes` and `pageInfo` will be updated when we're fetching a second page. -Setting `keyArgs` to `false` makes the update work as intended: +Here, the `groups` field doesn't have a good candidate for `keyArgs`: we don't want to account for `after` argument because it will change on requesting subsequent pages. Setting `keyArgs` to `false` makes the update work as intended: ```javascript typePolicies: { diff --git a/doc/development/fe_guide/merge_request_widget_extensions.md b/doc/development/fe_guide/merge_request_widget_extensions.md index e5e813bf921..61d3e79a080 100644 --- a/doc/development/fe_guide/merge_request_widget_extensions.md +++ b/doc/development/fe_guide/merge_request_widget_extensions.md @@ -85,7 +85,7 @@ special formatting is required. When the extension receives this data, it is set to `collapsedData`. You can access `collapsedData` in any computed property or method. -When the user clicks **Expand**, the `fetchFullData` method is called. This method +When the user selects **Expand**, the `fetchFullData` method is called. This method also gets called with the props as an argument. This method **must** also return the full data. However, this data must be correctly formatted to match the format mentioned in the data structure section. diff --git a/doc/development/fe_guide/registry_architecture.md b/doc/development/fe_guide/registry_architecture.md index a87d38634d5..d86f8416db6 100644 --- a/doc/development/fe_guide/registry_architecture.md +++ b/doc/development/fe_guide/registry_architecture.md @@ -71,7 +71,7 @@ main pieces of the desired UI and UX of a registry page. The most important comp secondary content, right primary and secondary content, right action, and details slots. - `metadata-item`: represents one piece of metadata, with an icon or a link. Used primarily in the title area. -- `persisted-dropdown-selection`: represents a dropdown menu that stores the user selection in the +- `persisted-dropdown-selection`: represents a menu that stores the user selection in the `localStorage`. - `registry-search`: implements `gl-filtered-search` with a sorting section on the right. - `title-area`: implements the top title area of the registry. Includes: a main title, an avatar, a diff --git a/doc/development/fe_guide/style/scss.md b/doc/development/fe_guide/style/scss.md index aacf07fda92..98f74813231 100644 --- a/doc/development/fe_guide/style/scss.md +++ b/doc/development/fe_guide/style/scss.md @@ -176,7 +176,7 @@ To check if any warnings are produced by your changes, run `yarn lint:stylelint` catch any warnings. If the Rake task is throwing warnings you don't understand, SCSS Lint's -documentation includes [a full list of their rules](https://stylelint.io/user-guide/rules/list/). +documentation includes [a full list of their rules](https://stylelint.io/user-guide/rules/). ### Fixing issues diff --git a/doc/development/fe_guide/view_component.md b/doc/development/fe_guide/view_component.md index 662d1ad32fc..b61c23cadef 100644 --- a/doc/development/fe_guide/view_component.md +++ b/doc/development/fe_guide/view_component.md @@ -15,7 +15,7 @@ watch this [introduction video](https://youtu.be/akRhUbvtnmo). ## Browse components with Lookbook -We have a [Lookbook](https://github.com/allmarkedup/lookbook) in [http://gdk.test:3000/rails/lookbook](http://gdk.test:3000/rails/lookbook) (only available in development mode) to browse and interact with ViewComponent previews. +We have a [Lookbook](https://github.com/allmarkedup/lookbook) in `http://gdk.test:3000/rails/lookbook` (only available in development mode) to browse and interact with ViewComponent previews. ## Pajamas components @@ -24,12 +24,12 @@ available as a ViewComponent in `app/components/pajamas`. NOTE: We are still in the process of creating these components, so not every Pajamas component is available as ViewComponent. -Reach out to the [Foundations team](https://about.gitlab.com/handbook/engineering/development/dev/ecosystem/foundations/) +Reach out to the [Foundations team](https://about.gitlab.com/handbook/engineering/development/dev/manage/foundations/) if the component you are looking for is not yet available. ### Available components -Consider this list a best effort. The full list can be found in [`app/components/pajamas`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/app/components/pajamas). Also see [our Lookbook](http://gdk.test:3000/rails/lookbook) for a more interactive way to browse our components. +Consider this list a best effort. The full list can be found in [`app/components/pajamas`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/app/components/pajamas). Also see our Lookbook (`http://gdk.test:3000/rails/lookbook`) for a more interactive way to browse our components. #### Alert @@ -155,7 +155,7 @@ For the full list of options, see its #### Checkbox tag -The `Pajamas::CheckboxTagComponent` follows the [Pajamas Checkbox](https://design.gitlab.com/components/checkbox) specification. +The `Pajamas::CheckboxTagComponent` follows the [Pajamas Checkbox](https://design.gitlab.com/components/checkbox/) specification. The `name` argument and `label` slot are required. @@ -176,7 +176,7 @@ For the full list of options, see its #### Checkbox -The `Pajamas::CheckboxComponent` follows the [Pajamas Checkbox](https://design.gitlab.com/components/checkbox) specification. +The `Pajamas::CheckboxComponent` follows the [Pajamas Checkbox](https://design.gitlab.com/components/checkbox/) specification. NOTE: `Pajamas::CheckboxComponent` is used internally by the [GitLab UI form builder](haml.md#use-the-gitlab-ui-form-builder) and requires an instance of [ActionView::Helpers::FormBuilder](https://api.rubyonrails.org/v6.1.0/classes/ActionView/Helpers/FormBuilder.html) to be passed as the `form` argument. diff --git a/doc/development/fe_guide/vue.md b/doc/development/fe_guide/vue.md index 00d9588d087..779010b8aa1 100644 --- a/doc/development/fe_guide/vue.md +++ b/doc/development/fe_guide/vue.md @@ -65,6 +65,9 @@ To do that, you can use the `data` attributes in the HTML element and query them You should only do this while initializing the application, because the mounted element is replaced with a Vue-generated DOM. +The `data` attributes are [only able to accept String values](https://developer.mozilla.org/en-US/docs/Learn/HTML/Howto/Use_data_attributes#javascript_access), +so you will need to cast or convert other variable types to String. + The advantage of providing data from the DOM to the Vue instance through `props` or `provide` in the `render` function, instead of querying the DOM inside the main Vue component, is that you avoid creating a fixture or an HTML element in the unit test. diff --git a/doc/development/fe_guide/vue3_migration.md b/doc/development/fe_guide/vue3_migration.md index c3ce42a80a5..aae7674d190 100644 --- a/doc/development/fe_guide/vue3_migration.md +++ b/doc/development/fe_guide/vue3_migration.md @@ -159,3 +159,232 @@ export default { [In Vue 3](https://v3-migration.vuejs.org/breaking-changes/props-default-this.html), the props default value factory is passed the raw props as an argument, and can also access injections. + +## Handling libraries that do not work with `@vue/compat` + +**Problem** + +Some libraries rely on Vue.js 2 internals. They might not work with `@vue/compat`, so we need a strategy to use an updated version with Vue.js 3 while maintaining compatibility with the current codebase. + +**Goals** + +- We should add as few changes as possible to existing code to support new libraries. Instead, we should **add*- new code, which will act as **facade**, making the new version compatible with the old one +- Switching between new and old versions should be hidden inside tooling (webpack / jest) and should not be exposed to the code +- All facades specific to migration should live in the same directory to simplify future migration steps + +### Step-by-step migration + +In the step-by-step guide, we will be migrating [VueApollo Demo](https://gitlab.com/gitlab-org/frontend/vue3-migration-vue-apollo/-/tree/main/src/vue3compat) project. It will allow us to focus on migration specifics while avoiding nuances of complex tooling setup in the GitLab project. The project intentionally uses the same tooling as GitLab: + +- webpack +- yarn +- Vue.js + VueApollo + +#### Initial state + +Right after cloning, you could run [VueApollo Demo](https://gitlab.com/gitlab-org/frontend/vue3-migration-vue-apollo/-/tree/main/src/vue3compat) with Vue.js 2 using `yarn serve` or with Vue.js 3 (compat build) using `yarn serve:vue3`. However latter immediately crashes: + +```javascript +Uncaught TypeError: Cannot read properties of undefined (reading 'loading') +``` + +VueApollo v3 (used for Vue.js 2) fails to initialize in Vue.js compat + +NOTE: +While stubbing `Vue.version` will solve VueApollo-related issues in the demo project, it will still lose reactivity on specific scenarios, so an upgrade is still needed + +#### Step 1. Perform upgrade according to library docs + +According to [VueApollo v4 installation guide](https://v4.apollo.vuejs.org/guide/installation.html), we need to install `@vue/apollo-option` (this package provides VueApollo support for Options API) and make changes to our application: + +```diff +--- a/src/index.js ++++ b/src/index.js +@@ -1,19 +1,17 @@ +-import Vue from "vue"; +-import VueApollo from "vue-apollo"; ++import { createApp, h } from "vue"; ++import { createApolloProvider } from "@vue/apollo-option"; + + import Demo from "./components/Demo.vue"; + import createDefaultClient from "./lib/graphql"; + +-Vue.use(VueApollo); +- +-const apolloProvider = new VueApollo({ ++const apolloProvider = createApolloProvider({ + defaultClient: createDefaultClient(), + }); + +-new Vue({ +- el: "#app", +- apolloProvider, +- render(h) { ++const app = createApp({ ++ render() { + return h(Demo); + }, + }); ++app.use(apolloProvider); ++app.mount("#app"); +``` + +You can view these changes in [01-upgrade-vue-apollo](https://gitlab.com/gitlab-org/frontend/vue3-migration-vue-apollo/-/compare/main...01-upgrade-vue-apollo) branch of demo project + +#### Step 2. Addressing differences in augmenting applications in Vue.js 2 and 3 + +In Vue.js 2 tooling like `VueApollo` is initialized in a "lazy" fashion: + +```javascript +// We are registering VueApollo "handler" to handle some data LATER +Vue.use(VueApollo) +// ... +// apolloProvider is provided at app instantiation, +// previously registered VueApollo will handle that +new Vue({ /- ... */, apolloProvider }) +``` + +In Vue.js 3 both steps were merged in one - we are immediately registering the handler and passing configuration: + +```javascript +app.use(apolloProvider) +``` + +In order to backport this behavior, we need the following knowledge: + +- We can access extra options provided to Vue instance via `$options`, so extra `apolloProvider` will be visible as `this.$options.apolloProvider` +- We can access the current `app` (in Vue.js 3 meaning) on the Vue instance via `this.$.appContext.app` + +NOTE: +We're relying on non-public Vue.js 3 API in this case. However, since `@vue/compat` builds are expected to be available only for 3.2.x branch, we have reduced risks that this API will be changed + +With this knowledge, we can move the initialization of our tooling as early as possible in Vue2 - in the `beforeCreate()` lifecycle hook: + +```diff +--- a/src/index.js ++++ b/src/index.js +@@ -1,4 +1,4 @@ +-import { createApp, h } from "vue"; ++import Vue from "vue"; + import { createApolloProvider } from "@vue/apollo-option"; + + import Demo from "./components/Demo.vue"; +@@ -8,10 +8,13 @@ const apolloProvider = createApolloProvider({ + defaultClient: createDefaultClient(), + }); + +-const app = createApp({ +- render() { ++new Vue({ ++ el: "#app", ++ apolloProvider, ++ render(h) { + return h(Demo); + }, ++ beforeCreate() { ++ this.$.appContext.app.use(this.$options.apolloProvider); ++ }, + }); +-app.use(apolloProvider); +-app.mount("#app"); +``` + +You can view these changes in [02-bring-back-new-vue](https://gitlab.com/gitlab-org/frontend/vue3-migration-vue-apollo/-/compare/01-upgrade-vue-apollo...02-bring-back-new-vue) branch of demo project + +#### Step 3. Recreating `VueApollo` class + +Vue.js 3 libraries (and Vue.js itself) have a preference for using factories like `createApp` instead of classes (previously `new Vue`) + +`VueApollo` class served two purposes: + +- constructor for creating `apolloProvider` +- installation of apollo-related logic in components + +We can utilize `Vue.use(VueApollo)` code, which existed in our codebase, to hide there our mixin and avoid modification of our app code: + +```diff +--- a/src/index.js ++++ b/src/index.js +@@ -4,7 +4,26 @@ import { createApolloProvider } from "@vue/apollo-option"; + import Demo from "./components/Demo.vue"; + import createDefaultClient from "./lib/graphql"; + +-const apolloProvider = createApolloProvider({ ++class VueApollo { ++ constructor(...args) { ++ return createApolloProvider(...args); ++ } ++ ++ // called by Vue.use ++ static install() { ++ Vue.mixin({ ++ beforeCreate() { ++ if (this.$options.apolloProvider) { ++ this.$.appContext.app.use(this.$options.apolloProvider); ++ } ++ }, ++ }); ++ } ++} ++ ++Vue.use(VueApollo); ++ ++const apolloProvider = new VueApollo({ + defaultClient: createDefaultClient(), + }); + +@@ -14,7 +33,4 @@ new Vue({ + render(h) { + return h(Demo); + }, +- beforeCreate() { +- this.$.appContext.app.use(this.$options.apolloProvider); +- }, + }); +``` + +You can view these changes in [03-recreate-vue-apollo](https://gitlab.com/gitlab-org/frontend/vue3-migration-vue-apollo/-/compare/02-bring-back-new-vue...03-recreate-vue-apollo) branch of demo project + +#### Step 4. Moving `VueApollo` class to a separate file and setting up an alias + +Now, we have almost the same code (excluding import) as in Vue.js 2 version. +We will move our facade to the separate file and set up `webpack` conditionally execute it if `vue-apollo` is imported when using Vue.js 3: + +```diff +--- a/src/index.js ++++ b/src/index.js +@@ -1,5 +1,5 @@ + import Vue from "vue"; +-import { createApolloProvider } from "@vue/apollo-option"; ++import VueApollo from "vue-apollo"; + + import Demo from "./components/Demo.vue"; + import createDefaultClient from "./lib/graphql"; +diff --git a/webpack.config.js b/webpack.config.js +index 6160d3f..b8b955f 100644 +--- a/webpack.config.js ++++ b/webpack.config.js +@@ -12,6 +12,7 @@ if (USE_VUE3) { + + VUE3_ALIASES = { + vue: "@vue/compat", ++ "vue-apollo": path.resolve("src/vue3compat/vue-apollo"), + }; + } +``` + +(moving `VueApollo` class from `index.js` to `vue3compat/vue-apollo.js` as default export is omitted for clarity) + +You can view these changes in [04-add-webpack-alias](https://gitlab.com/gitlab-org/frontend/vue3-migration-vue-apollo/-/compare/03-recreate-vue-apollo...04-add-webpack-alias) branch of demo project + +#### Step 5. Observe the results + +At this point, you should be able again to run **both*- Vue.js 2 version with `yarn serve` and Vue.js 3 one with `yarn serve:vue3` +[Final MR](https://gitlab.com/gitlab-org/frontend/vue3-migration-vue-apollo/-/merge_requests/1/diffs) with all changes from previous steps displays no changes to `index.js` (application code), which was our goal + +### Applying this approach in the GitLab project + +In [commit adding VueApollo v4 support](https://gitlab.com/gitlab-org/gitlab/-/commit/e0af7e6479695a28a4fe85a88f90815aa3ce2814) we can see additional nuances not covered by step-by-step guide: + +- We might need to add additional imports to our facades (our code in GitLab uses `ApolloMutation` component) +- We need to update aliases not only for webpack but also for jest so our tests could also consume our facade diff --git a/doc/development/fe_guide/vuex.md b/doc/development/fe_guide/vuex.md index 5047f1b7f89..19bbfa314ea 100644 --- a/doc/development/fe_guide/vuex.md +++ b/doc/development/fe_guide/vuex.md @@ -22,7 +22,7 @@ official [Vuex documentation](https://vuex.vuejs.org). Vuex is composed of State, Getters, Mutations, Actions, and Modules. -When a user clicks on an action, we need to `dispatch` it. This action `commits` a mutation that changes the state. The action itself does not update the state; only a mutation should update the state. +When a user selects an action, we need to `dispatch` it. This action `commits` a mutation that changes the state. The action itself does not update the state; only a mutation should update the state. ## File structure diff --git a/doc/development/feature_development.md b/doc/development/feature_development.md index 760ba033633..76447124177 100644 --- a/doc/development/feature_development.md +++ b/doc/development/feature_development.md @@ -197,6 +197,7 @@ The following integration guides are internal. Some integrations require access - [Preventing transient bugs](transient/prevention-patterns.md) - [GitLab Application SLIs](application_slis/index.md) - [Spam protection and CAPTCHA development guide](spam_protection_and_captcha/index.md) +- [RuboCop development guide](rubocop_development_guide.md) ## Other GitLab Development Kit (GDK) guides diff --git a/doc/development/feature_flags/controls.md b/doc/development/feature_flags/controls.md index e6c3c20d50b..e804a888e98 100644 --- a/doc/development/feature_flags/controls.md +++ b/doc/development/feature_flags/controls.md @@ -16,7 +16,7 @@ To turn on/off features behind feature flags in any of the GitLab-provided environments, like staging and production, you need to have access to the [ChatOps](../chatops_on_gitlabcom.md) bot. The ChatOps bot is currently running on the ops instance, which is different from -[GitLab.com](https://gitlab.com) or [`dev.gitlab.org`](https://dev.gitlab.org). +[GitLab.com](https://gitlab.com) or `dev.gitlab.org`. Follow the ChatOps document to [request access](../chatops_on_gitlabcom.md#requesting-access). @@ -55,8 +55,8 @@ change feature flags or you do not have access. ### Enabling a feature for pre-production testing As a first step in a feature rollout, you should enable the feature on -[`staging.gitlab.com`](https://staging.gitlab.com) -and [`dev.gitlab.org`](https://dev.gitlab.org). +`staging.gitlab.com` +and `dev.gitlab.org`. These two environments have different scopes. `dev.gitlab.org` is a production CE environment that has internal GitLab Inc. diff --git a/doc/development/feature_flags/index.md b/doc/development/feature_flags/index.md index f1cde4ae1ea..500afa8ba1d 100644 --- a/doc/development/feature_flags/index.md +++ b/doc/development/feature_flags/index.md @@ -435,7 +435,7 @@ For example, the following feature flags are enabled for a certain percentage of If a project A has `:feature-set-1` enabled, there is no guarantee that project A also has `:feature-set-2` enabled. -For more detail, see [This is how percentages work in Flipper](https://www.hackwithpassion.com/this-is-how-percentages-work-in-flipper). +For more detail, see [This is how percentages work in Flipper](https://www.hackwithpassion.com/this-is-how-percentages-work-in-flipper/). #### Use actors for verifying in production diff --git a/doc/development/fips_compliance.md b/doc/development/fips_compliance.md index 475c1a49000..c6208d45c77 100644 --- a/doc/development/fips_compliance.md +++ b/doc/development/fips_compliance.md @@ -67,7 +67,7 @@ listed here that also do not work properly in FIPS mode: - [Static Application Security Testing (SAST)](../user/application_security/sast/index.md) supports a reduced set of [analyzers](../user/application_security/sast/index.md#fips-enabled-images) when operating in FIPS-compliant mode. -- Advanced Search is currently not included in FIPS mode. It must not be enabled to be FIPS-compliant. +- Advanced Search is currently not included in FIPS mode. It must not be enabled to be FIPS-compliant. - [Gravatar or Libravatar-based profile images](../administration/libravatar.md) are not FIPS-compliant. Additionally, these package repositories are disabled in FIPS mode: @@ -203,7 +203,7 @@ This [GitHub pull request](https://github.com/awslabs/amazon-eks-ami/pull/898) m it possible to create an Amazon Linux 2 EKS AMI with FIPS enabled for Kubernetes v1.21. To build an image: -1. [Install Packer](https://learn.hashicorp.com/tutorials/packer/get-started-install-cli). +1. [Install Packer](https://developer.hashicorp.com/packer/tutorials/docker-get-started/get-started-install-cli). 1. Run the following: ```shell diff --git a/doc/development/gemfile.md b/doc/development/gemfile.md index 7d3531afb49..3c7dc19da8e 100644 --- a/doc/development/gemfile.md +++ b/doc/development/gemfile.md @@ -66,7 +66,7 @@ This means that new dependencies should, at a minimum, meet the following criter When adding a new gem to our `Gemfile` or even changing versions in `Gemfile.lock` it is strongly recommended that you -[request a Security review](https://about.gitlab.com/handbook/engineering/security/#how-to-request-a-security-review). +[request a Security review](https://about.gitlab.com/handbook/security/#how-to-request-a-security-review). New gems add an extra security risk for GitLab, and it is important to evaluate this risk before we ship this to production. Technically, just adding a new gem and pushing to a branch in our main `gitlab` project is a security diff --git a/doc/development/github_importer.md b/doc/development/github_importer.md index 9ba375439f4..edf78d5f83d 100644 --- a/doc/development/github_importer.md +++ b/doc/development/github_importer.md @@ -78,14 +78,21 @@ individually to import this information. A `Gitlab::GithubImport::ImportPullRequestMergedByWorker` job is scheduled for each fetched pull request. -### 6. Stage::ImportPullRequestsReviewsWorker +### 6. Stage::ImportPullRequestsReviewRequestsWorker -This worker imports the pull requests' reviews. For each pull request, this worker: +This worker imports assigned reviewers of pull requests. For each pull request, this worker: + +- Fetches all assigned review requests. +- Schedules a `Gitlab::GithubImport::PullRequests::ImportReviewRequestWorker` job for each fetched review request. + +### 7. Stage::ImportPullRequestsReviewsWorker + +This worker imports reviews of pull requests. For each pull request, this worker: - Fetches all the pages of reviews. - Schedules a `Gitlab::GithubImport::ImportPullRequestReviewWorker` job for each fetched review. -### 7. Stage::ImportIssuesAndDiffNotesWorker +### 8. Stage::ImportIssuesAndDiffNotesWorker This worker imports all issues and pull request comments. For every issue, we schedule a job for the `Gitlab::GithubImport::ImportIssueWorker` worker. For @@ -101,7 +108,7 @@ label links in the same worker removes the need for performing a separate crawl through the API data, reducing the number of API calls necessary to import a project. -### 8. Stage::ImportIssueEventsWorker +### 9. Stage::ImportIssueEventsWorker This worker imports all issues and pull request events. For every event, we schedule a job for the `Gitlab::GithubImport::ImportIssueEventWorker` worker. @@ -117,7 +124,7 @@ Therefore, both issues and pull requests have a common API for most related thin NOTE: This stage is optional and can consume significant extra import time (controlled by `Gitlab::GithubImport::Settings`). -### 9. Stage::ImportNotesWorker +### 10. Stage::ImportNotesWorker This worker imports regular comments for both issues and pull requests. For every comment, we schedule a job for the @@ -128,7 +135,7 @@ returns comments for both issues and pull requests. This means we have to wait for all issues and pull requests to be imported before we can import regular comments. -### 10. Stage::ImportAttachmentsWorker +### 11. Stage::ImportAttachmentsWorker This worker imports note attachments that are linked inside Markdown. For each entity with Markdown text in the project, we schedule a job of: @@ -147,7 +154,7 @@ Each job: NOTE: It's an optional stage that could consume significant extra import time (controlled by `Gitlab::GithubImport::Settings`). -### 11. Stage::ImportProtectedBranchesWorker +### 12. Stage::ImportProtectedBranchesWorker This worker imports protected branch rules. For every rule that exists on GitHub, we schedule a job of @@ -156,7 +163,7 @@ For every rule that exists on GitHub, we schedule a job of Each job compares the branch protection rules from GitHub and GitLab and applies the strictest of the rules to the branches in GitLab. -### 12. Stage::FinishImportWorker +### 13. Stage::FinishImportWorker This worker completes the import process by performing some housekeeping (such as flushing any caches) and by marking the import as completed. diff --git a/doc/development/gitlab_flavored_markdown/specification_guide/index.md b/doc/development/gitlab_flavored_markdown/specification_guide/index.md index 95d06907aa6..17afebcf6ee 100644 --- a/doc/development/gitlab_flavored_markdown/specification_guide/index.md +++ b/doc/development/gitlab_flavored_markdown/specification_guide/index.md @@ -34,7 +34,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w against the official specification. - They support [snapshot testing](#markdown-snapshot-testing) of GitLab internal GLFM processing logic. This is accomplished by automatically - generating YAML ["example snapshot files"](#example-snapshot-files) + generating YAML ["example snapshot files"](#output-example-snapshot-files) which are used as fixtures to drive automated testing within the GitLab app. - There are [various scripts and logic](#scripts) which are used to accomplish the above goals. @@ -104,13 +104,13 @@ serve as input to automated conformance tests. It is Here are the HTML-rendered versions of the specifications: -- [GitLab Flavored Markdown (GLFM) specification](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/output/spec.html), which extends the: +- [GitLab Flavored Markdown (GLFM) specification](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/output_spec/spec.html), which extends the: - [GitHub Flavored Markdown (GFM) specification](https://github.github.com/gfm/) (rendered from the [source `spec.txt` for GFM specification](https://github.com/github/cmark-gfm/blob/master/test/spec.txt)), which extends the: - [CommonMark specification](https://spec.commonmark.org/0.30/) (rendered from the [source `spec.txt` for CommonMark specification](https://github.com/commonmark/commonmark-spec/blob/master/spec.txt)) NOTE: The creation of the -[HTML-rendered version of the GitLab Flavored Markdown (GLFM) specification](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/output/spec.html) +[HTML-rendered version of the GitLab Flavored Markdown (GLFM) specification](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/output_spec/spec.html) file is still pending. However, GLFM has more complex parsing, rendering, and testing requirements than @@ -177,7 +177,7 @@ In this context, it should not be confused with other similar or related meaning _example_, such as [RSpec examples](https://relishapp.com/rspec/rspec-core/docs/example-groups/basic-structure-describe-it). -See the section on the [`glfm_official_specification_examples.md`](#glfm_official_specification_examplesmd) file +See the section on the [`glfm_official_specification.md`](#glfm_official_specificationmd) file for more details on the backtick-delimited Markdown+HTML example syntax. ### Parsers and renderers @@ -341,7 +341,7 @@ The GitLab [Markdown API](../../../api/markdown.md) generates HTML for a given Markdown string using this method. The Markdown specified in the [Markdown examples](#markdown-examples) is used to automatically generate HTML in -[`glfm_specification/example_snapshots/html.yml`](#glfm_specificationexample_snapshotshtmlyml) via +[`glfm_specification/output_example_snapshots/html.yml`](#htmlyml) via [`update-example-snapshots.rb`](#update-example-snapshotsrb-script). These examples are used when running [Markdown snapshot testing](#markdown-snapshot-testing). @@ -353,7 +353,7 @@ in the ProseMirror WYSIWYG editor. Just like static HTML, the Markdown specified in the [Markdown examples](#markdown-examples) is used to automatically generate HTML in -[`glfm_specification/example_snapshots/html.yml`](#glfm_specificationexample_snapshotshtmlyml) via +[`glfm_specification/output_example_snapshots/html.yml`](#htmlyml) via [`update-example-snapshots.rb`](#update-example-snapshotsrb-script). These examples are used when running [Markdown snapshot testing](#markdown-snapshot-testing). @@ -387,10 +387,10 @@ Here are more details on the sources of canonical HTML examples: version of [`spec.txt`](#spectxt). 1. For the examples which are part of the GLFM [_official specification_](#official-specifications), the canonical HTML is manually maintained and curated via the examples contained in the - [`glfm_official_specification_examples.md`](#glfm_official_specification_examplesmd) [input specification file](#input-specification-files). + [`glfm_official_specification.md`](#glfm_official_specificationmd) [input specification file](#input-specification-files). 1. For the examples which are part of the GLFM [_internal extensions_](#internal-extensions), the canonical HTML **is never specified**, and **must be left empty in all examples** contained in - the [`glfm_internal_extension_examples.md`](#glfm_internal_extension_examplesmd) [input specification file](#input-specification-files). + the [`glfm_internal_extensions.md`](#glfm_internal_extensionsmd) [input specification file](#input-specification-files). ### Canonicalization of HTML @@ -401,7 +401,7 @@ or HTML elements, to support specific appearance and behavioral requirements. Neither the backend nor the frontend rendering logic can directly render the clean, basic HTML which is necessary to perform comparison to the [canonical HTML](#canonical-html) when running [Markdown conformance testing](#markdown-conformance-testing) -for the [GLFM official specification examples](#glfm_official_specification_examplesmd). +for the [GLFM official specification examples](#glfm_official_specificationmd). Nor should they be able to, because: @@ -443,7 +443,7 @@ Fixture-based normalization should be used whenever possible, because it is simp understand than regex-based normalization. The [Markdown snapshot testing](#markdown-snapshot-testing) uses RSpec to generate the -[example snapshot files](#example-snapshot-files). RSpec enables you to: +[example snapshot files](#output-example-snapshot-files). RSpec enables you to: - Use the same powerful fixture support and helpers as all the rest of the GitLab RSpec suite. - Use fixtures to control the state of the database when the example snapshots are generated. @@ -575,10 +575,10 @@ The documentation on the implementation is split into three sections: 1. [Scripts](#scripts). 1. [Specification files](#specification-files). -1. [Example snapshot files](#example-snapshot-files): +1. [Example snapshot files](#output-example-snapshot-files): These YAML files are used as input data or fixtures to drive the various tests, and are located under - `glfm_specification/example_snapshots`. All example snapshot files are automatically + `glfm_specification/output_example_snapshots`. All example snapshot files are automatically generated based on the specification files and the implementation of the parsers and renderers. However, they can also be directly edited if necessary, such as to test-drive an incomplete implementation. @@ -620,8 +620,8 @@ end subgraph input:<br/>input specification files C[ghfm_spec_v_0.29.md] --> A D[glfm_intro.md] --> A - E[glfm_official_specification_examples.md] --> A - F[glfm_internal_extension_examples.md] --> A + E[glfm_official_specification.md] --> A + F[glfm_internal_extensions.md] --> A end subgraph output:<br/>GLFM specification files A --> G[spec.txt] @@ -646,7 +646,7 @@ script, which expects canonical HTML, against the GitLab renderer implementation `scripts/glfm/run-spec-tests.sh` is a convenience shell script which runs conformance specs via the CommonMark standard `spec_tests.py` script, -which uses the `glfm_specification/output/spec.txt` file and `scripts/glfm/canonicalize-html.rb` +which uses the `glfm_specification/output_spec/spec.txt` file and `scripts/glfm/canonicalize-html.rb` helper script to test the GLFM renderer implementations' support for rendering Markdown specification examples to canonical HTML. @@ -672,9 +672,9 @@ end #### `update-example-snapshots.rb` script The `scripts/glfm/update-example-snapshots.rb` script uses the GLFM -`glfm_specification/output/spec.txt` specification file and the +`glfm_specification/output_spec/spec.txt` specification file and the `glfm_specification/input/gitlab_flavored_markdown/glfm_example_status.yml` -file to create and update the [example snapshot](#example-snapshot-files) +file to create and update the [example snapshot](#output-example-snapshot-files) YAML files: ```mermaid @@ -746,7 +746,7 @@ runs the [`update-specification.rb`](#update-specificationrb-script). It fails with an exception and non-zero return code if running these scripts results in any diffs to the generated and committed [output specification files](#output-specification-files) or -[example snapshot files](#example-snapshot-files). +[example snapshot files](#output-example-snapshot-files). This script is run via the `glfm-verify` CI job to ensure that all changes to the [input specification files](#input-specification-files) @@ -766,12 +766,16 @@ subcategories based on their usage and purpose: [`ghfm_spec_v_0.29.md`](#github-flavored-markdown-specification) specification. - `gitlab_flavored_markdown`: Contains all `glfm_*` files. - `*.md` [input specification files](#input-specification-files), - which represent the GLFM specification itself. + which represent the source of truth for the GLFM specification and all associated examples. - `*.yml` [input specification configuration files](#input-specification-configuration-files), which control various aspects of the automated GLFM scripts and processes. - - `output`: Contains [output specification files](#output-specification-files), - which are automatically generated from the + - `output_spec`: Contains the `spec.txt` and `spec.html` [output specification files](#output-specification-files), + which represent the [GLFM official specification](#official-specifications), and are automatically generated from the input files by running the [`update-specification.rb`](#update-specificationrb-script) script. + - `output_example_snapshots`: Contains [output example snapshot files](#output-example-snapshot-files). + which are used to drive [snapshot testing](#markdown-snapshot-testing), and are automatically generated from the + input files by running the [`update-specification.rb`](#update-specificationrb-script) + and [`scripts/glfm/update-example-snapshots.rb`](#update-example-snapshotsrb-script) scripts. #### Input specification files @@ -790,31 +794,22 @@ is a copy of the official latest [GFM `spec.txt`](https://github.com/github/cmar - When it is downloaded, the version number is added to the filename. - The extension is changed from `*.txt` to `*.md` so that it can be handled better by Markdown editors. -NOTE: -For extra clarity, this file uses the `ghfm` acronym in its name instead of `gfm`, as -explained in the [Acronyms section](#acronyms-glfm-ghfm-gfm-commonmark). - -##### `glfm_intro.md` - -[`glfm_specification/input/gitlab_flavored_markdown/glfm_intro.md`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/input/gitlab_flavored_markdown/glfm_intro.md) -is the GitLab-specific version of the prose in the introduction section of the GLFM specification. +It currently contains additional **Introduction** and **Appendix** prose-only header sections which do not +contain any examples. -- It is manually updated. -- The `update-specification.rb` script inserts it into the generated GLFM `spec.txt` to replace - the GitHub-specific GFM version of the introductory section. - -##### `glfm_canonical_examples.txt` +All header sections which contain examples are expected to be contained within a contiguous file section +which is delimited by: -The `glfm_canonical_examples.txt` file is deprecated and no longer exists. It has been replaced by two files: +1. The beginning of the second H1 header (the first one after the **Introduction** section) +1. An `<!-- END TESTS -->` HTML comment line. -- [`glfm_official_specification_examples.md`](#glfm_official_specification_examplesmd) - which contains the [GLFM official specification](#official-specifications) examples. -- [`glfm_internal_extension_examples.md`](#glfm_internal_extension_examplesmd) - which contains the [GLFM internal extension](#internal-extensions) examples. +NOTE: +For extra clarity, this file uses the `ghfm` acronym in its name instead of `gfm`, as +explained in the [Acronyms section](#acronyms-glfm-ghfm-gfm-commonmark). -##### `glfm_official_specification_examples.md` +##### `glfm_official_specification.md` -[`glfm_specification/input/gitlab_flavored_markdown/glfm_official_specification_examples.md`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/input/gitlab_flavored_markdown/glfm_official_specification_examples.md) +[`glfm_specification/input/gitlab_flavored_markdown/glfm_official_specification.md`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/input/gitlab_flavored_markdown/glfm_official_specification.md) consists of the manually updated Markdown+HTML examples for the [GLFM official specification](#official-specifications), and their associated documentation and descriptions. @@ -832,7 +827,12 @@ consists of the manually updated Markdown+HTML examples for the - `H3` header sections must be nested within `H2` header sections. They cannot be nested directly within `H1` header sections. -`glfm_specification/input/gitlab_flavored_markdown/glfm_official_specification_examples.md` sample entries: +It _may_ contain additional prose-only header sections which do not contain any examples. + +All header sections which contain examples _must_ be contained within a contiguous file section which +is delimited by `<!-- BEGIN TESTS -->` and `<!-- END TESTS -->` HTML comment lines. + +`glfm_specification/input/gitlab_flavored_markdown/glfm_official_specification.md` sample entries: <!-- markdownlint-disable MD048 --> @@ -864,13 +864,13 @@ bold <!-- markdownlint-enable MD048 --> -##### `glfm_internal_extension_examples.md` +##### `glfm_internal_extensions.md` -[`glfm_specification/input/gitlab_flavored_markdown/glfm_internal_extension_examples.md`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/input/gitlab_flavored_markdown/glfm_internal_extension_examples.md) +[`glfm_specification/input/gitlab_flavored_markdown/glfm_internal_extensions.md`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/input/gitlab_flavored_markdown/glfm_internal_extensions.md) consists of the manually updated Markdown examples for the [GLFM internal extensions](#internal-extensions), and their associated documentation and descriptions. -Its general format is identical to [`glfm_official_specification_examples.md`](#glfm_official_specification_examplesmd), +Its general format is identical to [`glfm_official_specification.md`](#glfm_official_specificationmd), consisting of `H1`, `H2`, or `H3` sections containing [Markdown examples](#markdown-examples) in the [standard backtick-delimited `spec.txt` format](#various-markdown-specifications). @@ -878,7 +878,12 @@ However, as described in the [canonical HTML section](#canonical-html), only the example is specified, and the HTML portion is left empty, because internal extension examples are never used for [Markdown conformance testing](#markdown-conformance-testing). -`glfm_specification/input/gitlab_flavored_markdown/glfm_official_specification_examples.md` sample entries: +It _may_ contain additional prose-only header sections which do not contain any examples. + +All header sections which contain examples _must_ be contained within a contiguous file section which +is delimited by `<!-- BEGIN TESTS -->` and `<!-- END TESTS -->` HTML comment lines. + +`glfm_specification/input/gitlab_flavored_markdown/glfm_official_specification.md` sample entries: NOTE: All lines in this example are prefixed with a `|` character. This prefix helps avoid false @@ -908,7 +913,7 @@ See the main [specification files](#specification-files) section for more contex All of the manually curated example names in the configuration files must correspond to an existing [Markdown example](#markdown-examples) name found in -[`example_snapshots/examples_index.yml`](#glfm_specificationexample_snapshotsexamples_indexyml), +[`output_example_snapshots/examples_index.yml`](#examples_indexyml), which is automatically generated based on the [input specification files](#input-specification-files). If there is an invalid reference to an example name that does not exist, the @@ -943,16 +948,16 @@ controls the behavior of the [scripts](#scripts) and [tests](#types-of-markdown- The following optional entries are supported for each example. They all default to `false`: - `skip_update_example_snapshots`: When true, skips any addition or update of any this example's entries - in the [`glfm_specification/example_snapshots/html.yml`](#glfm_specificationexample_snapshotshtmlyml) file - or the [`glfm_specification/example_snapshots/prosemirror_json.yml`](#glfm_specificationexample_snapshotsprosemirror_jsonyml) file. + in the [`glfm_specification/output_example_snapshots/html.yml`](#htmlyml) file + or the [`glfm_specification/output_example_snapshots/prosemirror_json.yml`](#prosemirror_jsonyml) file. If this value is truthy, then no other `skip_update_example_snapshot_*` entries can be truthy, and an error is raised if any of them are. - `skip_update_example_snapshot_html_static`: When true, skips addition or update of this example's [static HTML](#static-html) - entry in the [`glfm_specification/example_snapshots/html.yml`](#glfm_specificationexample_snapshotshtmlyml) file. + entry in the [`glfm_specification/output_example_snapshots/html.yml`](#htmlyml) file. - `skip_update_example_snapshot_html_wysiwyg`: When true, skips addition or update of this example's [WYSIWYG HTML](#wysiwyg-html) - entry in the [`glfm_specification/example_snapshots/html.yml`](#glfm_specificationexample_snapshotshtmlyml) file. + entry in the [`glfm_specification/output_example_snapshots/html.yml`](#htmlyml) file. - `skip_update_example_snapshot_prosemirror_json`: When true, skips addition or update of this example's - entry in the [`glfm_specification/example_snapshots/prosemirror_json.yml`](#glfm_specificationexample_snapshotsprosemirror_jsonyml) file. + entry in the [`glfm_specification/output_example_snapshots/prosemirror_json.yml`](#prosemirror_jsonyml) file. - `skip_running_conformance_static_tests`: When true, skips running the [Markdown conformance tests](#markdown-conformance-testing) of the [static HTML](#static-html) for this example. - `skip_running_conformance_wysiwyg_tests`: When true, skips running the [Markdown conformance tests](#markdown-conformance-testing) @@ -962,7 +967,7 @@ The following optional entries are supported for each example. They all default - `skip_running_snapshot_wysiwyg_html_tests`: When true, skips running the [Markdown snapshot tests](#markdown-snapshot-testing) of the [WYSIWYG HTML](#wysiwyg-html) for this example. - `skip_running_snapshot_prosemirror_json_tests`: When true, skips running the [Markdown snapshot tests](#markdown-snapshot-testing) - of the [ProseMirror JSON](#glfm_specificationexample_snapshotsprosemirror_jsonyml) for this example. + of the [ProseMirror JSON](#prosemirror_jsonyml) for this example. `glfm_specification/input/gitlab_flavored_markdown/glfm_example_status.yml` sample entry: @@ -1033,9 +1038,9 @@ or [environment-variable-based normalization](#environment-variable-based-normal snapshot: 07_01_00_href: *07_01_00_href 07_01_00_id: *07_01_00_id - wysiwyg: - 07_01_00_href: *07_01_00_href - 07_01_00_id: *07_01_00_id + wysiwyg: + 07_01_00_href: *07_01_00_href + 07_01_00_id: *07_01_00_id prosemirror_json: 07_01_00_href: *07_01_00_href 07_01_00_id: *07_01_00_id @@ -1093,36 +1098,66 @@ move or copy a hosted version of the rendered HTML `spec.html` version to anothe ##### spec.txt -[`glfm_specification/output/spec.txt`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/output/spec.txt) +[`glfm_specification/output_spec/spec.txt`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/output_spec/spec.txt) is a Markdown specification file, in the standard format with prose and Markdown + canonical HTML examples. -It also serves as input for other scripts such as `update-example-snapshots.rb` -and `run-spec-tests.sh`. +It also serves as input for other scripts such as +`run-spec-tests.sh`. It is generated or updated by the `update-specification.rb` script, using the [input specification files](#input-specification-files) as input. See the [`update-specification.rb` script section](#update-specificationrb-script) for a diagram and more description on this process. +NOTE: +Even though `spec.txt` is a Markdown file, it is named with a `*.txt` extension +for consistency with the GFM and CommonMark specifications. All other GLFM +Markdown files are named with a `*.md` extension for compatibility with +various editors to enable Markdown formatting and syntax highlighting. + ##### spec.html -[`glfm_specification/output/spec.html`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/output/spec.html) +[`glfm_specification/output_spec/spec.html`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/output_spec/spec.html) is an HTML file, rendered based on `spec.txt`. It is -also generated (or updated) by the `update-specification.rb` script at the same time as +generated (or updated) by the `update-specification.rb` script at the same time as `spec.txt`. It corresponds to the HTML-rendered versions of the "GitHub Flavored Markdown" (<abbr title="GitHub Flavored Markdown">GFM</abbr>) [specification](https://github.github.com/gfm/) -and the [CommonMark specification](https://spec.commonmark.org/0.30/). +and the [CommonMark specification](https://spec.commonmark.org/0.30/), but only +contains GitLab Flavored Markdown (GLFM) examples. -### Example snapshot files +NOTE: + +The formatting of this HTML is currently not identical to the GFM and CommonMark +HTML-rendered specification. It is only the raw output of running `spec.txt` through +the GitLab Markdown renderer. Properly formatting the HTML will require +duplicating or reusing the Lua script and template from the CommonMark project: +[CommonMark Makefile](https://github.com/commonmark/commonmark-spec/blob/master/Makefile#L11) + +#### Output example snapshot files + +The `output_example_snapshots` directory contains files which are generated by the +`update-specification.rb` and `update-example-snapshots.rb` scripts based off of the files in the +`glfm_specification/input` directory. + +The `output-specification.rb` script generates +`output_snapshot_examples/snapshot_spec.md` and `output_snapshot_examples/snapshot_spec.html`. +These files are Markdown specification files containing examples generated based on input files, +similar to the `output_spec/spec.txt` and `output_spec/spec.html`, with the following differences: -The `example_snapshots` directory contains files which are generated by the -`update-example-snapshots.rb` script based off of the files in the -`glfm_specification/input` directory. They are used as fixtures to drive the -various Markdown snapshot tests. +1. They contain a superset of _all_ examples from + the Commonmark, GitHub Flavored Markdown, and GitLab Flavored Markdown specifications, whereas + `spec.*` only contains the GLFM specification. This is to provide a single place to refer to + all examples when working with [snapshot testing](#markdown-snapshot-testing). +1. They contain _only_ header sections which contain examples. They do not contain any prose-only + sections which do not contain examples. + +The `update-example-snapshots.rb` script generates the various +`output_snapshot_examples/*.yml` files, which +are used as fixtures to drive the [snapshot testing](#markdown-snapshot-testing). After the entire GLFM implementation is complete for both backend (Ruby) and frontend (JavaScript), all of these YAML files can be automatically generated. @@ -1131,9 +1166,47 @@ key in `glfm_specification/input/gitlab_flavored_markdown/glfm_example_status.ym can be used to disable automatic generation of some examples. They can instead be manually edited as necessary to help drive the implementations. -#### `glfm_specification/example_snapshots/examples_index.yml` +##### `snapshot_spec.md` + +[`glfm_specification/output_example_snapshots/snapshot_spec.md`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/output_example_snapshots/snapshot_spec.md) +is a Markdown file, containing standard Markdown + canonical HTML examples like [`spec.txt`](#spectxt). + +It is generated or updated by the `update-specification.rb` script, using the +[input specification files](#input-specification-files) as input. +See the [`update-specification.rb` script section](#update-specificationrb-script) +for a diagram and more description on this process. It also serves as input for other +scripts such as `update-example-snapshots.rb`. + +It is similar to [`spec.txt`](#spectxt), with the following differences: + +1. [`spec.txt`](#spectxt) contains only examples for GitLab Flavored Markdown, but + `snapshot_spec.md` also contains the full superset of examples from the + "GitHub Flavored Markdown" (<abbr title="GitHub Flavored Markdown">GFM</abbr>)[specification](https://github.github.com/gfm/) + and the [CommonMark specification](https://spec.commonmark.org/0.30/) specifications. +1. [`spec.txt`](#spectxt) represents the full GLFM specification, including additional header sections + containing only explanatory prose and no examples, but `snapshot_spec.md` consists of only + header sections which contain examples. This is because its purpose is to serve as input for + the other [`output example snapshot files`](#output-example-snapshot-files) - it is not intended + to serve as an actual [specification file](#output-specification-files) + like [`spec.txt`](#spectxt) or [`spec.html`](#spechtml). + +##### `snapshot_spec.html` + +[`glfm_specification/output_snapshot_examples/snapshot_spec.html`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/output_snapshot_examples/snapshot_spec.html) +is an HTML file, rendered based on `snapshot_spec.md`. It is +generated (or updated) by the `update-specification.rb` script at the same time as +`snapshot_spec.md`. + +NOTE: +The formatting of this HTML is currently not identical to the GFM and CommonMark +HTML-rendered specification. It is only the raw output of running `snapshot_spec.md` through +the GitLab Markdown renderer. Properly formatting the HTML will require +duplicating or reusing the Lua script and template from the CommonMark project: +[CommonMark Makefile](https://github.com/commonmark/commonmark-spec/blob/master/Makefile#L11) + +##### `examples_index.yml` -[`glfm_specification/example_snapshots/examples_index.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/example_snapshots/examples_index.yml) +[`glfm_specification/output_example_snapshots/examples_index.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/output_example_snapshots/examples_index.yml) is the main list of all CommonMark, GFM, and GLFM example names, each with a uniquely identifying name. @@ -1142,9 +1215,9 @@ CommonMark, GFM, and GLFM example names, each with a uniquely identifying name. - For CommonMark and GFM examples, these sections originally came from the GFM `spec.txt`. - For GLFM examples, it is generated from - [`glfm_official_specification_examples.md`](#glfm_official_specification_examplesmd) and [`glfm_internal_extension_examples.md`](#glfm_internal_extension_examplesmd). + [`glfm_official_specification.md`](#glfm_official_specificationmd) and [`glfm_internal_extensions.md`](#glfm_internal_extensionsmd). - It also contains extra metadata about each example, such as: - 1. `spec_txt_example_position` - The position of the example in the generated GLFM `spec.txt` file. + 1. `spec_example_position` - The position of the example in the generated GLFM `spec.txt` file. - This value is the index order of each individual Markdown + HTML5 example in the file. It is _not_ the line number in the file. - This value can be used to locate the example in the rendered `spec.html` file, because the standard @@ -1159,49 +1232,49 @@ CommonMark, GFM, and GLFM example names, each with a uniquely identifying name. examples where multiple examples exist for the same Section 7 subsection are added to the end of the sub-section. -`glfm_specification/example_snapshots/examples_index.yml` sample entries: +`examples_index.yml` sample entries: ```yaml 02_01_00_preliminaries_characters_and_lines_1: - spec_txt_example_position: 1 + spec_example_position: 1 source_specification: commonmark 03_01_00_blocks_and_inlines_precedence_1: - spec_txt_example_position: 12 + spec_example_position: 12 source_specification: commonmark 05_03_00_container_blocks_task_list_items_1: - spec_txt_example_position: 279 + spec_example_position: 279 source_specification: github 06_04_00_inlines_emphasis_and_strong_emphasis_1: - spec_txt_example_position: 360 + spec_example_position: 360 source_specification: github 07_01_00_audio_link_1: - spec_txt_example_position: 301 + spec_example_position: 301 source_specification: gitlab ``` -#### `glfm_specification/example_snapshots/markdown.yml` +##### `markdown.yml` -[`glfm_specification/example_snapshots/markdown.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/example_snapshots/markdown.yml) contains the original Markdown -for each entry in `glfm_specification/example_snapshots/examples_index.yml` +[`glfm_specification/output_example_snapshots/markdown.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/output_example_snapshots/markdown.yml) contains the original Markdown +for each entry in `glfm_specification/output_example_snapshots/examples_index.yml`: - For CommonMark and GFM Markdown, it is generated (or updated) from the standard GFM `spec.txt` using the `update-example-snapshots.rb` script. - For GLFM, it is generated (or updated) from the - [`glfm_official_specification_examples.md`](#glfm_official_specification_examplesmd) and [`glfm_internal_extension_examples.md`](#glfm_internal_extension_examplesmd) + [`glfm_official_specification.md`](#glfm_official_specificationmd) and [`glfm_internal_extensions.md`](#glfm_internal_extensionsmd) input specification files. -`glfm_specification/example_snapshots/markdown.yml` sample entry: +`glfm_specification/output_example_snapshots/markdown.yml` sample entry: ```yaml 06_04_00_inlines_emphasis_and_strong_emphasis_1: | *foo bar* ``` -#### `glfm_specification/example_snapshots/html.yml` +##### `html.yml` -[`glfm_specification/example_snapshots/html.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/example_snapshots/html.yml) -contains the HTML for each entry in `glfm_specification/example_snapshots/examples_index.yml` +[`glfm_specification/output_example_snapshots/html.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/output_example_snapshots/html.yml) +contains the HTML for each entry in `glfm_specification/output_example_snapshots/examples_index.yml`: Three types of entries exist, with different HTML for each: @@ -1209,16 +1282,16 @@ Three types of entries exist, with different HTML for each: - The ["Canonical"](#canonicalization-of-html) HTML. - For CommonMark and GFM examples, the HTML comes from the examples in `spec.txt`. - For [GLFM official specification](#official-specifications) examples, it is generated/updated from - [`glfm_official_specification_examples.md`](#glfm_official_specification_examplesmd). + [`glfm_official_specification.md`](#glfm_official_specificationmd). - **Static** - This is the static (backend (Ruby)-generated) HTML for each entry in - `glfm_specification/example_snapshots/examples_index.yml`. + `glfm_specification/output_example_snapshots/examples_index.yml`. - It is generated/updated from backend [Markdown API](../../../api/markdown.md) (or the underlying internal classes) via the `update-example-snapshots.rb` script, but can be manually updated for static examples with incomplete implementations. - **WYSIWYG** - The WYSIWYG (frontend, JavaScript-generated) HTML for each entry in - `glfm_specification/example_snapshots/examples_index.yml`. + `glfm_specification/output_example_snapshots/examples_index.yml`. - It is generated (or updated) from the frontend Content Editor implementation via the `update-example-snapshots.rb` script. It can be manually updated for WYSIWYG examples with incomplete implementations. @@ -1226,7 +1299,7 @@ Three types of entries exist, with different HTML for each: Any exceptions or failures which occur when generating HTML are replaced with an `Error - check implementation` value. -`glfm_specification/example_snapshots/html.yml` sample entry: +`glfm_specification/output_example_snapshots/html.yml` sample entry: ```yaml 06_04_00_inlines_emphasis_and_strong_emphasis_1: @@ -1242,16 +1315,16 @@ NOTE: The actual `static` or `WYSIWYG` entries may differ from the example `html.yml`, depending on how the implementations evolve. -#### `glfm_specification/example_snapshots/prosemirror_json.yml` +##### `prosemirror_json.yml` -[`glfm_specification/example_snapshots/prosemirror_json.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/example_snapshots/prosemirror_json.yml) -contains the ProseMirror JSON for each entry in `glfm_specification/example_snapshots/examples_index.yml` +[`glfm_specification/output_example_snapshots/prosemirror_json.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/output_example_snapshots/prosemirror_json.yml) +contains the ProseMirror JSON for each entry in `glfm_specification/output_example_snapshots/examples_index.yml` - It is generated (or updated) from the frontend code via the `update-example-snapshots.rb` script, but can be manually updated for examples with incomplete implementations. - Any exceptions or failures when generating are replaced with a `Error - check implementation` value. -`glfm_specification/example_snapshots/prosemirror_json.yml` sample entry: +`glfm_specification/output_example_snapshots/prosemirror_json.yml` sample entry: ```yaml 06_04_00_inlines_emphasis_and_strong_emphasis_1: |- @@ -1291,13 +1364,13 @@ This section describes how the scripts can be used to manage the GLFM specificat 1. If you are working on an in-progress feature or bug, make any necessary manual updates to the [input specification files](#input-specification-files). This may include: 1. Updating the canonical Markdown or HTML examples in - [`glfm_official_specification_examples.md`](#glfm_official_specification_examplesmd) or [`glfm_internal_extension_examples.md`](#glfm_internal_extension_examplesmd). + [`glfm_official_specification.md`](#glfm_official_specificationmd) or [`glfm_internal_extensions.md`](#glfm_internal_extensionsmd). 1. Updating `glfm_specification/input/gitlab_flavored_markdown/glfm_example_status.yml` to reflect the current status of the examples or tests. 1. Run [`update-specification.rb`](#update-specificationrb-script) to update the `spec.txt` to reflect any changes which were made to the [input specification files](#input-specification-files). 1. Visually inspect and confirm any resulting changes to the [output specification files](#output-specification-files). -1. Run [`update-example-snapshots.rb`](#update-example-snapshotsrb-script) to update the [example snapshot files](#example-snapshot-files). -1. Visually inspect and confirm any resulting changes to the [example snapshot files](#example-snapshot-files). +1. Run [`update-example-snapshots.rb`](#update-example-snapshotsrb-script) to update the [example snapshot files](#output-example-snapshot-files). +1. Visually inspect and confirm any resulting changes to the [example snapshot files](#output-example-snapshot-files). 1. Run [`run-snapshot-tests.sh`](#run-snapshot-testssh-script) as a convenience script to run all relevant frontend (RSpec) and backend (Jest) tests which use the example snapshots. 1. Any frontend or backend snapshot test may also be run individually. 1. All frontend and backend tests are also run as part of the continuous integration suite, as they normally are. -1. Commit any changes to the [input specification files](#input-specification-files), [output specification files](#output-specification-files), or [example snapshot files](#example-snapshot-files). +1. Commit any changes to the [input specification files](#input-specification-files), [output specification files](#output-specification-files), or [example snapshot files](#output-example-snapshot-files). diff --git a/doc/development/go_guide/index.md b/doc/development/go_guide/index.md index 197c7616d82..b561ebc4285 100644 --- a/doc/development/go_guide/index.md +++ b/doc/development/go_guide/index.md @@ -408,7 +408,7 @@ should be used in functions that can block and passed as the first parameter. Every project should have a `Dockerfile` at the root of their repository, to build and run the project. Since Go program are static binaries, they should not require any external dependency, and shells in the final image are useless. -We encourage [Multistage builds](https://docs.docker.com/develop/develop-images/multistage-build/): +We encourage [Multistage builds](https://docs.docker.com/build/building/multi-stage/): - They let the user build the project with the right Go version and dependencies. diff --git a/doc/development/graphql_guide/pagination.md b/doc/development/graphql_guide/pagination.md index 7b9d2158c60..daa39875119 100644 --- a/doc/development/graphql_guide/pagination.md +++ b/doc/development/graphql_guide/pagination.md @@ -20,7 +20,7 @@ See the [general pagination guidelines section](../database/pagination_guideline This is the traditional, page-by-page pagination, that is most common, and used across much of GitLab. You can recognize it by -a list of page numbers near the bottom of a page, which, when clicked, +a list of page numbers near the bottom of a page, which, when selected, take you to that page of results. For example, when you select **Page 100**, we send `100` to the @@ -248,7 +248,7 @@ incorrect sort order. There are times when the [complexity of sorting](#limitations-of-query-complexity) is more than our keyset pagination can handle. -For example, in [`IssuesResolver`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/graphql/resolvers/issues_resolver.rb), +For example, in [`ProjectIssuesResolver`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/graphql/resolvers/project_issues_resolver.rb), when sorting by `priority_asc`, we can't use keyset pagination as the ordering is much too complex. For more information, read [`issuable.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/concerns/issuable.rb). diff --git a/doc/development/i18n/externalization.md b/doc/development/i18n/externalization.md index 158eb19764b..91e2efcb2a3 100644 --- a/doc/development/i18n/externalization.md +++ b/doc/development/i18n/externalization.md @@ -463,12 +463,17 @@ use `%{created_at}` in Ruby but `%{createdAt}` in JavaScript. Make sure to The `n_` and `n__` methods should only be used to fetch pluralized translations of the same string, not to control the logic of showing different strings for different -quantities. Some languages have different quantities of target plural forms. +quantities. For similar strings, pluralize the entire sentence to provide the most context +when translating. Some languages have different quantities of target plural forms. For example, Chinese (simplified) has only one target plural form in our translation tool. This means the translator has to choose to translate only one of the strings, and the translation doesn't behave as intended in the other case. -For example, use this: +Below are some examples: + +Example 1: For different strings + +Use this: ```ruby if selected_projects.one? @@ -485,6 +490,27 @@ Instead of this: format(n_("%{project_name}", "%d projects selected", count), project_name: 'GitLab') ``` +Example 2: For similar strings + +Use this: + +```ruby +n__('Last day', 'Last %d days', days.length) +``` + +Instead of this: + +```ruby +# incorrect usage example +const pluralize = n__('day', 'days', days.length) + +if (days.length === 1 ) { + return sprintf(s__('Last %{pluralize}', pluralize) +} + +return sprintf(s__('Last %{dayNumber} %{pluralize}'), { dayNumber: days.length, pluralize }) +``` + ### Namespaces A namespace is a way to group translations that belong together. They provide context to our diff --git a/doc/development/i18n/merging_translations.md b/doc/development/i18n/merging_translations.md index 2a086c796c9..f98e5119916 100644 --- a/doc/development/i18n/merging_translations.md +++ b/doc/development/i18n/merging_translations.md @@ -36,7 +36,7 @@ it should be disapproved in Crowdin. Our strings must use [variables](externaliz for HTML instead. It might be useful to pause the integration on the Crowdin side for a -moment so translations don't keep coming. You can do this by clicking +moment so translations don't keep coming. You can do this by selecting **Pause sync** on the [Crowdin integration settings page](https://translate.gitlab.com/project/gitlab-ee/settings#integration). ## Merging translations diff --git a/doc/development/i18n/proofreader.md b/doc/development/i18n/proofreader.md index b5d57f9912b..8a4ad73efe3 100644 --- a/doc/development/i18n/proofreader.md +++ b/doc/development/i18n/proofreader.md @@ -58,6 +58,7 @@ are very appreciative of the work done by translators and proofreaders! - French - Davy Defaud - [GitLab](https://gitlab.com/DevDef), [Crowdin](https://crowdin.com/profile/DevDef) - Germain Gorisse - [GitLab](https://gitlab.com/ggorisse), [Crowdin](https://crowdin.com/profile/germaingorisse) + - Xavier Delatour - [GitLab](https://gitlab.com/xdelatour), [Crowdin](https://crowdin.com/profile/xdelatour) - Galician - Antón Méixome - [Crowdin](https://crowdin.com/profile/meixome) - Pedro Garcia - [GitLab](https://gitlab.com/pedgarrod), [Crowdin](https://crowdin.com/profile/breaking_pitt) diff --git a/doc/development/import_export.md b/doc/development/import_export.md index f864dd3b678..9aab7f38dbb 100644 --- a/doc/development/import_export.md +++ b/doc/development/import_export.md @@ -6,116 +6,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Import/Export development documentation -Troubleshooting and general development guidelines and tips for the [Import/Export feature](../user/project/settings/import_export.md). +General development guidelines and tips for the [Import/Export feature](../user/project/settings/import_export.md). <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> This document is originally based on the [Import/Export 201 presentation available on YouTube](https://www.youtube.com/watch?v=V3i1OfExotE). -## Troubleshooting commands - -Finds information about the status of the import and further logs using the JID: - -```ruby -# Rails console -Project.find_by_full_path('group/project').import_state.slice(:jid, :status, :last_error) -> {"jid"=>"414dec93f941a593ea1a6894", "status"=>"finished", "last_error"=>nil} -``` - -```shell -# Logs -grep JID /var/log/gitlab/sidekiq/current -grep "Import/Export error" /var/log/gitlab/sidekiq/current -grep "Import/Export backtrace" /var/log/gitlab/sidekiq/current -tail /var/log/gitlab/gitlab-rails/importer.log -``` - -## Troubleshooting performance issues - -Read through the current performance problems using the Import/Export below. - -### OOM errors - -Out of memory (OOM) errors are normally caused by the [Sidekiq Memory Killer](../administration/operations/sidekiq_memory_killer.md): - -```shell -SIDEKIQ_MEMORY_KILLER_MAX_RSS = 2000000 -SIDEKIQ_MEMORY_KILLER_HARD_LIMIT_RSS = 3000000 -SIDEKIQ_MEMORY_KILLER_GRACE_TIME = 900 -``` - -An import status `started`, and the following Sidekiq logs signal a memory issue: - -```shell -WARN: Work still in progress <struct with JID> -``` - -### Timeouts - -Timeout errors occur due to the `Gitlab::Import::StuckProjectImportJobsWorker` marking the process as failed: - -```ruby -module Gitlab - module Import - class StuckProjectImportJobsWorker - include Gitlab::Import::StuckImportJob - # ... - end - end -end - -module Gitlab - module Import - module StuckImportJob - # ... - IMPORT_JOBS_EXPIRATION = 15.hours.to_i - # ... - def perform - stuck_imports_without_jid_count = mark_imports_without_jid_as_failed! - stuck_imports_with_jid_count = mark_imports_with_jid_as_failed! - - track_metrics(stuck_imports_with_jid_count, stuck_imports_without_jid_count) - end - # ... - end - end -end -``` - -```shell -Marked stuck import jobs as failed. JIDs: xyz -``` - -```plaintext - +-----------+ +-----------------------------------+ - |Export Job |--->| Calls ActiveRecord `as_json` and | - +-----------+ | `to_json` on all project models | - +-----------------------------------+ - - +-----------+ +-----------------------------------+ - |Import Job |--->| Loads all JSON in memory, then | - +-----------+ | inserts into the DB in batches | - +-----------------------------------+ -``` - -### Problems and solutions - -| Problem | Possible solutions | -| -------- | -------- | -| [Slow JSON](https://gitlab.com/gitlab-org/gitlab/-/issues/25251) loading/dumping models from the database | [split the worker](https://gitlab.com/gitlab-org/gitlab/-/issues/25252) | -| | Batch export -| | Optimize SQL -| | Move away from `ActiveRecord` callbacks (difficult) -| High memory usage (see also some [analysis](https://gitlab.com/gitlab-org/gitlab/-/issues/18857) | DB Commit sweet spot that uses less memory | -| | [Netflix Fast JSON API](https://github.com/Netflix/fast_jsonapi) may help | -| | Batch reading/writing to disk and any SQL - -### Temporary solutions - -While the performance problems are not tackled, there is a process to workaround -importing big projects, using a foreground import: - -[Foreground import](https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/5384) of big projects for customers. -(Using the import template in the [infrastructure tracker](https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues)) - ## Security The Import/Export feature is constantly updated (adding new things to export), however diff --git a/doc/development/integrations/index.md b/doc/development/integrations/index.md index a0a81775391..2639da818c6 100644 --- a/doc/development/integrations/index.md +++ b/doc/development/integrations/index.md @@ -10,9 +10,9 @@ description: "GitLab's development guidelines for Integrations" This page provides development guidelines for implementing [GitLab integrations](../../user/project/integrations/index.md), which are part of our [main Rails project](https://gitlab.com/gitlab-org/gitlab). -Also see our [direction page](https://about.gitlab.com/direction/ecosystem/integrations/) for an overview of our strategy around integrations. +Also see our [direction page](https://about.gitlab.com/direction/manage/integrations/) for an overview of our strategy around integrations. -This guide is a work in progress. You're welcome to ping `@gitlab-org/ecosystem-stage/integrations` +This guide is a work in progress. You're welcome to ping `@gitlab-org/manage/integrations` if you need clarification or spot any outdated information. ## Add a new integration diff --git a/doc/development/integrations/jira_connect.md b/doc/development/integrations/jira_connect.md index fb0d239db46..d4215662db4 100644 --- a/doc/development/integrations/jira_connect.md +++ b/doc/development/integrations/jira_connect.md @@ -109,4 +109,4 @@ The following steps describe setting up an environment to test the GitLab OAuth 1. Go to **Admin > Settings > General**. 1. Scroll down and expand the GitLab for Jira App section. 1. Go to [gitpod.io/variables](https://gitpod.io/variables). -1. Paste the Application ID into the **Jira Connect Application ID** field and click **Save changes** +1. Paste the Application ID into the **Jira Connect Application ID** field and select **Save changes**. diff --git a/doc/development/integrations/secure.md b/doc/development/integrations/secure.md index f40caf29cfa..787b46133ad 100644 --- a/doc/development/integrations/secure.md +++ b/doc/development/integrations/secure.md @@ -157,7 +157,7 @@ If the scanner requires a fully functional Linux environment, it is recommended to use a [Debian](https://www.debian.org/intro/about) "slim" distribution or [Alpine Linux](https://www.alpinelinux.org/). If possible, it is recommended to build the image from scratch, using the `FROM scratch` instruction, and to compile the scanner with all the libraries it needs. -[Multi-stage builds](https://docs.docker.com/develop/develop-images/multistage-build/) +[Multi-stage builds](https://docs.docker.com/build/building/multi-stage/) might also help with keeping the image small. To keep an image size small, consider using [dive](https://github.com/wagoodman/dive#dive) to analyze layers in a Docker image to @@ -314,7 +314,7 @@ This documentation gives an overview of the report JSON format, as well as recommendations and examples to help integrators set its fields. The format is extensively described in the documentation of [SAST](../../user/application_security/sast/index.md#reports-json-format), -[DAST](../../user/application_security/dast/index.md#reports), +[DAST](../../user/application_security/dast/proxy-based.md#reports), [Dependency Scanning](../../user/application_security/dependency_scanning/index.md#reports-json-format), and [Container Scanning](../../user/application_security/container_scanning/index.md#reports-json-format) diff --git a/doc/development/integrations/secure_partner_integration.md b/doc/development/integrations/secure_partner_integration.md index ba0654971d3..bcbc02d4827 100644 --- a/doc/development/integrations/secure_partner_integration.md +++ b/doc/development/integrations/secure_partner_integration.md @@ -77,7 +77,7 @@ and complete an integration with the Secure stage. 1. Get a test account to begin developing your integration. You can request a [GitLab.com Subscription Sandbox](https://about.gitlab.com/partners/technology-partners/integrate/#gitlabcom-subscription-sandbox-request) or an [EE Developer License](https://about.gitlab.com/partners/technology-partners/integrate/#requesting-ultimate-dev-license-for-rd). -1. Provide a [pipeline job](../../development/pipelines.md) +1. Provide a [pipeline job](../../development/pipelines) template that users could integrate into their own GitLab pipelines. 1. Create a report artifact with your pipeline jobs. 1. Ensure your pipeline jobs create a report artifact that GitLab can process diff --git a/doc/development/internal_api/index.md b/doc/development/internal_api/index.md index 9b77a20a0d6..68d9b88bc05 100644 --- a/doc/development/internal_api/index.md +++ b/doc/development/internal_api/index.md @@ -496,8 +496,6 @@ metric counters. | Attribute | Type | Required | Description | |:---------------------------------------------------------------------------|:--------------|:---------|:-----------------------------------------------------------------------------------------------------------------| -| `gitops_sync_count` (DEPRECATED) | integer | no | The number to increase the `gitops_sync` counter by | -| `k8s_api_proxy_request_count` (DEPRECATED) | integer | no | The number to increase the `k8s_api_proxy_request` counter by | | `counters` | hash | no | The number to increase the `k8s_api_proxy_request` counter by | | `counters["k8s_api_proxy_request"]` | integer | no | The number to increase the `k8s_api_proxy_request` counter by | | `counters["gitops_sync"]` | integer | no | The number to increase the `gitops_sync` counter by | @@ -512,7 +510,7 @@ Example Request: ```shell curl --request POST --header "Gitlab-Kas-Api-Request: <JWT token>" --header "Content-Type: application/json" \ - --data '{"gitops_sync_count":1}' "http://localhost:3000/api/v4/internal/kubernetes/usage_metrics" + --data '{"counters": {"gitops_sync":1}}' "http://localhost:3000/api/v4/internal/kubernetes/usage_metrics" ``` ### Create Starboard vulnerability diff --git a/doc/development/kubernetes.md b/doc/development/kubernetes.md index e7764326db3..332a3c4ab09 100644 --- a/doc/development/kubernetes.md +++ b/doc/development/kubernetes.md @@ -155,15 +155,9 @@ Mitigation strategies include: ## Debugging Kubernetes integrations Logs related to the Kubernetes integration can be found in -[`kubernetes.log`](../administration/logs/index.md#kuberneteslog). On a local +[`kubernetes.log`](../administration/logs/index.md#kuberneteslog-deprecated). On a local GDK install, these logs are present in `log/kubernetes.log`. -Some services such as -[`Clusters::Applications::InstallService`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/clusters/applications/install_service.rb#L18) -rescues `StandardError` which can make it harder to debug issues in an -development environment. The current workaround is to temporarily -comment out the `rescue` in your local development source. - You can also follow the installation logs to debug issues related to installation. Once the installation/upgrade is underway, wait for the pod to be created. Then run the following to obtain the pods logs as diff --git a/doc/development/licensed_feature_availability.md b/doc/development/licensed_feature_availability.md deleted file mode 100644 index b007df8f1da..00000000000 --- a/doc/development/licensed_feature_availability.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'ee_features.md' -remove_date: '2022-10-08' ---- - -This document was moved to [another location](ee_features.md). - -<!-- This redirect file can be deleted after <2022-10-08>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/merge_request_concepts/widget_extensions.md b/doc/development/merge_request_concepts/widget_extensions.md deleted file mode 100644 index 097e9155f2b..00000000000 --- a/doc/development/merge_request_concepts/widget_extensions.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../fe_guide/merge_request_widget_extensions.md' -remove_date: '2022-10-27' ---- - -This document was moved to [another location](../fe_guide/merge_request_widget_extensions.md). - -<!-- This redirect file can be deleted after <2022-10-27>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/migration_style_guide.md b/doc/development/migration_style_guide.md index f59c1fd8368..5764c876e4d 100644 --- a/doc/development/migration_style_guide.md +++ b/doc/development/migration_style_guide.md @@ -52,9 +52,18 @@ work it needs to perform and how long it takes to complete: of release manager through the [post-deploy migration pipeline](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/post_deploy_migration/readme.md#how-to-determine-if-a-post-deploy-migration-has-been-executed-on-gitlabcom). These migrations can be used for schema changes that aren't critical for the application to operate, or data migrations that take at most a few minutes. Common examples for schema changes that should run post-deploy include: + - Clean-ups, like removing unused columns. - Adding non-critical indices on high-traffic tables. - Adding non-critical indices that take a long time to create. + + These migrations should not be used for schema changes that are critical for the application to operate. Making such + schema changes in a post-deployment migration have caused issues in the past, for example [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/378582). + Changes that should always be a regular schema migration and not be executed in a post-deployment migration include: + + - Creating a new table, example: `create_table`. + - Adding a new column to an existing table, example: `add_column`. + 1. [**Batched background migrations.**](database/batched_background_migrations.md) These aren't regular Rails migrations, but application code that is executed via Sidekiq jobs, although a post-deployment migration is used to schedule them. Use them only for data migrations that exceed the timing guidelines for post-deploy migrations. Batched background migrations should _not_ change the schema. @@ -897,9 +906,14 @@ end Table **has records** but **no foreign keys**: -- First release: Remove the application code related to the table, such as models, -controllers and services. -- Second release: Use the `drop_table` method in your migration. +- Remove the application code related to the table, such as models, + controllers and services. +- In a post-deployment migration, use `drop_table`. + +This can all be in a single migration if you're sure the code is not used. +If you want to reduce risk slightly, consider putting the migrations into a +second merge request after the application changes are merged. This approach +provides an opportunity to roll back. ```ruby def up @@ -913,12 +927,16 @@ end Table **has foreign keys**: -- First release: Remove the application code related to the table, such as models, -controllers, and services. -- Second release: Remove the foreign keys using the `with_lock_retries` -helper method. Use `drop_table` in another migration file. +- Remove the application code related to the table, such as models, + controllers, and services. +- In a post-deployment migration, remove the foreign keys using the + `with_lock_retries` helper method. In another subsequent post-deployment + migration, use `drop_table`. -**Migrations for the second release:** +This can all be in a single migration if you're sure the code is not used. +If you want to reduce risk slightly, consider putting the migrations into a +second merge request after the application changes are merged. This approach +provides an opportunity to roll back. Removing the foreign key on the `projects` table: diff --git a/doc/development/multi_version_compatibility.md b/doc/development/multi_version_compatibility.md index 808554d6027..8e2103b2f3e 100644 --- a/doc/development/multi_version_compatibility.md +++ b/doc/development/multi_version_compatibility.md @@ -258,7 +258,7 @@ For more information, see [the relevant issue](https://gitlab.com/gitlab-org/git We bumped the Markdown cache version and found a bug when a user edited a description or comment which was generated from a different Markdown cache version. The cached HTML wasn't generated properly after saving. In most cases, this wouldn't have happened because users would have -viewed the Markdown before clicking **Edit** and that would mean the Markdown cache is refreshed. But because we run mixed versions, this is +viewed the Markdown before selecting **Edit** and that would mean the Markdown cache is refreshed. But because we run mixed versions, this is more likely to happen. Another user on a different version could view the same page and refresh the cache to the other version behind the scenes. For more information, see [the relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/208255). diff --git a/doc/development/packages/cleanup_policies.md b/doc/development/packages/cleanup_policies.md new file mode 100644 index 00000000000..aa60cde450b --- /dev/null +++ b/doc/development/packages/cleanup_policies.md @@ -0,0 +1,122 @@ +--- +stage: Package +group: Container Registry +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Cleanup policies + +Cleanup policies are recurrent background processes that automatically remove +objects according to some parameters set by users. + +## Container Registry + +Cleanup policies for the container registry work on all the container repositories +hosted in a single project. All tags that match the cleanup parameters are removed. + +### Parameters + +The [ContainerExpirationPolicy](https://gitlab.com/gitlab-org/gitlab/-/blob/37a76cbfb54a9a3f0dba3c3748eaaac82fb8bf4b/app/models/container_expiration_policy.rb) +holds all parameters for the container registry cleanup policies. + +The parameters are split into two groups: + +- The parameters that define tags to keep: + - `keep_n`. Keep the `n` most recent tags. + - `name_regex_keep`. Keep tags matching this regular expression. +- The parameters that define tags to destroy: + - `older_than`. Destroy tags older than this timestamp. + - `name_regex`. Destroy tags matching this regular expression. + +The remaining parameters impact when the policy is executed: + +- `enabled`. Defines if the policy is enabled or not. +- `cadence`. Defines the execution cadence of the policy. +- `next_run_at`. Defines when the next execution should happen. + +### Execution + +Due to the large number of policies we need to process on GitLab.com, the execution +follows this design. + +- Policy executions are limited in time. +- Policy executions are either complete or partial. +- The background jobs will consider the next job to be executed based on two +priorities: + - Policy with a `next_run_at` in the past. + - Partially executed policies. + +To track the cleanup policy status on a container repository, +we have an `expiration_policy_cleanup_status` on the `ContainerRepository` +model. + +Background jobs for this execution are organized on: + +- A cron background job that runs every hour. +- A set of background jobs that will loop on container repositories that need +a policy execution. + +#### The cron background job + +The [cron background job](https://gitlab.com/gitlab-org/gitlab/-/blob/36454d77a8de76a25896efd7c051d6796985f579/app/workers/container_expiration_policy_worker.rb) +is quite simple. +Its main tasks are: + +1. Check if there are any container repositories in need of a cleanup. If any, +enqueue as many limited capacity jobs as necessary, up to a limit. +1. Compute metrics for cleanup policies and log them. + +#### The limited capacity job + +This [job](https://gitlab.com/gitlab-org/gitlab/-/blob/36454d77a8de76a25896efd7c051d6796985f579/app/workers/container_expiration_policies/cleanup_container_repository_worker.rb) +is based on the [limited capacity concern](../sidekiq/limited_capacity_worker.md). + +This job will run in parallel up to [a specific capacity](settings.md#container-registry). + +The primary responsibility of this job is to select the next container +repository that requires cleaning and call the related service on it. + +This is where the two priorities are evaluated in order. If a container repository +is found, the cleanup service is called on it. + +To ensure that only one cleaning is executed on a given container repository +at any time, we use a database lock along with the +`expiration_policy_cleanup_status` column. + +This job will re-enqueue itself until no more container repositories require cleanup. + +#### Services + +Here is the services call that will happen from the limited capacity job: + +```mermaid +flowchart TD + job[Limited capacity job] --> cleanup([ContainerExpirationPolicies::CleanupService]) + cleanup --> cleanup_tags([Projects::ContainerRepository::CleanupTagsService]) + cleanup_tags --> delete_tags([Projects::ContainerRepository::DeleteTagsService]) +``` + +- [`ContainerExpirationPolicies::CleanupService`](https://gitlab.com/gitlab-org/gitlab/-/blob/6546ffc6fe4e9b447a1b7f050edddb8926fe4a3d/app/services/container_expiration_policies/cleanup_service.rb). +This service mainly deals with container repository `expiration_policy_cleanup_status` +updates and will call the cleanup tags service. +- [`Projects::ContainerRepository::CleanupTagsService`](https://gitlab.com/gitlab-org/gitlab/-/blob/f23d70b7d638c38d71af102cfd32a3f6751596f9/app/services/projects/container_repository/cleanup_tags_service.rb). +This service receives the policy parameters and builds the list of tags to +destroy on the container registry. +- [`Projects::ContainerRepository::DeleteTagsService`](https://gitlab.com/gitlab-org/gitlab/-/blob/f23d70b7d638c38d71af102cfd32a3f6751596f9/app/services/projects/container_repository/delete_tags_service.rb). +This service receives a list of tags and loops on that list. For each tag, +the service will call the container registry API endpoint to destroy the target tag. + +The cleanup tags service uses a very specific [execution order](../../user/packages/container_registry/reduce_container_registry_storage.md#how-the-cleanup-policy-works) +to build the list of tags to destroy. + +Lastly, the cleanup tags service and delete tags service work using facades. +The actual implementation depends on the type of container registry connected. +If the GitLab container registry is connected, several improvements are available +and used during cleanup policies execution, such as [better use of the container registry API](https://gitlab.com/groups/gitlab-org/-/epics/8379). + +### Historic reference links + +- [First iteration](https://gitlab.com/gitlab-org/gitlab/-/issues/15398) +- [Throttling policy executions](https://gitlab.com/gitlab-org/gitlab/-/issues/208193) +- [Adding caching](https://gitlab.com/gitlab-org/gitlab/-/issues/339129) +- [Further improvements](https://gitlab.com/groups/gitlab-org/-/epics/8379) diff --git a/doc/development/packages/debian_repository.md b/doc/development/packages/debian_repository.md index 71f6c916652..26a33c548d8 100644 --- a/doc/development/packages/debian_repository.md +++ b/doc/development/packages/debian_repository.md @@ -1,6 +1,6 @@ --- stage: Package -group: Package +group: Package Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/packages/dependency_proxy.md b/doc/development/packages/dependency_proxy.md index f3e323fb9fa..d5cc219cba0 100644 --- a/doc/development/packages/dependency_proxy.md +++ b/doc/development/packages/dependency_proxy.md @@ -1,6 +1,6 @@ --- stage: Package -group: Package +group: Container Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -130,7 +130,7 @@ graph TD A[Receive manifest request] --> | We have the manifest cached.| B{Docker manifest HEAD request} A --> | We do not have manifest cached.| C{Docker manifest GET request} B --> | Digest matches the one in the DB | D[Fetch manifest from cache] - B --> | Network failure, cannot reach DockerHub | D[Fetch manifest from cache] + B --> | HEAD request error, network failure, cannot reach DockerHub | D[Fetch manifest from cache] B --> | Digest does not match the one in DB | C C --> E[Save manifest to cache, save digest to database] D --> F diff --git a/doc/development/packages/index.md b/doc/development/packages/index.md index 41d6e1b84fd..e6ec7e9654a 100644 --- a/doc/development/packages/index.md +++ b/doc/development/packages/index.md @@ -1,15 +1,18 @@ --- stage: Package -group: Package +group: Package Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Package Registry Development +# Package and container registry documentation -Development and Architectural documentation for the package registry +The documentation for package and container registry development is split into two groups. + +## Package registry development + +Development and architectural documentation for the package registry: - [Debian repository structure](debian_repository.md) -- [Dependency proxy structure](dependency_proxy.md) - [Developing a new format](new_format_development.md) - [Settings](settings.md) - [Structure / Schema](structure.md) @@ -25,3 +28,12 @@ Development and Architectural documentation for the package registry - [NuGet](../../api/packages/nuget.md) - [PyPI](../../api/packages/pypi.md) - [Ruby Gems](../../api/packages/rubygems.md) + +## Container registry development + +Development and architectural documentation for the container registry + +- [Dependency proxy structure](dependency_proxy.md) +- [Settings](settings.md) +- [Structure / Schema](structure.md) +- [Cleanup policies](cleanup_policies.md) diff --git a/doc/development/packages/new_format_development.md b/doc/development/packages/new_format_development.md index e9decea3094..f730d4f9476 100644 --- a/doc/development/packages/new_format_development.md +++ b/doc/development/packages/new_format_development.md @@ -1,6 +1,6 @@ --- stage: Package -group: Package +group: Package Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/packages/settings.md b/doc/development/packages/settings.md index d591f708954..33caa064ab3 100644 --- a/doc/development/packages/settings.md +++ b/doc/development/packages/settings.md @@ -1,6 +1,6 @@ --- stage: Package -group: Package +group: Package Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/packages/structure.md b/doc/development/packages/structure.md index e5426f8e2b8..281ec2e6c39 100644 --- a/doc/development/packages/structure.md +++ b/doc/development/packages/structure.md @@ -1,6 +1,6 @@ --- stage: Package -group: Package +group: Package Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- diff --git a/doc/development/performance.md b/doc/development/performance.md index 4f22d9ceb03..3881fad0528 100644 --- a/doc/development/performance.md +++ b/doc/development/performance.md @@ -21,8 +21,8 @@ consistent performance of GitLab. Refer to the [Index](#performance-documentatio - [Query performance guidelines](database/query_performance.md) - [Pagination performance guidelines](../development/database/pagination_performance_guidelines.md) - [Keyset pagination performance](../development/database/keyset_pagination.md#performance) - - [Troubleshooting import/export performance issues](../development/import_export.md#troubleshooting-performance-issues) - - [Pipelines performance in the `gitlab` project](../development/pipelines.md#performance) + - [Troubleshooting import/export performance issues](../user/project/settings/import_export_troubleshooting.md#troubleshooting-performance-issues) + - [Pipelines performance in the `gitlab` project](pipelines/performance.md) - Frontend: - [Performance guidelines and monitoring](../development/fe_guide/performance.md) - [Browser performance testing guidelines](../ci/testing/browser_performance_testing.md) diff --git a/doc/development/pipelines.md b/doc/development/pipelines.md index 01e42fda2c9..a4e06e98d14 100644 --- a/doc/development/pipelines.md +++ b/doc/development/pipelines.md @@ -1,871 +1,11 @@ --- -stage: none -group: Engineering Productivity -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +redirect_to: 'pipelines/index.md' +remove_date: '2023-01-20' --- -# Pipelines for the GitLab project +This document was moved to [another location](pipelines/index.md). -Pipelines for [`gitlab-org/gitlab`](https://gitlab.com/gitlab-org/gitlab) (as well as the `dev` instance's) is configured in the usual -[`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab-ci.yml) -which itself includes files under -[`.gitlab/ci/`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/.gitlab/ci) -for easier maintenance. - -We're striving to [dogfood](https://about.gitlab.com/handbook/engineering/development/principles/#dogfooding) -GitLab [CI/CD features and best-practices](../ci/yaml/index.md) -as much as possible. - -## Minimal test jobs before a merge request is approved - -**To reduce the pipeline cost and shorten the job duration, before a merge request is approved, the pipeline will run a minimal set of RSpec & Jest tests that are related to the merge request changes.** - -After a merge request has been approved, the pipeline would contain the full RSpec & Jest tests. This will ensure that all tests -have been run before a merge request is merged. - -### Overview of the GitLab project test dependency - -To understand how the minimal test jobs are executed, we need to understand the dependency between -GitLab code (frontend and backend) and the respective tests (Jest and RSpec). -This dependency can be visualized in the following diagram: - -```mermaid -flowchart LR - subgraph frontend - fe["Frontend code"]--tested with-->jest - end - subgraph backend - be["Backend code"]--tested with-->rspec - end - - be--generates-->fixtures["frontend fixtures"] - fixtures--used in-->jest -``` - -In summary: - -- RSpec tests are dependent on the backend code. -- Jest tests are dependent on both frontend and backend code, the latter through the frontend fixtures. - -### RSpec minimal jobs - -#### Determining related RSpec test files in a merge request - -To identify the minimal set of tests needed, we use the [`test_file_finder` gem](https://gitlab.com/gitlab-org/ci-cd/test_file_finder), with two strategies: - -- dynamic mapping from test coverage tracing (generated via the [`Crystalball` gem](https://github.com/toptal/crystalball)) - ([see where it's used](https://gitlab.com/gitlab-org/gitlab/-/blob/47d507c93779675d73a05002e2ec9c3c467cd698/tooling/bin/find_tests#L15)) -- static mapping maintained in the [`tests.yml` file](https://gitlab.com/gitlab-org/gitlab/-/blob/master/tests.yml) for special cases that cannot - be mapped via coverage tracing ([see where it's used](https://gitlab.com/gitlab-org/gitlab/-/blob/47d507c93779675d73a05002e2ec9c3c467cd698/tooling/bin/find_tests#L12)) - -The test mappings contain a map of each source files to a list of test files which is dependent of the source file. - -In the `detect-tests` job, we use this mapping to identify the minimal tests needed for the current merge request. - -Later on in [the `rspec fail-fast` job](#fail-fast-job-in-merge-request-pipelines), we run the minimal tests needed for the current merge request. - -#### Exceptional cases - -In addition, there are a few circumstances where we would always run the full RSpec tests: - -- when the `pipeline:run-all-rspec` label is set on the merge request -- when the `pipeline:run-full-rspec` label is set on the merge request, this label is assigned by triage automation when the merge request is approved by any reviewer -- when the merge request is created by an automation (for example, Gitaly update or MR targeting a stable branch) -- when the merge request is created in a security mirror -- when any CI configuration file is changed (for example, `.gitlab-ci.yml` or `.gitlab/ci/**/*`) - -### Jest minimal jobs - -#### Determining related Jest test files in a merge request - -To identify the minimal set of tests needed, we pass a list of all the changed files into `jest` using the [`--findRelatedTests`](https://jestjs.io/docs/cli#--findrelatedtests-spaceseparatedlistofsourcefiles) option. -In this mode, `jest` would resolve all the dependencies of related to the changed files, which include test files that have these files in the dependency chain. - -#### Exceptional cases - -In addition, there are a few circumstances where we would always run the full Jest tests: - -- when the `pipeline:run-all-jest` label is set on the merge request -- when the merge request is created by an automation (for example, Gitaly update or MR targeting a stable branch) -- when the merge request is created in a security mirror -- when any CI configuration file is changed (for example, `.gitlab-ci.yml` or `.gitlab/ci/**/*`) -- when any frontend "core" file is changed (for example, `package.json`, `yarn.lock`, `babel.config.js`, `jest.config.*.js`, `config/helpers/**/*.js`) -- when any vendored JavaScript file is changed (for example, `vendor/assets/javascripts/**/*`) -- when any backend file is changed ([see the patterns list for details](https://gitlab.com/gitlab-org/gitlab/-/blob/3616946936c1adbd9e754c1bd06f86ba670796d8/.gitlab/ci/rules.gitlab-ci.yml#L205-216)) - -### Fork pipelines - -We run only the minimal RSpec & Jest jobs for fork pipelines, unless the `pipeline:run-all-rspec` -label is set on the MR. The goal is to reduce the CI/CD minutes consumed by fork pipelines. - -See the [experiment issue](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/1170). - -## Fail-fast job in merge request pipelines - -To provide faster feedback when a merge request breaks existing tests, we are experimenting with a -fail-fast mechanism. - -An `rspec fail-fast` job is added in parallel to all other `rspec` jobs in a merge -request pipeline. This job runs the tests that are directly related to the changes -in the merge request. - -If any of these tests fail, the `rspec fail-fast` job fails, triggering a -`fail-pipeline-early` job to run. The `fail-pipeline-early` job: - -- Cancels the currently running pipeline and all in-progress jobs. -- Sets pipeline to have status `failed`. - -For example: - -```mermaid -graph LR - subgraph "prepare stage"; - A["detect-tests"] - end - - subgraph "test stage"; - B["jest"]; - C["rspec migration"]; - D["rspec unit"]; - E["rspec integration"]; - F["rspec system"]; - G["rspec fail-fast"]; - end - - subgraph "post-test stage"; - Z["fail-pipeline-early"]; - end - - A --"artifact: list of test files"--> G - G --"on failure"--> Z -``` - -The `rspec fail-fast` is a no-op if there are more than 10 test files related to the -merge request. This prevents `rspec fail-fast` duration from exceeding the average -`rspec` job duration and defeating its purpose. - -This number can be overridden by setting a CI/CD variable named `RSPEC_FAIL_FAST_TEST_FILE_COUNT_THRESHOLD`. - -## Faster feedback when reverting merge requests - -When you need to revert a merge request, to get accelerated feedback, you can add the `~pipeline:revert` label to your merge request. - -When this label is assigned, the following steps of the CI/CD pipeline are skipped: - -- The `e2e:package-and-test` job. -- The `rspec:undercoverage` job. -- The entire [Review Apps process](testing_guide/review_apps.md). - -Apply the label to the merge request, and run a new pipeline for the MR. - -## Test jobs - -We have dedicated jobs for each [testing level](testing_guide/testing_levels.md) and each job runs depending on the -changes made in your merge request. -If you want to force all the RSpec jobs to run regardless of your changes, you can add the `pipeline:run-all-rspec` label to the merge request. - -WARNING: -Forcing all jobs on docs only related MRs would not have the prerequisite jobs and would lead to errors - -### Test suite parallelization - -Our current RSpec tests parallelization setup is as follows: - -1. The `retrieve-tests-metadata` job in the `prepare` stage ensures we have a - `knapsack/report-master.json` file: - - The `knapsack/report-master.json` file is fetched from the latest `main` pipeline which runs `update-tests-metadata` - (for now it's the 2-hourly `maintenance` scheduled master pipeline), if it's not here we initialize the file with `{}`. -1. Each `[rspec|rspec-ee] [migration|unit|integration|system|geo] n m` job are run with - `knapsack rspec` and should have an evenly distributed share of tests: - - It works because the jobs have access to the `knapsack/report-master.json` - since the "artifacts from all previous stages are passed by default". - - the jobs set their own report path to - `"knapsack/${TEST_TOOL}_${TEST_LEVEL}_${DATABASE}_${CI_NODE_INDEX}_${CI_NODE_TOTAL}_report.json"`. - - if knapsack is doing its job, test files that are run should be listed under - `Report specs`, not under `Leftover specs`. -1. The `update-tests-metadata` job (which only runs on scheduled pipelines for - [the canonical project](https://gitlab.com/gitlab-org/gitlab) takes all the - `knapsack/rspec*.json` files and merge them all together into a single - `knapsack/report-master.json` file that is saved as artifact. - -After that, the next pipeline uses the up-to-date `knapsack/report-master.json` file. - -### Flaky tests - -#### Automatic skipping of flaky tests - -Tests that are [known to be flaky](testing_guide/flaky_tests.md#automatic-retries-and-flaky-tests-detection) are -skipped unless the `$SKIP_FLAKY_TESTS_AUTOMATICALLY` variable is set to `false` or if the `~"pipeline:run-flaky-tests"` -label is set on the MR. - -See the [experiment issue](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/1069). - -#### Automatic retry of failing tests in a separate process - -Unless `$RETRY_FAILED_TESTS_IN_NEW_PROCESS` variable is set to `false` (`true` by default), RSpec tests that failed are automatically retried once in a separate -RSpec process. The goal is to get rid of most side-effects from previous tests that may lead to a subsequent test failure. - -We keep track of retried tests in the `$RETRIED_TESTS_REPORT_FILE` file saved as artifact by the `rspec:flaky-tests-report` job. - -See the [experiment issue](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/1148). - -### Single database testing - -By default, all tests run with [multiple databases](database/multiple_databases.md). - -We also run tests with a single database in nightly scheduled pipelines, and in merge requests that touch database-related files. - -If you want to force tests to run with a single database, you can add the `pipeline:run-single-db` label to the merge request. - -### Monitoring - -The GitLab test suite is [monitored](performance.md#rspec-profiling) for the `main` branch, and any branch -that includes `rspec-profile` in their name. - -### Logging - -- Rails logging to `log/test.log` is disabled by default in CI - [for performance reasons](https://jtway.co/speed-up-your-rails-test-suite-by-6-in-1-line-13fedb869ec4). - To override this setting, provide the - `RAILS_ENABLE_TEST_LOG` environment variable. - -## Review app jobs - -Consult the [Review Apps](testing_guide/review_apps.md) dedicated page for more information. - -If you want to force a Review App to be deployed regardless of your changes, you can add the `pipeline:run-review-app` label to the merge request. - -## As-if-FOSS jobs - -The `* as-if-foss` jobs run the GitLab test suite "as if FOSS", meaning as if the jobs would run in the context -of `gitlab-org/gitlab-foss`. These jobs are only created in the following cases: - -- when the `pipeline:run-as-if-foss` label is set on the merge request -- when the merge request is created in the `gitlab-org/security/gitlab` project -- when any CI configuration file is changed (for example, `.gitlab-ci.yml` or `.gitlab/ci/**/*`) - -The `* as-if-foss` jobs are run in addition to the regular EE-context jobs. They have the `FOSS_ONLY='1'` variable -set and get the `ee/` folder removed before the tests start running. - -The intent is to ensure that a change doesn't introduce a failure after `gitlab-org/gitlab` is synced to `gitlab-org/gitlab-foss`. - -## As-if-JH jobs - -NOTE: -This is disabled for now. - -The `* as-if-jh` jobs run the GitLab test suite "as if JiHu", meaning as if the jobs would run in the context -of [GitLab JH](jh_features_review.md). These jobs are only created in the following cases: - -- when the `pipeline:run-as-if-jh` label is set on the merge request -- when the `pipeline:run-all-rspec` label is set on the merge request -- when any code or backstage file is changed -- when any startup CSS file is changed - -The `* as-if-jh` jobs are run in addition to the regular EE-context jobs. The `jh/` folder is added before the tests start running. - -The intent is to ensure that a change doesn't introduce a failure after `gitlab-org/gitlab` is synced to [GitLab JH](https://jihulab.com/gitlab-cn/gitlab). - -### When to consider applying `pipeline:run-as-if-jh` label - -NOTE: -This is disabled for now. - -If a Ruby file is renamed and there's a corresponding [`prepend_mod` line](jh_features_review.md#jh-features-based-on-ce-or-ee-features), -it's likely that GitLab JH is relying on it and requires a corresponding -change to rename the module or class it's prepending. - -### Corresponding JH branch - -NOTE: -This is disabled for now. - -You can create a corresponding JH branch on [GitLab JH](https://jihulab.com/gitlab-cn/gitlab) by -appending `-jh` to the branch name. If a corresponding JH branch is found, -`* as-if-jh` jobs grab the `jh` folder from the respective branch, -rather than from the default branch `main-jh`. - -NOTE: -For now, CI will try to fetch the branch on the [GitLab JH mirror](https://gitlab.com/gitlab-org/gitlab-jh-mirrors/gitlab), so it might take some time for the new JH branch to propagate to the mirror. - -## Ruby 3.0 jobs - -You can add the `pipeline:run-in-ruby3` label to the merge request to switch -the Ruby version used for running the whole test suite to 3.0. When you do -this, the test suite will no longer run in Ruby 2.7 (default), and an -additional job `verify-ruby-2.7` will also run and always fail to remind us to -remove the label and run in Ruby 2.7 before merging the merge request. - -This should let us: - -- Test changes for Ruby 3.0 -- Make sure it will not break anything when it's merged into the default branch - -## `undercover` RSpec test - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/74859) in GitLab 14.6. - -The `rspec:undercoverage` job runs [`undercover`](https://rubygems.org/gems/undercover) -to detect, and fail if any changes introduced in the merge request has zero coverage. - -The `rspec:undercoverage` job obtains coverage data from the `rspec:coverage` -job. - -In the event of an emergency, or false positive from this job, add the -`pipeline:skip-undercoverage` label to the merge request to allow this job to -fail. - -### Troubleshooting `rspec:undercoverage` failures - -The `rspec:undercoverage` job has [known bugs](https://gitlab.com/groups/gitlab-org/-/epics/8254) -that can cause false positive failures. You can test coverage locally to determine if it's -safe to apply `~"pipeline:skip-undercoverage"`. For example, using `<spec>` as the name of the -test causing the failure: - -1. Run `SIMPLECOV=1 bundle exec rspec <spec>`. -1. Run `scripts/undercoverage`. - -If these commands return `undercover: ✅ No coverage is missing in latest changes` then you can apply `~"pipeline:skip-undercoverage"` to bypass pipeline failures. - -## Ruby versions testing - -Our test suite runs against Ruby 2 in merge requests and default branch pipelines. - -We also run our test suite against Ruby 3 on another 2-hourly scheduled pipelines, as GitLab.com will soon run on Ruby 3. - -## PostgreSQL versions testing - -Our test suite runs against PG12 as GitLab.com runs on PG12 and -[Omnibus defaults to PG12 for new installs and upgrades](../administration/package_information/postgresql_versions.md). - -We do run our test suite against PG11 and PG13 on nightly scheduled pipelines. - -We also run our test suite against PG11 upon specific database library changes in MRs and `main` pipelines (with the `rspec db-library-code pg11` job). - -### Current versions testing - -| Where? | PostgreSQL version | Ruby version | -|------------------------------------------------------------------------------------------------|-------------------------------------------------|--------------| -| Merge requests | 12 (default version), 11 for DB library changes | 2.7 (default version) | -| `master` branch commits | 12 (default version), 11 for DB library changes | 2.7 (default version) | -| `maintenance` scheduled pipelines for the `master` branch (every even-numbered hour) | 12 (default version), 11 for DB library changes | 2.7 (default version) | -| `maintenance` scheduled pipelines for the `ruby3` branch (every odd-numbered hour), see below. | 12 (default version), 11 for DB library changes | 3.0 (coded in the branch) | -| `nightly` scheduled pipelines for the `master` branch | 12 (default version), 11, 13 | 2.7 (default version) | - -The pipeline configuration for the scheduled pipeline testing Ruby 3 is -stored in the `ruby3-sync` branch. The pipeline updates the `ruby3` branch -with latest `master`, and then it triggers a regular branch pipeline for -`ruby3`. Any changes in `ruby3` are only for running the pipeline. It should -never be merged back to `master`. Any other Ruby 3 changes should go into -`master` directly, which should be compatible with Ruby 2.7. - -Previously, `ruby3-sync` was using a project token stored in `RUBY3_SYNC_TOKEN` -(now backed up in `RUBY3_SYNC_TOKEN_NOT_USED`), however due to various -permissions issues, we ended up using an access token from `gitlab-bot` so now -`RUBY3_SYNC_TOKEN` is actually an access token from `gitlab-bot`. - -### Long-term plan - -We follow the [PostgreSQL versions shipped with Omnibus GitLab](../administration/package_information/postgresql_versions.md): - -| PostgreSQL version | 14.1 (July 2021) | 14.2 (August 2021) | 14.3 (September 2021) | 14.4 (October 2021) | 14.5 (November 2021) | 14.6 (December 2021) | -| -------------------| ---------------------- | ---------------------- | ---------------------- | ---------------------- | ---------------------- | ---------------------- | -| PG12 | MRs/`2-hour`/`nightly` | MRs/`2-hour`/`nightly` | MRs/`2-hour`/`nightly` | MRs/`2-hour`/`nightly` | MRs/`2-hour`/`nightly` | MRs/`2-hour`/`nightly` | -| PG11 | `nightly` | `nightly` | `nightly` | `nightly` | `nightly` | `nightly` | -| PG13 | `nightly` | `nightly` | `nightly` | `nightly` | `nightly` | `nightly` | - -## Redis versions testing - -Our test suite runs against Redis 6 as GitLab.com runs on Redis 6 and -[Omnibus defaults to Redis 6 for new installs and upgrades](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/config/software/redis.rb). - -We do run our test suite against Redis 5 on `nightly` scheduled pipelines, specifically when running backward-compatible and forward-compatible PostgreSQL jobs. - -### Current versions testing - -| Where? | Redis version | -| ------ | ------------------ | -| MRs | 6 | -| `default branch` (non-scheduled pipelines) | 6 | -| `nightly` scheduled pipelines | 5 | - -## Pipelines types for merge requests - -In general, pipelines for an MR fall into one of the following types (from shorter to longer), depending on the changes made in the MR: - -- [Documentation pipeline](#documentation-pipeline): For MRs that touch documentation. -- [Backend pipeline](#backend-pipeline): For MRs that touch backend code. -- [Frontend pipeline](#frontend-pipeline): For MRs that touch frontend code. -- [End-to-end pipeline](#end-to-end-pipeline): For MRs that touch code in the `qa/` folder. - -A "pipeline type" is an abstract term that mostly describes the "critical path" (for example, the chain of jobs for which the sum -of individual duration equals the pipeline's duration). -We use these "pipeline types" in [metrics dashboards](https://app.periscopedata.com/app/gitlab/858266/GitLab-Pipeline-Durations) to detect what types and jobs need to be optimized first. - -An MR that touches multiple areas would be associated with the longest type applicable. For instance, an MR that touches backend -and frontend would fall into the "Frontend" pipeline type since this type takes longer to finish than the "Backend" pipeline type. - -We use the [`rules:`](../ci/yaml/index.md#rules) and [`needs:`](../ci/yaml/index.md#needs) keywords extensively -to determine the jobs that need to be run in a pipeline. Note that an MR that includes multiple types of changes would -have a pipelines that include jobs from multiple types (for example, a combination of docs-only and code-only pipelines). - -Following are graphs of the critical paths for each pipeline type. Jobs that aren't part of the critical path are omitted. - -### Documentation pipeline - -[Reference pipeline](https://gitlab.com/gitlab-org/gitlab/-/pipelines/432049110). - -```mermaid -graph LR - classDef criticalPath fill:#f66; - - 1-3["docs-lint links (5 minutes)"]; - class 1-3 criticalPath; - click 1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356757&udv=0" -``` - -### Backend pipeline - -[Reference pipeline](https://gitlab.com/gitlab-org/gitlab/-/pipelines/433316063). - -```mermaid -graph RL; - classDef criticalPath fill:#f66; - - 1-3["compile-test-assets (6 minutes)"]; - class 1-3 criticalPath; - click 1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914317&udv=0" - 1-6["setup-test-env (4 minutes)"]; - click 1-6 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914315&udv=0" - 1-14["retrieve-tests-metadata"]; - click 1-14 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356697&udv=0" - 1-15["detect-tests"]; - click 1-15 "https://app.periscopedata.com/app/gitlab/652085/EP---Jobs-Durations?widget=10113603&udv=1005715" - - 2_5-1["rspec & db jobs (24 minutes)"]; - class 2_5-1 criticalPath; - click 2_5-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations" - 2_5-1 --> 1-3 & 1-6 & 1-14 & 1-15; - - 3_2-1["rspec:coverage (5.35 minutes)"]; - class 3_2-1 criticalPath; - click 3_2-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7248745&udv=0" - 3_2-1 -.->|"(don't use needs<br/>because of limitations)"| 2_5-1; - - 4_3-1["rspec:undercoverage (3.5 minutes)"]; - class 4_3-1 criticalPath; - click 4_3-1 "https://app.periscopedata.com/app/gitlab/652085/EP---Jobs-Durations?widget=13446492&udv=1005715" - 4_3-1 --> 3_2-1; - -``` - -### Frontend pipeline - -[Reference pipeline](https://gitlab.com/gitlab-org/gitlab/-/pipelines/431913287). - -```mermaid -graph RL; - classDef criticalPath fill:#f66; - - 1-2["build-qa-image (2 minutes)"]; - click 1-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914325&udv=0" - 1-5["compile-production-assets (16 minutes)"]; - class 1-5 criticalPath; - click 1-5 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914312&udv=0" - - 2_3-1["build-assets-image (1.3 minutes)"]; - class 2_3-1 criticalPath; - 2_3-1 --> 1-5 - - 2_6-1["start-review-app-pipeline (49 minutes)"]; - class 2_6-1 criticalPath; - click 2_6-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations" - 2_6-1 --> 2_3-1 & 1-2; -``` - -### End-to-end pipeline - -[Reference pipeline](https://gitlab.com/gitlab-org/gitlab/-/pipelines/431918463). - -```mermaid -graph RL; - classDef criticalPath fill:#f66; - - 1-2["build-qa-image (2 minutes)"]; - click 1-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914325&udv=0" - 1-5["compile-production-assets (16 minutes)"]; - class 1-5 criticalPath; - click 1-5 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914312&udv=0" - 1-15["detect-tests"]; - click 1-15 "https://app.periscopedata.com/app/gitlab/652085/EP---Jobs-Durations?widget=10113603&udv=1005715" - - 2_3-1["build-assets-image (1.3 minutes)"]; - class 2_3-1 criticalPath; - 2_3-1 --> 1-5 - - 2_4-1["e2e:package-and-test (102 minutes)"]; - class 2_4-1 criticalPath; - click 2_4-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914305&udv=0" - 2_4-1 --> 1-2 & 2_3-1 & 1-15; -``` - -## CI configuration internals - -### Workflow rules - -Pipelines for the GitLab project are created using the [`workflow:rules` keyword](../ci/yaml/index.md#workflow) -feature of the GitLab CI/CD. - -Pipelines are always created for the following scenarios: - -- `main` branch, including on schedules, pushes, merges, and so on. -- Merge requests. -- Tags. -- Stable, `auto-deploy`, and security branches. - -Pipeline creation is also affected by the following CI/CD variables: - -- If `$FORCE_GITLAB_CI` is set, pipelines are created. -- If `$GITLAB_INTERNAL` is not set, pipelines are not created. - -No pipeline is created in any other cases (for example, when pushing a branch with no -MR for it). - -The source of truth for these workflow rules is defined in [`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab-ci.yml). - -### Default image - -The default image is defined in [`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab-ci.yml). - -<!-- vale gitlab.Spelling = NO --> - -It includes Ruby, Go, Git, Git LFS, Chrome, Node, Yarn, PostgreSQL, and Graphics Magick. - -<!-- vale gitlab.Spelling = YES --> - -The images used in our pipelines are configured in the -[`gitlab-org/gitlab-build-images`](https://gitlab.com/gitlab-org/gitlab-build-images) -project, which is push-mirrored to [`gitlab/gitlab-build-images`](https://dev.gitlab.org/gitlab/gitlab-build-images) -for redundancy. - -The current version of the build images can be found in the -["Used by GitLab section"](https://gitlab.com/gitlab-org/gitlab-build-images/blob/master/.gitlab-ci.yml). - -### Default variables - -In addition to the [predefined CI/CD variables](../ci/variables/predefined_variables.md), -each pipeline includes default variables defined in -[`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab-ci.yml). - -### Stages - -The current stages are: - -- `sync`: This stage is used to synchronize changes from [`gitlab-org/gitlab`](https://gitlab.com/gitlab-org/gitlab) to - [`gitlab-org/gitlab-foss`](https://gitlab.com/gitlab-org/gitlab-foss). -- `prepare`: This stage includes jobs that prepare artifacts that are needed by - jobs in subsequent stages. -- `build-images`: This stage includes jobs that prepare Docker images - that are needed by jobs in subsequent stages or downstream pipelines. -- `fixtures`: This stage includes jobs that prepare fixtures needed by frontend tests. -- `lint`: This stage includes linting and static analysis jobs. -- `test`: This stage includes most of the tests, and DB/migration jobs. -- `post-test`: This stage includes jobs that build reports or gather data from - the `test` stage's jobs (for example, coverage, Knapsack metadata, and so on). -- `review`: This stage includes jobs that build the CNG images, deploy them, and - run end-to-end tests against Review Apps (see [Review Apps](testing_guide/review_apps.md) for details). - It also includes Docs Review App jobs. -- `qa`: This stage includes jobs that perform QA tasks against the Review App - that is deployed in stage `review`. -- `post-qa`: This stage includes jobs that build reports or gather data from - the `qa` stage's jobs (for example, Review App performance report). -- `pages`: This stage includes a job that deploys the various reports as - GitLab Pages (for example, [`coverage-ruby`](https://gitlab-org.gitlab.io/gitlab/coverage-ruby/), - and `webpack-report` (found at `https://gitlab-org.gitlab.io/gitlab/webpack-report/`, but there is - [an issue with the deployment](https://gitlab.com/gitlab-org/gitlab/-/issues/233458)). -- `notify`: This stage includes jobs that notify various failures to Slack. - -### Dependency Proxy - -Some of the jobs are using images from Docker Hub, where we also use -`${GITLAB_DEPENDENCY_PROXY_ADDRESS}` as a prefix to the image path, so that we pull -images from our [Dependency Proxy](../user/packages/dependency_proxy/index.md). -By default, this variable is set from the value of `${GITLAB_DEPENDENCY_PROXY}`. - -`${GITLAB_DEPENDENCY_PROXY}` is a group CI/CD variable defined in -[`gitlab-org`](https://gitlab.com/gitlab-org) as -`${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/`. This means when we use an image -defined as: - -```yaml -image: ${GITLAB_DEPENDENCY_PROXY_ADDRESS}alpine:edge -``` - -Projects in the `gitlab-org` group pull from the Dependency Proxy, while -forks that reside on any other personal namespaces or groups fall back to -Docker Hub unless `${GITLAB_DEPENDENCY_PROXY}` is also defined there. - -#### Work around for when a pipeline is started by a Project access token user - -When a pipeline is started by a Project access token user (e.g. the `release-tools approver bot` user which -automatically updates the Gitaly version used in the main project), -[the Dependency proxy isn't accessible](https://gitlab.com/gitlab-org/gitlab/-/issues/332411#note_1130388163) -and the job fails at the `Preparing the "docker+machine" executor` step. -To work around that, we have a special workflow rule, that overrides the -`${GITLAB_DEPENDENCY_PROXY_ADDRESS}` variable so that Depdendency proxy isn't used in that case: - -```yaml -- if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $GITLAB_USER_LOGIN =~ /project_\d+_bot\d*/' - variables: - GITLAB_DEPENDENCY_PROXY_ADDRESS: "" -``` - -NOTE: -We don't directly override the `${GITLAB_DEPENDENCY_PROXY}` variable because group-level -variables have higher precedence over `.gitlab-ci.yml` variables. - -### Common job definitions - -Most of the jobs [extend from a few CI definitions](../ci/yaml/index.md#extends) -defined in [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml) -that are scoped to a single [configuration keyword](../ci/yaml/index.md#job-keywords). - -| Job definitions | Description | -|------------------|-------------| -| `.default-retry` | Allows a job to [retry](../ci/yaml/index.md#retry) upon `unknown_failure`, `api_failure`, `runner_system_failure`, `job_execution_timeout`, or `stuck_or_timeout_failure`. | -| `.default-before_script` | Allows a job to use a default `before_script` definition suitable for Ruby/Rails tasks that may need a database running (for example, tests). | -| `.setup-test-env-cache` | Allows a job to use a default `cache` definition suitable for setting up test environment for subsequent Ruby/Rails tasks. | -| `.rails-cache` | Allows a job to use a default `cache` definition suitable for Ruby/Rails tasks. | -| `.static-analysis-cache` | Allows a job to use a default `cache` definition suitable for static analysis tasks. | -| `.coverage-cache` | Allows a job to use a default `cache` definition suitable for coverage tasks. | -| `.qa-cache` | Allows a job to use a default `cache` definition suitable for QA tasks. | -| `.yarn-cache` | Allows a job to use a default `cache` definition suitable for frontend jobs that do a `yarn install`. | -| `.assets-compile-cache` | Allows a job to use a default `cache` definition suitable for frontend jobs that compile assets. | -| `.use-pg11` | Allows a job to run the `postgres` 11 and `redis` services (see [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml) for the specific versions of the services). | -| `.use-pg11-ee` | Same as `.use-pg11` but also use an `elasticsearch` service (see [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml) for the specific version of the service). | -| `.use-pg12` | Allows a job to use the `postgres` 12 and `redis` services (see [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml) for the specific versions of the services). | -| `.use-pg12-ee` | Same as `.use-pg12` but also use an `elasticsearch` service (see [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml) for the specific version of the service). | -| `.use-pg13` | Allows a job to use the `postgres` 13 and `redis` services (see [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml) for the specific versions of the services). | -| `.use-pg13-ee` | Same as `.use-pg13` but also use an `elasticsearch` service (see [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml) for the specific version of the service). | -| `.use-kaniko` | Allows a job to use the `kaniko` tool to build Docker images. | -| `.as-if-foss` | Simulate the FOSS project by setting the `FOSS_ONLY='1'` CI/CD variable. | -| `.use-docker-in-docker` | Allows a job to use Docker in Docker. | - -### `rules`, `if:` conditions and `changes:` patterns - -We're using the [`rules` keyword](../ci/yaml/index.md#rules) extensively. - -All `rules` definitions are defined in -[`rules.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rules.gitlab-ci.yml), -then included in individual jobs via [`extends`](../ci/yaml/index.md#extends). - -The `rules` definitions are composed of `if:` conditions and `changes:` patterns, -which are also defined in -[`rules.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rules.gitlab-ci.yml) -and included in `rules` definitions via [YAML anchors](../ci/yaml/yaml_optimization.md#anchors) - -#### `if:` conditions - -<!-- vale gitlab.Substitutions = NO --> - -| `if:` conditions | Description | Notes | -|------------------|-------------|-------| -| `if-not-canonical-namespace` | Matches if the project isn't in the canonical (`gitlab-org/`) or security (`gitlab-org/security`) namespace. | Use to create a job for forks (by using `when: on_success|manual`), or **not** create a job for forks (by using `when: never`). | -| `if-not-ee` | Matches if the project isn't EE (that is, project name isn't `gitlab` or `gitlab-ee`). | Use to create a job only in the FOSS project (by using `when: on_success|manual`), or **not** create a job if the project is EE (by using `when: never`). | -| `if-not-foss` | Matches if the project isn't FOSS (that is, project name isn't `gitlab-foss`, `gitlab-ce`, or `gitlabhq`). | Use to create a job only in the EE project (by using `when: on_success|manual`), or **not** create a job if the project is FOSS (by using `when: never`). | -| `if-default-refs` | Matches if the pipeline is for `master`, `main`, `/^[\d-]+-stable(-ee)?$/` (stable branches), `/^\d+-\d+-auto-deploy-\d+$/` (auto-deploy branches), `/^security\//` (security branches), merge requests, and tags. | Note that jobs aren't created for branches with this default configuration. | -| `if-master-refs` | Matches if the current branch is `master` or `main`. | | -| `if-master-push` | Matches if the current branch is `master` or `main` and pipeline source is `push`. | | -| `if-master-schedule-maintenance` | Matches if the current branch is `master` or `main` and pipeline runs on a 2-hourly schedule. | | -| `if-master-schedule-nightly` | Matches if the current branch is `master` or `main` and pipeline runs on a nightly schedule. | | -| `if-auto-deploy-branches` | Matches if the current branch is an auto-deploy one. | | -| `if-master-or-tag` | Matches if the pipeline is for the `master` or `main` branch or for a tag. | | -| `if-merge-request` | Matches if the pipeline is for a merge request. | | -| `if-merge-request-title-as-if-foss` | Matches if the pipeline is for a merge request and the MR has label ~"pipeline:run-as-if-foss" | | -| `if-merge-request-title-update-caches` | Matches if the pipeline is for a merge request and the MR has label ~"pipeline:update-cache". | | -| `if-merge-request-title-run-all-rspec` | Matches if the pipeline is for a merge request and the MR has label ~"pipeline:run-all-rspec". | | -| `if-security-merge-request` | Matches if the pipeline is for a security merge request. | | -| `if-security-schedule` | Matches if the pipeline is for a security scheduled pipeline. | | -| `if-nightly-master-schedule` | Matches if the pipeline is for a `master` scheduled pipeline with `$NIGHTLY` set. | | -| `if-dot-com-gitlab-org-schedule` | Limits jobs creation to scheduled pipelines for the `gitlab-org` group on GitLab.com. | | -| `if-dot-com-gitlab-org-master` | Limits jobs creation to the `master` or `main` branch for the `gitlab-org` group on GitLab.com. | | -| `if-dot-com-gitlab-org-merge-request` | Limits jobs creation to merge requests for the `gitlab-org` group on GitLab.com. | | -| `if-dot-com-gitlab-org-and-security-tag` | Limits job creation to tags for the `gitlab-org` and `gitlab-org/security` groups on GitLab.com. | | -| `if-dot-com-gitlab-org-and-security-merge-request` | Limit jobs creation to merge requests for the `gitlab-org` and `gitlab-org/security` groups on GitLab.com. | | -| `if-dot-com-gitlab-org-and-security-tag` | Limit jobs creation to tags for the `gitlab-org` and `gitlab-org/security` groups on GitLab.com. | | -| `if-dot-com-ee-schedule` | Limits jobs to scheduled pipelines for the `gitlab-org/gitlab` project on GitLab.com. | | -| `if-security-pipeline-merge-result` | Matches if the pipeline is for a security merge request triggered by `@gitlab-release-tools-bot`. | | - -<!-- vale gitlab.Substitutions = YES --> - -#### `changes:` patterns - -| `changes:` patterns | Description | -|------------------------------|--------------------------------------------------------------------------| -| `ci-patterns` | Only create job for CI configuration-related changes. | -| `ci-build-images-patterns` | Only create job for CI configuration-related changes related to the `build-images` stage. | -| `ci-review-patterns` | Only create job for CI configuration-related changes related to the `review` stage. | -| `ci-qa-patterns` | Only create job for CI configuration-related changes related to the `qa` stage. | -| `yaml-lint-patterns` | Only create job for YAML-related changes. | -| `docs-patterns` | Only create job for docs-related changes. | -| `frontend-dependency-patterns` | Only create job when frontend dependencies are updated (that is, `package.json`, and `yarn.lock`). changes. | -| `frontend-patterns` | Only create job for frontend-related changes. | -| `backend-patterns` | Only create job for backend-related changes. | -| `db-patterns` | Only create job for DB-related changes. | -| `backstage-patterns` | Only create job for backstage-related changes (that is, Danger, fixtures, RuboCop, specs). | -| `code-patterns` | Only create job for code-related changes. | -| `qa-patterns` | Only create job for QA-related changes. | -| `code-backstage-patterns` | Combination of `code-patterns` and `backstage-patterns`. | -| `code-qa-patterns` | Combination of `code-patterns` and `qa-patterns`. | -| `code-backstage-qa-patterns` | Combination of `code-patterns`, `backstage-patterns`, and `qa-patterns`. | -| `static-analysis-patterns` | Only create jobs for Static Analytics configuration-related changes. | - -## Performance - -### Interruptible pipelines - -By default, all jobs are [interruptible](../ci/yaml/index.md#interruptible), except the -`dont-interrupt-me` job which runs automatically on `main`, and is `manual` -otherwise. - -If you want a running pipeline to finish even if you push new commits to a merge -request, be sure to start the `dont-interrupt-me` job before pushing. - -### Git fetch caching - -Because GitLab.com uses the [pack-objects cache](../administration/gitaly/configure_gitaly.md#pack-objects-cache), -concurrent Git fetches of the same pipeline ref are deduplicated on -the Gitaly server (always) and served from cache (when available). - -This works well for the following reasons: - -- The pack-objects cache is enabled on all Gitaly servers on GitLab.com. -- The CI/CD [Git strategy setting](../ci/pipelines/settings.md#choose-the-default-git-strategy) for `gitlab-org/gitlab` is **Git clone**, - causing all jobs to fetch the same data, which maximizes the cache hit ratio. -- We use [shallow clone](../ci/pipelines/settings.md#limit-the-number-of-changes-fetched-during-clone) to avoid downloading the full Git - history for every job. - -### Caching strategy - -1. All jobs must only pull caches by default. -1. All jobs must be able to pass with an empty cache. In other words, caches are only there to speed up jobs. -1. We currently have several different cache definitions defined in - [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml), - with fixed keys: - - `.setup-test-env-cache` - - `.ruby-cache` - - `.rails-cache` - - `.static-analysis-cache` - - `.rubocop-cache` - - `.coverage-cache` - - `.ruby-node-cache` - - `.qa-cache` - - `.yarn-cache` - - `.assets-compile-cache` (the key includes `${NODE_ENV}` so it's actually two different caches). -1. These cache definitions are composed of [multiple atomic caches](../ci/caching/index.md#use-multiple-caches). -1. Only the following jobs, running in 2-hourly `maintenance` scheduled pipelines, are pushing (that is, updating) to the caches: - - `update-setup-test-env-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rails.gitlab-ci.yml). - - `update-gitaly-binaries-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rails.gitlab-ci.yml). - - `update-rubocop-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rails.gitlab-ci.yml). - - `update-qa-cache`, defined in [`.gitlab/ci/qa.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/qa.gitlab-ci.yml). - - `update-assets-compile-production-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). - - `update-assets-compile-test-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). - - `update-yarn-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). - - `update-storybook-yarn-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). -1. These jobs can also be forced to run in merge requests with the `pipeline:update-cache` label (this can be useful to warm the caches in a MR that updates the cache keys). - -### Artifacts strategy - -We limit the artifacts that are saved and retrieved by jobs to the minimum to reduce the upload/download time and costs, as well as the artifacts storage. - -### Components caching - -Some external components (GitLab Workhorse and frontend assets) of GitLab need to be built from source as a preliminary step for running tests. - -#### `cache-workhorse` - -In [this MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/79766), and then -[this MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/96297), -we introduced a new `cache-workhorse` job that: - -- runs automatically for all GitLab.com `gitlab-org/gitlab` scheduled pipelines -- runs automatically for any `master` commit that touches the `workhorse/` folder -- is manual for GitLab.com's `gitlab-org`'s MRs that touches caching-related files - -This job tries to download a generic package that contains GitLab Workhorse binaries needed in the GitLab test suite (under `tmp/tests/gitlab-workhorse`). - -- If the package URL returns a 404: - 1. It runs `scripts/setup-test-env`, so that the GitLab Workhorse binaries are built. - 1. It then creates an archive which contains the binaries and upload it [as a generic package](https://gitlab.com/gitlab-org/gitlab/-/packages/). -- Otherwise, if the package already exists, it exits the job successfully. - -We also changed the `setup-test-env` job to: - -1. First download the GitLab Workhorse generic package build and uploaded by `cache-workhorse`. -1. If the package is retrieved successfully, its content is placed in the right folder (for example, `tmp/tests/gitlab-workhorse`), preventing the building of the binaries when `scripts/setup-test-env` is run later on. -1. If the package URL returns a 404, the behavior doesn't change compared to the current one: the GitLab Workhorse binaries are built as part of `scripts/setup-test-env`. - -NOTE: -The version of the package is the workhorse tree SHA (for example, `git rev-parse HEAD:workhorse`). - -#### `cache-assets` - -In [this MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/96297), -we introduced three new `cache-assets:test`, `cache-assets:test as-if-foss`, -and `cache-assets:production` jobs that: - -- never run unless `$CACHE_ASSETS_AS_PACKAGE == "true"` -- runs automatically for all GitLab.com `gitlab-org/gitlab` scheduled pipelines -- runs automatically for any `master` commit that touches the assets-related folders -- is manual for GitLab.com's `gitlab-org`'s MRs that touches caching-related files - -This job tries to download a generic package that contains GitLab compiled assets -needed in the GitLab test suite (under `app/assets/javascripts/locale/**/app.js`, -and `public/assets`). - -- If the package URL returns a 404: - 1. It runs `bin/rake gitlab:assets:compile`, so that the GitLab assets are compiled. - 1. It then creates an archive which contains the assets and uploads it [as a generic package](https://gitlab.com/gitlab-org/gitlab/-/packages/). - The package version is set to the assets folders' hash sum. -- Otherwise, if the package already exists, it exits the job successfully. - -#### `compile-*-assets` - -We also changed the `compile-test-assets`, `compile-test-assets as-if-foss`, -and `compile-production-assets` jobs to: - -1. First download the "native" cache assets, which contain: - - The [compiled assets](https://gitlab.com/gitlab-org/gitlab/-/blob/a6910c9086bb28e553f5e747ec2dd50af6da3c6b/.gitlab/ci/global.gitlab-ci.yml#L86-87). - - A [`cached-assets-hash.txt` file](https://gitlab.com/gitlab-org/gitlab/-/blob/a6910c9086bb28e553f5e747ec2dd50af6da3c6b/.gitlab/ci/global.gitlab-ci.yml#L85) - containing the `SHA256` hexdigest of all the source files on which the assets depend on. - This list of files is a pessimistic list and the assets might not depend on - some of these files. At worst we compile the assets more often, which is better than - using outdated assets. - - The file is [created after assets are compiled](https://gitlab.com/gitlab-org/gitlab/-/blob/a6910c9086bb28e553f5e747ec2dd50af6da3c6b/.gitlab/ci/frontend.gitlab-ci.yml#L83). -1. We then we compute the `SHA256` hexdigest of all the source files the assets depend on, **for the current checked out branch**. We [store the hexdigest in the `GITLAB_ASSETS_HASH` variable](https://gitlab.com/gitlab-org/gitlab/-/blob/a6910c9086bb28e553f5e747ec2dd50af6da3c6b/.gitlab/ci/frontend.gitlab-ci.yml#L27). -1. If `$CACHE_ASSETS_AS_PACKAGE == "true"`, we download the generic package built and uploaded by [`cache-assets:*`](#cache-assets). - - If the cache is up-to-date for the checked out branch, we download the native cache - **and** the cache package. We could optimize that by not downloading - the genetic package but the native cache is actually very often outdated because it's - rebuilt only every 2 hours. -1. We [run the `assets_compile_script` function](https://gitlab.com/gitlab-org/gitlab/-/blob/a6910c9086bb28e553f5e747ec2dd50af6da3c6b/.gitlab/ci/frontend.gitlab-ci.yml#L35), - which [itself runs](https://gitlab.com/gitlab-org/gitlab/-/blob/c023191ef412e868ae957f3341208a41ca678403/scripts/utils.sh#L76) - the [`assets:compile` Rake task](https://gitlab.com/gitlab-org/gitlab/-/blob/c023191ef412e868ae957f3341208a41ca678403/lib/tasks/gitlab/assets.rake#L80-109). - - This task is responsible for deciding if assets need to be compiled or not. - It [compares the `HEAD` `SHA256` hexdigest from `$GITLAB_ASSETS_HASH` with the `master` hexdigest from `cached-assets-hash.txt`](https://gitlab.com/gitlab-org/gitlab/-/blob/c023191ef412e868ae957f3341208a41ca678403/lib/tasks/gitlab/assets.rake#L86). -1. If the hashes are the same, we don't compile anything. If they're different, we compile the assets. - -### Pre-clone step - -NOTE: -We no longer use this optimization for `gitlab-org/gitlab` because the [pack-objects cache](../administration/gitaly/configure_gitaly.md#pack-objects-cache) -allows Gitaly to serve the full CI/CD fetch traffic now. See [Git fetch caching](#git-fetch-caching). - -The pre-clone step works by using the `CI_PRE_CLONE_SCRIPT` variable -[defined by GitLab.com shared runners](../ci/runners/saas/linux_saas_runner.md#pre-clone-script). - ---- - -[Return to Development documentation](index.md) +<!-- This redirect file can be deleted after <2023-01-20>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/pipelines/index.md b/doc/development/pipelines/index.md new file mode 100644 index 00000000000..01bb813e794 --- /dev/null +++ b/doc/development/pipelines/index.md @@ -0,0 +1,631 @@ +--- +stage: none +group: Engineering Productivity +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Pipelines for the GitLab project + +Pipelines for [`gitlab-org/gitlab`](https://gitlab.com/gitlab-org/gitlab) (as well as the `dev` instance's) is configured in the usual +[`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab-ci.yml) +which itself includes files under +[`.gitlab/ci/`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/.gitlab/ci) +for easier maintenance. + +We're striving to [dogfood](https://about.gitlab.com/handbook/engineering/development/principles/#dogfooding) +GitLab [CI/CD features and best-practices](../../ci/yaml/index.md) +as much as possible. + +## Minimal test jobs before a merge request is approved + +**To reduce the pipeline cost and shorten the job duration, before a merge request is approved, the pipeline will run a minimal set of RSpec & Jest tests that are related to the merge request changes.** + +After a merge request has been approved, the pipeline would contain the full RSpec & Jest tests. This will ensure that all tests +have been run before a merge request is merged. + +### Overview of the GitLab project test dependency + +To understand how the minimal test jobs are executed, we need to understand the dependency between +GitLab code (frontend and backend) and the respective tests (Jest and RSpec). +This dependency can be visualized in the following diagram: + +```mermaid +flowchart LR + subgraph frontend + fe["Frontend code"]--tested with-->jest + end + subgraph backend + be["Backend code"]--tested with-->rspec + end + + be--generates-->fixtures["frontend fixtures"] + fixtures--used in-->jest +``` + +In summary: + +- RSpec tests are dependent on the backend code. +- Jest tests are dependent on both frontend and backend code, the latter through the frontend fixtures. + +### RSpec minimal jobs + +#### Determining related RSpec test files in a merge request + +To identify the minimal set of tests needed, we use the [`test_file_finder` gem](https://gitlab.com/gitlab-org/ci-cd/test_file_finder), with two strategies: + +- dynamic mapping from test coverage tracing (generated via the [`Crystalball` gem](https://github.com/toptal/crystalball)) + ([see where it's used](https://gitlab.com/gitlab-org/gitlab/-/blob/47d507c93779675d73a05002e2ec9c3c467cd698/tooling/bin/find_tests#L15)) +- static mapping maintained in the [`tests.yml` file](https://gitlab.com/gitlab-org/gitlab/-/blob/master/tests.yml) for special cases that cannot + be mapped via coverage tracing ([see where it's used](https://gitlab.com/gitlab-org/gitlab/-/blob/47d507c93779675d73a05002e2ec9c3c467cd698/tooling/bin/find_tests#L12)) + +The test mappings contain a map of each source files to a list of test files which is dependent of the source file. + +In the `detect-tests` job, we use this mapping to identify the minimal tests needed for the current merge request. + +Later on in [the `rspec fail-fast` job](#fail-fast-job-in-merge-request-pipelines), we run the minimal tests needed for the current merge request. + +#### Exceptional cases + +In addition, there are a few circumstances where we would always run the full RSpec tests: + +- when the `pipeline:run-all-rspec` label is set on the merge request +- when the `pipeline:run-full-rspec` label is set on the merge request, this label is assigned by triage automation when the merge request is approved by any reviewer +- when the merge request is created by an automation (for example, Gitaly update or MR targeting a stable branch) +- when the merge request is created in a security mirror +- when any CI configuration file is changed (for example, `.gitlab-ci.yml` or `.gitlab/ci/**/*`) + +### Jest minimal jobs + +#### Determining related Jest test files in a merge request + +To identify the minimal set of tests needed, we pass a list of all the changed files into `jest` using the [`--findRelatedTests`](https://jestjs.io/docs/cli#--findrelatedtests-spaceseparatedlistofsourcefiles) option. +In this mode, `jest` would resolve all the dependencies of related to the changed files, which include test files that have these files in the dependency chain. + +#### Exceptional cases + +In addition, there are a few circumstances where we would always run the full Jest tests: + +- when the `pipeline:run-all-jest` label is set on the merge request +- when the merge request is created by an automation (for example, Gitaly update or MR targeting a stable branch) +- when the merge request is created in a security mirror +- when any CI configuration file is changed (for example, `.gitlab-ci.yml` or `.gitlab/ci/**/*`) +- when any frontend "core" file is changed (for example, `package.json`, `yarn.lock`, `babel.config.js`, `jest.config.*.js`, `config/helpers/**/*.js`) +- when any vendored JavaScript file is changed (for example, `vendor/assets/javascripts/**/*`) +- when any backend file is changed ([see the patterns list for details](https://gitlab.com/gitlab-org/gitlab/-/blob/3616946936c1adbd9e754c1bd06f86ba670796d8/.gitlab/ci/rules.gitlab-ci.yml#L205-216)) + +### Fork pipelines + +We run only the minimal RSpec & Jest jobs for fork pipelines, unless the `pipeline:run-all-rspec` +label is set on the MR. The goal is to reduce the CI/CD minutes consumed by fork pipelines. + +See the [experiment issue](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/1170). + +## Fail-fast job in merge request pipelines + +To provide faster feedback when a merge request breaks existing tests, we are experimenting with a +fail-fast mechanism. + +An `rspec fail-fast` job is added in parallel to all other `rspec` jobs in a merge +request pipeline. This job runs the tests that are directly related to the changes +in the merge request. + +If any of these tests fail, the `rspec fail-fast` job fails, triggering a +`fail-pipeline-early` job to run. The `fail-pipeline-early` job: + +- Cancels the currently running pipeline and all in-progress jobs. +- Sets pipeline to have status `failed`. + +For example: + +```mermaid +graph LR + subgraph "prepare stage"; + A["detect-tests"] + end + + subgraph "test stage"; + B["jest"]; + C["rspec migration"]; + D["rspec unit"]; + E["rspec integration"]; + F["rspec system"]; + G["rspec fail-fast"]; + end + + subgraph "post-test stage"; + Z["fail-pipeline-early"]; + end + + A --"artifact: list of test files"--> G + G --"on failure"--> Z +``` + +The `rspec fail-fast` is a no-op if there are more than 10 test files related to the +merge request. This prevents `rspec fail-fast` duration from exceeding the average +`rspec` job duration and defeating its purpose. + +This number can be overridden by setting a CI/CD variable named `RSPEC_FAIL_FAST_TEST_FILE_COUNT_THRESHOLD`. + +## Faster feedback for merge requests that fix a broken `master` + +When you need to [fix a broken `master`](https://about.gitlab.com/handbook/engineering/workflow/#resolution-of-broken-master), you can add the `pipeline:expedite-master-fixing` label to expedite the pipelines that run on the merge request. + +When this label is assigned, the following steps of the CI/CD pipeline are skipped: + +- The `e2e:package-and-test` job. +- The `rspec:undercoverage` job. +- The entire [Review Apps process](../testing_guide/review_apps.md). + +Apply the label to the merge request, and run a new pipeline for the MR. + +Note that the merge request also needs to have the `master:broken` or `master:foss-broken` label set. + +## Test jobs + +We have dedicated jobs for each [testing level](../testing_guide/testing_levels.md) and each job runs depending on the +changes made in your merge request. +If you want to force all the RSpec jobs to run regardless of your changes, you can add the `pipeline:run-all-rspec` label to the merge request. + +WARNING: +Forcing all jobs on docs only related MRs would not have the prerequisite jobs and would lead to errors + +### Test suite parallelization + +Our current RSpec tests parallelization setup is as follows: + +1. The `retrieve-tests-metadata` job in the `prepare` stage ensures we have a + `knapsack/report-master.json` file: + - The `knapsack/report-master.json` file is fetched from the latest `main` pipeline which runs `update-tests-metadata` + (for now it's the 2-hourly `maintenance` scheduled master pipeline), if it's not here we initialize the file with `{}`. +1. Each `[rspec|rspec-ee] [migration|unit|integration|system|geo] n m` job are run with + `knapsack rspec` and should have an evenly distributed share of tests: + - It works because the jobs have access to the `knapsack/report-master.json` + since the "artifacts from all previous stages are passed by default". + - the jobs set their own report path to + `"knapsack/${TEST_TOOL}_${TEST_LEVEL}_${DATABASE}_${CI_NODE_INDEX}_${CI_NODE_TOTAL}_report.json"`. + - if knapsack is doing its job, test files that are run should be listed under + `Report specs`, not under `Leftover specs`. +1. The `update-tests-metadata` job (which only runs on scheduled pipelines for + [the canonical project](https://gitlab.com/gitlab-org/gitlab) takes all the + `knapsack/rspec*.json` files and merge them all together into a single + `knapsack/report-master.json` file that is saved as artifact. + +After that, the next pipeline uses the up-to-date `knapsack/report-master.json` file. + +### Flaky tests + +#### Automatic skipping of flaky tests + +Tests that are [known to be flaky](../testing_guide/flaky_tests.md#automatic-retries-and-flaky-tests-detection) are +skipped unless the `$SKIP_FLAKY_TESTS_AUTOMATICALLY` variable is set to `false` or if the `~"pipeline:run-flaky-tests"` +label is set on the MR. + +See the [experiment issue](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/1069). + +#### Automatic retry of failing tests in a separate process + +Unless `$RETRY_FAILED_TESTS_IN_NEW_PROCESS` variable is set to `false` (`true` by default), RSpec tests that failed are automatically retried once in a separate +RSpec process. The goal is to get rid of most side-effects from previous tests that may lead to a subsequent test failure. + +We keep track of retried tests in the `$RETRIED_TESTS_REPORT_FILE` file saved as artifact by the `rspec:flaky-tests-report` job. + +See the [experiment issue](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/1148). + +### Compatibility testing + +By default, we run all tests with the versions that runs on GitLab.com. + +Other versions (usually one back-compatible version, and one forward-compatible version) should be running in nightly scheduled pipelines. + +Exceptions to this general guideline should be motivated and documented. + +#### Single database testing + +By default, all tests run with [multiple databases](../database/multiple_databases.md). + +We also run tests with a single database in nightly scheduled pipelines, and in merge requests that touch database-related files. + +If you want to force tests to run with a single database, you can add the `pipeline:run-single-db` label to the merge request. + +### Monitoring + +The GitLab test suite is [monitored](../performance.md#rspec-profiling) for the `main` branch, and any branch +that includes `rspec-profile` in their name. + +### Logging + +- Rails logging to `log/test.log` is disabled by default in CI + [for performance reasons](https://jtway.co/speed-up-your-rails-test-suite-by-6-in-1-line-13fedb869ec4). + To override this setting, provide the + `RAILS_ENABLE_TEST_LOG` environment variable. + +## Review app jobs + +Consult the [Review Apps](../testing_guide/review_apps.md) dedicated page for more information. + +If you want to force a Review App to be deployed regardless of your changes, you can add the `pipeline:run-review-app` label to the merge request. + +## As-if-FOSS jobs + +The `* as-if-foss` jobs run the GitLab test suite "as if FOSS", meaning as if the jobs would run in the context +of `gitlab-org/gitlab-foss`. These jobs are only created in the following cases: + +- when the `pipeline:run-as-if-foss` label is set on the merge request +- when the merge request is created in the `gitlab-org/security/gitlab` project +- when any CI configuration file is changed (for example, `.gitlab-ci.yml` or `.gitlab/ci/**/*`) + +The `* as-if-foss` jobs are run in addition to the regular EE-context jobs. They have the `FOSS_ONLY='1'` variable +set and get the `ee/` folder removed before the tests start running. + +The intent is to ensure that a change doesn't introduce a failure after `gitlab-org/gitlab` is synced to `gitlab-org/gitlab-foss`. + +## As-if-JH cross project downstream pipeline + +The `start-as-if-jh` job triggers a cross project downstream pipeline which +runs the GitLab test suite "as if JiHu", meaning as if the pipeline would run +in the context of [GitLab JH](../jh_features_review.md). These jobs are only +created in the following cases: + +- when the `pipeline:run-as-if-jh` label is set on the merge request + +This pipeline runs under the context of a generated branch in the +[GitLab JH validation](https://gitlab.com/gitlab-org-sandbox/gitlab-jh-validation) +project, which is a mirror of the +[GitLab JH mirror](https://gitlab.com/gitlab-org/gitlab-jh-mirrors/gitlab). + +The generated branch name is prefixed with `as-if-jh/` along with the branch +name in the merge request. This generated branch is based on the merge request +branch, additionally adding changes downloaded from the +[corresponding JH branch](#corresponding-jh-branch) on top to turn the whole +pipeline as if JiHu. + +The intent is to ensure that a change doesn't introduce a failure after +[GitLab](https://gitlab.com/gitlab-org/gitlab) is synchronized to +[GitLab JH](https://jihulab.com/gitlab-cn/gitlab). + +### When to consider applying `pipeline:run-as-if-jh` label + +If a Ruby file is renamed and there's a corresponding [`prepend_mod` line](../jh_features_review.md#jh-features-based-on-ce-or-ee-features), +it's likely that GitLab JH is relying on it and requires a corresponding +change to rename the module or class it's prepending. + +### Corresponding JH branch + +You can create a corresponding JH branch on [GitLab JH](https://jihulab.com/gitlab-cn/gitlab) by +appending `-jh` to the branch name. If a corresponding JH branch is found, +as-if-jh pipeline grabs files from the respective branch, rather than from the +default branch `main-jh`. + +NOTE: +For now, CI will try to fetch the branch on the [GitLab JH mirror](https://gitlab.com/gitlab-org/gitlab-jh-mirrors/gitlab), so it might take some time for the new JH branch to propagate to the mirror. + +NOTE: +While [GitLab JH validation](https://gitlab.com/gitlab-org-sandbox/gitlab-jh-validation) is a mirror of +[GitLab JH mirror](https://gitlab.com/gitlab-org/gitlab-jh-mirrors/gitlab), +it does not include any corresponding JH branch beside the default `main-jh`. +This is why when we want to fetch corresponding JH branch we should fetch it +from the main mirror, rather than the validation project. + +### How as-if-JH pipeline was configured + +The whole process looks like this: + +```mermaid +flowchart TD + subgraph "JiHuLab.com" + JH["gitlab-cn/gitlab"] + end + + subgraph "GitLab.com" + Mirror["gitlab-org/gitlab-jh-mirrors/gitlab"] + Validation["gitlab-org-sandbox/gitlab-jh-validation"] + + subgraph MR["gitlab-org/gitlab merge request"] + Add["add-jh-files job"] + Prepare["prepare-as-if-jh-branch job"] + Add --"download artifacts"--> Prepare + end + + Mirror --"pull mirror with master and main-jh"--> Validation + Mirror --"download JiHu files with ADD_JH_FILES_TOKEN"--> Add + Prepare --"push as-if-jh branches with AS_IF_JH_TOKEN"--> Validation + Validation --> Pipeline["as-if-jh pipeline"] + end + + JH --"pull mirror with corresponding JH branches"--> Mirror +``` + +#### Tokens set in the project variables + +- `ADD_JH_FILES_TOKEN`: This is a [GitLab JH mirror](https://gitlab.com/gitlab-org/gitlab-jh-mirrors/gitlab) + project token with `read_api` permission, to be able to download JiHu files. +- `AS_IF_JH_TOKEN`: This is a [GitLab JH validation](https://gitlab.com/gitlab-org-sandbox/gitlab-jh-validation) + project token with `write_repository` permission, to push generated `as-if-jh/*` branch. + +#### How we generate the as-if-JH branch + +First `add-jh-files` job will download the required JiHu files from the +corresponding JH branch, saving in artifacts. Next `prepare-as-if-jh-branch` +job will create a new branch from the merge request branch, commit the +changes, and finally push the branch to the +[validation project](https://gitlab.com/gitlab-org-sandbox/gitlab-jh-validation). + +#### How we trigger and run the as-if-JH pipeline + +After having the `as-if-jh/*` branch, `start-as-if-jh` job will trigger a pipeline +in the [validation project](https://gitlab.com/gitlab-org-sandbox/gitlab-jh-validation) +to run the cross-project downstream pipeline. + +#### How the GitLab JH mirror project is set up + +The [GitLab JH mirror](https://gitlab.com/gitlab-org/gitlab-jh-mirrors/gitlab) project is private and CI is disabled. + +It's a pull mirror pulling from [GitLab JH](https://jihulab.com/gitlab-cn/gitlab), +mirroring all branches, overriding divergent refs, triggering no pipelines +when mirror is updated. + +The pulling user is [`@gitlab-jh-bot`](https://gitlab.com/gitlab-jh-bot), who +is a maintainer in the project. The credentials can be found in the 1password +engineering vault. + +No password is used from mirroring because GitLab JH is a public project. + +#### How the GitLab JH validation project is set up + +This [GitLab JH validation](https://gitlab.com/gitlab-org-sandbox/gitlab-jh-validation) project is public and CI is enabled, without any project variables. + +It's a pull mirror pulling from [GitLab JH mirror](https://gitlab.com/gitlab-org/gitlab-jh-mirrors/gitlab), +mirroring only protected branches, `master` and `main-jh`, overriding +divergent refs, triggering no pipelines when mirror is updated. + +The pulling user is [`@gitlab-jh-validation-bot`](https://gitlab.com/gitlab-jh-validation-bot), who is a maintainer in the project, and also a +reporter in the +[GitLab JH mirror](https://gitlab.com/gitlab-org/gitlab-jh-mirrors/gitlab). +The credentials can be found in the 1password engineering vault. + +A personal access token from `@gitlab-jh-validation-bot` with +`write_repository` permission is used as the password to pull changes from +the GitLab JH mirror. Username is set with `gitlab-jh-validation-bot`. + +There is also a [pipeline schedule](https://gitlab.com/gitlab-org-sandbox/gitlab-jh-validation/-/pipeline_schedules) +to run maintenance pipelines with variable `SCHEDULE_TYPE` set to `maintenance` +running every day, updating cache. + +The default CI/CD configuration file is also set at `jh/.gitlab-ci.yml` so it +runs exactly like [GitLab JH](https://jihulab.com/gitlab-cn/gitlab/-/blob/main-jh/jh/.gitlab-ci.yml). + +## Ruby 3.0 jobs + +You can add the `pipeline:run-in-ruby3` label to the merge request to switch +the Ruby version used for running the whole test suite to 3.0. When you do +this, the test suite will no longer run in Ruby 2.7 (default), and an +additional job `verify-ruby-2.7` will also run and always fail to remind us to +remove the label and run in Ruby 2.7 before merging the merge request. + +This should let us: + +- Test changes for Ruby 3.0 +- Make sure it will not break anything when it's merged into the default branch + +## `undercover` RSpec test + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/74859) in GitLab 14.6. + +The `rspec:undercoverage` job runs [`undercover`](https://rubygems.org/gems/undercover) +to detect, and fail if any changes introduced in the merge request has zero coverage. + +The `rspec:undercoverage` job obtains coverage data from the `rspec:coverage` +job. + +In the event of an emergency, or false positive from this job, add the +`pipeline:skip-undercoverage` label to the merge request to allow this job to +fail. + +### Troubleshooting `rspec:undercoverage` failures + +The `rspec:undercoverage` job has [known bugs](https://gitlab.com/groups/gitlab-org/-/epics/8254) +that can cause false positive failures. You can test coverage locally to determine if it's +safe to apply `~"pipeline:skip-undercoverage"`. For example, using `<spec>` as the name of the +test causing the failure: + +1. Run `SIMPLECOV=1 bundle exec rspec <spec>`. +1. Run `scripts/undercoverage`. + +If these commands return `undercover: ✅ No coverage is missing in latest changes` then you can apply `~"pipeline:skip-undercoverage"` to bypass pipeline failures. + +## Ruby versions testing + +Our test suite runs against Ruby 2 in merge requests and default branch pipelines. + +We also run our test suite against Ruby 3 on another 2-hourly scheduled pipelines, as GitLab.com will soon run on Ruby 3. + +## PostgreSQL versions testing + +Our test suite runs against PG12 as GitLab.com runs on PG12 and +[Omnibus defaults to PG12 for new installs and upgrades](../../administration/package_information/postgresql_versions.md). + +We do run our test suite against PG11 and PG13 on nightly scheduled pipelines. + +We also run our test suite against PG11 upon specific database library changes in MRs and `main` pipelines (with the `rspec db-library-code pg11` job). + +### Current versions testing + +| Where? | PostgreSQL version | Ruby version | +|------------------------------------------------------------------------------------------------|-------------------------------------------------|--------------| +| Merge requests | 12 (default version), 11 for DB library changes | 2.7 (default version) | +| `master` branch commits | 12 (default version), 11 for DB library changes | 2.7 (default version) | +| `maintenance` scheduled pipelines for the `master` branch (every even-numbered hour) | 12 (default version), 11 for DB library changes | 2.7 (default version) | +| `maintenance` scheduled pipelines for the `ruby3` branch (every odd-numbered hour), see below. | 12 (default version), 11 for DB library changes | 3.0 (coded in the branch) | +| `nightly` scheduled pipelines for the `master` branch | 12 (default version), 11, 13 | 2.7 (default version) | + +There are 2 pipeline schedules used for testing Ruby 3. One is triggering a +pipeline in `ruby3-sync` branch, which updates the `ruby3` branch with latest +`master`, and no pipelines will be triggered by this push. The other schedule +is triggering a pipeline in `ruby3` 5 minutes after it, which is considered +the maintenance schedule to run test suites and update cache. + +Any changes in `ruby3` are only for running the pipeline. It should +never be merged back to `master`. Any other Ruby 3 changes should go into +`master` directly, which should be compatible with Ruby 2.7. + +Previously, `ruby3-sync` was using a project token stored in `RUBY3_SYNC_TOKEN` +(now backed up in `RUBY3_SYNC_TOKEN_NOT_USED`), however due to various +permissions issues, we ended up using an access token from `gitlab-bot` so now +`RUBY3_SYNC_TOKEN` is actually an access token from `gitlab-bot`. + +### Long-term plan + +We follow the [PostgreSQL versions shipped with Omnibus GitLab](../../administration/package_information/postgresql_versions.md): + +| PostgreSQL version | 14.1 (July 2021) | 14.2 (August 2021) | 14.3 (September 2021) | 14.4 (October 2021) | 14.5 (November 2021) | 14.6 (December 2021) | +| -------------------| ---------------------- | ---------------------- | ---------------------- | ---------------------- | ---------------------- | ---------------------- | +| PG12 | MRs/`2-hour`/`nightly` | MRs/`2-hour`/`nightly` | MRs/`2-hour`/`nightly` | MRs/`2-hour`/`nightly` | MRs/`2-hour`/`nightly` | MRs/`2-hour`/`nightly` | +| PG11 | `nightly` | `nightly` | `nightly` | `nightly` | `nightly` | `nightly` | +| PG13 | `nightly` | `nightly` | `nightly` | `nightly` | `nightly` | `nightly` | + +## Redis versions testing + +Our test suite runs against Redis 6 as GitLab.com runs on Redis 6 and +[Omnibus defaults to Redis 6 for new installs and upgrades](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/config/software/redis.rb). + +We do run our test suite against Redis 5 on `nightly` scheduled pipelines, specifically when running backward-compatible and forward-compatible PostgreSQL jobs. + +### Current versions testing + +| Where? | Redis version | +| ------ | ------------------ | +| MRs | 6 | +| `default branch` (non-scheduled pipelines) | 6 | +| `nightly` scheduled pipelines | 5 | + +## Pipelines types for merge requests + +In general, pipelines for an MR fall into one of the following types (from shorter to longer), depending on the changes made in the MR: + +- [Documentation pipeline](#documentation-pipeline): For MRs that touch documentation. +- [Backend pipeline](#backend-pipeline): For MRs that touch backend code. +- [Frontend pipeline](#frontend-pipeline): For MRs that touch frontend code. +- [End-to-end pipeline](#end-to-end-pipeline): For MRs that touch code in the `qa/` folder. + +A "pipeline type" is an abstract term that mostly describes the "critical path" (for example, the chain of jobs for which the sum +of individual duration equals the pipeline's duration). +We use these "pipeline types" in [metrics dashboards](https://app.periscopedata.com/app/gitlab/858266/GitLab-Pipeline-Durations) to detect what types and jobs need to be optimized first. + +An MR that touches multiple areas would be associated with the longest type applicable. For instance, an MR that touches backend +and frontend would fall into the "Frontend" pipeline type since this type takes longer to finish than the "Backend" pipeline type. + +We use the [`rules:`](../../ci/yaml/index.md#rules) and [`needs:`](../../ci/yaml/index.md#needs) keywords extensively +to determine the jobs that need to be run in a pipeline. Note that an MR that includes multiple types of changes would +have a pipelines that include jobs from multiple types (for example, a combination of docs-only and code-only pipelines). + +Following are graphs of the critical paths for each pipeline type. Jobs that aren't part of the critical path are omitted. + +### Documentation pipeline + +[Reference pipeline](https://gitlab.com/gitlab-org/gitlab/-/pipelines/432049110). + +```mermaid +graph LR + classDef criticalPath fill:#f66; + + 1-3["docs-lint links (5 minutes)"]; + class 1-3 criticalPath; + click 1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356757&udv=0" +``` + +### Backend pipeline + +[Reference pipeline](https://gitlab.com/gitlab-org/gitlab/-/pipelines/433316063). + +```mermaid +graph RL; + classDef criticalPath fill:#f66; + + 1-3["compile-test-assets (6 minutes)"]; + class 1-3 criticalPath; + click 1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914317&udv=0" + 1-6["setup-test-env (4 minutes)"]; + click 1-6 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914315&udv=0" + 1-14["retrieve-tests-metadata"]; + click 1-14 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356697&udv=0" + 1-15["detect-tests"]; + click 1-15 "https://app.periscopedata.com/app/gitlab/652085/EP---Jobs-Durations?widget=10113603&udv=1005715" + + 2_5-1["rspec & db jobs (24 minutes)"]; + class 2_5-1 criticalPath; + click 2_5-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations" + 2_5-1 --> 1-3 & 1-6 & 1-14 & 1-15; + + 3_2-1["rspec:coverage (5.35 minutes)"]; + class 3_2-1 criticalPath; + click 3_2-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7248745&udv=0" + 3_2-1 -.->|"(don't use needs<br/>because of limitations)"| 2_5-1; + + 4_3-1["rspec:undercoverage (3.5 minutes)"]; + class 4_3-1 criticalPath; + click 4_3-1 "https://app.periscopedata.com/app/gitlab/652085/EP---Jobs-Durations?widget=13446492&udv=1005715" + 4_3-1 --> 3_2-1; + +``` + +### Frontend pipeline + +[Reference pipeline](https://gitlab.com/gitlab-org/gitlab/-/pipelines/431913287). + +```mermaid +graph RL; + classDef criticalPath fill:#f66; + + 1-2["build-qa-image (2 minutes)"]; + click 1-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914325&udv=0" + 1-5["compile-production-assets (16 minutes)"]; + class 1-5 criticalPath; + click 1-5 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914312&udv=0" + + 2_3-1["build-assets-image (1.3 minutes)"]; + class 2_3-1 criticalPath; + 2_3-1 --> 1-5 + + 2_6-1["start-review-app-pipeline (49 minutes)"]; + class 2_6-1 criticalPath; + click 2_6-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations" + 2_6-1 --> 2_3-1 & 1-2; +``` + +### End-to-end pipeline + +[Reference pipeline](https://gitlab.com/gitlab-org/gitlab/-/pipelines/431918463). + +```mermaid +graph RL; + classDef criticalPath fill:#f66; + + 1-2["build-qa-image (2 minutes)"]; + click 1-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914325&udv=0" + 1-5["compile-production-assets (16 minutes)"]; + class 1-5 criticalPath; + click 1-5 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914312&udv=0" + 1-15["detect-tests"]; + click 1-15 "https://app.periscopedata.com/app/gitlab/652085/EP---Jobs-Durations?widget=10113603&udv=1005715" + + 2_3-1["build-assets-image (1.3 minutes)"]; + class 2_3-1 criticalPath; + 2_3-1 --> 1-5 + + 2_4-1["e2e:package-and-test (102 minutes)"]; + class 2_4-1 criticalPath; + click 2_4-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914305&udv=0" + 2_4-1 --> 1-2 & 2_3-1 & 1-15; +``` + +## CI configuration internals + +See the dedicated [CI configuration internals page](internals.md). + +## Performance + +See the dedicated [CI configuration performance page](performance.md). + +--- + +[Return to Development documentation](../index.md) diff --git a/doc/development/pipelines/internals.md b/doc/development/pipelines/internals.md new file mode 100644 index 00000000000..2861e2f266b --- /dev/null +++ b/doc/development/pipelines/internals.md @@ -0,0 +1,216 @@ +--- +stage: none +group: Engineering Productivity +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# CI configuration internals + +## Workflow rules + +Pipelines for the GitLab project are created using the [`workflow:rules` keyword](../../ci/yaml/index.md#workflow) +feature of the GitLab CI/CD. + +Pipelines are always created for the following scenarios: + +- `main` branch, including on schedules, pushes, merges, and so on. +- Merge requests. +- Tags. +- Stable, `auto-deploy`, and security branches. + +Pipeline creation is also affected by the following CI/CD variables: + +- If `$FORCE_GITLAB_CI` is set, pipelines are created. +- If `$GITLAB_INTERNAL` is not set, pipelines are not created. + +No pipeline is created in any other cases (for example, when pushing a branch with no +MR for it). + +The source of truth for these workflow rules is defined in [`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab-ci.yml). + +## Default image + +The default image is defined in [`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab-ci.yml). + +<!-- vale gitlab.Spelling = NO --> + +It includes Ruby, Go, Git, Git LFS, Chrome, Node, Yarn, PostgreSQL, and Graphics Magick. + +<!-- vale gitlab.Spelling = YES --> + +The images used in our pipelines are configured in the +[`gitlab-org/gitlab-build-images`](https://gitlab.com/gitlab-org/gitlab-build-images) +project, which is push-mirrored to [`gitlab/gitlab-build-images`](https://dev.gitlab.org/gitlab/gitlab-build-images) +for redundancy. + +The current version of the build images can be found in the +["Used by GitLab section"](https://gitlab.com/gitlab-org/gitlab-build-images/blob/master/.gitlab-ci.yml). + +## Default variables + +In addition to the [predefined CI/CD variables](../../ci/variables/predefined_variables.md), +each pipeline includes default variables defined in +[`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab-ci.yml). + +## Stages + +The current stages are: + +- `sync`: This stage is used to synchronize changes from [`gitlab-org/gitlab`](https://gitlab.com/gitlab-org/gitlab) to + [`gitlab-org/gitlab-foss`](https://gitlab.com/gitlab-org/gitlab-foss). +- `prepare`: This stage includes jobs that prepare artifacts that are needed by + jobs in subsequent stages. +- `build-images`: This stage includes jobs that prepare Docker images + that are needed by jobs in subsequent stages or downstream pipelines. +- `fixtures`: This stage includes jobs that prepare fixtures needed by frontend tests. +- `lint`: This stage includes linting and static analysis jobs. +- `test`: This stage includes most of the tests, and DB/migration jobs. +- `post-test`: This stage includes jobs that build reports or gather data from + the `test` stage's jobs (for example, coverage, Knapsack metadata, and so on). +- `review`: This stage includes jobs that build the CNG images, deploy them, and + run end-to-end tests against Review Apps (see [Review Apps](../testing_guide/review_apps.md) for details). + It also includes Docs Review App jobs. +- `qa`: This stage includes jobs that perform QA tasks against the Review App + that is deployed in stage `review`. +- `post-qa`: This stage includes jobs that build reports or gather data from + the `qa` stage's jobs (for example, Review App performance report). +- `pages`: This stage includes a job that deploys the various reports as + GitLab Pages (for example, [`coverage-ruby`](https://gitlab-org.gitlab.io/gitlab/coverage-ruby/), + and `webpack-report` (found at `https://gitlab-org.gitlab.io/gitlab/webpack-report/`, but there is + [an issue with the deployment](https://gitlab.com/gitlab-org/gitlab/-/issues/233458)). +- `notify`: This stage includes jobs that notify various failures to Slack. + +## Dependency Proxy + +Some of the jobs are using images from Docker Hub, where we also use +`${GITLAB_DEPENDENCY_PROXY_ADDRESS}` as a prefix to the image path, so that we pull +images from our [Dependency Proxy](../../user/packages/dependency_proxy/index.md). +By default, this variable is set from the value of `${GITLAB_DEPENDENCY_PROXY}`. + +`${GITLAB_DEPENDENCY_PROXY}` is a group CI/CD variable defined in +[`gitlab-org`](https://gitlab.com/gitlab-org) as +`${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/`. This means when we use an image +defined as: + +```yaml +image: ${GITLAB_DEPENDENCY_PROXY_ADDRESS}alpine:edge +``` + +Projects in the `gitlab-org` group pull from the Dependency Proxy, while +forks that reside on any other personal namespaces or groups fall back to +Docker Hub unless `${GITLAB_DEPENDENCY_PROXY}` is also defined there. + +### Work around for when a pipeline is started by a Project access token user + +When a pipeline is started by a Project access token user (e.g. the `release-tools approver bot` user which +automatically updates the Gitaly version used in the main project), +[the Dependency proxy isn't accessible](https://gitlab.com/gitlab-org/gitlab/-/issues/332411#note_1130388163) +and the job fails at the `Preparing the "docker+machine" executor` step. +To work around that, we have a special workflow rule, that overrides the +`${GITLAB_DEPENDENCY_PROXY_ADDRESS}` variable so that Depdendency proxy isn't used in that case: + +```yaml +- if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $GITLAB_USER_LOGIN =~ /project_\d+_bot\d*/' + variables: + GITLAB_DEPENDENCY_PROXY_ADDRESS: "" +``` + +NOTE: +We don't directly override the `${GITLAB_DEPENDENCY_PROXY}` variable because group-level +variables have higher precedence over `.gitlab-ci.yml` variables. + +## Common job definitions + +Most of the jobs [extend from a few CI definitions](../../ci/yaml/index.md#extends) +defined in [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml) +that are scoped to a single [configuration keyword](../../ci/yaml/index.md#job-keywords). + +| Job definitions | Description | +|------------------|-------------| +| `.default-retry` | Allows a job to [retry](../../ci/yaml/index.md#retry) upon `unknown_failure`, `api_failure`, `runner_system_failure`, `job_execution_timeout`, or `stuck_or_timeout_failure`. | +| `.default-before_script` | Allows a job to use a default `before_script` definition suitable for Ruby/Rails tasks that may need a database running (for example, tests). | +| `.setup-test-env-cache` | Allows a job to use a default `cache` definition suitable for setting up test environment for subsequent Ruby/Rails tasks. | +| `.rails-cache` | Allows a job to use a default `cache` definition suitable for Ruby/Rails tasks. | +| `.static-analysis-cache` | Allows a job to use a default `cache` definition suitable for static analysis tasks. | +| `.coverage-cache` | Allows a job to use a default `cache` definition suitable for coverage tasks. | +| `.qa-cache` | Allows a job to use a default `cache` definition suitable for QA tasks. | +| `.yarn-cache` | Allows a job to use a default `cache` definition suitable for frontend jobs that do a `yarn install`. | +| `.assets-compile-cache` | Allows a job to use a default `cache` definition suitable for frontend jobs that compile assets. | +| `.use-pg11` | Allows a job to run the `postgres` 11 and `redis` services (see [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml) for the specific versions of the services). | +| `.use-pg11-ee` | Same as `.use-pg11` but also use an `elasticsearch` service (see [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml) for the specific version of the service). | +| `.use-pg12` | Allows a job to use the `postgres` 12 and `redis` services (see [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml) for the specific versions of the services). | +| `.use-pg12-ee` | Same as `.use-pg12` but also use an `elasticsearch` service (see [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml) for the specific version of the service). | +| `.use-pg13` | Allows a job to use the `postgres` 13 and `redis` services (see [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml) for the specific versions of the services). | +| `.use-pg13-ee` | Same as `.use-pg13` but also use an `elasticsearch` service (see [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml) for the specific version of the service). | +| `.use-kaniko` | Allows a job to use the `kaniko` tool to build Docker images. | +| `.as-if-foss` | Simulate the FOSS project by setting the `FOSS_ONLY='1'` CI/CD variable. | +| `.use-docker-in-docker` | Allows a job to use Docker in Docker. | + +## `rules`, `if:` conditions and `changes:` patterns + +We're using the [`rules` keyword](../../ci/yaml/index.md#rules) extensively. + +All `rules` definitions are defined in +[`rules.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rules.gitlab-ci.yml), +then included in individual jobs via [`extends`](../../ci/yaml/index.md#extends). + +The `rules` definitions are composed of `if:` conditions and `changes:` patterns, +which are also defined in +[`rules.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rules.gitlab-ci.yml) +and included in `rules` definitions via [YAML anchors](../../ci/yaml/yaml_optimization.md#anchors) + +### `if:` conditions + +<!-- vale gitlab.Substitutions = NO --> + +| `if:` conditions | Description | Notes | +|------------------|-------------|-------| +| `if-not-canonical-namespace` | Matches if the project isn't in the canonical (`gitlab-org/`) or security (`gitlab-org/security`) namespace. | Use to create a job for forks (by using `when: on_success|manual`), or **not** create a job for forks (by using `when: never`). | +| `if-not-ee` | Matches if the project isn't EE (that is, project name isn't `gitlab` or `gitlab-ee`). | Use to create a job only in the FOSS project (by using `when: on_success|manual`), or **not** create a job if the project is EE (by using `when: never`). | +| `if-not-foss` | Matches if the project isn't FOSS (that is, project name isn't `gitlab-foss`, `gitlab-ce`, or `gitlabhq`). | Use to create a job only in the EE project (by using `when: on_success|manual`), or **not** create a job if the project is FOSS (by using `when: never`). | +| `if-default-refs` | Matches if the pipeline is for `master`, `main`, `/^[\d-]+-stable(-ee)?$/` (stable branches), `/^\d+-\d+-auto-deploy-\d+$/` (auto-deploy branches), `/^security\//` (security branches), merge requests, and tags. | Note that jobs aren't created for branches with this default configuration. | +| `if-master-refs` | Matches if the current branch is `master` or `main`. | | +| `if-master-push` | Matches if the current branch is `master` or `main` and pipeline source is `push`. | | +| `if-master-schedule-maintenance` | Matches if the current branch is `master` or `main` and pipeline runs on a 2-hourly schedule. | | +| `if-master-schedule-nightly` | Matches if the current branch is `master` or `main` and pipeline runs on a nightly schedule. | | +| `if-auto-deploy-branches` | Matches if the current branch is an auto-deploy one. | | +| `if-master-or-tag` | Matches if the pipeline is for the `master` or `main` branch or for a tag. | | +| `if-merge-request` | Matches if the pipeline is for a merge request. | | +| `if-merge-request-title-as-if-foss` | Matches if the pipeline is for a merge request and the MR has label ~"pipeline:run-as-if-foss" | | +| `if-merge-request-title-update-caches` | Matches if the pipeline is for a merge request and the MR has label ~"pipeline:update-cache". | | +| `if-merge-request-title-run-all-rspec` | Matches if the pipeline is for a merge request and the MR has label ~"pipeline:run-all-rspec". | | +| `if-security-merge-request` | Matches if the pipeline is for a security merge request. | | +| `if-security-schedule` | Matches if the pipeline is for a security scheduled pipeline. | | +| `if-nightly-master-schedule` | Matches if the pipeline is for a `master` scheduled pipeline with `$NIGHTLY` set. | | +| `if-dot-com-gitlab-org-schedule` | Limits jobs creation to scheduled pipelines for the `gitlab-org` group on GitLab.com. | | +| `if-dot-com-gitlab-org-master` | Limits jobs creation to the `master` or `main` branch for the `gitlab-org` group on GitLab.com. | | +| `if-dot-com-gitlab-org-merge-request` | Limits jobs creation to merge requests for the `gitlab-org` group on GitLab.com. | | +| `if-dot-com-gitlab-org-and-security-tag` | Limits job creation to tags for the `gitlab-org` and `gitlab-org/security` groups on GitLab.com. | | +| `if-dot-com-gitlab-org-and-security-merge-request` | Limit jobs creation to merge requests for the `gitlab-org` and `gitlab-org/security` groups on GitLab.com. | | +| `if-dot-com-gitlab-org-and-security-tag` | Limit jobs creation to tags for the `gitlab-org` and `gitlab-org/security` groups on GitLab.com. | | +| `if-dot-com-ee-schedule` | Limits jobs to scheduled pipelines for the `gitlab-org/gitlab` project on GitLab.com. | | +| `if-security-pipeline-merge-result` | Matches if the pipeline is for a security merge request triggered by `@gitlab-release-tools-bot`. | | + +<!-- vale gitlab.Substitutions = YES --> + +### `changes:` patterns + +| `changes:` patterns | Description | +|------------------------------|--------------------------------------------------------------------------| +| `ci-patterns` | Only create job for CI configuration-related changes. | +| `ci-build-images-patterns` | Only create job for CI configuration-related changes related to the `build-images` stage. | +| `ci-review-patterns` | Only create job for CI configuration-related changes related to the `review` stage. | +| `ci-qa-patterns` | Only create job for CI configuration-related changes related to the `qa` stage. | +| `yaml-lint-patterns` | Only create job for YAML-related changes. | +| `docs-patterns` | Only create job for docs-related changes. | +| `frontend-dependency-patterns` | Only create job when frontend dependencies are updated (that is, `package.json`, and `yarn.lock`). changes. | +| `frontend-patterns-for-as-if-foss` | Only create job for frontend-related changes that have impact on FOSS. | +| `backend-patterns` | Only create job for backend-related changes. | +| `db-patterns` | Only create job for DB-related changes. | +| `backstage-patterns` | Only create job for backstage-related changes (that is, Danger, fixtures, RuboCop, specs). | +| `code-patterns` | Only create job for code-related changes. | +| `qa-patterns` | Only create job for QA-related changes. | +| `code-backstage-patterns` | Combination of `code-patterns` and `backstage-patterns`. | +| `code-qa-patterns` | Combination of `code-patterns` and `qa-patterns`. | +| `code-backstage-qa-patterns` | Combination of `code-patterns`, `backstage-patterns`, and `qa-patterns`. | +| `static-analysis-patterns` | Only create jobs for Static Analytics configuration-related changes. | diff --git a/doc/development/pipelines/performance.md b/doc/development/pipelines/performance.md new file mode 100644 index 00000000000..1c6f9d78879 --- /dev/null +++ b/doc/development/pipelines/performance.md @@ -0,0 +1,151 @@ +--- +stage: none +group: Engineering Productivity +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# CI configuration performance + +## Interruptible pipelines + +By default, all jobs are [interruptible](../../ci/yaml/index.md#interruptible), except the +`dont-interrupt-me` job which runs automatically on `main`, and is `manual` +otherwise. + +If you want a running pipeline to finish even if you push new commits to a merge +request, be sure to start the `dont-interrupt-me` job before pushing. + +## Git fetch caching + +Because GitLab.com uses the [pack-objects cache](../../administration/gitaly/configure_gitaly.md#pack-objects-cache), +concurrent Git fetches of the same pipeline ref are deduplicated on +the Gitaly server (always) and served from cache (when available). + +This works well for the following reasons: + +- The pack-objects cache is enabled on all Gitaly servers on GitLab.com. +- The CI/CD [Git strategy setting](../../ci/pipelines/settings.md#choose-the-default-git-strategy) for `gitlab-org/gitlab` is **Git clone**, + causing all jobs to fetch the same data, which maximizes the cache hit ratio. +- We use [shallow clone](../../ci/pipelines/settings.md#limit-the-number-of-changes-fetched-during-clone) to avoid downloading the full Git + history for every job. + +## Caching strategy + +1. All jobs must only pull caches by default. +1. All jobs must be able to pass with an empty cache. In other words, caches are only there to speed up jobs. +1. We currently have several different cache definitions defined in + [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml), + with fixed keys: + - `.setup-test-env-cache` + - `.ruby-cache` + - `.rails-cache` + - `.static-analysis-cache` + - `.rubocop-cache` + - `.coverage-cache` + - `.ruby-node-cache` + - `.qa-cache` + - `.yarn-cache` + - `.assets-compile-cache` (the key includes `${NODE_ENV}` so it's actually two different caches). +1. These cache definitions are composed of [multiple atomic caches](../../ci/caching/index.md#use-multiple-caches). +1. Only the following jobs, running in 2-hourly `maintenance` scheduled pipelines, are pushing (that is, updating) to the caches: + - `update-setup-test-env-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rails.gitlab-ci.yml). + - `update-gitaly-binaries-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rails.gitlab-ci.yml). + - `update-rubocop-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rails.gitlab-ci.yml). + - `update-qa-cache`, defined in [`.gitlab/ci/qa.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/qa.gitlab-ci.yml). + - `update-assets-compile-production-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). + - `update-assets-compile-test-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). + - `update-yarn-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). + - `update-storybook-yarn-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). +1. These jobs can also be forced to run in merge requests with the `pipeline:update-cache` label (this can be useful to warm the caches in a MR that updates the cache keys). + +## Artifacts strategy + +We limit the artifacts that are saved and retrieved by jobs to the minimum to reduce the upload/download time and costs, as well as the artifacts storage. + +## Components caching + +Some external components (GitLab Workhorse and frontend assets) of GitLab need to be built from source as a preliminary step for running tests. + +## `cache-workhorse` + +In [this MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/79766), and then +[this MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/96297), +we introduced a new `cache-workhorse` job that: + +- runs automatically for all GitLab.com `gitlab-org/gitlab` scheduled pipelines +- runs automatically for any `master` commit that touches the `workhorse/` folder +- is manual for GitLab.com's `gitlab-org`'s MRs that touches caching-related files + +This job tries to download a generic package that contains GitLab Workhorse binaries needed in the GitLab test suite (under `tmp/tests/gitlab-workhorse`). + +- If the package URL returns a 404: + 1. It runs `scripts/setup-test-env`, so that the GitLab Workhorse binaries are built. + 1. It then creates an archive which contains the binaries and upload it [as a generic package](https://gitlab.com/gitlab-org/gitlab/-/packages/). +- Otherwise, if the package already exists, it exits the job successfully. + +We also changed the `setup-test-env` job to: + +1. First download the GitLab Workhorse generic package build and uploaded by `cache-workhorse`. +1. If the package is retrieved successfully, its content is placed in the right folder (for example, `tmp/tests/gitlab-workhorse`), preventing the building of the binaries when `scripts/setup-test-env` is run later on. +1. If the package URL returns a 404, the behavior doesn't change compared to the current one: the GitLab Workhorse binaries are built as part of `scripts/setup-test-env`. + +NOTE: +The version of the package is the workhorse tree SHA (for example, `git rev-parse HEAD:workhorse`). + +## `cache-assets` + +In [this MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/96297), +we introduced three new `cache-assets:test`, `cache-assets:test as-if-foss`, +and `cache-assets:production` jobs that: + +- never run unless `$CACHE_ASSETS_AS_PACKAGE == "true"` +- runs automatically for all GitLab.com `gitlab-org/gitlab` scheduled pipelines +- runs automatically for any `master` commit that touches the assets-related folders +- is manual for GitLab.com's `gitlab-org`'s MRs that touches caching-related files + +This job tries to download a generic package that contains GitLab compiled assets +needed in the GitLab test suite (under `app/assets/javascripts/locale/**/app.js`, +and `public/assets`). + +- If the package URL returns a 404: + 1. It runs `bin/rake gitlab:assets:compile`, so that the GitLab assets are compiled. + 1. It then creates an archive which contains the assets and uploads it [as a generic package](https://gitlab.com/gitlab-org/gitlab/-/packages/). + The package version is set to the assets folders' hash sum. +- Otherwise, if the package already exists, it exits the job successfully. + +## `compile-*-assets` + +We also changed the `compile-test-assets`, `compile-test-assets as-if-foss`, +and `compile-production-assets` jobs to: + +1. First download the "native" cache assets, which contain: + - The [compiled assets](https://gitlab.com/gitlab-org/gitlab/-/blob/a6910c9086bb28e553f5e747ec2dd50af6da3c6b/.gitlab/ci/global.gitlab-ci.yml#L86-87). + - A [`cached-assets-hash.txt` file](https://gitlab.com/gitlab-org/gitlab/-/blob/a6910c9086bb28e553f5e747ec2dd50af6da3c6b/.gitlab/ci/global.gitlab-ci.yml#L85) + containing the `SHA256` hexdigest of all the source files on which the assets depend on. + This list of files is a pessimistic list and the assets might not depend on + some of these files. At worst we compile the assets more often, which is better than + using outdated assets. + + The file is [created after assets are compiled](https://gitlab.com/gitlab-org/gitlab/-/blob/a6910c9086bb28e553f5e747ec2dd50af6da3c6b/.gitlab/ci/frontend.gitlab-ci.yml#L83). +1. We then we compute the `SHA256` hexdigest of all the source files the assets depend on, **for the current checked out branch**. We [store the hexdigest in the `GITLAB_ASSETS_HASH` variable](https://gitlab.com/gitlab-org/gitlab/-/blob/a6910c9086bb28e553f5e747ec2dd50af6da3c6b/.gitlab/ci/frontend.gitlab-ci.yml#L27). +1. If `$CACHE_ASSETS_AS_PACKAGE == "true"`, we download the generic package built and uploaded by [`cache-assets:*`](#cache-assets). + - If the cache is up-to-date for the checked out branch, we download the native cache + **and** the cache package. We could optimize that by not downloading + the genetic package but the native cache is actually very often outdated because it's + rebuilt only every 2 hours. +1. We [run the `assets_compile_script` function](https://gitlab.com/gitlab-org/gitlab/-/blob/a6910c9086bb28e553f5e747ec2dd50af6da3c6b/.gitlab/ci/frontend.gitlab-ci.yml#L35), + which [itself runs](https://gitlab.com/gitlab-org/gitlab/-/blob/c023191ef412e868ae957f3341208a41ca678403/scripts/utils.sh#L76) + the [`assets:compile` Rake task](https://gitlab.com/gitlab-org/gitlab/-/blob/c023191ef412e868ae957f3341208a41ca678403/lib/tasks/gitlab/assets.rake#L80-109). + + This task is responsible for deciding if assets need to be compiled or not. + It [compares the `HEAD` `SHA256` hexdigest from `$GITLAB_ASSETS_HASH` with the `master` hexdigest from `cached-assets-hash.txt`](https://gitlab.com/gitlab-org/gitlab/-/blob/c023191ef412e868ae957f3341208a41ca678403/lib/tasks/gitlab/assets.rake#L86). +1. If the hashes are the same, we don't compile anything. If they're different, we compile the assets. + +## Pre-clone step + +NOTE: +We no longer use this optimization for `gitlab-org/gitlab` because the [pack-objects cache](../../administration/gitaly/configure_gitaly.md#pack-objects-cache) +allows Gitaly to serve the full CI/CD fetch traffic now. See [Git fetch caching](#git-fetch-caching). + +The pre-clone step works by using the `CI_PRE_CLONE_SCRIPT` variable +[defined by GitLab.com shared runners](../../ci/runners/saas/linux_saas_runner.md#pre-clone-script). diff --git a/doc/development/prometheus_metrics.md b/doc/development/prometheus_metrics.md index c2caa354567..d3d809c5386 100644 --- a/doc/development/prometheus_metrics.md +++ b/doc/development/prometheus_metrics.md @@ -36,7 +36,7 @@ After you add or change an existing common metric, you must [re-run the import s Or, you can create a database migration: ```ruby -class ImportCommonMetrics < Gitlab::Database::Migration[1.0] +class ImportCommonMetrics < Gitlab::Database::Migration[2.0] def up ::Gitlab::DatabaseImporters::CommonMetrics::Importer.new.execute end diff --git a/doc/development/rake_tasks.md b/doc/development/rake_tasks.md index 505f480c410..14fbe0e875b 100644 --- a/doc/development/rake_tasks.md +++ b/doc/development/rake_tasks.md @@ -85,6 +85,18 @@ To import these metrics, you can run: bundle exec rake 'gitlab:seed:development_metrics[your_project_id]' ``` +#### Seed a project with vulnerabilities + +You can seed a project with [security vulnerabilities](../user/application_security/vulnerabilities/index.md). + +```shell +# Seed all projects +bin/rake 'gitlab:seed:vulnerabilities' + +# Seed a specific project +bin/rake 'gitlab:seed:vulnerabilities[group-path/project-path]' +``` + ### Automation If you're very sure that you want to **wipe the current database** and refill @@ -208,7 +220,7 @@ bundle exec rake rubocop:todo:generate\[Gitlab/NamespacedClass,Lint/Syntax\] Some shells require brackets to be escaped or quoted. -See [Resolving RuboCop exceptions](contributing/style_guides.md#resolving-rubocop-exceptions) +See [Resolving RuboCop exceptions](../development/rubocop_development_guide.md#resolving-rubocop-exceptions) on how to proceed from here. ### Run RuboCop in graceful mode diff --git a/doc/development/reusing_abstractions.md b/doc/development/reusing_abstractions.md index 2701192137c..58d1e20394c 100644 --- a/doc/development/reusing_abstractions.md +++ b/doc/development/reusing_abstractions.md @@ -151,7 +151,8 @@ When implementing a service class, consider: developer's discretion, such as: `issue`, `project`, `merge_request`. 1. When service represents an action initiated by a user or executed in the context of a user, the initializer must have the `current_user:` keyword argument. - Services with `current_user:` argument run high-level business logic. + Services with the `current_user:` argument run high-level business logic + and must validate user authorization to perform their operations. 1. When service does not have a user context and it's not directly initiated by a user (like background service or side-effects), the `current_user:` argument is not needed. This describes low-level domain logic or instance-wide logic. diff --git a/doc/development/routing.md b/doc/development/routing.md index 16ed15fdcc6..8f475674c39 100644 --- a/doc/development/routing.md +++ b/doc/development/routing.md @@ -39,7 +39,7 @@ You can view and find routes from the console by running: rails routes | grep crm ``` -You can also view routes in your browser by going to [http://gdk.test:3000/rails/info/routes](http://gdk.test:3000/rails/info/routes). +You can also view routes in your browser by going to `http://gdk.test:3000/rails/info/routes`. ## Global routes diff --git a/doc/development/rubocop_development_guide.md b/doc/development/rubocop_development_guide.md new file mode 100644 index 00000000000..2ff94f65232 --- /dev/null +++ b/doc/development/rubocop_development_guide.md @@ -0,0 +1,114 @@ +--- +type: reference, dev +stage: none +group: Development +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# RuboCop rule development guide + +Our codebase style is defined and enforced by [RuboCop](https://github.com/rubocop-hq/rubocop). + +You can check for any offenses locally with `bundle exec rubocop --parallel`. +On the CI, this is automatically checked by the `static-analysis` jobs. + +In addition, you can [integrate RuboCop](developing_with_solargraph.md) into +supported IDEs using the [Solargraph](https://github.com/castwide/solargraph) gem. + +For RuboCop rules that we have not taken a decision on, follow the [Ruby style guide](backend/ruby_style_guide.md) to write idiomatic Ruby. + +Reviewers/maintainers should be tolerant and not too pedantic about style. + +Some RuboCop rules are disabled, and for those, +reviewers/maintainers must not ask authors to use one style or the other, as both +are accepted. This isn't an ideal situation because this leaves space for +[bike-shedding](https://en.wiktionary.org/wiki/bikeshedding). Ideally we +should enable all RuboCop rules to avoid style-related +discussions, nitpicking, or back-and-forth in reviews. The +[GitLab Ruby style guide](backend/ruby_style_guide.md) includes a non-exhaustive +list of styles that commonly come up in reviews and are not enforced. + +Additionally, we have dedicated +[test-specific style guides and best practices](testing_guide/index.md). + +## Creating new RuboCop cops + +Typically it is better for the linting rules to be enforced programmatically as it +reduces the aforementioned [bike-shedding](https://en.wiktionary.org/wiki/bikeshedding). + +To that end, we encourage creation of new RuboCop rules in the codebase. + +Before adding a new cop to enforce a given style, make sure to discuss it with your team. + +We maintain cops across several Ruby code bases, and not all of them are +specific to the GitLab application. +When creating a new cop that could be applied to multiple applications, we encourage you +to add it to our [`gitlab-styles`](https://gitlab.com/gitlab-org/gitlab-styles) gem. +If the cop targets rules that only apply to the main GitLab application, +it should be added to [GitLab](https://gitlab.com/gitlab-org/gitlab) instead. + +## Cop grace period + +A cop is in a _grace period_ if it is enabled and has `Details: grace period` defined in its TODO YAML configuration. + +On the default branch, offenses from cops in the [grace period](rake_tasks.md#run-rubocop-in-graceful-mode) do not fail the RuboCop CI job. Instead, the job notifies the `#f_rubocop` Slack channel. However, on other branches, the RuboCop job fails. + +A grace period can safely be lifted as soon as there are no warnings for 2 weeks in the `#f_rubocop` channel on Slack. + +## Enabling a new cop + +1. Enable the new cop in `.rubocop.yml` (if not already done via [`gitlab-styles`](https://gitlab.com/gitlab-org/ruby/gems/gitlab-styles)). +1. [Generate TODOs for the new cop](rake_tasks.md#generate-initial-rubocop-todo-list). +1. [Set the new cop to `grace period`](#cop-grace-period). +1. Create an issue to fix TODOs and encourage community contributions (via ~"good for new contributors" and/or ~"Seeking community contributions"). [See some examples](https://gitlab.com/gitlab-org/gitlab/-/issues/?sort=created_date&state=opened&label_name%5B%5D=good%20for%20new%20contributors&label_name%5B%5D=static%20code%20analysis&first_page_size=20). +1. Create an issue to remove `grace period` after 2 weeks of silence in the `#f_rubocop` Slack channel. [See an example](https://gitlab.com/gitlab-org/gitlab/-/issues/374903). + +## Silenced offenses + +When offenses are silenced for cops in the [grace period](#cop-grace-period), +the `#f_rubocop` Slack channel receives a notification message every 2 hours. + +To fix this issue: + +1. Find cops with silenced offenses in the linked CI job. +1. [Generate TODOs](rake_tasks.md#generate-initial-rubocop-todo-list) for these cops. + +### RuboCop node pattern + +When creating [node patterns](https://docs.rubocop.org/rubocop-ast/node_pattern.html) to match +Ruby's AST, you can use [`scripts/rubocop-parse`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/scripts/rubocop-parse). +This displays the AST of a Ruby expression to help you create the matcher. +See also [!97024](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/97024). + +## Resolving RuboCop exceptions + +When the number of RuboCop exceptions exceeds the default [`exclude-limit` of 15](https://docs.rubocop.org/rubocop/1.2/usage/basic_usage.html#command-line-flags), +we may want to resolve exceptions over multiple commits. To minimize confusion, +we should track our progress through the exception list. + +The preferred way to [generate the initial list or a list for specific RuboCop rules](rake_tasks.md#generate-initial-rubocop-todo-list) +is to run the Rake task `rubocop:todo:generate`: + +```shell +# Initial list +bundle exec rake rubocop:todo:generate + +# List for specific RuboCop rules +bundle exec rake 'rubocop:todo:generate[Gitlab/NamespacedClass,Lint/Syntax]' +``` + +This Rake task creates or updates the exception list in `.rubocop_todo/`. For +example, the configuration for the RuboCop rule `Gitlab/NamespacedClass` is +located in `.rubocop_todo/gitlab/namespaced_class.yml`. + +Make sure to commit any changes in `.rubocop_todo/` after running the Rake task. + +## Reveal existing RuboCop exceptions + +To reveal existing RuboCop exceptions in the code that have been excluded via `.rubocop_todo.yml` and +`.rubocop_todo/**/*.yml`, set the environment variable `REVEAL_RUBOCOP_TODO` to `1`. + +This allows you to reveal existing RuboCop exceptions during your daily work cycle and fix them along the way. + +NOTE: +Define permanent `Exclude`s in `.rubocop.yml` instead of `.rubocop_todo/**/*.yml`. diff --git a/doc/development/sec/analyzer_development_guide.md b/doc/development/sec/analyzer_development_guide.md index a35bc2b7237..af3a6f2b7d7 100644 --- a/doc/development/sec/analyzer_development_guide.md +++ b/doc/development/sec/analyzer_development_guide.md @@ -78,11 +78,23 @@ go build -o analyzer ./analyzer convert test/fixtures/app/spotbugsXml.Xml > ./gl-sast-report.json ``` +### Execution criteria + +[Enabling SAST](../../user/application_security/sast/index.md#configure-sast-manually) requires including a pre-defined [template](https://gitlab.com/gitlab-org/gitlab/-/blob/ee4d473eb9a39f2f84b719aa0ca13d2b8e11dc7e/lib/gitlab/ci/templates/Jobs/SAST.gitlab-ci.yml) to your GitLab CI/CD configuration. + +The following independent criteria determine which analyzer needs to be run on a project: + +1. The SAST template uses [`rules:exists`](../../ci/yaml/index.md#rulesexists) to determine which analyzer will be run based on the presence of certain files. For example, the Brakeman analyzer [runs when there are](https://gitlab.com/gitlab-org/gitlab/-/blob/ee4d473eb9a39f2f84b719aa0ca13d2b8e11dc7e/lib/gitlab/ci/templates/Jobs/SAST.gitlab-ci.yml#L60) `.rb` files and a `Gemfile`. +1. Each analyzer runs a customizable [match interface](https://gitlab.com/gitlab-org/security-products/analyzers/common/-/blob/master/search/search.go) before it performs the actual analysis. For example: [Flawfinder checks for C/C++ files](https://gitlab.com/gitlab-org/security-products/analyzers/flawfinder/-/blob/f972ac786268fb649553056a94cda05cdc1248b2/plugin/plugin.go#L14). +1. For some analyzers that run on generic file extensions, there is a check based on a CI/CD variable. For example: Kubernetes manifests are written in YAML, so [Kubesec](https://gitlab.com/gitlab-org/security-products/analyzers/kubesec) runs only when [`SCAN_KUBERNETES_MANIFESTS` is set to true](../../user/application_security/sast/index.md#enabling-kubesec-analyzer). + +Step 1 helps prevent wastage of CI/CD minutes that would be spent running analyzers not suitable for the project. However, due to [technical limitations](https://gitlab.com/gitlab-org/gitlab/-/issues/227632), it cannot be used for large projects. Therefore, step 2 acts as final check to ensure a mismatched analyzer is able to exit early. + ## How to test the analyzers Video walkthrough of how Dependency Scanning analyzers are using [downstream pipeline](../../ci/pipelines/downstream_pipelines.md) feature to test analyzers using test projects: -[](http://www.youtube.com/watch?v=KauRBlfUbDE) +[](https://www.youtube.com/watch?v=KauRBlfUbDE) ### Testing local changes @@ -118,6 +130,12 @@ To use Docker with `replace` in the `go.mod` file: 1. Update the `replace` statement to make sure it matches the destination of the `COPY` statement in the step above: `replace gitlab.com/gitlab-org/security-products/analyzers/command/v3 => /command` +## Analyzer scripts + +The [analyzer-scripts](https://gitlab.com/gitlab-org/secure/tools/analyzer-scripts) repository contains scripts that can be used to interact with most analyzers. They enable you to build, run, and debug analyzers in a GitLab CI-like environment, and are particularly useful for locally validating changes to an analyzer. + +For more information, refer to the [project README](https://gitlab.com/gitlab-org/secure/tools/analyzer-scripts/-/blob/master/README.md). + ## Versioning and release process Analyzers are independent projects that follow their own versioning. `Patch` version bumps tend to correspond to a `Minor` version bump of the underlying tools (i.e. [`bandit`](https://wiki.openstack.org/wiki/Security/Projects/Bandit)), allowing us greater flexibility in reserving `Minor` bumps for more significant changes to our scanners. In case of breaking changes imposed by the wrapped scanner, creating a new analyzer on a separate repository must be considered. diff --git a/doc/development/sec/img/primary_identifier_changed_v15_6.png b/doc/development/sec/img/primary_identifier_changed_v15_6.png Binary files differnew file mode 100644 index 00000000000..aa1a116c801 --- /dev/null +++ b/doc/development/sec/img/primary_identifier_changed_v15_6.png diff --git a/doc/development/sec/index.md b/doc/development/sec/index.md index c9805044e58..fc13c960451 100644 --- a/doc/development/sec/index.md +++ b/doc/development/sec/index.md @@ -64,7 +64,97 @@ After the data is available as a Report Artifact it can be processed by the GitL Depending on the context, the security reports may be stored either in the database or stay as Report Artifacts for on-demand access. +#### Security report ingestion overview + +For details on how GitLab processes the reports generated by the scanners, see +[Security report ingestion overview](security_report_ingestion_overview.md). + ## CI/CD template development While CI/CD templates are the responsibility of the Verify section, many are critical to the Sec Section's feature usage. If you are working with CI/CD templates, please read the [development guide for GitLab CI/CD templates](../cicd/templates.md). + +## Importance of the primary identifier + +Within analyzer JSON reports, the [`identifiers` field](../integrations/secure.md#identifiers) contains a collection of types and categories by which +a vulnerability can be described (that is, a CWE family). + +The first item in the `identifiers` collection is known as the [primary identifier](../../user/application_security/terminology#primary-identifier), +a critical component to both describing and tracking vulnerabilities. + +In most other cases, the `identifiers` collection is unordered, where the remaining secondary identifiers act as metadata for grouping vulnerabilities +(see [Analyzer vulnerability translation](#analyzer-vulnerability-translation) below for the exception). + +Any time the primary identifier changes and a project pipeline is re-run, ingestion of the new report will “orphan” the previous DB record. +Because our processing logic relies on generating a delta of two different vulnerabilities, it can end up looking rather confusing. For example: + +[!Screenshot of primary identifier mismatch in MR widget](img/primary_identifier_changed_v15_6.png) + +After being [merged](../integrations/secure.md#tracking-and-merging-vulnerabilities), the previous vulnerability is listed as "remediated" and the introduced as ["detected"](../../user/application_security/vulnerabilities/index.md#vulnerability-status-values). + +### Guiding principles for ensuring primary identifier stability + +- A primary identifier should never change unless we have a compelling reason. +- Analyzer supporting vulnerability translation must include the legacy primary identifiers in a secondary position to prevent “orphaning” of results. +- Beyond the primary identifier, the order of secondary identifiers does not matter. +- The identifier is unique based on a combination of the `Type` and `Value` fields (see [identifier fingerprint](https://gitlab.com/gitlab-org/gitlab/-/blob/v15.5.1-ee/lib/gitlab/ci/reports/security/identifier.rb#L63)). +- If we change the primary identifier, rolling back analyzers to previous versions will not fix the orphaned results. The data previously ingested into our database is an artifact of previous jobs with few ways of automating data migrations. + +### Analyzer vulnerability translation + +In the case of SAST's semgrep analyzer, there is a secondary identifier of particular importance: the identifier linking the report’s vulnerability +to the legacy analyzer (that is, bandit or eslint). + +To [enable vulnerability translation](../../user/application_security/sast/analyzers.md#vulnerability-translation) +the semgrep analyzer relies on a secondary identifier exactly matching the primary identifier of the legacy analyzer. + +For example, when [`eslint`](https://gitlab.com/gitlab-org/security-products/analyzers/eslint) was previously used to generate vulnerability records, +the [`semgrep`](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) analyzer must produce an identifier collection containing the +original eslint primary identifier. + +Given the original `eslint` report: + +```json +{ + "version": "14.0.4", + "vulnerabilities": [ + { + "identifiers": [ + { + "type": "eslint_rule_id", + "name": "ESLint rule ID security/detect-eval-with-expression", + "value": "security/detect-eval-with-expression" + } + ] + } + ] +} +``` + +The corresponding semgrep report must contain the `eslint_rule_id`: + +```json +{ + "version": "14.0.4", + "vulnerabilities": [ + { + "identifiers": [ + { + "type": "semgrep_id", + "name": "eslint.detect-eval-with-expression", + "value": "eslint.detect-eval-with-expression", + "url": "https://semgrep.dev/r/gitlab.eslint.detect-eval-with-expression" + }, + { + "type": "eslint_rule_id", + "name": "ESLint rule ID security/detect-eval-with-expression", + "value": "security/detect-eval-with-expression" + } + ] + } + ] +} +``` + +[Tracking of vulnerabilities](../integrations/secure.md#tracking-and-merging-vulnerabilities) relies on a combination of the two identifiers +to remap DB records previously generated with the legacy analyzers to those generated with the new `semgrep` ones. diff --git a/doc/development/sec/security_report_ingestion_overview.md b/doc/development/sec/security_report_ingestion_overview.md new file mode 100644 index 00000000000..c1d977c2a17 --- /dev/null +++ b/doc/development/sec/security_report_ingestion_overview.md @@ -0,0 +1,74 @@ +--- +stage: Secure +group: Threat Insights +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: concepts +--- + +# Security report ingestion overview + +## Definitions + +- **Vulnerability Finding** – an instance of `Vulnerabilities::Finding` class. This class was previously called `Vulnerabilities::Occurrence`; after renaming the class, we kept the associated table name `vulnerability_occurrences` due to the effort involved in renaming large tables. +- **Vulnerability** – an instance of `Vulnerability` class. They are created based on information available in `Vulnerabilities::Finding` class. Every `Vulnerability` **must have** a corresponding `Vulnerabilities::Finding` object to be valid, however this is not enforced at the database level. +- **Security Finding** – an instance of `Security::Finding` class. They store **partial** finding data to improve performance of the pipeline security report. We are working on extending this class to store almost all required information so we can stop relying on job artifacts. +- **Feedback** – an instance of `Vulnerabilities::Feedback` class. They are created to keep track of users' interactions with Vulnerability Findings before they are promoted to a Vulnerability. We are in the process of removing this model via [Deprecate and remove Vulnerabilities::Feedback epic](https://gitlab.com/groups/gitlab-org/-/epics/5629). +- **Issue Link** – an instance of `Vulnerabilities::IssueLink` class. They are used to link `Vulnerability` objects to `Issue` objects. + +## Vulnerability creation from security reports + +Assumptions: + +- Project uses GitLab CI +- Project uses [security scanning tools](../../user/application_security) +- No Vulnerabilities are present in the database +- All pipelines perform security scans + +1. Code is pushed to a branch that's **not** the default branch. +1. GitLab CI runs a new pipeline for that branch. +1. Pipeline status transitions to any of [`::Ci::Pipeline.completed_statuses`](https://gitlab.com/gitlab-org/gitlab/-/blob/354261b2fe4fc5b86d1408467beadd90e466ce0a/app/models/concerns/ci/has_status.rb#L12). +1. `Security::StoreScansWorker` is called and it schedules `Security::StoreScansService`. +1. `Security::StoreScansService` calls `Security::StoreGroupedScansService`. +1. `Security::StoreGroupedScansService` calls `Security::StoreScanService`. +1. `Security::StoreScanService` calls `Security::StoreFindingsService`. +1. At this point we have `Security::Finding` objects **only**. + +At this point, the following things can happen to the `Security::Finding`: + +- Dismissal +- Issue creation +- Promotion to a Vulnerability + +If the pipeline ran on the default branch then the following, additional steps are done: + +1. `Security::StoreScansService` gets called and schedules `Security::StoreSecurityReportsWorker`. +1. `Security::StoreSecurityReportsWorker` executes `Security::Ingestion::IngestReportsService`. +1. `Security::Ingestion::IngestReportsService` takes all reports from a given Pipeline and calls `Security::Ingestion::IngestReportService` and then calls `Security::Ingestion::MarkAsResolvedService`. +1. `Security::Ingestion::IngestReportService` calls `Security::Ingestion::IngestReportSliceService` which executes a number of tasks for a report slice. + +### Dismissal + +If you select `Dismiss vulnerability`, a Feedback is created. You can also dismiss it with a comment. + +#### After Feedback removal + +If there is only a Security Finding, a Vulnerability Finding and a Vulnerability get created. At the same time we create a `Vulnerabilities::StateTransition` record to indicate the Vulnerability was dismissed. + +### Issue creation + +If you select `Create issue`, a Vulnerabilities::Feedback record is created as well. The Feedback has a different `feedback_type` and an `issue_id` that’s not `NULL`. + +NOTE: +Vulnerabilities::Feedback are in the process of being [deprecated](https://gitlab.com/groups/gitlab-org/-/epics/5629). This will later create a `Vulnerabilities::IssueLink` record. + +#### After Feedback removal + +If there's only a Security Finding, a Vulnerability Finding and a Vulnerability gets created. At the same time, we create an Issue and a Issue Link. + +## Promotion to a Vulnerability + +If the branch with a Security Finding gets merged into the default branch, all Security Findings get promoted into Vulnerabilities. Promotion is the process of creating Vulnerability Findings and Vulnerability records from those Security Findings. + +If there's a dismissal Feedback present for that Security Finding, the created Vulnerability is marked as dismissed. + +If there's an issue Feedback present for that Security Finding, we also create an Issue Link for that Vulnerability. diff --git a/doc/development/secure_coding_guidelines.md b/doc/development/secure_coding_guidelines.md index 67f7c556055..c102e99720f 100644 --- a/doc/development/secure_coding_guidelines.md +++ b/doc/development/secure_coding_guidelines.md @@ -856,7 +856,7 @@ Working with archive files like `zip`, `tar`, `jar`, `war`, `cpio`, `apk`, `rar` ### Zip Slip -In 2018, the security company Snyk [released a blog post](https://snyk.io/research/zip-slip-vulnerability) describing research into a widespread and critical vulnerability present in many libraries and applications which allows an attacker to overwrite arbitrary files on the server file system which, in many cases, can be leveraged to achieve remote code execution. The vulnerability was dubbed Zip Slip. +In 2018, the security company Snyk [released a blog post](https://security.snyk.io/research/zip-slip-vulnerability) describing research into a widespread and critical vulnerability present in many libraries and applications which allows an attacker to overwrite arbitrary files on the server file system which, in many cases, can be leveraged to achieve remote code execution. The vulnerability was dubbed Zip Slip. A Zip Slip vulnerability happens when an application extracts an archive without validating and sanitizing the filenames inside the archive for directory traversal sequences that change the file location when the file is extracted. @@ -1218,7 +1218,7 @@ GitLab-specific example can be found in [this issue](https://gitlab.com/gitlab-o **Example 3:** you need to fetch a remote file, and perform a `HEAD` request to get and validate the content length and content type. When you subsequently make a `GET` request, though, the file delivered is a different size or different file type. (This is stretching the definition of TOCTOU, but things _have_ changed between time of check and time of use). -**Example 4:** you allow users to upvote a comment if they haven't already. The server is multi-threaded, and you aren't using transactions or an applicable database index. By repeatedly clicking upvote in quick succession a malicious user is able to add multiple upvotes: the requests arrive at the same time, the checks run in parallel and confirm that no upvote exists yet, and so each upvote is written to the database. +**Example 4:** you allow users to upvote a comment if they haven't already. The server is multi-threaded, and you aren't using transactions or an applicable database index. By repeatedly selecting upvote in quick succession a malicious user is able to add multiple upvotes: the requests arrive at the same time, the checks run in parallel and confirm that no upvote exists yet, and so each upvote is written to the database. Here's some pseudocode showing an example of a potential TOCTOU bug: @@ -1271,7 +1271,7 @@ This sensitive data must be handled carefully to avoid leaks which could lead to - The [Gitleaks Git hook](https://gitlab.com/gitlab-com/gl-security/security-research/gitleaks-endpoint-installer) is recommended for preventing credentials from being committed. - Never log credentials under any circumstance. Issue [#353857](https://gitlab.com/gitlab-org/gitlab/-/issues/353857) is an example of credential leaks through log file. - When credentials are required in a CI/CD job, use [masked variables](../ci/variables/index.md#mask-a-cicd-variable) to help prevent accidental exposure in the job logs. Be aware that when [debug logging](../ci/variables/index.md#debug-logging) is enabled, all masked CI/CD variables are visible in job logs. Also consider using [protected variables](../ci/variables/index.md#protected-cicd-variables) when possible so that sensitive CI/CD variables are only available to pipelines running on protected branches or protected tags. -- Proper scanners must be enabled depending on what data those credentials are protecting. See the [Application Security Inventory Policy](https://about.gitlab.com/handbook/engineering/security/security-engineering-and-research/application-security/inventory.html#policies) and our [Data Classification Standards](https://about.gitlab.com/handbook/engineering/security/data-classification-standard.html#data-classification-standards). +- Proper scanners must be enabled depending on what data those credentials are protecting. See the [Application Security Inventory Policy](https://about.gitlab.com/handbook/security/security-engineering-and-research/application-security/inventory.html#policies) and our [Data Classification Standards](https://about.gitlab.com/handbook/security/data-classification-standard.html#data-classification-standards). - To store and/or share credentials between teams, refer to [1Password for Teams](https://about.gitlab.com/handbook/security/#1password-for-teams) and follow [the 1Password Guidelines](https://about.gitlab.com/handbook/security/#1password-guidelines). - If you need to share a secret with a team member, use 1Password. Do not share a secret over email, Slack, or other service on the Internet. @@ -1281,7 +1281,7 @@ This sensitive data must be handled carefully to avoid leaks which could lead to - Avoid including credentials as part of an HTTP response unless it is absolutely necessary as part of the workflow. For example, generating a PAT for users. - Avoid sending credentials in URL parameters, as these can be more easily logged inadvertently during transit. -In the event of credential leak through an MR, issue, or any other medium, [reach out to SIRT team](https://about.gitlab.com/handbook/engineering/security/security-operations/sirt/#-engaging-sirt). +In the event of credential leak through an MR, issue, or any other medium, [reach out to SIRT team](https://about.gitlab.com/handbook/security/security-operations/sirt/#-engaging-sirt). ## Serialization diff --git a/doc/development/service_ping/implement.md b/doc/development/service_ping/implement.md index 88a1454393f..5a564b2d83e 100644 --- a/doc/development/service_ping/implement.md +++ b/doc/development/service_ping/implement.md @@ -231,13 +231,6 @@ Examples: estimate_batch_distinct_count(::Note.with_suggestions.where(time_period), :author_id, start: ::Note.minimum(:id), finish: ::Note.maximum(:id)) ``` -1. Execution of estimated batch counter with joined relation (`joins(:cluster)`), - for a custom column (`'clusters.user_id'`): - - ```ruby - estimate_batch_distinct_count(::Clusters::Applications::CertManager.where(time_period).available.joins(:cluster), 'clusters.user_id') - ``` - When instrumenting metric with usage of estimated batch counter please add `_estimated` suffix to its name, for example: @@ -266,7 +259,7 @@ Arguments: #### Ordinary Redis counters -Example of implementation: [`Gitlab::UsageDataCounters::WikiPageCounter`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/wiki_page_counter.rb), using Redis methods [`INCR`](https://redis.io/commands/incr) and [`GET`](https://redis.io/commands/get). +Example of implementation: [`Gitlab::UsageDataCounters::WikiPageCounter`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/wiki_page_counter.rb), using Redis methods [`INCR`](https://redis.io/commands/incr/) and [`GET`](https://redis.io/commands/get/). Events are handled by counter classes in the `Gitlab::UsageDataCounters` namespace, inheriting from `BaseCounter`, that are either: @@ -813,7 +806,7 @@ and run a local container instance: 1. On your local machine, make sure you are signed in to the GitLab Docker registry. You can find the instructions for this in [Authenticate to the GitLab Container Registry](../../user/packages/container_registry/index.md#authenticate-with-the-container-registry). 1. Once signed in, download the new image by using `docker pull registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:<VERSION>` -1. For more information about working with and running Omnibus GitLab containers in Docker, refer to [GitLab Docker images](https://docs.gitlab.com/omnibus/docker/README.html) in the Omnibus documentation. +1. For more information about working with and running Omnibus GitLab containers in Docker, refer to [GitLab Docker images](../../install/docker.md) documentation. ### Test with GitLab development toolkits @@ -849,105 +842,6 @@ You can then count each user that performed any combination of these actions. To add data for aggregated metrics to the Service Ping payload, create metric YAML definition file following [Aggregated metric instrumentation guide](metrics_instrumentation.md#aggregated-metrics). -### (DEPRECATED) Defining aggregated metric via aggregated metric YAML config file - -WARNING: -This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/98206) in GitLab 15.5 -and is planned for removal in 15.5. Use [metrics definition YAMLs](https://gitlab.com/gitlab-org/gitlab/-/issues/370963) instead. - -To add data for aggregated metrics to the Service Ping payload, add a corresponding definition to: - -- [`config/metrics/aggregates/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/aggregates/) for metrics available in the Community Edition. -- [`ee/config/metrics/aggregates/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/aggregates/) for metrics available in the Enterprise Edition. - -Each aggregate definition includes following parts: - -- `name`: Unique name under which the aggregate metric is added to the Service Ping payload. -- `operator`: Operator that defines how the aggregated metric data is counted. Available operators are: - - `OR`: Removes duplicates and counts all entries that triggered any of listed events. - - `AND`: Removes duplicates and counts all elements that were observed triggering all of following events. -- `time_frame`: One or more valid time frames. Use these to limit the data included in aggregated metric to events within a specific date-range. Valid time frames are: - - `7d`: Last seven days of data. - - `28d`: Last twenty eight days of data. - - `all`: All historical data, only available for `database` sourced aggregated metrics. -- `source`: Data source used to collect all events data included in aggregated metric. Valid data sources are: - - [`database`](#database-sourced-aggregated-metrics) - - [`redis`](#redis-sourced-aggregated-metrics) -- `events`: list of events names to aggregate into metric. All events in this list must - relay on the same data source. Additional data source requirements are described in the - [Database sourced aggregated metrics](#database-sourced-aggregated-metrics) and - [Redis sourced aggregated metrics](#redis-sourced-aggregated-metrics) sections. -- `feature_flag`: Name of [development feature flag](../feature_flags/index.md#development-type) - that is checked before metrics aggregation is performed. Corresponding feature flag - should have `default_enabled` attribute set to `false`. The `feature_flag` attribute - is optional and can be omitted. When `feature_flag` is missing, no feature flag is checked. - -Example aggregated metric entries: - -```yaml -- name: example_metrics_union - operator: OR - events: - - 'users_expanding_secure_security_report' - - 'users_expanding_testing_code_quality_report' - - 'users_expanding_testing_accessibility_report' - source: redis - time_frame: - - 7d - - 28d -- name: example_metrics_intersection - operator: AND - source: database - time_frame: - - 28d - - all - events: - - 'dependency_scanning_pipeline_all_time' - - 'container_scanning_pipeline_all_time' - feature_flag: example_aggregated_metric -``` - -Aggregated metrics collected in `7d` and `28d` time frames are added into Service Ping payload under the `aggregated_metrics` sub-key in the `counts_weekly` and `counts_monthly` top level keys. - -```ruby -{ - :counts_monthly => { - :deployments => 1003, - :successful_deployments => 78, - :failed_deployments => 275, - :packages => 155, - :personal_snippets => 2106, - :project_snippets => 407, - :aggregated_metrics => { - :example_metrics_union => 7, - :example_metrics_intersection => 2 - }, - :snippets => 2513 - } -} -``` - -Aggregated metrics for `all` time frame are present in the `count` top level key, with the `aggregate_` prefix added to their name. - -For example: - -`example_metrics_intersection` - -Becomes: - -`counts.aggregate_example_metrics_intersection` - -```ruby -{ - :counts => { - :deployments => 11003, - :successful_deployments => 178, - :failed_deployments => 1275, - :aggregate_example_metrics_intersection => 12 - } -} -``` - ### Redis sourced aggregated metrics > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/45979) in GitLab 13.6. @@ -963,9 +857,7 @@ you must fulfill the following requirements: ### Database sourced aggregated metrics -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/52784) in GitLab 13.9. -> - It's [deployed behind a feature flag](../../user/feature_flags.md), disabled by default. -> - It's enabled on GitLab.com. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/52784) in GitLab 13.9. To declare an aggregate of metrics based on events collected from database, follow these steps: @@ -1018,25 +910,9 @@ end #### Add new aggregated metric definition -After all metrics are persisted, you can add an aggregated metric definition at -[`aggregated_metrics/`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/aggregates/). - +After all metrics are persisted, you can add an aggregated metric definition following [Aggregated metric instrumentation guide](metrics_instrumentation.md#aggregated-metrics). To declare the aggregate of metrics collected with [Estimated Batch Counters](#estimated-batch-counters), you must fulfill the following requirements: - Metrics names listed in the `events:` attribute, have to use the same names you passed in the `metric_name` argument while persisting metrics in previous step. - Every metric listed in the `events:` attribute, has to be persisted for **every** selected `time_frame:` value. - -Example definition: - -```yaml -- name: example_metrics_intersection_database_sourced - operator: AND - source: database - events: - - 'dependency_scanning_pipeline' - - 'container_scanning_pipeline' - time_frame: - - 28d - - all -``` diff --git a/doc/development/service_ping/index.md b/doc/development/service_ping/index.md index 0a4e5998345..37e0b753448 100644 --- a/doc/development/service_ping/index.md +++ b/doc/development/service_ping/index.md @@ -115,7 +115,7 @@ sequenceDiagram > - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/295289) in GitLab 15.2. [Feature flag `measure_service_ping_metric_collection`](https://gitlab.com/gitlab-org/gitlab/-/issues/358128) removed. ```ruby - { + { "metadata"=> { "uuid"=>"0000000-0000-0000-0000-000000000000", @@ -504,7 +504,7 @@ Service Ping reporting process state is monitored with [internal SiSense dashboa - [Product Intelligence Guide](https://about.gitlab.com/handbook/product/product-intelligence-guide/) - [Snowplow Guide](../snowplow/index.md) -- [Product Intelligence Direction](https://about.gitlab.com/direction/product-intelligence/) +- [Product Intelligence Direction](https://about.gitlab.com/direction/analytics/product-intelligence/) - [Data Analysis Process](https://about.gitlab.com/handbook/business-technology/data-team/#data-analysis-process/) - [Data for Product Managers](https://about.gitlab.com/handbook/business-technology/data-team/programs/data-for-product-managers/) - [Data Infrastructure](https://about.gitlab.com/handbook/business-technology/data-team/platform/infrastructure/) diff --git a/doc/development/service_ping/metrics_instrumentation.md b/doc/development/service_ping/metrics_instrumentation.md index 7a3f291460c..a9f236819fe 100644 --- a/doc/development/service_ping/metrics_instrumentation.md +++ b/doc/development/service_ping/metrics_instrumentation.md @@ -41,7 +41,7 @@ We have built a domain-specific language (DSL) to define the metrics instrumenta You can use database metrics to track data kept in the database, for example, a count of issues that exist on a given instance. - `operation`: Operations for the given `relation`, one of `count`, `distinct_count`, `sum`, and `average`. -- `relation`: `ActiveRecord::Relation` for the objects we want to perform the `operation`. +- `relation`: Assigns lambda that returns the `ActiveRecord::Relation` for the objects we want to perform the `operation`. The assigned lambda can accept up to one parameter. The parameter is hashed and stored under the `options` key in the metric definition. - `start`: Specifies the start value of the batch counting, by default is `relation.minimum(:id)`. - `finish`: Specifies the end value of the batch counting, by default is `relation.maximum(:id)`. - `cache_start_and_finish_as`: Specifies the cache key for `start` and `finish` values and sets up caching them. Use this call when `start` and `finish` are expensive queries that should be reused between different metric calculations. @@ -55,10 +55,10 @@ module Gitlab module Usage module Metrics module Instrumentations - class CountBoardsMetric < DatabaseMetric + class CountIssuesMetric < DatabaseMetric operation :count - relation { Board } + relation ->(options) { Issue.where(confidential: options[:confidential]) } end end end @@ -156,7 +156,7 @@ You can use Redis metrics to track events not kept in the database, for example, [Example of a merge request that adds a `Redis` metric](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/97009). -Please note that `RedisMetric` class can only be used as the `instrumentation_class` for Redis metrics with simple counters classes (classes that only inherit `BaseCounter` and set `PREFIX` and `KNOWN_EVENTS` constants). In case the counter class has additional logic included in it, a new `instrumentation_class`, inheriting from `RedisMetric`, needs to be created. This new class needs to include the additional logic from the counter class. +The `RedisMetric` class can only be used as the `instrumentation_class` for Redis metrics with simple counters classes (classes that only inherit `BaseCounter` and set `PREFIX` and `KNOWN_EVENTS` constants). In case the counter class has additional logic included in it, a new `instrumentation_class`, inheriting from `RedisMetric`, needs to be created. This new class needs to include the additional logic from the counter class. Count unique values for `source_code_pushes` event. @@ -256,6 +256,13 @@ options: ## Aggregated metrics +<div class="video-fallback"> + See the video from: <a href="https://www.youtube.com/embed/22LbYqHwtUQ">Product Intelligence Office Hours Oct 6th</a> for an aggregated metrics walk-through. +</div> +<figure class="video-container"> + <iframe src="https://www.youtube.com/embed/22LbYqHwtUQ" frameborder="0" allowfullscreen="true"> </iframe> +</figure> + The aggregated metrics feature provides insight into the number of data attributes, for example `pseudonymized_user_ids`, that occurred in a collection of events. For example, you can aggregate the number of users who perform multiple actions such as creating a new issue and opening a new merge request. diff --git a/doc/development/service_ping/metrics_lifecycle.md b/doc/development/service_ping/metrics_lifecycle.md index f13aebeb16e..92c5d2d317d 100644 --- a/doc/development/service_ping/metrics_lifecycle.md +++ b/doc/development/service_ping/metrics_lifecycle.md @@ -25,6 +25,10 @@ Any such changes lead to inconsistent reports from multiple GitLab instances. If there is a problem with an existing metric, it's best to deprecate the existing metric, and use it, side by side, with the desired new metric. +If you do need to change a metric, please notify the Customer Success Ops team (`@csops-team`), Analytics Engineers (`@gitlab-data/analytics-engineers`), and Product Analysts (`@gitlab-data/product-analysts`) teams by `@` mentioning those groups in a comment on the MR. +Many Service Ping metrics are relied upon for health score and XMAU reporting and +unexpected changes to those metrics could break reporting. + Example: Consider following change. Before GitLab 12.6, the `example_metric` was implemented as: @@ -135,3 +139,6 @@ To remove a metric: 1. Remove any other records related to the metric: - The feature flag YAML file at [`config/feature_flags/*/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/config/feature_flags). - The entry in the known events YAML file at [`lib/gitlab/usage_data_counters/known_events/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/usage_data_counters/known_events). + +1. Notify the Customer Success Ops team (`@csops-team`), Analytics Engineers (`@gitlab-data/analytics-engineers`), and Product Analysts (`@gitlab-data/product-analysts`) by `@` mentioning those groups in a comment on the MR. + Many Service Ping metrics are relied upon for health score and XMAU reporting and unexpected changes to those metrics could break reporting. diff --git a/doc/development/service_ping/review_guidelines.md b/doc/development/service_ping/review_guidelines.md index a1806551303..70f7f3dca54 100644 --- a/doc/development/service_ping/review_guidelines.md +++ b/doc/development/service_ping/review_guidelines.md @@ -68,6 +68,7 @@ are regular backend changes. Read the [stages file](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/stages.yml). - Check the file location. Consider the time frame, and if the file should be under `ee`. - Check the tiers. +- If a metric was changed or removed: Make sure the MR author notified the Customer Success Ops team (`@csops-team`), Analytics Engineers (`@gitlab-data/analytics-engineers`), and Product Analysts (`@gitlab-data/product-analysts`) by `@` mentioning those groups in a comment on the MR. - Metrics instrumentations - Recommend using metrics instrumentation for new metrics, [if possible](metrics_instrumentation.md#support-for-instrumentation-classes). - Approve the MR, and relabel the MR with `~"product intelligence::approved"`. diff --git a/doc/development/service_ping/troubleshooting.md b/doc/development/service_ping/troubleshooting.md index 201d6a385eb..f8fd45e6062 100644 --- a/doc/development/service_ping/troubleshooting.md +++ b/doc/development/service_ping/troubleshooting.md @@ -120,3 +120,45 @@ To work around this bug, you have two options: 1. Expand **Usage Statistics**. 1. Clear the **Enable Service Ping** checkbox. 1. Select **Save Changes**. + +## Generate Service Ping + +### Generate or get the cached Service Ping in rails console + +Use the following method in the [rails console](../../administration/operations/rails_console.md#starting-a-rails-console-session). + +```ruby +Gitlab::Usage::ServicePingReport.for(output: :all_metrics_values, cached: true) +``` + +### Generate a fresh new Service Ping + +Use the following method in the [rails console](../../administration/operations/rails_console.md#starting-a-rails-console-session). + +This also refreshes the cached Service Ping displayed in the Admin Area. + +```ruby +Gitlab::Usage::ServicePingReport.for(output: :all_metrics_values) +``` + +### Generate and print + +Generates Service Ping data in JSON format. + +```shell +gitlab-rake gitlab:usage_data:generate +``` + +Generates Service Ping data in YAML format: + +```shell +gitlab-rake gitlab:usage_data:dump_sql_in_yaml +``` + +### Generate and send Service Ping + +Prints the metrics saved in `conversational_development_index_metrics`. + +```shell +gitlab-rake gitlab:usage_data:generate_and_send +``` diff --git a/doc/development/service_ping/usage_data.md b/doc/development/service_ping/usage_data.md index 2f2df14f56d..1c7a212ed64 100644 --- a/doc/development/service_ping/usage_data.md +++ b/doc/development/service_ping/usage_data.md @@ -66,5 +66,4 @@ Examples: ```ruby distinct_count(::Project, :creator_id) distinct_count(::Note.with_suggestions.where(time_period), :author_id, start: ::User.minimum(:id), finish: ::User.maximum(:id)) -distinct_count(::Clusters::Applications::CertManager.where(time_period).available.joins(:cluster), 'clusters.user_id') ``` diff --git a/doc/development/shell_commands.md b/doc/development/shell_commands.md index 3935e98199a..d78a005d76b 100644 --- a/doc/development/shell_commands.md +++ b/doc/development/shell_commands.md @@ -71,6 +71,8 @@ FileUtils.touch myfile This coding style could have prevented CVE-2013-4546. +See also <https://gitlab.com/gitlab-org/gitlab/-/merge_requests/93030>, and <https://starlabs.sg/blog/2022/07-gitlab-project-import-rce-analysis-cve-2022-2185/> for another example. + ## Separate options from arguments with -- Make the difference between options and arguments clear to the argument parsers of system commands with `--`. This is supported by many but not all Unix commands. diff --git a/doc/development/sidekiq/index.md b/doc/development/sidekiq/index.md index e8c939571cf..a95e94cdd34 100644 --- a/doc/development/sidekiq/index.md +++ b/doc/development/sidekiq/index.md @@ -36,6 +36,31 @@ with back-off between each retry. 25 retries means that the last retry would happen around three weeks after the first attempt (assuming all 24 prior retries failed). +This means that a lot can happen in between the job being scheduled +and its execution. Therefore, we must guard workers so they don't +fail 25 times when the state changes after they are scheduled. For +example, a job should not fail when the project it was scheduled for +is deleted. + +Instead of: + +```ruby +def perform(project_id) + project = Project.find(project_id) + # ... +end +``` + +Do this: + +```ruby +def perform(project_id) + project = Project.find_by_id(project_id) + return unless project + # ... +end +``` + For most workers - especially [idempotent workers](idempotent_jobs.md) - the default of 25 retries is more than sufficient. Many of our older workers declare 3 retries, which used to be the default within the diff --git a/doc/development/sidekiq/worker_attributes.md b/doc/development/sidekiq/worker_attributes.md index 48a222d65a0..4fcd8e33d5c 100644 --- a/doc/development/sidekiq/worker_attributes.md +++ b/doc/development/sidekiq/worker_attributes.md @@ -277,7 +277,7 @@ they prefer read replicas and will wait for replicas to catch up: | **Data Consistency** | **Description** | |--------------|-----------------------------| -| `:always` | The job is required to use the primary database (default). It should be used for workers that primarily perform writes or that have strict requirements around data consistency when reading their own writes. | +| `:always` | The job is required to use the primary database (default). It should be used for workers that primarily perform writes, have strict requirements around data consistency when reading their own writes, or are cron jobs. | | `:sticky` | The job prefers replicas, but switches to the primary for writes or when encountering replication lag. It should be used for jobs that require to be executed as fast as possible but can sustain a small initial queuing delay. | | `:delayed` | The job prefers replicas, but switches to the primary for writes. When encountering replication lag before the job starts, the job is retried once. If the replica is still not up to date on the next retry, it switches to the primary. It should be used for jobs where delaying execution further typically does not matter, such as cache expiration or web hooks execution. | diff --git a/doc/development/snowplow/event_dictionary_guide.md b/doc/development/snowplow/event_dictionary_guide.md index c5a21f5a081..794a9a0160c 100644 --- a/doc/development/snowplow/event_dictionary_guide.md +++ b/doc/development/snowplow/event_dictionary_guide.md @@ -22,24 +22,24 @@ All event definitions are stored in the following directories: Each event is defined in a separate YAML file consisting of the following fields: -| Field | Required | Additional information | -|------------------------|----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `description` | yes | A description of the event. | -| `category` | yes | The event category (see [Structured event taxonomy](index.md#structured-event-taxonomy)). | -| `action` | yes | The event action (see [Structured event taxonomy](index.md#structured-event-taxonomy)). | -| `label_description` | no | A description of the event label (see [Structured event taxonomy](index.md#structured-event-taxonomy)). | -| `property_description` | no | A description of the event property (see [Structured event taxonomy](index.md#structured-event-taxonomy)). | -| `value_description` | no | A description of the event value (see [Structured event taxonomy](index.md#structured-event-taxonomy)). | -| `extra_properties` | no | The type and description of each extra property sent with the event. | -| `identifiers` | no | A list of identifiers sent with the event. Can be set to one or more of `project`, `user`, or `namespace`. | -| `iglu_schema_url` | no | The URL to the custom schema sent with the event, for example, `iglu:com.gitlab/gitlab_experiment/jsonschema/1-0-0`. | -| `product_section` | yes | The [section](https://gitlab.com/gitlab-com/www-gitlab-com/-/blob/master/data/sections.yml). | -| `product_stage` | no | The [stage](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/stages.yml) for the event. | -| `product_group` | yes | The [group](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/stages.yml) that owns the event. | -| `product_category` | no | The [product category](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/categories.yml) for the event. | -| `milestone` | no | The milestone when the event is introduced. | -| `introduced_by_url` | no | The URL to the merge request that introduced the event. | -| `distributions` | yes | The [distributions](https://about.gitlab.com/handbook/marketing/strategic-marketing/tiers/#definitions) where the tracked feature is available. Can be set to one or more of `ce` or `ee`. | +| Field | Required | Additional information | +|------------------------|----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `description` | yes | A description of the event. | +| `category` | yes | The event category (see [Event schema](index.md#event-schema)). | +| `action` | yes | The event action (see [Event schema](index.md#event-schema)). | +| `label_description` | no | A description of the event label (see [Event schema](index.md#event-schema)). | +| `property_description` | no | A description of the event property (see [Event schema](index.md#event-schema)). | +| `value_description` | no | A description of the event value (see [Event schema](index.md#event-schema)). | +| `extra_properties` | no | The type and description of each extra property sent with the event. | +| `identifiers` | no | A list of identifiers sent with the event. Can be set to one or more of `project`, `user`, or `namespace`. | +| `iglu_schema_url` | no | The URL to the custom schema sent with the event, for example, `iglu:com.gitlab/gitlab_experiment/jsonschema/1-0-0`. | +| `product_section` | yes | The [section](https://gitlab.com/gitlab-com/www-gitlab-com/-/blob/master/data/sections.yml). | +| `product_stage` | no | The [stage](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/stages.yml) for the event. | +| `product_group` | yes | The [group](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/stages.yml) that owns the event. | +| `product_category` | no | The [product category](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/categories.yml) for the event. | +| `milestone` | no | The milestone when the event is introduced. | +| `introduced_by_url` | no | The URL to the merge request that introduced the event. | +| `distributions` | yes | The [distributions](https://about.gitlab.com/handbook/marketing/strategic-marketing/tiers/#definitions) where the tracked feature is available. Can be set to one or more of `ce` or `ee`. | | `tiers` | yes | The [tiers]( https://about.gitlab.com/handbook/marketing/strategic-marketing/tiers/) where the tracked feature is available. Can be set to one or more of `free`, `premium`, or `ultimate`. | ### Example event definition diff --git a/doc/development/snowplow/implementation.md b/doc/development/snowplow/implementation.md index 26016eeba84..a80d6fe70ff 100644 --- a/doc/development/snowplow/implementation.md +++ b/doc/development/snowplow/implementation.md @@ -13,7 +13,7 @@ This page describes how to: ## Snowplow JavaScript frontend tracking -GitLab provides a `Tracking` interface that wraps the [Snowplow JavaScript tracker](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/javascript-trackers/) +GitLab provides a `Tracking` interface that wraps the [Snowplow JavaScript tracker](https://docs.snowplow.io/docs/collecting-data/collecting-from-own-applications/javascript-trackers/) to track custom events. For the recommended frontend tracking implementation, see [Usage recommendations](#usage-recommendations). @@ -42,14 +42,14 @@ and the custom `extra` object. You can modify this object for any subsequent structured event that fires, although this is not recommended. Tracking implementations must have an `action` and a `category`. You can provide additional -properties from the [structured event taxonomy](index.md#structured-event-taxonomy), in +properties from the [event schema](index.md#event-schema), in addition to an `extra` object that accepts key-value pairs. -| Property | Type | Default value | Description | -|:-----------|:-------|:---------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `category` | string | `document.body.dataset.page` | Page or subsection of a page in which events are captured. | +| Property | Type | Default value | Description | +|:-----------|:-------|:---------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `category` | string | `document.body.dataset.page` | Page or subsection of a page in which events are captured. | | `action` | string | `'generic'` | Action the user is taking. Clicks must be `click` and activations must be `activate`. For example, focusing a form field is `activate_form_input`, and clicking a button is `click_button`. | -| `data` | object | `{}` | Additional data such as `label`, `property`, `value` as described in [Structured event taxonomy](index.md#structured-event-taxonomy), `context` for custom contexts, and `extra` (key-value pairs object). | +| `data` | object | `{}` | Additional data such as `label`, `property`, `value` as described in [Event schema](index.md#event-schema), `context` for custom contexts, and `extra` (key-value pairs object). | ### Usage recommendations @@ -81,7 +81,7 @@ The following example shows `data-track-*` attributes assigned to a button: | Attribute | Required | Description | |:----------------------|:---------|:------------| | `data-track-action` | true | Action the user is taking. Clicks must be prepended with `click` and activations must be prepended with `activate`. For example, focusing a form field is `activate_form_input` and clicking a button is `click_button`. Replaces `data-track-event`, which was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/290962) in GitLab 13.11. | -| `data-track-label` | false | The specific element or object to act on. This can be: the label of the element, for example, a tab labeled 'Create from template' for `create_from_template`; a unique identifier if no text is available, for example, `groups_dropdown_close` for closing the Groups dropdown in the top bar; or the name or title attribute of a record being created. | +| `data-track-label` | false | The specific element or object to act on. This can be: the label of the element, for example, a tab labeled 'Create from template' for `create_from_template`; a unique identifier if no text is available, for example, `groups_dropdown_close` for closing the Groups dropdown list in the top bar; or the name or title attribute of a record being created. | | `data-track-property` | false | Any additional property of the element, or object being acted on. | | `data-track-value` | false | Describes a numeric value (decimal) directly related to the event. This could be the value of an input. For example, `10` when clicking `internal` visibility. If omitted, this is the element's `value` property or `undefined`. For checkboxes, the default value is the element's checked attribute or `0` when unchecked. The value is parsed as numeric before sending the event. | | `data-track-extra` | false | A key-value pair object passed as a valid JSON string. This attribute is added to the `extra` property in our [`gitlab_standard`](schemas.md#gitlab_standard) schema. | @@ -349,7 +349,7 @@ describe('MyTracking', () => { ### Form tracking -To enable Snowplow automatic [form tracking](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/javascript-trackers/javascript-tracker/javascript-tracker-v2/tracking-specific-events/#form-tracking): +To enable Snowplow automatic [form tracking](https://docs.snowplow.io/docs/collecting-data/collecting-from-own-applications/javascript-trackers/javascript-tracker/javascript-tracker-v2/tracking-specific-events/#form-tracking): 1. Call `Tracking.enableFormTracking` when the DOM is ready. 1. Provide a `config` object that includes at least one of the following elements: @@ -389,7 +389,7 @@ describe('MyFormTracking', () => { ## Implement Ruby backend tracking -`Gitlab::Tracking` is an interface that wraps the [Snowplow Ruby Tracker](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/ruby-tracker/) for tracking custom events. +`Gitlab::Tracking` is an interface that wraps the [Snowplow Ruby Tracker](https://docs.snowplow.io/docs/collecting-data/collecting-from-own-applications/ruby-tracker/) for tracking custom events. Backend tracking provides: - User behavior tracking @@ -415,7 +415,7 @@ Use the following arguments: |------------|---------------------------|---------------|-----------------------------------------------------------------------------------------------------------------------------------| | `category` | String | | Area or aspect of the application. For example, `HealthCheckController` or `Lfs::FileTransformer`. | | `action` | String | | The action being taken. For example, a controller action such as `create`, or an Active Record callback. | -| `label` | String | `nil` | The specific element or object to act on. This can be one of the following: the label of the element, for example, a tab labeled 'Create from template' for `create_from_template`; a unique identifier if no text is available, for example, `groups_dropdown_close` for closing the Groups dropdown in the top bar; or the name or title attribute of a record being created. | +| `label` | String | `nil` | The specific element or object to act on. This can be one of the following: the label of the element, for example, a tab labeled 'Create from template' for `create_from_template`; a unique identifier if no text is available, for example, `groups_dropdown_close` for closing the Groups dropdown list in the top bar; or the name or title attribute of a record being created. | | `property` | String | `nil` | Any additional property of the element, or object being acted on. | | `value` | Numeric | `nil` | Describes a numeric value (decimal) directly related to the event. This could be the value of an input. For example, `10` when clicking `internal` visibility. | | `context` | Array\[SelfDescribingJSON\] | `nil` | An array of custom contexts to send with this event. Most events should not have any custom contexts. | @@ -451,7 +451,7 @@ Before you test frontend events in development, you must: 1. Turn off ad blockers that could prevent Snowplow JavaScript from loading in your environment. 1. Turn off "Do Not Track" (DNT) in your browser. -All URLs are pseudonymized. The entity identifier [replaces](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/javascript-trackers/javascript-tracker/javascript-tracker-v2/tracker-setup/other-parameters-2/#Setting_a_custom_page_URL_and_referrer_URL) personally identifiable +All URLs are pseudonymized. The entity identifier [replaces](https://docs.snowplow.io/docs/collecting-data/collecting-from-own-applications/javascript-trackers/javascript-tracker/javascript-tracker-v2/tracker-setup/other-parameters-2/#setting-a-custom-page-url-and-referrer-url) personally identifiable information (PII). PII includes usernames, group, and project names. Page titles are hardcoded as `GitLab` for the same reason. @@ -477,7 +477,7 @@ For a video tutorial, see the [Snowplow plugin walk through](https://www.youtube #### Snowplow Micro -[Snowplow Micro](https://snowplowanalytics.com/blog/2019/07/17/introducing-snowplow-micro/) is a +[Snowplow Micro](https://snowplow.io/blog/introducing-snowplow-micro/) is a Docker-based solution for testing backend and frontend in a local development environment. Snowplow Micro records the same events as the full Snowplow pipeline. To query events, use the Snowplow Micro API. diff --git a/doc/development/snowplow/index.md b/doc/development/snowplow/index.md index 7327f18fcaf..8d05e05e592 100644 --- a/doc/development/snowplow/index.md +++ b/doc/development/snowplow/index.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w Snowplow is an enterprise-grade marketing and Product Intelligence platform that tracks how users engage with our website and application. -[Snowplow](https://snowplowanalytics.com) consists of several loosely-coupled sub-systems: +[Snowplow](https://snowplow.io/) consists of several loosely-coupled sub-systems: - **Trackers** fire Snowplow events. Snowplow has twelve trackers that cover web, mobile, desktop, server, and IoT. - **Collectors** receive Snowplow events from trackers. We use different event collectors that synchronize events to Amazon S3, Apache Kafka, or Amazon Kinesis. @@ -80,20 +80,21 @@ sequenceDiagram For more details about the architecture, see [Snowplow infrastructure](infrastructure.md). -## Structured event taxonomy +## Event schema -Click events must be consistent. If each feature captures events differently, it can be difficult +All the events must be consistent. If each feature captures events differently, it can be difficult to perform analysis. -Each click event provides attributes that describe the event. +Each event provides attributes that describe the event. | Attribute | Type | Required | Description | | --------- | ------- | -------- | ----------- | | category | text | true | The page or backend section of the application. Unless infeasible, use the Rails page attribute by default in the frontend, and namespace + class name on the backend, for example, `Notes::CreateService`. | | action | text | true | The action the user takes, or aspect that's being instrumented. The first word must describe the action or aspect. For example, clicks must be `click`, activations must be `activate`, creations must be `create`. Use underscores to describe what was acted on. For example, activating a form field is `activate_form_input`, an interface action like clicking on a dropdown list is `click_dropdown`, a behavior like creating a project record from the backend is `create_project`. | -| label | text | false | The specific element or object to act on. This can be one of the following: the label of the element, for example, a tab labeled 'Create from template' for `create_from_template`; a unique identifier if no text is available, for example, `groups_dropdown_close` for closing the Groups dropdown in the top bar; or the name or title attribute of a record being created. For Service Ping metrics adapted to Snowplow events, this should be the full metric [key path](../service_ping/metrics_dictionary.md#metric-key_path) taken from its definition file. | +| label | text | false | The specific element or object to act on. This can be one of the following: the label of the element, for example, a tab labeled 'Create from template' for `create_from_template`; a unique identifier if no text is available, for example, `groups_dropdown_close` for closing the Groups dropdown list in the top bar; or the name or title attribute of a record being created. For Service Ping metrics adapted to Snowplow events, this should be the full metric [key path](../service_ping/metrics_dictionary.md#metric-key_path) taken from its definition file. | | property | text | false | Any additional property of the element, or object being acted on. For Service Ping metrics adapted to Snowplow events, this should be additional information or context that can help analyze the event. For example, in the case of `usage_activity_by_stage_monthly.create.merge_requests_users`, there are four different possible merge request actions: "create", "merge", "comment", and "close". Each of these would be a possible property value. | | value | decimal | false | Describes a numeric value (decimal) directly related to the event. This could be the value of an input. For example, `10` when clicking `internal` visibility. | +| context | vector | false | Additional data in the form of a [self-describing JSON](https://docs.snowplow.io/docs/pipeline-components-and-applications/iglu/common-architecture/self-describing-json-schemas/) to describe the event if the attributes are not sufficient. Each context must have its schema defined to assure data integrity. Refer to the list of GitLab-defined contexts for more details. | ### Examples @@ -185,16 +186,16 @@ LIMIT 100 ### Web-specific parameters -Snowplow JavaScript adds [web-specific parameters](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/snowplow-tracker-protocol/#Web-specific_parameters) to all web events by default. +Snowplow JavaScript adds [web-specific parameters](https://docs.snowplow.io/docs/collecting-data/collecting-from-own-applications/snowplow-tracker-protocol/#Web-specific_parameters) to all web events by default. ## Related topics -- [Snowplow data structure](https://docs.snowplowanalytics.com/docs/understanding-your-pipeline/canonical-event/) +- [Snowplow data structure](https://docs.snowplow.io/docs/understanding-your-pipeline/canonical-event/) - [Our Iglu schema registry](https://gitlab.com/gitlab-org/iglu) - [List of events used in our codebase (Event Dictionary)](https://metrics.gitlab.com/snowplow/) - [Product Intelligence Guide](https://about.gitlab.com/handbook/product/product-intelligence-guide/) - [Service Ping Guide](../service_ping/index.md) -- [Product Intelligence Direction](https://about.gitlab.com/direction/product-intelligence/) +- [Product Intelligence Direction](https://about.gitlab.com/direction/analytics/product-intelligence/) - [Data Analysis Process](https://about.gitlab.com/handbook/business-technology/data-team/#data-analysis-process/) - [Data for Product Managers](https://about.gitlab.com/handbook/business-technology/data-team/programs/data-for-product-managers/) - [Data Infrastructure](https://about.gitlab.com/handbook/business-technology/data-team/platform/infrastructure/) diff --git a/doc/development/snowplow/review_guidelines.md b/doc/development/snowplow/review_guidelines.md index 756dbd0ca92..ee2c1eb95df 100644 --- a/doc/development/snowplow/review_guidelines.md +++ b/doc/development/snowplow/review_guidelines.md @@ -35,7 +35,7 @@ events or touches Snowplow related files. #### The Product Intelligence **reviewer** should -- Check that the [event taxonomy](index.md#structured-event-taxonomy) is correct. +- Check that the [event schema](index.md#event-schema) is correct. - Check the [usage recommendations](implementation.md#usage-recommendations). - Check that the [Event Dictionary](event_dictionary_guide.md) is up-to-date. - If needed, check that the events are firing locally using one of the diff --git a/doc/development/snowplow/schemas.md b/doc/development/snowplow/schemas.md index 482c0e0105b..5a6cdea9fee 100644 --- a/doc/development/snowplow/schemas.md +++ b/doc/development/snowplow/schemas.md @@ -33,8 +33,8 @@ _\* Default value present for frontend events only_ ## Default Schema -Frontend events include a [web-specific schema](https://docs.snowplowanalytics.com/docs/understanding-your-pipeline/canonical-event/#Web-specific_fields) provided by Snowplow. -All URLs are pseudonymized. The entity identifier [replaces](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/javascript-trackers/javascript-tracker/javascript-tracker-v2/tracker-setup/other-parameters-2/#Setting_a_custom_page_URL_and_referrer_URL) personally identifiable +Frontend events include a [web-specific schema](https://docs.snowplow.io/docs/understanding-your-pipeline/canonical-event/#web-specific-fields) provided by Snowplow. +All URLs are pseudonymized. The entity identifier [replaces](https://docs.snowplow.io/docs/collecting-data/collecting-from-own-applications/javascript-trackers/javascript-tracker/javascript-tracker-v2/tracker-setup/other-parameters-2/#setting-a-custom-page-url-and-referrer-url) personally identifiable information (PII). PII includes usernames, group, and project names. Page titles are hardcoded as `GitLab` for the same reason. @@ -172,3 +172,18 @@ Page titles are hardcoded as `GitLab` for the same reason. | `v_collector` | **{dotted-circle}** | string | Collector version | | `v_etl` | **{dotted-circle}** | string | ETL version | | `v_tracker` | **{dotted-circle}** | string | Identifier for Snowplow tracker | + +## `gitlab_service_ping` + +Backend events converted from ServicePing (`redis` and `redis_hll`) must include [ServicePing context](https://gitlab.com/gitlab-org/iglu/-/tree/master/public/schemas/com.gitlab/gitlab_service_ping/jsonschema) +using the [helper class](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/tracking/service_ping_context.rb). + +An example of converted `redis_hll` [event with context](https://gitlab.com/gitlab-org/gitlab/-/edit/master/app/controllers/concerns/product_analytics_tracking.rb#L58). + +| Field Name | Required | Type | Description | +|---------------|:-------------------:|------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `data_source` | **{check-circle}** | string (max 64 chars) | The `data_source` attribute from the metrics YAML definition. | +| `event_name`* | **{dotted-circle}** | string (max 128 chars) | When there is a many-to-many relationship between events and metrics, this field contains the name of a Redis event that can be used for aggregations in downstream systems | +| `key_path`* | **{dotted-circle}** | string (max 256 chars) | The `key_path` attribute from the metrics YAML definition | + +_\* Either `event_name` or `key_path` is required_ diff --git a/doc/development/spam_protection_and_captcha/exploratory_testing.md b/doc/development/spam_protection_and_captcha/exploratory_testing.md index e17d25acb9d..f6e3e6814a8 100644 --- a/doc/development/spam_protection_and_captcha/exploratory_testing.md +++ b/doc/development/spam_protection_and_captcha/exploratory_testing.md @@ -91,7 +91,7 @@ CAPTCHA response string does not matter. It can be any string. If you use a real, valid key pair, you must solve the CAPTCHA to obtain a valid CAPTCHA response to use. You can do this once only, and only before it expires. -To directly test the GraphQL API via [GraphQL Explorer](http://gdk.test:3000/-/graphql-explorer), +To directly test the GraphQL API via GraphQL Explorer (`http://gdk.test:3000/-/graphql-explorer`), get a reCAPTCHA response string via this form: `public/recaptcha.html` (`http://gdk.test:3000/recaptcha.html`): ```html diff --git a/doc/development/sql.md b/doc/development/sql.md index 5829d27b8ee..cdc952eb08b 100644 --- a/doc/development/sql.md +++ b/doc/development/sql.md @@ -103,7 +103,7 @@ transaction. Transactions for migrations can be disabled using the following pattern: ```ruby -class MigrationName < Gitlab::Database::Migration[1.0] +class MigrationName < Gitlab::Database::Migration[2.0] disable_ddl_transaction! end ``` @@ -111,7 +111,7 @@ end For example: ```ruby -class AddUsersLowerUsernameEmailIndexes < Gitlab::Database::Migration[1.0] +class AddUsersLowerUsernameEmailIndexes < Gitlab::Database::Migration[2.0] disable_ddl_transaction! def up diff --git a/doc/development/stage_group_observability/dashboards/index.md b/doc/development/stage_group_observability/dashboards/index.md index 9f9ba609271..c2da46bde7d 100644 --- a/doc/development/stage_group_observability/dashboards/index.md +++ b/doc/development/stage_group_observability/dashboards/index.md @@ -43,7 +43,7 @@ All metrics recorded in the GitLab production system have [one-year retention](https://gitlab.com/gitlab-cookbooks/gitlab-prometheus/-/blob/31526b03fef823e2f9b3cda7c75dcd28a12418a3/attributes/prometheus.rb#L40). You can also zoom in and filter the time range directly on a graph. For more information, see the -[Grafana Time Range Controls](https://grafana.com/docs/grafana/latest/dashboards/time-range-controls/) +[Grafana Time Range Controls](https://grafana.com/docs/grafana/latest/dashboards/use-dashboards/#set-dashboard-time-range) documentation. ## Filters and annotations @@ -51,7 +51,7 @@ documentation. On each dashboard, there are two filters and some annotation switches on the top of the page. Some special events are meaningful to development and operational activities. -[Grafana annotations](https://grafana.com/docs/grafana/latest/dashboards/annotations/) mark them +[Grafana annotations](https://grafana.com/docs/grafana/latest/dashboards/build-dashboards/annotate-visualizations/) mark them directly on the graphs.  diff --git a/doc/development/stage_group_observability/dashboards/stage_group_dashboard.md b/doc/development/stage_group_observability/dashboards/stage_group_dashboard.md index fb5d5bbe379..abc46c52407 100644 --- a/doc/development/stage_group_observability/dashboards/stage_group_dashboard.md +++ b/doc/development/stage_group_observability/dashboards/stage_group_dashboard.md @@ -56,7 +56,7 @@ description, note the following: To inspect the raw data of the panel for further calculation, select **Inspect** from the dropdown list of a panel. Queries, raw data, and panel JSON structure are available. -Read more at [Grafana panel inspection](http://grafana.com/docs/grafana/next/panels/query-a-data-source/). +Read more at [Grafana panel inspection](https://grafana.com/docs/grafana/latest/panels-visualizations/query-transform-data/). All the dashboards are powered by [Grafana](https://grafana.com/), a frontend for displaying metrics. Grafana consumes the data returned from queries to backend Prometheus data source, then presents it diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index f1a23f06699..b6bf3c7805a 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -110,7 +110,7 @@ If the specs fail the check they must be fixed before than can run in random ord ### Test speed -GitLab has a massive test suite that, without [parallelization](../pipelines.md#test-suite-parallelization), can take hours +GitLab has a massive test suite that, without [parallelization](../pipelines/index.md#test-suite-parallelization), can take hours to run. It's important that we make an effort to write tests that are accurate and effective _as well as_ fast. @@ -282,7 +282,7 @@ end #### Stubbing methods within factories -You should avoid using `allow(object).to receive(:method)` in factories, as this makes the factory unable to be used with `let_it_be`. +You should avoid using `allow(object).to receive(:method)` in factories, as this makes the factory unable to be used with `let_it_be`, as described in [common test setup](#common-test-setup). Instead, you can use `stub_method` to stub the method: @@ -425,6 +425,11 @@ results are available, and not just the first failure. when you need an ID/IID/access level that doesn't actually exists. Using 123, 1234, or even 999 is brittle as these IDs could actually exist in the database in the context of a CI run. +- All top-level `RSpec.describe` blocks should have [`feature_category`](https://about.gitlab.com/categories.json) metadata set. + Consider splitting the file in the case there are identified multiple feature categories in same file. + If no `feature_category` is identified then use `not_owned`. This information is used in flaky test + issues created in order to identify the group owning the feature. + Eg: `RSpec.describe Admin::Geo::SettingsController, :geo, feature_category: :geo_replication do`. ### Coverage @@ -921,7 +926,8 @@ them unspecified, and look up the value after the row is created. #### Redis GitLab stores two main categories of data in Redis: cached items, and Sidekiq -jobs. +jobs. [View the full list of `Gitlab::Redis::Wrapper` descendants](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/redis.rb) that are backed by +a separate Redis instance. In most specs, the Rails cache is actually an in-memory store. This is replaced between specs, so calls to `Rails.cache.read` and `Rails.cache.write` are safe. @@ -961,6 +967,14 @@ it "really connects to Prometheus", :permit_dns do And if you need more specific control, the DNS blocking is implemented in `spec/support/helpers/dns_helpers.rb` and these methods can be called elsewhere. +#### Rate Limiting + +[Rate limiting](../../security/rate_limits.md) is enabled in the test suite. Rate limits +may be triggered in feature specs that use the `:js` trait. In most cases, triggering rate +limiting can be avoided by marking the spec with the `:clean_gitlab_redis_rate_limiting` +trait. This trait clears the rate limiting data stored in Redis cache between specs. If +a single test triggers the rate limit, the `:disable_rate_limit` can be used instead. + #### Stubbing File methods In the situations where you need to @@ -1437,9 +1451,10 @@ GitLab uses [factory_bot](https://github.com/thoughtbot/factory_bot) as a test f resulting record to pass validation. - When instantiating from a factory, don't supply attributes that aren't required by the test. -- Prefer [implicit](https://github.com/thoughtbot/factory_bot/blob/master/GETTING_STARTED.md#implicit-definition) - or [explicit](https://github.com/thoughtbot/factory_bot/blob/master/GETTING_STARTED.md#explicit-definition) - association definitions instead of using `create` / `build` for association setup. +- Prefer [implicit](https://github.com/thoughtbot/factory_bot/blob/master/GETTING_STARTED.md#implicit-definition), + [explicit](https://github.com/thoughtbot/factory_bot/blob/master/GETTING_STARTED.md#explicit-definition), or + [inline](https://github.com/thoughtbot/factory_bot/blob/master/GETTING_STARTED.md#inline-definition) associations + over `create` / `build` for association setup in callbacks. See [issue #262624](https://gitlab.com/gitlab-org/gitlab/-/issues/262624) for further context. - Factories don't have to be limited to `ActiveRecord` objects. [See example](https://gitlab.com/gitlab-org/gitlab-foss/commit/0b8cefd3b2385a21cfed779bd659978c0402766d). diff --git a/doc/development/testing_guide/contract/consumer_tests.md b/doc/development/testing_guide/contract/consumer_tests.md index 67bb7160749..9c72e6835bd 100644 --- a/doc/development/testing_guide/contract/consumer_tests.md +++ b/doc/development/testing_guide/contract/consumer_tests.md @@ -213,7 +213,7 @@ pactWith( const discussions = await getDiscussions({ url: provider.mockService.baseUrl, }); - + expect(discussions).toEqual(Matchers.eachLike({ id: 'fd73763cbcbf7b29eb8765d969a38f7d735e222a', project_id: 6954442, diff --git a/doc/development/testing_guide/end_to_end/best_practices.md b/doc/development/testing_guide/end_to_end/best_practices.md index 37eda7f76f5..777a9f47339 100644 --- a/doc/development/testing_guide/end_to_end/best_practices.md +++ b/doc/development/testing_guide/end_to_end/best_practices.md @@ -17,8 +17,8 @@ In case custom inflection logic is needed, custom inflectors are added in the [q ## Link a test to its test case Every test should have a corresponding test case in the [GitLab project Test Cases](https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases) as well as a results issue in the [Quality Test Cases project](https://gitlab.com/gitlab-org/quality/testcases/-/issues). -If a test case issue does not yet exist you can create one yourself. To do so, create a new -issue in the [Test Cases](https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases) GitLab project +If a test case issue does not yet exist, any GitLab team member can create a new test case in +the [Test Cases](https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases) GitLab project with a placeholder title. After the test case URL is linked to a test in the code, when the test is run in a pipeline that has reporting enabled, the `report-results` script automatically updates the test case and the results issue. diff --git a/doc/development/testing_guide/end_to_end/feature_flags.md b/doc/development/testing_guide/end_to_end/feature_flags.md index 8e25c817938..6d826e170f6 100644 --- a/doc/development/testing_guide/end_to_end/feature_flags.md +++ b/doc/development/testing_guide/end_to_end/feature_flags.md @@ -160,7 +160,7 @@ For example: ```ruby def initialize - name_of_the_future_flag_activated = false + name_of_the_feature_flag_activated = false ... end ``` diff --git a/doc/development/testing_guide/end_to_end/index.md b/doc/development/testing_guide/end_to_end/index.md index 1853ad47779..8ffe044c4d8 100644 --- a/doc/development/testing_guide/end_to_end/index.md +++ b/doc/development/testing_guide/end_to_end/index.md @@ -123,9 +123,8 @@ test or a group of tests that are different than the groups in any of the existi For example, when we [dequarantine](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/debugging-qa-test-failures/#dequarantining-tests) a flaky test we first want to make sure that it's no longer flaky. -We can do that using the `ce:custom-parallel` and `ee:custom-parallel` jobs. -Both are manual jobs that you can configure using custom variables. -When selecting the name (not the play icon) of one of the parallel jobs, +We can do that by running `_ee:quarantine` manual job. +When selecting the name (not the play icon) of manual job, you are prompted to enter variables. You can use any of [the variables that can be used with `gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#supported-gitlab-environment-variables) as well as these: @@ -134,7 +133,7 @@ as well as these: |-|-| | `QA_SCENARIO` | The scenario to run (default `Test::Instance::Image`) | | `QA_TESTS` | The tests to run (no default, which means run all the tests in the scenario). Use file paths as you would when running tests via RSpec, for example, `qa/specs/features/ee/browser_ui` would include all the `EE` UI tests. | -| `QA_RSPEC_TAGS` | The RSpec tags to add (no default) | +| `QA_RSPEC_TAGS` | The RSpec tags to add (default `--tag quarantine`) | For now, [manual jobs with custom variables don't use the same variable when retried](https://gitlab.com/gitlab-org/gitlab/-/issues/31367), @@ -157,6 +156,19 @@ against the [Review App](../review_apps.md). See [Review Apps](../review_apps.md) for more details about Review Apps. +#### Selective test execution + +In order to limit amount of tests executed in a merge request, dynamic selection of which tests to execute is present. Algorithm of which tests to run is based +on changed files and merge request labels. Following criteria determine which tests will run: + +1. Changes in `qa` framework code would execute the full suite +1. Changes in particular `_spec.rb` file in `qa` folder would execute only that particular test +1. Merge request with backend changes and label `devops::manage` would execute all e2e tests related to `manage` stage + +#### Overriding selective test execution + +To override selective test execution and trigger the full suite, label `pipeline:run-all-e2e` should be added to particular merge request. + ### Run tests in parallel To run tests in parallel on CI, the [Knapsack](https://github.com/KnapsackPro/knapsack) @@ -184,7 +196,8 @@ Use these environment variables to configure metrics export: | `QA_INFLUXDB_URL` | `true` | Should be set to `https://influxdb.quality.gitlab.net`. No default value. | | `QA_INFLUXDB_TOKEN` | `true` | InfluxDB write token that can be found under `Influxdb auth tokens` document in `Gitlab-QA` `1Password` vault. No default value. | | `QA_RUN_TYPE` | `false` | Arbitrary name for test execution, like `package-and-test`. Automatically inferred from the project name for live environment test executions. No default value. | -| `QA_EXPORT_TEST_METRICS` | `false` | Flag to enable or disable metrics export. Defaults to `true`. | +| `QA_EXPORT_TEST_METRICS` | `false` | Flag to enable or disable metrics export to InfluxDB. Defaults to `false`. | +| `QA_SAVE_TEST_METRICS` | `false` | Flag to enable or disable saving metrics as JSON file. Defaults to `false`. | ## Test reports diff --git a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md index b12c6bd443a..a7d1ece77b2 100644 --- a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md +++ b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md @@ -36,7 +36,7 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec | `:only` | The test is only to be run in specific execution contexts. See [test execution context selection](execution_context_selection.md) for more information. | | `:orchestrated` | The GitLab instance under test may be [configured by `gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#orchestrated-tests) to be different to the default GitLab configuration, or `gitlab-qa` may launch additional services in separate Docker containers, or both. Tests tagged with `:orchestrated` are excluded when testing environments where we can't dynamically modify the GitLab configuration (for example, Staging). | | `:packages` | The test requires a GitLab instance that has the [Package Registry](../../../administration/packages/index.md#gitlab-package-registry-administration) enabled. | -| `:product_group` | Specifies what product group the test belongs to. See [Product sections, stages, groups, and categories](https://about.gitlab.com/handbook/product/categories) for the comprehensive groups list. | +| `:product_group` | Specifies what product group the test belongs to. See [Product sections, stages, groups, and categories](https://about.gitlab.com/handbook/product/categories/) for the comprehensive groups list. | | `:quarantine` | The test has been [quarantined](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/debugging-qa-test-failures/#quarantining-tests), runs in a separate job that only includes quarantined tests, and is allowed to fail. The test is skipped in its regular job so that if it fails it doesn't hold up the pipeline. Note that you can also [quarantine a test only when it runs in a specific context](execution_context_selection.md#quarantine-a-test-for-a-specific-environment). | | `:relative_url` | The test requires a GitLab instance to be installed under a [relative URL](../../../install/relative_url.md). | | `:reliable` | The test has been [promoted to a reliable test](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/reliable-tests/#promoting-an-existing-test-to-reliable) meaning it passes consistently in all pipelines, including merge requests. | diff --git a/doc/development/testing_guide/end_to_end/style_guide.md b/doc/development/testing_guide/end_to_end/style_guide.md index f380b24d7a5..419942d6b8f 100644 --- a/doc/development/testing_guide/end_to_end/style_guide.md +++ b/doc/development/testing_guide/end_to_end/style_guide.md @@ -12,7 +12,7 @@ This document describes the conventions used at GitLab for writing End-to-end (E ### When to use `click_`? -When clicking in a single link to navigate, use `click_`. +When selecting a single link to navigate, use `click_`. For example: @@ -24,9 +24,9 @@ def click_ci_cd_pipelines end ``` -From a testing perspective, if we want to check that clicking a link, or a button (a single interaction) is working as intended, we would want the test to read as: +From a testing perspective, if we want to check that selecting a link, or a button (a single interaction) is working as intended, we would want the test to read as: -- Click a certain element +- Select a certain element - Verify the action took place ### When to use `go_to_`? @@ -47,7 +47,7 @@ end `go_to_` fits the definition of interacting with multiple elements very well given it's more of a meta-navigation action that includes multiple interactions. -Notice that in the above example, before clicking the `:operations_environments_link`, another element is hovered over. +Notice that in the above example, before selecting the `:operations_environments_link`, another element is hovered over. > We can create these methods as helpers to abstract multi-step navigation. diff --git a/doc/development/testing_guide/flaky_tests.md b/doc/development/testing_guide/flaky_tests.md index 45f1d0ddf7e..cc62a0ebf03 100644 --- a/doc/development/testing_guide/flaky_tests.md +++ b/doc/development/testing_guide/flaky_tests.md @@ -11,9 +11,144 @@ info: To determine the technical writer assigned to the Stage/Group associated w It's a test that sometimes fails, but if you retry it enough times, it passes, eventually. +## What are the potential cause for a test to be flaky? + +### Unclean environment + +**Label:** `flaky-test::unclean environment` + +**Description:** The environment got dirtied by a previous test. The actual cause is probably not the flaky test here. + +**Difficulty to reproduce:** Moderate. Usually, running the same spec files until the one that's failing reproduces the problem. + +**Resolution:** Fix the previous tests and/or places where the environment is modified, so that +it's reset to a pristine test after each test. + +**Examples:** + +- [Example 1](https://gitlab.com/gitlab-org/gitlab/-/issues/378414#note_1142026988): A migration + test might roll-back the database, perform its testing, and then roll-up the database in an + inconsistent state, so that following tests might not know about certain columns. +- [Example 2](https://gitlab.com/gitlab-org/gitlab/-/issues/368500): A test modifies data that is + used by a following test. +- [Example 3](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/103434#note_1172316521): A test for a database query passes in a fresh database, but in a + CI/CD pipeline where the database is used to process previous test sequences, the test fails. This likely + means that the query itself needs to be updated to work in a non-clean database. + +### Ordering assertion + +**Label:** `flaky-test::ordering assertion` + +**Description:** The test is expecting a specific order in the data under test yet the data is in +a non-deterministic order. + +**Difficulty to reproduce:** Easy. Usually, running the test locally several times would reproduce +the problem. + +**Resolution:** Depending on the problem, you might want to: + +- loosen the assertion if the test shouldn't care about ordering but only on the elements +- fix the test by specifying a deterministic ordering +- fix the app code by specifying a deterministic ordering + +**Examples:** + +- [Example 1](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10148/diffs): Without + specifying `ORDER BY`, database will not give deterministic ordering, or data race happening + in the tests. + +### Dataset-specific + +**Label:** `flaky-test::dataset-specific` + +**Description:** The test assumes the dataset is in a particular (usually limited) state, which +might not be true depending on when the test run during the test suite. + +**Difficulty to reproduce:** Moderate, as the amount of data needed to reproduce the issue might be +difficult to achieve locally. + +**Resolution:** Fix the test to not assume that the dataset is in a particular state, don't hardcode IDs. + +**Examples:** + +- [Example 1](https://gitlab.com/gitlab-org/gitlab/-/issues/378381): The database is recreated when + any table has more than 500 columns. It could pass in the merge request, but fail later in + `master` if the order of tests changes. +- [Example 2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91016/diffs): A test asserts + that trying to find a record with an unexisting ID retuns an error message. The test uses an + hardcoded ID that's supposed to not exist (e.g. `42`). If the test is run early in the test + suite, it might pass as not enough records were created before it, but as soon as it would run + later in the suite, there could be a record that actually has the ID `42`, hence the test would + start to fail. + +### Random input + +**Label:** `flaky-test::random input` + +**Description:** The test use random values, that sometimes match the expectations, and sometimes not. + +**Difficulty to reproduce:** Easy, as the test can be modified locally to use the "random value" +used at the time the test failed + +**Resolution:** Once the problem is reproduced, it should be easy to debug and fix either the test +or the app. + +**Examples:** + +- [Example 1](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/20121): The test isn't robust enough to handle a specific data, that only appears sporadically since the data input is random. + +### Unreliable DOM Selector + +**Label:** `flaky-test::unreliable dom selector` + +**Description:** The DOM selector used in the test is unreliable. + +**Difficulty to reproduce:** Moderate to difficult. Depending on whether the DOM selector is duplicated, or appears after a delay etc. + +**Resolution:** It really depends on the problem here. It could be to wait for requests to finish, to scroll down the page etc. + +**Examples:** + +- [Example 1](https://gitlab.com/gitlab-org/gitlab/-/issues/338341): A non-unique CSS selector + matching more than one element, or a non-waiting selector method that does not allow rendering + time before throwing an `element not found` error. +- [Example 2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/101728/diffs): A CSS selector + only appears after a GraphQL requests has finished, and the UI has updated. + +### Datetime-sensitive + +**Label:** `flaky-test::datetime-sensitive` + +**Description:** The test is assuming a specific date or time. + +**Difficulty to reproduce:** Easy to moderate, depending on whether the test consistently fails after a certain date, or only fails at a given time or date. + +**Resolution:** Freezing the time is usually a good solution. + +**Examples:** + +- [Example 1](https://gitlab.com/gitlab-org/gitlab/-/issues/118612): A test that breaks after some time passed. + +### Unstable infrastructure + +**Label:** `flaky-test::unstable infrastructure` + +**Description:** The test fails from time to time due to infrastructure issues. + +**Difficulty to reproduce:** Hard. It's really hard to reproduce CI infrastructure issues. It might +be possible by using containers locally. + +**Resolution:** Starting a conversation with the Infrastructure department in a dedicated issue is +usually a good idea. + +**Examples:** + +- [Example 1](https://gitlab.com/gitlab-org/gitlab/-/issues/363214): The runner is under heavy load at this time. +- [Example 2](https://gitlab.com/gitlab-org/gitlab/-/issues/360559): The runner is having networking issues, making a job failing early + ## Quarantined tests -When a test frequently fails in `main`, +When a test frequently fails in `master`, create [a ~"failure::flaky-test" issue](https://about.gitlab.com/handbook/engineering/workflow/#broken-master). If the test cannot be fixed in a timely fashion, there is an impact on the @@ -62,7 +197,7 @@ For example, `FLAKY_RSPEC_GENERATE_REPORT=1 bin/rspec ...`. The `rspec/flaky/report-suite.json` report is: -- Used for [automatically skipping known flaky tests](../pipelines.md#automatic-skipping-of-flaky-tests). +- Used for [automatically skipping known flaky tests](../pipelines/index.md#automatic-skipping-of-flaky-tests). - [Imported into Snowflake](https://gitlab.com/gitlab-data/analytics/-/blob/master/extract/gitlab_flaky_tests/upload.py) once per day, for monitoring with the [internal dashboard](https://app.periscopedata.com/app/gitlab/888968/EP---Flaky-tests). diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md index d7627e2064f..041b0f0a4f4 100644 --- a/doc/development/testing_guide/frontend_testing.md +++ b/doc/development/testing_guide/frontend_testing.md @@ -11,7 +11,7 @@ at GitLab. We use Jest for JavaScript unit and integration testing, and RSpec feature tests with Capybara for e2e (end-to-end) integration testing. Unit and feature tests need to be written for all new features. -Most of the time, you should use [RSpec](https://github.com/rspec/rspec-rails#feature-specs) for your feature tests. +Most of the time, you should use [RSpec](https://github.com/rspec/rspec-rails#feature-specs) for your feature tests. For more information on how to get started with feature tests, see [get started with feature tests](#get-started-with-feature-tests) Regression tests should be written for bug fixes to prevent them from recurring in the future. @@ -458,7 +458,7 @@ If you cannot register handlers to the `Promise`, for example because it is exec ```javascript it('waits for an Ajax call', async () => { synchronousFunction(); - + await waitForPromises(); expect(something).toBe('done'); @@ -962,7 +962,7 @@ If an integration test depends on JavaScript to run correctly, you need to make sure the spec is configured to enable JavaScript when the tests are run. If you don't do this, the spec runner displays vague error messages. -To enable a JavaScript driver in an `rspec` test, add `:js` to the +To enable a JavaScript driver in an `RSpec` test, add `:js` to the individual spec or the context block containing multiple specs that need JavaScript enabled: @@ -1242,6 +1242,332 @@ A good guideline to follow: the more complex the component you may want to steer - To just have some kind of test written - To capture highly volatile UI elements without stubbing them (Think of GitLab UI version updates) +## Get started with feature tests + +### What is a feature test + +A [feature test](testing_levels.md#white-box-tests-at-the-system-level-formerly-known-as-system--feature-tests), also known as `white-box testing`, is a test that spawns a browser and has Capybara helpers. This means the test can: + +- Locate an element in the browser. +- Click that element. +- Call the API. + +Feature tests are expensive to run. You should make sure that you **really want** this type of test before running one. + +All of our feature tests are written in `Ruby` but often end up being written by `JavaScript` engineers, as they implement the user-facing feature. So, the following section assumes no prior knowledge of `Ruby` or `Capybara`, and provide a clear guideline on when and how to use these tests. + +### When to use feature tests + +You should use a feature test when the test: + +- Is across multiple components. +- Requires that a user navigate across pages. +- Is submitting a form and observing results elsewhere. +- Would result in a huge number of mocking and stubbing with fake data and components if done as a unit test. + +Feature tests are especially useful when you want to test: + +- That multiple components are working together successfully. +- Complex API interactions. Feature tests interact with the API, so they are slower but do not need any level of mocking or fixtures. + +### When not to use feature tests + +You should use `jest` and `vue-test-utils` unit tests instead of a feature test if you can get the same test results from these methods. Feature tests are quite expensive to run. + +You should use a unit test if: + +- The behavior you are implementing is all in one component. +- You can simulate other components' behavior to trigger the desired effect. +- You can already select UI elements in the virtual DOM to trigger the desired effects. + +Also, if a behavior in your new code needs multiple components to work together, you should consider testing your behavior higher in the component tree. For example, let's say that we have a component called `ParentComponent` with the code: + +```vue + <script> + export default{ + name: ParentComponent, + data(){ + return { + internalData: 'oldValue' + } + }, + methods:{ + changeSomeInternalData(newVal){ + this.internalData = newVal + } + } + } + </script> + <template> + <div> + <child-component-1 @child-event="changeSomeInternalData" /> + <child-component-2 :parent-data="internalData" /> + </div> + </template> +``` + +In this example: + +- `ChildComponent1` emits an event. +- `ParentComponent` changes its `internalData` value. +- `ParentComponent` passes the props down to `ChildComponent2`. + +You can use a unit test instead by: + +- From inside the `ParentComponent` unit test file, emitting the expected event from `childComponent1` +- Making sure the prop is passed down to `childComponent2`. + +Then each child component unit tests what happens when the event is emitted and when the prop changes. + +This example also applies at larger scale and with deeper component trees. It is definitely worth using unit tests and avoiding the extra cost of feature tests if you can: + +- Confidently mount child components. +- Emit events or select elements in the virtual DOM. +- Get the test behavior that you want. + +### Where to create your test + +Feature tests live in `spec/features` folder. You should look for existing files that can test the page you are adding a feature to. Within that folder, you can locate your section. For example, if you wanted to add a new feature test for the pipeline page, you would look in `spec/features/projects/pipelines` and see if the test you want to write exists here. + +### How to run a feature test + +1. Make sure that you have a working GDK environment. +1. Start your `gdk` environment with `gdk start` command. +1. In your terminal, run: + + ```shell + bundle exec rspec path/to/file:line_of_my_test + ``` + +You can also prefix this command with `WEBDRIVER_HEADLESS=0` which will run the test by opening an actual browser on your computer that you can see, which is very useful for debugging. + +### How to write a test + +#### Basic file structure + +1. Make all string literals unchangeable + + In all feature tests, the very first line should be: + + ```ruby + # frozen_string_literal: true + ``` + + This is in every `Ruby` file and makes all string literals unchangeable. There are also some performance benefits, but this is beyond the scope of this section. + +1. Import dependencies. + + You should import the modules you need. You will most likely always need to require `spec_helper`: + + ```ruby + require 'spec_helper' + ``` + + Import any other relevant module. + +1. Create a global scope for RSpec to define our tests, just like what we do in jest with the initial describe block. + +Then, you need to create the very first `RSpec` scope. + +```ruby +RSpec.describe 'Pipeline', :js do + ... +end +``` + +What is different though, is that just like everything in Ruby, this is actually a `class`. Which means that right at the top, you can `include` modules that you'd need for your test. For example, you could include the `RoutesHelpers` to navigate more easily. + +```ruby +RSpec.describe 'Pipeline', :js do + include RoutesHelpers + ... +end +``` + +After all of this implementation, we have a file that looks something like this: + +```ruby +# frozen_string_literal: true + +require 'spec_helper' + +RSpec.describe 'Pipeline', :js do + include RoutesHelpers +end +``` + +#### Seeding data + +Each test is in its own environment and so you must use a factory to seed the required data. To continue on with the pipeline example, let's say that you want a test that takes you to the main pipeline page, which is at the route `/namespace/project/-/pipelines/:id/`. + +Most feature tests at least require you to create a user, because you want to be signed in. You can skip this step if you don't have to be signed in, but as a general rule, you should **always create a user unless you are specifically testing a feature looked at by an anonymous user**. This makes sure that you explicitly set a level of permission that you can edit in the test as needed to change or test a new level of permission as the section changes. To create a user: + +```ruby + let(:user) { create(:user) } +``` + +This creates a variable that holds the newly created user and we can use `create` because we imported the `spec_helper`. + +However, we have not done anything with this user yet because it's just a variable. So, in the `before do` block of the spec, we could sign in with the user so that every spec starts with a signed in user. + +```ruby + let(:user) { create(:user) } + + before do + sign_in(user) + end +``` + +Now that we have a user, we should look at what else we'd need before asserting anything on a pipeline page. If you look at the route `/namespace/project/-/pipelines/:id/` we can determine we need a project and a pipeline. + +So we'd create a project and pipeline, and link them together. Usually in factories, the child element requires its parent as an argument. In this case, a pipeline is a child of a project. So we can create the project first, and then when we create the pipeline, we are pass the project as an argument which "binds" the pipeline to the project. A pipeline is also owned by a user, so we need the user as well. For example, this creates a project and a pipeline: + +```ruby + let(:user) { create(:user) } + let(:project) { create(:project, :repository) } + let(:pipeline) { create(:ci_pipeline, project: project, ref: 'master', sha: project.commit.id, user: user) } +``` + +In the same spirit, you could then create a job (build) by using the build factory and passing the parent pipeline: + +```ruby + create(:ci_build, pipeline: pipeline, stage_idx: 10, stage: 'publish', name: 'CentOS') +``` + +There are many factories that already exists, so make sure to look at other existing files to see if what you need is available. + +#### Navigation + +You can navigate to a page by using the `visit` method and passing the path as an argument. Rails automatically generates helper paths, so make sure to use these instead of a hardcoded string. They are generated using the route model, so if we want to go to a pipeline, we'd use: + +```ruby + visit project_pipeline_path(project, pipeline) +``` + +Before executing any page interaction when navigating or making asynchronous call through the UI, make sure to use `wait_for_requests` before proceeding with further instructions. + +#### Elements interaction + +There are a lot of different ways to find and interact with elements. For example, you could use the basic `find` method with the `selector` and `text` parameter and then use the `.click` method + +```ruby + find('.gl-tab-nav-item', text: 'Tests').click +``` + +Alternatively, you could use `click_button` with a string of text that is found within the button, which is a more semantically meaningful way of clicking the element. + +```ruby + click_button 'Text inside the button element' +``` + +If you want to follow a link, then there is `click_link`: + +```ruby + click_link 'Text inside the link tag' +``` + +You can use `fill_in` to fill input / form elements. The first argument is the selector, the second is `with:` which is the value to pass in. + +```ruby + fill_in 'current_password', with: '123devops' +``` + +Alternatively, you can use the `find` selector paired with `send_keys` to add keys in a field without removing previous text, or `set` which completely replaces the value of the input element. + +All of these are valid selectors and methods. Pick whichever suits your needs and look around as there are many more useful ones! + +#### Assertions + +To assert anything in a page, you can always access `page` variable, which is automatically defines and actually means the page document. This means you can expect the `page` to have certain components like selectors or content. Here are a few examples: + +```ruby + # Finding an element by ID + expect(page).to have_selector('#js-pipeline-graph') +``` + +```ruby + # Finding by text + expect(page).to have_content('build') +``` + +```ruby + # Finding by `href` value + expect(page).to have_link(pipeline.ref) +``` + +```ruby + # Finding by CSS selector. This is a last resort. + # For example, when you cannot add attributes on the desired element. + expect(page).to have_css('.js-icon-retry') +``` + +```ruby + # Find by data-testid + # Like CSS selector, this is acceptable when there isn't a specific matcher available. + expect(page).to have_selector('[data-testid="pipeline-multi-actions-dropdown"]') +``` + +```ruby + # You can combine any of these selectors with `not_to` instead + expect(page).not_to have_selector('#js-pipeline-graph') +``` + +```ruby + # When a test case has back to back expectations, + # it is recommended to group them using `:aggregate_failures` + it 'shows the issue description and design references', :aggregate_failures do + expect(page).to have_text('The designs I mentioned') + expect(page).to have_link(design_tab_ref) + expect(page).to have_link(design_ref_a) + expect(page).to have_link(design_ref_b) + end +``` + +You can also create a sub-block to look into, to: + +- Scope down where you are making your assertions and reduce the risk of finding another element that was not intended. +- Make sure an element is found within the right boundaries. + +```ruby + page.within('#js-pipeline-graph') do + ... + end +``` + +#### Feature flags + +By default, every feature flag is enabled **regardless of the YAML definition or the flags you've set manually in your GDK**. To test when a feature flag is disabled, you must manually stub the flag, ideally in a `before do` block. + +```ruby + stub_feature_flags(my_feature_flag: false) +``` + +If you are stubbing an `ee` feature flag, then use: + +```ruby + stub_licensed_features(my_feature_flag: false) +``` + +### Debugging + +You can run your spec with the prefix `WEBDRIVER_HEADLESS=0` to open an actual browser. However, the specs goes though the commands quickly and leaves you no time to look around. + +To avoid this problem, you can write `binding.pry` on the line where you want Capybara to stop execution. You are then inside the browser with normal usage. To understand why you cannot find certain elements, you can: + +- Select elements. +- Use the console and network tab. +- Execute selectors inside the browser console. + +Inside the terminal, where capybara is running, you can also execute `next` which goes line by line through the test. This way you can check every single interaction one by one to see what might be causing an issue. + +### Updating ChromeDriver + +On MacOS, if you get a ChromeDriver error, make sure to update it by running + +```shell + brew upgrade chromedriver +``` + --- [Return to Testing documentation](index.md) diff --git a/doc/development/testing_guide/index.md b/doc/development/testing_guide/index.md index 753089fa673..5008564de01 100644 --- a/doc/development/testing_guide/index.md +++ b/doc/development/testing_guide/index.md @@ -43,12 +43,17 @@ system tests, parameterized tests etc. Everything you should know about how to write good Frontend tests: Jest, testing promises, stubbing etc. +## [Getting started with Feature tests](frontend_testing.md#get-started-with-feature-tests) + +Need to get started with feature tests? Here are some general guidelines, +tips and tricks to get the most out of white-box testing. + ## [Flaky tests](flaky_tests.md) What are flaky tests, the different kind of flaky tests we encountered, and what we do about them. -## [GitLab pipelines](../pipelines.md) +## [GitLab pipelines](../pipelines/index.md) How GitLab test suite is run in the CI context: setup, caches, artifacts, parallelization, monitoring. diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md index d86d1c3c7b4..5edf18725c0 100644 --- a/doc/development/testing_guide/review_apps.md +++ b/doc/development/testing_guide/review_apps.md @@ -4,9 +4,9 @@ group: unassigned info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Review Apps +# Using review apps in the development of GitLab -Review Apps are deployed using the `start-review-app-pipeline` job which triggers a child pipeline containing a series of jobs to perform the various tasks needed to deploy a Review App. +Review apps are deployed using the `start-review-app-pipeline` job which triggers a child pipeline containing a series of jobs to perform the various tasks needed to deploy a review app.  @@ -21,7 +21,7 @@ For any of the following scenarios, the `start-review-app-pipeline` job would be - for scheduled pipelines - the MR has the `pipeline:run-review-app` label set -## QA runs on Review Apps +## QA runs on review apps On every [pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/125315730) in the `qa` stage (which comes after the `review` stage), the `review-qa-smoke` and `review-qa-reliable` jobs are automatically started. The `review-qa-smoke` runs @@ -34,7 +34,7 @@ You can also manually start the `review-qa-all`: it runs the full QA suite. After the end-to-end test runs have finished, [Allure reports](https://github.com/allure-framework/allure2) are generated and published by the `allure-report-qa-smoke`, `allure-report-qa-reliable`, and `allure-report-qa-all` jobs. A comment with links to the reports are added to the merge request. -Errors can be found in the `gitlab-review-apps` Sentry project and [filterable by Review App URL](https://sentry.gitlab.net/gitlab/gitlab-review-apps/?query=url%3A%22https%3A%2F%2Fgitlab-review-require-ve-u92nn2.gitlab-review.app%2F%22) or [commit SHA](https://sentry.gitlab.net/gitlab/gitlab-review-apps/releases/6095b501da7/all-events/). +Errors can be found in the `gitlab-review-apps` Sentry project and [filterable by review app URL](https://sentry.gitlab.net/gitlab/gitlab-review-apps/?query=url%3A%22https%3A%2F%2Fgitlab-review-require-ve-u92nn2.gitlab-review.app%2F%22) or [commit SHA](https://sentry.gitlab.net/gitlab/gitlab-review-apps/releases/6095b501da7/all-events/). ### Bypass failed review app deployment to merge a broken `master` fix @@ -47,7 +47,7 @@ On every [pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/125315730) in browser performance testing using a [Sitespeed.io Container](../../ci/testing/browser_performance_testing.md). -## Sample Data for Review Apps +## Sample Data for review apps Upon deployment of a review app, project data is created from the [`sample-gitlab-project`](https://gitlab.com/gitlab-org/sample-data-templates/sample-gitlab-project) template project. This aims to provide projects with prepopulated resources to facilitate manual and exploratory testing. @@ -55,16 +55,16 @@ The sample projects will be created in the `root` user namespace and can be acce ## How to -### Redeploy Review App from a clean slate +### Redeploy review app from a clean slate -To reset Review App and redeploy from a clean slate, do the following: +To reset review app and redeploy from a clean slate, do the following: 1. Run `review-stop` job. 1. Re-deploy by running or retrying `review-deploy` job. -Doing this will remove all existing data from a previously deployed Review App. +Doing this will remove all existing data from a previously deployed review app. -### Get access to the GCP Review Apps cluster +### Get access to the GCP review apps cluster You need to [open an access request (internal link)](https://gitlab.com/gitlab-com/access-requests/-/issues/new) for the `gcp-review-apps-dev` GCP group and role. @@ -74,7 +74,7 @@ This grants you the following permissions for: - [Retrieving pod logs](#dig-into-a-pods-logs). Granted by [Viewer (`roles/viewer`)](https://cloud.google.com/iam/docs/understanding-roles#kubernetes-engine-roles). - [Running a Rails console](#run-a-rails-console). Granted by [Kubernetes Engine Developer (`roles/container.pods.exec`)](https://cloud.google.com/iam/docs/understanding-roles#kubernetes-engine-roles). -### Log into my Review App +### Log into my review app For GitLab Team Members only. If you want to sign in to the review app, review the GitLab handbook information for the [shared 1Password account](https://about.gitlab.com/handbook/security/#1password-for-teams). @@ -82,23 +82,23 @@ the GitLab handbook information for the [shared 1Password account](https://about - The default username is `root`. - The password can be found in the 1Password login item named `GitLab EE Review App`. -### Enable a feature flag for my Review App +### Enable a feature flag for my review app -1. Open your Review App and sign in as documented above. +1. Open your review app and sign in as documented above. 1. Create a personal access token. 1. Enable the feature flag using the [Feature flag API](../../api/features.md). -### Find my Review App slug +### Find my review app slug 1. Open the `review-deploy` job. 1. Look for `** Deploying review-*`. 1. For instance for `** Deploying review-1234-abc-defg... **`, - your Review App slug would be `review-1234-abc-defg` in this case. + your review app slug would be `review-1234-abc-defg` in this case. ### Run a Rails console 1. Make sure you [have access to the cluster](#get-access-to-the-gcp-review-apps-cluster) and the `container.pods.exec` permission first. -1. [Filter Workloads by your Review App slug](https://console.cloud.google.com/kubernetes/workload?project=gitlab-review-apps). For example, `review-qa-raise-e-12chm0`. +1. [Filter Workloads by your review app slug](https://console.cloud.google.com/kubernetes/workload?project=gitlab-review-apps). For example, `review-qa-raise-e-12chm0`. 1. Find and open the `toolbox` Deployment. For example, `review-qa-raise-e-12chm0-toolbox`. 1. Select the Pod in the "Managed pods" section. For example, `review-qa-raise-e-12chm0-toolbox-d5455cc8-2lsvz`. 1. Select the `KUBECTL` dropdown list, then `Exec` -> `toolbox`. @@ -111,7 +111,7 @@ the GitLab handbook information for the [shared 1Password account](https://about ### Dig into a Pod's logs 1. Make sure you [have access to the cluster](#get-access-to-the-gcp-review-apps-cluster) and the `container.pods.getLogs` permission first. -1. [Filter Workloads by your Review App slug](https://console.cloud.google.com/kubernetes/workload?project=gitlab-review-apps). For example, `review-qa-raise-e-12chm0`. +1. [Filter Workloads by your review app slug](https://console.cloud.google.com/kubernetes/workload?project=gitlab-review-apps). For example, `review-qa-raise-e-12chm0`. 1. Find and open the `migrations` Deployment. For example, `review-qa-raise-e-12chm0-migrations.1`. 1. Select the Pod in the "Managed pods" section. For example, `review-qa-raise-e-12chm0-migrations.1-nqwtx`. 1. Select `Container logs`. @@ -149,11 +149,11 @@ subgraph "2. gitlab `review-prepare` stage" end subgraph "3. gitlab `review` stage" - C["review-deploy<br><br>Helm deploys the Review App using the Cloud<br/>Native images built by the CNG-mirror pipeline.<br><br>Cloud Native images are deployed to the `review-apps`<br>Kubernetes (GKE) cluster, in the GCP `gitlab-review-apps` project."] + C["review-deploy<br><br>Helm deploys the review app using the Cloud<br/>Native images built by the CNG-mirror pipeline.<br><br>Cloud Native images are deployed to the `review-apps`<br>Kubernetes (GKE) cluster, in the GCP `gitlab-review-apps` project."] end subgraph "4. gitlab `qa` stage" - E[review-qa-smoke, review-qa-reliable<br><br>gitlab-qa runs the smoke and reliable suites against the Review App.] + E[review-qa-smoke, review-qa-reliable<br><br>gitlab-qa runs the smoke and reliable suites against the review app.] end subgraph "CNG-mirror pipeline" @@ -172,7 +172,7 @@ subgraph "CNG-mirror pipeline" job [triggers a pipeline](https://gitlab.com/gitlab-org/build/CNG-mirror/pipelines/44364657) in the [`CNG-mirror`](https://gitlab.com/gitlab-org/build/CNG-mirror) project. - The `review-build-cng` job automatically starts only if your MR includes - [CI or frontend changes](../pipelines.md#changes-patterns). In other cases, the job is manual. + [CI or frontend changes](../pipelines/internals.md#changes-patterns). In other cases, the job is manual. - The [`CNG-mirror`](https://gitlab.com/gitlab-org/build/CNG-mirror/pipelines/44364657) pipeline creates the Docker images of each component (for example, `gitlab-rails-ee`, `gitlab-shell`, `gitaly` etc.) based on the commit from the [GitLab pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/125315730) and stores @@ -180,10 +180,10 @@ subgraph "CNG-mirror pipeline" - We use the [`CNG-mirror`](https://gitlab.com/gitlab-org/build/CNG-mirror) project so that the `CNG`, (Cloud Native GitLab), project's registry is not overloaded with a lot of transient Docker images. 1. Once `review-build-cng` is done, the [`review-deploy`](https://gitlab.com/gitlab-org/gitlab/-/jobs/467724810) job - deploys the Review App using [the official GitLab Helm chart](https://gitlab.com/gitlab-org/charts/gitlab/) to + deploys the review app using [the official GitLab Helm chart](https://gitlab.com/gitlab-org/charts/gitlab/) to the [`review-apps`](https://console.cloud.google.com/kubernetes/clusters/details/us-central1-b/review-apps?project=gitlab-review-apps) Kubernetes cluster on GCP. - - The actual scripts used to deploy the Review App can be found at + - The actual scripts used to deploy the review app can be found at [`scripts/review_apps/review-apps.sh`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/scripts/review_apps/review-apps.sh). - These scripts are basically [our official Auto DevOps scripts](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml) where the @@ -192,11 +192,11 @@ subgraph "CNG-mirror pipeline" - Since we're using [the official GitLab Helm chart](https://gitlab.com/gitlab-org/charts/gitlab/), this means you get a dedicated environment for your branch that's very close to what it would look in production. - - Each review app is deployed to its own Kubernetes namespace. The namespace is based on the Review App slug that is + - Each review app is deployed to its own Kubernetes namespace. The namespace is based on the review app slug that is unique to each branch. 1. Once the [`review-deploy`](https://gitlab.com/gitlab-org/gitlab/-/jobs/467724810) job succeeds, you should be able to - use your Review App thanks to the direct link to it from the MR widget. To log - into the Review App, see "Log into my Review App?" below. + use your review app thanks to the direct link to it from the MR widget. To log + into the review app, see "Log into my review app?" below. **Additional notes:** @@ -213,23 +213,23 @@ subgraph "CNG-mirror pipeline" `#quality` channel and/or create a ~Quality ~"type::bug" issue with a link to your merge request. - The manual `review-stop` can be used to - stop a Review App manually, and is also started by GitLab once a merge + stop a review app manually, and is also started by GitLab once a merge request's branch is deleted after being merged. - The Kubernetes cluster is connected to the `gitlab` projects using the [GitLab Kubernetes integration](../../user/infrastructure/clusters/index.md). This basically - allows to have a link to the Review App directly from the merge request widget. + allows to have a link to the review app directly from the merge request widget. -### Auto-stopping of Review Apps +### Auto-stopping of review apps -Review Apps are automatically stopped 2 days after the last deployment thanks to +Review apps are automatically stopped 2 days after the last deployment thanks to the [Environment auto-stop](../../ci/environments/index.md#stop-an-environment-after-a-certain-time-period) feature. -If you need your Review App to stay up for a longer time, you can +If you need your review app to stay up for a longer time, you can [pin its environment](../../ci/environments/index.md#override-a-deployments-scheduled-stop-time) or retry the `review-deploy` job to update the "latest deployed at" time. The `review-cleanup` job that automatically runs in scheduled -pipelines stops stale Review Apps after 5 days, +pipelines stops stale review apps after 5 days, deletes their environment after 6 days, and cleans up any dangling Helm releases and Kubernetes resources after 7 days. @@ -246,13 +246,13 @@ The Helm version used is defined in the [`registry.gitlab.com/gitlab-org/gitlab-build-images:gitlab-helm3.5-kubectl1.17` image](https://gitlab.com/gitlab-org/gitlab-build-images/-/blob/master/Dockerfile.gitlab-helm3.5-kubectl1.17#L6) used by the `review-deploy` and `review-stop` jobs. -## Diagnosing unhealthy Review App releases +## Diagnosing unhealthy review app releases -If [Review App Stability](https://app.periscopedata.com/app/gitlab/496118/Engineering-Productivity-Sandbox?widget=6690556&udv=785399) +If [review app stability](https://app.periscopedata.com/app/gitlab/496118/Engineering-Productivity-Sandbox?widget=6690556&udv=785399) dips this may be a signal that the `review-apps` cluster is unhealthy. -Leading indicators may be health check failures leading to restarts or majority failure for Review App deployments. +Leading indicators may be health check failures leading to restarts or majority failure for review app deployments. -The [Review Apps Overview dashboard](https://console.cloud.google.com/monitoring/classic/dashboards/6798952013815386466?project=gitlab-review-apps&timeDomain=1d) +The [review apps Overview dashboard](https://console.cloud.google.com/monitoring/classic/dashboards/6798952013815386466?project=gitlab-review-apps&timeDomain=1d) aids in identifying load spikes on the cluster, and if nodes are problematic or the entire cluster is trending towards unhealthy. See the [review apps page of the Engineering Productivity Runbook](https://gitlab.com/gitlab-org/quality/engineering-productivity/team/-/blob/main/runbook/review-apps.md) for troubleshooting review app releases. @@ -273,7 +273,7 @@ find a way to limit it to only us.** ## Other resources -- [Review Apps integration for CE/EE (presentation)](https://docs.google.com/presentation/d/1QPLr6FO4LduROU8pQIPkX1yfGvD13GEJIBOenqoKxR8/edit?usp=sharing) +- [Review apps integration for CE/EE (presentation)](https://docs.google.com/presentation/d/1QPLr6FO4LduROU8pQIPkX1yfGvD13GEJIBOenqoKxR8/edit?usp=sharing) - [Stability issues](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/212) ### Helpful command line tools diff --git a/doc/development/utilities.md b/doc/development/utilities.md index 45a6b74f33a..551834670b3 100644 --- a/doc/development/utilities.md +++ b/doc/development/utilities.md @@ -199,10 +199,26 @@ Refer to [`strong_memoize.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/maste end strong_memoize_attr :result - strong_memoize_attr :enabled?, :enabled def enabled? Feature.enabled?(:some_feature) end + strong_memoize_attr :enabled?, :enabled + end + ``` + + There's also `strong_memoize_with` to help memoize methods that take arguments. + This should be used for methods that have a low number of possible values + as arguments or with consistent repeating arguments in a loop. + + ```ruby + class Find + include Gitlab::Utils::StrongMemoize + + def result(basic: true) + strong_memoize_with(:result, basic) do + search(basic) + end + end end ``` diff --git a/doc/development/windows.md b/doc/development/windows.md index 5f32848da79..bf56e16344a 100644 --- a/doc/development/windows.md +++ b/doc/development/windows.md @@ -132,7 +132,7 @@ PowerShell has aliases for all of the following commands so you don't have to le - `ls` ---> `dir` - `rm` ---> `del` - `rm -rf nonemptydir` ---> `rmdir /S nonemptydir` -- `/` ---> `\` (path separator) +- `/` ---> <code>\</code> (path separator) - `cat` ---> `type` - `mv` ---> `move` - Redirection works the same (for example, `>` and `2>&1`) diff --git a/doc/development/work_items.md b/doc/development/work_items.md index eabad175bf7..a417e1d1349 100644 --- a/doc/development/work_items.md +++ b/doc/development/work_items.md @@ -55,6 +55,7 @@ To avoid confusion and ensure communication is efficient, we will use the follow | legacy issue view | The existing view used to render issues and incidents | | | | issue | The existing issue model | | | | issuable | Any model currently using the issueable module (issues, epics and MRs) | _Incidents are an **issuable**_ | _Incidents are a **work item type**_ | +| widget | A UI element to present or allow interaction with specific work item data | | | Some terms have been used in the past but have since become confusing and are now discouraged. @@ -138,6 +139,20 @@ To introduce a new WIT there are two options: ### Work item type widgets +A widget is a single component that can exist on a work item. This component can be used on one or +many work item types and can be lightly customized at the point of implementation. + +A widget contains both the frontend UI (if present) and the associated logic for presenting and +managing any data used by the widget. There can be a one-to-many connection between the data model +and widgets. It means there can be multiple widgets that use or manage the same data, and they could +be present at the same time (for example, a read-only summary widget and an editable detail widget, +or two widgets showing two different filtered views of the same model). + +Widgets should be differentiated by their **purpose**. When possible, this purpose should be +abstracted to the highest reasonable level to maximize reusability. For example, the widget for +managing "tasks" was built as "child items". Rather than managing one type of child, it's abstracted +up to managing any children. + All WITs will share the same pool of predefined widgets and will be customized by which widgets are active on a specific WIT. Every attribute (column or association) will become a widget with self-encapsulated functionality regardless of the WIT it belongs to. @@ -164,13 +179,17 @@ define these various types in a very flexible manner. Having GitLab use this system first (without introducing customer customization) allows us to better build out the initial system. -NOTE: -Currently work item's `base_type` is used to define static mapping of what +Work item's `base_type` is used to define static mapping of what widgets are available for each type (current status), this definition should be -rather stored in database table. The exact structure of the WIT widgets -metadata is still to be defined. `base_type` was added to help converting other -types of resources (requirements and incidents) into work items. Eventually (when -these resources become regular work items), `base_type` will be removed. +rather stored in a database table. The exact structure of the WIT widgets metadata +is [still to be defined](https://gitlab.com/gitlab-org/gitlab/-/issues/370599). +`base_type` was added to help convert other types of resources (requirements +and incidents) into work items. Eventually (when these resources become regular +work items), `base_type` will be removed. + +Until the architecture of WIT widgets is finalized, we are holding off on the creation of new work item +types. If a new work item type is absolutely necessary, please reach out to a +member of the [Project Management Engineering Team](https://gitlab.com/gitlab-org/gitlab/-/issues/370599). ### Custom work item types diff --git a/doc/development/workhorse/index.md b/doc/development/workhorse/index.md index 0f4e55a002a..91795336f78 100644 --- a/doc/development/workhorse/index.md +++ b/doc/development/workhorse/index.md @@ -21,7 +21,7 @@ but that repository is no longer used for development. ## Install Workhorse -To install GitLab Workhorse you need [Go 1.15 or newer](https://go.dev/dl) and +To install GitLab Workhorse you need [Go 1.18 or newer](https://go.dev/dl) and [GNU Make](https://www.gnu.org/software/make/). To install into `/usr/local/bin` run `make install`. |
