diff options
Diffstat (limited to 'doc/development')
183 files changed, 1590 insertions, 1232 deletions
diff --git a/doc/development/adding_database_indexes.md b/doc/development/adding_database_indexes.md deleted file mode 100644 index 7ab846cce3e..00000000000 --- a/doc/development/adding_database_indexes.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'database/adding_database_indexes.md' -remove_date: '2022-11-05' ---- - -This document was moved to [another location](database/adding_database_indexes.md). - -<!-- This redirect file can be deleted after <2022-11-05>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/api_graphql_styleguide.md b/doc/development/api_graphql_styleguide.md index c19341a1404..e0db0d7e34d 100644 --- a/doc/development/api_graphql_styleguide.md +++ b/doc/development/api_graphql_styleguide.md @@ -2115,7 +2115,7 @@ end Authenticating a user with the `current_user:` argument for `post_graphql` generates more queries on the first request than on subsequent requests on that same user. If you are testing for N+1 queries using - [QueryRecorder](query_recorder.md), use a **different** user for each request. + [QueryRecorder](database/query_recorder.md), use a **different** user for each request. The below example shows how a test for avoiding N+1 queries should look: @@ -2223,7 +2223,7 @@ end [Adding field with resolver on a Type causes "Can't determine the return type " error on a different Type](https://github.com/rmosolgo/graphql-ruby/issues/3974). Unfortunately, the errors generated don't really indicate what the problem is. For example, - remove the quotes from the `Rspec.descrbe` in + remove the quotes from the `Rspec.describe` in [ee/spec/graphql/resolvers/compliance_management/merge_requests/compliance_violation_resolver_spec.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/spec/graphql/resolvers/compliance_management/merge_requests/compliance_violation_resolver_spec.rb). Then run `rspec ee/spec/graphql/resolvers/compliance_management/merge_requests/compliance_violation_resolver_spec.rb`. @@ -2263,7 +2263,7 @@ end 1. 95% of the resolver specs use arguments that are Ruby objects, as opposed to when using the GraphQL API only strings and integers are used. This works fine in most cases. - 1. If your resolver takes arguments that use a `prepare` proc, such as a resolver that accepts timeframe + 1. If your resolver takes arguments that use a `prepare` proc, such as a resolver that accepts time frame arguments (`TimeFrameArguments`), you must pass the `arg_style: :internal_prepared` parameter into the `resolve` method. This tells the code to convert the arguments into strings and integers and pass them through regular argument handling, ensuring that the `prepare` proc is called correctly. diff --git a/doc/development/api_styleguide.md b/doc/development/api_styleguide.md index f74ed8db50f..006d0a01abb 100644 --- a/doc/development/api_styleguide.md +++ b/doc/development/api_styleguide.md @@ -152,6 +152,34 @@ For non-200 HTTP responses, use the provided helpers in `lib/api/helpers.rb` to For `DELETE` requests, you should also generally use the `destroy_conditionally!` helper which by default returns a `204 No Content` response on success, or a `412 Precondition Failed` response if the given `If-Unmodified-Since` header is out of range. This helper calls `#destroy` on the passed resource, but you can also implement a custom deletion method by passing a block. +## Choosing HTTP verbs + +When defining a new [API route](https://github.com/ruby-grape/grape#routes), use +the correct [HTTP request method](https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods). + +### Deciding between `PATCH` and `PUT` + +In a Rails application, both the `PATCH` and `PUT` request methods are routed to +the `update` method in controllers. With Grape, the framework we use to write +the GitLab API, you must explicitly set the `PATCH` or `PUT` HTTP verb for an +endpoint that does updates. + +If the endpoint updates *all* attributes of a given resource, use the +[`PUT`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/PUT) request +method. If the endpoint updates *some* attributes of a given resource, use the +[`PATCH`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/PATCH) +request method. + +Here is a good example for `PATCH`: [`PATCH /projects/:id/protected_branches/:name`](../api/protected_branches.md#update-a-protected-branch) +Here is a good example for `PUT`: [`PUT /projects/:id/merge_requests/:merge_request_iid/approve`](../api/merge_request_approvals.md#approve-merge-request) + +Often, a good `PUT` endpoint only has ids and a verb (in the example above, "approve"). +Or, they only have a single value and represent a key/value pair. + +The [Rails blog](https://rubyonrails.org/2012/2/26/edge-rails-patch-is-the-new-primary-http-method-for-updates) +has a detailed explanation of why `PATCH` is usually the most apt verb for web +API endpoints that perform an update. + ## Using API path helpers in GitLab Rails codebase Because we support [installing GitLab under a relative URL](../install/relative_url.md), one must take this @@ -271,7 +299,7 @@ which introduced the scope. When an API endpoint returns collections, always add a test to verify that the API endpoint does not have an N+1 problem, now and in the future. -We can do this using [`ActiveRecord::QueryRecorder`](query_recorder.md). +We can do this using [`ActiveRecord::QueryRecorder`](database/query_recorder.md). Example: diff --git a/doc/development/application_limits.md b/doc/development/application_limits.md index b64d25ccf64..b1efc11db62 100644 --- a/doc/development/application_limits.md +++ b/doc/development/application_limits.md @@ -38,7 +38,7 @@ It's recommended to create two separate migration script files. desired limit using `create_or_update_plan_limit` migration helper, such as: ```ruby - class InsertProjectHooksPlanLimits < Gitlab::Database::Migration[2.0] + class InsertProjectHooksPlanLimits < Gitlab::Database::Migration[2.1] def up create_or_update_plan_limit('project_hooks', 'default', 0) create_or_update_plan_limit('project_hooks', 'free', 10) diff --git a/doc/development/application_slis/rails_request_apdex.md b/doc/development/application_slis/rails_request_apdex.md index 2fa9f5f4869..8fcd725f74d 100644 --- a/doc/development/application_slis/rails_request_apdex.md +++ b/doc/development/application_slis/rails_request_apdex.md @@ -108,7 +108,7 @@ a case-by-case basis. Take the following into account: should try to keep as short as possible. 1. Traffic characteristics should also be taken into account. If the - traffic to the endpoint is bursty, like CI traffic spinning up a + traffic to the endpoint sometimes bursts, like CI traffic spinning up a big batch of jobs hitting the same endpoint, then having these endpoints take five seconds is unacceptable from an infrastructure point of view. We cannot scale up the fleet fast enough to accommodate for diff --git a/doc/development/approval_rules.md b/doc/development/approval_rules.md index ce774b1e8f9..312bf2b1bb7 100644 --- a/doc/development/approval_rules.md +++ b/doc/development/approval_rules.md @@ -1,6 +1,6 @@ --- stage: Create -group: Source Code +group: Code Review info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -221,7 +221,7 @@ It is responsible for parsing `approval_rules_attributes` parameter to: - Filter the group IDs whether they are visible to user. - Identify the `any_approver` rule. - Append hidden groups to it when specified. -- Append user defined inapplicable (rules that does not apply to MR's target +- Append user defined inapplicable (rules that do not apply to the merge request's target branch) approval rules. ## Flow diff --git a/doc/development/architecture.md b/doc/development/architecture.md index a731e661a80..5eb1dcc3208 100644 --- a/doc/development/architecture.md +++ b/doc/development/architecture.md @@ -592,8 +592,6 @@ Grafana is an open source, feature rich metrics dashboard and graph editor for G Jaeger, inspired by Dapper and OpenZipkin, is a distributed tracing system. It can be used for monitoring microservices-based distributed systems. -For monitoring deployed apps, see [Jaeger tracing documentation](../operations/tracing.md) - #### Logrotate - [Project page](https://github.com/logrotate/logrotate/blob/master/README.md) @@ -626,7 +624,7 @@ Mattermost is an open source, private cloud, Slack-alternative from <https://mat - Layer: Core Service (Data) - GitLab.com: [Storage Architecture](https://about.gitlab.com/handbook/engineering/infrastructure/production/architecture/#storage-architecture) -MinIO is an object storage server released under the GNU AGPL v3.0. It is compatible with Amazon S3 cloud storage service. It is best suited for storing unstructured data such as photos, videos, log files, backups, and container / VM images. Size of an object can range from a few KBs to a maximum of 5TB. +MinIO is an object storage server released under the GNU AGPL v3.0. It is compatible with Amazon S3 cloud storage service. It is best suited for storing unstructured data such as photos, videos, log files, backups, and container / VM images. Size of an object can range from a few KB to a maximum of 5 TB. #### NGINX diff --git a/doc/development/caching.md b/doc/development/caching.md index 36fbfc7010e..58ec7a77591 100644 --- a/doc/development/caching.md +++ b/doc/development/caching.md @@ -80,7 +80,7 @@ indicates we have plenty of headroom. - Generic data can be cached for everyone. - You must keep this in mind when building new features. 1. Try to preserve cache data as much as possible: - - Use nested caches to maintain as much cached data as possible across expiries. + - Use nested caches to maintain as much cached data as possible across expires. 1. Perform as few requests to the cache as possible: - This reduces variable latency caused by network issues. - Lower overhead for each read on the cache. @@ -166,7 +166,7 @@ Is the cache being added "worthy"? This can be hard to measure, but you can cons - Calling the same method multiple times but only calculating the value once. - Stored in Ruby memory. - `@article ||= Article.find(params[:id])` - - `strong_memoize { Article.find(params[:id]) }` + - `strong_memoize_attr :method_name` 1. Request caching: - Return the same value for a key for the duration of a web request. - `Gitlab::SafeRequestStore.fetch` @@ -252,7 +252,7 @@ All the time! ### When to use method caching -- Using instance variables, or [strong_memoize](utilities.md#strongmemoize) is something we all tend to do anyway. +- Use instance variables, or [`StrongMemoize`](utilities.md#strongmemoize). - Useful when the same value is needed multiple times in a request. - Can be used to prevent multiple cache calls for the same key. - Can cause issues with ActiveRecord objects where a value doesn't change until you call diff --git a/doc/development/cascading_settings.md b/doc/development/cascading_settings.md index 1a0f0ec5b5f..389623e68d8 100644 --- a/doc/development/cascading_settings.md +++ b/doc/development/cascading_settings.md @@ -38,7 +38,7 @@ Settings are not cascading by default. To define a cascading setting, take the f `application_settings`. ```ruby - class AddDelayedProjectRemovalCascadingSetting < Gitlab::Database::Migration[2.0] + class AddDelayedProjectRemovalCascadingSetting < Gitlab::Database::Migration[2.1] include Gitlab::Database::MigrationHelpers::CascadingNamespaceSettings enable_lock_retries! @@ -153,7 +153,7 @@ Renders the label for a checkbox setting. [`_setting_label_fieldset.html.haml`](https://gitlab.com/gitlab-org/gitlab/-/blob/c2736823b8e922e26fd35df4f0cd77019243c858/app/views/shared/namespaces/cascading_settings/_setting_label_fieldset.html.haml) -Renders the label for a fieldset setting. +Renders the label for a `fieldset` setting. | Local | Description | Type | Required (default value) | |:-----------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------|:-------------------------| diff --git a/doc/development/cicd/pipeline_wizard.md b/doc/development/cicd/pipeline_wizard.md index eb6d1c60bb1..213a9fe1762 100644 --- a/doc/development/cicd/pipeline_wizard.md +++ b/doc/development/cicd/pipeline_wizard.md @@ -127,7 +127,7 @@ is planned to add the ability to create a MR from here. ### Props -- `template` (required): The template content as an unparsed String. See +- `template` (required): The template content as an un-parsed string. See [Template file location](#template-file-location) for more information. - `project-path` (required): The full path of the project the final file should be committed to diff --git a/doc/development/code_review.md b/doc/development/code_review.md index 90f33319365..30d9d671038 100644 --- a/doc/development/code_review.md +++ b/doc/development/code_review.md @@ -28,7 +28,7 @@ The reviewer can: - Give you a second opinion on the chosen solution and implementation. - Help look for bugs, logic problems, or uncovered edge cases. -If the merge request is trivial (for example, fixing a typo or a tiny refactor that doesn't change the behavior or any data), +If the merge request is trivial to review (for example, fixing a typo or a tiny refactor that doesn't change the behavior or any data), you can skip the reviewer step and directly ask a [maintainer](https://about.gitlab.com/handbook/engineering/workflow/code-review/#maintainer). Otherwise, a merge request should always be first reviewed by a reviewer in each [category (e.g. backend, database)](#approval-guidelines) @@ -124,8 +124,10 @@ page, with these behaviors: branch name (unless their out-of-office (`OOO`) status changes, as in point 1). It removes leading `ce-` and `ee-`, and trailing `-ce` and `-ee`, so that it can be stable for backport branches. +- People whose Slack or [GitLab status](../user/profile/index.md#set-your-current-status) emoji + is Ⓜ `:m:`are only suggested as reviewers on projects they are a maintainer of. -The [Roulette dashboard](https://gitlab-org.gitlab.io/gitlab-roulette) contains: +The [Roulette dashboard](https://gitlab-org.gitlab.io/gitlab-roulette/) contains: - Assignment events in the last 7 and 30 days. - Currently assigned merge requests per person. @@ -155,7 +157,7 @@ with [domain expertise](#domain-experts). | `~frontend` changes (*1*) | [Frontend maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_maintainers_frontend). | | `~UX` user-facing changes (*3*) | [Product Designer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_reviewers_UX). Refer to the [design and user interface guidelines](contributing/design.md) for details. | | Adding a new JavaScript library (*1*) | - [Frontend foundations member](https://about.gitlab.com/direction/manage/foundations/) if the library significantly increases the [bundle size](https://gitlab.com/gitlab-org/frontend/playground/webpack-memory-metrics/-/blob/master/doc/report.md).<br/>- A [legal department member](https://about.gitlab.com/handbook/legal/) if the license used by the new library hasn't been approved for use in GitLab.<br/><br/>More information about license compatibility can be found in our [GitLab Licensing and Compatibility documentation](licensing.md). | -| A new dependency or a file system change | - [Distribution team member](https://about.gitlab.com/company/team/). See how to work with the [Distribution team](https://about.gitlab.com/handbook/engineering/development/enablement/systems/distribution/#how-to-work-with-distribution) for more details.<br/>- For Rubygems, request an [AppSec review](gemfile.md#request-an-appsec-review). | +| A new dependency or a file system change | - [Distribution team member](https://about.gitlab.com/company/team/). See how to work with the [Distribution team](https://about.gitlab.com/handbook/engineering/development/enablement/systems/distribution/#how-to-work-with-distribution) for more details.<br/>- For RubyGems, request an [AppSec review](gemfile.md#request-an-appsec-review). | | `~documentation` or `~UI text` changes | [Technical writer](https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments) based on assignments in the appropriate [DevOps stage group](https://about.gitlab.com/handbook/product/categories/#devops-stages). | | Changes to development guidelines | Follow the [review process](development_processes.md#development-guidelines-review) and get the approvals accordingly. | | End-to-end **and** non-end-to-end changes (*4*) | [Software Engineer in Test](https://about.gitlab.com/handbook/engineering/quality/#individual-contributors). | @@ -185,49 +187,52 @@ Using checklists improves quality in software engineering. This checklist is a s See the [test engineering process](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/test-engineering/) for further quality guidelines. -1. I have self-reviewed this MR per [code review guidelines](code_review.md). -1. For the code that this change impacts, I believe that the automated tests ([Testing Guide](testing_guide/index.md)) validate functionality that is highly important to users (including consideration of [all test levels](testing_guide/testing_levels.md)). -1. If the existing automated tests do not cover the above functionality, I have added the necessary additional tests or added an issue to describe the automation testing gap and linked it to this MR. -1. I have considered the technical aspects of this change's impact on GitLab.com hosted customers and self-managed customers. -1. I have considered the impact of this change on the frontend, backend, and database portions of the system where appropriate and applied the `~ux`, `~frontend`, `~backend`, and `~database` labels accordingly. -1. I have tested this MR in [all supported browsers](../install/requirements.md#supported-web-browsers), or determined that this testing is not needed. -1. I have confirmed that this change is [backwards compatible across updates](multi_version_compatibility.md), or I have decided that this does not apply. -1. I have properly separated EE content from FOSS, or this MR is FOSS only. +1. You have self-reviewed this MR per [code review guidelines](code_review.md). +1. For the code that this change impacts, you believe that the automated tests ([Testing Guide](testing_guide/index.md)) validate functionality that is highly important to users (including consideration of [all test levels](testing_guide/testing_levels.md)). +1. If the existing automated tests do not cover the above functionality, you have added the necessary additional tests or added an issue to describe the automation testing gap and linked it to this MR. +1. You have considered the technical aspects of this change's impact on GitLab.com hosted customers and self-managed customers. +1. You have considered the impact of this change on the frontend, backend, and database portions of the system where appropriate and applied the `~ux`, `~frontend`, `~backend`, and `~database` labels accordingly. +1. You have tested this MR in [all supported browsers](../install/requirements.md#supported-web-browsers), or determined that this testing is not needed. +1. You have confirmed that this change is [backwards compatible across updates](multi_version_compatibility.md), or you have decided that this does not apply. +1. You have properly separated EE content from FOSS, or this MR is FOSS only. - [Where should EE code go?](ee_features.md) -1. I have considered that existing data may be surprisingly varied. For example, a new model validation can break existing records. Consider making validation on existing data optional rather than required if you haven't confirmed that existing data will pass validation. +1. You have considered that existing data may be surprisingly varied. For example, a new model validation can break existing records. Consider making validation on existing data optional rather than required if you haven't confirmed that existing data will pass validation. ##### Performance, reliability, and availability -1. I am confident that this MR does not harm performance, or I have asked a reviewer to help assess the performance impact. ([Merge request performance guidelines](merge_request_performance_guidelines.md)) -1. I have added [information for database reviewers in the MR description](database_review.md#required), or I have decided that it is unnecessary. +1. You are confident that this MR does not harm performance, or you have asked a reviewer to help assess the performance impact. ([Merge request performance guidelines](merge_request_performance_guidelines.md)) +1. You have added [information for database reviewers in the MR description](database_review.md#required), or you have decided that it is unnecessary. - [Does this MR have database-related changes?](database_review.md) -1. I have considered the availability and reliability risks of this change. -1. I have considered the scalability risk based on future predicted growth. -1. I have considered the performance, reliability, and availability impacts of this change on large customers who may have significantly more data than the average customer. +1. You have considered the availability and reliability risks of this change. +1. You have considered the scalability risk based on future predicted growth. +1. You have considered the performance, reliability, and availability impacts of this change on large customers who may have significantly more data than the average customer. ##### Observability instrumentation -1. I have included enough instrumentation to facilitate debugging and proactive performance improvements through observability. +1. You have included enough instrumentation to facilitate debugging and proactive performance improvements through observability. See [example](https://gitlab.com/gitlab-org/gitlab/-/issues/346124#expectations) of adding feature flags, logging, and instrumentation. ##### Documentation -1. I have included changelog trailers, or I have decided that they are not needed. +1. You have included changelog trailers, or you have decided that they are not needed. - [Does this MR need a changelog?](changelog.md#what-warrants-a-changelog-entry) -1. I have added/updated documentation or decided that documentation changes are unnecessary for this MR. +1. You have added/updated documentation or decided that documentation changes are unnecessary for this MR. - [Is documentation required?](https://about.gitlab.com/handbook/product/ux/technical-writing/workflow/#when-documentation-is-required) ##### Security -1. I have confirmed that if this MR contains changes to processing or storing of credentials or tokens, authorization, and authentication methods, or other items described in [the security review guidelines](https://about.gitlab.com/handbook/security/#when-to-request-a-security-review), I have added the `~security` label and I have `@`-mentioned `@gitlab-com/gl-security/appsec`. -1. I have reviewed the documentation regarding [internal application security reviews](https://about.gitlab.com/handbook/security/#internal-application-security-reviews) for **when** and **how** to request a security review and requested a security review if this is warranted for this change. +1. You have confirmed that if this MR contains changes to processing or storing of credentials or tokens, authorization, and authentication methods, or other items described in [the security review guidelines](https://about.gitlab.com/handbook/security/#when-to-request-a-security-review), you have added the `~security` label and you have `@`-mentioned `@gitlab-com/gl-security/appsec`. +1. You have reviewed the documentation regarding [internal application security reviews](https://about.gitlab.com/handbook/security/#internal-application-security-reviews) for **when** and **how** to request a security review and requested a security review if this is warranted for this change. +1. If there are security scan results that are blocking the MR (due to the [scan result policies](https://gitlab.com/gitlab-com/gl-security/security-policies)): + - For true positive findings, they should be corrected before the merge request is merged. This will remove the AppSec approval required by the scan result policy. + - For false positive findings, something that should be discussed for risk acceptance, or anything questionable, please ping `@gitlab-com/gl-security/appsec`. ##### Deployment -1. I have considered using a feature flag for this change because the change may be high risk. -1. If I am using a feature flag, I plan to test the change in staging before I test it in production, and I have considered rolling it out to a subset of production customers before rolling it out to all customers. +1. You have considered using a feature flag for this change because the change may be high risk. +1. If you are using a feature flag, you plan to test the change in staging before you test it in production, and you have considered rolling it out to a subset of production customers before rolling it out to all customers. - [When to use a feature flag](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#when-to-use-feature-flags) -1. I have informed the Infrastructure department of a default setting or new setting change per [definition of done](contributing/merge_request_workflow.md#definition-of-done), or decided that this is unnecessary. +1. You have informed the Infrastructure department of a default setting or new setting change per [definition of done](contributing/merge_request_workflow.md#definition-of-done), or decided that this is unnecessary. ### The responsibility of the merge request author diff --git a/doc/development/contributing/design.md b/doc/development/contributing/design.md index e6b6b56cf73..b8d7a8eef39 100644 --- a/doc/development/contributing/design.md +++ b/doc/development/contributing/design.md @@ -24,8 +24,17 @@ screenshots (or videos) of your changes in the description, as explained in our [MR workflow](merge_request_workflow.md). These screenshots/videos are very helpful for all reviewers and can speed up the review process, especially if the changes are small. -- Attach the ~UX label to any merge request that impacts the user experience. This will enable Product Designers to [review](https://about.gitlab.com/handbook/product/ux/product-designer/mr-reviews/#stage-group-mrs/) any user facing changes. -- Assign the Product Designer suggested by Reviewer Roulette as the reviewer of your merge request. The reviewer does not have to be the domain expert unless this is a community contribution. +- Attach the ~UX label to any merge request that has any user facing changes. This will trigger our +Reviewer Roulette to suggest a UX [reviewer](https://about.gitlab.com/handbook/product/ux/product-designer/mr-reviews/#stage-group-mrs). + +If you are a **team member**: We recommend assigning the Product Designer suggested by the +[Reviewer Roulette](../code_review.md#reviewer-roulette) as reviewer. [This helps us](https://about.gitlab.com/handbook/product/ux/product-designer/mr-reviews/#benefits) spread work evenly, improve communication, and make our UI more +consistent. If you have a reason to choose a different reviewer, add a comment to mention you assigned +it to a Product Designer of your choice. + +If you are a **community contributor**: We favor choosing the Product Designer that is a +[domain expert](../code_review.md#domain-experts) in the area you are contributing, to regardless +of the Reviewer Roulette. ## Checklist diff --git a/doc/development/contributing/index.md b/doc/development/contributing/index.md index 8c0d18f877b..1a5b801a95a 100644 --- a/doc/development/contributing/index.md +++ b/doc/development/contributing/index.md @@ -100,7 +100,7 @@ If you have any questions or need help, visit [Getting Help](https://about.gitla communicate with the GitLab community. GitLab prefers [asynchronous communication](https://about.gitlab.com/handbook/communication/#internal-communication) over real-time communication. We do encourage you to connect and hang out with us. GitLab has a Gitter room dedicated for [contributors](https://gitter.im/gitlab/contributors), which is bridged with our -internal Slack. We actively monitor this channel. There is also a community-run [Discord server](http://discord.gg/gitlab) where you can +internal Slack. We actively monitor this channel. There is also a community-run [Discord server](https://discord.gg/gitlab) where you can find other contributors in the `#contributors` channel. Thanks for your contribution! @@ -229,4 +229,4 @@ For information on how to contribute documentation, see GitLab ## Getting an Enterprise Edition License If you need a license for contributing to an EE-feature, see -[relevant information](https://about.gitlab.com/handbook/marketing/community-relations/code-contributor-program/#contributing-to-the-gitlab-enterprise-edition-ee). +[relevant information](https://about.gitlab.com/handbook/marketing/community-relations/code-contributor-program/operations/#contributing-to-the-gitlab-enterprise-edition-ee). diff --git a/doc/development/contributing/merge_request_workflow.md b/doc/development/contributing/merge_request_workflow.md index f2c06e289c9..f06e8825660 100644 --- a/doc/development/contributing/merge_request_workflow.md +++ b/doc/development/contributing/merge_request_workflow.md @@ -213,6 +213,7 @@ To reach the definition of done, the merge request must create no regressions an - Verified as working in production on GitLab.com. - Verified as working for self-managed instances. +- Verified as supporting [Geo](../../administration/geo/index.md) via the [self-service framework](../geo/framework.md). To learn more see [here](../geo/framework.md#geo-is-a-requirement-in-the-definition-of-done). If a regression occurs, we prefer you revert the change. Your contribution is *incomplete* until you have made sure it meets all of these @@ -313,7 +314,7 @@ The following items are checked after the merge request has been merged: 1. Confirmed to be working in staging before implementing the change in production, where possible. 1. Confirmed to be working in the production with no new [Sentry](https://about.gitlab.com/handbook/engineering/monitoring/#sentry) errors after the contribution is deployed. 1. Confirmed that the [rollout plan](https://about.gitlab.com/handbook/engineering/development/processes/rollout-plans/) has been completed. -1. If there is a performance risk in the change, I have analyzed the performance of the system before and after the change. +1. If there is a performance risk in the change, you have analyzed the performance of the system before and after the change. 1. *If the merge request uses feature flags, per-project or per-group enablement, and a staged rollout:* - Confirmed to be working on GitLab projects. - Confirmed to be working at each stage for all projects added. diff --git a/doc/development/contributing/style_guides.md b/doc/development/contributing/style_guides.md index fe1aa8449c2..0a030d1f3bc 100644 --- a/doc/development/contributing/style_guides.md +++ b/doc/development/contributing/style_guides.md @@ -47,7 +47,7 @@ We were using Overcommit prior to Lefthook, so you may want to uninstall it firs bundle exec lefthook run pre-push ``` -This should return the lefthook version and the list of executable commands with output. +This should return the Lefthook version and the list of executable commands with output. ### Lefthook configuration diff --git a/doc/development/creating_enums.md b/doc/development/creating_enums.md deleted file mode 100644 index d3892c4c44e..00000000000 --- a/doc/development/creating_enums.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'database/creating_enums.md' -remove_date: '2022-11-06' ---- - -This document was moved to [another location](database/creating_enums.md). - -<!-- This redirect file can be deleted after <2022-11-06>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/dangerbot.md b/doc/development/dangerbot.md index 52503f2d9c8..b568809ea4e 100644 --- a/doc/development/dangerbot.md +++ b/doc/development/dangerbot.md @@ -132,7 +132,7 @@ have had the chance to add to `helper.labels_to_add`. #### Shared rules and plugins If the rule or plugin you implement can be useful for other projects, think about -upstreaming them to the [`gitlab-dangerfiles`](https://gitlab.com/gitlab-org/ruby/gems/gitlab-dangerfiles) project. +adding them upstream to the [`gitlab-dangerfiles`](https://gitlab.com/gitlab-org/ruby/gems/gitlab-dangerfiles) project. #### Enable Danger on a project diff --git a/doc/development/database/add_foreign_key_to_existing_column.md b/doc/development/database/add_foreign_key_to_existing_column.md index 1609e00531e..07fa8133496 100644 --- a/doc/development/database/add_foreign_key_to_existing_column.md +++ b/doc/development/database/add_foreign_key_to_existing_column.md @@ -69,7 +69,7 @@ In the example above, you'd be still able to update records in the `emails` tabl Migration file for adding `NOT VALID` foreign key: ```ruby -class AddNotValidForeignKeyToEmailsUser < Gitlab::Database::Migration[2.0] +class AddNotValidForeignKeyToEmailsUser < Gitlab::Database::Migration[2.1] def up add_concurrent_foreign_key :emails, :users, column: :user_id, on_delete: :cascade, validate: false end @@ -100,7 +100,7 @@ In case the data volume is higher (>1000 records), it's better to create a backg Example for cleaning up records in the `emails` table in a database migration: ```ruby -class RemoveRecordsWithoutUserFromEmailsTable < Gitlab::Database::Migration[2.0] +class RemoveRecordsWithoutUserFromEmailsTable < Gitlab::Database::Migration[2.1] disable_ddl_transaction! class Email < ActiveRecord::Base @@ -133,7 +133,7 @@ Migration file for validating the foreign key: ```ruby # frozen_string_literal: true -class ValidateForeignKeyOnEmailUsers < Gitlab::Database::Migration[2.0] +class ValidateForeignKeyOnEmailUsers < Gitlab::Database::Migration[2.1] def up validate_foreign_key :emails, :user_id end diff --git a/doc/development/database/adding_database_indexes.md b/doc/development/database/adding_database_indexes.md index d4cd807ef22..6a401c804f5 100644 --- a/doc/development/database/adding_database_indexes.md +++ b/doc/development/database/adding_database_indexes.md @@ -107,11 +107,10 @@ determining whether existing indexes are still required. More information on the meaning of the various columns can be found at <https://www.postgresql.org/docs/current/monitoring-stats.html>. -To determine if an index is still being used on production, use the following -Thanos query with your index name: +To determine if an index is still being used on production, use [Thanos](https://thanos-query.ops.gitlab.net/graph?g0.expr=sum%20by%20(type)(rate(pg_stat_user_indexes_idx_scan%7Benv%3D%22gprd%22%2C%20indexrelname%3D%22INSERT%20INDEX%20NAME%20HERE%22%7D%5B30d%5D))&g0.tab=1&g0.stacked=0&g0.range_input=1h&g0.max_source_resolution=0s&g0.deduplicate=1&g0.partial_response=0&g0.store_matches=%5B%5D): ```sql -sum(rate(pg_stat_user_indexes_idx_tup_read{env="gprd", indexrelname="index_ci_name", type="patroni-ci"}[5m])) +sum by (type)(rate(pg_stat_user_indexes_idx_scan{env="gprd", indexrelname="INSERT INDEX NAME HERE"}[30d])) ``` Because the query output relies on the actual usage of your database, it @@ -232,7 +231,7 @@ A Rails migration example: ```ruby # in db/post_migrate/ -class AddIndexToPartitionedTable < Gitlab::Database::Migration[2.0] +class AddIndexToPartitionedTable < Gitlab::Database::Migration[2.1] include Gitlab::Database::PartitioningMigrationHelpers disable_ddl_transaction! @@ -355,7 +354,7 @@ Use the asynchronous index helpers on your local environment to test changes for For very large tables, index destruction can be a challenge to manage. While `remove_concurrent_index` removes indexes in a way that does not block normal traffic, it can still be problematic if index destruction runs for -during autovacuum. Necessary database operations like `autovacuum` cannot run, and +during `autovacuum`. Necessary database operations like `autovacuum` cannot run, and the deployment process on GitLab.com is blocked while waiting for index destruction to finish. diff --git a/doc/development/database/avoiding_downtime_in_migrations.md b/doc/development/database/avoiding_downtime_in_migrations.md index 57f5a66a9ee..b34c0bbf728 100644 --- a/doc/development/database/avoiding_downtime_in_migrations.md +++ b/doc/development/database/avoiding_downtime_in_migrations.md @@ -82,7 +82,7 @@ to write a migration that removes a column: In this case, a **transactional migration** can be used. Something as simple as: ```ruby -class RemoveUsersUpdatedAtColumn < Gitlab::Database::Migration[2.0] +class RemoveUsersUpdatedAtColumn < Gitlab::Database::Migration[2.1] def up remove_column :users, :updated_at end @@ -103,7 +103,7 @@ If the `down` method requires adding back any dropped indexes or constraints, th be done within a transactional migration, then the migration would look like this: ```ruby -class RemoveUsersUpdatedAtColumn < Gitlab::Database::Migration[2.0] +class RemoveUsersUpdatedAtColumn < Gitlab::Database::Migration[2.1] disable_ddl_transaction! def up @@ -158,7 +158,7 @@ renaming. For example ```ruby # A regular migration in db/migrate -class RenameUsersUpdatedAtToUpdatedAtTimestamp < Gitlab::Database::Migration[2.0] +class RenameUsersUpdatedAtToUpdatedAtTimestamp < Gitlab::Database::Migration[2.1] disable_ddl_transaction! def up @@ -186,7 +186,7 @@ We can perform this cleanup using ```ruby # A post-deployment migration in db/post_migrate -class CleanupUsersUpdatedAtRename < Gitlab::Database::Migration[2.0] +class CleanupUsersUpdatedAtRename < Gitlab::Database::Migration[2.1] disable_ddl_transaction! def up @@ -233,7 +233,7 @@ as follows: ```ruby # A regular migration in db/migrate -class ChangeUsersUsernameStringToText < Gitlab::Database::Migration[2.0] +class ChangeUsersUsernameStringToText < Gitlab::Database::Migration[2.1] disable_ddl_transaction! def up @@ -252,7 +252,7 @@ Next we need to clean up our changes using a post-deployment migration: ```ruby # A post-deployment migration in db/post_migrate -class ChangeUsersUsernameStringToTextCleanup < Gitlab::Database::Migration[2.0] +class ChangeUsersUsernameStringToTextCleanup < Gitlab::Database::Migration[2.1] disable_ddl_transaction! def up @@ -319,6 +319,11 @@ This operation is safe as there's no code using the table just yet. Dropping tables can be done safely using a post-deployment migration, but only if the application no longer uses the table. +Add the table to `DELETED_TABLES` in +[gitlab_schema.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/database/gitlab_schema.rb), +along with its `gitlab_schema`. Even though the table is deleted, it is still +referenced in database migrations. + ## Renaming Tables Renaming tables requires downtime as an application may continue @@ -418,7 +423,7 @@ Check how the migration is performing while it's running. Multiple ways to do th #### High-level status of batched background migrations -See how to [check the status of batched background migrations](../../update/index.md#checking-for-background-migrations-before-upgrading). +See how to [check the status of batched background migrations](../../update/background_migrations.md). #### Query the database @@ -478,7 +483,7 @@ for batched background migration: To monitor the health of the database, use these additional metrics: -- [PostgreSQL Tuple Statistics](https://dashboards.gitlab.net/d/000000167/postgresql-tuple-statistics?orgId=1&refresh=1m): if you see high rate of updates for the tables being actively converted, or increasing percentage of dead tuples for this table, it might mean that autovacuum cannot keep up. +- [PostgreSQL Tuple Statistics](https://dashboards.gitlab.net/d/000000167/postgresql-tuple-statistics?orgId=1&refresh=1m): if you see high rate of updates for the tables being actively converted, or increasing percentage of dead tuples for this table, it might mean that `autovacuum` cannot keep up. - [PostgreSQL Overview](https://dashboards.gitlab.net/d/000000144/postgresql-overview?orgId=1): if you see high system usage or transactions per second (TPS) on the primary database server, it might mean that the migration is causing problems. ### Prometheus metrics @@ -499,8 +504,8 @@ If the migration has not completed, the subsequent steps fail anyway. By checkin aim to have more helpful error message. 1. Create indexes using the `bigint` columns that match the existing indexes using the `integer` column ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L28-34)). -1. Create foreign keys (FK) using the `bigint` columns that match the existing FKs using the -`integer` column. Do this both for FK referencing other tables, and FKs that reference the table +1. Create foreign keys (FK) using the `bigint` columns that match the existing FK using the +`integer` column. Do this both for FK referencing other tables, and FK that reference the table that is being migrated ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L36-43)). 1. Inside a transaction, swap the columns: 1. Lock the tables involved. To reduce the chance of hitting a deadlock, we recommended to do this in parent to child order ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L47)). @@ -509,7 +514,7 @@ that is being migrated ([see an example](https://gitlab.com/gitlab-org/gitlab/-/ 1. Swap the defaults ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L59-62)). 1. Swap the PK constraint (if any) ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L64-68)). 1. Remove old indexes and rename new ones ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L70-72)). - 1. Remove old FKs (if still present) and rename new ones ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L74)). + 1. Remove old foreign keys (if still present) and rename new ones ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L74)). See example [merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66088), and [migration](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb). diff --git a/doc/development/database/background_migrations.md b/doc/development/database/background_migrations.md index fe62bbc6b14..457694c7abd 100644 --- a/doc/development/database/background_migrations.md +++ b/doc/development/database/background_migrations.md @@ -236,7 +236,7 @@ Next we need a post-deployment migration that schedules the migration for existing data. ```ruby -class ScheduleExtractIntegrationsUrl < Gitlab::Database::Migration[2.0] +class ScheduleExtractIntegrationsUrl < Gitlab::Database::Migration[2.1] disable_ddl_transaction! MIGRATION = 'ExtractIntegrationsUrl' @@ -263,7 +263,7 @@ jobs and manually run on any un-migrated rows. Such a migration would look like this: ```ruby -class ConsumeRemainingExtractIntegrationsUrlJobs < Gitlab::Database::Migration[2.0] +class ConsumeRemainingExtractIntegrationsUrlJobs < Gitlab::Database::Migration[2.1] disable_ddl_transaction! def up @@ -412,7 +412,7 @@ When looking at the batch execution time versus the delay time, the execution ti should fit comfortably within the delay time for a few reasons: - To allow for a variance in query times. -- To allow autovacuum to catch up after periods of high churn. +- To allow `autovacuum` to catch up after periods of high churn. Never try to optimize by fully filling the delay window even if you are confident the queries themselves have no timing variance. diff --git a/doc/development/database/batched_background_migrations.md b/doc/development/database/batched_background_migrations.md index ca11e9c8dd3..71df4da59c3 100644 --- a/doc/development/database/batched_background_migrations.md +++ b/doc/development/database/batched_background_migrations.md @@ -285,7 +285,7 @@ In the second (filtered) example, we know exactly 100 will be updated with each 1. In the post-deployment migration, enqueue the batched background migration: ```ruby - class BackfillNamespaceType < Gitlab::Database::Migration[2.0] + class BackfillNamespaceType < Gitlab::Database::Migration[2.1] MIGRATION = 'BackfillNamespaceType' DELAY_INTERVAL = 2.minutes @@ -308,7 +308,7 @@ In the second (filtered) example, we know exactly 100 will be updated with each NOTE: When applying additional filters, it is important to ensure they are properly covered by an index to optimize `EachBatch` performance. -In the example above we need an index on `(type, id)` to support the filters. See [the `EachBatch` docs for more information](../iterating_tables_in_batches.md). +In the example above we need an index on `(type, id)` to support the filters. See [the `EachBatch` docs for more information](iterating_tables_in_batches.md). ## Example @@ -366,7 +366,7 @@ background migration. 1. Create a post-deployment migration that queues the migration for existing data: ```ruby - class QueueBackfillRoutesNamespaceId < Gitlab::Database::Migration[2.0] + class QueueBackfillRoutesNamespaceId < Gitlab::Database::Migration[2.1] MIGRATION = 'BackfillRouteNamespaceId' DELAY_INTERVAL = 2.minutes @@ -403,7 +403,7 @@ background migration. that checks that the batched background migration is completed. For example: ```ruby - class FinalizeBackfillRouteNamespaceId < Gitlab::Database::Migration[2.0] + class FinalizeBackfillRouteNamespaceId < Gitlab::Database::Migration[2.1] MIGRATION = 'BackfillRouteNamespaceId' disable_ddl_transaction! @@ -452,7 +452,7 @@ the batching column. Database post-migration: ```ruby -class ProjectsWithIssuesMigration < Gitlab::Database::Migration[2.0] +class ProjectsWithIssuesMigration < Gitlab::Database::Migration[2.1] MIGRATION = 'BatchProjectsWithIssues' INTERVAL = 2.minutes BATCH_SIZE = 5000 diff --git a/doc/development/database/creating_enums.md b/doc/development/database/creating_enums.md index 73c3f546728..e2ae36f7481 100644 --- a/doc/development/database/creating_enums.md +++ b/doc/development/database/creating_enums.md @@ -79,7 +79,7 @@ This works as-is, however, it has a couple of downside that: - When it happens, we have to ship a database migration to fix the data integrity, which might be impossible if you cannot recover the original value. -Also, you might observe a workaround for this concern by setting an offset in EE's values. +Also, you might observe a workaround for this concern by setting an offset in the `EE` module's values. For example, this example sets `1000` as the offset: ```ruby diff --git a/doc/development/database/database_debugging.md b/doc/development/database/database_debugging.md index 0d6e9955a19..edc35dd95e8 100644 --- a/doc/development/database/database_debugging.md +++ b/doc/development/database/database_debugging.md @@ -53,7 +53,7 @@ bundle exec rake db:reset RAILS_ENV=test - `bundle exec rake db:migrate:up:main VERSION=20170926203418 RAILS_ENV=development`: Set up a migration - `bundle exec rake db:migrate:redo:main VERSION=20170926203418 RAILS_ENV=development`: Re-run a specific migration -Replace `main` in the above commands to execute agains the `ci` database instead of `main`. +Replace `main` in the above commands to execute against the `ci` database instead of `main`. ## Manually access the database @@ -107,7 +107,7 @@ The new connection should be working now. Use these instructions for exploring the GitLab database while developing with the GDK: 1. Install or open [Visual Studio Code](https://code.visualstudio.com/download). -1. Install the [PostgreSQL VSCode Extension](https://marketplace.visualstudio.com/items?itemName=ckolkman.vscode-postgres). +1. Install the [PostgreSQL VS Code Extension](https://marketplace.visualstudio.com/items?itemName=ckolkman.vscode-postgres). 1. In Visual Studio Code select **PostgreSQL Explorer** in the left toolbar. 1. In the top bar of the new window, select `+` to **Add Database Connection**, and follow the prompts to fill in the details: 1. **Hostname**: the path to the PostgreSQL folder in your GDK directory (for example `/dev/gitlab-development-kit/postgresql`). diff --git a/doc/development/database/database_dictionary.md b/doc/development/database/database_dictionary.md index bd6dbc54316..d74d7e77edb 100644 --- a/doc/development/database/database_dictionary.md +++ b/doc/development/database/database_dictionary.md @@ -11,7 +11,8 @@ locate the feature categories responsible for specific database tables. ## Location -Database dictionary metadata files are stored in the `gitlab` project under `db/docs/`. +Database dictionary metadata files are stored in the `gitlab` project under `db/docs/` for the `main` and `ci` databases. +For the `geo` database, the dictionary files are stored under `ee/db/docs/`. ## Example dictionary file @@ -29,23 +30,43 @@ milestone: '13.0' ## Schema -| Attribute | Type | Required | Description | -|----------------------|---------------|----------|--------------------------------------------------------------------------| -| `table_name` | String | yes | Database table name | -| `classes` | Array(String) | no | List of classes that respond to `.table_name` with the `table_name` | -| `feature_categories` | Array(String) | yes | List of feature categories using this table | -| `description` | String | no | Text description of the information stored in the table and it's purpose | -| `introduced_by_url` | URL | no | URL to the merge request or commit which introduced this table | -| `milestone` | String | no | The milestone that introduced this table | +| Attribute | Type | Required | Description | +|----------------------------|---------------|----------|-----------------------------------------------------------------------------------| +| `table_name` / `view_name` | String | yes | Database table name or view name | +| `classes` | Array(String) | no | List of classes that are associated to this table or view. | +| `feature_categories` | Array(String) | yes | List of feature categories using this table or view. | +| `description` | String | no | Text description of the information stored in the table or view, and its purpose. | +| `introduced_by_url` | URL | no | URL to the merge request or commit which introduced this table or view. | +| `milestone` | String | no | The milestone that introduced this table or view. | +| `gitlab_schema` | String | yes | GitLab schema name. | ## Adding tables -When adding a new table, create a new file under `db/docs/` named -`<table_name>.yml` containing as much information as you know about the table. +When adding a new table, create a new file under `db/docs/` for the `main` and `ci` databases. +For the `geo` database use `ee/db/docs/`. +Name the file as `<table_name>.yml`, containing as much information as you know about the table. Include this file in the commit with the migration that creates the table. ## Dropping tables -When dropping a table, you must remove the metadata file from `db/docs/` -in the same commit with the migration that drops the table. +When dropping a table, you must remove the metadata file from `db/docs/` for `main` and `ci` databases. +For the `geo` database, you must remove the file from `ee/db/docs/`. +Use the same commit with the migration that drops the table. + +## Adding views + +When adding a new view, you should: + +1. Create a new file for this view in the appropriate directory: + - `main` database: `db/docs/views/` + - `ci` database: `db/docs/views/` + - `geo` database: `ee/db/docs/views/` +1. Name the file `<view_name>.yml`, and include as much information as you know about the view. +1. Include this file in the commit with the migration that creates the view. + +## Dropping views + +When dropping a view, you must remove the metadata file from `db/docs/views/`. +For the `geo` database, you must remove the file from `ee/db/docs/views/`. +Use the same commit with the migration that drops the view. diff --git a/doc/development/database/efficient_in_operator_queries.md b/doc/development/database/efficient_in_operator_queries.md index fb7ff3c1cc2..78b310ae708 100644 --- a/doc/development/database/efficient_in_operator_queries.md +++ b/doc/development/database/efficient_in_operator_queries.md @@ -126,8 +126,7 @@ For very large groups the database queries can easily time out, causing HTTP 500 ## Optimizing ordered `IN` queries -In the talk -["How to teach an elephant to dance rock'n'roll"](https://www.youtube.com/watch?v=Ha38lcjVyhQ), +In the talk ["How to teach an elephant to dance rock and roll"](https://www.youtube.com/watch?v=Ha38lcjVyhQ), Maxim Boguk demonstrated a technique to optimize a special class of ordered `IN` queries, such as our ordered group-level queries. @@ -160,7 +159,7 @@ The technique can only optimize `IN` queries that satisfy the following requirem in the following order: `column_for_the_in_query`, `order by column 1`, and `order by column 2`. - The columns in the `ORDER BY` clause are distinct - (the combination of the columns uniquely identifies one particular column in the table). + (the combination of the columns uniquely identifies one particular row in the table). WARNING: This technique does not improve the performance of the `COUNT(*)` queries. @@ -918,11 +917,11 @@ the `LIMIT` is reached or no more data can be found. Here's an outline of the steps we take in the recursive CTE query (expressing the steps in SQL is non-trivial but is explained next): -1. Sort the initial resultset according to the `ORDER BY` clause. +1. Sort the initial `resultset` according to the `ORDER BY` clause. 1. Pick the top cursor to fetch the record, this is our first record. In the example, this cursor would be (`2020-01-05`, `3`) for `project_id=9`. 1. We can use (`2020-01-05`, `3`) to fetch the next issue respecting the `ORDER BY` clause -`project_id=9` filter. This produces an updated resultset. +`project_id=9` filter. This produces an updated `resultset`. | `project_ids` | `created_at_values` | `id_values` | | ------------- | ------------------- | ----------- | @@ -931,7 +930,7 @@ this cursor would be (`2020-01-05`, `3`) for `project_id=9`. | 10 | 2020-01-15 | 7 | | **9** | **2020-01-06** | **6** | -1. Repeat 1 to 3 with the updated resultset until we have fetched `N=20` records. +1. Repeat 1 to 3 with the updated `resultset` until we have fetched `N=20` records. ### Initializing the recursive CTE query diff --git a/doc/development/database/foreign_keys.md b/doc/development/database/foreign_keys.md index d9506ae614a..25b3d815d7a 100644 --- a/doc/development/database/foreign_keys.md +++ b/doc/development/database/foreign_keys.md @@ -49,7 +49,7 @@ To replace a foreign key: foreign key before removing the old one. ```ruby - class ReplaceFkOnPackagesPackagesProjectId < Gitlab::Database::Migration[2.0] + class ReplaceFkOnPackagesPackagesProjectId < Gitlab::Database::Migration[2.1] disable_ddl_transaction! NEW_CONSTRAINT_NAME = 'fk_new' @@ -69,7 +69,7 @@ To replace a foreign key: 1. [Validate the new foreign key](add_foreign_key_to_existing_column.md#validate-the-foreign-key) ```ruby - class ValidateFkNew < Gitlab::Database::Migration[2.0] + class ValidateFkNew < Gitlab::Database::Migration[2.1] NEW_CONSTRAINT_NAME = 'fk_new' # foreign key added in <link to MR or path to migration adding new FK> @@ -86,7 +86,7 @@ To replace a foreign key: 1. Remove the old foreign key: ```ruby - class RemoveFkOld < Gitlab::Database::Migration[2.0] + class RemoveFkOld < Gitlab::Database::Migration[2.1] OLD_CONSTRAINT_NAME = 'fk_old' # new foreign key added in <link to MR or path to migration adding new FK> diff --git a/doc/development/database/index.md b/doc/development/database/index.md index 87b1b4a9d87..c244d784422 100644 --- a/doc/development/database/index.md +++ b/doc/development/database/index.md @@ -75,12 +75,24 @@ info: To determine the technical writer assigned to the Stage/Group associated w - [Pagination performance guidelines](pagination_performance_guidelines.md) - [Efficient `IN` operator queries](efficient_in_operator_queries.md) - [Data layout and access patterns](layout_and_access_patterns.md) +- [Check for background migrations before upgrading](../../update/background_migrations.md) ## Case studies - [Database case study: Filtering by label](filtering_by_label.md) - [Database case study: Namespaces storage statistics](namespaces_storage_statistics.md) +## PostgreSQL information for GitLab administrators + +- [Configure GitLab using an external PostgreSQL service](../../administration/postgresql/external.md) +- [Configuring PostgreSQL for scaling](../../administration/postgresql/index.md) +- [Database Load Balancing](../../administration/postgresql/database_load_balancing.md) +- [Moving GitLab databases to a different PostgreSQL instance](../../administration/postgresql/moving.md) +- [Replication and failover with Omnibus GitLab](../../administration/postgresql/replication_and_failover.md) +- [Standalone PostgreSQL using Omnibus GitLab](../../administration/postgresql/standalone.md) +- [Troubleshooting PostgreSQL](../../administration/troubleshooting/postgresql.md) +- [Working with the bundled PgBouncer service](../../administration/postgresql/pgbouncer.md) + ## Miscellaneous - [Maintenance operations](maintenance_operations.md) diff --git a/doc/development/database/keyset_pagination.md b/doc/development/database/keyset_pagination.md index 21bce41012e..42d7458b45a 100644 --- a/doc/development/database/keyset_pagination.md +++ b/doc/development/database/keyset_pagination.md @@ -159,7 +159,7 @@ configuration is necessary: - Function-based ordering. - Ordering with a custom tie-breaker column, like `iid`. -These order objects can be defined in the model classes as normal ActiveRecord scopes, there is no special behavior that prevents using these scopes elsewhere (kaminari, background jobs). +These order objects can be defined in the model classes as normal ActiveRecord scopes, there is no special behavior that prevents using these scopes elsewhere (Kaminari, background jobs). ### `NULLS LAST` ordering diff --git a/doc/development/database/loose_foreign_keys.md b/doc/development/database/loose_foreign_keys.md index 962cd2602bc..daa022a3de2 100644 --- a/doc/development/database/loose_foreign_keys.md +++ b/doc/development/database/loose_foreign_keys.md @@ -134,7 +134,7 @@ scripts/decomposition/generate-loose-foreign-key -c ci_job_token_project_scope_l ``` To swap all the foreign keys (all having `_id` appended), but not create a new branch (only commit -the changes) and not create rspecs, run: +the changes) and not create RSpec tests, run: ```shell scripts/decomposition/generate-loose-foreign-key -c --no-branch --no-rspec _id @@ -192,7 +192,7 @@ trigger needs to be configured only once. If the model already has at least one `loose_foreign_key` definition, then this step can be skipped: ```ruby -class TrackProjectRecordChanges < Gitlab::Database::Migration[2.0] +class TrackProjectRecordChanges < Gitlab::Database::Migration[2.1] include Gitlab::Database::MigrationHelpers::LooseForeignKeyHelpers enable_lock_retries! @@ -227,7 +227,7 @@ trigger. If the foreign key is deleted earlier, there is a good chance of introducing data inconsistency which needs manual cleanup: ```ruby -class RemoveProjectsCiPipelineFk < Gitlab::Database::Migration[2.0] +class RemoveProjectsCiPipelineFk < Gitlab::Database::Migration[2.1] disable_ddl_transaction! def up @@ -258,7 +258,7 @@ records in the database. Migration for removing the trigger: ```ruby -class UnTrackProjectRecordChanges < Gitlab::Database::Migration[2.0] +class UnTrackProjectRecordChanges < Gitlab::Database::Migration[2.1] include Gitlab::Database::MigrationHelpers::LooseForeignKeyHelpers enable_lock_retries! @@ -278,7 +278,7 @@ table however, there is still a chance for having leftover pending records in th must be removed with an inline data migration. ```ruby -class RemoveLeftoverProjectDeletions < Gitlab::Database::Migration[2.0] +class RemoveLeftoverProjectDeletions < Gitlab::Database::Migration[2.1] disable_ddl_transaction! def up diff --git a/doc/development/database/migrations_for_multiple_databases.md b/doc/development/database/migrations_for_multiple_databases.md index b4d2656121b..bc0ef654336 100644 --- a/doc/development/database/migrations_for_multiple_databases.md +++ b/doc/development/database/migrations_for_multiple_databases.md @@ -58,7 +58,7 @@ Example migration adding a concurrent index that is treated as change of the str that is executed on all configured databases. ```ruby -class AddUserIdAndStateIndexToMergeRequestReviewers < Gitlab::Database::Migration[2.0] +class AddUserIdAndStateIndexToMergeRequestReviewers < Gitlab::Database::Migration[2.1] disable_ddl_transaction! INDEX_NAME = 'index_on_merge_request_reviewers_user_id_and_state' @@ -75,16 +75,24 @@ end #### Example: Add a new table to store in a single database -1. Define the [GitLab Schema](multiple_databases.md#gitlab-schema) of the table in [`lib/gitlab/database/gitlab_schemas.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/database/gitlab_schemas.yml): +1. Add the table to the [database dictionary](database_dictionary.md) in [`db/docs/`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/db/docs): ```yaml - ssh_signatures: :gitlab_main + table_name: ssh_signatures + description: Description example + introduced_by_url: Merge request link + milestone: Milestone example + feature_categories: + - Feature category example + classes: + - Class example + gitlab_schema: gitlab_main ``` 1. Create the table in a schema migration: ```ruby - class CreateSshSignatures < Gitlab::Database::Migration[2.0] + class CreateSshSignatures < Gitlab::Database::Migration[2.1] def change create_table :ssh_signatures do |t| t.timestamps_with_timezone null: false @@ -125,7 +133,7 @@ Example migration updating `archived` column of `projects` that is executed only for the database containing `gitlab_main` schema. ```ruby -class UpdateProjectsArchivedState < Gitlab::Database::Migration[2.0] +class UpdateProjectsArchivedState < Gitlab::Database::Migration[2.1] disable_ddl_transaction! restrict_gitlab_migration gitlab_schema: :gitlab_main @@ -157,7 +165,7 @@ databases. For example, running migration in context of `ci:` and reading featur from `main:`, as no established connection to another database is present. ```ruby -class UpdateProjectsArchivedState < Gitlab::Database::Migration[2.0] +class UpdateProjectsArchivedState < Gitlab::Database::Migration[2.1] disable_ddl_transaction! restrict_gitlab_migration gitlab_schema: :gitlab_main @@ -195,7 +203,7 @@ that is marked in `lib/gitlab/database/gitlab_schemas.yml` as `gitlab_shared`. This migration is executed across all configured databases. ```ruby -class DeleteAllLooseForeignKeyRecords < Gitlab::Database::Migration[2.0] +class DeleteAllLooseForeignKeyRecords < Gitlab::Database::Migration[2.1] disable_ddl_transaction! def up @@ -211,13 +219,13 @@ end #### Example: run DML `gitlab_shared` only on the database containing the given `gitlab_schema` Example migration updating `loose_foreign_keys_deleted_records` table -that is marked in `lib/gitlab/database/gitlab_schemas.yml` as `gitlab_shared`. +that is marked in `db/docs/loose_foreign_keys_deleted_records.yml` as `gitlab_shared`. This migration since it configures restriction on `gitlab_ci` is executed only in context of database containing `gitlab_ci` schema. ```ruby -class DeleteCiBuildsLooseForeignKeyRecords < Gitlab::Database::Migration[2.0] +class DeleteCiBuildsLooseForeignKeyRecords < Gitlab::Database::Migration[2.1] disable_ddl_transaction! restrict_gitlab_migration gitlab_schema: :gitlab_ci @@ -261,7 +269,7 @@ the `database_tasks: false` set. `gitlab:db:validate_config` always runs before ## Validation Validation in a nutshell uses [`pg_query`](https://github.com/pganalyze/pg_query) to analyze -each query and classify tables with information from [`gitlab_schema.yml`](multiple_databases.md#gitlab-schema). +each query and classify tables with information from [`db/docs/`](database_dictionary.md). The migration is skipped if the specified `gitlab_schema` is outside of a list of schemas managed by a given database connection (`Gitlab::Database::gitlab_schemas_for_connection`). @@ -279,7 +287,7 @@ as part of the migration run and prevent the migration from being completed. ### Exception 1: migration running in DDL mode does DML select ```ruby -class UpdateProjectsArchivedState < Gitlab::Database::Migration[2.0] +class UpdateProjectsArchivedState < Gitlab::Database::Migration[2.1] disable_ddl_transaction! # Missing: @@ -310,7 +318,7 @@ running in **DDL** mode, but the executed payload appears to be reading data fro ### Exception 2: migration running in DML mode changes the structure ```ruby -class AddUserIdAndStateIndexToMergeRequestReviewers < Gitlab::Database::Migration[2.0] +class AddUserIdAndStateIndexToMergeRequestReviewers < Gitlab::Database::Migration[2.1] disable_ddl_transaction! # restrict_gitlab_migration if defined indicates DML, it should be removed @@ -341,7 +349,7 @@ but the executed payload appears to be doing structure changes (DDL). ### Exception 3: migration running in DML mode accesses data from a table in another schema ```ruby -class UpdateProjectsArchivedState < Gitlab::Database::Migration[2.0] +class UpdateProjectsArchivedState < Gitlab::Database::Migration[2.1] disable_ddl_transaction! # Since it modifies `projects` it should use `gitlab_main` @@ -372,7 +380,7 @@ data in `gitlab_main`. ### Exception 4: mixing DDL and DML mode ```ruby -class UpdateProjectsArchivedState < Gitlab::Database::Migration[2.0] +class UpdateProjectsArchivedState < Gitlab::Database::Migration[2.1] disable_ddl_transaction! # This migration is invalid regardless of specification @@ -435,4 +443,4 @@ tables in any database, just like any ordinary Sidekiq worker can. ## How to determine `gitlab_schema` for a given table -See [GitLab Schema](multiple_databases.md#gitlab-schema). +See [database dictionary](database_dictionary.md). diff --git a/doc/development/database/multiple_databases.md b/doc/development/database/multiple_databases.md index e5b6cfb8866..d22e3209096 100644 --- a/doc/development/database/multiple_databases.md +++ b/doc/development/database/multiple_databases.md @@ -14,9 +14,9 @@ On GitLab.com we are using two separate databases. ## GitLab Schema For properly discovering allowed patterns between different databases -the GitLab application implements the `lib/gitlab/database/gitlab_schemas.yml` YAML file. +the GitLab application implements the [database dictionary](database_dictionary.md). -This file provides a virtual classification of tables into a `gitlab_schema` +The database dictionary provides a virtual classification of tables into a `gitlab_schema` which conceptually is similar to [PostgreSQL Schema](https://www.postgresql.org/docs/current/ddl-schemas.html). We decided as part of [using database schemas to better isolated CI decomposed features](https://gitlab.com/gitlab-org/gitlab/-/issues/333415) that we cannot use PostgreSQL schema due to complex migration procedures. Instead we implemented diff --git a/doc/development/database/not_null_constraints.md b/doc/development/database/not_null_constraints.md index 53ab9a83d60..77fa23bbb19 100644 --- a/doc/development/database/not_null_constraints.md +++ b/doc/development/database/not_null_constraints.md @@ -25,7 +25,7 @@ For example, consider a migration that creates a table with two `NOT NULL` colum `db/migrate/20200401000001_create_db_guides.rb`: ```ruby -class CreateDbGuides < Gitlab::Database::Migration[2.0] +class CreateDbGuides < Gitlab::Database::Migration[2.1] def change create_table :db_guides do |t| t.bigint :stars, default: 0, null: false @@ -44,7 +44,7 @@ For example, consider a migration that adds a new `NOT NULL` column `active` to `db/migrate/20200501000001_add_active_to_db_guides.rb`: ```ruby -class AddExtendedTitleToSprints < Gitlab::Database::Migration[2.0] +class AddExtendedTitleToSprints < Gitlab::Database::Migration[2.1] def change add_column :db_guides, :active, :boolean, default: true, null: false end @@ -116,7 +116,7 @@ with `validate: false` in a post-deployment migration, `db/post_migrate/20200501000001_add_not_null_constraint_to_epics_description.rb`: ```ruby -class AddNotNullConstraintToEpicsDescription < Gitlab::Database::Migration[2.0] +class AddNotNullConstraintToEpicsDescription < Gitlab::Database::Migration[2.1] disable_ddl_transaction! def up @@ -147,7 +147,7 @@ so we add a post-deployment migration for the 13.0 milestone (current), `db/post_migrate/20200501000002_cleanup_epics_with_null_description.rb`: ```ruby -class CleanupEpicsWithNullDescription < Gitlab::Database::Migration[2.0] +class CleanupEpicsWithNullDescription < Gitlab::Database::Migration[2.1] # With BATCH_SIZE=1000 and epics.count=29500 on GitLab.com # - 30 iterations will be run # - each requires on average ~150ms @@ -185,7 +185,7 @@ migration helper in a final post-deployment migration, `db/post_migrate/20200601000001_validate_not_null_constraint_on_epics_description.rb`: ```ruby -class ValidateNotNullConstraintOnEpicsDescription < Gitlab::Database::Migration[2.0] +class ValidateNotNullConstraintOnEpicsDescription < Gitlab::Database::Migration[2.1] disable_ddl_transaction! def up diff --git a/doc/development/database/pagination_performance_guidelines.md b/doc/development/database/pagination_performance_guidelines.md index 0f98b50d95c..b06839979da 100644 --- a/doc/development/database/pagination_performance_guidelines.md +++ b/doc/development/database/pagination_performance_guidelines.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Pagination performance guidelines -The following document gives a few ideas for improving the pagination (sorting) performance. These apply both on [offset](pagination_guidelines.md#offset-pagination) and [keyset](pagination_guidelines.md#keyset-pagination) paginations. +The following document gives a few ideas for improving the pagination (sorting) performance. These apply both on [offset](pagination_guidelines.md#offset-pagination) and [keyset](pagination_guidelines.md#keyset-pagination) pagination. ## Tie-breaker column diff --git a/doc/development/database/polymorphic_associations.md b/doc/development/database/polymorphic_associations.md index f3c9bf1276f..bb7eb46b448 100644 --- a/doc/development/database/polymorphic_associations.md +++ b/doc/development/database/polymorphic_associations.md @@ -98,10 +98,10 @@ AND source_id = 4 Instead such a table should be broken up into separate tables. For example, you may end up with 4 tables in this case: -- project_members -- group_members -- pending_project_members -- pending_group_members +- `project_members` +- `group_members` +- `pending_project_members` +- `pending_group_members` This makes querying data trivial. For example, to get the members of a group you'd run: diff --git a/doc/development/database/query_performance.md b/doc/development/database/query_performance.md index 61fd80338fe..73a6a40f801 100644 --- a/doc/development/database/query_performance.md +++ b/doc/development/database/query_performance.md @@ -15,13 +15,14 @@ When you are optimizing your SQL queries, there are two dimensions to pay attent ## Timing guidelines for queries -| Query Type | Maximum Query Time | Notes | -|----|----|---| -| General queries | `100ms` | This is not a hard limit, but if a query is getting above it, it is important to spend time understanding why it can or cannot be optimized. | -| Queries in a migration | `100ms` | This is different than the total [migration time](../migration_style_guide.md#how-long-a-migration-should-take). | -| Concurrent operations in a migration | `5min` | Concurrent operations do not block the database, but they block the GitLab update. This includes operations such as `add_concurrent_index` and `add_concurrent_foreign_key`. | -| Background migrations | `1s` | | -| Service Ping | `1s` | See the [Service Ping docs](../service_ping/implement.md) for more details. | +| Query Type | Maximum Query Time | Notes | +|-------------------------------------------|--------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| General queries | `100ms` | This is not a hard limit, but if a query is getting above it, it is important to spend time understanding why it can or cannot be optimized. | +| Queries in a migration | `100ms` | This is different than the total [migration time](../migration_style_guide.md#how-long-a-migration-should-take). | +| Concurrent operations in a migration | `5min` | Concurrent operations do not block the database, but they block the GitLab update. This includes operations such as `add_concurrent_index` and `add_concurrent_foreign_key`. | +| Concurrent operations in a post migration | `20min` | Concurrent operations do not block the database, but they block the GitLab post update process. This includes operations such as `add_concurrent_index` and `add_concurrent_foreign_key`. If index creation exceeds 20 minutes, consider [async index creation](adding_database_indexes.md#create-indexes-asynchronously). | +| Background migrations | `1s` | | +| Service Ping | `1s` | See the [Service Ping docs](../service_ping/implement.md) for more details. | - When analyzing your query's performance, pay attention to if the time you are seeing is on a [cold or warm cache](#cold-and-warm-cache). These guidelines apply for both cache types. - When working with batched queries, change the range and batch size to see how it effects the query timing and caching. diff --git a/doc/development/database/query_recorder.md b/doc/development/database/query_recorder.md index f1540e7e2ae..84bd0fc938f 100644 --- a/doc/development/database/query_recorder.md +++ b/doc/development/database/query_recorder.md @@ -47,7 +47,7 @@ the longest common prefix, grouping similar queries together. In some cases, N+1 specs have been written to include three requests: first one to warm the cache, second one to establish a control, third one to validate that -ther are no N+1 queries. Rather than make an extra request to warm the cache, prefer two requests +there are no N+1 queries. Rather than make an extra request to warm the cache, prefer two requests (control and test) and configure your test to ignore [cached queries](#cached-queries) in N+1 specs. ## Cached queries diff --git a/doc/development/database/single_table_inheritance.md b/doc/development/database/single_table_inheritance.md index 32de1fdea35..dcf696b85bc 100644 --- a/doc/development/database/single_table_inheritance.md +++ b/doc/development/database/single_table_inheritance.md @@ -31,7 +31,7 @@ could result in loading unexpected code or associations which may cause unintend side effects or failures during upgrades. ```ruby -class SomeMigration < Gitlab::Database::Migration[2.0] +class SomeMigration < Gitlab::Database::Migration[2.1] class Services < MigrationRecord self.table_name = 'services' self.inheritance_column = :_type_disabled @@ -47,7 +47,7 @@ This ensures that the migration loads the columns for the migration in isolation and the helper disables STI by default. ```ruby -class EnqueueSomeBackgroundMigration < Gitlab::Database::Migration[2.0] +class EnqueueSomeBackgroundMigration < Gitlab::Database::Migration[2.1] disable_ddl_transaction! def up diff --git a/doc/development/database/strings_and_the_text_data_type.md b/doc/development/database/strings_and_the_text_data_type.md index fb005e51902..47e89c1ce0f 100644 --- a/doc/development/database/strings_and_the_text_data_type.md +++ b/doc/development/database/strings_and_the_text_data_type.md @@ -50,7 +50,7 @@ For example, consider a migration that creates a table with two text columns, `db/migrate/20200401000001_create_db_guides.rb`: ```ruby -class CreateDbGuides < Gitlab::Database::Migration[2.0] +class CreateDbGuides < Gitlab::Database::Migration[2.1] def change create_table :db_guides do |t| t.bigint :stars, default: 0, null: false @@ -74,7 +74,7 @@ For example, consider a migration that adds a new text column `extended_title` t `db/migrate/20200501000001_add_extended_title_to_sprints.rb`: ```ruby -class AddExtendedTitleToSprints < Gitlab::Database::Migration[2.0] +class AddExtendedTitleToSprints < Gitlab::Database::Migration[2.1] # rubocop:disable Migration/AddLimitToTextColumns # limit is added in 20200501000002_add_text_limit_to_sprints_extended_title @@ -89,7 +89,7 @@ A second migration should follow the first one with a limit added to `extended_t `db/migrate/20200501000002_add_text_limit_to_sprints_extended_title.rb`: ```ruby -class AddTextLimitToSprintsExtendedTitle < Gitlab::Database::Migration[2.0] +class AddTextLimitToSprintsExtendedTitle < Gitlab::Database::Migration[2.1] disable_ddl_transaction! def up @@ -165,7 +165,7 @@ in a post-deployment migration, `db/post_migrate/20200501000001_add_text_limit_migration.rb`: ```ruby -class AddTextLimitMigration < Gitlab::Database::Migration[2.0] +class AddTextLimitMigration < Gitlab::Database::Migration[2.1] disable_ddl_transaction! def up @@ -196,7 +196,7 @@ to add a background migration for the 13.0 milestone (current), `db/post_migrate/20200501000002_schedule_cap_title_length_on_issues.rb`: ```ruby -class ScheduleCapTitleLengthOnIssues < Gitlab::Database::Migration[2.0] +class ScheduleCapTitleLengthOnIssues < Gitlab::Database::Migration[2.1] # Info on how many records will be affected on GitLab.com # time each batch needs to run on average, etc ... BATCH_SIZE = 5000 @@ -236,7 +236,7 @@ helper in a final post-deployment migration, `db/post_migrate/20200601000001_validate_text_limit_migration.rb`: ```ruby -class ValidateTextLimitMigration < Gitlab::Database::Migration[2.0] +class ValidateTextLimitMigration < Gitlab::Database::Migration[2.1] disable_ddl_transaction! def up @@ -255,7 +255,7 @@ Increasing text limits on existing database columns can be safely achieved by fi and then dropping the previous limit: ```ruby -class ChangeMaintainerNoteLimitInCiRunner < Gitlab::Database::Migration[2.0] +class ChangeMaintainerNoteLimitInCiRunner < Gitlab::Database::Migration[2.1] disable_ddl_transaction! def up diff --git a/doc/development/database/table_partitioning.md b/doc/development/database/table_partitioning.md index ac715b871da..5f1deb77b6c 100644 --- a/doc/development/database/table_partitioning.md +++ b/doc/development/database/table_partitioning.md @@ -173,7 +173,7 @@ An example migration of partitioning the `audit_events` table by its `created_at` column would look like: ```ruby -class PartitionAuditEvents < Gitlab::Database::Migration[2.0] +class PartitionAuditEvents < Gitlab::Database::Migration[2.1] include Gitlab::Database::PartitioningMigrationHelpers def up @@ -200,7 +200,7 @@ into the partitioned copy. Continuing the above example, the migration would look like: ```ruby -class BackfillPartitionAuditEvents < Gitlab::Database::Migration[2.0] +class BackfillPartitionAuditEvents < Gitlab::Database::Migration[2.1] include Gitlab::Database::PartitioningMigrationHelpers def up @@ -233,7 +233,7 @@ failed jobs. Once again, continuing the example, this migration would look like: ```ruby -class CleanupPartitionedAuditEventsBackfill < Gitlab::Database::Migration[2.0] +class CleanupPartitionedAuditEventsBackfill < Gitlab::Database::Migration[2.1] include Gitlab::Database::PartitioningMigrationHelpers def up @@ -273,7 +273,7 @@ Include the partitioning key in the following constraints: Add the partitioning key column. For example, in a rails migration: ```ruby -class AddPartitionNumberForPartitioning < Gitlab::Database::Migration[2.0] +class AddPartitionNumberForPartitioning < Gitlab::Database::Migration[2.1] enable_lock_retries! TABLE_NAME = :table_name @@ -291,7 +291,7 @@ end Add indexes including the partitioning key column. For example, in a rails migration: ```ruby -class PrepareIndexesForPartitioning < Gitlab::Database::Migration[2.0] +class PrepareIndexesForPartitioning < Gitlab::Database::Migration[2.1] disable_ddl_transaction! TABLE_NAME = :table_name @@ -312,7 +312,7 @@ end Swap the primary key including the partitioning key column. For example, in a rails migration: ```ruby -class PreparePrimaryKeyForPartitioning < Gitlab::Database::Migration[2.0] +class PreparePrimaryKeyForPartitioning < Gitlab::Database::Migration[2.1] disable_ddl_transaction! TABLE_NAME = :table_name @@ -347,7 +347,7 @@ end Enforce unique indexes including the partitioning key column. For example, in a rails migration: ```ruby -class PrepareUniqueContraintForPartitioning < Gitlab::Database::Migration[2.0] +class PrepareUniqueContraintForPartitioning < Gitlab::Database::Migration[2.1] disable_ddl_transaction! TABLE_NAME = :table_name @@ -373,7 +373,7 @@ end Enforce foreign keys including the partitioning key column. For example, in a rails migration: ```ruby -class PrepareForeignKeyForPartitioning < Gitlab::Database::Migration[2.0] +class PrepareForeignKeyForPartitioning < Gitlab::Database::Migration[2.1] disable_ddl_transaction! SOURCE_TABLE_NAME = :source_table_name @@ -410,7 +410,7 @@ partition by using the following helpers provided by the database team. For example, using list partitioning in Rails post migrations: ```ruby -class PrepareTableConstraintsForListPartitioning < Gitlab::Database::Migration[2.0] +class PrepareTableConstraintsForListPartitioning < Gitlab::Database::Migration[2.1] include Gitlab::Database::PartitioningMigrationHelpers::TableManagementHelpers disable_ddl_transaction! @@ -441,7 +441,7 @@ end ``` ```ruby -class ConvertTableToListPartitioning < Gitlab::Database::Migration[2.0] +class ConvertTableToListPartitioning < Gitlab::Database::Migration[2.1] include Gitlab::Database::PartitioningMigrationHelpers::TableManagementHelpers disable_ddl_transaction! diff --git a/doc/development/database/understanding_explain_plans.md b/doc/development/database/understanding_explain_plans.md index fff9d755e9a..094bd6b346f 100644 --- a/doc/development/database/understanding_explain_plans.md +++ b/doc/development/database/understanding_explain_plans.md @@ -826,4 +826,4 @@ A more extensive guide on understanding query plans can be found in the [presentation](https://public.dalibo.com/exports/conferences/_archives/_2012/201211_explain/understanding_explain.pdf) from [Dalibo.org](https://www.dalibo.com/en/). -Depesz's blog also has a good [section](https://www.depesz.com/tag/unexplainable/) dedicated to query plans. +The Depesz blog also has a good [section](https://www.depesz.com/tag/unexplainable/) dedicated to query plans. diff --git a/doc/development/database_debugging.md b/doc/development/database_debugging.md deleted file mode 100644 index f18830ee7ca..00000000000 --- a/doc/development/database_debugging.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'database/database_debugging.md' -remove_date: '2022-11-06' ---- - -This document was moved to [another location](database/database_debugging.md). - -<!-- This redirect file can be deleted after <2022-11-06>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/database_query_comments.md b/doc/development/database_query_comments.md deleted file mode 100644 index 7f9def7e567..00000000000 --- a/doc/development/database_query_comments.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'database/database_query_comments.md' -remove_date: '2022-11-05' ---- - -This document was moved to [another location](database/database_query_comments.md). - -<!-- This redirect file can be deleted after <2022-11-05>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/database_review.md b/doc/development/database_review.md index 7fbc48af91c..e66be062986 100644 --- a/doc/development/database_review.md +++ b/doc/development/database_review.md @@ -150,7 +150,7 @@ Include in the MR description: - Write the raw SQL in the MR description. Preferably formatted nicely with [pgFormatter](https://sqlformat.darold.net) or - [paste.depesz.com](https://paste.depesz.com) and using regular quotes + <https://paste.depesz.com> and using regular quotes <!-- vale gitlab.NonStandardQuotes = NO --> (for example, `"projects"."id"`) and avoiding smart quotes (for example, `“projects”.“id”`). <!-- vale gitlab.NonStandardQuotes = YES --> diff --git a/doc/development/db_dump.md b/doc/development/db_dump.md deleted file mode 100644 index c632302329a..00000000000 --- a/doc/development/db_dump.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'database/db_dump.md' -remove_date: '2022-11-06' ---- - -This document was moved to [another location](database/db_dump.md). - -<!-- This redirect file can be deleted after <2022-11-06>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/directory_structure.md b/doc/development/directory_structure.md index 27384fa559f..34ee86d9ee5 100644 --- a/doc/development/directory_structure.md +++ b/doc/development/directory_structure.md @@ -1,94 +1,11 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +redirect_to: 'software_design.md' +remove_date: '2023-01-24' --- -# Backend directory structure +This document was moved to [another location](software_design.md) -## Use namespaces to define bounded contexts - -A healthy application is divided into macro and sub components that represent the contexts at play, -whether they are related to business domain or infrastructure code. - -As GitLab code has so many features and components it's hard to see what contexts are involved. -We should expect any class to be defined inside a module/namespace that represents the contexts where it operates. - -When we namespace classes inside their domain: - -- Similar terminology becomes unambiguous as the domain clarifies the meaning: - For example, `MergeRequests::Diff` and `Notes::Diff`. -- Top-level namespaces could be associated to one or more groups identified as domain experts. -- We can better identify the interactions and coupling between components. - For example, several classes inside `MergeRequests::` domain interact more with `Ci::` - domain and less with `ImportExport::`. - -```ruby -# bad -class MyClass -end - -# good -module MyDomain - class MyClass - end -end -``` - -### About namespace naming - -A good guideline for naming a top-level namespace (bounded context) is to use the related -[feature category](https://gitlab.com/gitlab-com/www-gitlab-com/-/blob/master/data/categories.yml). For example, `Continuous Integration` feature category maps to `Ci::` namespace. - -Alternatively a new class could be added to `Projects::` or `Groups::` if it's either: - -- Strictly related to one of these domains. For example `Projects::Alias`. -- A new component that does not have yet a more specific domain. In this case, when - a more explicit domain does emerge we would need to move the class to a more specific - namespace. - -Do not use the [stage or group name](https://about.gitlab.com/handbook/product/categories/#devops-stages) -since a feature category could be reassigned to a different group in the future. - -```ruby -# bad -module Create - class Commit - end -end - -# good -module Repositories - class Commit - end -end -``` - -On the other hand, a feature category may sometimes be too granular. Features tend to be -treated differently according to Product and Marketing, while they may share a lot of -domain models and behavior under the hood. In this case, having too many bounded contexts -could make them shallow and more coupled with other contexts. - -Bounded contexts (or top-level namespaces) can be seen as macro-components in the overall app. -Good bounded contexts should be [deep](https://medium.com/@nakabonne/depth-of-module-f62dac3c2fdb) -so consider having nested namespaces to further break down complex parts of the domain. -For example, `Ci::Config::`. - -For example, instead of having separate and granular bounded contexts like: `ContainerScanning::`, -`ContainerHostSecurity::`, `ContainerNetworkSecurity::`, we could have: - -```ruby -module ContainerSecurity - module HostSecurity - end - - module NetworkSecurity - end - - module Scanning - end -end -``` - -If classes that are defined into a namespace have a lot in common with classes in other namespaces, -chances are that these two namespaces are part of the same bounded context. +<!-- This redirect file can be deleted after <2023-01-24>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/documentation/index.md b/doc/development/documentation/index.md index d52db71b633..a9f2726ea93 100644 --- a/doc/development/documentation/index.md +++ b/doc/development/documentation/index.md @@ -12,7 +12,7 @@ The GitLab documentation is [intended as the single source of truth (SSOT)](http In addition to this page, the following resources can help you craft and contribute to documentation: - [Style Guide](styleguide/index.md) - What belongs in the docs, language guidelines, Markdown standards to follow, links, and more. -- [Topic type template](structure.md) - Learn about the different types of topics. +- [Topic types](topic_types/index.md) - Learn about the different types of topics. - [Documentation process](workflow.md). - [Markdown Guide](../../user/markdown.md) - A reference for all Markdown syntax supported by GitLab. - [Site architecture](site_architecture/index.md) - How <https://docs.gitlab.com> is built. @@ -159,26 +159,17 @@ You can use a Rake task to update the `CODEOWNERS` file. To update the `CODEOWNERS` file: -1. Open a merge request to update - [the Rake task](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/tasks/gitlab/tw/codeowners.rake) - with the latest [TW team assignments](https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments). -1. Assign the merge request to a backend maintainer for review and merge. -1. After the MR is merged, go to the root of the `gitlab` repository. -1. Run the Rake task and save the output in a file: - - ```shell - bundle exec rake tw:codeowners > ~/Desktop/updates.md - ``` - -1. Open the file (for example, `~/Desktop/updates.md`) and copy everything - except the errors at the bottom of the file. -1. Open the [`CODEOWNERS`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/CODEOWNERS) - file and paste the lines into the `^[Documentation Pages]` section. - - WARNING: - The documentation section is not the last section of the `CODEOWNERS` file. Don't - delete data that isn't ours! - +1. Review the [TW team assignments](https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments) + in the [`codeowners.rake`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/tasks/gitlab/tw/codeowners.rake) + file. If any assignments have changed: + 1. Update the `codeowners.rake` file with the changes. + 1. Assign the merge request to a technical writing manager for review and merge. +1. After the changes to `codeowners.rake` are merged, go to the root of the `gitlab` repository. +1. Run the Rake task with this command: `bundle exec rake tw:codeowners` +1. Review the command output for any pages that need attention to + their metadata. Handle any needed changes in a separate merge request. +1. Add the changes to the CODEOWNERS file to Git: `git add .gitlab/CODEOWNERS` +1. Commit your changes to your branch, and push your branch up to `origin`. 1. Create a merge request and assign it to a technical writing manager for review. ## Move, rename, or delete a page diff --git a/doc/development/documentation/site_architecture/deployment_process.md b/doc/development/documentation/site_architecture/deployment_process.md index 18cc27adaaa..2ba69ca0987 100644 --- a/doc/development/documentation/site_architecture/deployment_process.md +++ b/doc/development/documentation/site_architecture/deployment_process.md @@ -167,30 +167,5 @@ If you do not have the Maintainer role to perform this task, ask for help in the ## Docker files -The [`dockerfiles` directory](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/dockerfiles/) contains all needed -Dockerfiles to build and deploy <https://docs.gitlab.com>. It is heavily inspired by Docker's -[Dockerfile](https://github.com/docker/docker.github.io/blob/06ed03db13895bfe867761b6fc2ad40acf6026dd/Dockerfile). - -| Dockerfile | Docker image | Description | -|:---------------------------------------------------------------------------------------------------------------------------|:------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| [`bootstrap.Dockerfile`](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/dockerfiles/bootstrap.Dockerfile) | `gitlab-docs:bootstrap` | Contains all the dependencies that are needed to build the website. If the gems are updated and `Gemfile{,.lock}` changes, the image must be rebuilt. | -| [`builder.onbuild.Dockerfile`](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/dockerfiles/builder.onbuild.Dockerfile) | `gitlab-docs:builder-onbuild` | Base image to build the docs website. It uses `ONBUILD` to perform all steps and depends on `gitlab-docs:bootstrap`. | -| [`nginx.onbuild.Dockerfile`](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/dockerfiles/nginx.onbuild.Dockerfile) | `gitlab-docs:nginx-onbuild` | Base image to use for building documentation archives. It uses `ONBUILD` to perform all required steps to copy the archive, and relies upon its parent `Dockerfile.builder.onbuild` that is invoked when building single documentation archives (see the `Dockerfile` of each branch) | -| [`archives.Dockerfile`](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/dockerfiles/archives.Dockerfile) | `gitlab-docs:archives` | Contains all the versions of the website in one archive. It copies all generated HTML files from every version in one location. | - -### How to build the images - -Although build images are built automatically via GitLab CI/CD, you can build and tag all tooling images locally: - -1. Make sure you have [Docker installed](https://docs.docker.com/get-docker/). -1. Make sure you're in the `dockerfiles/` directory of the `gitlab-docs` repository. -1. Build the images: - - ```shell - docker build -t registry.gitlab.com/gitlab-org/gitlab-docs:bootstrap -f Dockerfile.bootstrap ../ - docker build -t registry.gitlab.com/gitlab-org/gitlab-docs:builder-onbuild -f Dockerfile.builder.onbuild ../ - docker build -t registry.gitlab.com/gitlab-org/gitlab-docs:nginx-onbuild -f Dockerfile.nginx.onbuild ../ - ``` - -For each image, there's a manual job under the `images` stage in -[`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/.gitlab-ci.yml) which can be invoked at any time. +The [`dockerfiles` directory](https://gitlab.com/gitlab-org/gitlab-docs/-/tree/main/dockerfiles) contains Dockerfiles needed +to build, test, and deploy <https://docs.gitlab.com>. diff --git a/doc/development/documentation/site_architecture/global_nav.md b/doc/development/documentation/site_architecture/global_nav.md index d95ca720119..ef6f3c0ae18 100644 --- a/doc/development/documentation/site_architecture/global_nav.md +++ b/doc/development/documentation/site_architecture/global_nav.md @@ -18,12 +18,14 @@ Global navigation is the left-most pane in the documentation. You can use the Research shows that people use Google to search for GitLab product documentation. When they land on a result, we want them to find topics nearby that are related to the content they're reading. The global nav provides this information. -At the highest level, our global nav is workflow-based. Navigation needs to help users build a mental model of how to use GitLab. +At the highest level, our global nav is **workflow-based**. Navigation needs to help users build a mental model of how to use GitLab. The levels under each of the higher workflow-based topics are the names of features. For example: **Use GitLab** (_workflow_) **> Build your application** (_workflow_) **> CI/CD** (_feature_) **> Pipelines** (_feature_) +While some older sections of the nav are alphabetical, the nav should primarily be workflow-based. + ## Choose the right words for your navigation entry Before you add an item to the left nav, choose the parts of speech you want to use. diff --git a/doc/development/documentation/structure.md b/doc/development/documentation/structure.md deleted file mode 100644 index 35a93f08f66..00000000000 --- a/doc/development/documentation/structure.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'topic_types/index.md' -remove_date: '2022-11-16' ---- - -This document was moved to [another location](topic_types/index.md). - -<!-- This redirect file can be deleted after <2022-11-16>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
\ No newline at end of file diff --git a/doc/development/documentation/styleguide/img/tier_badge.png b/doc/development/documentation/styleguide/img/tier_badge.png Binary files differdeleted file mode 100644 index 5fc38e08172..00000000000 --- a/doc/development/documentation/styleguide/img/tier_badge.png +++ /dev/null diff --git a/doc/development/documentation/styleguide/index.md b/doc/development/documentation/styleguide/index.md index ef934186981..3e55b334992 100644 --- a/doc/development/documentation/styleguide/index.md +++ b/doc/development/documentation/styleguide/index.md @@ -97,7 +97,7 @@ move in this direction, so we can address these issues: information into a format that is geared toward helping others, rather than documenting how a feature was implemented. -GitLab uses these [topic types](../structure.md). +GitLab uses these [topic types](../topic_types/index.md). ### Link instead of repeating text @@ -147,7 +147,7 @@ help benefit translation. For example, we: - [since and because](word_list.md#since) - [once and after](word_list.md#once) - [it](word_list.md#it) -- Avoid [ing](word_list.md#-ing-words) words. +- Avoid [-ing](word_list.md#-ing-words) words. [The GitLab voice](#the-gitlab-voice) dictates that we write clearly and directly, and with translation in mind. [The word list](word_list.md) and our Vale rules @@ -158,7 +158,7 @@ also aid in consistency, which is important for localization. All GitLab documentation is written using [Markdown](https://en.wikipedia.org/wiki/Markdown). The [documentation website](https://docs.gitlab.com) uses [GitLab Kramdown](https://gitlab.com/gitlab-org/gitlab_kramdown), -a "flavored" Kramdown engine to render pages from Markdown to HTML. The use of Kramdown's +a "flavored" Kramdown engine to render pages from Markdown to HTML. The use of Kramdown features is limited by our linters, so, use regular Markdown and follow the rules in the linked style guide. You can't use Kramdown-specific markup (for example, `{:.class}`). @@ -345,10 +345,36 @@ Some contractions, however, should be avoided: <!-- vale gitlab.Possessive = YES --> +### Possessives + +Try to avoid using possessives (`'s`) for proper nouns, like organization or product names. + +For example, instead of `Docker's CLI`, use `the Docker CLI`. + +For details, see [the Google documentation style guide](https://developers.google.com/style/possessives#product,-feature,-and-company-names). + +### Prepositions + +Use prepositions at the end of the sentence when needed. +Dangling or stranded prepositions are fine. For example: + +- You can leave the group you're a member of. +- Share the credentials with users you want to give access to. + +These constructions are more casual than the alternatives: + +- You can leave the group of which you're a member. +- Share the credentials with users to which you want to give access. + ### Acronyms If you use an acronym, spell it out on first use on a page. You do not need to spell it out more than once on a page. -When possible, try to avoid acronyms in topic titles. + +- **Titles:** Try to avoid acronyms in topic titles, especially if the acronym is not widely used. +- **Plurals:** Try not to make acronyms plural. For example, use `YAML files`, not `YAMLs`. If you must make an acronym plural, do not use an apostrophe. For example, use `APIs`, not `API's`. +- **Possessives:** Use caution when making an acronym possessive. If possible, + write the sentence to avoid making the acronym possessive. If you must make the + acronym possessive, consider spelling out the words. ### Numbers @@ -383,8 +409,12 @@ when published. Example: ### Emphasis +<!-- vale gitlab.Spelling = NO --> + Use **bold** rather than italic to provide emphasis. GitLab uses a sans-serif font and italic text does not stand out as much as it would in a serif font. For details, see [Butterick's Practical Typography guide on bold or italic](https://practicaltypography.com/bold-or-italic.html). +<!-- vale gitlab.Spelling = YES --> + You can use italics when you are introducing a term for the first time. Otherwise, use bold. - Use double asterisks (`**`) to mark a word or text in bold (`**bold**`). @@ -700,7 +730,7 @@ Put the entire link on a single line so that [linters](../testing.md) can find i ### Links in separate repositories To link to a page in a different repository, use an absolute URL. -For example, to link from a page in the GitLab repo to the Charts repo, +For example, to link from a page in the GitLab repository to the Charts repository, use a URL like `https://docs.gitlab.com/charts/`. ### Anchor links @@ -737,6 +767,28 @@ in your merge request fails. ### Text for links +Follow these guidelines for link text. + +#### Standard text + +As much as possible, use text that follows one of these patterns: + +- `For more information, see [LINK TEXT](LINK)`. +- `To [DO THIS THING], see [LINK TEXT](LINK)` + +For example: + +- `For more information, see [merge requests](../../../user/project/merge_requests/index.md).` +- `To create a review app, see [review apps](../../../ci/review_apps/index.md).` + +You can expand on this text by using phrases like +`For more information about this feature, see...` + +Do not to use alternate phrases, like `Learn more about...` or +`To read more...`. + +#### Descriptive text rather than `here` + Use descriptive text for links, rather than words like `here` or `this page.` For example, instead of: @@ -748,6 +800,14 @@ Use: - `For more information, see [merge requests](LINK)`. +#### Links to issues + +When linking to an issue, include the issue number in the link. For example: + +- `For more information, see [issue 12345](LINK).` + +Do not use the pound sign (`issue #12345`). + ### Links to external documentation When possible, avoid links to external documentation. These links can easily become outdated, and are difficult to maintain. @@ -1382,10 +1442,12 @@ Here's some other content in tab two. For tab titles, be brief and consistent. Ensure they are parallel, and start each with a capital letter. For example: -- `Omnibus package`, `Helm chart`, `Source` +- `Linux package (Omnibus)`, `Helm chart (Kubernetes)` (when documenting configuration edits, follow the + [configuration edits guide](#configuration-documentation-for-different-installation-methods)) - `15.1 and earlier`, `15.2 and later` -See [Pajamas](https://design.gitlab.com/components/tabs/#guidelines) for details. +See [Pajamas](https://design.gitlab.com/components/tabs/#guidelines) for more +details on tabs. ## Terms @@ -1414,9 +1476,7 @@ When names change, it is more complicated to search or grep text that has line b ### Product tier badges Tier badges are displayed as orange text next to a topic title. These badges link to the GitLab -pricing page. For example: - - +pricing page. You must assign a tier badge: @@ -1436,17 +1496,17 @@ functionality is described. #### Available product tier badges -| Tier in which feature is available | Tier badge | -|:------------------------------------------------------------------------|:----------------------| -| GitLab Free self-managed and SaaS, and higher tiers | `**(FREE)**` | -| GitLab Premium self-managed and SaaS, and their higher tiers | `**(PREMIUM)**` | -| GitLab Ultimate self-managed and SaaS | `**(ULTIMATE)**` | -| Only GitLab Free self-managed and higher tiers (no SaaS-based tiers) | `**(FREE SELF)**` | -| Only GitLab Premium self-managed and higher tiers (no SaaS-based tiers) | `**(PREMIUM SELF)**` | -| Only GitLab Ultimate self-managed (no SaaS-based tiers) | `**(ULTIMATE SELF)**` | -| Only GitLab Free SaaS and higher tiers (no self-managed instances) | `**(FREE SAAS)**` | -| Only GitLab Premium SaaS and higher tiers (no self-managed instances) | `**(PREMIUM SAAS)**` | -| Only GitLab Ultimate SaaS (no self-managed instances) | `**(ULTIMATE SAAS)**` | +| Where feature is available | Tier badge | +|:-----------------------------------------------------------------------------------------|:----------------------| +| On GitLab self-managed and GitLab SaaS, available in all tiers. | `**(FREE)**` | +| On GitLab self-managed and GitLab SaaS, available in Premium and Ultimate. | `**(PREMIUM)**` | +| On GitLab self-managed and GitLab SaaS, available in Ultimate. | `**(ULTIMATE)**` | +| On GitLab self-managed, available in all tiers. Not available on GitLab SaaS. | `**(FREE SELF)**` | +| On GitLab self-managed, available in Premium and Ultimate. Not available on GitLab SaaS. | `**(PREMIUM SELF)**` | +| On GitLab self-managed, available in Ultimate. Not available on GitLab SaaS. | `**(ULTIMATE SELF)**` | +| On GitLab SaaS, available in all tiers. Not available on self-managed. | `**(FREE SAAS)**` | +| On GitLab SaaS, available in Premium and Ultimate. Not available on self-managed. | `**(PREMIUM SAAS)**` | +| On GitLab SaaS, available in Ultimate. Not available on self-managed. | `**(ULTIMATE SAAS)**` | Topics that are only for instance administrators should be badged `<TIER> SELF`. Instance administrator documentation often includes sections that mention: @@ -1487,21 +1547,35 @@ we install Ruby from source. To update the guide for a new Ruby version: - Replace the sha256sum. It's available on the [downloads page](https://www.ruby-lang.org/en/downloads/) of the Ruby website. -### Configuration documentation for source and Omnibus installations +### Configuration documentation for different installation methods -GitLab supports two installation methods: installations from source, and Omnibus -packages. Possible configuration settings include: +GitLab supports four installation methods: -- Settings that touch configuration files in `config/`. -- NGINX settings. -- Other settings in `lib/support/`. +- Linux package (Omnibus) +- Helm chart (Kubernetes) +- Docker +- Self-compiled (source) Configuration procedures can require users to edit configuration files, reconfigure -GitLab, or restart GitLab. Use these styles to document these steps, replacing -`PATH/TO` with the appropriate path: +GitLab, or restart GitLab. In this case: + +- Use [tabs](#tabs) to differentiate among the various installation methods. +- Use the installation methods names exactly as described in the previous list. +- Use them in the order described below. +- Indent the code blocks to line up with the list item they belong to. +- Use the appropriate syntax highlighting for each code block (`ruby`, `shell`, or `yaml`). +- For the YAML files, always include the parent settings. +- The final step to reconfigure or restart GitLab can be used verbatim since it's + the same every time. + +You can copy and paste the following snippet when describing a configuration +edit: +<!-- markdownlint-disable tabs-blank-lines --> ````markdown -**For Omnibus installations** +::Tabs + +:::TabTitle Linux package (Omnibus) 1. Edit `/etc/gitlab/gitlab.rb`: @@ -1509,32 +1583,159 @@ GitLab, or restart GitLab. Use these styles to document these steps, replacing external_url "https://gitlab.example.com" ``` -1. Save the file and [reconfigure](PATH/TO/administration/restart_gitlab.md#omnibus-gitlab-reconfigure) - GitLab for the changes to take effect. +1. Save the file and reconfigure GitLab: ---- + ```shell + sudo gitlab-ctl reconfigure + ``` + +:::TabTitle Helm chart (Kubernetes) + +1. Export the Helm values: + + ```shell + helm get values gitlab > gitlab_values.yaml + ``` + +1. Edit `gitlab_values.yaml`: + + ```yaml + global: + hosts: + gitlab: + name: gitlab.example.com + ``` + +1. Save the file and apply the new values: + + ```shell + helm upgrade -f gitlab_values.yaml gitlab gitlab/gitlab + ``` + +:::TabTitle Docker + +1. Edit `docker-compose.yml`: + + ```yaml + version: "3.6" + services: + gitlab: + environment: + GITLAB_OMNIBUS_CONFIG: | + external_url "https://gitlab.example.com" + ``` -**For installations from source** +1. Save the file and restart GitLab: -1. Edit `config/gitlab.yml`: + ```shell + docker compose up -d + ``` + +:::TabTitle Self-compiled (source) + +1. Edit `/home/git/gitlab/config/gitlab.yml`: ```yaml - gitlab: - host: "gitlab.example.com" + production: &base + gitlab: + host: "gitlab.example.com" + ``` + +1. Save the file and restart GitLab: + + ```shell + # For systems running systemd + sudo systemctl restart gitlab.target + + # For systems running SysV init + sudo service gitlab restart ``` -1. Save the file and [restart](PATH/TO/administration/restart_gitlab.md#installations-from-source) - GitLab for the changes to take effect. +::EndTabs ```` +<!-- markdownlint-enable tabs-blank-lines --> + +It renders as: + +::Tabs + +:::TabTitle Linux package (Omnibus) + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + external_url "https://gitlab.example.com" + ``` + +1. Save the file and reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +:::TabTitle Helm chart (Kubernetes) + +1. Export the Helm values: + + ```shell + helm get values gitlab > gitlab_values.yaml + ``` + +1. Edit `gitlab_values.yaml`: + + ```yaml + global: + hosts: + gitlab: + name: gitlab.example.com + ``` + +1. Save the file and apply the new values: + + ```shell + helm upgrade -f gitlab_values.yaml gitlab gitlab/gitlab + ``` + +:::TabTitle Docker + +1. Edit `docker-compose.yml`: + + ```yaml + version: "3.6" + services: + gitlab: + environment: + GITLAB_OMNIBUS_CONFIG: | + external_url "https://gitlab.example.com" + ``` -In this case: +1. Save the file and restart GitLab: -- Bold the installation method's name. -- Separate the methods with three dashes (`---`) to create a horizontal line. -- Indent the code blocks to line up with the list item they belong to.. -- Use the appropriate syntax highlighting for each code block. -- Use the [GitLab Restart](#gitlab-restart) section to explain any required - restart or reconfigure of GitLab. + ```shell + docker compose up -d + ``` + +:::TabTitle Self-compiled (source) + +1. Edit `/home/git/gitlab/config/gitlab.yml`: + + ```yaml + production: &base + gitlab: + host: "gitlab.example.com" + ``` + +1. Save the file and restart GitLab: + + ```shell + # For systems running systemd + sudo systemctl restart gitlab.target + + # For systems running SysV init + sudo service gitlab restart + ``` + +::EndTabs ## Feature flags diff --git a/doc/development/documentation/styleguide/word_list.md b/doc/development/documentation/styleguide/word_list.md index d28972a644b..333a5521536 100644 --- a/doc/development/documentation/styleguide/word_list.md +++ b/doc/development/documentation/styleguide/word_list.md @@ -745,6 +745,11 @@ For **MB** and **GB**, follow the [Microsoft guidance](https://learn.microsoft.c Do not use first-person singular. Use **you**, **we**, or **us** instead. ([Vale](../testing.md#vale) rule: [`FirstPerson.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/FirstPerson.yml)) +## member + +When you add a [user account](#user-account) to a group or project, +the user account becomes a **member**. + ## merge requests Use lowercase for **merge requests**. If you use **MR** as the acronym, spell it out on first use. @@ -767,20 +772,41 @@ Do not use **navigate**. Use **go** instead. For example: ([Vale](../testing.md#vale) rule: [`SubstitutionSuggestions.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/SubstitutionSuggestions.yml)) -## need to, should +## need to + +Try to avoid **need to**, because it's wordy. + +For example, when a variable is **required**, +instead of **You need to set the variable**, use: + +- Set the variable. +- You must set the variable. + +When the variable is **recommended**: + +- You should set the variable. -Try to avoid **needs to**, because it's wordy. If something is recommended, use **should** instead. If something is required, use **must**. +When the variable is **optional**: + +- You can set the variable. + +## normal, normally + +Don't use **normal** to mean the usual, typical, or standard way of doing something. +Use those terms instead. Use: -- You should set the variable. (recommended) -- You must set the variable. (required) -- Set the variable. (required) +- Typically, you specify a certificate. +- Usually, you specify a certificate. +- Follow the standard Git workflow. Instead of: -- You need to set the variable. -- We recommend that you set the variable. +- Normally, you specify a certificate. +- Follow the normal Git workflow. + +([Vale](../testing.md#vale) rule: [`Normal.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/Normal.yml)) ## note that @@ -898,6 +924,15 @@ For example, you might write something like: Use lowercase for **push rules**. +## recommend, we recommend + +Instead of **we recommend**, use **you should**. We want to talk to the user the way +we would talk to a colleague, and to avoid differentiation between `we` and `them`. + +- You should set the variable. (It's recommended.) +- Set the variable. (It's required.) +- You can set the variable. (It's optional.) + ## register Use **register** instead of **sign up** when talking about creating an account. @@ -1068,6 +1103,21 @@ Do not use **slave**. Another option is **secondary**. ([Vale](../testing.md#val Use **subgroup** (no hyphen) instead of **sub-group**. ([Vale](../testing.md#vale) rule: [`SubstitutionSuggestions.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/SubstitutionSuggestions.yml)) +## subscription tier + +Do not confuse **subscription** or **subscription tier** with **[license](#license)**. +A user purchases a **subscription**. That subscription has a **tier**. + +To describe tiers: + +| Instead of | Use | +|---------------------------------|----------------------------------------| +| In the Free tier or greater | In all tiers | +| In the Free tier or higher | In all tiers | +| In the Premium tier or greater | In the Premium and Ultimate tier | +| In the Premium tier or higher | In the Premium and Ultimate tier | +| In the Premium tier or lower | In the Free and Premium tier | + ## that Do not use **that** when describing a noun. For example: @@ -1127,6 +1177,11 @@ Always follow these words with a noun. For example: - Use: **Those settings** need to be configured. (Or even better, **Configure those settings.**) - Instead of: **Those** need to be configured. +## to which, of which + +Try to avoid **to which** and **of which**, and let the preposition dangle at the end of the sentence instead. +For examples, see [Prepositions](index.md#prepositions). + ## to-do item Use lowercase and hyphenate **to-do** item. ([Vale](../testing.md#vale) rule: [`ToDo.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/ToDo.yml)) @@ -1198,18 +1253,10 @@ See also [downgrade](#downgrade) and [roll back](#roll-back). Do not use **useful**. If the user doesn't find the process to be useful, we lose their trust. ([Vale](../testing.md#vale) rule: [`Simplicity.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/Simplicity.yml)) -## user, users - -When possible, address the reader directly, instead of calling them **users**. -Use the [second person](#you-your-yours), **you**, instead. +## user account -Use: - -- You can configure a pipeline. - -Instead of: - -- Users can configure a pipeline. +You create a **user account**. The user account has an [access level](#access-level). +When you add a **user account** to a group or project, the user account becomes a **member**. ## utilize @@ -1270,7 +1317,7 @@ in present tense, active voice. ## you, your, yours -Use **you**, **your**, and **yours** instead of [**the user** and **the user's**](#user-users). +Use **you**, **your**, and **yours** instead of **the user** and **the user's**. Documentation should be from the [point of view](https://design.gitlab.com/content/voice-tone/#point-of-view) of the reader. Use: diff --git a/doc/development/documentation/topic_types/concept.md b/doc/development/documentation/topic_types/concept.md index 7be6bef4fad..e01b06c2c07 100644 --- a/doc/development/documentation/topic_types/concept.md +++ b/doc/development/documentation/topic_types/concept.md @@ -11,7 +11,7 @@ A concept introduces a single feature or concept. A concept should answer the questions: - What is this? -- Why would I use it? +- Why would you use it? Think of everything someone might want to know if they've never heard of this concept before. diff --git a/doc/development/documentation/topic_types/index.md b/doc/development/documentation/topic_types/index.md index 8e8c474ce3c..964b41303cb 100644 --- a/doc/development/documentation/topic_types/index.md +++ b/doc/development/documentation/topic_types/index.md @@ -6,20 +6,19 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Documentation topic types (CTRT) -At GitLab, we have not traditionally used types for our content. However, we are starting to -move in this direction, and we now use four primary topic types: +Each topic on a page should be one of the following topic types: - [Concept](concept.md) - [Task](task.md) - [Reference](reference.md) - [Troubleshooting](troubleshooting.md) +Even if a page is short, the page usually starts with a concept and then +includes a task or reference topic. + The tech writing team sometimes uses the acronym `CTRT` to refer to our topic types. The acronym refers to the first letter of each topic type. -In general, each page in the GitLab documentation contains multiple topics. -Each topic on a page should be recognizable as a specific topic type. - In addition to the four primary topic types, we also have a page type for [Tutorials](tutorial.md) and [Get started](#get-started). @@ -66,9 +65,9 @@ Some pages are solely a list of links to other documentation. We do not encourage this page type. Lists of links can get out-of-date quickly and offer little value to users, who prefer to search to find information. -## Topic text guidelines +## Topic title guidelines -In general, for topic text: +In general, for topic titles: - Be clear and direct. Make every word count. - Use articles and prepositions. diff --git a/doc/development/documentation/topic_types/task.md b/doc/development/documentation/topic_types/task.md index 78d670a16d6..0dba3e079b6 100644 --- a/doc/development/documentation/topic_types/task.md +++ b/doc/development/documentation/topic_types/task.md @@ -69,6 +69,12 @@ For example, `Create an issue when you want to track bugs or future work`. To start the task steps, use a succinct action followed by a colon. For example, `To create an issue:` +## Task prerequisites + +As a best practice, if the task requires the user to have a role other than Guest, +put the minimum role in the prerequisites. See [the Word list](../styleguide/word_list.md) for +how to write the phrase for each role. + ## Related topics - [View the format for writing task steps](../styleguide/index.md#navigation). diff --git a/doc/development/documentation/versions.md b/doc/development/documentation/versions.md index 030bdec0361..334dcd73ea5 100644 --- a/doc/development/documentation/versions.md +++ b/doc/development/documentation/versions.md @@ -46,7 +46,7 @@ The item text must include these words in order. Capitalization doesn't matter. - `introduced`, `enabled`, `deprecated`, `changed`, `moved`, `recommended`, `removed`, or `renamed` - `in` or `to` -- `GitLab` +- `GitLab` (or, for external projects, the name of the project) If possible, include a link to the related issue, merge request, or epic. Do not link to the pricing page. Do not include the subscription tier. @@ -203,8 +203,8 @@ We cannot guarantee future feature work, and promises like these can raise legal issues. Instead, say that an issue exists. For example: -- Support for improvements is proposed in issue `[issue-number](LINK-TO-ISSUE)`. -- You cannot do this thing, but issue `[issue-number](LINK-TO-ISSUE)` proposes to change this behavior. +- Support for improvements is proposed in `[issue <issue_number>](LINK-TO-ISSUE)`. +- You cannot do this thing, but `[issue 12345](LINK-TO-ISSUE)` proposes to change this behavior. You can say that we plan to remove a feature. diff --git a/doc/development/documentation/workflow.md b/doc/development/documentation/workflow.md index 9d8d25607c8..2effa21b266 100644 --- a/doc/development/documentation/workflow.md +++ b/doc/development/documentation/workflow.md @@ -13,29 +13,32 @@ Anyone can contribute to the GitLab documentation! You can create a merge reques accomplish their work with GitLab. If you are working on a feature or enhancement, use the -[feature workflow process described in the GitLab Handbook](https://about.gitlab.com/handbook/product/ux/technical-writing/workflow/#for-a-product-change). +[feature workflow process described in the GitLab Handbook](https://about.gitlab.com/handbook/product/ux/technical-writing/workflow/#documentation-for-a-product-change). ## How to update the docs If you are not a GitLab team member, or do not have the Developer role for the GitLab repository, to update GitLab documentation: -1. Select an issue you'd like to work on. +1. Select an [issue](https://about.gitlab.com/handbook/product/ux/technical-writing/#community-contribution-opportunities) you'd like to work on. - You don't need an issue to open a merge request. - For a Hackathon, in the issue, in a comment, mention the person who opened the issue and ask for the issue to be assigned to you. To be fair to other contributors, if you see someone has already asked to work on the issue, choose another issue. If you are looking for issues to work on and don't see any that suit you, you can always fix [Vale](testing.md#vale) issues. 1. Go to the [GitLab repository](https://gitlab.com/gitlab-org/gitlab). -1. In the top-right, select **Fork**. Forking makes a copy of the repository on GitLab.com. -1. In your fork, find the documentation page by going to the `\doc` directory. +1. In the top right, select **Fork**. Forking makes a copy of the repository on GitLab.com. +1. In your fork, find the documentation page in the `\doc` directory. 1. If you know Git, make your changes and open a merge request. If not, follow these steps: - 1. In the top right, select **Edit**, make the changes, and **Save**. - 1. From the left menu, select **Merge requests**. + 1. On the top right, select **Edit** if it is visible. If it is not, select the down arrow (**{chevron-lg-down}**) next to **Open in Web IDE** or **Gitpod**, and select **Edit**. + 1. In the **Commit message** text box, enter a commit message. Use 3-5 words, start with a capital letter, and do not end with a period. + 1. Select **Commit changes**. + 1. On the left sidebar, select **Merge requests**. + 1. Select **New merge request**. 1. For the source branch, select your fork and branch. If you did not create a branch, select `master`. For the target branch, select the [GitLab repository](https://gitlab.com/gitlab-org/gitlab) `master` branch. - 1. For the commit message, use 3-5 words, start with a capital letter, and do not end with a period. - 1. Select **Commit changes**. A merge request opens. + 1. Select **Compare branches and continue**. A new merge request opens. 1. Select the **Documentation** template. In the description, write a brief summary of the changes and link to the related issue, if there is one. + 1. Select **Create merge request**. If you need help while working on the page, view: @@ -65,7 +68,7 @@ If you are a member of the GitLab Slack workspace, you can request help in `#doc When you author an issue or merge request, you must add these labels: -- A [type label](../contributing/issue_workflow.md#type-labels). +- A [type label](../contributing/issue_workflow.md#type-labels), either `~"type::feature"` or `~"type::maintenance"`. - A [stage label](../contributing/issue_workflow.md#stage-labels) and [group label](../contributing/issue_workflow.md#group-labels). For example, `~devops::create` and `~group::source code`. - A `~documentation` [specialization label](../contributing/issue_workflow.md#specialization-labels). @@ -75,9 +78,8 @@ A member of the Technical Writing team adds these labels: - A [documentation scoped label](../../user/project/labels.md#scoped-labels) with the `docs::` prefix. For example, `~docs::improvement`. - The [`~Technical Writing` team label](../contributing/issue_workflow.md#team-labels). -- A type label: either `~"type::feature"` or `~"type::maintenance"`. -### Reviewing and merging +## Reviewing and merging Anyone with the Maintainer role to the relevant GitLab project can merge documentation changes. Maintainers must make a good-faith effort to ensure that the content: @@ -111,13 +113,24 @@ The process involves the following: The process is reflected in the **Documentation** [merge request template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/merge_request_templates/Documentation.md). -## Other ways to help +### Before merging -If you have ideas for further documentation resources please -[create an issue](https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Documentation) -using the Documentation template. +Ensure the following if skipping an initial Technical Writer review: + +- [Product badges](styleguide/index.md#product-tier-badges) are applied. +- The GitLab [version](versions.md) that + introduced the feature is included. +- Changes to topic titles don't affect in-app hyperlinks. +- Specific [user permissions](../../user/permissions.md) are documented. +- New documents are linked from higher-level indexes, for discoverability. +- The style guide is followed: + - For [directories and files](site_architecture/folder_structure.md). + - For [images](styleguide/index.md#images). + +Merge requests that change the location of documentation must always be reviewed by a Technical +Writer before merging. -## Post-merge reviews +### Post-merge reviews If not assigned to a Technical Writer for review prior to merging, a review must be scheduled immediately after merge by the developer or maintainer. For this, @@ -146,19 +159,8 @@ Remember: - The Technical Writer can also help decide that documentation can be merged without Technical writer review, with the review to occur soon after merge. -### Before merging - -Ensure the following if skipping an initial Technical Writer review: - -- [Product badges](styleguide/index.md#product-tier-badges) are applied. -- The GitLab [version](versions.md) that - introduced the feature is included. -- Changes to topic titles don't affect in-app hyperlinks. -- Specific [user permissions](../../user/permissions.md) are documented. -- New documents are linked from higher-level indexes, for discoverability. -- The style guide is followed: - - For [directories and files](site_architecture/folder_structure.md). - - For [images](styleguide/index.md#images). +## Other ways to help -Merge requests that change the location of documentation must always be reviewed by a Technical -Writer before merging. +If you have ideas for further documentation resources please +[create an issue](https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Documentation) +using the Documentation template. diff --git a/doc/development/ee_features.md b/doc/development/ee_features.md index 14df73b8779..5e236c3e322 100644 --- a/doc/development/ee_features.md +++ b/doc/development/ee_features.md @@ -94,6 +94,25 @@ setting the [`FOSS_ONLY` environment variable](https://gitlab.com/gitlab-org/git to something that evaluates as `true`. The same works for running tests (for example `FOSS_ONLY=1 yarn jest`). +### Simulate a CE instance with a licensed GDK + +To simulate a CE instance without deleting the license in a GDK: + +1. Create an `env.runit` file in the root of your GDK with the line: + + ```shell + export FOSS_ONLY=1 + ``` + +1. Then restart the GDK: + + ```shell + gdk restart rails && gdk restart webpack + ``` + +Remove the line in `env.runit` if you want to revert back to an EE +installation, and repeat step 2. + #### Run feature specs as CE When running [feature specs](testing_guide/best_practices.md#system--feature-tests) diff --git a/doc/development/elasticsearch.md b/doc/development/elasticsearch.md index ab2d241a781..88a417b4745 100644 --- a/doc/development/elasticsearch.md +++ b/doc/development/elasticsearch.md @@ -69,7 +69,7 @@ The `whitespace` tokenizer was selected to have more control over how tokens are Please see the `code` filter for an explanation on how tokens are split. NOTE: -The [Elasticsearch code_analyzer doesn't account for all code cases](../integration/advanced_search/elasticsearch_troubleshooting.md#elasticsearch-code_analyzer-doesnt-account-for-all-code-cases). +The [Elasticsearch `code_analyzer` doesn't account for all code cases](../integration/advanced_search/elasticsearch_troubleshooting.md#elasticsearch-code_analyzer-doesnt-account-for-all-code-cases). #### `code_search_analyzer` @@ -113,9 +113,9 @@ Uses a [Pattern Capture token filter](https://www.elastic.co/guide/en/elasticsea Patterns: -- `"(\\p{Ll}+|\\p{Lu}\\p{Ll}+|\\p{Lu}+)"`: captures CamelCased and lowedCameCased strings as separate tokens +- `"(\\p{Ll}+|\\p{Lu}\\p{Ll}+|\\p{Lu}+)"`: captures CamelCase and lowerCamelCase strings as separate tokens - `"(\\d+)"`: extracts digits -- `"(?=([\\p{Lu}]+[\\p{L}]+))"`: captures CamelCased strings recursively. Ex: `ThisIsATest` => `[ThisIsATest, IsATest, ATest, Test]` +- `"(?=([\\p{Lu}]+[\\p{L}]+))"`: captures CamelCase strings recursively. For example: `ThisIsATest` => `[ThisIsATest, IsATest, ATest, Test]` - `'"((?:\\"|[^"]|\\")*)"'`: captures terms inside quotes, removing the quotes - `"'((?:\\'|[^']|\\')*)'"`: same as above, for single-quotes - `'\.([^.]+)(?=\.|\s|\Z)'`: separate terms with periods in-between diff --git a/doc/development/fe_guide/accessibility.md b/doc/development/fe_guide/accessibility.md index 67166a93cb4..af45603782f 100644 --- a/doc/development/fe_guide/accessibility.md +++ b/doc/development/fe_guide/accessibility.md @@ -109,15 +109,15 @@ Text input examples: </gl-form-group> ``` -Textarea examples: +`textarea` examples: ```html -<!-- Textarea with label --> +<!-- textarea with label --> <gl-form-group :label="__('Issue description')" label-for="issue-description"> <gl-form-textarea id="issue-description" v-model="description" /> </gl-form-group> -<!-- Textarea with hidden label --> +<!-- textarea with hidden label --> <gl-form-group :label="__('Issue description')" label-for="issue-description" label-sr-only> <gl-form-textarea id="issue-description" v-model="description" /> </gl-form-group> @@ -347,7 +347,7 @@ Keep in mind that: See the [Pajamas Keyboard-only page](https://design.gitlab.com/accessibility-audits/keyboard-only/) for more detail. -## Tabindex +## `tabindex` Prefer **no** `tabindex` to using `tabindex`, since: diff --git a/doc/development/fe_guide/content_editor.md b/doc/development/fe_guide/content_editor.md index 8cc274c732e..982033cf2ad 100644 --- a/doc/development/fe_guide/content_editor.md +++ b/doc/development/fe_guide/content_editor.md @@ -11,7 +11,7 @@ experience for [GitLab Flavored Markdown](../../user/markdown.md) in the GitLab It also serves as the foundation for implementing Markdown-focused editors that target other engines, like static site generators. -We use [tiptap 2.0](https://tiptap.dev/) and [ProseMirror](https://prosemirror.net/) +We use [Tiptap 2.0](https://tiptap.dev/) and [ProseMirror](https://prosemirror.net/) to build the Content Editor. These frameworks provide a level of abstraction on top of the native [`contenteditable`](https://developer.mozilla.org/en-US/docs/Web/Guide/HTML/Editable_content) web technology. @@ -209,7 +209,7 @@ the following events: - `blur` - `error`. -Learn more about these events in [Tiptap's event guide](https://tiptap.dev/api/events/). +Learn more about these events in [the Tiptap event guide](https://tiptap.dev/api/events/). ```html <script> @@ -255,13 +255,13 @@ provides all the necessary extensions to support #### Implement new extensions Extensions are the building blocks of the Content Editor. You can learn how to implement -new ones by reading [Tiptap's guide](https://tiptap.dev/guide/custom-extensions). +new ones by reading [the Tiptap guide](https://tiptap.dev/guide/custom-extensions). We recommend checking the list of built-in [nodes](https://tiptap.dev/api/nodes) and [marks](https://tiptap.dev/api/marks) before implementing a new extension from scratch. Store the Content Editor extensions in the `~/content_editor/extensions` directory. -When using a Tiptap's built-in extension, wrap it in a ES6 module inside this directory: +When using a Tiptap built-in extension, wrap it in a ES6 module inside this directory: ```javascript export { Bold as default } from '@tiptap/extension-bold'; @@ -326,10 +326,10 @@ sequenceDiagram A->>E: setContent(document) ``` -Deserializers live in the extension modules. Read Tiptap's -[parseHTML](https://tiptap.dev/guide/custom-extensions#parse-html) and -[addAttributes](https://tiptap.dev/guide/custom-extensions#attributes) documentation to -learn how to implement them. Titap's API is a wrapper around ProseMirror's +Deserializers live in the extension modules. Read Tiptap documentation about +[`parseHTML`](https://tiptap.dev/guide/custom-extensions#parse-html) and +[`addAttributes`](https://tiptap.dev/guide/custom-extensions#attributes) to +learn how to implement them. The Tiptap API is a wrapper around ProseMirror's [schema spec API](https://prosemirror.net/docs/ref/#model.SchemaSpec). #### Serialization diff --git a/doc/development/fe_guide/customizable_dashboards.md b/doc/development/fe_guide/customizable_dashboards.md index 38ee750d421..807f83f5bec 100644 --- a/doc/development/fe_guide/customizable_dashboards.md +++ b/doc/development/fe_guide/customizable_dashboards.md @@ -41,16 +41,12 @@ export default { // All values are grid row/column numbers up to 12. // We use the default 12 column grid https://github.com/gridstack/gridstack.js#change-grid-columns. gridAttributes: { - size: { - height: 4, - width: 6, - minHeight: 4, - minWidth: 6, - }, - position: { - xPos: 0, - yPos: 0, - }, + height: 4, + width: 6, + minHeight: 4, + minWidth: 6, + xPos: 0, + yPos: 0, }, // Options that are used to set bespoke values for each widget. // Available customizations are determined by the widget itself. diff --git a/doc/development/fe_guide/dark_mode.md b/doc/development/fe_guide/dark_mode.md index d687d9740c9..55181edd64c 100644 --- a/doc/development/fe_guide/dark_mode.md +++ b/doc/development/fe_guide/dark_mode.md @@ -19,9 +19,9 @@ Note the following: - We define two types of variables in `_dark.scss`: - SCSS variables are used in framework, components, and utility classes. - CSS variables are used for any colors within the `app/assets/stylesheets/page_bundles` directory. -- `app/views/layouts/_head.html.haml` then loads application or application_dark based on the user's theme preference. +- `app/views/layouts/_head.html.haml` then loads `application` or `application_dark` based on the user's theme preference. -As we do not want to generate separate `_dark.css` variants of every page_bundle file, +As we do not want to generate separate `_dark.css` variants of every `page_bundle` file, we use CSS variables with SCSS variables as fallbacks. This is because when we generate the `page_bundles` CSS, we get the variable values from `_variables.scss`, so any SCSS variables have light mode values. diff --git a/doc/development/fe_guide/development_process.md b/doc/development/fe_guide/development_process.md index fc91ff55b24..232689080ea 100644 --- a/doc/development/fe_guide/development_process.md +++ b/doc/development/fe_guide/development_process.md @@ -45,7 +45,7 @@ Use your best judgment when to use it and contribute new points through merge re - [ ] Check off tasks on your created task list to keep everyone updated on the progress - [ ] [Share your work early with reviewers/maintainers](#share-your-work-early) -- [ ] Share your work with UXer and Product Manager with Screenshots and/or [GIF's](https://about.gitlab.com/handbook/product/making-gifs/). They are easy to create for you and keep them up to date. +- [ ] Share your work with UXer and Product Manager with Screenshots and/or [GIF images](https://about.gitlab.com/handbook/product/making-gifs/). They are easy to create for you and keep them up to date. - [ ] If you are blocked on something let everyone on the issue know through a comment. - [ ] Are you unable to work on this issue for a longer period of time, also let everyone know. - [ ] **Documentation** Update/add docs for the new feature, see `docs/`. Ping one of the documentation experts/reviewers @@ -58,7 +58,7 @@ Use your best judgment when to use it and contribute new points through merge re - [ ] Did you check the mobile view? - [ ] Check the built webpack bundle (For the report run `WEBPACK_REPORT=true gdk start`, then open `webpack-report/index.html`) if we have unnecessary bloat due to wrong references, including libraries multiple times, etc.. If you need help contact the webpack [domain expert](https://about.gitlab.com/handbook/engineering/frontend/#frontend-domain-experts) - [ ] **Tests** Not only greenfield tests - Test also all bad cases that come to your mind. -- [ ] If you have multiple MR's then also smoke test against the final merge. +- [ ] If you have multiple MRs then also smoke test against the final merge. - [ ] Are there any big changes on how and especially how frequently we use the API then let production know about it - [ ] Smoke test of the RC on dev., staging., canary deployments and .com - [ ] Follow up on issues that came out of the review. Create issues for discovered edge cases that should be covered in future iterations. diff --git a/doc/development/fe_guide/graphql.md b/doc/development/fe_guide/graphql.md index 28ea84301a6..0fec38b1200 100644 --- a/doc/development/fe_guide/graphql.md +++ b/doc/development/fe_guide/graphql.md @@ -109,7 +109,7 @@ Default client accepts two parameters: `resolvers` and `config`. If you are making multiple queries to the same Apollo client object you might encounter the following error: `Cache data may be lost when replacing the someProperty field of a Query object. To address this problem, either ensure all objects of SomeEntityhave an id or a custom merge function`. We are already checking `ID` presence for every GraphQL type that has an `ID`, so this shouldn't be the case. Most likely, the `SomeEntity` type doesn't have an `ID` property, and to fix this warning we need to define a custom merge function. -We have some client-wide types with `merge: true` defined in the default client as [typePolicies](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/javascripts/lib/graphql.js) (this means that Apollo will merge existing and incoming responses in the case of subsequent queries). Consider adding `SomeEntity` there or defining a custom merge function for it. +We have some client-wide types with `merge: true` defined in the default client as [`typePolicies`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/javascripts/lib/graphql.js) (this means that Apollo will merge existing and incoming responses in the case of subsequent queries). Consider adding `SomeEntity` there or defining a custom merge function for it. ## GraphQL Queries @@ -1138,7 +1138,7 @@ query getPipelineEtag { ```javascript /* pipeline_editor/components/header/pipeline_status.vue */ -import getPipelineEtag from '~/pipeline_editor/graphql/queries/client/pipeline_etag.query.graphql'; +import getPipelineEtag from '~/ci/pipeline_editor/graphql/queries/client/pipeline_etag.query.graphql'; apollo: { pipelineEtag: { diff --git a/doc/development/fe_guide/haml.md b/doc/development/fe_guide/haml.md index d76d718e8e5..9dc5b265783 100644 --- a/doc/development/fe_guide/haml.md +++ b/doc/development/fe_guide/haml.md @@ -81,11 +81,7 @@ When using the GitLab UI form builder, the following components are available fo NOTE: Currently only the listed components are available but more components are planned. -<!-- vale gitlab.Spelling = NO --> - -#### gitlab_ui_checkbox_component - -<!-- vale gitlab.Spelling = YES --> +#### `gitlab_ui_checkbox_component` [GitLab UI Docs](https://gitlab-org.gitlab.io/gitlab-ui/?path=/story/base-form-form-checkbox--default) @@ -110,11 +106,7 @@ This component supports [ViewComponent slots](https://viewcomponent.org/guide/sl | `label` | Checkbox label content. This slot can be used instead of the `label` argument. | | `help_text` | Help text content displayed below the checkbox. This slot can be used instead of the `help_text` argument. | -<!-- vale gitlab.Spelling = NO --> - -#### gitlab_ui_radio_component - -<!-- vale gitlab.Spelling = YES --> +#### `gitlab_ui_radio_component` [GitLab UI Docs](https://gitlab-org.gitlab.io/gitlab-ui/?path=/story/base-form-form-radio--default) diff --git a/doc/development/fe_guide/merge_request_widget_extensions.md b/doc/development/fe_guide/merge_request_widget_extensions.md index 61d3e79a080..49c6664c6d6 100644 --- a/doc/development/fe_guide/merge_request_widget_extensions.md +++ b/doc/development/fe_guide/merge_request_widget_extensions.md @@ -355,7 +355,12 @@ To generate these known events for a single widget: 1. `redis_slot` = `code_review` 1. `category` = `code_review` 1. `aggregation` = `weekly` -1. Add each event to the appropriate aggregates in `config/metrics/aggregates/code_review.yml` +1. Add each event (those listed in the command in step 7, replacing `test_reports` + with the appropriate name slug) to the aggregate files: + 1. `config/metrics/counts_7d/{timestamp}_code_review_category_monthly_active_users.yml` + 1. `config/metrics/counts_7d/{timestamp}_code_review_group_monthly_active_users.yml` + 1. `config/metrics/counts_28d/{timestamp}_code_review_category_monthly_active_users.yml` + 1. `config/metrics/counts_28d/{timestamp}_code_review_group_monthly_active_users.yml` ### Add new events diff --git a/doc/development/fe_guide/security.md b/doc/development/fe_guide/security.md index 4b7ce6d11e3..d578449e578 100644 --- a/doc/development/fe_guide/security.md +++ b/doc/development/fe_guide/security.md @@ -88,7 +88,7 @@ readability. If you need to output raw HTML, you should sanitize it. -If you are using Vue, you can use the[`v-safe-html` directive](https://gitlab-org.gitlab.io/gitlab-ui/?path=/story/directives-safe-html-directive--default) from GitLab UI. +If you are using Vue, you can use the[`v-safe-html` directive](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/javascripts/vue_shared/directives/safe_html.js). For other use cases, wrap a preconfigured version of [`dompurify`](https://www.npmjs.com/package/dompurify) that also allows the icons to be rendered: diff --git a/doc/development/fe_guide/source_editor.md b/doc/development/fe_guide/source_editor.md index b7cd903485e..4cfc68553e0 100644 --- a/doc/development/fe_guide/source_editor.md +++ b/doc/development/fe_guide/source_editor.md @@ -66,9 +66,9 @@ with additional functions on the instance level: | Function | Arguments | Description | --------------------- | ----- | ----- | -| `updateModelLanguage` | `path`: String | Updates the instance's syntax highlighting to follow the extension of the passed `path`. Available only on the instance level.| -| `use` | Array of objects | Array of extensions to apply to the instance. Accepts only the array of _objects_. You must fetch the extensions' ES6 modules must be fetched and resolved in your views or components before they are passed to `use`. This property is available on _instance_ (applies extension to this particular instance) and _global editor_ (applies the same extension to all instances) levels. | -| Monaco Editor options | See [documentation](https://microsoft.github.io/monaco-editor/api/interfaces/monaco.editor.IStandaloneCodeEditor.html) | Default Monaco editor options | +| `updateModelLanguage` | `path`: String | Updates the instance's syntax highlighting to follow the extension of the passed `path`. Available only on the instance level. | +| `use` | Array of objects | Array of extensions to apply to the instance. Accepts only an array of **objects**. The extensions' ES6 modules must be fetched and resolved in your views or components before they're passed to `use`. Available on the instance and global editor (all instances) levels. | +| Monaco Editor options | See [documentation](https://microsoft.github.io/monaco-editor/api/interfaces/monaco.editor.IStandaloneCodeEditor.html) | Default Monaco editor options. | ## Tips diff --git a/doc/development/fe_guide/style/javascript.md b/doc/development/fe_guide/style/javascript.md index 38fb926197b..0536d1c5c77 100644 --- a/doc/development/fe_guide/style/javascript.md +++ b/doc/development/fe_guide/style/javascript.md @@ -7,7 +7,7 @@ disqus_identifier: 'https://docs.gitlab.com/ee/development/fe_guide/style_guide_ # JavaScript style guide -We use [Airbnb's JavaScript Style Guide](https://github.com/airbnb/javascript) and its accompanying +We use [the Airbnb JavaScript Style Guide](https://github.com/airbnb/javascript) and its accompanying linter to manage most of our JavaScript style guidelines. In addition to the style guidelines set by Airbnb, we also have a few specific rules @@ -16,11 +16,11 @@ listed below. NOTE: You can run ESLint locally by running `yarn run lint:eslint:all` or `yarn run lint:eslint $PATH_TO_FILE`. -## Avoid forEach +## Avoid `forEach` -Avoid forEach when mutating data. Use `map`, `reduce` or `filter` instead of `forEach` +Avoid `forEach` when mutating data. Use `map`, `reduce` or `filter` instead of `forEach` when mutating data. This minimizes mutations in functions, -which aligns with [Airbnb's style guide](https://github.com/airbnb/javascript#testing--for-real). +which aligns with [the Airbnb style guide](https://github.com/airbnb/javascript#testing--for-real). ```javascript // bad diff --git a/doc/development/fe_guide/style/scss.md b/doc/development/fe_guide/style/scss.md index 98f74813231..7a5c955db93 100644 --- a/doc/development/fe_guide/style/scss.md +++ b/doc/development/fe_guide/style/scss.md @@ -41,13 +41,13 @@ GitLab differs from the scale used in the Bootstrap library. For a Bootstrap pad utility, you may need to double the size of the applied utility to achieve the same visual result (such as `ml-1` becoming `gl-ml-2`). -#### Where should I put new utility classes? +#### Where should you put new utility classes? -If a class you need has not been added to GitLab UI, you get to add it! Follow the naming patterns documented in the [utility files](https://gitlab.com/gitlab-org/gitlab-ui/-/tree/main/src/scss/utility-mixins) and refer to [GitLab UI's CSS documentation](https://gitlab.com/gitlab-org/gitlab-ui/-/blob/main/doc/contributing/adding_css.md#adding-utility-mixins) for more details, especially about adding responsive and stateful rules. +If a class you need has not been added to GitLab UI, you get to add it! Follow the naming patterns documented in the [utility files](https://gitlab.com/gitlab-org/gitlab-ui/-/tree/main/src/scss/utility-mixins) and refer to the [GitLab UI CSS documentation](https://gitlab.com/gitlab-org/gitlab-ui/-/blob/main/doc/contributing/adding_css.md#adding-utility-mixins) for more details, especially about adding responsive and stateful rules. If it is not possible to wait for a GitLab UI update (generally one day), add the class to [`utilities.scss`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/stylesheets/utilities.scss) following the same naming conventions documented in GitLab UI. A follow-up issue to backport the class to GitLab UI and delete it from GitLab should be opened. -#### When should I create component classes? +#### When should you create component classes? We recommend a "utility-first" approach. diff --git a/doc/development/fe_guide/tooling.md b/doc/development/fe_guide/tooling.md index c9efb8e939d..762ef852d74 100644 --- a/doc/development/fe_guide/tooling.md +++ b/doc/development/fe_guide/tooling.md @@ -212,7 +212,7 @@ yarn run lint:prettier:fix Formats all files in the repository with Prettier. -### VSCode Settings +### VS Code Settings #### Select Prettier as default formatter diff --git a/doc/development/fe_guide/troubleshooting.md b/doc/development/fe_guide/troubleshooting.md index 873a14b8f14..3e4c5007b64 100644 --- a/doc/development/fe_guide/troubleshooting.md +++ b/doc/development/fe_guide/troubleshooting.md @@ -10,13 +10,13 @@ Running into a problem? Maybe this will help ¯\＿(ツ)＿/¯. ## Troubleshooting issues -### This guide doesn't contain the issue I ran into +### This guide doesn't contain the issue you ran into If you run into a Frontend development issue that is not in this guide, consider updating this guide with your issue and possible remedies. This way future adventurers can face these dragons with more success, being armed with your experience and knowledge. ## Testing issues -### ``Property or method `nodeType` is not defined`` but I'm not using `nodeType` anywhere +### ``Property or method `nodeType` is not defined`` but you're not using `nodeType` anywhere This issue can happen in Vue component tests, when an expectation fails, but there is an error thrown when Jest tries to pretty print the diff in the console. It's been noted that using `toEqual` with an array as a diff --git a/doc/development/fe_guide/view_component.md b/doc/development/fe_guide/view_component.md index b61c23cadef..90bb75514d8 100644 --- a/doc/development/fe_guide/view_component.md +++ b/doc/development/fe_guide/view_component.md @@ -180,7 +180,7 @@ The `Pajamas::CheckboxComponent` follows the [Pajamas Checkbox](https://design.g NOTE: `Pajamas::CheckboxComponent` is used internally by the [GitLab UI form builder](haml.md#use-the-gitlab-ui-form-builder) and requires an instance of [ActionView::Helpers::FormBuilder](https://api.rubyonrails.org/v6.1.0/classes/ActionView/Helpers/FormBuilder.html) to be passed as the `form` argument. -It is preferred to use the [gitlab_ui_checkbox_component](haml.md#gitlab_ui_checkbox_component) method to render this ViewComponent. +It is preferred to use the [`gitlab_ui_checkbox_component`](haml.md#gitlab_ui_checkbox_component) method to render this ViewComponent. To use a checkbox without an instance of [ActionView::Helpers::FormBuilder](https://api.rubyonrails.org/v6.1.0/classes/ActionView/Helpers/FormBuilder.html) use [CheckboxTagComponent](#checkbox-tag). For the full list of options, see its diff --git a/doc/development/fe_guide/vue3_migration.md b/doc/development/fe_guide/vue3_migration.md index aae7674d190..b9e819a95bd 100644 --- a/doc/development/fe_guide/vue3_migration.md +++ b/doc/development/fe_guide/vue3_migration.md @@ -182,13 +182,13 @@ In the step-by-step guide, we will be migrating [VueApollo Demo](https://gitlab. #### Initial state -Right after cloning, you could run [VueApollo Demo](https://gitlab.com/gitlab-org/frontend/vue3-migration-vue-apollo/-/tree/main/src/vue3compat) with Vue.js 2 using `yarn serve` or with Vue.js 3 (compat build) using `yarn serve:vue3`. However latter immediately crashes: +Right after cloning, you could run [VueApollo Demo](https://gitlab.com/gitlab-org/frontend/vue3-migration-vue-apollo/-/tree/main/src/vue3compat) with Vue.js 2 using `yarn serve` or with Vue.js 3 (`compat` build) using `yarn serve:vue3`. However latter immediately crashes: ```javascript Uncaught TypeError: Cannot read properties of undefined (reading 'loading') ``` -VueApollo v3 (used for Vue.js 2) fails to initialize in Vue.js compat +VueApollo v3 (used for Vue.js 2) fails to initialize in Vue.js `compat` NOTE: While stubbing `Vue.version` will solve VueApollo-related issues in the demo project, it will still lose reactivity on specific scenarios, so an upgrade is still needed diff --git a/doc/development/fe_guide/widgets.md b/doc/development/fe_guide/widgets.md index 3364c778d76..edb8559da48 100644 --- a/doc/development/fe_guide/widgets.md +++ b/doc/development/fe_guide/widgets.md @@ -44,7 +44,7 @@ All editable sidebar widgets should use [`SidebarEditableItem`](https://gitlab.c We aim to make widgets as reusable as possible. That's why we should avoid adding any external state bindings to widgets or to their child components. This includes Vuex mappings and mediator stores. -## Widget's responsibility +## Widget responsibility A widget is responsible for fetching and updating an entity it's designed for (assignees, iterations, and so on). This means a widget should **always** fetch data (if it's not in Apollo cache already). @@ -78,7 +78,7 @@ To handle the same logic for query updates, we **alias** query fields. For examp - `group` or `project` become `workspace` - `issue`, `epic`, or `mergeRequest` become `issuable` -Unfortunately, Apollo assigns aliased fields a typename of `undefined`, so we need to fetch `__typename` explicitly: +Unfortunately, Apollo assigns aliased fields a `typename` of `undefined`, so we need to fetch `__typename` explicitly: ```plaintext query issueConfidential($fullPath: ID!, $iid: String) { diff --git a/doc/development/feature_categorization/index.md b/doc/development/feature_categorization/index.md index 0bf506b53ba..47663915ea7 100644 --- a/doc/development/feature_categorization/index.md +++ b/doc/development/feature_categorization/index.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Introduced](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/269) in GitLab 13.2. -Each Sidekiq worker, controller action, or API endpoint +Each Sidekiq worker, controller action, [test example](../testing_guide/best_practices.md#feature-category-metadata) or API endpoint must declare a `feature_category` attribute. This attribute maps each of these to a [feature category](https://about.gitlab.com/handbook/product/categories/). This is done for error budgeting, alert routing, and team attribution. @@ -166,3 +166,40 @@ end As with Rails controllers, an API class must specify the category for every single action unless the same category is used for every action within that class. + +## RSpec Examples + +You must set feature category metadata for each RSpec example. This information is used for flaky test +issues to identify the group that owns the feature. + +The `feature_category` should be a value from [`categories.json`](https://about.gitlab.com/categories.json). + +The `feature_category` metadata can be set: + +- [In the top-level `RSpec.describe` blocks](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/104274/diffs#6bd01173381e873f3e1b6c55d33cdaa3d897156b_5_5). +- [In `describe` blocks](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/104274/diffs#a520db2677a30e7f1f5593584f69c49031b894b9_12_12). + +Consider splitting the file in the case there are multiple feature categories identified in the same file. + +Example: + + ```ruby + RSpec.describe Admin::Geo::SettingsController, :geo, feature_category: :geo_replication do + ``` + +For examples that don't have a `feature_category` set we add a warning when running them in local environment. + +In order to disable the warning use `RSPEC_WARN_MISSING_FEATURE_CATEGORY=false` when running RSpec tests: + +```shell +RSPEC_WARN_MISSING_FEATURE_CATEGORY=false bin/rspec spec/<test_file> +``` + +### Excluding specs from feature categorization + +In the rare case an action cannot be tied to a feature category this +can be done using the `not_owned` feature category. + +```ruby +RSpec.describe Utils, feature_category: :not_owned do +``` diff --git a/doc/development/feature_development.md b/doc/development/feature_development.md index 76447124177..b9e093b9479 100644 --- a/doc/development/feature_development.md +++ b/doc/development/feature_development.md @@ -19,7 +19,7 @@ Consult these topics for information on contributing to specific GitLab features ### General -- [Directory structure](directory_structure.md) +- [Software design guides](software_design.md) - [GitLab EventStore](event_store.md) to publish/subscribe to domain events - [GitLab utilities](utilities.md) - [Newlines style guide](newlines_styleguide.md) @@ -54,7 +54,7 @@ Consult these topics for information on contributing to specific GitLab features ### Debugging - [Pry debugging](pry_debugging.md) -- [Sidekiq debugging](../administration/troubleshooting/sidekiq.md) +- [Sidekiq debugging](../administration/sidekiq/sidekiq_troubleshooting.md) ### Git specifics @@ -127,7 +127,7 @@ See [database guidelines](database/index.md). - [Security Scanners](integrations/secure.md) - [Secure Partner Integration](integrations/secure_partner_integration.md) - [How to run Jenkins in development environment](integrations/jenkins.md) -- [How to run local `Codesandbox` integration for Web IDE Live Preview](integrations/codesandbox.md) +- [How to run local CodeSandbox integration for Web IDE Live Preview](integrations/codesandbox.md) The following integration guides are internal. Some integrations require access to administrative accounts of third-party services and are available only for GitLab team members to contribute to: @@ -204,3 +204,4 @@ The following integration guides are internal. Some integrations require access - [Run full Auto DevOps cycle in a GDK instance](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/main/doc/howto/auto_devops.md) - [Using GitLab Runner with the GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/main/doc/howto/runner.md) - [Using the Web IDE terminal with the GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/web_ide_terminal_gdk_setup.md) +- [Gitpod configuration internals page](gitpod_internals.md) diff --git a/doc/development/feature_flags/controls.md b/doc/development/feature_flags/controls.md index e804a888e98..3e6491a92b5 100644 --- a/doc/development/feature_flags/controls.md +++ b/doc/development/feature_flags/controls.md @@ -67,10 +67,8 @@ a (very) rough estimate of how your feature will look and behave on GitLab.com. Both of these instances are connected to Sentry so make sure you check the projects there for any exceptions while testing your feature after enabling the feature flag. -For these pre-production environments, the commands should be run in a -Slack channel for the stage the feature is relevant to. For example, use the -`#s_monitor` channel for features developed by the Monitor stage, Health -group. +For these pre-production environments, it's strongly encouraged to run the command in +`#staging`, `#production`, or `#chatops-ops-test`, for improved visibility. To enable a feature for 25% of the time, run the following in Slack: @@ -289,10 +287,47 @@ To disable a feature flag that has been enabled for a specific project you can r /chatops run feature set --project=gitlab-org/gitlab some_feature false ``` -You cannot selectively disable feature flags for a specific project/group/user without applying a [specific method of implementing](index.md#selectively-disable-by-actor) the feature flags. +You cannot selectively disable feature flags for a specific project/group/user without applying a [specific method of implementing](controls.md#selectively-disable-by-actor) the feature flags. If a feature flag is disabled via ChatOps, that will take precedence over the `default_enabled` value in the YAML. In other words, you could have a feature enabled for on-premise installations but not for GitLab.com. +#### Selectively disable by actor + +By default you cannot selectively disable a feature flag by actor. + +```shell +# This will not work how you would expect. +/chatops run feature set some_feature true +/chatops run feature set --project=gitlab-org/gitlab some_feature false +``` + +However, if you add two feature flags, you can write your conditional statement in such a way that the equivalent selective disable is possible. + +```ruby +Feature.enabled?(:a_feature, project) && Feature.disabled?(:a_feature_override, project) +``` + +```shell +# This will enable a feature flag globally, except for gitlab-org/gitlab +/chatops run feature set a_feature true +/chatops run feature set --project=gitlab-org/gitlab a_feature_override true +``` + +#### Percentage-based actor selection + +When using the percentage rollout of actors on multiple feature flags, the actors for each feature flag are selected separately. + +For example, the following feature flags are enabled for a certain percentage of actors: + +```plaintext +/chatops run feature set feature-set-1 25 --actors +/chatops run feature set feature-set-2 25 --actors +``` + +If a project A has `:feature-set-1` enabled, there is no guarantee that project A also has `:feature-set-2` enabled. + +For more detail, see [This is how percentages work in Flipper](https://www.hackwithpassion.com/this-is-how-percentages-work-in-flipper/). + ### Feature flag change logging #### ChatOps level diff --git a/doc/development/feature_flags/index.md b/doc/development/feature_flags/index.md index 500afa8ba1d..5ff4292dfb6 100644 --- a/doc/development/feature_flags/index.md +++ b/doc/development/feature_flags/index.md @@ -187,7 +187,7 @@ Only feature flags that have a YAML definition file can be used when running the ```shell $ bin/feature-flag my_feature_flag >> Specify the group introducing the feature flag, like `group::apm`: -?> group::memory +?> group::application performance >> URL of the MR introducing the feature flag (enter to skip): ?> https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38602 @@ -202,7 +202,7 @@ create config/feature_flags/development/my_feature_flag.yml name: my_feature_flag introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38602 rollout_issue_url: https://gitlab.com/gitlab-org/gitlab/-/issues/232533 -group: group::memory +group: group::application performance type: development default_enabled: false ``` @@ -398,44 +398,8 @@ Feature.enabled?(:feature_flag, group) Feature.enabled?(:feature_flag, user) ``` -Please see [Feature flag controls](controls.md#process) for more details on working with feature flags. - -#### Selectively disable by actor - -By default you cannot selectively disable a feature flag by actor. - -```shell -# This will not work how you would expect. -/chatops run feature set some_feature true -/chatops run feature set --project=gitlab-org/gitlab some_feature false -``` - -However, if you add two feature flags, you can write your conditional statement in such a way that the equivalent selective disable is possible. - -```ruby -Feature.enabled?(:a_feature, project) && Feature.disabled?(:a_feature_override, project) -``` - -```shell -# This will enable a feature flag globally, except for gitlab-org/gitlab -/chatops run feature set a_feature true -/chatops run feature set --project=gitlab-org/gitlab a_feature_override true -``` - -#### Percentage-based actor selection - -When using the percentage rollout of actors on multiple feature flags, the actors for each feature flag are selected separately. - -For example, the following feature flags are enabled for a certain percentage of actors: - -```plaintext -/chatops run feature set feature-set-1 25 --actors -/chatops run feature set feature-set-2 25 --actors -``` - -If a project A has `:feature-set-1` enabled, there is no guarantee that project A also has `:feature-set-2` enabled. - -For more detail, see [This is how percentages work in Flipper](https://www.hackwithpassion.com/this-is-how-percentages-work-in-flipper/). +See [Feature flags in the development of GitLab](controls.md#process) for details on how to use ChatOps +to selectively enable or disable feature flags in GitLab-provided environments, like staging and production. #### Use actors for verifying in production diff --git a/doc/development/file_storage.md b/doc/development/file_storage.md index 11acb8a2161..c346d55f639 100644 --- a/doc/development/file_storage.md +++ b/doc/development/file_storage.md @@ -36,7 +36,7 @@ There are many places where file uploading is used, according to contexts: GitLab started saving everything on local disk. While directory location changed from previous versions, they are still not 100% standardized. You can see them below: -| Description | In DB? | Relative path (from CarrierWave.root) | Uploader class | model_type | +| Description | In DB? | Relative path (from CarrierWave.root) | Uploader class | Model type | | ------------------------------------- | ------ | ----------------------------------------------------------- | ---------------------- | ---------- | | Instance logo | yes | `uploads/-/system/appearance/logo/:id/:filename` | `AttachmentUploader` | Appearance | | Header logo | yes | `uploads/-/system/appearance/header_logo/:id/:filename` | `AttachmentUploader` | Appearance | diff --git a/doc/development/filtering_by_label.md b/doc/development/filtering_by_label.md deleted file mode 100644 index 675fe004c22..00000000000 --- a/doc/development/filtering_by_label.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'database/filtering_by_label.md' -remove_date: '2022-11-05' ---- - -This document was moved to [another location](database/filtering_by_label.md). - -<!-- This redirect file can be deleted after <2022-11-05>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/fips_compliance.md b/doc/development/fips_compliance.md index c6208d45c77..187a9b0cc93 100644 --- a/doc/development/fips_compliance.md +++ b/doc/development/fips_compliance.md @@ -69,6 +69,7 @@ listed here that also do not work properly in FIPS mode: when operating in FIPS-compliant mode. - Advanced Search is currently not included in FIPS mode. It must not be enabled to be FIPS-compliant. - [Gravatar or Libravatar-based profile images](../administration/libravatar.md) are not FIPS-compliant. +- [Personal Access Tokens](../user/profile/personal_access_tokens.md) are not available for use or creation. Additionally, these package repositories are disabled in FIPS mode: @@ -441,13 +442,27 @@ def default_min_key_size(name) end ``` -## Nightly Omnibus FIPS builds +## Omnibus FIPS packages -The Distribution team has created [nightly FIPS Omnibus builds](https://packages.gitlab.com/gitlab/nightly-fips-builds). These -GitLab builds are compiled to use the system OpenSSL instead of the Omnibus-embedded version of OpenSSL. +GitLab has a dedicated repository +([`gitlab/gitlab-fips`](https://packages.gitlab.com/gitlab/gitlab-fips)) +for builds of the Omnibus GitLab which are built with FIPS compliance. +These GitLab builds are compiled to use the system OpenSSL, instead of +the Omnibus-embedded version of OpenSSL. These packages are built for: + +- RHEL 8 (and compatible) +- AmazonLinux 2 +- Ubuntu + +These are [consumed by the GitLab Environment Toolkit](#install-gitlab-with-fips-compliance) (GET). See [the section on how FIPS builds are created](#how-fips-builds-are-created). +### Nightly Omnibus FIPS builds + +The Distribution team has created [nightly FIPS Omnibus builds](https://packages.gitlab.com/gitlab/nightly-fips-builds), +which can be used for *testing* purposes. These should never be used for production environments. + ## Runner See the [documentation on installing a FIPS-compliant GitLab Runner](https://docs.gitlab.com/runner/install/#fips-compliant-gitlab-runner). diff --git a/doc/development/foreign_keys.md b/doc/development/foreign_keys.md deleted file mode 100644 index cdf655bf0bf..00000000000 --- a/doc/development/foreign_keys.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'database/foreign_keys.md' -remove_date: '2022-11-05' ---- - -This document was moved to [another location](database/foreign_keys.md). - -<!-- This redirect file can be deleted after <2022-11-05>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/gemfile.md b/doc/development/gemfile.md index 3c7dc19da8e..36ef1bcd834 100644 --- a/doc/development/gemfile.md +++ b/doc/development/gemfile.md @@ -47,7 +47,8 @@ This needs to be done for any new, or updated gems. We do not allow gems that are fetched from Git repositories. All gems have to be available in the RubyGems index. We want to minimize external build -dependencies and build times. +dependencies and build times. It's enforced by the RuboCop rule +[`Cop/GemFetcher`](https://gitlab.com/gitlab-org/ruby/gems/gitlab-styles/-/blob/master/lib/rubocop/cop/gem_fetcher.rb). ## Review the new dependency for quality @@ -56,7 +57,7 @@ This means that new dependencies should, at a minimum, meet the following criter - They have an active developer community. At the minimum a maintainer should still be active to merge change requests in case of emergencies. -- There are no issues open that we know may impact the availablity or performance of GitLab. +- There are no issues open that we know may impact the availability or performance of GitLab. - The project is tested using some form of test automation. The test suite must be passing using the Ruby version currently used by GitLab. - If the project uses a C extension, consider requesting an additional review from a C or MRI diff --git a/doc/development/geo.md b/doc/development/geo.md index 884c09cc174..76c75cb1c6a 100644 --- a/doc/development/geo.md +++ b/doc/development/geo.md @@ -200,7 +200,7 @@ sequenceDiagram S->>TDB: Insert to `job_artifact_registry` ``` -- [Sidekiq-cron](https://github.com/ondrejbartas/sidekiq-cron) enqueues a `Geo::Secondary::RegistryConsistencyWorker` job every minute. As long as it is actively doing work (creating and deleting rows), this job immediately reenqueues itself. This job uses an exclusive lease to prevent multiple instances of itself from running simultaneously. +- [Sidekiq-cron](https://github.com/ondrejbartas/sidekiq-cron) enqueues a `Geo::Secondary::RegistryConsistencyWorker` job every minute. As long as it is actively doing work (creating and deleting rows), this job immediately re-enqueues itself. This job uses an exclusive lease to prevent multiple instances of itself from running simultaneously. - [Sidekiq](architecture.md#sidekiq) picks up `Geo::Secondary::RegistryConsistencyWorker` job - Sidekiq queries `ci_job_artifacts` table for up to 10000 rows - Sidekiq queries `job_artifact_registry` table for up to 10000 rows @@ -308,7 +308,7 @@ sequenceDiagram - Sidekiq queries `job_artifact_registry` in the [PostgreSQL Geo Tracking Database](#tracking-database) for the number of rows marked "pending verification" or "failed verification and ready to retry" - Sidekiq enqueues one or more `Geo::VerificationBatchWorker` jobs, limited by the "maximum verification concurrency" setting - Sidekiq picks up `Geo::VerificationBatchWorker` job - - Sidekiq queries `job_artifact_registry` in the PostgreSQL Geo Tracking Databasef for rows marked "pending verification" + - Sidekiq queries `job_artifact_registry` in the PostgreSQL Geo Tracking Database for rows marked "pending verification" - If the previous step yielded less than 10 rows, then Sidekiq queries `job_artifact_registry` for rows marked "failed verification and ready to retry" - For each row - Sidekiq marks it "started verification" @@ -588,23 +588,45 @@ When some write actions are not allowed because the site is a The database itself will already be read-only in a replicated setup, so we don't need to take any extra step for that. -## Steps needed to replicate a new data type - -As GitLab evolves, we constantly need to add new resources to the Geo replication system. -The implementation depends on resource specifics, but there are several things -that need to be taken care of: - -- Event generation on the primary site. Whenever a new resource is changed/updated, we need to - create a task for the Log Cursor. -- Event handling. The Log Cursor needs to have a handler for every event type generated by the primary site. -- Dispatch worker (cron job). Make sure the backfill condition works well. -- Sync worker. -- Registry with all possible states. -- Verification. -- Cleaner. When sync settings are changed for the secondary site, some resources need to be cleaned up. -- Geo Node Status. We need to provide API endpoints as well as some presentation in the GitLab Admin Area. -- Health Check. If we can perform some pre-cheсks and make site unhealthy if something is wrong, we should do that. - The `rake gitlab:geo:check` command has to be updated too. +## Ensuring a new feature has Geo support + +Geo depends on PostgreSQL replication of the main and CI databases, so if you add a new table or field, it should already work on secondary Geo sites. + +However, if you introduce a new kind of data which is stored outside of the main and CI PostgreSQL databases, then you need to ensure that this data is replicated and verified by Geo. This is necessary for customers to be able to rely on their secondary sites for [disaster recovery](../administration/geo/disaster_recovery/index.md). + +The following subsections describe how to determine whether work is needed, and if so, how to proceed. If you have any questions, [contact the Geo team](https://about.gitlab.com/handbook/product/categories/#geo-group). + +For comparison with your own features, see [Supported Geo data types](../administration/geo/replication/datatypes.md). It has a detailed, up-to-date list of the types of data that Geo replicates and verifies. + +### Git repositories + +If you add a feature that is backed by Git repositories, then you must add Geo support. See [the repository replicator strategy of the Geo self-service framework](geo/framework.md#repository-replicator-strategy). + +### Blobs + +If you add a subclass of `CarrierWave::Uploader::Base`, then you are adding what Geo calls a blob. If you specifically subclass [`AttachmentUploader` as generally recommended](uploads/working_with_uploads.md#recommendations), then the data has Geo support with no work needed. This is because `AttachmentUploader` tracks blobs with the `Upload` model using the `uploads` table, and Geo support is already implemented for that model. + +If your blobs are tracked in a new table, perhaps because you expect millions of rows at GitLab.com scale, then you must add Geo support. See [the blob replicator strategy of the Geo self-service framework](geo/framework.md#blob-replicator-strategy). + +[Geo detects new blobs with a spec](https://gitlab.com/gitlab-org/gitlab/-/blob/eeba0e4d231ae39012a5bbaeac43a72c2bd8affb/ee/spec/uploaders/every_gitlab_uploader_spec.rb) that fails when an `Uploader` does not have a corresponding `Replicator`. + +### Features with more than one kind of data + +If a new complex feature is backed by multiple kinds of data, for example, a Git repository and a blob, then you can likely consider each kind of data separately. + +Taking [Designs](../user/project/issues/design_management.md) as an example, each issue has a Git repository which can have many LFS objects, and each LFS object may have an automatically generated thumbnail. + +- LFS objects were already supported by Geo, so no Geo-specific work was needed. +- The implementation of thumbnails reused the `Upload` model, so no Geo-specific work was needed. +- Design Git repositories were not inherently supported by Geo, so work was needed. + +As another example, [Dependency Proxy](../administration/packages/dependency_proxy.md) is backed by two kinds of blobs, `DependencyProxy::Blob` and `DependencyProxy::Manifest`. We can use [the blob replicator strategy of the Geo self-service framework](geo/framework.md#blob-replicator-strategy) on each type, independent of each other. + +### Other kinds of data + +If a new feature introduces a new kind of data which is not a Git repository, or a blob, or a combination of the two, then contact the Geo team to discuss how to handle it. + +As an example, Container Registry data does not easily fit into the above categories. It is backed by a registry service which owns the data, and GitLab interacts with the registry service's API. So a one off approach is required for Geo support of Container Registry. Still, we are able to reuse much of the glue code of [the Geo self-service framework](geo/framework.md#repository-replicator-strategy). ## History of communication channel diff --git a/doc/development/geo/framework.md b/doc/development/geo/framework.md index 3624d280f86..60529db5ce6 100644 --- a/doc/development/geo/framework.md +++ b/doc/development/geo/framework.md @@ -18,6 +18,14 @@ across Geo sites. This API is presented as a Ruby Domain-Specific Language (DSL) and aims to make it possible to replicate data with minimal effort of the engineer who created a data type. +## Geo is a requirement in the definition of done + +Geo is the GitLab solution for [disaster recovery](https://about.gitlab.com/direction/geo/disaster_recovery/). A robust disaster recovery solution must replicate **all GitLab data** such that all GitLab services can be successfully restored in their entirety with minimal data loss in the event of a disaster. + +For this reason, Geo replication and verification support for GitLab generated data is part of the [definition of done](../contributing/merge_request_workflow.md#definition-of-done). This ensures that new features ship with Geo support and our customers are not exposed to data loss. + +Adding Geo support with the Self Service Framework (SSF) is easy and outlined in detail on this page for various types of data. However, for a more general guide that can help you decide if and how you need to add Geo support for a new GitLab feature, [you may start here](../geo.md#ensuring-a-new-feature-has-geo-support). + ## Nomenclature Before digging into the API, developers need to know some Geo-specific diff --git a/doc/development/geo/proxying.md b/doc/development/geo/proxying.md index 67d4129a9d0..f3136890788 100644 --- a/doc/development/geo/proxying.md +++ b/doc/development/geo/proxying.md @@ -260,12 +260,7 @@ S-->>C: return Git response from primary ## Git push -### Unified URLs - -With unified URLs, a push will redirect to a local path formatted as `/-/push_from_secondary/$SECONDARY_ID/*`. Further -requests through this path will be proxied to the primary, which will handle the push. - -#### Git push over SSH +### Git push over SSH As SSH operations go through GitLab Shell instead of Workhorse, they are not proxied through the mechanism used for Workhorse requests. With SSH operations, they are proxied as Git HTTP requests to the primary site by the secondary @@ -293,7 +288,12 @@ I-->>S: <response> S-->>C: return Git response from primary ``` -#### Git push over HTTP(s) +### Git push over HTTP(S) + +#### Git push over HTTP(S) unified URLs + +With unified URLs, a push redirects to a local path formatted as `/-/push_from_secondary/$SECONDARY_ID/*`. Further +requests through this path are proxied to the primary, which will handle the push. ```mermaid sequenceDiagram @@ -326,7 +326,7 @@ W-->>Wsec: Pipe messages to the Git client Wsec-->>C: Return piped messages from Git ``` -### Git push over HTTP with Separate URLs +#### Git push over HTTP(S) with separate URLs With separate URLs, the secondary will redirect to a URL formatted like `$PRIMARY/-/push_from_secondary/$SECONDARY_ID/*`. diff --git a/doc/development/git_object_deduplication.md b/doc/development/git_object_deduplication.md index dcfcd2e864a..6014ccbfb39 100644 --- a/doc/development/git_object_deduplication.md +++ b/doc/development/git_object_deduplication.md @@ -171,7 +171,7 @@ There are three different things that can go wrong here. #### 1. SQL says repository A belongs to pool P but Gitaly says A has no alternate objects -In this case, we miss out on disk space savings but all RPC's on A +In this case, we miss out on disk space savings but all RPCs on A itself function fine. The next time garbage collection runs on A, the alternates connection gets established in Gitaly. This is done by `Projects::GitDeduplicationService` in GitLab Rails. diff --git a/doc/development/gitaly.md b/doc/development/gitaly.md index a570b5dd7eb..a23f1dd2d80 100644 --- a/doc/development/gitaly.md +++ b/doc/development/gitaly.md @@ -56,7 +56,7 @@ they are merged. ### `gitaly-ruby` -It is possible to implement and test RPC's in Gitaly using Ruby code, +It is possible to implement and test RPCs in Gitaly using Ruby code, in [`gitaly-ruby`](https://gitlab.com/gitlab-org/gitaly/tree/master/ruby). This should make it easier to contribute for developers who are less @@ -324,7 +324,7 @@ the integration by using GDK: 1. The state of the flag must be observable. To check it, you must enable it by fetching the Prometheus metrics: - 1. Navigate to GDK's root directory. + 1. Navigate to the GDK root directory. 1. Make sure you have the proper branch checked out for Gitaly. 1. Recompile it with `make gitaly-setup` and restart the service with `gdk restart gitaly`. 1. Make sure your setup is running: `gdk status | grep praefect`. @@ -342,7 +342,7 @@ the integration by using GDK: 1. After you observe the metrics for the new feature flag and it increments, you can enable the new feature: - 1. Navigate to GDK's root directory. + 1. Navigate to the GDK root directory. 1. Start a Rails console: ```shell diff --git a/doc/development/github_importer.md b/doc/development/github_importer.md index edf78d5f83d..17b4a28f57d 100644 --- a/doc/development/github_importer.md +++ b/doc/development/github_importer.md @@ -192,7 +192,7 @@ briefly waits for jobs to complete before deciding what the next action should be. For small projects, this may slow down the import process a bit, but it also reduces pressure on the system as a whole. -## Refreshing import JIDs +## Refreshing import job IDs GitLab includes a worker called `Gitlab::Import::StuckProjectImportJobsWorker` that periodically runs and marks project imports as failed if they have been @@ -203,7 +203,7 @@ often we hit the GitHub rate limit (more on this below), but we don't want To prevent this from happening we periodically refresh the expiration time of the import process. This works by storing the JID of the import job in the -database, then refreshing this JID's TTL at various stages throughout the import +database, then refreshing this JID TTL at various stages throughout the import process. This is done by calling `ProjectImportState#refresh_jid_expiration`. By refreshing this TTL we can ensure our import does not get marked as failed so long we're still performing work. diff --git a/doc/development/gitlab_flavored_markdown/specification_guide/index.md b/doc/development/gitlab_flavored_markdown/specification_guide/index.md index 17afebcf6ee..85c6653180b 100644 --- a/doc/development/gitlab_flavored_markdown/specification_guide/index.md +++ b/doc/development/gitlab_flavored_markdown/specification_guide/index.md @@ -157,7 +157,7 @@ official specifications, but are part of the GitHub and GitLab internal Markdown implementations. These internal extensions are often dependent upon the GitHub or GitLab implementations or environments, and may depend upon metadata which is only available via interacting with those environments. For example, -[GitHub supports GitHub-specific autolinked references](https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/autolinked-references-and-urls), +[GitHub supports GitHub-specific automatically linked references](https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/autolinked-references-and-urls), and [GitLab also supports GitLab-specific references](../../../user/markdown.md#gitlab-specific-references). These may also be implemented by third-party Markdown rendering engines which integrate with @@ -458,7 +458,7 @@ You can see the RSpec shared context containing these fixtures in In some cases, fixtures may not be usable, because they do not provide control over the varying values. In these cases, we can introduce support for a environment variable into the production -code, which allows us to override the randommness in our test environment when we are +code, which allows us to override the randomness in our test environment when we are generating the HTML for footnote examples. Even though it is in the production code path, it has no effect unless it is explicitly set, therefore it is innocuous. It allows us to avoid the more-complex regex-based normalization described below. @@ -1056,7 +1056,7 @@ allows control over other aspects of the snapshot example generation process. the example will only be run by `ee/spec/requests/api/markdown_snapshot_spec.rb`, not by `spec/requests/api/markdown_snapshot_spec.rb`. - The `api_request_override_path` field overrides the API endpoint path which is used to - generate the `static` HTML for the specifed example. Different endpoints can generate different + generate the `static` HTML for the specified example. Different endpoints can generate different HTML in some cases, so we want to be able to exercise different API endpoints for the same Markdown. By default, the `/markdown` endpoint is used. @@ -1149,7 +1149,7 @@ These files are Markdown specification files containing examples generated based similar to the `output_spec/spec.txt` and `output_spec/spec.html`, with the following differences: 1. They contain a superset of _all_ examples from - the Commonmark, GitHub Flavored Markdown, and GitLab Flavored Markdown specifications, whereas + the CommonMark, GitHub Flavored Markdown, and GitLab Flavored Markdown specifications, whereas `spec.*` only contains the GLFM specification. This is to provide a single place to refer to all examples when working with [snapshot testing](#markdown-snapshot-testing). 1. They contain _only_ header sections which contain examples. They do not contain any prose-only diff --git a/doc/development/gitpod_internals.md b/doc/development/gitpod_internals.md new file mode 100644 index 00000000000..a4674df758d --- /dev/null +++ b/doc/development/gitpod_internals.md @@ -0,0 +1,30 @@ +--- +stage: none +group: Engineering Productivity +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Gitpod internal configuration + +## Settings + +The settings for `gitlab-org/gitlab` are defined under a [project's settings in a Gitpod dashboard](https://gitpod.io/t/gitlab-org/gitlab/settings). To view the settings, you must first join the `gitlab-org` team on [Gitpod.io](https://gitpod.io/). You can join the team using the bookmark link at the top of `#gitpod-gdk` internal Slack channel. + +The current settings are: + +- `Enable Incremental Prebuilds`: Uses an earlier successful prebuild to create new prebuilds faster. +- `Use Last Successful Prebuild`: Skips waiting for prebuilds currently in progress and uses the last successful prebuild from previous commits on the same branch. + +## Webhooks + +A webhook that starts with `https://gitpod.io/` is created to enable prebuilds (see [Enabling Prebuilds](https://www.gitpod.io/docs/configure/authentication/gitlab#enabling-prebuilds) for more details). The webhook is maintained by an [Engineering Productivity team](https://about.gitlab.com/handbook/engineering/quality/engineering-productivity/). + +You can find this webhook in [Webhook Settings in `gitlab-org/gitlab`](https://gitlab.com/gitlab-org/gitlab/-/hooks). If you cannot access this setting, please chat to the [Engineering Productivity team](https://about.gitlab.com/handbook/engineering/quality/engineering-productivity/). + +### Troubleshooting a failed webhook + +If a webhook failed to connect for a long time, then it may have been disabled in the project. + +To re-enable a failing or failed webhook, send a test request in [Webhook Settings](https://gitlab.com/gitlab-org/gitlab/-/hooks). See [Re-enable disabled webhooks page](https://docs.gitlab.com/15.4/ee/user/project/integrations/webhooks.html#re-enable-disabled-webhooks) for more details. + +After re-enabling, check the prebuilds' health in a [project's prebuilds](https://gitpod.io/t/gitlab-org/gitlab/prebuilds) and confirm that prebuilds start without any errors. diff --git a/doc/development/go_guide/index.md b/doc/development/go_guide/index.md index b561ebc4285..e7e628f5293 100644 --- a/doc/development/go_guide/index.md +++ b/doc/development/go_guide/index.md @@ -391,7 +391,7 @@ functionality: This gives us a thin abstraction over underlying implementations that is consistent across Workhorse, Gitaly, and, in future, other Go servers. For example, in the case of `gitlab.com/gitlab-org/labkit/tracing` we can switch -from using `Opentracing` directly to using `Zipkin` or Gokit's own tracing wrapper +from using `Opentracing` directly to using `Zipkin` or the Go kit's own tracing wrapper without changes to the application code, while still keeping the same consistent configuration mechanism (that is, the `GITLAB_TRACING` environment variable). diff --git a/doc/development/graphql_guide/batchloader.md b/doc/development/graphql_guide/batchloader.md index ef0b97f4f62..6a716de61b8 100644 --- a/doc/development/graphql_guide/batchloader.md +++ b/doc/development/graphql_guide/batchloader.md @@ -180,7 +180,7 @@ def resolve(args = {}, context = { current_user: current_user }) end ``` -We can also use [QueryRecorder](../query_recorder.md) to make sure we are performing only **one SQL query** per call. +We can also use [QueryRecorder](../database/query_recorder.md) to make sure we are performing only **one SQL query** per call. ```ruby it 'executes only 1 SQL query' do diff --git a/doc/development/hash_indexes.md b/doc/development/hash_indexes.md deleted file mode 100644 index 2a9f7e5a25d..00000000000 --- a/doc/development/hash_indexes.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'database/hash_indexes.md' -remove_date: '2022-11-06' ---- - -This document was moved to [another location](database/hash_indexes.md). - -<!-- This redirect file can be deleted after <2022-11-06>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/i18n/externalization.md b/doc/development/i18n/externalization.md index 91e2efcb2a3..2269a28e496 100644 --- a/doc/development/i18n/externalization.md +++ b/doc/development/i18n/externalization.md @@ -284,9 +284,7 @@ expect(wrapper.text()).toEqual(MSG_ALERT_SETTINGS_FORM_ERROR); ### Dynamic translations -Sometimes there are dynamic translations that the parser can't find when running -`bin/rake gettext:find`. For these scenarios you can use the [`N_` method](https://github.com/grosser/gettext_i18n_rails/blob/c09e38d481e0899ca7d3fc01786834fa8e7aab97/Readme.md#unfound-translations-with-rake-gettextfind). -There's also an alternative method to [translate messages from validation errors](https://github.com/grosser/gettext_i18n_rails/blob/c09e38d481e0899ca7d3fc01786834fa8e7aab97/Readme.md#option-a). +For more details you can see how we [keep translations dynamic](#keep-translations-dynamic). ## Working with special content @@ -764,6 +762,10 @@ class MyPresenter end ``` +Sometimes there are dynamic translations that the parser can't find when running +`bin/rake gettext:find`. For these scenarios you can use the [`N_` method](https://github.com/grosser/gettext_i18n_rails/blob/c09e38d481e0899ca7d3fc01786834fa8e7aab97/Readme.md#unfound-translations-with-rake-gettextfind). +There's also an alternative method to [translate messages from validation errors](https://github.com/grosser/gettext_i18n_rails/blob/c09e38d481e0899ca7d3fc01786834fa8e7aab97/Readme.md#option-a). + ### Splitting sentences Never split a sentence, as it assumes the sentence's grammar and structure is the same in all diff --git a/doc/development/image_scaling.md b/doc/development/image_scaling.md index 4b19c21a457..502a18fecd7 100644 --- a/doc/development/image_scaling.md +++ b/doc/development/image_scaling.md @@ -1,6 +1,6 @@ --- -stage: Data Stores -group: Application Performance +stage: Manage +group: Workspace info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -26,7 +26,7 @@ Whether we allow an image to be rescaled or not is decided by combination of har The hard-coded rules only permit: - [Project, group and user avatars](https://gitlab.com/gitlab-org/gitlab/-/blob/fd08748862a5fe5c25b919079858146ea85843ae/app/controllers/concerns/send_file_upload.rb#L65-67) -- [PNGs or JPEGs](https://gitlab.com/gitlab-org/gitlab/-/blob/5dff8fa3814f2a683d8884f468cba1ec06a60972/lib/gitlab/file_type_detection.rb#L23) +- [PNG or JPEG images](https://gitlab.com/gitlab-org/gitlab/-/blob/5dff8fa3814f2a683d8884f468cba1ec06a60972/lib/gitlab/file_type_detection.rb#L23) - [Specific dimensions](https://gitlab.com/gitlab-org/gitlab/-/blob/5dff8fa3814f2a683d8884f468cba1ec06a60972/app/models/concerns/avatarable.rb#L6) Furthermore, configuration in Workhorse can lead to the image scaler rejecting a request if: diff --git a/doc/development/import_project.md b/doc/development/import_project.md index b879d635350..7f5a0faf8fb 100644 --- a/doc/development/import_project.md +++ b/doc/development/import_project.md @@ -57,7 +57,7 @@ This method takes longer to import than the other methods and depends on several This script was introduced in GitLab 12.6 for importing large GitLab project exports. -As part of this script we also disable direct and background upload to avoid situations where a huge archive is being uploaded to GCS (while being inside a transaction, which can cause idle transaction timeouts). +As part of this script we also disable direct upload to avoid situations where a huge archive is being uploaded to GCS (while being inside a transaction, which can cause idle transaction timeouts). We can run this script from the terminal: diff --git a/doc/development/insert_into_tables_in_batches.md b/doc/development/insert_into_tables_in_batches.md deleted file mode 100644 index ced5332e880..00000000000 --- a/doc/development/insert_into_tables_in_batches.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'database/insert_into_tables_in_batches.md' -remove_date: '2022-11-05' ---- - -This document was moved to [another location](database/insert_into_tables_in_batches.md). - -<!-- This redirect file can be deleted after <2022-11-05>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/integrations/codesandbox.md b/doc/development/integrations/codesandbox.md index d7d0ea48db0..b7fe3fbd1c4 100644 --- a/doc/development/integrations/codesandbox.md +++ b/doc/development/integrations/codesandbox.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Set up local CodeSandbox development environment This guide walks through setting up a local [CodeSandbox repository](https://github.com/codesandbox/codesandbox-client) and integrating it with a local GitLab instance. CodeSandbox -is used to power the Web IDE's [Live Preview feature](../../user/project/web_ide/index.md#live-preview). Having a local CodeSandbox setup is useful for debugging upstream issues or +is used to power the Web IDE [Live Preview feature](../../user/project/web_ide/index.md#live-preview). Having a local CodeSandbox setup is useful for debugging upstream issues or creating upstream contributions like [this one](https://github.com/codesandbox/codesandbox-client/pull/5137). ## Initial setup @@ -43,7 +43,7 @@ Before using CodeSandbox with your local GitLab instance, you must: GitLab integrates with two parts of CodeSandbox: - An npm package called `smooshpack` (called `sandpack` in the `codesandbox-client` project). - This exposes an entrypoint for us to kick off Codesandbox's bundler. + This exposes an entrypoint for us to kick off CodeSandbox's bundler. - A server that houses CodeSandbox assets for bundling and previewing. This is hosted on a separate server for security. diff --git a/doc/development/integrations/index.md b/doc/development/integrations/index.md index 2639da818c6..1c9144a1163 100644 --- a/doc/development/integrations/index.md +++ b/doc/development/integrations/index.md @@ -58,7 +58,7 @@ end `Integration.prop_accessor` installs accessor methods on the class. Here we would have `#url`, `#url=` and `#url_changed?`, to manage the `url` field. Fields stored in `Integration#properties` should be accessed by these accessors directly on the model, just like other ActiveRecord attributes. -You should always access the properties through their getters, and not interact with the `properties` hash directly. +You should always access the properties through their `getters`, and not interact with the `properties` hash directly. You **must not** write to the `properties` hash, you **must** use the generated setter method instead. Direct writes to this hash are not persisted. @@ -124,6 +124,39 @@ module Integrations end ``` +## Define configuration test + +Optionally, you can define a configuration test of an integration's settings. The test is executed from the integration form's **Test** button, and results are returned to the user. + +A good configuration test: + +- Does not change data on the service. For example, it should not trigger a CI build. Sending a message is okay. +- Is meaningful and as thorough as possible. + +If it's not possible to follow the above guidelines, consider not adding a configuration test. + +To add a configuration test, define a `#test` method for the integration model. + +The method receives `data`, which is a test push event payload. +It should return a hash, containing the keys: + +- `success` (required): a boolean to indicate if the configuration test has passed. +- `result` (optional): a message returned to the user if the configuration test has failed. + +For example: + +```ruby +module Integrations + class FooBar < Integration + def test(data) + success = test_api_key(data) + + { success: success, result: 'API key is invalid' } + end + end +end +``` + ### Customize the frontend form The frontend form is generated dynamically based on metadata defined in the model. @@ -282,6 +315,8 @@ You can also refer to our general [documentation guidelines](../documentation/in ## Testing +Testing should not be confused with [defining configuration tests](#define-configuration-test). + It is often sufficient to add tests for the integration model in `spec/models/integrations`, and a factory with example settings in `spec/factories/integrations.rb`. diff --git a/doc/development/integrations/jira_connect.md b/doc/development/integrations/jira_connect.md index d4215662db4..c7bb77a6a5d 100644 --- a/doc/development/integrations/jira_connect.md +++ b/doc/development/integrations/jira_connect.md @@ -59,6 +59,19 @@ To install the app in Jira: _Note that any changes to the app descriptor requires you to uninstall then reinstall the app._ +## Simple setup + +To avoid external dependencies like Gitpod and a Jira Cloud instance, use the [Jira connect test tool](https://gitlab.com/gitlab-org/manage/integrations/jira-connect-test-tool) and your local GDK: + +1. Clone the [**Jira-connect-test-tool**](https://gitlab.com/gitlab-org/manage/integrations/jira-connect-test-tool) `git clone git@gitlab.com:gitlab-org/manage/integrations/jira-connect-test-tool.git`. +1. Start the app `bundle exec rackup`. (The app requires your GDK GitLab to be available on `http://127.0.0.1:3000`.). +1. Open `config/gitlab.yml` and uncomment the `jira_connect` config. +1. Restart GDK. +1. Go to `http://127.0.0.1:3000/-/profile/personal_access_tokens`. +1. Create a new token with the `api` scope and copy the token. +1. Go to `http://localhost:9292`. +1. Paste the token and select **Install GitLab.com Jira Cloud app**. + ### Troubleshooting If the app install failed, you might need to delete `jira_connect_installations` from your database. @@ -68,7 +81,7 @@ If the app install failed, you might need to delete `jira_connect_installations` #### Not authorized to access the file -If you use Gitpod and you get an error about Jira not being able to access the descriptor file, you might need to make the GDK's port public by following these steps: +If you use Gitpod and you get an error about Jira not being able to access the descriptor file, you might need to make the GDK port public by following these steps: 1. Open your GitLab workspace in Gitpod. 1. When the GDK is running, select **Ports** in the bottom-right corner. diff --git a/doc/development/integrations/secure.md b/doc/development/integrations/secure.md index 787b46133ad..190a6f6eda2 100644 --- a/doc/development/integrations/secure.md +++ b/doc/development/integrations/secure.md @@ -102,7 +102,7 @@ it's declared under the `reports:sast` key in the job definition, not because of ### Policies -Certain GitLab workflows, such as [AutoDevOps](../../topics/autodevops/customize.md#disable-jobs), +Certain GitLab workflows, such as [AutoDevOps](../../topics/autodevops/cicd_variables.md#job-disabling-variables), define CI/CD variables to indicate that given scans should be disabled. You can check for this by looking for variables such as: @@ -328,21 +328,6 @@ You can find the schemas for these scanners here: - [SAST](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/dist/sast-report-format.json) - [Secret Detection](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/dist/secret-detection-report-format.json) -### Retention period for vulnerabilities - -GitLab has the following retention policies for vulnerabilities on non-default branches. Vulnerabilities are no longer available: - -- When the related CI job artifact expires. -- 90 days after the pipeline is created, even if the related CI job artifacts are locked. - -To view vulnerabilities, either: - -- Run a new pipeline. -- Download the related CI job artifacts if they are available. - -NOTE: -This does not apply for the vulnerabilities existing on the default branch. - ### Report validation > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/351000) in GitLab 15.0. diff --git a/doc/development/integrations/secure_partner_integration.md b/doc/development/integrations/secure_partner_integration.md index bcbc02d4827..853541144fb 100644 --- a/doc/development/integrations/secure_partner_integration.md +++ b/doc/development/integrations/secure_partner_integration.md @@ -90,7 +90,7 @@ and complete an integration with the Secure stage. - Documentation for [SAST reports](../../user/application_security/sast/index.md#reports-json-format). - Documentation for [Dependency Scanning reports](../../user/application_security/dependency_scanning/index.md#reports-json-format). - Documentation for [Container Scanning reports](../../user/application_security/container_scanning/index.md#reports-json-format). - - See this [example secure job definition that also defines the artifact created](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/Container-Scanning.gitlab-ci.yml). + - See this [example secure job definition that also defines the artifact created](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Container-Scanning.gitlab-ci.yml). - If you need a new kind of scan or report, [create an issue](https://gitlab.com/gitlab-org/gitlab/-/issues/new#) and add the label `devops::secure`. - Once the job is completed, the data can be seen: diff --git a/doc/development/iterating_tables_in_batches.md b/doc/development/iterating_tables_in_batches.md deleted file mode 100644 index 589e38a5cb0..00000000000 --- a/doc/development/iterating_tables_in_batches.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'database/iterating_tables_in_batches.md' -remove_date: '2022-11-06' ---- - -This document was moved to [another location](database/iterating_tables_in_batches.md). - -<!-- This redirect file can be deleted after <2022-11-06>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/lfs.md b/doc/development/lfs.md index 4d3371af1d4..dd7687cd28b 100644 --- a/doc/development/lfs.md +++ b/doc/development/lfs.md @@ -4,7 +4,10 @@ group: Source Code info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Git LFS **(FREE)** +# Git LFS developer information **(FREE)** + +This page contains developer-centric information for GitLab team members. For the +user documentation, see [Git Large File Storage](../topics/git/lfs/index.md). ## Deep Dive @@ -86,3 +89,9 @@ request is not preserved for the internal API requests made by Gitaly correlation IDs for those API requests are random values until [this Workhorse issue](https://gitlab.com/gitlab-org/gitlab-workhorse/-/issues/309) is resolved. + +## Related topics + +- Blog post: [Getting started with Git LFS](https://about.gitlab.com/blog/2017/01/30/getting-started-with-git-lfs-tutorial/) +- User documentation: [Git Large File Storage (LFS)](../topics/git/lfs/index.md) +- [GitLab Git Large File Storage (LFS) Administration](../administration/lfs/index.md) for self-managed instances diff --git a/doc/development/licensing.md b/doc/development/licensing.md index d7dfdb7357d..cb703e98264 100644 --- a/doc/development/licensing.md +++ b/doc/development/licensing.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w ## Automated Testing -In order to comply with the terms the libraries we use are licensed under, we have to make sure to check new gems for compatible licenses whenever they're added. To automate this process, we use the [license_finder](https://github.com/pivotal/LicenseFinder) gem by Pivotal. It runs every time a new commit is pushed and verifies that all gems and node modules in the bundle use a license that doesn't conflict with the licensing of either GitLab Community Edition or GitLab Enterprise Edition. +To comply with the terms the libraries we use are licensed under, we have to make sure to check new gems for compatible licenses whenever they're added. To automate this process, we use the [License Finder](https://github.com/pivotal/LicenseFinder) gem by Pivotal. It runs every time a new commit is pushed and verifies that all gems and node modules in the bundle use a license that doesn't conflict with the licensing of either GitLab Community Edition or GitLab Enterprise Edition. There are some limitations with the automated testing, however. CSS, JavaScript, or Ruby libraries which are not included by way of Bundler, npm, or Yarn (for instance those manually copied into our source tree in the `vendor` directory), must be verified manually and independently. Take care whenever one such library is used, as automated tests don't catch problematic licenses from them. diff --git a/doc/development/merge_request_performance_guidelines.md b/doc/development/merge_request_performance_guidelines.md index 9d1489836fb..fd78d02202f 100644 --- a/doc/development/merge_request_performance_guidelines.md +++ b/doc/development/merge_request_performance_guidelines.md @@ -145,7 +145,7 @@ end This means running one query for every object to update. This code can easily overload a database given enough rows to update or many instances of this code running in parallel. This particular problem is known as the -["N+1 query problem"](https://guides.rubyonrails.org/active_record_querying.html#eager-loading-associations). You can write a test with [QueryRecorder](query_recorder.md) to detect this and prevent regressions. +["N+1 query problem"](https://guides.rubyonrails.org/active_record_querying.html#eager-loading-associations). You can write a test with [QueryRecorder](database/query_recorder.md) to detect this and prevent regressions. In this particular case the workaround is fairly easy: @@ -224,7 +224,7 @@ The total number of the queries (including cached ones) executed by the code mod should not increase unless absolutely necessary. The number of executed queries (including cached queries) should not depend on collection size. -You can write a test by passing the `skip_cached` variable to [QueryRecorder](query_recorder.md) to detect this and prevent regressions. +You can write a test by passing the `skip_cached` variable to [QueryRecorder](database/query_recorder.md) to detect this and prevent regressions. As an example, say you have a CI pipeline. All pipeline builds belong to the same pipeline, thus they also belong to the same project (`pipeline.project`): @@ -351,7 +351,7 @@ Post.all.includes(:author).each do |post| end ``` -Also consider using [QueryRecoder tests](query_recorder.md) to prevent a regression when eager loading. +Also consider using [QueryRecoder tests](database/query_recorder.md) to prevent a regression when eager loading. ## Memory Usage @@ -463,7 +463,7 @@ Read more about when and how feature flags should be used in We can consider the following types of storages: -- **Local temporary storage** (very-very short-term storage) This type of storage is system-provided storage, ex. `/tmp` folder. +- **Local temporary storage** (very-very short-term storage) This type of storage is system-provided storage, like a `/tmp` folder. This is the type of storage that you should ideally use for all your temporary tasks. The fact that each node has its own temporary storage makes scaling significantly easier. This storage is also very often SSD-based, thus is significantly faster. @@ -480,7 +480,7 @@ We can consider the following types of storages: Be respectful of that. - **Shared persistent storage** (long-term storage) This type of storage uses - shared network-based storage (ex. NFS). This solution is mostly used by customers running small + shared network-based storage (for example, NFS). This solution is mostly used by customers running small installations consisting of a few nodes. The files on shared storage are easily accessible, but any job that is uploading or downloading data can create a serious contention to all other jobs. This is also an approach by default used by Omnibus. @@ -525,13 +525,13 @@ end The usage of shared temporary storage is required if your intent is to persistent file for a disk-based storage, and not Object Storage. -[Workhorse direct_upload](uploads/index.md#direct-upload) when accepting file +[Workhorse direct upload](uploads/index.md#direct-upload) when accepting file can write it to shared storage, and later GitLab Rails can perform a move operation. The move operation on the same destination is instantaneous. The system instead of performing `copy` operation just re-attaches file into a new place. Since this introduces extra complexity into application, you should only try -to re-use well established patterns (ex.: `ObjectStorage` concern) instead of re-implementing it. +to re-use well established patterns (for example, `ObjectStorage` concern) instead of re-implementing it. The usage of shared temporary storage is otherwise deprecated for all other usages. @@ -549,7 +549,7 @@ that implements a seamless support for Shared and Object Storage-based persisten #### Data access Each feature that accepts data uploads or allows to download them needs to use -[Workhorse direct_upload](uploads/index.md#direct-upload). It means that uploads needs to be +[Workhorse direct upload](uploads/index.md#direct-upload). It means that uploads needs to be saved directly to Object Storage by Workhorse, and all downloads needs to be served by Workhorse. @@ -561,5 +561,5 @@ can time out, which is especially problematic for slow clients. If clients take to upload/download the processing slot might be killed due to request processing timeout (usually between 30s-60s). -For the above reasons it is required that [Workhorse direct_upload](uploads/index.md#direct-upload) is implemented +For the above reasons it is required that [Workhorse direct upload](uploads/index.md#direct-upload) is implemented for all file uploads and downloads. diff --git a/doc/development/migration_style_guide.md b/doc/development/migration_style_guide.md index 5764c876e4d..8f035d4aa13 100644 --- a/doc/development/migration_style_guide.md +++ b/doc/development/migration_style_guide.md @@ -184,13 +184,21 @@ git checkout origin/master db/structure.sql VERSION=<migration ID> bundle exec rails db:migrate:main ``` -### Adding new tables to GitLab Schema - -GitLab connects to two different Postgres databases: `main` and `ci`. New tables should be defined in [`lib/gitlab/database/gitlab_schemas.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/database/gitlab_schemas.yml) with the databases they need to be added to. - - ```yaml - <TABLE_NAME>: :gitlab_main - ``` +### Adding new tables to the database dictionary + +GitLab connects to two different Postgres databases: `main` and `ci`. New tables should be defined in [`db/docs/`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/db/docs): + +```yaml +table_name: table name exmaple +description: Description example +introduced_by_url: Merge request link +milestone: Milestone example +feature_categories: +- Feature category example +classes: +- Class example +gitlab_schema: gitlab_main +``` ## Avoiding downtime @@ -305,7 +313,7 @@ of migration helpers. In this example, we use version 2.0 of the migration class: ```ruby -class TestMigration < Gitlab::Database::Migration[2.0] +class TestMigration < Gitlab::Database::Migration[2.1] def change end end @@ -338,7 +346,7 @@ is concurrently accessed and modified by other processes, acquiring the lock may a while. The lock request is waiting in a queue and it may also block other queries on the `users` table once it has been enqueued. -More information about PostgresSQL locks: [Explicit Locking](https://www.postgresql.org/docs/current/explicit-locking.html) +More information about PostgreSQL locks: [Explicit Locking](https://www.postgresql.org/docs/current/explicit-locking.html) For stability reasons, GitLab.com has a specific [`statement_timeout`](../user/gitlab_com/index.md#postgresql) set. When the migration is invoked, any database query has @@ -376,7 +384,7 @@ Occasionally a migration may need to acquire multiple locks on different objects To prevent catalog bloat, ask for all those locks explicitly before performing any DDL. A better strategy is to split the migration, so that we only need to acquire one lock at the time. -**Removing a column:** +#### Removing a column ```ruby enable_lock_retries! @@ -386,7 +394,7 @@ def change end ``` -**Multiple changes on the same table:** +#### Multiple changes on the same table With the lock-retry methodology enabled, all operations wrap into a single transaction. When you have the lock, you should do as much as possible inside the transaction rather than trying to get another lock later. @@ -406,7 +414,7 @@ def down end ``` -**Removing a foreign key:** +#### Removing a foreign key ```ruby enable_lock_retries! @@ -420,7 +428,7 @@ def down end ``` -**Changing default value for a column:** +#### Changing default value for a column ```ruby enable_lock_retries! @@ -434,7 +442,7 @@ def down end ``` -**Creating a new table with a foreign key:** +#### Creating a new table with a foreign key We can wrap the `create_table` method with `with_lock_retries`: @@ -453,7 +461,7 @@ def down end ``` -**Creating a new table when we have two foreign keys:** +#### Creating a new table when we have two foreign keys Only one foreign key should be created per transaction. This is because [the addition of a foreign key constraint requires a `SHARE ROW EXCLUSIVE` lock on the referenced table](https://www.postgresql.org/docs/12/sql-createtable.html#:~:text=The%20addition%20of%20a%20foreign%20key%20constraint%20requires%20a%20SHARE%20ROW%20EXCLUSIVE%20lock%20on%20the%20referenced%20table), and locking multiple tables in the same transaction should be avoided. @@ -600,7 +608,7 @@ by calling the method `disable_ddl_transaction!` in the body of your migration class like so: ```ruby -class MyMigration < Gitlab::Database::Migration[2.0] +class MyMigration < Gitlab::Database::Migration[2.1] disable_ddl_transaction! INDEX_NAME = 'index_name' @@ -611,10 +619,10 @@ class MyMigration < Gitlab::Database::Migration[2.0] end ``` -Verify the index is not being used anymore with this Thanos query: +You can verify that the index is not being used with [Thanos](https://thanos-query.ops.gitlab.net/graph?g0.expr=sum%20by%20(type)(rate(pg_stat_user_indexes_idx_scan%7Benv%3D%22gprd%22%2C%20indexrelname%3D%22INSERT%20INDEX%20NAME%20HERE%22%7D%5B30d%5D))&g0.tab=1&g0.stacked=0&g0.range_input=1h&g0.max_source_resolution=0s&g0.deduplicate=1&g0.partial_response=0&g0.store_matches=%5B%5D): ```sql -sum by (type)(rate(pg_stat_user_indexes_idx_scan{env="gprd", indexrelname="index_groups_on_parent_id_id"}[5m])) +sum by (type)(rate(pg_stat_user_indexes_idx_scan{env="gprd", indexrelname="INSERT INDEX NAME HERE"}[30d])) ``` Note that it is not necessary to check if the index exists prior to @@ -657,7 +665,7 @@ by calling the method `disable_ddl_transaction!` in the body of your migration class like so: ```ruby -class MyMigration < Gitlab::Database::Migration[2.0] +class MyMigration < Gitlab::Database::Migration[2.1] disable_ddl_transaction! INDEX_NAME = 'index_name' @@ -700,7 +708,7 @@ The easiest way to test for existence of an index by name is to use the be used with a name option. For example: ```ruby -class MyMigration < Gitlab::Database::Migration[2.0] +class MyMigration < Gitlab::Database::Migration[2.1] INDEX_NAME = 'index_name' def up @@ -735,7 +743,7 @@ Here's an example where we add a new column with a foreign key constraint. Note it includes `index: true` to create an index for it. ```ruby -class Migration < Gitlab::Database::Migration[2.0] +class Migration < Gitlab::Database::Migration[2.1] def change add_reference :model, :other_model, index: true, foreign_key: { on_delete: :cascade } @@ -766,12 +774,7 @@ With PostgreSQL 11 being the minimum version in GitLab 13.0 and later, adding co the standard `add_column` helper should be used in all cases. Before PostgreSQL 11, adding a column with a default was problematic as it would -have caused a full table rewrite. The corresponding helper `add_column_with_default` -has been deprecated and is scheduled to be removed in a later release. - -If a backport adding a column with a default value is needed for %12.9 or earlier versions, -it should use `add_column_with_default` helper. If a [large table](https://gitlab.com/gitlab-org/gitlab/-/blob/master/rubocop/rubocop-migrations.yml#L3) -is involved, backporting to %12.9 is contraindicated. +have caused a full table rewrite. ## Removing the column default for non-nullable columns @@ -796,7 +799,7 @@ expensive and disruptive operation for larger tables, but in reality it's not. Take the following migration as an example: ```ruby -class DefaultRequestAccessGroups < Gitlab::Database::Migration[2.0] +class DefaultRequestAccessGroups < Gitlab::Database::Migration[2.1] def change change_column_default(:namespaces, :request_access_enabled, from: false, to: true) end @@ -818,7 +821,7 @@ in the `namespaces` table. Only when creating a new column with a default, all t NOTE: A faster [ALTER TABLE ADD COLUMN with a non-null default](https://www.depesz.com/2018/04/04/waiting-for-postgresql-11-fast-alter-table-add-column-with-a-non-null-default/) -was introduced on PostgresSQL 11.0, removing the need of rewriting the table when a new column with a default value is added. +was introduced on PostgreSQL 11.0, removing the need of rewriting the table when a new column with a default value is added. For the reasons mentioned above, it's safe to use `change_column_default` in a single-transaction migration without requiring `disable_ddl_transaction!`. @@ -852,8 +855,7 @@ update_column_in_batches(:projects, :foo, update_value) do |table, query| end ``` -Like `add_column_with_default`, there is a RuboCop cop to detect usage of this -on large tables. In the case of `update_column_in_batches`, it may be acceptable +In the case of `update_column_in_batches`, it may be acceptable to run on a large table, as long as it is only updating a small subset of the rows in the table, but do not ignore that without validating on the GitLab.com staging environment - or asking someone else to do so for you - beforehand. @@ -991,7 +993,7 @@ Re-add a sequence: A Rails migration example: ```ruby -class DropSequenceTest < Gitlab::Database::Migration[2.0] +class DropSequenceTest < Gitlab::Database::Migration[2.1] def up drop_sequence(:ci_pipelines_config, :pipeline_id, :ci_pipelines_config_pipeline_id_seq) end @@ -1022,7 +1024,7 @@ Under the hood, it works like this: - Add the primary key using the index defined beforehand. ```ruby -class SwapPrimaryKey < Gitlab::Database::Migration[2.0] +class SwapPrimaryKey < Gitlab::Database::Migration[2.1] TABLE_NAME = :table_name PRIMARY_KEY = :table_name_pkey OLD_INDEX_NAME = :old_index_name @@ -1113,18 +1115,18 @@ The Rails 5 natively supports `JSONB` (binary JSON) column type. Example migration adding this column: ```ruby -class AddOptionsToBuildMetadata < Gitlab::Database::Migration[2.0] +class AddOptionsToBuildMetadata < Gitlab::Database::Migration[2.1] def change add_column :ci_builds_metadata, :config_options, :jsonb end end ``` -You have to use a serializer to provide a translation layer: +By default hash keys will be strings. Optionally you can add a custom data type to provide different access to keys. ```ruby class BuildMetadata - serialize :config_options, Serializers::Json # rubocop:disable Cop/ActiveRecordSerialize + attribute :config_options, :ind_jsonb # for indifferent accesss or :sym_jsonb if you need symbols only as keys. end ``` @@ -1145,7 +1147,7 @@ Do not store `attr_encrypted` attributes as `:text` in the database; use efficient: ```ruby -class AddSecretToSomething < Gitlab::Database::Migration[2.0] +class AddSecretToSomething < Gitlab::Database::Migration[2.1] def change add_column :something, :encrypted_secret, :binary add_column :something, :encrypted_secret_iv, :binary @@ -1203,7 +1205,7 @@ If you need more complex logic, you can define and use models local to a migration. For example: ```ruby -class MyMigration < Gitlab::Database::Migration[2.0] +class MyMigration < Gitlab::Database::Migration[2.1] class Project < MigrationRecord self.table_name = 'projects' end @@ -1227,7 +1229,7 @@ Be aware of the limitations [when using models in migrations](#using-models-in-m In most circumstances, prefer migrating data in **batches** when modifying data in the database. -We introduced a new helper [each_batch_range](https://gitlab.com/gitlab-org/gitlab/-/blob/cd3e0a5cddcb464cb9b8c6e3275839cf57dfa6e2/lib/gitlab/database/dynamic_model_helpers.rb#L28-32) which facilitates the process of iterating over a collection in a performant way. The default size of the batch is defined in the `BATCH_SIZE` constant. +We introduced a new helper [`each_batch_range`](https://gitlab.com/gitlab-org/gitlab/-/blob/cd3e0a5cddcb464cb9b8c6e3275839cf57dfa6e2/lib/gitlab/database/dynamic_model_helpers.rb#L28-32) which facilitates the process of iterating over a collection in a performant way. The default size of the batch is defined in the `BATCH_SIZE` constant. See the following example to get an idea. @@ -1302,7 +1304,7 @@ in a previous migration. It is important not to leave out the `User.reset_column_information` command, to ensure that the old schema is dropped from the cache and ActiveRecord loads the updated schema information. ```ruby -class AddAndSeedMyColumn < Gitlab::Database::Migration[2.0] +class AddAndSeedMyColumn < Gitlab::Database::Migration[2.1] class User < MigrationRecord self.table_name = 'users' end diff --git a/doc/development/multi_version_compatibility.md b/doc/development/multi_version_compatibility.md index 8e2103b2f3e..36b392a0037 100644 --- a/doc/development/multi_version_compatibility.md +++ b/doc/development/multi_version_compatibility.md @@ -112,7 +112,7 @@ During an update, there will be [two different versions of GitLab running in dif Yes! We have specific instructions for [zero-downtime updates](../update/index.md#upgrading-without-downtime) because it allows us to ignore some permutations of compatibility. This is why we don't worry about Rails code making DB calls to an old PostgreSQL database schema. -## I've identified a potential backwards compatibility problem, what can I do about it? +## You've identified a potential backwards compatibility problem, what can you do about it? ### Coordinate diff --git a/doc/development/namespaces_storage_statistics.md b/doc/development/namespaces_storage_statistics.md deleted file mode 100644 index 75e79d1f693..00000000000 --- a/doc/development/namespaces_storage_statistics.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'database/namespaces_storage_statistics.md' -remove_date: '2022-11-05' ---- - -This document was moved to [another location](database/namespaces_storage_statistics.md). - -<!-- This redirect file can be deleted after <2022-11-05>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/new_fe_guide/development/accessibility.md b/doc/development/new_fe_guide/development/accessibility.md deleted file mode 100644 index 9575acd20c7..00000000000 --- a/doc/development/new_fe_guide/development/accessibility.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../../fe_guide/accessibility.md' -remove_date: '2022-11-15' ---- - -This document was moved to [another location](../../fe_guide/accessibility.md). - -<!-- This redirect file can be deleted after <2022-11-15>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/new_fe_guide/development/components.md b/doc/development/new_fe_guide/development/components.md deleted file mode 100644 index 9ad742272d1..00000000000 --- a/doc/development/new_fe_guide/development/components.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../../fe_guide/index.md' -remove_date: '2022-11-15' ---- - -This document was moved to [another location](../../fe_guide/index.md). - -<!-- This redirect file can be deleted after <2022-11-15>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/new_fe_guide/development/index.md b/doc/development/new_fe_guide/development/index.md deleted file mode 100644 index 9ad742272d1..00000000000 --- a/doc/development/new_fe_guide/development/index.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../../fe_guide/index.md' -remove_date: '2022-11-15' ---- - -This document was moved to [another location](../../fe_guide/index.md). - -<!-- This redirect file can be deleted after <2022-11-15>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/new_fe_guide/development/performance.md b/doc/development/new_fe_guide/development/performance.md deleted file mode 100644 index c72f3ded896..00000000000 --- a/doc/development/new_fe_guide/development/performance.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../../fe_guide/performance.md' -remove_date: '2022-11-15' ---- - -This document was moved to [another location](../../fe_guide/performance.md). - -<!-- This redirect file can be deleted after <2022-11-15>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/new_fe_guide/index.md b/doc/development/new_fe_guide/index.md deleted file mode 100644 index 83c1db696b4..00000000000 --- a/doc/development/new_fe_guide/index.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../fe_guide/index.md' -remove_date: '2022-11-15' ---- - -This document was moved to [another location](../fe_guide/index.md). - -<!-- This redirect file can be deleted after <2022-11-15>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/new_fe_guide/modules/dirty_submit.md b/doc/development/new_fe_guide/modules/dirty_submit.md deleted file mode 100644 index 9ad742272d1..00000000000 --- a/doc/development/new_fe_guide/modules/dirty_submit.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../../fe_guide/index.md' -remove_date: '2022-11-15' ---- - -This document was moved to [another location](../../fe_guide/index.md). - -<!-- This redirect file can be deleted after <2022-11-15>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/new_fe_guide/modules/index.md b/doc/development/new_fe_guide/modules/index.md deleted file mode 100644 index 9ad742272d1..00000000000 --- a/doc/development/new_fe_guide/modules/index.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../../fe_guide/index.md' -remove_date: '2022-11-15' ---- - -This document was moved to [another location](../../fe_guide/index.md). - -<!-- This redirect file can be deleted after <2022-11-15>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/new_fe_guide/modules/widget_extensions.md b/doc/development/new_fe_guide/modules/widget_extensions.md deleted file mode 100644 index 3741ee8c38a..00000000000 --- a/doc/development/new_fe_guide/modules/widget_extensions.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../../fe_guide/merge_request_widget_extensions.md' -remove_date: '2022-11-15' ---- - -This document was moved to [another location](../../fe_guide/merge_request_widget_extensions.md). - -<!-- This redirect file can be deleted after <2022-11-15>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/new_fe_guide/tips.md b/doc/development/new_fe_guide/tips.md deleted file mode 100644 index 83c1db696b4..00000000000 --- a/doc/development/new_fe_guide/tips.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../fe_guide/index.md' -remove_date: '2022-11-15' ---- - -This document was moved to [another location](../fe_guide/index.md). - -<!-- This redirect file can be deleted after <2022-11-15>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/ordering_table_columns.md b/doc/development/ordering_table_columns.md deleted file mode 100644 index b665cb0d4c7..00000000000 --- a/doc/development/ordering_table_columns.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'database/ordering_table_columns.md' -remove_date: '2022-11-04' ---- - -This document was moved to [another location](database/ordering_table_columns.md). - -<!-- This redirect file can be deleted after <2022-11-04>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/packages/dependency_proxy.md b/doc/development/packages/dependency_proxy.md index d5cc219cba0..cc8b202e556 100644 --- a/doc/development/packages/dependency_proxy.md +++ b/doc/development/packages/dependency_proxy.md @@ -117,7 +117,7 @@ Manifests are more complicated, partially due to [rate limiting on DockerHub](ht A manifest is essentially a recipe for creating an image. It has a list of blobs to create a certain image. So `alpine:latest` has a manifest associated with it that specifies the blobs needed to create the `alpine:latest` image. The interesting part is that `alpine:latest` can change over time, so we can't just cache the manifest and -assume it is OK to use forever. Instead, we must check the digest of the manifest, which is an Etag. This gets +assume it is OK to use forever. Instead, we must check the digest of the manifest, which is an ETag. This gets interesting because the requests for manifests often don't include the digest. So how do we know if the manifest we have cached is still the most up-to-date `alpine:latest`? DockerHub allows free HEAD requests that don't count toward their rate limit. The HEAD request returns the manifest digest so we can tell whether or not the one we @@ -143,7 +143,7 @@ Management of file uploads and caching happens in [Workhorse](../workhorse/index [`POST` routes](https://gitlab.com/gitlab-org/gitlab/-/blob/3f76455ac9cf90a927767e55c837d6b07af818df/config/routes/group.rb#L170-173) that we have for the Dependency Proxy. -The [send_dependency](https://gitlab.com/gitlab-org/gitlab/-/blob/7359d23f4e078479969c872924150219c6f1665f/app/helpers/workhorse_helper.rb#L46-53) +The [`send_dependency`](https://gitlab.com/gitlab-org/gitlab/-/blob/7359d23f4e078479969c872924150219c6f1665f/app/helpers/workhorse_helper.rb#L46-53) method makes a request to Workhorse including the previously fetched JWT from the external registry. Workhorse then can use that token to request the manifest or blob the user originally requested. The Workhorse code lives in [`workhorse/internal/dependencyproxy/dependencyproxy.go`](https://gitlab.com/gitlab-org/gitlab/-/blob/b8f44a8f3c26efe9932c2ada2df75ef7acb8417b/workhorse/internal/dependencyproxy/dependencyproxy.go#L4). diff --git a/doc/development/pages/index.md b/doc/development/pages/index.md index 6ee8a9ac433..5e56264330a 100644 --- a/doc/development/pages/index.md +++ b/doc/development/pages/index.md @@ -178,7 +178,7 @@ The `redirect-uri` must not contain any GitLab Pages site domain. auth-redirect-uri=http://pages.gdk.test:3010/auth # the authentication callback url for GitLab Pages ``` -1. If running Pages inside the GDK, you can use GDK's `protected_config_files` section under `gdk` in +1. If running Pages inside the GDK, you can use GDK `protected_config_files` section under `gdk` in your `gdk.yml` to avoid getting `gitlab-pages.conf` configuration rewritten: ```yaml diff --git a/doc/development/performance.md b/doc/development/performance.md index 3881fad0528..127cade9fee 100644 --- a/doc/development/performance.md +++ b/doc/development/performance.md @@ -37,7 +37,7 @@ consistent performance of GitLab. Refer to the [Index](#performance-documentatio - [Service measurement](../development/service_measurement.md) - Self-managed administration and customer-focused: - [File system performance benchmarking](../administration/operations/filesystem_benchmarking.md) - - [Sidekiq performance troubleshooting](../administration/troubleshooting/sidekiq.md) + - [Sidekiq performance troubleshooting](../administration/sidekiq/sidekiq_troubleshooting.md) ## Workflow @@ -74,7 +74,7 @@ GitLab provides built-in tools to help improve performance and availability: - [Profiling](profiling.md). - [Distributed Tracing](distributed_tracing.md) - [GitLab Performance Monitoring](../administration/monitoring/performance/index.md). -- [QueryRecoder](query_recorder.md) for preventing `N+1` regressions. +- [QueryRecoder](database/query_recorder.md) for preventing `N+1` regressions. - [Chaos endpoints](chaos_endpoints.md) for testing failure scenarios. Intended mainly for testing availability. - [Service measurement](service_measurement.md) for measuring and logging service execution. @@ -312,7 +312,7 @@ To export the flame graph to an SVG file, use [Brendan Gregg's FlameGraph tool]( stackprof --stackcollapse /tmp/group_member_policy_spec.rb.dump | flamegraph.pl > flamegraph.svg ``` -It's also possible to view flame graphs through [speedscope](https://github.com/jlfwong/speedscope). +It's also possible to view flame graphs through [Speedscope](https://github.com/jlfwong/speedscope). You can do this when using the [performance bar](profiling.md#speedscope-flamegraphs) and when [profiling code blocks](https://github.com/jlfwong/speedscope/wiki/Importing-from-stackprof-(ruby)). This option isn't supported by `bin/rspec-stackprof`. diff --git a/doc/development/pipelines/index.md b/doc/development/pipelines/index.md index 01bb813e794..a7b8c99bd13 100644 --- a/doc/development/pipelines/index.md +++ b/doc/development/pipelines/index.md @@ -68,8 +68,8 @@ Later on in [the `rspec fail-fast` job](#fail-fast-job-in-merge-request-pipeline In addition, there are a few circumstances where we would always run the full RSpec tests: -- when the `pipeline:run-all-rspec` label is set on the merge request -- when the `pipeline:run-full-rspec` label is set on the merge request, this label is assigned by triage automation when the merge request is approved by any reviewer +- when the `pipeline:run-all-rspec` label is set on the merge request. This label will trigger all RSpec tests including those run in the `as-if-foss` jobs. +- when the `pipeline:mr-approved` label is set on the merge request, and if the code changes satisfy the `backend-patterns` rule. Note that this label is assigned by triage automation when the merge request is approved by any reviewer. It is not recommended to apply this label manually. - when the merge request is created by an automation (for example, Gitaly update or MR targeting a stable branch) - when the merge request is created in a security mirror - when any CI configuration file is changed (for example, `.gitlab-ci.yml` or `.gitlab/ci/**/*`) @@ -88,10 +88,12 @@ In addition, there are a few circumstances where we would always run the full Je - when the `pipeline:run-all-jest` label is set on the merge request - when the merge request is created by an automation (for example, Gitaly update or MR targeting a stable branch) - when the merge request is created in a security mirror -- when any CI configuration file is changed (for example, `.gitlab-ci.yml` or `.gitlab/ci/**/*`) -- when any frontend "core" file is changed (for example, `package.json`, `yarn.lock`, `babel.config.js`, `jest.config.*.js`, `config/helpers/**/*.js`) +- when relevant CI configuration file is changed (`.gitlab/ci/rules.gitlab-ci.yml`, `.gitlab/ci/frontend.gitlab-ci.yml`) +- when any frontend dependency file is changed (for example, `package.json`, `yarn.lock`, `config/webpack.config.js`, `config/helpers/**/*.js`) - when any vendored JavaScript file is changed (for example, `vendor/assets/javascripts/**/*`) -- when any backend file is changed ([see the patterns list for details](https://gitlab.com/gitlab-org/gitlab/-/blob/3616946936c1adbd9e754c1bd06f86ba670796d8/.gitlab/ci/rules.gitlab-ci.yml#L205-216)) + +The `rules` definitions for full Jest tests are defined at `.frontend:rules:jest` in +[`rules.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/42321b18b946c64d2f6f788c38844499a5ae9141/.gitlab/ci/rules.gitlab-ci.yml#L938-955). ### Fork pipelines @@ -146,9 +148,37 @@ merge request. This prevents `rspec fail-fast` duration from exceeding the avera This number can be overridden by setting a CI/CD variable named `RSPEC_FAIL_FAST_TEST_FILE_COUNT_THRESHOLD`. +## Re-run previously failed tests in merge request pipelines + +In order to reduce the feedback time after resolving failed tests for a merge request, the `rspec rspec-pg12-rerun-previous-failed-tests` +and `rspec rspec-ee-pg12-rerun-previous-failed-tests` jobs run the failed tests from the previous MR pipeline. + +This was introduced on August 25th 2021, with <https://gitlab.com/gitlab-org/gitlab/-/merge_requests/69053>. + +### How it works? + +1. The `detect-previous-failed-tests` job (`prepare` stage) detects the test files associated with failed RSpec + jobs from the previous MR pipeline. +1. The `rspec rspec-pg12-rerun-previous-failed-tests` and `rspec rspec-ee-pg12-rerun-previous-failed-tests` jobs + will run the test files gathered by the `detect-previous-failed-tests` job. + +```mermaid +graph LR + subgraph "prepare stage"; + A["detect-previous-failed-tests"] + end + + subgraph "test stage"; + B["rspec rspec-pg12-rerun-previous-failed-tests"]; + C["rspec rspec-ee-pg12-rerun-previous-failed-tests"]; + end + + A --"artifact: list of test files"--> B & C +``` + ## Faster feedback for merge requests that fix a broken `master` -When you need to [fix a broken `master`](https://about.gitlab.com/handbook/engineering/workflow/#resolution-of-broken-master), you can add the `pipeline:expedite-master-fixing` label to expedite the pipelines that run on the merge request. +When you need to [fix a broken `master`](https://about.gitlab.com/handbook/engineering/workflow/#resolution-of-broken-master), you can add the `pipeline:expedite` label to expedite the pipelines that run on the merge request. When this label is assigned, the following steps of the CI/CD pipeline are skipped: diff --git a/doc/development/pipelines/internals.md b/doc/development/pipelines/internals.md index 2861e2f266b..71492ed9f5d 100644 --- a/doc/development/pipelines/internals.md +++ b/doc/development/pipelines/internals.md @@ -107,7 +107,7 @@ automatically updates the Gitaly version used in the main project), [the Dependency proxy isn't accessible](https://gitlab.com/gitlab-org/gitlab/-/issues/332411#note_1130388163) and the job fails at the `Preparing the "docker+machine" executor` step. To work around that, we have a special workflow rule, that overrides the -`${GITLAB_DEPENDENCY_PROXY_ADDRESS}` variable so that Depdendency proxy isn't used in that case: +`${GITLAB_DEPENDENCY_PROXY_ADDRESS}` variable so that Dependency proxy isn't used in that case: ```yaml - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $GITLAB_USER_LOGIN =~ /project_\d+_bot\d*/' @@ -165,9 +165,9 @@ and included in `rules` definitions via [YAML anchors](../../ci/yaml/yaml_optimi | `if:` conditions | Description | Notes | |------------------|-------------|-------| -| `if-not-canonical-namespace` | Matches if the project isn't in the canonical (`gitlab-org/`) or security (`gitlab-org/security`) namespace. | Use to create a job for forks (by using `when: on_success|manual`), or **not** create a job for forks (by using `when: never`). | -| `if-not-ee` | Matches if the project isn't EE (that is, project name isn't `gitlab` or `gitlab-ee`). | Use to create a job only in the FOSS project (by using `when: on_success|manual`), or **not** create a job if the project is EE (by using `when: never`). | -| `if-not-foss` | Matches if the project isn't FOSS (that is, project name isn't `gitlab-foss`, `gitlab-ce`, or `gitlabhq`). | Use to create a job only in the EE project (by using `when: on_success|manual`), or **not** create a job if the project is FOSS (by using `when: never`). | +| `if-not-canonical-namespace` | Matches if the project isn't in the canonical (`gitlab-org/`) or security (`gitlab-org/security`) namespace. | Use to create a job for forks (by using `when: on_success` or `when: manual`), or **not** create a job for forks (by using `when: never`). | +| `if-not-ee` | Matches if the project isn't EE (that is, project name isn't `gitlab` or `gitlab-ee`). | Use to create a job only in the FOSS project (by using `when: on_success` or `when: manual`), or **not** create a job if the project is EE (by using `when: never`). | +| `if-not-foss` | Matches if the project isn't FOSS (that is, project name isn't `gitlab-foss`, `gitlab-ce`, or `gitlabhq`). | Use to create a job only in the EE project (by using `when: on_success` or `when: manual`), or **not** create a job if the project is FOSS (by using `when: never`). | | `if-default-refs` | Matches if the pipeline is for `master`, `main`, `/^[\d-]+-stable(-ee)?$/` (stable branches), `/^\d+-\d+-auto-deploy-\d+$/` (auto-deploy branches), `/^security\//` (security branches), merge requests, and tags. | Note that jobs aren't created for branches with this default configuration. | | `if-master-refs` | Matches if the current branch is `master` or `main`. | | | `if-master-push` | Matches if the current branch is `master` or `main` and pipeline source is `push`. | | @@ -203,7 +203,7 @@ and included in `rules` definitions via [YAML anchors](../../ci/yaml/yaml_optimi | `ci-qa-patterns` | Only create job for CI configuration-related changes related to the `qa` stage. | | `yaml-lint-patterns` | Only create job for YAML-related changes. | | `docs-patterns` | Only create job for docs-related changes. | -| `frontend-dependency-patterns` | Only create job when frontend dependencies are updated (that is, `package.json`, and `yarn.lock`). changes. | +| `frontend-dependency-patterns` | Only create job when frontend dependencies are updated (for example, `package.json`, and `yarn.lock`) changes. | | `frontend-patterns-for-as-if-foss` | Only create job for frontend-related changes that have impact on FOSS. | | `backend-patterns` | Only create job for backend-related changes. | | `db-patterns` | Only create job for DB-related changes. | diff --git a/doc/development/pipelines/performance.md b/doc/development/pipelines/performance.md index 1c6f9d78879..5f2df91edf3 100644 --- a/doc/development/pipelines/performance.md +++ b/doc/development/pipelines/performance.md @@ -54,7 +54,6 @@ This works well for the following reasons: - `update-qa-cache`, defined in [`.gitlab/ci/qa.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/qa.gitlab-ci.yml). - `update-assets-compile-production-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). - `update-assets-compile-test-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). - - `update-yarn-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). - `update-storybook-yarn-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). 1. These jobs can also be forced to run in merge requests with the `pipeline:update-cache` label (this can be useful to warm the caches in a MR that updates the cache keys). diff --git a/doc/development/polymorphic_associations.md b/doc/development/polymorphic_associations.md deleted file mode 100644 index 6b9158b8408..00000000000 --- a/doc/development/polymorphic_associations.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'database/polymorphic_associations.md' -remove_date: '2022-11-04' ---- - -This document was moved to [another location](database/polymorphic_associations.md). - -<!-- This redirect file can be deleted after <2022-11-04>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/product_qualified_lead_guide/index.md b/doc/development/product_qualified_lead_guide/index.md index 89a05d59fc9..1ad4f622829 100644 --- a/doc/development/product_qualified_lead_guide/index.md +++ b/doc/development/product_qualified_lead_guide/index.md @@ -17,7 +17,7 @@ A hand-raise PQL is a user who requests to speak to sales from within the produc 1. Set up CustomersDot using the [normal install instructions](https://gitlab.com/gitlab-org/customers-gitlab-com/-/blob/staging/doc/setup/installation_steps.md). 1. Set the `CUSTOMER_PORTAL_URL` environment variable to your local (or ngrok) URL of your CustomersDot instance. -1. Place `export CUSTOMER_PORTAL_URL='https://XXX.ngrok.io/'` in your shell rc script (`~/.zshrc` or `~/.bash_profile` or `~/.bashrc`) and restart GDK. +1. Place `export CUSTOMER_PORTAL_URL='https://XXX.ngrok.io/'` in your shell `rc` script (`~/.zshrc` or `~/.bash_profile` or `~/.bashrc`) and restart GDK. 1. Enter the credentials on CustomersDot development to Platypus in your `/config/secrets.yml` and restart. Credentials for the Platypus Staging are in the 1Password Growth vault. The URL for staging is `https://staging.ci.nexus.gitlabenvironment.cloud`. ```yaml @@ -95,16 +95,16 @@ The `ctaTracking` parameters follow [the `data-track` attributes](../snowplow/im When embedding a new hand raise form, use a unique `glmContent` or `glm_content` field that is different to any existing values. -We currently use the following `glm content` values: +We currently use the following `glm_content` values: -| glm_content value | Notes | -| ------ | ------ | -| discover-group-security | This value is used in the group security feature discovery page. | -| discover-group-security-pqltest | This value is used in the group security feature discovery page [experiment with 3 CTAs](https://gitlab.com/gitlab-org/gitlab/-/issues/349799). | -| discover-project-security | This value is used in the project security feature discovery page. | -| discover-project-security-pqltest | This value is used in the project security feature discovery page [experiment with 3 CTAs](https://gitlab.com/gitlab-org/gitlab/-/issues/349799). | -| group-billing | This value is used in the group billing page. | -| trial-status-show-group | This value is used in the top left nav when a namespace has an active trial. | +| `glm_content` value | Notes | +|-------------------------------------|-------| +| `discover-group-security` | This value is used in the group security feature discovery page. | +| `discover-group-security-pqltest` | This value is used in the group security feature discovery page [experiment with 3 CTAs](https://gitlab.com/gitlab-org/gitlab/-/issues/349799). | +| `discover-project-security` | This value is used in the project security feature discovery page. | +| `discover-project-security-pqltest` | This value is used in the project security feature discovery page [experiment with 3 CTAs](https://gitlab.com/gitlab-org/gitlab/-/issues/349799). | +| `group-billing` | This value is used in the group billing page. | +| `trial-status-show-group` | This value is used in the top left nav when a namespace has an active trial. | ### Test the component @@ -144,7 +144,7 @@ sequenceDiagram ``` -#### Trial lead flow on CustomersDot (sync_to_gl) +#### Trial lead flow on CustomersDot (`sync_to_gl`) ```mermaid sequenceDiagram diff --git a/doc/development/profiling.md b/doc/development/profiling.md index 3eb2c7c9144..efee6ff3cd5 100644 --- a/doc/development/profiling.md +++ b/doc/development/profiling.md @@ -21,7 +21,7 @@ The first argument to the profiler is either a full URL leading slash. By default the report dump will be stored in a temporary file, which can be -interacted with using the [stackprof API](#reading-a-gitlabprofiler-report). +interacted with using the [Stackprof API](#reading-a-gitlabprofiler-report). When using the script, command-line documentation is available by passing no arguments. @@ -62,7 +62,7 @@ Pass in a `profiler_options` hash to configure the output file (`out`) of the sa Gitlab::Profiler.profile('/gitlab-org/gitlab-test', user: User.first, profiler_options: { out: 'tmp/profile.dump' }) ``` -## Reading a GitLab::Profiler report +## Reading a `GitLab::Profiler` report You can get a summary of where time was spent by running Stackprof against the sampling data. For example: @@ -131,25 +131,39 @@ Find more information about different sampling modes in the [Stackprof docs](htt This is enabled for all users that can access the performance bar. +<!-- vale gitlab.SubstitutionWarning = NO --> +<!-- Here, "bullet" is a false positive --> + ## Bullet -Bullet is a Gem that can be used to track down N+1 query problems. Bullet section is -displayed on the [performance-bar](../administration/monitoring/performance/performance_bar.md). +Bullet is a Gem that can be used to track down N+1 query problems. It logs +query problems to the Rails log and the browser console. The **Bullet** section is +displayed on the [performance bar](../administration/monitoring/performance/performance_bar.md).  -Because Bullet adds quite a bit of logging noise the logging is disabled by default. -To enable the logging, set the environment variable `ENABLE_BULLET` to a non-empty value before -starting GitLab. For example: +Bullet is enabled only in development mode by default. However, logging is disabled, +because Bullet logging is noisy. To configure Bullet and its logging: -```shell -ENABLE_BULLET=true bundle exec rails s -``` +- To manually enable or disable Bullet on an environment, add these lines to + `config/gitlab.yml`, changing the `enabled` value as needed: + + ```yaml + bullet: + enabled: false + ``` + +- To enable Bullet logging, set the `ENABLE_BULLET` environment variable to a + non-empty value before starting GitLab: + + ```shell + ENABLE_BULLET=true bundle exec rails s + ``` -Bullet logs query problems to both the Rails log as well as the browser -console. +As a follow-up to finding `N+1` queries with Bullet, consider writing a +[QueryRecoder test](database/query_recorder.md) to prevent a regression. -As a follow up to finding `N+1` queries with Bullet, consider writing a [QueryRecoder test](query_recorder.md) to prevent a regression. +<!-- vale gitlab.SubstitutionWarning = YES --> ## System stats diff --git a/doc/development/project_templates.md b/doc/development/project_templates.md index 2f1ded23e38..269724c0a7f 100644 --- a/doc/development/project_templates.md +++ b/doc/development/project_templates.md @@ -4,7 +4,9 @@ group: Workspace info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments" --- -# Contribute a built-in project template +# Contribute to built-in project templates + +## Adding a new built-in project template This page provides instructions about how to contribute a [built-in project template](../user/project/working_with_projects.md#create-a-project-from-a-built-in-template). @@ -20,7 +22,7 @@ You can contribute the following types of project templates: - Enterprise: For users with GitLab Premium and above. - Non-enterprise: For users with GitLab Free and above. -## Prerequisites +### Prerequisites To add or update an existing template, you must have the following tools installed: @@ -28,22 +30,22 @@ installed: - `wget` - `tar` -## Create a project template for review +### Create a project template for review 1. In your selected namespace, create a public project. 1. Add the project content you want to use in the template. Do not include unnecessary assets or dependencies. For an example, [see this project](https://gitlab.com/gitlab-org/project-templates/dotnetcore). 1. When the project is ready for review, [create an issue](https://gitlab.com/gitlab-org/gitlab/issues) with a link to your project. - In your issue, mention the relevant [Backend Engineering Manager and Product Manager](https://about.gitlab.com/handbook/product/categories/#source-code-group) + In your issue, mention the Create:Source Code [Backend Engineering Manager and Product Manager](https://about.gitlab.com/handbook/product/categories/#source-code-group) for the Templates feature. -## Add the template SVG icon to GitLab SVGs +### Add the template SVG icon to GitLab SVGs If the project template has an SVG icon, you must add it to the [GitLab SVGs project](https://gitlab.com/gitlab-org/gitlab-svgs/-/blob/main/README.md#adding-icons-or-illustrations) before you can create a merge request with vendor details. -## Create a merge request with vendor details +### Create a merge request with vendor details Before GitLab can implement the project template, you must [create a merge request](../user/project/merge_requests/creating_merge_requests.md) in [`gitlab-org/gitlab`](https://gitlab.com/gitlab-org/gitlab) that includes vendor details about the project. @@ -111,7 +113,7 @@ Before GitLab can implement the project template, you must [create a merge reque 1. After you run the scripts, there is one new file in `vendor/project_templates/` and four changed files. Commit all changes and push your branch to update the merge request. For an example, see this [merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25318). -## Test your built-in project with the GitLab Development Kit +### Test your built-in project with the GitLab Development Kit Complete the following steps to test the project template in your own GitLab Development Kit instance: @@ -124,9 +126,24 @@ Complete the following steps to test the project template in your own GitLab Dev ## Contribute an improvement to an existing template -To update an existing built-in project template: +To update an existing built-in project template, changes are usually made to the existing template, found in the [project-templates](https://gitlab.com/gitlab-org/project-templates) group. A merge request is made directly against the template and the Create:Source Code [Backend Engineering Manager and Product Manager](https://about.gitlab.com/handbook/product/categories/#source-code-group) pinged for review. + +Sometimes it is necessary to completely replace the template files. In this case the process would be: 1. Create a merge request in the relevant project of the `project-templates` and `pages` group and mention `@gitlab-org/manage/import/backend` when you are ready for a review. 1. If your merge request is accepted, either: - [Create an issue](https://gitlab.com/gitlab-org/gitlab/-/issues) to ask for the template to get updated. - [Create a merge request with vendor details](#create-a-merge-request-with-vendor-details) to update the template. + +## For GitLab team members + +Please ensure the merge request has been reviewed by the Security Counterpart before merging. + +To review a merge request which changes a vendored project template, run the `check-template-changes` script: + +```shell +scripts/check-template-changes vendor/project_templates/<template_name>.tar.gz +``` + +This script outputs a diff of the file changes against the default branch and also verifies that +the template repository matches the source template project. diff --git a/doc/development/projections.md b/doc/development/projections.md index e45a16c267d..7c727fc0901 100644 --- a/doc/development/projections.md +++ b/doc/development/projections.md @@ -20,11 +20,11 @@ Projections are a way to define relations between files. Every file can have a You can find a basic list of projection options in [projectionist.txt](https://github.com/tpope/vim-projectionist/blob/master/doc/projectionist.txt) -## Which plugins can I use +## Which plugins can you use - vim - [vim-projectionist](https://github.com/tpope/vim-projectionist) -- VSCode +- VS Code - [Alternate File](https://marketplace.visualstudio.com/items?itemName=will-wow.vscode-alternate-file) - [projectionist](https://github.com/jarsen/projectionist) - [`jumpto`](https://github.com/gmdayley/jumpto) diff --git a/doc/development/prometheus_metrics.md b/doc/development/prometheus_metrics.md index d3d809c5386..456f2eb50aa 100644 --- a/doc/development/prometheus_metrics.md +++ b/doc/development/prometheus_metrics.md @@ -36,7 +36,7 @@ After you add or change an existing common metric, you must [re-run the import s Or, you can create a database migration: ```ruby -class ImportCommonMetrics < Gitlab::Database::Migration[2.0] +class ImportCommonMetrics < Gitlab::Database::Migration[2.1] def up ::Gitlab::DatabaseImporters::CommonMetrics::Importer.new.execute end diff --git a/doc/development/query_count_limits.md b/doc/development/query_count_limits.md deleted file mode 100644 index f16c8cfc6cd..00000000000 --- a/doc/development/query_count_limits.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'database/query_count_limits.md' -remove_date: '2022-11-06' ---- - -This document was moved to [another location](database/query_count_limits.md). - -<!-- This redirect file can be deleted after <2022-11-06>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/query_performance.md b/doc/development/query_performance.md deleted file mode 100644 index 618d007f766..00000000000 --- a/doc/development/query_performance.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'database/query_performance.md' -remove_date: '2022-11-06' ---- - -This document was moved to [another location](database/query_performance.md). - -<!-- This redirect file can be deleted after <2022-11-06>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/query_recorder.md b/doc/development/query_recorder.md deleted file mode 100644 index cb05bc604af..00000000000 --- a/doc/development/query_recorder.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'database/query_recorder.md' -remove_date: '2022-11-06' ---- - -This document was moved to [another location](database/query_recorder.md). - -<!-- This redirect file can be deleted after <2022-11-06>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/reactive_caching.md b/doc/development/reactive_caching.md index 3239748193f..9e3b1f58abe 100644 --- a/doc/development/reactive_caching.md +++ b/doc/development/reactive_caching.md @@ -208,7 +208,7 @@ There are some `class_attribute` options which can be tweaked. self.reactive_cache_key = ->(integration) { [integration.class.model_name.singular, integration.project_id] } ``` - If your reactive_cache_key is exactly like the above, you can use the existing + If your `reactive_cache_key` is exactly like the above, you can use the existing `Integrations::ReactivelyCached` concern instead. #### `self.reactive_cache_lease_timeout` diff --git a/doc/development/redis/new_redis_instance.md b/doc/development/redis/new_redis_instance.md index d9d4bb6a9f8..0a030a0877f 100644 --- a/doc/development/redis/new_redis_instance.md +++ b/doc/development/redis/new_redis_instance.md @@ -181,7 +181,7 @@ and the [old (fallback-instance)](#fallback-instance). If we fail to fetch data from the new instance, we will fallback and read from the old Redis instance. We can monitor logs for `Gitlab::Redis::MultiStore::ReadFromPrimaryError`, and also the Prometheus counter `gitlab_redis_multi_store_read_fallback_total`. -For pipelined commands (`pipelined` and `multi`), we execute the entire operation in both stores and then compare the results. If they differ, we emit a +For `pipelined` commands (`pipelined` and `multi`), we execute the entire operation in both stores and then compare the results. If they differ, we emit a `Gitlab::Redis::MultiStore:PipelinedDiffError` error, and track it in the `gitlab_redis_multi_store_pipelined_diff_error_total` Prometheus counter. Once we stop seeing those errors, this means that we are no longer relying on the data stored on the old Redis store. @@ -218,7 +218,7 @@ MultiStore implements read and write Redis commands separately. - `flushdb` - `rpush` -##### Pipelined commands +##### `pipelined` commands **NOTE:** The Ruby block passed to these commands will be executed twice, once per each store. Thus, excluding the Redis operations performed, the block should be idempotent. @@ -238,16 +238,16 @@ a developer will need to add an implementation for missing Redis commands before | error | message | |---------------------------------------------------|---------------------------------------------------------------------------------------------| | `Gitlab::Redis::MultiStore::ReadFromPrimaryError` | Value not found on the Redis primary store. Read from the Redis secondary store successful. | -| `Gitlab::Redis::MultiStore::PipelinedDiffError` | Pipelined command executed on both stores successfully but results differ between them. | +| `Gitlab::Redis::MultiStore::PipelinedDiffError` | `pipelined` command executed on both stores successfully but results differ between them. | | `Gitlab::Redis::MultiStore::MethodMissingError` | Method missing. Falling back to execute method on the Redis secondary store. | ##### Metrics -| metrics name | type | labels | description | -|-------------------------------------------------------|--------------------|------------------------|--------------------------------------------------------| -| `gitlab_redis_multi_store_read_fallback_total` | Prometheus Counter | command, instance_name | Client side Redis MultiStore reading fallback total | -| `gitlab_redis_multi_store_pipelined_diff_error_total` | Prometheus Counter | command, instance_name | Redis MultiStore pipelined command diff between stores | -| `gitlab_redis_multi_store_method_missing_total` | Prometheus Counter | command, instance_name | Client side Redis MultiStore method missing total | +| Metrics name | Type | Labels | Description | +|-------------------------------------------------------|--------------------|----------------------------|----------------------------------------------------------| +| `gitlab_redis_multi_store_read_fallback_total` | Prometheus Counter | `command`, `instance_name` | Client side Redis MultiStore reading fallback total | +| `gitlab_redis_multi_store_pipelined_diff_error_total` | Prometheus Counter | `command`, `instance_name` | Redis MultiStore `pipelined` command diff between stores | +| `gitlab_redis_multi_store_method_missing_total` | Prometheus Counter | `command`, `instance_name` | Client side Redis MultiStore method missing total | ## Step 4: clean up after the migration diff --git a/doc/development/reusing_abstractions.md b/doc/development/reusing_abstractions.md index 58d1e20394c..e3f523fc6a7 100644 --- a/doc/development/reusing_abstractions.md +++ b/doc/development/reusing_abstractions.md @@ -198,7 +198,7 @@ Several base classes implement the service classes convention. You may consider - `BaseGroupService` for services scoped to groups. Classes that are not service objects should be -[created elsewhere](directory_structure.md#use-namespaces-to-define-bounded-contexts), +[created elsewhere](software_design.md#use-namespaces-to-define-bounded-contexts), such as in `lib`. #### ServiceResponse diff --git a/doc/development/scalability.md b/doc/development/scalability.md index 671086f33b2..de9c57c2f2a 100644 --- a/doc/development/scalability.md +++ b/doc/development/scalability.md @@ -267,7 +267,7 @@ However, there are a number of strategies to ensure queues get drained in a timely manner: - Add more processing capacity. This can be done by spinning up more - instances of Sidekiq or [Sidekiq Cluster](../administration/operations/extra_sidekiq_processes.md). + instances of Sidekiq or [Sidekiq Cluster](../administration/sidekiq/extra_sidekiq_processes.md). - Split jobs into smaller units of work. For example, `PostReceive` used to process each commit message in the push, but now it farms out this to `ProcessCommitWorker`. diff --git a/doc/development/sec/analyzer_development_guide.md b/doc/development/sec/analyzer_development_guide.md index af3a6f2b7d7..4fb32785b9f 100644 --- a/doc/development/sec/analyzer_development_guide.md +++ b/doc/development/sec/analyzer_development_guide.md @@ -21,7 +21,7 @@ There are a number of shared Go modules shared across analyzers for common behav ## How to use the analyzers Analyzers are shipped as Docker images. For example, to run the -[semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) Docker image to scan the working directory: +[Semgrep](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) Docker image to scan the working directory: 1. `cd` into the directory of the source code you want to scan. 1. Run `docker login registry.gitlab.com` and provide username plus diff --git a/doc/development/sec/index.md b/doc/development/sec/index.md index fc13c960451..3f52020701f 100644 --- a/doc/development/sec/index.md +++ b/doc/development/sec/index.md @@ -102,15 +102,15 @@ After being [merged](../integrations/secure.md#tracking-and-merging-vulnerabilit ### Analyzer vulnerability translation -In the case of SAST's semgrep analyzer, there is a secondary identifier of particular importance: the identifier linking the report’s vulnerability -to the legacy analyzer (that is, bandit or eslint). +In the case of the SAST Semgrep analyzer, there is a secondary identifier of particular importance: the identifier linking the report’s vulnerability +to the legacy analyzer (that is, bandit or ESLint). To [enable vulnerability translation](../../user/application_security/sast/analyzers.md#vulnerability-translation) -the semgrep analyzer relies on a secondary identifier exactly matching the primary identifier of the legacy analyzer. +the Semgrep analyzer relies on a secondary identifier exactly matching the primary identifier of the legacy analyzer. For example, when [`eslint`](https://gitlab.com/gitlab-org/security-products/analyzers/eslint) was previously used to generate vulnerability records, the [`semgrep`](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep) analyzer must produce an identifier collection containing the -original eslint primary identifier. +original ESLint primary identifier. Given the original `eslint` report: @@ -131,7 +131,7 @@ Given the original `eslint` report: } ``` -The corresponding semgrep report must contain the `eslint_rule_id`: +The corresponding Semgrep report must contain the `eslint_rule_id`: ```json { diff --git a/doc/development/secure_coding_guidelines.md b/doc/development/secure_coding_guidelines.md index c102e99720f..bccdda9ca04 100644 --- a/doc/development/secure_coding_guidelines.md +++ b/doc/development/secure_coding_guidelines.md @@ -410,7 +410,7 @@ References: #### XSS mitigation and prevention in JavaScript and Vue - When updating the content of an HTML element using JavaScript, mark user-controlled values as `textContent` or `nodeValue` instead of `innerHTML`. -- Avoid using `v-html` with user-controlled data, use [`v-safe-html`](https://gitlab-org.gitlab.io/gitlab-ui/?path=/story/directives-safe-html-directive--default) instead. +- Avoid using `v-html` with user-controlled data, use [`v-safe-html`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/javascripts/vue_shared/directives/safe_html.js) instead. - Render unsafe or unsanitized content using [`dompurify`](fe_guide/security.md#sanitize-html-output). - Consider using [`gl-sprintf`](../../ee/development/i18n/externalization.md#interpolation) to interpolate translated strings securely. - Avoid `__()` with translations that contain user-controlled values. @@ -422,7 +422,7 @@ References: ##### Vue - [isSafeURL](https://gitlab.com/gitlab-org/gitlab/-/blob/v12.7.5-ee/app/assets/javascripts/lib/utils/url_utility.js#L190-207) -- [GlSprintf](https://gitlab-org.gitlab.io/gitlab-ui/?path=/story/utilities-sprintf--default) +- [GlSprintf](https://gitlab-org.gitlab.io/gitlab-ui/?path=/docs/utilities-sprintf--sentence-with-link) #### Content Security Policy @@ -718,13 +718,12 @@ There are some cases where `users` passed in the code is actually referring to a ```ruby def find_user_from_sources - strong_memoize(:find_user_from_sources) do - deploy_token_from_request || - find_user_from_bearer_token || - find_user_from_job_token || - user_from_warden - end + deploy_token_from_request || + find_user_from_bearer_token || + find_user_from_job_token || + user_from_warden end + strong_memoize_attr :find_user_from_sources ``` ### Past Vulnerable Code @@ -1210,7 +1209,7 @@ These types of bugs are often seen in environments which allow multi-threading a ### Examples -**Example 1:** you have a model which accepts a URL as input. When the model is created you verify that the URL's host resolves to a public IP address, to prevent attackers making internal network calls. But DNS records can change ([DNS rebinding](#server-side-request-forgery-ssrf)]). An attacker updates the DNS record to `127.0.0.1`, and when your code resolves those URL's host it results in sending a potentially malicious request to a server on the internal network. The property was valid at the "time of check", but invalid and malicious at "time of use". +**Example 1:** you have a model which accepts a URL as input. When the model is created you verify that the URL host resolves to a public IP address, to prevent attackers making internal network calls. But DNS records can change ([DNS rebinding](#server-side-request-forgery-ssrf)]). An attacker updates the DNS record to `127.0.0.1`, and when your code resolves those URL host it results in sending a potentially malicious request to a server on the internal network. The property was valid at the "time of check", but invalid and malicious at "time of use". GitLab-specific example can be found in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/214401) where, although `Gitlab::UrlBlocker.validate!` was called, the returned value was not used. This made it vulnerable to TOCTOU bug and SSRF protection bypass through [DNS rebinding](#server-side-request-forgery-ssrf). The fix was to [use the validated IP address](https://gitlab.com/gitlab-org/gitlab/-/commit/7af8abd4df9a98f7a1ae7c4ec9840d0a7a8c684d). diff --git a/doc/development/serializing_data.md b/doc/development/serializing_data.md deleted file mode 100644 index aa8b20eded7..00000000000 --- a/doc/development/serializing_data.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'database/serializing_data.md' -remove_date: '2022-11-06' ---- - -This document was moved to [another location](database/serializing_data.md). - -<!-- This redirect file can be deleted after <2022-11-06>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/service_ping/implement.md b/doc/development/service_ping/implement.md index 5a564b2d83e..70da97502bb 100644 --- a/doc/development/service_ping/implement.md +++ b/doc/development/service_ping/implement.md @@ -181,7 +181,7 @@ The highest encountered error rate is 4.9%. When correctly used, the `estimate_batch_distinct_count` method enables efficient counting over columns that contain non-unique values, which cannot be assured by other counters. -##### estimate_batch_distinct_count method +##### `estimate_batch_distinct_count` method Method: @@ -316,7 +316,7 @@ HyperLogLog (HLL) is a probabilistic algorithm and its **results always includes used HLL implementation is "approximated with a standard error of 0.81%". NOTE: - A user's consent for usage_stats (`User.single_user&.requires_usage_stats_consent?`) is not checked during the data tracking stage due to performance reasons. Keys corresponding to those counters are present in Redis even if `usage_stats_consent` is still required. However, no metric is collected from Redis and reported back to GitLab as long as `usage_stats_consent` is required. + A user's consent for `usage_stats` (`User.single_user&.requires_usage_stats_consent?`) is not checked during the data tracking stage due to performance reasons. Keys corresponding to those counters are present in Redis even if `usage_stats_consent` is still required. However, no metric is collected from Redis and reported back to GitLab as long as `usage_stats_consent` is required. With `Gitlab::UsageDataCounters::HLLRedisCounter` we have available data structures used to count unique values. @@ -509,7 +509,7 @@ We have the following recommendations for [adding new events](#add-new-events): Events are tracked behind optional [feature flags](../feature_flags/index.md) due to concerns for Redis performance and scalability. -For a full list of events and corresponding feature flags see, [known_events](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events/) files. +For a full list of events and corresponding feature flags, see the [`known_events/`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events/) files. To enable or disable tracking for specific event in <https://gitlab.com> or <https://about.staging.gitlab.com>, run commands such as the following to [enable or disable the corresponding feature](../feature_flags/index.md). @@ -870,7 +870,7 @@ these steps: Only metrics calculated with [Estimated Batch Counters](#estimated-batch-counters) can be persisted for database sourced aggregated metrics. To persist a metric, inject a Ruby block into the -[estimate_batch_distinct_count](#estimate_batch_distinct_count-method) method. +[`estimate_batch_distinct_count`](#estimate_batch_distinct_count-method) method. This block should invoke the `Gitlab::Usage::Metrics::Aggregates::Sources::PostgresHll.save_aggregated_metrics` [method](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage/metrics/aggregates/sources/postgres_hll.rb#L21), diff --git a/doc/development/service_ping/metrics_dictionary.md b/doc/development/service_ping/metrics_dictionary.md index 3439f581e7f..49f8a5ac465 100644 --- a/doc/development/service_ping/metrics_dictionary.md +++ b/doc/development/service_ping/metrics_dictionary.md @@ -41,7 +41,7 @@ Each metric is defined in a separate YAML file consisting of a number of fields: | `value_type` | yes | `string`; one of [`string`, `number`, `boolean`, `object`](https://json-schema.org/understanding-json-schema/reference/type.html). | | `status` | yes | `string`; [status](#metric-statuses) of the metric, may be set to `active`, `removed`, `broken`. | | `time_frame` | yes | `string`; may be set to a value like `7d`, `28d`, `all`, `none`. | -| `data_source` | yes | `string`; may be set to a value like `database`, `redis`, `redis_hll`, `prometheus`, `system`. | +| `data_source` | yes | `string`; may be set to a value like `database`, `redis`, `redis_hll`, `prometheus`, `system`, `license`. | | `data_category` | yes | `string`; [categories](#data-category) of the metric, may be set to `operational`, `optional`, `subscription`, `standard`. The default value is `optional`.| | `instrumentation_class` | yes | `string`; [the class that implements the metric](metrics_instrumentation.md). | | `distribution` | yes | `array`; may be set to one of `ce, ee` or `ee`. The [distribution](https://about.gitlab.com/handbook/marketing/strategic-marketing/tiers/#definitions) where the tracked feature is available. | @@ -55,7 +55,7 @@ Each metric is defined in a separate YAML file consisting of a number of fields: | `options` | no | `object`: options information needed to calculate the metric value. | | `skip_validation` | no | This should **not** be set. [Used for imported metrics until we review, update and make them valid](https://gitlab.com/groups/gitlab-org/-/epics/5425). | -### Metric key_path +### Metric `key_path` The `key_path` of the metric is the location in the JSON Service Ping payload. @@ -108,7 +108,7 @@ Metric definitions can have one of the following statuses: - `broken`: Metric reports broken data (for example, -1 fallback), or does not report data at all. A metric marked as `broken` must also have the `repair_issue_url` attribute. - `removed`: Metric was removed, but it may appear in Service Ping payloads sent from instances running on older versions of GitLab. -### Metric value_type +### Metric `value_type` Metric definitions can have one of the following values for `value_type`: @@ -120,12 +120,23 @@ In general, we avoid complex objects and prefer one of the `boolean`, `number`, An example of a metric that uses `value_type: object` is `topology` (`/config/metrics/settings/20210323120839_topology.yml`), which has a related schema in `/config/metrics/objects_schemas/topology_schema.json`. -### Metric time_frame - -- `7d`: The metric data applies to the most recent 7-day interval. For example, the following metric counts the number of users that create epics over a 7-day interval: `ee/config/metrics/counts_7d/20210305145820_g_product_planning_epic_created_weekly.yml`. -- `28d`: The metric data applies to the most recent 28-day interval. For example, the following metric counts the number of unique users that create issues over a 28-day interval: `config/metrics/counts_28d/20210216181139_issues.yml`. -- `all`: The metric data applies for the whole time the metric has been active (all-time interval). For example, the following metric counts all users that create issues: `/config/metrics/counts_all/20210216181115_issues.yml`. -- `none`: The metric collects a type of data that's not tracked over time, such as settings and configuration information. Therefore, a time interval is not applicable. For example, `uuid` has no time interval applicable: `config/metrics/license/20210201124933_uuid.yml`. +### Metric `time_frame` + +A metric's time frame is calculated based on the `time_frame` field and the `data_source` of the metric. +For `redis_hll` metrics, the type of aggregation is also taken into consideration. In this context, the term "aggregation" refers to [chosen events data storage interval](implement.md#add-new-events), and is **NOT** related to the Aggregated Metrics feature. +For more information about the aggregation type of each feature, see the [`common.yml` file](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events/common.yml). Weeks run from Monday to Sunday. + +| data_source | time_frame | aggregation | Description | +|------------------------|------------|----------------|-------------------------------------------------| +| any | `none` | not applicable | A type of data that’s not tracked over time, such as settings and configuration information | +| `database` | `all` | not applicable | The whole time the metric has been active (all-time interval) | +| `database` | `7d` | not applicable | 9 days ago to 2 days ago | +| `database` | `28d` | not applicable | 30 days ago to 2 days ago | +| `redis` | `all` | not applicable | The whole time the metric has been active (all-time interval) | +| `redis_hll` | `7d` | `daily` | Most recent 7 complete days | +| `redis_hll` | `7d` | `weekly` | Most recent complete week | +| `redis_hll` | `28d` | `daily` | Most recent 28 complete days | +| `redis_hll` | `28d` | `weekly` | Most recent 4 complete weeks | ### Data category diff --git a/doc/development/service_ping/metrics_instrumentation.md b/doc/development/service_ping/metrics_instrumentation.md index a9f236819fe..5cc8a7811d7 100644 --- a/doc/development/service_ping/metrics_instrumentation.md +++ b/doc/development/service_ping/metrics_instrumentation.md @@ -464,3 +464,15 @@ This guide describes how to migrate a Service Ping metric from [`lib/gitlab/usag 1. Remove the code from [`lib/gitlab/usage_data.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data.rb) or [`ee/lib/ee/gitlab/usage_data.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/ee/gitlab/usage_data.rb). 1. Remove the tests from [`spec/lib/gitlab/usage_data.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/lib/gitlab/usage_data_spec.rb) or [`ee/spec/lib/ee/gitlab/usage_data.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/spec/lib/ee/gitlab/usage_data_spec.rb). + +## Troubleshoot metrics + +Sometimes metrics fail for reasons that are not immediately clear. The failures can be related to performance issues or other problems. +The following pairing session video gives you an example of an investigation in to a real-world failing metric. + +<div class="video-fallback"> + See the video from: <a href="https://www.youtube.com/watch?v=y_6m2POx2ug">Product Intelligence Office Hours Oct 27th</a> to learn more about the metrics troubleshooting process. +</div> +<figure class="video-container"> + <iframe src="https://www.youtube.com/embed/y_6m2POx2ug" frameborder="0" allowfullscreen="true"> </iframe> +</figure> diff --git a/doc/development/service_ping/troubleshooting.md b/doc/development/service_ping/troubleshooting.md index f8fd45e6062..3b7cd092d97 100644 --- a/doc/development/service_ping/troubleshooting.md +++ b/doc/development/service_ping/troubleshooting.md @@ -30,7 +30,7 @@ For results about an investigation conducted into an unexpected drop in Service Check if the [export jobs](https://gitlab.com/gitlab-services/version-gitlab-com#data-export-using-pipeline-schedules) are successful. -Check [Service Ping errors](https://app.periscopedata.com/app/gitlab/968489/Product-Intelligence---Service-Ping-Health?widget=14609989&udv=0) in the [Service Ping Health Dahsboard](https://app.periscopedata.com/app/gitlab/968489/Product-Intelligence---Service-Ping-Health). +Check [Service Ping errors](https://app.periscopedata.com/app/gitlab/968489/Product-Intelligence---Service-Ping-Health?widget=14609989&udv=0) in the [Service Ping Health Dashboard](https://app.periscopedata.com/app/gitlab/968489/Product-Intelligence---Service-Ping-Health). ### Troubleshoot Google Storage layer diff --git a/doc/development/sha1_as_binary.md b/doc/development/sha1_as_binary.md deleted file mode 100644 index 7f928d09470..00000000000 --- a/doc/development/sha1_as_binary.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'database/sha1_as_binary.md' -remove_date: '2022-11-06' ---- - -This document was moved to [another location](database/sha1_as_binary.md). - -<!-- This redirect file can be deleted after <2022-11-06>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/sidekiq/compatibility_across_updates.md b/doc/development/sidekiq/compatibility_across_updates.md index cfb709d9110..b417a099228 100644 --- a/doc/development/sidekiq/compatibility_across_updates.md +++ b/doc/development/sidekiq/compatibility_across_updates.md @@ -123,14 +123,62 @@ uses a parameter hash. end ``` -## Removing workers +## Removing worker classes -Try to avoid removing workers and their queues in minor and patch -releases. +To remove a worker class, follow these steps over two minor releases: -During online update instance can have pending jobs and removing the queue can -lead to those jobs being stuck forever. If you can't write migration for those -Sidekiq jobs, please consider removing the worker in a major release only. +### In the first minor release + +1. Remove any code that enqueues the jobs. + + For example, if there is a UI component or an API endpoint that a user can interact with that results in the worker instance getting enqueued, make sure those surface areas are either removed or updated in a way that the worker instance is no longer enqueued. + + This ensures that instances related to the worker class are no longer being enqueued. + +1. Ensure both the frontend and backend code no longer relies on any of the work that used to be done by the worker. +1. In the relevant worker classes, replace the contents of the `perform` method with a no-op, while keeping any arguments in tact. + + For example, if you're working with the following `ExampleWorker`: + + ```ruby + class ExampleWorker + def perform(object_id) + SomeService.run!(object_id) + end + end + ``` + + Implementing the no-op might look like this: + + ```ruby + class ExampleWorker + def perform(object_id); end + end + ``` + + By implementing this no-op, you can avoid unnecessary cycles once any deprecated jobs that are still enqueued eventually get processed. + +### In a subsequent, separate minor release + +1. Delete the worker class file and follow the guidance in our [Sidekiq queues documentation](../sidekiq/index.md#sidekiq-queues) around running Rake tasks to regenerate/update related files. +1. Add a migration (not a post-deployment migration) that uses `sidekiq_remove_jobs`: + + ```ruby + class RemoveMyDeprecatedWorkersJobInstances < Gitlab::Database::Migration[2.0] + DEPRECATED_JOB_CLASSES = %w[ + MyDeprecatedWorkerOne + MyDeprecatedWorkerTwo + ] + + def up + sidekiq_remove_jobs(job_klasses: DEPRECATED_JOB_CLASSES) + end + + def down + # This migration removes any instances of deprecated workers and cannot be undone. + end + end + ``` ## Renaming queues @@ -141,7 +189,7 @@ When renaming queues, use the `sidekiq_queue_migrate` helper migration method in a **post-deployment migration**: ```ruby -class MigrateTheRenamedSidekiqQueue < Gitlab::Database::Migration[2.0] +class MigrateTheRenamedSidekiqQueue < Gitlab::Database::Migration[2.1] restrict_gitlab_migration gitlab_schema: :gitlab_main disable_ddl_transaction! diff --git a/doc/development/sidekiq/idempotent_jobs.md b/doc/development/sidekiq/idempotent_jobs.md index 80c6c403549..1c4f4ba44a8 100644 --- a/doc/development/sidekiq/idempotent_jobs.md +++ b/doc/development/sidekiq/idempotent_jobs.md @@ -156,7 +156,7 @@ end ## Setting the deduplication time-to-live (TTL) -Deduplication depends on an idempotency key that is stored in Redis. This is normally +Deduplication depends on an idempotent key that is stored in Redis. This is normally cleared by the configured deduplication strategy. However, the key can remain until its TTL in certain cases like: @@ -189,7 +189,7 @@ that can tolerate some duplication. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/69372) in GitLab 14.3. > - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/338350) in GitLab 14.4. > - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/338350) in GitLab 14.6. -> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/346598) in GitLab 14.9. [Feature flag preserve_latest_wal_locations_for_idempotent_jobs](https://gitlab.com/gitlab-org/gitlab/-/issues/346598) removed. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/346598) in GitLab 14.9. [Feature flag `preserve_latest_wal_locations_for_idempotent_jobs`](https://gitlab.com/gitlab-org/gitlab/-/issues/346598) removed. The deduplication always take into account the latest binary replication pointer, not the first one. This happens because we drop the same job scheduled for the second time and the Write-Ahead Log (WAL) is lost. diff --git a/doc/development/sidekiq/index.md b/doc/development/sidekiq/index.md index a95e94cdd34..c9d783377bd 100644 --- a/doc/development/sidekiq/index.md +++ b/doc/development/sidekiq/index.md @@ -14,7 +14,7 @@ information on administering GitLab, see [configuring Sidekiq](../../administrat There are pages with additional detail on the following topics: 1. [Compatibility across updates](compatibility_across_updates.md) -1. [Job idempotency and job deduplication](idempotent_jobs.md) +1. [Job idempotence and job deduplication](idempotent_jobs.md) 1. [Limited capacity worker: continuously performing work with a specified concurrency](limited_capacity_worker.md) 1. [Logging](logging.md) 1. [Worker attributes](worker_attributes.md) @@ -27,7 +27,7 @@ There are pages with additional detail on the following topics: All workers should include `ApplicationWorker` instead of `Sidekiq::Worker`, which adds some convenience methods and automatically sets the queue based on -the [routing rules](../../administration/sidekiq/extra_sidekiq_routing.md#queue-routing-rules). +the [routing rules](../../administration/sidekiq/processing_specific_job_classes.md#routing-rules). ## Retries @@ -88,7 +88,7 @@ error rate. Previously, each worker had its own queue, which was automatically set based on the worker class name. For a worker named `ProcessSomethingWorker`, the queue name would be `process_something`. You can now route workers to a specific queue using -[queue routing rules](../../administration/sidekiq/extra_sidekiq_routing.md#queue-routing-rules). +[queue routing rules](../../administration/sidekiq/processing_specific_job_classes.md#routing-rules). In GDK, new workers are routed to a queue named `default`. If you're not sure what queue a worker uses, diff --git a/doc/development/single_table_inheritance.md b/doc/development/single_table_inheritance.md deleted file mode 100644 index da8d48f2a42..00000000000 --- a/doc/development/single_table_inheritance.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'database/single_table_inheritance.md' -remove_date: '2022-11-06' ---- - -This document was moved to [another location](database/single_table_inheritance.md). - -<!-- This redirect file can be deleted after <2022-11-06>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/snowplow/index.md b/doc/development/snowplow/index.md index 8d05e05e592..32628744a23 100644 --- a/doc/development/snowplow/index.md +++ b/doc/development/snowplow/index.md @@ -17,7 +17,7 @@ Snowplow is an enterprise-grade marketing and Product Intelligence platform that - **Data modeling** joins event-level data with other data sets, aggregates them into smaller data sets, and applies business logic. This produces a clean set of tables for data analysis. We use data models for Redshift and Looker. - **Analytics** are performed on Snowplow events or on aggregate tables. - + ## Enable Snowplow tracking diff --git a/doc/development/snowplow/schemas.md b/doc/development/snowplow/schemas.md index 5a6cdea9fee..da58cd5f2e5 100644 --- a/doc/development/snowplow/schemas.md +++ b/doc/development/snowplow/schemas.md @@ -68,8 +68,8 @@ Page titles are hardcoded as `GitLab` for the same reason. | `doc_charset` | **{dotted-circle}** | string | Web page's character encoding | | `doc_height` | **{dotted-circle}** | string | Web page height | | `doc_width` | **{dotted-circle}** | string | Web page width | -| `domain_sessionid` | **{dotted-circle}** | string | Unique identifier (UUID) for this visit of this user_id to this domain | -| `domain_sessionidx` | **{dotted-circle}** | integer | Index of number of visits that this user_id has made to this domain (The first visit is `1`) | +| `domain_sessionid` | **{dotted-circle}** | string | Unique identifier (UUID) for this visit of this `user_id` to this domain | +| `domain_sessionidx` | **{dotted-circle}** | integer | Index of number of visits that this `user_id` has made to this domain (The first visit is `1`) | | `domain_userid` | **{dotted-circle}** | string | Unique identifier for a user, based on a first party cookie (so domain specific) | | `dvce_created_tstamp` | **{dotted-circle}** | timestamp | Timestamp when event occurred, as recorded by client device | | `dvce_ismobile` | **{dotted-circle}** | boolean | Indicates whether device is mobile | diff --git a/doc/development/snowplow/troubleshooting.md b/doc/development/snowplow/troubleshooting.md index 306040f8c9c..47df3e43d57 100644 --- a/doc/development/snowplow/troubleshooting.md +++ b/doc/development/snowplow/troubleshooting.md @@ -35,7 +35,7 @@ Drop occurring at application layer can be symptom of some issue, but it might b or even a result of a public holiday in some regions of the world with a larger user-base. To verify if there is an underlying problem to solve, you can check following things: 1. Check `about.gitlab.com` website traffic on [Google Analytics](https://analytics.google.com/analytics/web/) to verify if some public holiday might impact overall use of GitLab system - 1. You may require to open an access request for Google Analytics access first eg: [access request internal issue](https://gitlab.com/gitlab-com/team-member-epics/access-requests/-/issues/1772) + 1. You may require to open an access request for Google Analytics access first, for example: [access request internal issue](https://gitlab.com/gitlab-com/team-member-epics/access-requests/-/issues/1772) 1. Plot `select date(dvce_created_tstamp) , event , count(*) from legacy.snowplow_unnested_events_90 where dvce_created_tstamp > '2021-06-15' and dvce_created_tstamp < '2021-07-10' group by 1 , 2 order by 1 , 2` in SiSense to see what type of events was responsible for drop 1. Plot `select date(dvce_created_tstamp) ,se_category , count(*) from legacy.snowplow_unnested_events_90 where dvce_created_tstamp > '2021-06-15' and dvce_created_tstamp < '2021-07-31' and event = 'struct' group by 1 , 2 order by 1, 2` what events recorded the biggest drops in suspected category 1. Check if there was any MR merged that might cause reduction in reported events, pay an attention to ~"product intelligence" and ~"growth experiment" labeled MRs diff --git a/doc/development/software_design.md b/doc/development/software_design.md new file mode 100644 index 00000000000..03cbbb13d9f --- /dev/null +++ b/doc/development/software_design.md @@ -0,0 +1,141 @@ +--- +stage: none ++group: Engineering Productivity +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Software design guides + +## Use ubiquitous language instead of CRUD terminology + +The code should use the same [ubiquitous language](https://about.gitlab.com/handbook/communication/#ubiquitous-language) +as used in the product and user documentation. Failure to use ubiquitous language correctly +can be a major cause of confusion for contributors and customers when there is constant translation +or use of multiple terms. +This also goes against our [communication strategy](https://about.gitlab.com/handbook/communication/#mecefu-terms). + +In the example below, [CRUD](https://en.wikipedia.org/wiki/Create,_read,_update_and_delete) +terminology introduces ambiguity. The name says we are creating an `epic_issues` +association record, but we are adding an existing issue to an epic. The name `epic_issues`, +used from Rails convention, leaks to higher abstractions such as service objects. +The code speaks the framework jargon rather than ubiquitous language. + +```ruby +# Bad +EpicIssues::CreateService +``` + +Using ubiquitous language makes the code clear and doesn't introduce any +cognitive load to a reader trying to translate the framework jargon. + +```ruby +# Good +Epic::AddExistingIssueService +``` + +You can use CRUD when representing simple concepts that are not ambiguous, +like creating a project, and when matching the existing ubiquitous language. + +```ruby +# OK: Matches the product language. +Projects::CreateService +``` + +New classes and database tables should use ubiquitous language. In this case the model name +and table name follow the Rails convention. + +Existing classes that don't follow ubiquitous language should be renamed, when possible. +Some low level abstractions such as the database tables don't need to be renamed. +For example, use `self.table_name=` when the model name diverges from the table name. + +We can allow exceptions only when renaming is challenging. For example, when the naming is used +for STI, exposed to the user, or if it would be a breaking change. + +## Use namespaces to define bounded contexts + +A healthy application is divided into macro and sub components that represent the contexts at play, +whether they are related to business domain or infrastructure code. + +As GitLab code has so many features and components it's hard to see what contexts are involved. +We should expect any class to be defined inside a module/namespace that represents the contexts where it operates. + +When we namespace classes inside their domain: + +- Similar terminology becomes unambiguous as the domain clarifies the meaning: + For example, `MergeRequests::Diff` and `Notes::Diff`. +- Top-level namespaces could be associated to one or more groups identified as domain experts. +- We can better identify the interactions and coupling between components. + For example, several classes inside `MergeRequests::` domain interact more with `Ci::` + domain and less with `ImportExport::`. + +A good guideline for naming a top-level namespace (bounded context) is to use the related +[feature category](https://gitlab.com/gitlab-com/www-gitlab-com/-/blob/master/data/categories.yml). +For example, `Continuous Integration` feature category maps to `Ci::` namespace. + +```ruby +# bad +class JobArtifact +end + +# good +module Ci + class JobArtifact + end +end +``` + +Projects and Groups are generally container concepts because they identify tenants. +They allow features to exist at the project or group level, like repositories or runners, +but do not nest such features under `Projects::` or `Groups::`. + +`Projects::` and `Groups::` namespaces should be used only for concepts that are strictly related to them: +for example `Project::CreateService` or `Groups::TransferService`. + +For controllers we allow `app/controllers/projects` and `app/controllers/groups` to be exceptions. +We use this convention to indicate the scope of a given web endpoint. + +Do not use the [stage or group name](https://about.gitlab.com/handbook/product/categories/#devops-stages) +because a feature category could be reassigned to a different group in the future. + +```ruby +# bad +module Create + class Commit + end +end + +# good +module Repositories + class Commit + end +end +``` + +On the other hand, a feature category may sometimes be too granular. Features tend to be +treated differently according to Product and Marketing, while they may share a lot of +domain models and behavior under the hood. In this case, having too many bounded contexts +could make them shallow and more coupled with other contexts. + +Bounded contexts (or top-level namespaces) can be seen as macro-components in the overall app. +Good bounded contexts should be [deep](https://medium.com/@nakabonne/depth-of-module-f62dac3c2fdb) +so consider having nested namespaces to further break down complex parts of the domain. +For example, `Ci::Config::`. + +For example, instead of having separate and granular bounded contexts like: `ContainerScanning::`, +`ContainerHostSecurity::`, `ContainerNetworkSecurity::`, we could have: + +```ruby +module ContainerSecurity + module HostSecurity + end + + module NetworkSecurity + end + + module Scanning + end +end +``` + +If classes that are defined into a namespace have a lot in common with classes in other namespaces, +chances are that these two namespaces are part of the same bounded context. diff --git a/doc/development/spam_protection_and_captcha/exploratory_testing.md b/doc/development/spam_protection_and_captcha/exploratory_testing.md index f6e3e6814a8..1bcd336ce93 100644 --- a/doc/development/spam_protection_and_captcha/exploratory_testing.md +++ b/doc/development/spam_protection_and_captcha/exploratory_testing.md @@ -353,7 +353,7 @@ GraphQL response: } ``` -### Scenario: allow_possible_spam feature flag enabled +### Scenario: `allow_possible_spam` feature flag enabled With the `allow_possible_spam` feature flag enabled, the API returns a 200 response. Any valid request is successful and no CAPTCHA is presented, even if the request is considered diff --git a/doc/development/sql.md b/doc/development/sql.md index cdc952eb08b..5dbfd8f3ddb 100644 --- a/doc/development/sql.md +++ b/doc/development/sql.md @@ -103,7 +103,7 @@ transaction. Transactions for migrations can be disabled using the following pattern: ```ruby -class MigrationName < Gitlab::Database::Migration[2.0] +class MigrationName < Gitlab::Database::Migration[2.1] disable_ddl_transaction! end ``` @@ -111,7 +111,7 @@ end For example: ```ruby -class AddUsersLowerUsernameEmailIndexes < Gitlab::Database::Migration[2.0] +class AddUsersLowerUsernameEmailIndexes < Gitlab::Database::Migration[2.1] disable_ddl_transaction! def up diff --git a/doc/development/swapping_tables.md b/doc/development/swapping_tables.md deleted file mode 100644 index eaa6568dc36..00000000000 --- a/doc/development/swapping_tables.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'database/swapping_tables.md' -remove_date: '2022-11-04' ---- - -This document was moved to [another location](database/swapping_tables.md). - -<!-- This redirect file can be deleted after <2022-11-04>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index b6bf3c7805a..aee3e2871c2 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -425,11 +425,10 @@ results are available, and not just the first failure. when you need an ID/IID/access level that doesn't actually exists. Using 123, 1234, or even 999 is brittle as these IDs could actually exist in the database in the context of a CI run. -- All top-level `RSpec.describe` blocks should have [`feature_category`](https://about.gitlab.com/categories.json) metadata set. - Consider splitting the file in the case there are identified multiple feature categories in same file. - If no `feature_category` is identified then use `not_owned`. This information is used in flaky test - issues created in order to identify the group owning the feature. - Eg: `RSpec.describe Admin::Geo::SettingsController, :geo, feature_category: :geo_replication do`. + +### Feature category metadata + +You must [set feature category metadata for each RSpec example](../feature_categorization/index.md#rspec-examples). ### Coverage @@ -845,6 +844,8 @@ it 'is overdue' do travel_to(3.days.from_now) do expect(issue).to be_overdue end + + travel_back # Returns the current time back to its original state end ``` @@ -923,6 +924,21 @@ sequence-generated column. To avoid accidental conflicts, specs should also avoid manually specifying any values in these kinds of columns. Instead, leave them unspecified, and look up the value after the row is created. +##### TestProf in migration specs + +Because of what is described above, migration specs can't be run inside +a database transaction. Our test suite uses +[TestProf](https://github.com/test-prof/test-prof) to improve the runtime of the +test suite, but `TestProf` uses database transactions to perform these optimizations. +For this reason, we can't use `TestProf` methods in our migration specs. +These are the methods that should not be used and should be replaced with +default RSpec methods instead: + +- `let_it_be`: use `let` or `let!` instead. +- `let_it_be_with_reload`: use `let` or `let!` instead. +- `let_it_be_with_refind`: use `let` or `let!` instead. +- `before_all`: use `before` or `before(:all)` instead. + #### Redis GitLab stores two main categories of data in Redis: cached items, and Sidekiq @@ -1327,7 +1343,7 @@ Testing query performance allows us to: `QueryRecorder` allows profiling and testing of the number of database queries performed in a given block of code. -See the [`QueryRecorder`](../query_recorder.md) section for more details. +See the [`QueryRecorder`](../database/query_recorder.md) section for more details. #### GitalyClient @@ -1439,7 +1455,7 @@ or [cause the universe to implode](../contributing/verify/index.md#do-not-cause- ### Factories -GitLab uses [factory_bot](https://github.com/thoughtbot/factory_bot) as a test fixture replacement. +GitLab uses [`factory_bot`](https://github.com/thoughtbot/factory_bot) as a test fixture replacement. - Factory definitions live in `spec/factories/`, named using the pluralization of their corresponding model (`User` factories are defined in `users.rb`). @@ -1451,11 +1467,51 @@ GitLab uses [factory_bot](https://github.com/thoughtbot/factory_bot) as a test f resulting record to pass validation. - When instantiating from a factory, don't supply attributes that aren't required by the test. -- Prefer [implicit](https://github.com/thoughtbot/factory_bot/blob/master/GETTING_STARTED.md#implicit-definition), +- Use [implicit](https://github.com/thoughtbot/factory_bot/blob/master/GETTING_STARTED.md#implicit-definition), [explicit](https://github.com/thoughtbot/factory_bot/blob/master/GETTING_STARTED.md#explicit-definition), or [inline](https://github.com/thoughtbot/factory_bot/blob/master/GETTING_STARTED.md#inline-definition) associations - over `create` / `build` for association setup in callbacks. + instead of `create` / `build` for association setup in callbacks. See [issue #262624](https://gitlab.com/gitlab-org/gitlab/-/issues/262624) for further context. + + When creating factories with a [`has_many`](https://github.com/thoughtbot/factory_bot/blob/master/GETTING_STARTED.md#has_many-associations) and `belongs_to` association, use the `instance` method to refer to the object being built. + This prevents [creation of unnecessary records](https://gitlab.com/gitlab-org/gitlab/-/issues/378183) by using [interconnected associations](https://github.com/thoughtbot/factory_bot/blob/master/GETTING_STARTED.md#interconnected-associations). + + For example, if we have the following classes: + + ```ruby + class Car < ApplicationRecord + has_many :wheels, inverse_of: :car, foreign_key: :car_id + end + + class Wheel < ApplicationRecord + belongs_to :car, foreign_key: :car_id, inverse_of: :wheel, optional: false + end + ``` + + We can create the following factories: + + ```ruby + FactoryBot.define do + factory :car do + transient do + wheels_count { 2 } + end + + wheels do + Array.new(wheels_count) do + association(:wheel, car: instance) + end + end + end + end + + FactoryBot.define do + factory :wheel do + car { association :car } + end + end + ``` + - Factories don't have to be limited to `ActiveRecord` objects. [See example](https://gitlab.com/gitlab-org/gitlab-foss/commit/0b8cefd3b2385a21cfed779bd659978c0402766d). - Factories and their traits should produce valid objects that are [verified by specs](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/models/factories_spec.rb). diff --git a/doc/development/testing_guide/contract/index.md b/doc/development/testing_guide/contract/index.md index 8412a260c7d..08a21e58a52 100644 --- a/doc/development/testing_guide/contract/index.md +++ b/doc/development/testing_guide/contract/index.md @@ -26,6 +26,8 @@ The contracts themselves are stored in [`/spec/contracts/contracts`](https://git Before running the consumer tests, go to `spec/contracts/consumer` and run `npm install`. To run all the consumer tests, you just need to run `npm test -- /specs`. Otherwise, to run a specific spec file, replace `/specs` with the specific spec filename. +You can also run tests from the root directory of the project, using the command `yarn jest:contract`. + ### Run the provider tests Before running the provider tests, make sure your GDK (GitLab Development Kit) is fully set up and running. You can follow the setup instructions detailed in the [GDK repository](https://gitlab.com/gitlab-org/gitlab-development-kit/-/tree/main). To run the provider tests, you use Rake tasks that can be found in [`./lib/tasks/contracts`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/tasks/contracts). To get a list of all the Rake tasks related to the provider tests, run `bundle exec rake -T contracts`. For example: diff --git a/doc/development/testing_guide/end_to_end/feature_flags.md b/doc/development/testing_guide/end_to_end/feature_flags.md index 6d826e170f6..e473b158087 100644 --- a/doc/development/testing_guide/end_to_end/feature_flags.md +++ b/doc/development/testing_guide/end_to_end/feature_flags.md @@ -30,7 +30,7 @@ feature flag is under test. - Format: `feature_flag: { name: 'feature_flag_name', scope: :project }` - When `scope` is set to `:global`, the test will be **skipped on all live .com environments**. This is to avoid issues with feature flag changes affecting other tests or users on that environment. -- When `scope` is set to any other value (such as `:project`, `:group` or `:user`), or if no `scope` is specified, the test will only be **skipped on canary, production, and preprod**. +- When `scope` is set to any other value (such as `:project`, `:group` or `:user`), or if no `scope` is specified, the test will only be **skipped on canary, production, and pre-production**. This is due to the fact that administrator access is not available there. **WARNING:** You are strongly advised to first try and [enable feature flags only for a group, project, user](../../feature_flags/index.md#feature-actors), @@ -42,7 +42,7 @@ or [feature group](../../feature_flags/index.md#feature-groups). with administrator access, such as staging. **Note on `requires_admin`:** This tag should still be applied if there are other actions within the test that require administrator access that are unrelated to updating a -feature flag (ex: creating a user via the API). +feature flag (like creating a user via the API). The code below would enable a feature flag named `:feature_flag_name` for the project created by the test: diff --git a/doc/development/testing_guide/end_to_end/index.md b/doc/development/testing_guide/end_to_end/index.md index 8ffe044c4d8..55d725ba4ae 100644 --- a/doc/development/testing_guide/end_to_end/index.md +++ b/doc/development/testing_guide/end_to_end/index.md @@ -238,7 +238,7 @@ Each type of scheduled pipeline generates a static link for the latest test repo - [`production`](https://storage.googleapis.com/gitlab-qa-allure-reports/production-full/master/index.html) - [`production-sanity`](https://storage.googleapis.com/gitlab-qa-allure-reports/production-sanity/master/index.html) -## How do I run the tests? +## How do you run the tests? If you are not [testing code in a merge request](#testing-code-in-merge-requests), there are two main options for running the tests. If you want to run @@ -255,12 +255,12 @@ and the section below. Learn how to perform [tests that require special setup or consideration to run on your local environment](running_tests_that_require_special_setup.md). -## How do I write tests? +## How do you write tests? In order to write new tests, you first need to learn more about GitLab QA architecture. See the [documentation about it](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/architecture.md). -Once you decided where to put [test environment orchestration scenarios](https://gitlab.com/gitlab-org/gitlab-qa/tree/master/lib/gitlab/qa/scenario) and +After you've decided where to put [test environment orchestration scenarios](https://gitlab.com/gitlab-org/gitlab-qa/tree/master/lib/gitlab/qa/scenario) and [instance-level scenarios](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/qa/qa/specs/features), take a look at the [GitLab QA README](https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa/README.md), the [GitLab QA orchestrator README](https://gitlab.com/gitlab-org/gitlab-qa/tree/master/README.md), and [the already existing instance-level scenarios](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/qa/qa/specs/features). @@ -283,7 +283,7 @@ Continued reading: - [Execution context selection](execution_context_selection.md) - [Troubleshooting](troubleshooting.md) -## Where can I ask for help? +## Where can you ask for help? You can ask question in the `#quality` channel on Slack (GitLab internal) or you can find an issue you would like to work on in diff --git a/doc/development/testing_guide/end_to_end/page_objects.md b/doc/development/testing_guide/end_to_end/page_objects.md index 5fbf2ffbcde..6a599ce9a50 100644 --- a/doc/development/testing_guide/end_to_end/page_objects.md +++ b/doc/development/testing_guide/end_to_end/page_objects.md @@ -155,7 +155,7 @@ In our case, `data-qa-selector="login_field"`, `data-qa-selector="password_field Things to note: -- The name of the element and the `qa_selector` must match and be snake_cased +- The name of the element and the `qa_selector` must match and be snake cased - If the element appears on the page unconditionally, add `required: true` to the element. See [Dynamic element validation](dynamic_element_validation.md) - You may see `.qa-selector` classes in existing Page Objects. We should prefer the [`data-qa-selector`](#data-qa-selector-vs-qa-selector) diff --git a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md index a7d1ece77b2..c1389b3ac0e 100644 --- a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md +++ b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md @@ -15,7 +15,8 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec |-----------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `:elasticsearch` | The test requires an Elasticsearch service. It is used by the [instance-level scenario](https://gitlab.com/gitlab-org/gitlab-qa#definitions) [`Test::Integration::Elasticsearch`](https://gitlab.com/gitlab-org/gitlab/-/blob/72b62b51bdf513e2936301cb6c7c91ec27c35b4d/qa/qa/ee/scenario/test/integration/elasticsearch.rb) to include only tests that require Elasticsearch. | | `:except` | The test is to be run in their typical execution contexts _except_ as specified. See [test execution context selection](execution_context_selection.md) for more information. | -| `:feature_flag` | The test uses a feature flag and therefore requires an administrator account to run. When `scope` is set to `:global`, the test will be skipped on all live .com environments. Otherwise, it will be skipped only on Canary, Production, and Preprod. See [testing with feature flags](../../../development/testing_guide/end_to_end/feature_flags.md) for more details. | +| `:feature_flag` | The test uses a feature flag and therefore requires an administrator account to run. When `scope` is set to `:global`, the test will be skipped on all live .com environments. Otherwise, it will be skipped only on Canary, Production, and Pre-production. See [testing with feature flags](../../../development/testing_guide/end_to_end/feature_flags.md) for more details. | +| `:framework` | The test makes sanity assertions around the QA framework itself | | `:geo` | The test requires two GitLab Geo instances - a primary and a secondary - to be spun up. | | `:gitaly_cluster` | The test runs against a GitLab instance where repositories are stored on redundant Gitaly nodes behind a Praefect node. All nodes are [separate containers](../../../administration/gitaly/praefect.md#requirements). Tests that use this tag have a longer setup time since there are three additional containers that need to be started. | | `:github` | The test requires a GitHub personal access token. | diff --git a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md index 81e1c7d5dc0..4a947e59d5f 100644 --- a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md +++ b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md @@ -15,8 +15,8 @@ The project also has instructions for forking and building the images automatica Some extra environment variables for the location of the forked repository are also needed. -- `QA_THIRD_PARTY_DOCKER_REGISTRY` (the container registry where the repository/images are hosted, eg `registry.gitlab.com`) -- `QA_THIRD_PARTY_DOCKER_REPOSITORY` (the base repository path where the images are hosted, eg `registry.gitlab.com/<project path>`) +- `QA_THIRD_PARTY_DOCKER_REGISTRY` (the container registry where the repository/images are hosted, for example `registry.gitlab.com`) +- `QA_THIRD_PARTY_DOCKER_REPOSITORY` (the base repository path where the images are hosted, for example `registry.gitlab.com/<project path>`) - `QA_THIRD_PARTY_DOCKER_USER` (a username that has access to the container registry for this repository) - `QA_THIRD_PARTY_DOCKER_PASSWORD` (a password/token for the username to authenticate with) diff --git a/doc/development/testing_guide/end_to_end/troubleshooting.md b/doc/development/testing_guide/end_to_end/troubleshooting.md index e0925cb71f4..b4c47b90f0b 100644 --- a/doc/development/testing_guide/end_to_end/troubleshooting.md +++ b/doc/development/testing_guide/end_to_end/troubleshooting.md @@ -59,7 +59,7 @@ Net::ReadTimeout: Net::ReadTimeout with #<TCPSocket:(closed)> ``` This error can happen if GitLab runs on an address that does not resolve from -`localhost`. For example, if you set GDK's `hostname` +`localhost`. For example, if you set the GDK `hostname` [to a specific local IP address](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/run_qa_against_gdk.md#run-qa-tests-against-your-gdk-setup), you must use that IP address instead of `localhost` in the command. For example, if your IP is `192.168.0.12`: diff --git a/doc/development/testing_guide/flaky_tests.md b/doc/development/testing_guide/flaky_tests.md index cc62a0ebf03..ef5b75d166f 100644 --- a/doc/development/testing_guide/flaky_tests.md +++ b/doc/development/testing_guide/flaky_tests.md @@ -31,7 +31,7 @@ it's reset to a pristine test after each test. inconsistent state, so that following tests might not know about certain columns. - [Example 2](https://gitlab.com/gitlab-org/gitlab/-/issues/368500): A test modifies data that is used by a following test. -- [Example 3](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/103434#note_1172316521): A test for a database query passes in a fresh database, but in a +- [Example 3](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/103434#note_1172316521): A test for a database query passes in a fresh database, but in a CI/CD pipeline where the database is used to process previous test sequences, the test fails. This likely means that the query itself needs to be updated to work in a non-clean database. @@ -56,6 +56,7 @@ the problem. - [Example 1](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10148/diffs): Without specifying `ORDER BY`, database will not give deterministic ordering, or data race happening in the tests. +- [Example 2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/106936/diffs). ### Dataset-specific @@ -75,7 +76,7 @@ difficult to achieve locally. any table has more than 500 columns. It could pass in the merge request, but fail later in `master` if the order of tests changes. - [Example 2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91016/diffs): A test asserts - that trying to find a record with an unexisting ID retuns an error message. The test uses an + that trying to find a record with an nonexistent ID returns an error message. The test uses an hardcoded ID that's supposed to not exist (e.g. `42`). If the test is run early in the test suite, it might pass as not enough records were created before it, but as soon as it would run later in the suite, there could be a record that actually has the ID `42`, hence the test would @@ -104,6 +105,7 @@ or the app. **Description:** The DOM selector used in the test is unreliable. **Difficulty to reproduce:** Moderate to difficult. Depending on whether the DOM selector is duplicated, or appears after a delay etc. +Adding a delay in API or controller could help reproducing the issue. **Resolution:** It really depends on the problem here. It could be to wait for requests to finish, to scroll down the page etc. @@ -207,10 +209,10 @@ The `rspec/flaky/report-suite.json` report is: - [Sporadic RSpec failures due to `PG::UniqueViolation`](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/28307#note_24958837): <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/9846> - Follow-up: <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10688> - [Capybara.reset_session! should be called before requests are blocked](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/33779): <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12224> -- FFaker generates funky data that tests are not ready to handle (and tests should be predictable so that's bad!): +- ffaker generates funky data that tests are not ready to handle (and tests should be predictable so that's bad!): - [Make `spec/mailers/notify_spec.rb` more robust](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/20121): <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10015> - [Transient failure in `spec/requests/api/commits_spec.rb`](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/27988#note_25342521): <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/9944> - - [Replace FFaker factory data with sequences](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/29643): <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10184> + - [Replace ffaker factory data with sequences](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/29643): <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10184> - [Transient failure in spec/finders/issues_finder_spec.rb](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/30211#note_26707685): <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10404> ### Order-dependent flaky tests diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md index 041b0f0a4f4..2fa5fdeab7d 100644 --- a/doc/development/testing_guide/frontend_testing.md +++ b/doc/development/testing_guide/frontend_testing.md @@ -90,7 +90,7 @@ function getFahrenheit(celsius) { } ``` -It does not make sense to test our `getFahrenheit` function because underneath it does nothing else but invoking the library function, and we can expect that one is working as intended. (Simplified, I know) +It does not make sense to test our `getFahrenheit` function because underneath it does nothing else but invoking the library function, and we can expect that one is working as intended. Let's take a short look into Vue land. Vue is a critical part of the GitLab JavaScript codebase. When writing specs for Vue components, a common gotcha is to actually end up testing Vue provided functionality, because it appears to be the easiest thing to test. Here's an example taken from our codebase. @@ -522,7 +522,7 @@ it('waits for an event', () => { ### Ensuring that tests are isolated -Tests are normally architected in a pattern which requires a recurring setup and breakdown of the component under test. This is done by making use of the `beforeEach` and `afterEach` hooks. +Tests are normally architected in a pattern which requires a recurring setup of the component under test. This is often achieved by making use of the `beforeEach` hook. Example @@ -532,16 +532,22 @@ Example beforeEach(() => { wrapper = mount(Component); }); +``` + +With [enableAutoDestroy](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/100389), it is no longer necessary to manually call `wrapper.destroy()`. +However, some mocks, spies, and fixtures do need to be torn down, and we can leverage the `afterEach` hook. + +Example + +```javascript + let wrapper; afterEach(() => { - wrapper.destroy(); + fakeApollo = null; + store = null; }); ``` -When looking at this initially you'd suspect that the component is setup before each test and then broken down afterwards, providing isolation between tests. - -This is however not entirely true as the `destroy` method does not remove everything which has been mutated on the `wrapper` object. For functional components, destroy only removes the rendered DOM elements from the document. - ### Jest best practices > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/34209) in GitLab 13.2. @@ -713,7 +719,7 @@ unit testing by mocking out modules that cannot be easily consumed in our test e > Instead, consider using [`jest.mock(..)`](https://jestjs.io/docs/jest-object#jestmockmodulename-factory-options) > (or a similar mocking function) in the relevant spec file. -#### Where should I put manual mocks? +#### Where should you put manual mocks? Jest supports [manual module mocks](https://jestjs.io/docs/manual-mocks) by placing a mock in a `__mocks__/` directory next to the source module (for example, `app/assets/javascripts/ide/__mocks__`). **Don't do this.** We want to keep all of our test-related code in one place (the `spec/` folder). @@ -1159,7 +1165,7 @@ By now you've probably heard of [Jest snapshot tests](https://jestjs.io/docs/sna To use them within GitLab, there are a few guidelines that should be highlighted: - Treat snapshots as code -- Don't think of a snapshot file as a Blackbox +- Don't think of a snapshot file as a black box - Care for the output of the snapshot, otherwise, it's not providing any real value. This will usually involve reading the generated snapshot file as you would read any other piece of code Think of a snapshot test as a simple way to store a raw `String` representation of what you've put into the item being tested. This can be used to evaluate changes in a component, a store, a complex piece of generated output, etc. You can see more in the list below for some recommended `Do's and Don'ts`. @@ -1169,7 +1175,7 @@ Jest provides a great set of docs on [best practices](https://jestjs.io/docs/sna ### How does a snapshot work? -A snapshot is purely a stringified version of what you ask to be tested on the lefthand side of the function call. This means any kind of changes you make to the formatting of the string has an impact on the outcome. This process is done by leveraging serializers for an automatic transform step. For Vue this is already taken care of by leveraging the `vue-jest` package, which offers the proper serializer. +A snapshot is purely a stringified version of what you ask to be tested on the left-hand side of the function call. This means any kind of changes you make to the formatting of the string has an impact on the outcome. This process is done by leveraging serializers for an automatic transform step. For Vue this is already taken care of by leveraging the `vue-jest` package, which offers the proper serializer. Should the outcome of your spec be different from what is in the generated snapshot file, you'll be notified about it by a failing test in your test suite. @@ -1448,13 +1454,10 @@ Before executing any page interaction when navigating or making asynchronous cal #### Elements interaction -There are a lot of different ways to find and interact with elements. For example, you could use the basic `find` method with the `selector` and `text` parameter and then use the `.click` method - -```ruby - find('.gl-tab-nav-item', text: 'Tests').click -``` +There are a lot of different ways to find and interact with elements. +For best practises, refer to the [UI testing](best_practices.md#ui-testing) section. -Alternatively, you could use `click_button` with a string of text that is found within the button, which is a more semantically meaningful way of clicking the element. +To click a button, use `click_button` with the string of text found in the button: ```ruby click_button 'Text inside the button element' @@ -1474,20 +1477,20 @@ You can use `fill_in` to fill input / form elements. The first argument is the s Alternatively, you can use the `find` selector paired with `send_keys` to add keys in a field without removing previous text, or `set` which completely replaces the value of the input element. -All of these are valid selectors and methods. Pick whichever suits your needs and look around as there are many more useful ones! +You can find a more comprehensive list of actions in the [feature tests actions](best_practices.md#actions) documentation. #### Assertions To assert anything in a page, you can always access `page` variable, which is automatically defines and actually means the page document. This means you can expect the `page` to have certain components like selectors or content. Here are a few examples: ```ruby - # Finding an element by ID - expect(page).to have_selector('#js-pipeline-graph') + # Finding a button + expect(page).to have_button('Submit review') ``` ```ruby # Finding by text - expect(page).to have_content('build') + expect(page).to have_text('build') ``` ```ruby @@ -1496,20 +1499,20 @@ To assert anything in a page, you can always access `page` variable, which is au ``` ```ruby - # Finding by CSS selector. This is a last resort. - # For example, when you cannot add attributes on the desired element. - expect(page).to have_css('.js-icon-retry') + # Find by data-testid + # Like CSS selector, this is acceptable when there isn't a specific matcher available. + expect(page).to have_css('[data-testid="pipeline-multi-actions-dropdown"]') ``` ```ruby - # Find by data-testid - # Like CSS selector, this is acceptable when there isn't a specific matcher available. - expect(page).to have_selector('[data-testid="pipeline-multi-actions-dropdown"]') + # Finding by CSS selector. This is a last resort. + # For example, when you cannot add attributes on the desired element. + expect(page).to have_css('.js-icon-retry') ``` ```ruby # You can combine any of these selectors with `not_to` instead - expect(page).not_to have_selector('#js-pipeline-graph') + expect(page).not_to have_button('Submit review') ``` ```ruby @@ -1529,11 +1532,13 @@ You can also create a sub-block to look into, to: - Make sure an element is found within the right boundaries. ```ruby - page.within('#js-pipeline-graph') do + page.within('[data-testid="pipeline-multi-actions-dropdown"]') do ... end ``` +You can find a more comprehensive list of matchers in the [feature tests matchers](best_practices.md#matchers) documentation. + #### Feature flags By default, every feature flag is enabled **regardless of the YAML definition or the flags you've set manually in your GDK**. To test when a feature flag is disabled, you must manually stub the flag, ideally in a `before do` block. diff --git a/doc/development/testing_guide/img/testing_triangle.png b/doc/development/testing_guide/img/testing_triangle.png Binary files differindex 7a9a848c2ee..3ac4955eaff 100644 --- a/doc/development/testing_guide/img/testing_triangle.png +++ b/doc/development/testing_guide/img/testing_triangle.png diff --git a/doc/development/understanding_explain_plans.md b/doc/development/understanding_explain_plans.md deleted file mode 100644 index 72c3df11a96..00000000000 --- a/doc/development/understanding_explain_plans.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'database/understanding_explain_plans.md' -remove_date: '2022-11-04' ---- - -This document was moved to [another location](database/understanding_explain_plans.md). - -<!-- This redirect file can be deleted after <2022-11-04>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/uploads/index.md b/doc/development/uploads/index.md index 41ec71451fb..b5509f5934e 100644 --- a/doc/development/uploads/index.md +++ b/doc/development/uploads/index.md @@ -40,21 +40,9 @@ When using object storage, administrators can control how those files are moved This move can happen in one of these ways: - [Rails controller upload](#rails-controller-upload). -- [Background upload](#background-upload). - [Direct upload](#direct-upload). -These strategies activate as per the following `<feature>.object_store.*` settings: - -| | `background_upload` = `false` | `background_upload` = `true` | -| ------------------------- | ----------------------------- | ------------------------------- | -| `direct_upload` = `false` | Controller upload | Background upload | -| `direct_upload` = `true` | Direct upload | Direct upload (takes precedence)| - Individual Sidekiq workers might also store files in object storage, which is not something we cover here. -More importantly, `background_upload` does not imply _all files are uploaded by Sidekiq._ -Sidekiq workers that store files in object storage could still exist when this setting is `false`. -Those cases are never user-initiated uploads, but they might occur in response to another user-initiated -action, such as exporting a GitLab repository. Finally, Workhorse assists most user-initiated uploads using an upload buffering mechanism to keep slow work out of Rails controllers. This mechanism is explained in [Workhorse assisted uploads](#workhorse-assisted-uploads), @@ -98,12 +86,12 @@ GitLab to the object store provider. As mentioned above, there are three differe this HTTP request is sent. - [Rails controller upload](#rails-controller-upload). -- [Background upload](#background-upload). - [Direct upload](#direct-upload). +- [Workhorse assisted uploads](#workhorse-assisted-uploads). ### Rails controller upload -When neither background upload nor direct upload are available, Rails uploads the file to object storage +When direct upload is not available, Rails uploads the file to object storage as part of the controller `create` action. Which controller is responsible depends on the kind of file uploaded. A Rails controller upload is very similar to uploading to local storage. The main difference: Rails must @@ -115,25 +103,6 @@ keep some of the costly I/O work out of Ruby and Rails. Direct upload does a bet This strategy is only suitable for small file uploads, as it is subject to Puma's 60 second request timeout. -### Background upload - -WARNING: -This strategy is deprecated in GitLab 14.9 and later, and is scheduled to [be removed in GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/26600). - -With background uploads enabled: - -1. Files are uploaded as if they were to reside in local storage. -1. When Rails saves the upload metadata and the transaction completes, a Sidekiq job is scheduled. -1. The Sidekiq job transfers the file to the object store bucket. - - If the job completes, the upload record is updated to reflect the file's new location. - - If the job fails or gets lost, the upload stays in local storage and has the lifecycle of a normal local storage upload. - -As Rails and Sidekiq must cooperate to move the file to its final destination, it requires shared -storage and as such is unsuitable for CNG installations. We do not use background upload in GitLab SaaS. - -As background upload is an extension of local storage, it benefits from the same [Workhorse assistance](#workhorse-assisted-uploads) to -keep costly I/O work out of Ruby and Rails. - ### Direct upload Direct upload is the recommended way to move large files into object storage in CNG installations like GitLab SaaS. diff --git a/doc/development/uploads/working_with_uploads.md b/doc/development/uploads/working_with_uploads.md index c88762e6bd5..a3951fb4c7e 100644 --- a/doc/development/uploads/working_with_uploads.md +++ b/doc/development/uploads/working_with_uploads.md @@ -8,9 +8,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w ## Recommendations -- When creating an uploader, [make it a subclass](#where-should-i-store-my-files) of `AttachmentUploader` +- When creating an uploader, [make it a subclass](#where-should-you-store-your-files) of `AttachmentUploader` - Add your uploader to the [tables](#tables) in this document -- Do not add [new object storage buckets](#where-should-i-store-my-files) +- Do not add [new object storage buckets](#where-should-you-store-your-files) - Implement [direct upload](#implementing-direct-upload-support) - If you need to process your uploads, decide [where to do that](#processing-uploads) @@ -19,14 +19,14 @@ info: To determine the technical writer assigned to the Stage/Group associated w - [CarrierWave Uploaders](#carrierwave-uploaders) - [GitLab modifications to CarrierWave](#gitlab-modifications-to-carrierwave) -## Where should I store my files? +## Where should you store your files? CarrierWave Uploaders determine where files get stored. When you create a new Uploader class you are deciding where to store the files of your new feature. First of all, ask yourself if you need a new Uploader class. It is OK -to use the same Uploader class for different mountpoints or different +to use the same Uploader class for different mount points or different models. If you do want or need your own Uploader class then you should make it @@ -160,8 +160,8 @@ we modified it. The central concept of CarrierWave is the **Uploader** class. The Uploader defines where files get stored, and optionally contains validation and processing logic. To use an Uploader you must associate -it with a text column on an ActiveRecord model. This called "mounting" -and the column is called the "mountpoint". For example: +it with a text column on an ActiveRecord model. This is called "mounting" +and the column is called `mountpoint`. For example: ```ruby class Project < ApplicationRecord @@ -169,8 +169,8 @@ class Project < ApplicationRecord end ``` -Now if I upload an avatar called `tanuki.png` the idea is that in the -`projects.avatar` column for my project, CarrierWave stores the string +Now if you upload an avatar called `tanuki.png` the idea is that in the +`projects.avatar` column for your project, CarrierWave stores the string `tanuki.png`, and that the AttachmentUploader class contains the configuration data and directory schema. For example if the project ID is 123, the actual file may be in @@ -179,12 +179,12 @@ The directory `/var/opt/gitlab/gitlab-rails/uploads/-/system/project/avatar/123/` was chosen by the Uploader using among others configuration (`/var/opt/gitlab/gitlab-rails/uploads`), the model name (`project`), -the model ID (`123`) and the mountpoint (`avatar`). +the model ID (`123`) and the mount point (`avatar`). > The Uploader determines the individual storage directory of your -> upload. The mountpoint column in your model contains the filename. +> upload. The `mountpoint` column in your model contains the filename. -You never access the mountpoint column directly because CarrierWave +You never access the `mountpoint` column directly because CarrierWave defines a getter and setter on your model that operates on file handle objects. @@ -213,7 +213,7 @@ CarrierWave has 2 storage engines: |CarrierWave class|GitLab name|Description| |---|---|---| -|`CarrierWave::Storage::File`|`ObjectStorage::Store::LOCAL` |Local files, accessed through the Ruby stdlib| +|`CarrierWave::Storage::File`|`ObjectStorage::Store::LOCAL` |Local files, accessed through the Ruby `stdlib` | | `CarrierWave::Storage::Fog`|`ObjectStorage::Store::REMOTE`|Cloud files, accessed through the [Fog gem](https://github.com/fog/fog)| GitLab uses both of these engines, depending on configuration. @@ -227,8 +227,8 @@ storage engine file by file. An Uploader is associated with two storage areas: regular storage and cache storage. Each has its own storage engine. If you assign a file -to a mountpoint setter (`project.avatar = -File.open('/tmp/tanuki.png')`) you will copy/move the file to cache +to a mount point setter (`project.avatar = File.open('/tmp/tanuki.png')`) +you will copy/move the file to cache storage as a side effect via the `cache!` method. To persist the file you must somehow call the `store!` method. This either happens via [ActiveRecord callbacks](https://github.com/carrierwaveuploader/carrierwave/blob/v1.3.2/lib/carrierwave/orm/activerecord.rb#L55) diff --git a/doc/development/utilities.md b/doc/development/utilities.md index 551834670b3..58954101890 100644 --- a/doc/development/utilities.md +++ b/doc/development/utilities.md @@ -181,20 +181,6 @@ Refer to [`strong_memoize.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/maste include Gitlab::Utils::StrongMemoize def result - strong_memoize(:result) do - search - end - end - end - ``` - - Alternatively, use the `strong_memoize_attr` helper to memoize the method for you: - - ```ruby - class Find - include Gitlab::Utils::StrongMemoize - - def result search end strong_memoize_attr :result @@ -206,6 +192,26 @@ Refer to [`strong_memoize.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/maste end ``` + Using `strong_memoize_attr` on methods with parameters is not supported. + It does not work when combined with [`override`](#override) and might memoize wrong results. + + Use `strong_memoize_with` instead. + + ```ruby + # bad + def expensive_method(arg) + # ... + end + strong_memoize_attr :expensive_method + + # good + def expensive_method(arg) + strong_memoize_with(:expensive_method, arg) + # ... + end + end + ``` + There's also `strong_memoize_with` to help memoize methods that take arguments. This should be used for methods that have a low number of possible values as arguments or with consistent repeating arguments in a loop. diff --git a/doc/development/value_stream_analytics.md b/doc/development/value_stream_analytics.md index 0d321133705..2d5f33b5dae 100644 --- a/doc/development/value_stream_analytics.md +++ b/doc/development/value_stream_analytics.md @@ -71,7 +71,7 @@ which are always available to the end-users regardless of the subscription. ### Value streams Value streams are container objects for the stages. There can be multiple value streams per group -focusing on different aspects of the Dev Ops lifecycle. +focusing on different aspects of the DevOps lifecycle. ### Events @@ -89,7 +89,8 @@ They're responsible for defining a timestamp expression that is used in the calc #### Implementing an `Event` class -There are a few methods that are required to be implemented, the `StageEvent` base class describes them in great detail. The most important ones are: +You must implement a few methods, as described in the `StageEvent` base class. +The most important methods are: - `object_type` - `timestamp_projection` diff --git a/doc/development/value_stream_analytics/value_stream_analytics_aggregated_backend.md b/doc/development/value_stream_analytics/value_stream_analytics_aggregated_backend.md index a07998550bf..5bcadc6f39b 100644 --- a/doc/development/value_stream_analytics/value_stream_analytics_aggregated_backend.md +++ b/doc/development/value_stream_analytics/value_stream_analytics_aggregated_backend.md @@ -54,9 +54,9 @@ database with a minimal development effort. ### Example configuration - + -In this example, there are two independent value streams set up for two teams that are using +In this example, two independent value streams are set up for two teams that are using different development workflows within the `Test Group` (top-level namespace). The first value stream uses standard timestamp-based events for defining the stages. The second @@ -102,7 +102,7 @@ High-level overview for each top-level namespace with Premium or Ultimate licens 1. `INSERT` or `UPDATE` the data into the VSA database tables. The data loading is implemented within the [`Analytics::CycleAnalytics::DataLoaderService`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/services/analytics/cycle_analytics/data_loader_service.rb) -class. There are groups containing a lot of data, so to avoid overloading the primary database, +class. Some groups contain a lot of data, so to avoid overloading the primary database, the service performs operations in batches and enforces strict application limits: - Load records in batches. diff --git a/doc/development/verifying_database_capabilities.md b/doc/development/verifying_database_capabilities.md deleted file mode 100644 index 0217eb96e5a..00000000000 --- a/doc/development/verifying_database_capabilities.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'database/verifying_database_capabilities.md' -remove_date: '2022-11-06' ---- - -This document was moved to [another location](database/verifying_database_capabilities.md). - -<!-- This redirect file can be deleted after <2022-11-06>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/work_items.md b/doc/development/work_items.md index a417e1d1349..90e454bec85 100644 --- a/doc/development/work_items.md +++ b/doc/development/work_items.md @@ -54,7 +54,7 @@ To avoid confusion and ensure communication is efficient, we will use the follow | work item view | The new frontend view that renders work items of any type | | | | legacy issue view | The existing view used to render issues and incidents | | | | issue | The existing issue model | | | -| issuable | Any model currently using the issueable module (issues, epics and MRs) | _Incidents are an **issuable**_ | _Incidents are a **work item type**_ | +| issuable | Any model currently using the issuable module (issues, epics and MRs) | _Incidents are an **issuable**_ | _Incidents are a **work item type**_ | | widget | A UI element to present or allow interaction with specific work item data | | | Some terms have been used in the past but have since become confusing and are now discouraged. @@ -92,7 +92,7 @@ NOTE: At first, defining a WIT will only be possible at the root-level group, which would then be inherited by subgroups. We will investigate the possibility of defining new WITs at subgroup levels at a later iteration. -### Introducing work_item_types table +### Introducing `work_item_types` table For example, suppose there are three root-level groups with IDs: `11`, `12`, and `13`. Also, assume the following base types: `issue: 0`, `incident: 1`, `test_case: 2`. @@ -228,7 +228,7 @@ So, migrating epics to a work item type requires providing feature parity betwee The main missing features are: -- Get WIs to the group level. This is dependent on [Consolidate Groups and Projects](https://gitlab.com/gitlab-org/architecture/tasks/-/issues/7) +- Get work items to the group level. This is dependent on [Consolidate Groups and Projects](https://gitlab.com/gitlab-org/architecture/tasks/-/issues/7) initiative. - A hierarchy widget: the ability to structure work items into hierarchies. - Inherited date widget. diff --git a/doc/development/work_items_widgets.md b/doc/development/work_items_widgets.md index 89602d969e6..ba15a3f0163 100644 --- a/doc/development/work_items_widgets.md +++ b/doc/development/work_items_widgets.md @@ -80,7 +80,7 @@ mutation { ``` -### Widget's responsibility and structure +### Widget responsibility and structure A widget is responsible for displaying and updating a single attribute, such as title, description, or labels. Widgets must support any type of work item. diff --git a/doc/development/workhorse/configuration.md b/doc/development/workhorse/configuration.md index 82e44a6f995..9fc106b8f04 100644 --- a/doc/development/workhorse/configuration.md +++ b/doc/development/workhorse/configuration.md @@ -162,7 +162,7 @@ addr = "localhost:9229" max_version = "tls1.3" ``` -## Interaction of authBackend and authSocket +## Interaction of `authBackend` and `authSocket` The interaction between `authBackend` and `authSocket` can be confusing. If `authSocket` is set, it overrides the host portion of `authBackend`, but not @@ -170,7 +170,7 @@ the relative path. In table form: -| authBackend | authSocket | Workhorse connects to | Rails relative URL | +| `authBackend` | `authSocket` | Workhorse connects to | Rails relative URL | |--------------------------------|-------------------|-----------------------|--------------------| | unset | unset | `localhost:8080` | `/` | | `http://localhost:3000` | unset | `localhost:3000` | `/` | |
