diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2020-11-19 08:27:35 +0000 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2020-11-19 08:27:35 +0000 |
| commit | 7e9c479f7de77702622631cff2628a9c8dcbc627 (patch) | |
| tree | c8f718a08e110ad7e1894510980d2155a6549197 /doc/development | |
| parent | e852b0ae16db4052c1c567d9efa4facc81146e88 (diff) | |
| download | gitlab-ce-7e9c479f7de77702622631cff2628a9c8dcbc627.tar.gz | |
Add latest changes from gitlab-org/gitlab@13-6-stable-eev13.6.0-rc42
Diffstat (limited to 'doc/development')
221 files changed, 6483 insertions, 3283 deletions
diff --git a/doc/development/README.md b/doc/development/README.md index 7b8a5cd5f75..75ea6ae5f3b 100644 --- a/doc/development/README.md +++ b/doc/development/README.md @@ -1,6 +1,10 @@ --- comments: false -description: 'Learn how to contribute to GitLab.' +type: index, dev +stage: none +group: Development +info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +description: "Development Guidelines: learn how to contribute to GitLab." --- # Contributor and Development Docs @@ -60,6 +64,19 @@ Complementary reads: - [Patch release process for developers](https://gitlab.com/gitlab-org/release/docs/blob/master/general/patch/process.md#process-for-developers) - [Adding a new service component to GitLab](adding_service_component.md) +### Development guidelines review + +When you submit a change to GitLab's development guidelines, request a review +from: + +- A member of your team or group, to check for technical accuracy. +- For **significant** changes or proposals, request review from: + - Engineering managers (FE, BE, DB, Security, UX, and others), according to the subject or process you're proposing. + - The VP of Development (DRI) ([@clefelhocz1](https://gitlab.com/clefelhocz1)), for + final approval of the new or changed guidelines. +- The [Technical Writer assigned to dev guidelines](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines), + to review the content for consistency and adherence to documentation guidelines. + ## UX and Frontend guides - [GitLab Design System](https://design.gitlab.com/) for building GitLab with existing CSS styles and elements @@ -93,7 +110,6 @@ Complementary reads: - [Working with Merge Request diffs](diffs.md) - [Kubernetes integration guidelines](kubernetes.md) - [Permissions](permissions.md) -- [Prometheus](prometheus.md) - [Guidelines for reusing abstractions](reusing_abstractions.md) - [DeclarativePolicy framework](policies.md) - [How Git object deduplication works in GitLab](git_object_deduplication.md) @@ -117,17 +133,21 @@ Complementary reads: - [Approval Rules](approval_rules.md) - [Feature categorization](feature_categorization/index.md) - [Wikis development guide](wikis.md) +- [Newlines style guide](newlines_styleguide.md) +- [Image scaling guide](image_scaling.md) ## Performance guides - [Instrumentation](instrumentation.md) for Ruby code running in production - environments + environments. - [Performance guidelines](performance.md) for writing code, benchmarks, and - certain patterns to avoid + certain patterns to avoid. - [Merge request performance guidelines](merge_request_performance_guidelines.md) for ensuring merge requests do not negatively impact GitLab performance - [Profiling](profiling.md) a URL, measuring performance using Sherlock, or - tracking down N+1 queries using Bullet + tracking down N+1 queries using Bullet. +- [Cached queries guidelines](cached_queries.md), for tracking down N+1 queries masked by query caching, memory profiling and why should + we avoid cached queries. ## Database guides @@ -156,7 +176,7 @@ See [database guidelines](database/index.md). ## Documentation guides - [Writing documentation](documentation/index.md) -- [Documentation style guide](documentation/styleguide.md) +- [Documentation style guide](documentation/styleguide/index.md) - [Markdown](../user/markdown.md) ## Internationalization (i18n) guides @@ -167,7 +187,7 @@ See [database guidelines](database/index.md). ## Product Analytics guides -- [Product Analytics guide](product_analytics/index.md) +- [Product Analytics guide](https://about.gitlab.com/handbook/product/product-analytics-guide/) - [Usage Ping guide](product_analytics/usage_ping.md) - [Snowplow guide](product_analytics/snowplow.md) @@ -198,9 +218,9 @@ See [database guidelines](database/index.md). ## Other Development guides - [Defining relations between files using projections](projections.md) -- [Reference processing](./reference_processing.md) +- [Reference processing](reference_processing.md) - [Compatibility with multiple versions of the application running at the same time](multi_version_compatibility.md) -- [Features inside `.gitlab/`](./features_inside_dot_gitlab.md) +- [Features inside `.gitlab/`](features_inside_dot_gitlab.md) ## Other GitLab Development Kit (GDK) guides diff --git a/doc/development/adding_service_component.md b/doc/development/adding_service_component.md index bb5af286c1d..74309a5c616 100644 --- a/doc/development/adding_service_component.md +++ b/doc/development/adding_service_component.md @@ -1,10 +1,16 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Adding a new Service Component to GitLab The GitLab product is made up of several service components that run as independent system processes in communication with each other. These services can be run on the same instance, or spread across different instances. A list of the existing components can be found in the [GitLab architecture overview](architecture.md). ## Integration phases -The following outline re-uses the [maturity metric](https://about.gitlab.com/direction/maturity) naming as an example of the various phases of integrating a component. These are only loosely coupled to a components actual maturity, and are intended as a guide for implementation order (for example, a component does not need to be enabled by default to be Lovable, and being enabled by default does not on its own cause a component to be Lovable). +The following outline re-uses the [maturity metric](https://about.gitlab.com/direction/maturity/) naming as an example of the various phases of integrating a component. These are only loosely coupled to a components actual maturity, and are intended as a guide for implementation order (for example, a component does not need to be enabled by default to be Lovable, and being enabled by default does not on its own cause a component to be Lovable). - Proposed - [Proposing a new component](#proposing-a-new-component) @@ -53,10 +59,8 @@ TIP: **Tip:** ## Bundling a service with GitLab -NOTE: **Note:** Code shipped with GitLab needs to use a license approved by the Legal team. See the list of [existing approved licenses](https://about.gitlab.com/handbook/engineering/open-source/#using-open-source-libraries). -NOTE: **Note:** Notify the [Distribution team](https://about.gitlab.com/handbook/engineering/development/enablement/distribution/) when adding a new dependency that must be compiled. We must be able to compile the dependency on all supported platforms. New services to be bundled with GitLab need to be available in the following environments. diff --git a/doc/development/api_graphql_styleguide.md b/doc/development/api_graphql_styleguide.md index 3d4c033e676..0e81a332d6c 100644 --- a/doc/development/api_graphql_styleguide.md +++ b/doc/development/api_graphql_styleguide.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # GraphQL API style guide This document outlines the style guide for GitLab's [GraphQL API](../api/graphql/index.md). @@ -126,8 +132,23 @@ Non-nullable fields should only be used when a field is required, very unlikely to become optional in the future, and very easy to calculate. An example would be `id` fields. +A non-nullable GraphQL schema field is an object type followed by the exclamation point (bang) `!`. Here's an example from the `gitlab_schema.graphql` file: + +```graphql + id: ProjectID! +``` + +Here's an example of a non-nullable GraphQL array: + +```graphql + + errors: [String!]! +``` + Further reading: +- [GraphQL Best Practices Guide](https://graphql.org/learn/best-practices/#nullability). +- GraphQL documentation on [Object types and fields](https://graphql.org/learn/schema/#object-types-and-fields). - [GraphQL Best Practices Guide](https://graphql.org/learn/best-practices/#nullability) - [Using nullability in GraphQL](https://www.apollographql.com/blog/using-nullability-in-graphql-2254f84c4ed7) @@ -376,7 +397,7 @@ field :foo, GraphQL::STRING_TYPE, 'if `my_feature_flag` feature flag is disabled' def foo - object.foo unless Feature.enabled?(:my_feature_flag, object) + object.foo if Feature.enabled?(:my_feature_flag, object) end ``` @@ -683,7 +704,7 @@ end ``` Fields can also be authorized against multiple abilities, in which case -all of ability checks must pass. **Note:** This requires explicitly +all of ability checks must pass. This requires explicitly passing a block to `field`: ```ruby @@ -696,7 +717,6 @@ module Types end ``` -NOTE: **Note:** If the field's type already [has a particular authorization](#type-authorization) then there is no need to add that same authorization to the field. @@ -736,7 +756,142 @@ To find objects to display in a field, we can add resolvers to Arguments can be defined within the resolver in the same way as in a mutation. See the [Mutation arguments](#object-identifier-arguments) section. -To limit the amount of queries performed, we can use `BatchLoader`. +To limit the amount of queries performed, we can use [BatchLoader](graphql_guide/batchloader.md). + +### Writing resolvers + +Our code should aim to be thin declarative wrappers around finders and services. You can +repeat lists of arguments, or extract them to concerns. Composition is preferred over +inheritance in most cases. Treat resolvers like controllers: resolvers should be a DSL +that compose other application abstractions. + +For example: + +```ruby +class PostResolver < BaseResolver + type Post.connection_type, null: true + authorize :read_blog + description 'Blog posts, optionally filtered by name' + + argument :name, [::GraphQL::STRING_TYPE], required: false, as: :slug + + alias_method :blog, :object + + def resolve(**args) + PostFinder.new(blog, current_user, args).execute + end +end +``` + +You should never re-use resolvers directly. Resolvers have a complex life-cycle, with +authorization, readiness and resolution orchestrated by the framework, and at +each stage lazy values can be returned to take advantage of batching +opportunities. Never instantiate a resolver or a mutation in application code. + +Instead, the units of code reuse are much the same as in the rest of the +application: + +- Finders in queries to look up data. +- Services in mutations to apply operations. +- Loaders (batch-aware finders) specific to queries. + +Note that there is never any reason to use batching in a mutation. Mutations are +executed in series, so there are no batching opportunities. All values are +evaluated eagerly as soon as they are requested, so batching is unnecessary +overhead. If you are writing: + +- A `Mutation`, feel free to lookup objects directly. +- A `Resolver` or methods on a `BaseObject`, then you want to allow for batching. + +### Deriving resolvers (`BaseResolver.single` and `BaseResolver.last`) + +For some simple use cases, we can derive resolvers from others. +The main use case for this is one resolver to find all items, and another to +find one specific one. For this, we supply convenience methods: + +- `BaseResolver.single`, which constructs a new resolver that selects the first item. +- `BaseResolver.last`, with constructs a resolver that selects the last item. + +The correct singular type is inferred from the collection type, so we don't have +to define the `type` here. + +Before you make use of these methods, consider if it would be simpler to either: + +- Write another resolver that defines its own arguments. +- Write a concern that abstracts out the query. + +Using `BaseResolver.single` too freely is an anti-pattern. It can lead to +non-sensical fields, such as a `Project.mergeRequest` field that just returns +the first MR if no arguments are given. Whenever we derive a single resolver +from a collection resolver, it must have more restrictive arguments. + +To make this possible, use the `when_single` block to customize the single +resolver. Every `when_single` block must: + +- Define (or re-define) at least one argument. +- Make optional filters required. + +For example, we can do this by redefining an existing optional argument, +changing its type and making it required: + +```ruby +class JobsResolver < BaseResolver + type JobType.connection_type, null: true + authorize :read_pipeline + + argument :name, [::GraphQL::STRING_TYPE], required: false + + when_single do + argument :name, ::GraphQL::STRING_TYPE, required: true + end + + def resolve(**args) + JobsFinder.new(pipeline, current_user, args.compact).execute + end +``` + +Here we have a simple resolver for getting pipeline jobs. The `name` argument is +optional when getting a list, but required when getting a single job. + +If there are multiple arguments, and neither can be made required, we can use +the block to add a ready condition: + +```ruby +class JobsResolver < BaseResolver + alias_method :pipeline, :object + + type JobType.connection_type, null: true + authorize :read_pipeline + + argument :name, [::GraphQL::STRING_TYPE], required: false + argument :id, [::Types::GlobalIDType[::Job]], + required: false, + prepare: ->(ids, ctx) { ids.map(&:model_id) } + + when_single do + argument :name, ::GraphQL::STRING_TYPE, required: false + argument :id, ::Types::GlobalIDType[::Job], + required: false + prepare: ->(id, ctx) { id.model_id } + + def ready?(**args) + raise ::Gitlab::Graphql::Errors::ArgumentError, 'Only one argument may be provided' unless args.size == 1 + end + end + + def resolve(**args) + JobsFinder.new(pipeline, current_user, args.compact).execute + end +``` + +Then we can use these resolver on fields: + +```ruby +# In PipelineType + +field :jobs, resolver: JobsResolver, description: 'All jobs' +field :job, resolver: JobsResolver.single, description: 'A single job' +``` ### Correct use of `Resolver#ready?` @@ -835,7 +990,29 @@ To avoid duplicated argument definitions, you can place these arguments in a reu class, if the arguments are nested). Alternatively, you can consider to add a [helper resolver method](https://gitlab.com/gitlab-org/gitlab/-/issues/258969). -## Pass a parent object into a child Presenter +### Metadata + +When using resolvers, they can and should serve as the SSoT for field metadata. +All field options (apart from the field name) can be declared on the resolver. +These include: + +- `type` (this is particularly important, and will soon be mandatory) +- `extras` +- `description` + +Example: + +```ruby +module Resolvers + MyResolver < BaseResolver + type Types::MyType, null: true + extras [:lookahead] + description 'Retrieve a single MyType' + end +end +``` + +### Pass a parent object into a child Presenter Sometimes you need to access the resolved query parent in a child context to compute fields. Usually the parent is only available in the `Resolver` class as `parent`. @@ -850,7 +1027,7 @@ To find the parent object in your `Presenter` class: end ``` -1. Declare that your fields require the `parent` field context. For example: +1. Declare that your resolver or fields require the `parent` field context. For example: ```ruby # in ChildType @@ -858,6 +1035,14 @@ To find the parent object in your `Presenter` class: method: :my_computing_method, extras: [:parent], # Necessary description: 'My field description' + + field :resolver_field, resolver: SomeTypeResolver + + # In SomeTypeResolver + + extras [:parent] + type SomeType, null: true + description 'My field description' ``` 1. Declare your field's method in your Presenter class and have it accept the `parent` keyword argument. @@ -869,6 +1054,12 @@ This argument contains the parent **GraphQL context**, so you have to access the def my_computing_method(parent:) # do something with `parent[:parent_object]` here end + + # In SomeTypeResolver + + def resolve(parent:) + # ... + end ``` For an example of real-world use, check [this MR that added `scopedPath` and `scopedUrl` to `IterationPresenter`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/39543) @@ -1273,6 +1464,11 @@ tested for within the unit test of `Types::MutationType`. The merge request can be referred to as an example of this, including the method of testing deprecated aliased mutations. +#### Deprecating EE mutations + +EE mutations should follow the same process. For an example of the merge request +process, read [merge request !42588](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/42588). + ## Pagination implementation To learn more, visit [GraphQL pagination](graphql_guide/pagination.md). @@ -1381,6 +1577,62 @@ it 'returns a successful response' do end ``` +### Testing tips and tricks + +- Avoid false positives: + + Authenticating a user with the `current_user:` argument for `post_graphql` + generates more queries on the first request than on subsequent requests on that + same user. If you are testing for N+1 queries using + [QueryRecorder](query_recorder.md), use a **different** user for each request. + + The below example shows how a test for avoiding N+1 queries should look: + + ```ruby + RSpec.describe 'Query.project(fullPath).pipelines' do + include GraphqlHelpers + + let(:project) { create(:project) } + + let(:query) do + %( + { + project(fullPath: "#{project.full_path}") { + pipelines { + nodes { + id + } + } + } + } + ) + end + + it 'avoids N+1 queries' do + first_user = create(:user) + second_user = create(:user) + create(:ci_pipeline, project: project) + + control_count = ActiveRecord::QueryRecorder.new do + post_graphql(query, current_user: first_user) + end + + create(:ci_pipeline, project: project) + + expect do + post_graphql(query, current_user: second_user) # use a different user to avoid a false positive from authentication queries + end.not_to exceed_query_limit(control_count) + end + end + ``` + +- Mimic the folder structure of `app/graphql/types`: + + For example, tests for fields on `Types::Ci::PipelineType` + in `app/graphql/types/ci/pipeline_type.rb` should live in + `spec/requests/api/graphql/ci/pipeline_spec.rb` regardless of the query being + used to fetch the pipeline data. + ## Notes about Query flow and GraphQL infrastructure GitLab's GraphQL infrastructure can be found in `lib/gitlab/graphql`. @@ -1446,3 +1698,7 @@ For information on generating GraphQL documentation and schema files, see To help our readers, you should also add a new page to our [GraphQL API](../api/graphql/index.md) documentation. For guidance, see the [GraphQL API](documentation/graphql_styleguide.md) page. + +## Include a changelog entry + +All client-facing changes **must** include a [changelog entry](changelog.md). diff --git a/doc/development/api_styleguide.md b/doc/development/api_styleguide.md index 47c06377d88..d21975a43d2 100644 --- a/doc/development/api_styleguide.md +++ b/doc/development/api_styleguide.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # API style guide This style guide recommends best practices for API development. @@ -201,6 +207,12 @@ guide on how you can add a new custom validator. checks if the value of the given parameter is either an `Array`, `None`, or `Any`. It allows only either of these mentioned values to move forward in the request. +- `EmailOrEmailList`: + + The [`EmailOrEmailList` validator](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/validations/validators/email_or_email_list.rb) + checks if the value of a string or a list of strings contains only valid + email addresses. It allows only lists with all valid email addresses to move forward in the request. + ### Adding a new custom validator Custom validators are a great way to validate parameters before sending @@ -228,7 +240,7 @@ it's own file in the [`validators`](https://gitlab.com/gitlab-org/gitlab/-/blob/ ## Internal API -The [internal API](./internal_api.md) is documented for internal use. Please keep it up to date so we know what endpoints +The [internal API](internal_api.md) is documented for internal use. Please keep it up to date so we know what endpoints different components are making use of. ## Avoiding N+1 problems @@ -285,10 +297,15 @@ end ## Testing -When writing tests for new API endpoints, consider using a schema [fixture](./testing_guide/best_practices.md#fixtures) located in `/spec/fixtures/api/schemas`. You can `expect` a response to match a given schema: +When writing tests for new API endpoints, consider using a schema [fixture](testing_guide/best_practices.md#fixtures) located in `/spec/fixtures/api/schemas`. You can `expect` a response to match a given schema: ```ruby expect(response).to match_response_schema('merge_requests') ``` Also see [verifying N+1 performance](#verifying-with-tests) in tests. + +## Include a changelog entry + +All client-facing changes **must** include a [changelog entry](changelog.md). +This does not include internal APIs. diff --git a/doc/development/application_limits.md b/doc/development/application_limits.md index f96ed2e7f57..41fcf5301ad 100644 --- a/doc/development/application_limits.md +++ b/doc/development/application_limits.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Application limits development This document provides a development guide for contributors to add application @@ -27,7 +33,6 @@ It's recommended to create two separate migration script files. add_column(:plan_limits, :project_hooks, :integer, default: 100, null: false) ``` - NOTE: **Note:** Plan limits entries set to `0` mean that limits are not enabled. You should use this setting only in special and documented circumstances. @@ -58,7 +63,6 @@ It's recommended to create two separate migration script files. end ``` - NOTE: **Note:** Some plans exist only on GitLab.com. This will be a no-op for plans that do not exist. @@ -97,7 +101,6 @@ can be used to validate that a model does not exceed the limits. It ensures that the count of the records for the current model does not exceed the defined limit. -NOTE: **Note:** You must specify the limit scope of the object being validated and the limit name if it's different from the pluralized model name. @@ -146,5 +149,4 @@ GitLab.com: - `silver` - Namespaces and projects with a Silver subscription - `gold` - Namespaces and projects with a Gold subscription -NOTE: **Note:** -The test environment doesn't have any plans. +The `test` environment doesn't have any plans. diff --git a/doc/development/application_secrets.md b/doc/development/application_secrets.md index 24755586cf8..abc5ff7b985 100644 --- a/doc/development/application_secrets.md +++ b/doc/development/application_secrets.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Application secrets This page is a development guide for application secrets. diff --git a/doc/development/approval_rules.md b/doc/development/approval_rules.md index f295c20a36f..d190f2b7c63 100644 --- a/doc/development/approval_rules.md +++ b/doc/development/approval_rules.md @@ -1,3 +1,9 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Approval Rules **(STARTER)** This document explains the backend design and flow of all related functionality @@ -14,7 +20,7 @@ feature to work. NOTE: **Note:** This is a living document and should be updated accordingly when parts -of the codebase touched in this document changed/removed or when new components +of the codebase touched in this document are changed or removed, or when new components are added. ## Data Model @@ -141,7 +147,7 @@ Whenever an approval is given/revoked, a record is created/deleted. ## Controllers and Services -The following controllers and services below are being utilized for the approval +The following controllers and services below are being used for the approval rules feature to work. ### `API::ProjectApprovalSettings` diff --git a/doc/development/architecture.md b/doc/development/architecture.md index 513af491576..a1097ad4ed6 100644 --- a/doc/development/architecture.md +++ b/doc/development/architecture.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # GitLab architecture overview ## Software delivery @@ -252,6 +258,7 @@ Table description links: | [NGINX](#nginx) | Routes requests to appropriate components, terminates SSL | ✅ | ✅ | ⚙ | ✅ | ⤓ | ❌ | CE & EE | | [Node Exporter](#node-exporter) | Prometheus endpoint with system metrics | ✅ | N/A | N/A | ✅ | ❌ | ❌ | CE & EE | | [Outbound email (SMTP)](#outbound-email) | Send email messages to users | ⤓ | ⚙ | ⤓ | ✅ | ⤓ | ⤓ | CE & EE | +| [Patroni](#patroni) | Manage PostgreSQL HA cluster leader selection and replication | ⚙ | ❌ | ❌ | ✅ | ❌ | ❌ | EE Only | | [PgBouncer Exporter](#pgbouncer-exporter) | Prometheus endpoint with PgBouncer metrics | ⚙ | ❌ | ❌ | ✅ | ❌ | ❌ | CE & EE | | [PgBouncer](#pgbouncer) | Database connection pooling, failover | ⚙ | ❌ | ❌ | ✅ | ❌ | ❌ | EE Only | | [PostgreSQL Exporter](#postgresql-exporter) | Prometheus endpoint with PostgreSQL metrics | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | CE & EE | @@ -539,6 +546,15 @@ NGINX has an Ingress port for all HTTP requests and routes them to the appropria [Node Exporter](https://github.com/prometheus/node_exporter) is a Prometheus tool that gives us metrics on the underlying machine (think CPU/Disk/Load). It's just a packaged version of the common open source offering from the Prometheus project. +#### Patroni + +- [Project Page](https://github.com/zalando/patroni) +- Configuration: + - [Omnibus](../administration/postgresql/replication_and_failover.md#patroni) +- Layer: Core Service (Data) +- Process: `patroni` +- GitLab.com: [Database Architecture](https://about.gitlab.com/handbook/engineering/infrastructure/production/architecture/#database-architecture) + #### PgBouncer - [Project page](https://github.com/pgbouncer/pgbouncer/blob/master/README.md) @@ -681,7 +697,6 @@ Sidekiq is a Ruby background job processor that pulls jobs from the Redis queue #### Puma -NOTE: **Note:** Starting with GitLab 13.0, Puma is the default web server and Unicorn has been disabled by default. @@ -699,7 +714,6 @@ disabled by default. #### Unicorn -NOTE: **Note:** Starting with GitLab 13.0, Puma is the default web server and Unicorn has been disabled by default. @@ -931,7 +945,7 @@ processes: `puma master` (1 process), `puma cluster worker` ### Repository access -Repositories get accessed via HTTP or SSH. HTTP cloning/push/pull utilizes the GitLab API and SSH cloning is handled by GitLab Shell (previously explained). +Repositories get accessed via HTTP or SSH. HTTP cloning/push/pull uses the GitLab API and SSH cloning is handled by GitLab Shell (previously explained). ## Troubleshooting @@ -1015,9 +1029,9 @@ PostgreSQL: GitLab has configuration files located in `/home/git/gitlab/config/*`. Commonly referenced configuration files include: -- `gitlab.yml` - GitLab configuration -- `puma.rb` - Puma web server settings -- `database.yml` - Database connection settings +- `gitlab.yml`: GitLab configuration +- `puma.rb`: Puma web server settings +- `database.yml`: Database connection settings GitLab Shell has a configuration file at `/home/git/gitlab-shell/config.yml`. @@ -1033,9 +1047,12 @@ bundle exec rake gitlab:env:info RAILS_ENV=production bundle exec rake gitlab:check RAILS_ENV=production ``` -Note: It is recommended to log into the `git` user using `sudo -i -u git` or `sudo su - git`. While -the `sudo` commands provided by GitLab work in Ubuntu they do not always work in RHEL. +It's recommended to sign in to the `git` user using either `sudo -i -u git` or +`sudo su - git`. Although the `sudo` commands provided by GitLab work in Ubuntu, +they don't always work in RHEL. ## GitLab.com -We've also detailed [our architecture of GitLab.com](https://about.gitlab.com/handbook/engineering/infrastructure/production/architecture/) but this is probably over the top unless you have millions of users. +The [GitLab.com architecture](https://about.gitlab.com/handbook/engineering/infrastructure/production/architecture/) +is detailed for your reference, but this architecture is only useful if you have +millions of users. diff --git a/doc/development/background_migrations.md b/doc/development/background_migrations.md index e5cc2ae4d1d..86a2aed5ea1 100644 --- a/doc/development/background_migrations.md +++ b/doc/development/background_migrations.md @@ -343,10 +343,32 @@ for more details. 1. Make sure that tests you write are not false positives. 1. Make sure that if the data being migrated is critical and cannot be lost, the clean-up migration also checks the final state of the data before completing. -1. Make sure to know how much time it'll take to run all scheduled migrations. 1. When migrating many columns, make sure it won't generate too many dead tuples in the process (you may need to directly query the number of dead tuples and adjust the scheduling according to this piece of data). 1. Make sure to discuss the numbers with a database specialist, the migration may add more pressure on DB than you expect (measure on staging, or ask someone to measure on production). +1. Make sure to know how much time it'll take to run all scheduled migrations. +1. Provide an estimation section in the description, explaining timings from the + linked query plans and batches as described in the migration. + + For example, assuming a migration that deletes data, include information similar to + the following section: + + ```ruby + Background Migration Details: + + 47600 items to delete + batch size = 1000 + 47600 / 1000 = 48 loops + + Estimated times per batch: + - 900ms for select statement with 1000 items + - 2100ms for delete statement with 1000 items + Total: ~3sec per batch + + 2 mins delay per loop (safe for the given total time per batch) + + 48 * ( 120 + 3) = ~98.4 mins to run all the scheduled jobs + ``` diff --git a/doc/development/cached_queries.md b/doc/development/cached_queries.md new file mode 100644 index 00000000000..812e5f88754 --- /dev/null +++ b/doc/development/cached_queries.md @@ -0,0 +1,139 @@ +# Cached queries guidelines + +Rails provides an [SQL query cache](https://guides.rubyonrails.org/caching_with_rails.html#sql-caching), +used to cache the results of database queries for the duration of the request. + +If Rails encounters the same query again for that request, +it will use the cached result set as opposed to running the query against the database again. + +The query results are only cached for the duration of that single request, it does not persist across multiple requests. + +## Why cached queries are considered bad + +The cached queries help with reducing DB load, but they still: + +- Consume memory. +- Require as to re-instantiate each `ActiveRecord` object. +- Require as to re-instantiate each relation of the object. +- Make us spend additional CPU-cycles to look into a list of cached queries. + +The Cached SQL queries are cheaper, but they are not cheap at all from `memory` perspective. +They could mask [N+1 query problem](https://guides.rubyonrails.org/active_record_querying.html#eager-loading-associations), +so we should threat them the same way we threat regular N+1 queries. + +In case of N+1 queries, masked with cached queries, we are executing the same query N times. +It will not hit the database N times, it will return the cached results instead. +This is still expensive since we need to re-initialize objects each time, and this is CPU/Memory expensive. +Instead, we should use the same in-memory objects, if possible. + +When we introduce a new feature, we should avoid N+1 problems, +minimize the [query count](merge_request_performance_guidelines.md#query-counts), and pay special attention that [cached +queries](merge_request_performance_guidelines.md#cached-queries) are not masking N+1 problems. + +## How to detect + +### Detect potential offenders by using Kibana + +On GitLab.com, we are logging entries with the number of executed cached queries in the +`pubsub-redis-inf-gprd*` index with the [`db_cached_count`](https://log.gprd.gitlab.net/goto/77d18d80ad84c5df1bf1da5c2cd35b82). +We can filter endpoints that have a large number of executed cached queries. For example, if we encounter an endpoint +that has 100+ `db_cached_count`, this could indicate that there is an N+1 problem masked with cached queries. +We should probably investigate this endpoint further, to check if we are executing duplicated cached queries. + +For more cached queries Kibana visualizations see [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/259007). + +### Inspect suspicious endpoint using Performance Bar + +When building features, you could use the [performance bar](../administration/monitoring/performance/performance_bar.md) +to list database queries, which will include cached queries as well. The performance bar will show a warning +when threshold of total executed queries (including cached ones) has exceeded 100 queries. + +## What to look for + +Using [Kibana](cached_queries.md#detect-potential-offenders-by-using-kibana), you can look for a large number +of executed cached queries. End-points with large number of `db_cached_count` could indicate that there +are probably a lot of duplicated cached queries, which often indicates a masked N+1 problem. + +When you investigate specific endpoint, you could use +the [performance bar](cached_queries.md#inspect-suspicious-endpoint-using-performance-bar). +If you see a lot of similar queries, this often indicates an N+1 query issue (or a similar kind of query batching problem). +If you see same cached query executed multiple times, this often indicates a masked N+1 query problem. + +For example, let's say you wanted to debug `GroupMembers` page. + +In the left corner of the performance bar you could see **Database queries** showing the total number of database queries +and the number of executed cached queries: + + + +We can see that there are 55 cached queries. By clicking on the number, a modal window with more details is shown. +Cached queries are marked with the `cached` label, so they are easy to spot. We can see that there are multiple duplicated +cached queries: + + + +If we click on `...` for one of them, it will expand the actual stack trace: + +```shell +[ + "app/models/group.rb:305:in `has_owner?'", + "ee/app/views/shared/members/ee/_license_badge.html.haml:1", + "app/helpers/application_helper.rb:19:in `render_if_exists'", + "app/views/shared/members/_member.html.haml:31", + "app/views/groups/group_members/index.html.haml:75", + "app/controllers/application_controller.rb:134:in `render'", + "ee/lib/gitlab/ip_address_state.rb:10:in `with'", + "ee/app/controllers/ee/application_controller.rb:44:in `set_current_ip_address'", + "app/controllers/application_controller.rb:493:in `set_current_admin'", + "lib/gitlab/session.rb:11:in `with_session'", + "app/controllers/application_controller.rb:484:in `set_session_storage'", + "app/controllers/application_controller.rb:478:in `set_locale'", + "lib/gitlab/error_tracking.rb:52:in `with_context'", + "app/controllers/application_controller.rb:543:in `sentry_context'", + "app/controllers/application_controller.rb:471:in `block in set_current_context'", + "lib/gitlab/application_context.rb:54:in `block in use'", + "lib/gitlab/application_context.rb:54:in `use'", + "lib/gitlab/application_context.rb:21:in `with_context'", + "app/controllers/application_controller.rb:463:in `set_current_context'", + "lib/gitlab/jira/middleware.rb:19:in `call'" +] +``` + +The stack trace, shows us that we obviously have an N+1 problem, since we are repeatably executing for each group member: + +```ruby +group.has_owner?(current_user) +``` + +This is easily solvable by extracting this check, above the loop. + +After [the fix](https://gitlab.com/gitlab-org/gitlab/-/issues/231468), we now have: + + + +## How to measure the impact of the change + +We can use the [memory profiler](performance.md#using-memory-profiler) to profile our code. +For the previous example, we could wrap the profiler around the `Groups::GroupMembersController#index` action. + +We had: + +- Total allocated: 7133601 bytes (84858 objects) +- Total retained: 757595 bytes (6070 objects) +- `db_count`: 144 +- `db_cached_count`: 55 +- `db_duration`: 303ms + +After the fix, we can see that we have reduced the allocated memory as well as the number of cached queries and improved execution time: + +- Total allocated: 5313899 bytes (65290 objects), 1810KB (25%) less +- Total retained: 685593 bytes (5278 objects), 72KB (9%) less +- `db_count`: 95 (34% less) +- `db_cached_count`: 6 (89% less) +- `db_duration`: 162ms (87% faster) + +## See also + +- [Metrics that would help us detect the potential N+1 Cached SQL calls](https://gitlab.com/gitlab-org/gitlab/-/issues/259007) +- [Merge Request performance guidelines for cached queries](merge_request_performance_guidelines.md#cached-queries) +- [Improvements for biggest offenders](https://gitlab.com/groups/gitlab-org/-/epics/4508) diff --git a/doc/development/changelog.md b/doc/development/changelog.md index 922c4814d91..b8e879c1826 100644 --- a/doc/development/changelog.md +++ b/doc/development/changelog.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Changelog entries This guide contains instructions for when and how to generate a changelog entry @@ -38,9 +44,10 @@ the `author` field. GitLab team members **should not**. - [Security fixes](https://gitlab.com/gitlab-org/release/docs/blob/master/general/security/developer.md) **must** have a changelog entry, without `merge_request` value and with `type` set to `security`. -- Any user-facing change **should** have a changelog entry. This includes both visual changes (regardless of how minor), and changes to the rendered DOM which impact how a screen reader may announce the content. +- Any user-facing change **must** have a changelog entry. This includes both visual changes (regardless of how minor), and changes to the rendered DOM which impact how a screen reader may announce the content. +- Any client-facing change to our REST and GraphQL APIs **must** have a changelog entry. - Performance improvements **should** have a changelog entry. -- Changes that need to be documented in the Product Analytics [Event Dictionary](product_analytics/event_dictionary.md) +- Changes that need to be documented in the Product Analytics [Event Dictionary](https://about.gitlab.com/handbook/product/product-analytics-guide#event-dictionary) also require a changelog entry. - _Any_ contribution from a community member, no matter how small, **may** have a changelog entry regardless of these guidelines if the contributor wants one. @@ -48,7 +55,8 @@ the `author` field. GitLab team members **should not**. - Any docs-only changes **should not** have a changelog entry. - Any change behind a disabled feature flag **should not** have a changelog entry. - Any change behind an enabled feature flag **should** have a changelog entry. -- Any change that adds new usage data metrics and changes that needs to be documented in Product Analytics [Event Dictionary](telemetry/event_dictionary.md) **should** have a changelog entry. +- Any change that adds new usage data metrics and changes that needs to be documented in Product Analytics [Event Dictionary](https://about.gitlab.com/handbook/product/product-analytics-guide#event-dictionary) **should** have a changelog entry. +- A change that adds snowplow events **should** have a changelog entry - - A change that [removes a feature flag](feature_flags/development.md) **should** have a changelog entry - only if the feature flag did not default to true already. - A fix for a regression introduced and then fixed in the same release (i.e., @@ -117,7 +125,6 @@ the `--ee` option: bin/changelog --ee 'Hey DZ, I added a feature to GitLab!' ``` -NOTE: **Note:** All entries in the `CHANGELOG.md` file apply to all editions of GitLab. Changelog updates are based on a common [GitLab codebase](https://gitlab.com/gitlab-org/gitlab/), and are mirrored without proprietary code to [GitLab FOSS](https://gitlab.com/gitlab-org/gitlab-foss/) (also known as GitLab Community Edition). diff --git a/doc/development/chaos_endpoints.md b/doc/development/chaos_endpoints.md index 26ff3d2def7..63218af857d 100644 --- a/doc/development/chaos_endpoints.md +++ b/doc/development/chaos_endpoints.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Generating chaos in a test GitLab instance As [Werner Vogels](https://twitter.com/Werner), the CTO at Amazon Web Services, famously put it, **Everything fails, all the time**. @@ -18,7 +24,7 @@ Currently, there are four endpoints for simulating the following conditions: For obvious reasons, these endpoints are not enabled by default on `production`. They are enabled by default on **development** environments. -DANGER: **Danger:** +DANGER: **Warning:** It is required that you secure access to the chaos endpoints using a secret token. You should not enable them in production unless you absolutely know what you're doing. @@ -44,8 +50,7 @@ each endpoint can be set to `true`. This will run the chaos process in a Sidekiq To simulate a memory leak in your application, use the `/-/chaos/leakmem` endpoint. -NOTE: **Note:** -The memory is not retained after the request finishes. Once the request has completed, the Ruby garbage collector will attempt to recover the memory. +The memory is not retained after the request finishes. After the request has completed, the Ruby garbage collector will attempt to recover the memory. ```plaintext GET /-/chaos/leakmem @@ -80,7 +85,7 @@ GET /-/chaos/cpu_spin?duration_s=50&async=true | Attribute | Type | Required | Description | | ------------ | ------- | -------- | --------------------------------------------------------------------- | -| `duration_s` | integer | no | Duration, in seconds, that the core will be utilized. Defaults to 30s | +| `duration_s` | integer | no | Duration, in seconds, that the core will be used. Defaults to 30s | | `async` | boolean | no | Set to true to consume CPU in a Sidekiq background worker process | ```shell @@ -105,7 +110,7 @@ GET /-/chaos/db_spin?duration_s=50&async=true | Attribute | Type | Required | Description | | ------------ | ------- | -------- | --------------------------------------------------------------------------- | | `interval_s` | float | no | Interval, in seconds, for every DB request. Defaults to 1s | -| `duration_s` | integer | no | Duration, in seconds, that the core will be utilized. Defaults to 30s | +| `duration_s` | integer | no | Duration, in seconds, that the core will be used. Defaults to 30s | | `async` | boolean | no | Set to true to perform the operation in a Sidekiq background worker process | ```shell @@ -139,8 +144,8 @@ curl http://localhost:3000/-/chaos/sleep?duration_s=60&token=secret This endpoint will simulate the unexpected death of a worker process using a `kill` signal. -NOTE: **Note:** -Since this endpoint uses the `KILL` signal, the worker is not given a chance to cleanup or shutdown. +Because this endpoint uses the `KILL` signal, the worker isn't given an +opportunity to cleanup or shutdown. ```plaintext GET /-/chaos/kill diff --git a/doc/development/cicd/templates.md b/doc/development/cicd/templates.md index 7d522a55559..e4c3e622ede 100644 --- a/doc/development/cicd/templates.md +++ b/doc/development/cicd/templates.md @@ -174,3 +174,7 @@ is updated in a major version GitLab release. A template could contain malicious code. For example, a template that contains the `export` shell command in a job might accidentally expose project secret variables in a job log. If you're unsure if it's secure or not, you need to ask security experts for cross-validation. + +## Contribute CI/CD Template Merge Requests + +After your CI/CD Template MR is created and labeled with `ci::templates`, DangerBot suggests one reviewer and one maintainer that can review your code. When your merge request is ready for review, please `@mention` the reviewer and ask them to review your CI/CD Template changes. See details in the merge request that added [a DangerBot task for CI/CD Template MRs](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/44688). diff --git a/doc/development/code_comments.md b/doc/development/code_comments.md index b0e83b29cb0..d9ab719d18a 100644 --- a/doc/development/code_comments.md +++ b/doc/development/code_comments.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Code comments Whenever you add comment to the code that is expected to be addressed at any time diff --git a/doc/development/code_intelligence/index.md b/doc/development/code_intelligence/index.md index bd11f0bff79..24abf57e9d6 100644 --- a/doc/development/code_intelligence/index.md +++ b/doc/development/code_intelligence/index.md @@ -1,3 +1,9 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Code Intelligence > [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/1576) in GitLab 13.1. diff --git a/doc/development/code_review.md b/doc/development/code_review.md index 3ff802d3b23..63a47240435 100644 --- a/doc/development/code_review.md +++ b/doc/development/code_review.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Code Review Guidelines This guide contains advice and best practices for performing code review, and @@ -91,8 +97,14 @@ with [domain expertise](#domain-experts). **approved by a [frontend maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_maintainers_frontend)**. 1. If your merge request includes UX changes (*1*), it must be **approved by a [UX team member](https://about.gitlab.com/company/team/)**. -1. If your merge request includes adding a new JavaScript library (*1*), it must be - **approved by a [frontend lead](https://about.gitlab.com/company/team/)**. +1. If your merge request includes adding a new JavaScript library (*1*)... + - If the library significantly increases the + [bundle size](https://gitlab.com/gitlab-org/frontend/playground/webpack-memory-metrics/-/blob/master/doc/report.md), it must + be **approved by a [frontend foundations member](https://about.gitlab.com/direction/create/ecosystem/frontend-ux-foundations/)**. + - If the license used by the new library hasn't been approved for use in + GitLab, the license must be **approved by a [legal department member](https://about.gitlab.com/handbook/legal/)**. + More information about license compatiblity can be found in our + [GitLab Licensing and Compatibility documentation](licensing.md). 1. If your merge request includes adding a new UI/UX paradigm (*1*), it must be **approved by a [UX lead](https://about.gitlab.com/company/team/)**. 1. If your merge request includes a new dependency or a filesystem change, it must be @@ -378,8 +390,7 @@ When ready to merge: - When you set the MR to "Merge When Pipeline Succeeds", you should take over subsequent revisions for anything that would be spotted after that. -NOTE: **Note:** -Thanks to "Pipeline for Merged Results", authors won't have to rebase their +Thanks to **Pipeline for Merged Results**, authors won't have to rebase their branch as frequently anymore (only when there are conflicts) since the Merge Results Pipeline will already incorporate the latest changes from `master`. This results in faster review/merge cycles since maintainers don't have to ask diff --git a/doc/development/contributing/index.md b/doc/development/contributing/index.md index 6cbe57bf926..17431195c3d 100644 --- a/doc/development/contributing/index.md +++ b/doc/development/contributing/index.md @@ -37,7 +37,7 @@ Report suspected security vulnerabilities in private to `support@gitlab.com`, also see the [disclosure section on the GitLab.com website](https://about.gitlab.com/security/disclosure/). -DANGER: **Danger:** +DANGER: **Warning:** Do **NOT** create publicly viewable issues for suspected security vulnerabilities. ## Code of conduct @@ -146,7 +146,7 @@ Keep the following in mind when submitting merge requests: reviewers. - If the code quality is found to not meet GitLab’s standards, the merge request reviewer will provide guidance and refer the author to our: - - [Documentation](../documentation/styleguide.md) style guide. + - [Documentation](../documentation/styleguide/index.md) style guide. - Code style guides. - Sometimes style guides will be followed but the code will lack structural integrity, or the reviewer will have reservations about the code’s overall quality. When there is a reservation, diff --git a/doc/development/contributing/issue_workflow.md b/doc/development/contributing/issue_workflow.md index b7c05a369f0..99650c24661 100644 --- a/doc/development/contributing/issue_workflow.md +++ b/doc/development/contributing/issue_workflow.md @@ -90,25 +90,6 @@ explain what falls under each type label. The GitLab handbook documents [when something is a bug](https://about.gitlab.com/handbook/product/product-processes/#bug-issues) and [when it is a feature request](https://about.gitlab.com/handbook/product/product-processes/#feature-issues). -### Facet labels - -Sometimes it's useful to refine the type of an issue. In those cases, you can -add facet labels. - -Following is a non-exhaustive list of facet labels: - -- ~enhancement: This label can refine an issue that has the ~feature label. -- ~"master:broken": This label can refine an issue that has the ~bug label. -- ~"failure::flaky-test": This label can refine an issue that has the ~bug label. -- ~"technical debt": This label can refine an issue that has the ~tooling label. -- ~"static analysis": This label can refine an issue that has the ~tooling label. -- ~"ci-build": This label can refine an issue that has the ~tooling label. -- ~performance: A performance issue could describe a ~bug or a ~feature. -- ~security: A security issue could describe a ~bug or a ~feature. -- ~database: A database issue could describe a ~bug or a ~feature. -- ~customer: This relates to an issue that was created by a customer, or that is of interest for a customer. -- ~"UI text": Issues that add or modify any text within the UI such as user-assistance microcopy, button/menu labels, or error messages. - ### Stage labels Stage labels specify which [stage](https://about.gitlab.com/handbook/product/product-categories/#hierarchy) the issue belongs to. @@ -226,6 +207,13 @@ Examples of feature labels are `~wiki`, `~ldap`, `~api`, `~issues`, `~"merge req Feature labels are all-lowercase. +### Facet labels + +To track additional information or context about created issues, developers may +add _facet labels_. Facet labels are also sometimes used for issue prioritization +or for measurements (such as time to close). An example of a facet label is the +~customer label, which indicates customer interest. + ### Department labels The current department labels are: diff --git a/doc/development/contributing/merge_request_workflow.md b/doc/development/contributing/merge_request_workflow.md index 31f59ad875c..d5ffff7bfc8 100644 --- a/doc/development/contributing/merge_request_workflow.md +++ b/doc/development/contributing/merge_request_workflow.md @@ -123,24 +123,37 @@ document from the Kubernetes team also has some great points regarding this. ### Commit messages guidelines -When writing commit messages, please follow the guidelines below: +Commit messages should follow the guidelines below, for reasons explained by Chris Beams in [How to Write a Git Commit Message](https://chris.beams.io/posts/git-commit/): -- The commit subject must contain at least 3 words. -- The commit subject must not be longer than 72 characters. +- The commit subject and body must be separated by a blank line. - The commit subject must start with a capital letter. +- The commit subject must not be longer than 72 characters. - The commit subject must not end with a period. -- The commit subject and body must be separated by a blank line. - The commit body must not contain more than 72 characters per line. - Commits that change 30 or more lines across at least 3 files must describe these changes in the commit body. - The commit subject or body must not contain Emojis. - Use issues and merge requests' full URLs instead of short references, as they are displayed as plain text outside of GitLab. -- The merge request must not contain more than 10 commit messages. +- The merge request should not contain more than 10 commit messages. +- The commit subject should contain at least 3 words. + +**Important notes:** + +- If the guidelines are not met, the MR may not pass the [Danger checks](https://gitlab.com/gitlab-org/gitlab/blob/master/danger/commit_messages/Dangerfile). +- Consider enabling [Squash and merge](../../user/project/merge_requests/squash_and_merge.md#squash-and-merge) + if your merge request includes "Applied suggestion to X files" commits, so that Danger can ignore those. +- The prefixes in the form of `[prefix]` and `prefix:` are allowed (they can be all lowercase, as long + as the message itself is capitalized). For instance, `danger: Improve Danger behavior` and + `[API] Improve the labels endpoint` are valid commit messages. + +#### Why these standards matter + +1. Consistent commit messages that follow these guidelines make the history more readable. +1. Concise standard commit messages helps to identify breaking changes for a deployment or ~"master:broken" quicker when + reviewing commits between two points in time. -If the guidelines are not met, the MR will not pass the -[Danger checks](https://gitlab.com/gitlab-org/gitlab/blob/master/danger/commit_messages/Dangerfile). -For more information see [How to Write a Git Commit Message](https://chris.beams.io/posts/git-commit/). +#### Commit message template Example commit message template that can be used on your machine that embodies the above (guide for [how to apply template](https://codeinthehole.com/tips/a-useful-template-for-commit-messages/)): diff --git a/doc/development/contributing/style_guides.md b/doc/development/contributing/style_guides.md index 773c1a771cd..fd3fe239110 100644 --- a/doc/development/contributing/style_guides.md +++ b/doc/development/contributing/style_guides.md @@ -15,25 +15,49 @@ settings automatically by default. If your editor/IDE does not automatically sup we suggest investigating to see if a plugin exists. For instance here is the [plugin for vim](https://github.com/editorconfig/editorconfig-vim). -## Pre-commit static analysis +## Pre-push static analysis -You should install [`overcommit`](https://github.com/sds/overcommit) to automatically check for -static analysis offenses before committing locally. +We strongly recommend installing [Lefthook](https://github.com/Arkweid/lefthook) to automatically check +for static analysis offenses before pushing your changes. -After installing `overcommit`, run the following in your GitLab source directory: +To install `lefthook`, run the following in your GitLab source directory: ```shell -make -C tooling/overcommit +# 1. Make sure to uninstall Overcommit first +overcommit --uninstall + +# If using rbenv, at this point you may need to do: rbenv rehash + +# 2. Install lefthook... + +## With Homebrew (macOS) +brew install Arkweid/lefthook/lefthook + +## Or with Go +go get github.com/Arkweid/lefthook + +## Or with Rubygems +gem install lefthook + +# 3. Install the Git hooks +lefthook install -f ``` -Then before a commit is created, `overcommit` automatically checks for RuboCop (and other checks) -offenses on every modified file. +Before you push your changes, Lefthook then automatically run Danger checks, and other checks +for changed files. This saves you time as you don't have to wait for the same errors to be detected +by CI/CD. + +Lefthook relies on a pre-push hook to prevent commits that violate its ruleset. +To override this behavior, pass the environment variable `LEFTHOOK=0`. That is, +`LEFTHOOK=0 git push`. -This saves you time as you don't have to wait for the same errors to be detected by CI/CD. +You can also: -`overcommit` relies on a pre-commit hook to prevent commits that violate its ruleset. To override -this behavior, pass the `OVERCOMMIT_DISABLE` environment variable. For example, -`OVERCOMMIT_DISABLE=1 git rebase master` to rebase while disabling the Git hook. +- Define [local configuration](https://github.com/Arkweid/lefthook/blob/master/docs/full_guide.md#local-config). +- Skip [checks per tag on the fly](https://github.com/Arkweid/lefthook/blob/master/docs/full_guide.md#skip-some-tags-on-the-fly). + For example, `LEFTHOOK_EXCLUDE=frontend git push origin`. +- Run [hooks manually](https://github.com/Arkweid/lefthook/blob/master/docs/full_guide.md#run-githook-group-directly). + For example, `lefthook run pre-push`. ## Ruby, Rails, RSpec @@ -100,7 +124,7 @@ We're following [Ciro Santilli's Markdown Style Guide](https://cirosantilli.com/ ## Documentation -See the dedicated [Documentation Style Guide](../documentation/styleguide.md). +See the dedicated [Documentation Style Guide](../documentation/styleguide/index.md). ## Python diff --git a/doc/development/creating_enums.md b/doc/development/creating_enums.md index af9bf919b2b..fbf35171ecb 100644 --- a/doc/development/creating_enums.md +++ b/doc/development/creating_enums.md @@ -1,3 +1,9 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Creating enums When creating a new enum, it should use the database type `SMALLINT`. diff --git a/doc/development/dangerbot.md b/doc/development/dangerbot.md index 6fda394c10d..4feec5c093a 100644 --- a/doc/development/dangerbot.md +++ b/doc/development/dangerbot.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Danger bot The GitLab CI/CD pipeline includes a `danger-review` job that uses [Danger](https://github.com/danger/danger) diff --git a/doc/development/database/constraint_naming_convention.md b/doc/development/database/constraint_naming_convention.md new file mode 100644 index 00000000000..63a2d607ac5 --- /dev/null +++ b/doc/development/database/constraint_naming_convention.md @@ -0,0 +1,26 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + +# Constraints naming conventions + +The most common option is to let Rails pick the name for database constraints and indexes or let PostgreSQL use the defaults (when applicable). However, when needing to define custom names in Rails or working in Go applications where no ORM is used, it is important to follow strict naming conventions to improve consistency and discoverability. + +The table below describes the naming conventions for custom PostgreSQL constraints. +Please note that the intent is not to retroactively change names in existing databases but rather ensure consistency of future changes. + +| Type | Syntax | Notes | Examples | +|--------------------------|---------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------| +| **Primary Key** | `pk_<table name>` | | `pk_projects` | +| **Foreign Key** | `fk_<table name>_<column name>[_and_<column name>]*_<foreign table name>` | | `fk_projects_group_id_groups` | +| **Index** | `index_<table name>_on_<column name>[_and_<column name>]*[_and_<column name in partial clause>]*` | | `index_repositories_on_group_id` | +| **Unique Constraint** | `unique_<table name>_<column name>[_and_<column name>]*` | | `unique_projects_group_id_and_name` | +| **Check Constraint** | `check_<table name>_<column name>[_and_<column name>]*[_<suffix>]?` | The optional suffix should denote the type of validation, such as `length` and `enum`. It can also be used to desambiguate multiple `CHECK` constraints on the same column. | `check_projects_name_length`<br />`check_projects_type_enum`<br />`check_projects_admin1_id_and_admin2_id_differ` | +| **Exclusion Constraint** | `excl_<table name>_<column name>[_and_<column name>]*_[_<suffix>]?` | The optional suffix should denote the type of exclusion being performed. | `excl_reservations_start_at_end_at_no_overlap` | + +## Observations + +- Prefixes are preferred over suffices because they make it easier to identify the type of a given constraint quickly, as well as group them alphabetically; +- The `_and_` that joins column names can be omitted to keep the identifiers under the 63 characters' length limit defined by PostgreSQL. Additionally, the notation may be abbreviated to the best of our ability if struggling to keep under this limit. diff --git a/doc/development/database/index.md b/doc/development/database/index.md index 4bcefefe7a7..19159c6c0ff 100644 --- a/doc/development/database/index.md +++ b/doc/development/database/index.md @@ -57,9 +57,14 @@ info: To determine the technical writer assigned to the Stage/Group associated w - [Query Count Limits](../query_count_limits.md) - [Creating enums](../creating_enums.md) - [Client-side connection-pool](client_side_connection_pool.md) -- [Updating multiple values](./setting_multiple_values.md) +- [Updating multiple values](setting_multiple_values.md) +- [Constraints naming conventions](constraint_naming_convention.md) ## Case studies - [Database case study: Filtering by label](../filtering_by_label.md) - [Database case study: Namespaces storage statistics](../namespaces_storage_statistics.md) + +## Miscellaneous + +- [Maintenance operations](maintenance_operations.md) diff --git a/doc/development/database/maintenance_operations.md b/doc/development/database/maintenance_operations.md new file mode 100644 index 00000000000..c84ec31471e --- /dev/null +++ b/doc/development/database/maintenance_operations.md @@ -0,0 +1,46 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + +# Maintenance operations + +This page details various database related operations that may relate to development. + +## Disabling an index + +There are certain situations in which you might want to disable an index before removing it: + +- The index is on a large table and rebuilding it in the case of a revert would take a long time. +- It is uncertain whether or not the index is being used in ways that are not fully visible. + +To disable an index before removing it: + +1. Open a [production infrastructure issue](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/new) +and use the "Production Change" template. +1. Inform the database team in the issue `@gl-database` or in Slack `#database`. +1. Add a step to verify the index is used (this would likely be an `EXPLAIN` command known to use the index). +1. Add the step to disable the index: + + ```sql + UPDATE pg_index SET indisvalid = false WHERE indexrelid = 'index_issues_on_foo'::regclass; + ``` + +1. Add a step to verify the index is invalid (this would likely be the same as used to verify before disabling the index). +1. Verify the index is invalid on replicas: + + ```sql + SELECT indisvalid FROM pg_index WHERE indexrelid = 'index_issues_on_foo'::regclass; + ``` + +1. Add steps for rolling back the invalidation: + 1. Rollback the index invalidation + + ```sql + UPDATE pg_index SET indisvalid = true WHERE indexrelid = 'index_issues_on_foo'::regclass; + ``` + + 1. Verify the index is being used again. + +See this [example infrastructure issue](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2795) for reference. diff --git a/doc/development/database/setting_multiple_values.md b/doc/development/database/setting_multiple_values.md index 5569a0e10b7..c354247a9f8 100644 --- a/doc/development/database/setting_multiple_values.md +++ b/doc/development/database/setting_multiple_values.md @@ -1,9 +1,15 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Setting Multiple Values > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32921) in GitLab 13.5. -Frequently, we will want to update multiple objects with new values for one -or more columns. The obvious way to do this is using `Relation#update_all`: +There's often a need to update multiple objects with new values for one +or more columns. One method of doing this is using `Relation#update_all`: ```ruby user.issues.open.update_all(due_date: 7.days.from_now) # (1) @@ -28,11 +34,11 @@ update issues where id = obj_id ``` -The bad news: There is no way to express this in ActiveRecord or even dropping -down to ARel - the `UpdateManager` just does not support `update from`, so this +The bad news: there is no way to express this in ActiveRecord or even dropping +down to ARel. The `UpdateManager` does not support `update from`, so this is not expressible. -The good news: We supply an abstraction to help you generate these kinds of +The good news: we supply an abstraction to help you generate these kinds of updates, called `Gitlab::Database::BulkUpdate`. This constructs queries such as the above, and uses binding parameters to avoid SQL injection. @@ -44,7 +50,7 @@ To use this, we need: - a mapping from object/ID to the new values to set for that object - a way to determine the table for each object -So for example, we can express the query above as: +For example, we can express the query above as: ```ruby issue_a = Issue.find(..) @@ -87,7 +93,7 @@ objects = Foo.from_union([ ]) # At this point, all the objects are instances of Foo, even the ones from the # Bar table -mapping = objects.to_h { |obj| [obj, bazzes[obj.id] } +mapping = objects.to_h { |obj| [obj, bazzes[obj.id]] } # Issues at most 2 queries ::Gitlab::Database::BulkUpdate.execute(%i[baz], mapping) do |obj| @@ -100,4 +106,4 @@ end Note that this is a **very low level** tool, and operates on the raw column values. Enumerations and state fields must be translated into their underlying representations, for example, and nested associations are not supported. No -validations or hooks will be called. +validations or hooks are called. diff --git a/doc/development/database_review.md b/doc/development/database_review.md index 3f5f36b0b6e..d1ec32af464 100644 --- a/doc/development/database_review.md +++ b/doc/development/database_review.md @@ -27,7 +27,7 @@ A database review is required for: database review. - Changes in usage data metrics that use `count` and `distinct_count`. These metrics could have complex queries over large tables. - See the [Product Analytics Guide](product_analytics/usage_ping.md#implementing-usage-ping) + See the [Product Analytics Guide](https://about.gitlab.com/handbook/product/product-analytics-guide/) for implementation details. A database reviewer is expected to look out for obviously complex @@ -106,7 +106,8 @@ test its execution using `CREATE INDEX CONCURRENTLY` in the `#database-lab` Slac - Write the raw SQL in the MR description. Preferably formatted nicely with [pgFormatter](https://sqlformat.darold.net) or - [paste.depesz.com](https://paste.depesz.com). + [paste.depesz.com](https://paste.depesz.com) and using regular quotes + (e.g. `"projects"."id"`) and avoiding smart quotes (e.g. `“projects”.“id”`). - Include the output of `EXPLAIN (ANALYZE, BUFFERS)` of the relevant queries in the description. If the output is too long, wrap it in `<details>` blocks, paste it in a GitLab Snippet, or provide the @@ -201,8 +202,8 @@ estimated to keep migration timing to a minimum. NOTE: **Note:** Keep in mind that all runtimes should be measured against GitLab.com. -| Migration Type | Execution Time Recommended | Notes | +| Migration Type | Execution Time Recommended | Notes | |----|----|---| | Regular migrations on `db/migrate` | `3 minutes` | A valid exception are index creation as this can take a long time. | -| Post migrations on `db/post_migrate` | `10 minutes` | | -| Background migrations | --- | Since these are suitable for larger tables, it's not possible to set a precise timing guideline, however, any single query must stay below `1 second` execution time with cold caches. | +| Post migrations on `db/post_migrate` | `10 minutes` | | +| Background migrations | --- | Since these are suitable for larger tables, it's not possible to set a precise timing guideline, however, any single query must stay below `1 second` execution time with cold caches. | diff --git a/doc/development/deleting_migrations.md b/doc/development/deleting_migrations.md index b8f23019ac2..df36715fa6a 100644 --- a/doc/development/deleting_migrations.md +++ b/doc/development/deleting_migrations.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Delete existing migrations When removing existing migrations from the GitLab project, you have to take into account diff --git a/doc/development/deprecation_guidelines/index.md b/doc/development/deprecation_guidelines/index.md index 1ee22644bbc..5cc408bb57b 100644 --- a/doc/development/deprecation_guidelines/index.md +++ b/doc/development/deprecation_guidelines/index.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Deprecation guidelines This page includes information about how and when to remove or make breaking diff --git a/doc/development/diffs.md b/doc/development/diffs.md index eb070cbf4d7..ece7bb9e9ee 100644 --- a/doc/development/diffs.md +++ b/doc/development/diffs.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Working with diffs Currently we rely on different sources to present diffs, these include: @@ -93,7 +99,6 @@ Gitlab::Git::DiffCollection.collection_limits[:max_bytes] = Gitlab::Git::DiffCol No more files will be rendered at all if 5 megabytes have already been rendered. -NOTE: **Note:** All collection limit parameters are currently sent and applied on Gitaly. That is, once the limit is surpassed, Gitaly will only return the safe amount of data to be persisted on `merge_request_diff_files`. @@ -108,7 +113,6 @@ That is, it's equivalent to 10kb if the maximum allowed value is 100kb. The diff will still be persisted and expandable if the patch size doesn't surpass `ApplicationSettings#diff_max_patch_bytes`. -NOTE: **Note:** Although this nomenclature (Collapsing) is also used on Gitaly, this limit is only used on GitLab (hardcoded - not sent to Gitaly). Gitaly will only return `Diff.Collapsed` (RPC) when surpassing collection limits. @@ -123,7 +127,6 @@ Commit::DIFF_SAFE_LINES = Gitlab::Git::DiffCollection::DEFAULT_LIMITS[:max_lines File diff will be suppressed (technically different from collapsed, but behaves the same, and is expandable) if it has more than 5000 lines. -NOTE: **Note:** This limit is currently hardcoded and only applied on GitLab. ## Viewers @@ -132,3 +135,51 @@ Diff Viewers, which can be found on `models/diff_viewer/*` are classes used to m whether it's a binary, which partial should be used to render it or which File extensions this class accounts for. `DiffViewer::Base` validates _blobs_ (old and new versions) content, extension and file type in order to check if it can be rendered. + +## Merge request diffs against the `HEAD` of the target branch + +Historically, merge request diffs have been calculated by `git diff target...source` which compares the +`HEAD` of the source branch with the merge base (or a common ancestor) of the target branch and the source's. +This solution works well until the target branch starts containing some of the +changes introduced by the source branch: Consider the following case, in which the source branch +is `feature_a` and the target is `master`: + +1. Checkout a new branch `feature_a` from `master` and remove `file_a` and `file_b` in it. +1. Add a commit that removes `file_a` to `master`. + +The merge request diff still contains the `file_a` removal while the actual diff compared to +`master`'s `HEAD` has only the `file_b` removal. The diff with such redundant +changes is harder to review. + +In order to display an up-to-date diff, in GitLab 12.9 we +[introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/27008) merge request +diffs compared against `HEAD` of the target branch: the +target branch is artificially merged into the source branch, then the resulting +merge ref is compared to the source branch in order to calculate an accurate +diff. + +Until we complete the epics ["use merge refs for diffs"](https://gitlab.com/groups/gitlab-org/-/epics/854) +and ["merge conflicts in diffs"](https://gitlab.com/groups/gitlab-org/-/epics/4893), +both options `master (base)` and `master (HEAD)` are available to be displayed in merge requests: + + + +The `master (HEAD)` option is meant to replace `master (base)` in the future. + +In order to support comments for both options, diff note positions are stored for +both `master (base)` and `master (HEAD)` versions ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/198457) in 12.10). +The position for `master (base)` version is stored in `Note#position` and +`Note#original_position` columns, for `master (HEAD)` version `DiffNotePosition` +has been introduced. + +One of the key challenges to deal with when working on merge ref diffs are merge +conflicts. If the target and source branch contains a merge conflict, the branches +cannot be automatically merged. The [recording on +YouTube](https://www.youtube.com/watch?v=GFXIFA4ZuZw&feature=youtu.be&ab_channel=GitLabUnfiltered) +is a quick introduction to the problem and the motivation behind the [epic](https://gitlab.com/groups/gitlab-org/-/epics/854). + +In 13.5 a solution for both-modified merge +conflict has been +[introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/232484). However, +there are more classes of merge conflicts that are to be +[addressed](https://gitlab.com/groups/gitlab-org/-/epics/4893) in the future. diff --git a/doc/development/distributed_tracing.md b/doc/development/distributed_tracing.md index 952435150d6..6d277f9ae99 100644 --- a/doc/development/distributed_tracing.md +++ b/doc/development/distributed_tracing.md @@ -178,6 +178,7 @@ This configuration string uses the Jaeger driver `opentracing://jaeger` with the | `udp_endpoint` | `localhost:6831` | This is the default. Configures Jaeger to send trace information to the UDP listener on port `6831` using compact thrift protocol. Note that we've experienced some issues with the [Jaeger Client for Ruby](https://github.com/salemove/jaeger-client-ruby) when using this protocol. | | `sampler` | `probabalistic` | Configures Jaeger to use a probabilistic random sampler. The rate of samples is configured by the `sampler_param` value. | | `sampler_param` | `0.01` | Use a ratio of `0.01` to configure the `probabalistic` sampler to randomly sample _1%_ of traces. | +| `service_name` | `api` | Override the service name used by the Jaeger backend. This parameter will take precedence over the application-supplied value. | NOTE: **Note:** The same `GITLAB_TRACING` value should to be configured in the environment @@ -185,7 +186,7 @@ variables for all GitLab processes, including Workhorse, Gitaly, Rails, and Side ### 3. Start the GitLab application -Once the `GITLAB_TRACING` environment variable is exported to all GitLab services, start the +After the `GITLAB_TRACING` environment variable is exported to all GitLab services, start the application. When `GITLAB_TRACING` is configured properly, the application will log this on startup: diff --git a/doc/development/documentation/feature_flags.md b/doc/development/documentation/feature_flags.md index 0f03ceeb4b5..5bc3e1d59e2 100644 --- a/doc/development/documentation/feature_flags.md +++ b/doc/development/documentation/feature_flags.md @@ -30,8 +30,7 @@ See how to document them below, according to the state of the flag: - [Features that can be enabled or disabled for a single project](#features-enabled-by-project). - [Features with the feature flag removed](#features-with-flag-removed). -NOTE: **Note:** -The [`**(CORE ONLY)**`](styleguide.md#product-badges) badge or equivalent for +The [`**(CORE ONLY)**`](styleguide/index.md#product-badges) badge or equivalent for the feature's tier should be added to the line and heading that refers to enabling/disabling feature flags as Admin access is required to do so, therefore, it indicates that it cannot be done by regular users of GitLab.com. diff --git a/doc/development/documentation/graphql_styleguide.md b/doc/development/documentation/graphql_styleguide.md index 11e6b159359..d658794f7e0 100644 --- a/doc/development/documentation/graphql_styleguide.md +++ b/doc/development/documentation/graphql_styleguide.md @@ -67,7 +67,7 @@ Set up the section with the following: ``` - Include a screenshot of the result in the GraphiQL explorer. Follow the naming - convention described in the [Save the image](styleguide.md#save-the-image) section of the Documentation style guide. + convention described in the [Save the image](styleguide/index.md#save-the-image) section of the Documentation style guide. - Follow up with an example of what you can do with the output. Make sure the example is something that readers can do on their own deployments. - Include a link to the [GraphQL API resources](../../api/graphql/reference/index.md). diff --git a/doc/development/documentation/index.md b/doc/development/documentation/index.md index e51f966ee6e..69a8ff10878 100644 --- a/doc/development/documentation/index.md +++ b/doc/development/documentation/index.md @@ -11,7 +11,7 @@ GitLab's documentation is [intended as the single source of truth (SSOT)](https: In addition to this page, the following resources can help you craft and contribute to documentation: -- [Style Guide](styleguide.md) - What belongs in the docs, language guidelines, Markdown standards to follow, links, and more. +- [Style Guide](styleguide/index.md) - What belongs in the docs, language guidelines, Markdown standards to follow, links, and more. - [Structure and template](structure.md) - Learn the typical parts of a doc page and how to write each one. - [Documentation process](workflow.md). - [Markdown Guide](../../user/markdown.md) - A reference for all Markdown syntax supported by GitLab. @@ -64,11 +64,11 @@ However, anyone can contribute [documentation improvements](workflow.md) that ar [GitLab docs](https://gitlab.com/gitlab-org/gitlab-docs) uses [GitLab Kramdown](https://gitlab.com/gitlab-org/gitlab_kramdown) as its Markdown rendering engine. See the [GitLab Markdown Guide](https://about.gitlab.com/handbook/markdown-guide/) for a complete Kramdown reference. -Adhere to the [Documentation Style Guide](styleguide.md). If a style standard is missing, you are welcome to suggest one via a merge request. +Adhere to the [Documentation Style Guide](styleguide/index.md). If a style standard is missing, you are welcome to suggest one via a merge request. ## Folder structure and files -See the [Structure](styleguide.md#structure) section of the [Documentation Style Guide](styleguide.md). +See the [Structure](styleguide/index.md#structure) section of the [Documentation Style Guide](styleguide/index.md). ## Metadata @@ -229,7 +229,7 @@ Things to note: it in for `workflow/lfs/lfs_administration` and `lfs/lfs_administration` and will print the file and the line where this file is mentioned. You may ask why the two greps. Since [we use relative paths to link to - documentation](styleguide.md#links), sometimes it might be useful to search a path deeper. + documentation](styleguide/index.md#links), sometimes it might be useful to search a path deeper. - The `*.md` extension is not used when a document is linked to GitLab's built-in help page, which is why we omit it in `git grep`. - Use the checklist on the "Change documentation location" MR description template. @@ -472,269 +472,7 @@ The following GitLab features are used among others: ## Testing -We treat documentation as code, and so use tests in our CI pipeline to maintain the -standards and quality of the docs. The current tests, which run in CI jobs when a -merge request with new or changed docs is submitted, are: - -- [`docs lint`](https://gitlab.com/gitlab-org/gitlab/-/blob/0b562014f7b71f98540e682c8d662275f0011f2f/.gitlab/ci/docs.gitlab-ci.yml#L41): - Runs several tests on the content of the docs themselves: - - [`lint-doc.sh` script](https://gitlab.com/gitlab-org/gitlab/blob/master/scripts/lint-doc.sh) - runs the following checks and linters: - - All cURL examples use the long flags (ex: `--header`, not `-H`). - - The `CHANGELOG.md` does not contain duplicate versions. - - No files in `doc/` are executable. - - No new `README.md` was added. - - [markdownlint](#markdownlint). - - [Vale](#vale). - - Nanoc tests: - - [`internal_links`](https://gitlab.com/gitlab-org/gitlab/-/blob/0b562014f7b71f98540e682c8d662275f0011f2f/.gitlab/ci/docs.gitlab-ci.yml#L58) - checks that all internal links (ex: `[link](../index.md)`) are valid. - - [`internal_anchors`](https://gitlab.com/gitlab-org/gitlab/-/blob/0b562014f7b71f98540e682c8d662275f0011f2f/.gitlab/ci/docs.gitlab-ci.yml#L60) - checks that all internal anchors (ex: `[link](../index.md#internal_anchor)`) - are valid. - - [`ui-docs-links lint`](https://gitlab.com/gitlab-org/gitlab/-/blob/0b562014f7b71f98540e682c8d662275f0011f2f/.gitlab/ci/docs.gitlab-ci.yml#L62) - checks that all links to docs from UI elements (`app/views` files, for example) - are linking to valid docs and anchors. - -### Run tests locally - -Apart from [previewing your changes locally](#previewing-the-changes-live), you can also run all lint checks -and Nanoc tests locally. - -#### Lint checks - -Lint checks are performed by the [`lint-doc.sh`](https://gitlab.com/gitlab-org/gitlab/blob/master/scripts/lint-doc.sh) -script and can be executed as follows: - -1. Navigate to the `gitlab` directory. -1. Run: - - ```shell - MD_DOC_PATH=path/to/my_doc.md scripts/lint-doc.sh - ``` - -Where `MD_DOC_PATH` points to the file or directory you would like to run lint checks for. -If you omit it completely, it will default to the `doc/` directory. -The output should be similar to: - -```plaintext -=> Linting documents at path /path/to/gitlab as <user>... -=> Checking for cURL short options... -=> Checking for CHANGELOG.md duplicate entries... -=> Checking /path/to/gitlab/doc for executable permissions... -=> Checking for new README.md files... -=> Linting markdown style... -=> Linting prose... -✔ 0 errors, 0 warnings and 0 suggestions in 1 file. -✔ Linting passed -``` - -Note that this requires you to either have the required lint tools installed on your machine, -or a working Docker installation, in which case an image with these tools pre-installed will be used. - -#### Nanoc tests - -To execute Nanoc tests locally: - -1. Navigate to the [`gitlab-docs`](https://gitlab.com/gitlab-org/gitlab-docs) directory. -1. Run: - - ```shell - # Check for broken internal links - bundle exec nanoc check internal_links - - # Check for broken external links (might take a lot of time to complete). - # This test is set to be allowed to fail and is run only in the gitlab-docs project CI - bundle exec nanoc check internal_anchors - ``` - -#### `ui-docs-links` test - -The `ui-docs-links lint` job uses `haml-lint` to test that all links to docs from -UI elements (`app/views` files, for example) are linking to valid docs and anchors. - -To run the `ui-docs-links` test locally: - -1. Open the `gitlab` directory in a terminal window. -1. Run: - - ```shell - bundle exec haml-lint -i DocumentationLinks - ``` - -If you receive an error the first time you run this test, run `bundle install`, which -installs GitLab's dependencies, and try again. - -If you don't want to install all of GitLab's dependencies to test the links, you can: - -1. Open the `gitlab` directory in a terminal window. -1. Install `haml-lint`: - - ```shell - gem install haml_lint - ``` - -1. Run: - - ```shell - haml-lint -i DocumentationLinks - ``` - -If you manually install `haml-lint` with this process, it will not update automatically -and you should make sure your version matches the version used by GitLab. - -### Local linters - -To help adhere to the [documentation style guidelines](styleguide.md), and improve the content -added to documentation, [install documentation linters](#install-linters) and -[integrate them with your code editor](#configure-editors). - -At GitLab, we mostly use: - -- [markdownlint](#markdownlint) -- [Vale](#vale) - -#### markdownlint - -[markdownlint](https://github.com/DavidAnson/markdownlint) checks that Markdown syntax follows -[certain rules](https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#rules), and is -used by the [`docs-lint` test](#testing). - -Our [Documentation Style Guide](styleguide.md#markdown) and -[Markdown Guide](https://about.gitlab.com/handbook/markdown-guide/) elaborate on which choices must -be made when selecting Markdown syntax for GitLab documentation. This tool helps catch deviations -from those guidelines. - -markdownlint configuration is found in the following projects: - -- [`gitlab`](https://gitlab.com/gitlab-org/gitlab/blob/master/.markdownlint.json) -- [`gitlab-runner`](https://gitlab.com/gitlab-org/gitlab-runner/blob/master/.markdownlint.json) -- [`omnibus-gitlab`](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/.markdownlint.json) -- [`charts`](https://gitlab.com/gitlab-org/charts/gitlab/-/blob/master/.markdownlint.json) -- [`gitlab-development-kit`](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/master/.markdownlint.json) - -This configuration is also used within build pipelines. - -You can use markdownlint: - -- [On the command line](https://github.com/igorshubovych/markdownlint-cli#markdownlint-cli--). -- [Within a code editor](#configure-editors). -- [In a `pre-commit` hook](#configure-pre-commit-hooks). - -#### Vale - -[Vale](https://errata-ai.gitbook.io/vale/) is a grammar, style, and word usage linter for the -English language. Vale's configuration is stored in the -[`.vale.ini`](https://gitlab.com/gitlab-org/gitlab/blob/master/.vale.ini) file located in the root -directory of projects. - -Vale supports creating [custom tests](https://errata-ai.github.io/vale/styles/) that extend any of -several types of checks, which we store in the `.linting/vale/styles/gitlab` directory within the -documentation directory of projects. - -Vale configuration is found in the following projects: - -- [`gitlab`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/doc/.vale/gitlab) -- [`gitlab-runner`](https://gitlab.com/gitlab-org/gitlab-runner/-/tree/master/docs/.vale/gitlab) -- [`omnibus-gitlab`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/tree/master/doc/.vale/gitlab) -- [`charts`](https://gitlab.com/gitlab-org/charts/gitlab/-/tree/master/doc/.vale/gitlab) -- [`gitlab-development-kit`](https://gitlab.com/gitlab-org/gitlab-development-kit/-/tree/master/doc/.vale/gitlab) - -This configuration is also used within build pipelines. - -You can use Vale: - -- [On the command line](https://errata-ai.gitbook.io/vale/getting-started/usage). -- [Within a code editor](#configure-editors). -- [In a `pre-commit` hook](#configure-pre-commit-hooks). Vale only reports errors in the - `pre-commit` hook (the same configuration as the CI/CD pipelines), and does not report suggestions - or warnings. - -#### Install linters - -At a minimum, install [markdownlint](#markdownlint) and [Vale](#vale) to match the checks run in -build pipelines: - -1. Install `markdownlint-cli`, using either: - - - `npm`: - - ```shell - npm install -g markdownlint-cli - ``` - - - `yarn`: - - ```shell - yarn global add markdownlint-cli - ``` - - We recommend installing the version of `markdownlint-cli` currently used in the documentation - linting [Docker image](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/master/.gitlab-ci.yml#L420). - -1. Install [`vale`](https://github.com/errata-ai/vale/releases). For example, to install using - `brew` for macOS, run: - - ```shell - brew install vale - ``` - - We recommend installing the version of Vale currently used in the documentation linting - [Docker image](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/master/.gitlab-ci.yml#L419). - -In addition to using markdownlint and Vale at the command line, these tools can be -[integrated with your code editor](#configure-editors). - -#### Configure editors - -To configure markdownlint within your editor, install one of the following as appropriate: - -- [Sublime Text](https://packagecontrol.io/packages/SublimeLinter-contrib-markdownlint) -- [Visual Studio Code](https://marketplace.visualstudio.com/items?itemName=DavidAnson.vscode-markdownlint) -- [Atom](https://atom.io/packages/linter-node-markdownlint) -- [Vim](https://github.com/dense-analysis/ale) - -To configure Vale within your editor, install one of the following as appropriate: - -- The Sublime Text [`SublimeLinter-contrib-vale` plugin](https://packagecontrol.io/packages/SublimeLinter-contrib-vale). -- The Visual Studio Code [`errata-ai.vale-server` extension](https://marketplace.visualstudio.com/items?itemName=errata-ai.vale-server). You don't need Vale Server to use the plugin. -- [Vim](https://github.com/dense-analysis/ale). - -We don't use [Vale Server](https://errata-ai.github.io/vale/#using-vale-with-a-text-editor-or-another-third-party-application). - -#### Configure pre-commit hooks - -Git [pre-commit hooks](https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks) allow Git users to -run tests or other processes before committing to a branch, with the ability to not commit to the branch if -failures occur with these tests. - -[`overcommit`](https://github.com/sds/overcommit) is a Git hooks manager, making configuring, -installing, and removing Git hooks easy. - -Sample configuration for `overcommit` is available in the -[`.overcommit.yml.example`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.overcommit.yml.example) -file for the [`gitlab`](https://gitlab.com/gitlab-org/gitlab) project. - -To set up `overcommit` for documentation linting, see -[Pre-commit static analysis](../contributing/style_guides.md#pre-commit-static-analysis). - -#### Disable Vale tests - -You can disable a specific Vale linting rule or all Vale linting rules for any portion of a -document: - -- To disable a specific rule, add a `<!-- vale gitlab.rulename = NO -->` tag before the text, and a - `<!-- vale gitlab.rulename = YES -->` tag after the text, replacing `rulename` with the filename - of a test in the - [GitLab styles](https://gitlab.com/gitlab-org/gitlab/-/tree/master/doc/.linting/vale/styles/gitlab) - directory. -- To disable all Vale linting rules, add a `<!-- vale off -->` tag before the text, and a - `<!-- vale on -->` tag after the text. - -Whenever possible, exclude only the problematic rule and line(s). - -For more information, see -[Vale's documentation](https://errata-ai.gitbook.io/vale/getting-started/markup#markup-based-configuration). +For more information on documentation testing, see [Documentation testing](testing.md) ## Danger Bot diff --git a/doc/development/documentation/restful_api_styleguide.md b/doc/development/documentation/restful_api_styleguide.md index b12578b5d98..13c6140114f 100644 --- a/doc/development/documentation/restful_api_styleguide.md +++ b/doc/development/documentation/restful_api_styleguide.md @@ -38,12 +38,16 @@ The following can be used as a template to get started: ````markdown ## Descriptive title +> Version history note. + One or two sentence description of what endpoint does. ```plaintext METHOD /endpoint ``` +Supported attributes: + | Attribute | Type | Required | Description | |:------------|:---------|:---------|:----------------------| | `attribute` | datatype | yes/no | Detailed description. | @@ -65,6 +69,9 @@ Example response: ``` ```` +Adjust the [version history note accordingly](styleguide/index.md#version-text-in-the-version-history) +to describe the GitLab release that introduced the API call. + ## Method description Use the following table headers to describe the methods. Attributes should @@ -105,8 +112,8 @@ you can use in the API documentation. CAUTION: **Caution:** Do not use information for real users, URLs, or tokens. For documentation, refer to our -relevant style guide sections on [Fake user information](styleguide.md#fake-user-information), -[Fake URLs](styleguide.md#fake-urls), and [Fake tokens](styleguide.md#fake-tokens). +relevant style guide sections on [Fake user information](styleguide/index.md#fake-user-information), +[Fake URLs](styleguide/index.md#fake-urls), and [Fake tokens](styleguide/index.md#fake-tokens). ### Simple cURL command @@ -136,14 +143,26 @@ curl --data "name=foo" --header "PRIVATE-TOKEN: <your_access_token>" "https://gi ### Post data using JSON content -NOTE: **Note:** -In this example we create a new group. Watch carefully the single and double -quotes. +This example creates a new group. Be aware of the use of single (`'`) and double +(`"`) quotes. ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" --data '{"path": "my-group", "name": "My group"}' "https://gitlab.example.com/api/v4/groups" ``` +For readability, you can also set up the `--data` by using the following format: + +```shell +curl --request POST \ +--url "https://gitlab.example.com/api/v4/groups" \ +--header "content-type: application/json" \ +--header "PRIVATE-TOKEN: <your_access_token>" \ +--data '{ + "path": "my-group", + "name": "My group" +}' +``` + ### Post data using form-data Instead of using JSON or urlencode you can use multipart/form-data which diff --git a/doc/development/documentation/site_architecture/deployment_process.md b/doc/development/documentation/site_architecture/deployment_process.md index 00cdc69d422..f101a669968 100644 --- a/doc/development/documentation/site_architecture/deployment_process.md +++ b/doc/development/documentation/site_architecture/deployment_process.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Documentation deployment process The [`dockerfiles` directory](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/dockerfiles/) diff --git a/doc/development/documentation/site_architecture/global_nav.md b/doc/development/documentation/site_architecture/global_nav.md index 9fce9b4e4b3..21b0c4b6b43 100644 --- a/doc/development/documentation/site_architecture/global_nav.md +++ b/doc/development/documentation/site_architecture/global_nav.md @@ -1,4 +1,7 @@ --- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers description: "Learn how GitLab docs' global navigation works and how to add new items." --- @@ -67,7 +70,6 @@ With these groups in mind, the following are general rules for where new items s - Other documentation belongs at the top-level, but care must be taken to not create an enormously long top-level navigation, which defeats the purpose of it. -NOTE: **Note:** Making all documentation and navigation items adhere to these principles is being progressively rolled out. @@ -114,7 +116,6 @@ for clarity. To see the improvements planned, check the [global nav epic](https://gitlab.com/groups/gitlab-com/-/epics/21). -NOTE: **Note:** **Do not** [add items](#adding-new-items) to the global nav without the consent of one of the technical writers. diff --git a/doc/development/documentation/site_architecture/index.md b/doc/development/documentation/site_architecture/index.md index 5d3af6721d1..3772746e25b 100644 --- a/doc/development/documentation/site_architecture/index.md +++ b/doc/development/documentation/site_architecture/index.md @@ -1,4 +1,7 @@ --- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers description: "Learn how GitLab's documentation website is architectured." --- @@ -227,8 +230,9 @@ for its search function. This is how it works: there's a JavaScript snippet which initiates DocSearch by using an API key and an index name (`gitlab`) that are needed for Algolia to show the results. -NOTE: **For GitLab Team Members:** -If you’re a GitLab Team Member, find credentials for the Algolia dashboard +### Algolia notes for GitLab team members + +If you’re a GitLab team member, find credentials for the Algolia dashboard in the shared [GitLab 1Password account](https://about.gitlab.com/handbook/security/#1password-for-teams). To receive weekly reports of the search usage, search the Google doc with title `Email, Slack, and GitLab Groups and Aliases`, search for `docsearch`, diff --git a/doc/development/documentation/site_architecture/release_process.md b/doc/development/documentation/site_architecture/release_process.md index d04d34ff786..547adc89a08 100644 --- a/doc/development/documentation/site_architecture/release_process.md +++ b/doc/development/documentation/site_architecture/release_process.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # GitLab Docs monthly release process When a new GitLab version is released on the 22nd, we need to create the respective @@ -9,7 +15,6 @@ Since the charts use a different version number than all the other GitLab products, we need to add a [version mapping](https://docs.gitlab.com/charts/installation/version_mappings.html): -NOTE: **Note:** The charts stable branch is not created automatically like the other products. There's an [issue to track this](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/1442). It is usually created on the 21st or the 22nd. @@ -158,7 +163,7 @@ Releasing a new version is a long process that involves many moving parts. ### `test_internal_links_and_anchors` failing on dropdown merge requests -NOTE: **Note:** +DANGER: **Deprecated:** We now pin versions in the `.gitlab-ci.yml` of the respective branch, so the steps below are deprecated. diff --git a/doc/development/documentation/structure.md b/doc/development/documentation/structure.md index e454a401a9d..3cc7f49f05e 100644 --- a/doc/development/documentation/structure.md +++ b/doc/development/documentation/structure.md @@ -10,7 +10,7 @@ description: What to include in GitLab documentation pages. Use these standards to contribute content to the GitLab documentation. Before getting started, familiarize yourself with [GitLab's Documentation guidelines](index.md) -and the [Documentation Style Guide](styleguide.md). +and the [Documentation Style Guide](styleguide/index.md). ## Components of a documentation page @@ -39,7 +39,7 @@ pre-deployment and post-deployment tasks. ## Template for new docs -Follow the [folder structure and file name guidelines](styleguide.md#folder-structure-overview) +Follow the [folder structure and file name guidelines](styleguide/index.md#folder-structure-overview) and create a new topic by using this template: ```markdown @@ -160,9 +160,9 @@ commented out to help encourage others to add to it in the future. --> Notes: -- (1): Apply the [tier badges](styleguide.md#product-badges) accordingly. +- (1): Apply the [tier badges](styleguide/index.md#product-badges) accordingly. - (2): Apply the correct format for the - [GitLab version that introduces the feature](styleguide.md#gitlab-versions-and-tiers). + [GitLab version that introduces the feature](styleguide/index.md#gitlab-versions-and-tiers). ``` ## Help and feedback section @@ -230,7 +230,6 @@ Consider the following guidelines when offering examples: - Better and best cases can be considered part of the good case(s) code block. In the same code block, precede each with comments: `# Better` and `# Best`. -NOTE: **Note:** Although the bad-then-good approach is acceptable for the GitLab development guidelines, do not use it for user documentation. For user documentation, use *Do* and *Don't*. For examples, see the [Pajamas Design System](https://design.gitlab.com/content/punctuation/). diff --git a/doc/development/documentation/styleguide.md b/doc/development/documentation/styleguide.md index 6075124ef40..a12c2740083 100644 --- a/doc/development/documentation/styleguide.md +++ b/doc/development/documentation/styleguide.md @@ -1,2063 +1,5 @@ --- -stage: none -group: Style Guide -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers -description: 'Writing styles, markup, formatting, and other standards for GitLab Documentation.' +redirect_to: 'styleguide/index.md' --- -# Documentation Style Guide - -This document defines the standards for GitLab's documentation content and -files. - -For broader information about the documentation, see the [Documentation guidelines](index.md). - -For guidelines specific to text in the GitLab interface, see the Pajamas [Content](https://design.gitlab.com/content/error-messages) section. - -For information on how to validate styles locally or by using GitLab CI/CD, see [Testing](index.md#testing). - -Use this guide for standards on grammar, formatting, word usage, and more. - -You can also view a list of [recent updates to this guide](https://gitlab.com/dashboard/merge_requests?scope=all&utf8=%E2%9C%93&state=merged&label_name[]=tw-style¬[label_name][]=docs%3A%3Afix). - -If you can't find what you need: - -- View the GitLab Handbook for [writing style guidelines](https://about.gitlab.com/handbook/communication/#writing-style-guidelines) that apply to all GitLab content. -- Refer to one of the following: - - - [Microsoft Style Guide](https://docs.microsoft.com/en-us/style-guide/). - - [Google Developer Documentation Style Guide](https://developers.google.com/style). - -If you have questions about style, mention `@tw-style` in an issue or merge request, or, if you have access to the GitLab Slack workspace, use `#docs-process`. - -## Documentation is the single source of truth (SSOT) - -### Why a single source of truth - -The documentation of GitLab products and features is the SSOT for all -information related to implementation, usage, and troubleshooting. It evolves -continuously, in keeping with new products and features, and with improvements -for clarity, accuracy, and completeness. - -This policy prevents information silos, making it easier to find information -about GitLab products. - -It also informs decisions about the kinds of content we include in our -documentation. - -### All information - -Include problem-solving actions that may address rare cases or be considered -*risky*, so long as proper context is provided in the form of fully detailed -warnings and caveats. This kind of content should be included as it could be -helpful to others and, when properly explained, its benefits outweigh the risks. -If you think you have found an exception to this rule, contact the -Technical Writing team. - -We will add all troubleshooting information to the documentation, no matter how -unlikely a user is to encounter a situation. For the [Troubleshooting sections](#troubleshooting), -people in GitLab Support can merge additions themselves. - -### All media types - -Include any media types/sources if the content is relevant to readers. You can -freely include or link presentations, diagrams, videos, and so on; no matter who -it was originally composed for, if it is helpful to any of our audiences, we can -include it. - -- If you use an image that has a separate source file (for example, a vector or - diagram format), link the image to the source file so that it may be reused or - updated by anyone. -- Do not copy and paste content from other sources unless it is a limited - quotation with the source cited. Typically it is better to either rephrase - relevant information in your own words or link out to the other source. - -### No special types - -In the software industry, it is a best practice to organize documentation in -different types. For example, [Divio recommends](https://www.divio.com/blog/documentation/): - -- Tutorials -- How-to guides -- Explanation -- Reference (for example, a glossary) - -At GitLab, we have so many product changes in our monthly releases that we can't -afford to continuously update multiple types of information. If we have multiple -types, the information will become outdated. Therefore, we have a -[single template](structure.md) for documentation. - -We currently do not distinguish specific document types, although we are open to -reconsidering this policy after the documentation has reached a future stage of -maturity and quality. If you are reading this, then despite our continuous -improvement efforts, that point hasn't been reached. - -### Link instead of summarize - -There is a temptation to summarize the information on another page. This will -cause the information to live in two places. Instead, link to the single source -of truth and explain why it is important to consume the information. - -### Organize by topic, not by type - -Beyond top-level audience-type folders (for example, `administration`), we -organize content by topic, not by type, so it can be located as easily as -possible within the single-source-of-truth (SSOT) section for the subject -matter. - -For example, do not create groupings of similar media types. For example: - -- Glossaries. -- FAQs. -- Sets of all articles or videos. - -Such grouping of content by type makes it difficult to browse for the information -you need and difficult to maintain up-to-date content. Instead, organize content -by its subject (for example, everything related to CI goes together) and -cross-link between any related content. - -### Docs-first methodology - -We employ a *documentation-first methodology* to help ensure the documentation -remains a complete and trusted resource, and to make communicating about the use -of GitLab more efficient. - -- If the answer to a question exists in documentation, share the link to the - documentation instead of rephrasing the information. -- When you encounter new information not available in GitLab’s documentation (for - example, when working on a support case or testing a feature), your first step - should be to create a merge request (MR) to add this information to the - documentation. You can then share the MR in order to communicate this - information. - -New information that would be useful toward the future usage or troubleshooting -of GitLab should not be written directly in a forum or other messaging system, -but added to a documentation MR and then referenced, as described above. Note -that among any other documentation changes, you can either: - -- Add a [Troubleshooting section](#troubleshooting) to a doc if none exists. -- Un-comment and use the placeholder Troubleshooting section included as part of - our [documentation template](structure.md#template-for-new-docs), if present. - -The more we reflexively add useful information to the documentation, the more -(and more successfully) the documentation will be used to efficiently accomplish -tasks and solve problems. - -If you have questions when considering, authoring, or editing documentation, ask -the Technical Writing team on Slack in `#docs` or in GitLab by mentioning the -writer for the applicable [DevOps stage](https://about.gitlab.com/handbook/product/product-categories/#devops-stages). -Otherwise, forge ahead with your best effort. It does not need to be perfect; -the team is happy to review and improve upon your content. Please review the -[Documentation guidelines](index.md) before you begin your first documentation MR. - -Having a knowledge base in any form that is separate from the documentation would -be against the documentation-first methodology because the content would overlap with -the documentation. - -## Markdown - -All GitLab documentation is written using [Markdown](https://en.wikipedia.org/wiki/Markdown). - -The [documentation website](https://docs.gitlab.com) uses GitLab Kramdown as its -Markdown rendering engine. For a complete Kramdown reference, see the -[GitLab Markdown Kramdown Guide](https://about.gitlab.com/handbook/markdown-guide/). - -The [`gitlab-kramdown`](https://gitlab.com/gitlab-org/gitlab_kramdown) Ruby gem -will support all [GFM markup](../../user/markdown.md) in the future. That is, -all markup supported for display in the GitLab application itself. For now, use -regular Markdown markup, following the rules in the linked style guide. - -Note that Kramdown-specific markup (for example, `{:.class}`) will not render -properly on GitLab instances under [`/help`](index.md#gitlab-help). - -### HTML in Markdown - -Hard-coded HTML is valid, although it's discouraged from being used while we -have `/help`. HTML is permitted as long as: - -- There's no equivalent markup in Markdown. -- Advanced tables are necessary. -- Special styling is required. -- Reviewed and approved by a technical writer. - -### Markdown Rules - -GitLab ensures that the Markdown used across all documentation is consistent, as -well as easy to review and maintain, by [testing documentation changes](index.md#testing) -with [markdownlint](index.md#markdownlint). This lint test fails when any -document has an issue with Markdown formatting that may cause the page to render -incorrectly within GitLab. It will also fail when a document is using -non-standard Markdown (which may render correctly, but is not the current -standard for GitLab documentation). - -#### Markdown rule `MD044/proper-names` (capitalization) - -A rule that could cause confusion is `MD044/proper-names`, as it might not be -immediately clear what caused markdownlint to fail, or how to correct the -failure. This rule checks a list of known words, listed in the `.markdownlint.json` -file in each project, to verify proper use of capitalization and backticks. -Words in backticks will be ignored by markdownlint. - -In general, product names should follow the exact capitalization of the official -names of the products, protocols, and so on. See [`.markdownlint.json`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.markdownlint.json) -for the words tested for proper capitalization in GitLab documentation. - -Some examples fail if incorrect capitalization is used: - -- MinIO (needs capital `IO`) -- NGINX (needs all capitals) -- runit (needs lowercase `r`) - -Additionally, commands, parameters, values, filenames, and so on must be -included in backticks. For example: - -- "Change the `needs` keyword in your `.gitlab.yml`..." - - `needs` is a parameter, and `.gitlab.yml` is a file, so both need backticks. - Additionally, `.gitlab.yml` will fail markdownlint without backticks as it - does not have capital G or L. -- "Run `git clone` to clone a Git repository..." - - `git clone` is a command, so it must be lowercase, while Git is the product, - so it must have a capital G. - -## Structure - -Because we want documentation to be a SSOT, we should [organize by topic, not by -type](#organize-by-topic-not-by-type). - -### Folder structure overview - -The documentation is separated by top-level audience folders [`user`](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/doc/user), -[`administration`](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/doc/administration), -and [`development`](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/doc/development) -(contributing) folders. - -Beyond that, we primarily follow the structure of the GitLab user interface or -API. - -Our goal is to have a clear hierarchical structure with meaningful URLs like -`docs.gitlab.com/user/project/merge_requests/`. With this pattern, you can -immediately tell that you are navigating to user-related documentation about -Project features; specifically about Merge Requests. Our site's paths match -those of our repository, so the clear structure also makes documentation easier to update. - -The table below shows what kind of documentation goes where. - -| Directory | What belongs here | -|:----------------------|:----------------------------------------------------------------------------------------------------------------------------------------| -| `doc/user/` | User related documentation. Anything that can be done within the GitLab user interface goes here, including usage of the `/admin` interface. | -| `doc/administration/` | Documentation that requires the user to have access to the server where GitLab is installed. The admin settings that can be accessed via GitLab's interface exist under `doc/user/admin_area/`. | -| `doc/api/` | API related documentation. | -| `doc/development/` | Documentation related to the development of GitLab, whether contributing code or documentation. Related process and style guides should go here. | -| `doc/legal/` | Legal documents about contributing to GitLab. | -| `doc/install/` | Contains instructions for installing GitLab. | -| `doc/update/` | Contains instructions for updating GitLab. | -| `doc/topics/` | Indexes per topic (`doc/topics/topic_name/index.md`): all resources for that topic. | - -### Work with directories and files - -1. When you create a new directory, always start with an `index.md` file. - Do not use another file name and *do not* create `README.md` files. -1. *Do not* use special characters and spaces, or capital letters in file - names, directory names, branch names, and anything that generates a path. -1. When creating or renaming a file or directory and it has more than one word - in its name, use underscores (`_`) instead of spaces or dashes. For example, - proper naming would be `import_project/import_from_github.md`. This applies - to both image files and Markdown files. -1. For image files, do not exceed 100KB. -1. Do not upload video files to the product repositories. - [Link or embed videos](#videos) instead. -1. There are four main directories: `user`, `administration`, `api`, and - `development`. -1. The `doc/user/` directory has five main subdirectories: `project/`, `group/`, - `profile/`, `dashboard/` and `admin_area/`. - 1. `doc/user/project/` should contain all project related documentation. - 1. `doc/user/group/` should contain all group related documentation. - 1. `doc/user/profile/` should contain all profile related documentation. - Every page you would navigate under `/profile` should have its own document, - for example, `account.md`, `applications.md`, or `emails.md`. - 1. `doc/user/dashboard/` should contain all dashboard related documentation. - 1. `doc/user/admin_area/` should contain all admin related documentation - describing what can be achieved by accessing GitLab's admin interface - (_not to be confused with `doc/administration` where server access is - required_). - 1. Every category under `/admin/application_settings/` should have its - own document located at `doc/user/admin_area/settings/`. For example, - the **Visibility and Access Controls** category should have a document - located at `doc/user/admin_area/settings/visibility_and_access_controls.md`. -1. The `doc/topics/` directory holds topic-related technical content. Create - `doc/topics/topic_name/subtopic_name/index.md` when subtopics become necessary. - General user- and admin- related documentation, should be placed accordingly. -1. The `/university/` directory is *deprecated* and the majority of its documentation - has been moved. - -If you are unsure where to place a document or a content addition, this should -not stop you from authoring and contributing. You can use your best judgment and -then ask the reviewer of your MR to confirm your decision, and/or ask a -technical writer at any stage in the process. The technical writing team will -review all documentation changes, regardless, and can move content if there is a -better place for it. - -### Avoid duplication - -Do not include the same information in multiple places. -[Link to a single source of truth instead.](#link-instead-of-summarize) - -### References across documents - -- Give each folder an `index.md` page that introduces the topic, introduces the - pages within, and links to the pages within (including to the index pages of - any next-level subpaths). -- To ensure discoverability, ensure each new or renamed doc is linked from its - higher-level index page and other related pages. -- When making reference to other GitLab products and features, link to their - respective documentation, at least on first mention. -- When making reference to third-party products or technologies, link out to - their external sites, documentation, and resources. - -### Structure within documents - -- Include any and all applicable subsections as described on the - [structure and template](structure.md) page. -- Structure content in alphabetical order in tables, lists, and so on, unless - there's a logical reason not to (for example, when mirroring the user - interface or an otherwise ordered sequence). - -## Language - -GitLab documentation should be clear and easy to understand. - -- Be clear, concise, and stick to the goal of the documentation. -- Write in US English with US grammar. (Tested in [`British.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/British.yml).) -- Use [inclusive language](#inclusive-language). - -### Point of view - -In most cases, it’s appropriate to use the second-person (you, yours) point of -view, because it’s friendly and easy to understand. (Tested in -[`FirstPerson.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/FirstPerson.yml).) - -<!-- How do we harmonize the second person in Pajamas with our first person plural in our doc guide? --> - -### Capitalization - -#### Headings - -Use sentence case. For example: - -- `# Use variables to configure pipelines` -- `## Use the To-Do List` - -#### UI text - -When referring to specific user interface text, like a button label or menu -item, use the same capitalization that is displayed in the user interface. -Standards for this content are listed in the [Pajamas Design System Content section](https://design.gitlab.com/content/punctuation/) -and typically match what is called for in this Documentation Style Guide. - -If you think there is a mistake in the way the user interface text is styled, -create an issue or an MR to propose a change to the user interface text. - -#### Feature names - -- *Feature names are typically lowercase*, like those describing actions and - types of objects in GitLab. For example: - - epics - - issues - - issue weights - - merge requests - - milestones - - reorder issues - - runner, runners, shared runners - - a to-do item, to dos -- *Some features are capitalized*, typically nouns naming GitLab-specific - capabilities or tools. For example: - - GitLab CI/CD - - Repository Mirroring - - Value Stream Analytics - - the To-Do List - - the Web IDE - - Geo - - GitLab Runner (see [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/233529) for details) - -Document any exceptions in this style guide. If you're not sure, ask a GitLab -Technical Writer so that they can help decide and document the result. - -Do not match the capitalization of terms or phrases on the [Features page](https://about.gitlab.com/features/) -or [features.yml](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/features.yml) -by default. - -#### Other terms - -Capitalize names of: - -- GitLab [product tiers](https://about.gitlab.com/pricing/). For example, - GitLab Core and GitLab Ultimate. (Tested in [`BadgeCapitalization.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/BadgeCapitalization.yml).) -- Third-party organizations, software, and products. For example, Prometheus, - Kubernetes, Git, and The Linux Foundation. -- Methods or methodologies. For example, Continuous Integration, - Continuous Deployment, Scrum, and Agile. (Tested in [`.markdownlint.json`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.markdownlint.json).) - -Follow the capitalization style listed at the [authoritative source](#links-to-external-documentation) -for the entity, which may use non-standard case styles. For example: GitLab and -npm. - -Use forms of *sign in*, instead of *log in* or *login*. For example: - -- Sign in to GitLab. -- Open the sign-in page. - -Exceptions to this rule include the concept of *single sign-on* and -references to user interface elements. For example: - -- To sign in to product X, enter your credentials, and then select **Log in**. - -### Inclusive language - -We strive to create documentation that is inclusive. This section includes -guidance and examples in the following categories: - -- [Gender-specific wording](#avoid-gender-specific-wording). - (Tested in [`InclusionGender.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/InclusionGender.yml).) -- [Ableist language](#avoid-ableist-language). - (Tested in [`InclusionAbleism.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/InclusionAbleism.yml).) -- [Cultural sensitivity](#culturally-sensitive-language). - (Tested in [`InclusionCultural.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/InclusionCultural.yml).) - -We write our developer documentation with inclusivity and diversity in mind. This -page is not an exhaustive reference, but describes some general guidelines and -examples that illustrate some best practices to follow. - -#### Avoid gender-specific wording - -When possible, use gender-neutral pronouns. For example, you can use a singular -[they](https://developers.google.com/style/pronouns#gender-neutral-pronouns) as -a gender-neutral pronoun. - -Avoid the use of gender-specific pronouns, unless referring to a specific person. - -<!-- vale gitlab.InclusionGender = NO --> - -| Use | Avoid | -|-----------------------------------|---------------------------------| -| People, humanity | Mankind | -| GitLab Team Members | Manpower | -| You can install; They can install | He can install; She can install | - -<!-- vale gitlab.InclusionGender = YES --> - -If you need to set up [Fake user information](#fake-user-information), use -diverse or non-gendered names with common surnames. - -#### Avoid ableist language - -Avoid terms that are also used in negative stereotypes for different groups. - -<!-- vale gitlab.InclusionAbleism = NO --> - -| Use | Avoid | -|------------------------|----------------------| -| Check for completeness | Sanity check | -| Uncertain outliers | Crazy outliers | -| Slows the service | Cripples the service | -| Placeholder variable | Dummy variable | -| Active/Inactive | Enabled/Disabled | -| On/Off | Enabled/Disabled | - -<!-- vale gitlab.InclusionAbleism = YES --> - -Credit: [Avoid ableist language](https://developers.google.com/style/inclusive-documentation#ableist-language) -in the Google Developer Style Guide. - -#### Culturally sensitive language - -Avoid terms that reflect negative cultural stereotypes and history. In most -cases, you can replace terms such as `master` and `slave` with terms that are -more precise and functional, such as `primary` and `secondary`. - -<!-- vale gitlab.InclusionCultural = NO --> - -| Use | Avoid | -|----------------------|-----------------------| -| Primary / secondary | Master / slave | -| Allowlist / denylist | Blacklist / whitelist | - -<!-- vale gitlab.InclusionCultural = YES --> - -For more information see the following [Internet Draft specification](https://tools.ietf.org/html/draft-knodel-terminology-02). - -### Fake user information - -You may need to include user information in entries such as a REST call or user profile. -**Do not** use real user information or email addresses in GitLab documentation. For email -addresses and names, do use: - -- **Email addresses**: Use an email address ending in `example.com`. -- **Names**: Use strings like `example_username`. Alternatively, use diverse or - non-gendered names with common surnames, such as `Sidney Jones`, `Zhang Wei`, - or `Alex Garcia`. - -### Fake URLs - -When including sample URLs in the documentation, use: - -- `example.com` when the domain name is generic. -- `gitlab.example.com` when referring to self-managed instances of GitLab. - -### Fake tokens - -There may be times where a token is needed to demonstrate an API call using -cURL or a variable used in CI. It is strongly advised not to use real tokens in -documentation even if the probability of a token being exploited is low. - -You can use the following fake tokens as examples: - -| Token type | Token value | -|:----------------------|:-------------------------------------------------------------------| -| Private user token | `<your_access_token>` | -| Personal access token | `n671WNGecHugsdEDPsyo` | -| Application ID | `2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6` | -| Application secret | `04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df` | -| CI/CD variable | `Li8j-mLUVA3eZYjPfd_H` | -| Specific runner token | `yrnZW46BrtBFqM7xDzE7dddd` | -| Shared runner token | `6Vk7ZsosqQyfreAxXTZr` | -| Trigger token | `be20d8dcc028677c931e04f3871a9b` | -| Webhook secret token | `6XhDroRcYPM5by_h-HLY` | -| Health check token | `Tu7BgjR9qeZTEyRzGG2P` | -| Request profile token | `7VgpS4Ax5utVD2esNstz` | - -### Language to avoid - -When creating documentation, limit or avoid the use of the following verb -tenses, words, and phrases: - -- Avoid jargon when possible, and when not possible, define the term or - [link to a definition](#links-to-external-documentation). -- Avoid uncommon words when a more-common alternative is possible, ensuring that - content is accessible to more readers. -- Don't write in the first person singular. - (Tested in [`FirstPerson.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/FirstPerson.yml).) - - Instead of "I" or "me," use "we," "you," "us," or "one." - - When possible, stay user focused by writing in the second person ("you" or - the imperative). -- Don't overuse "that". In many cases, you can remove "that" from a sentence - and improve readability. -- Avoid use of the future tense: - - Instead of "after you execute this command, GitLab will display the - result", use "after you execute this command, GitLab displays the result". - - Only use the future tense to convey when the action or result will actually - occur at a future time. -- Don't use slashes to clump different words together or as a replacement for - the word "or": - - Instead of "and/or," consider using "or," or use another sensible - construction. - - Other examples include "clone/fetch," author/assignee," and - "namespace/repository name." Break apart any such instances in an - appropriate way. - - Exceptions to this rule include commonly accepted technical terms, such as - CI/CD and TCP/IP. -- <!-- vale gitlab.LatinTerms = NO --> - We discourage use of Latin abbreviations, such as "e.g.," "i.e.," or "etc.," - as even native users of English might misunderstand them. - (Tested in [`LatinTerms.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/LatinTerms.yml).) - - Instead of "i.e.," use "that is." - - Instead of "e.g.," use "for example," "such as," "for instance," or "like." - - Instead of "etc.," either use "and so on" or consider editing it out, since - it can be vague. - <!-- vale gitlab.LatinTerms = YES --> -- Avoid using the word *currently* when talking about the product or its - features. The documentation describes the product as it is, and not as it - will be at some indeterminate point in the future. -- Avoid the using the word *scalability* with increasing GitLab's performance - for additional users. Using the words *scale* or *scaling* in other cases is - acceptable, but references to increasing GitLab's performance for additional - users should direct readers to the GitLab - [reference architectures](../../administration/reference_architectures/index.md) - page. -- Avoid all forms of the phrases *high availability* and *HA*, and instead - direct readers to the GitLab [reference architectures](../../administration/reference_architectures/index.md) - for information about configuring GitLab to have the performance needed for - additional users over time. -- Don't use profanity or obscenities. Doing so may negatively affect other - users and contributors, which is contrary to GitLab's value of - [Diversity, Inclusion, and Belonging](https://about.gitlab.com/handbook/values/#diversity-inclusion). -- Avoid the use of [racially-insensitive terminology or phrases](https://www.marketplace.org/2020/06/17/tech-companies-update-language-to-avoid-offensive-terms/). For example: - - Use *primary* and *secondary* for database and server relationships. - - Use *allowlist* and *denylist* to describe access control lists. -- Avoid the word *please*. For details, see [the Microsoft style guide](https://docs.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/p/please). - -### Word usage clarifications - -- Don't use "may" and "might" interchangeably: - - Use "might" to indicate the probability of something occurring. "If you - skip this step, the import process might fail." - - Use "may" to indicate giving permission for someone to do something, or - consider using "can" instead. "You may select either option on this - screen." Or, "You can select either option on this screen." - -### Contractions - -Contractions are encouraged, and can create a friendly and informal tone, -especially in tutorials, instructional documentation, and -[user interfaces](https://design.gitlab.com/content/punctuation/#contractions). - -Some contractions, however, should be avoided: - -- Do not use contractions with a proper noun and a verb. For example: - - | Do | Don't | - |------------------------------------------|-----------------------------------------| - | GitLab is creating X. | GitLab's creating X. | - -- Do not use contractions when you need to emphasize a negative. For example: - - | Do | Don't | - |------------------------------------------|-----------------------------------------| - | Do *not* install X with Y. | *Don't* install X with Y. | - -- Do not use contractions in reference documentation. For example: - - | Do | Don't | - |------------------------------------------|-----------------------------------------| - | Do *not* set a limit greater than 1000. | *Don't* set a limit greater than 1000. | - | For `parameter1`, the default is 10. | For `parameter1`, the default's 10. | - -- Avoid contractions in error messages. Examples: - - | Do | Don't | - |------------------------------------------|-----------------------------------------| - | Requests to localhost are not allowed. | Requests to localhost aren't allowed. | - | Specified URL cannot be used. | Specified URL can't be used. | - -## Text - -- [Write in Markdown](#markdown). -- Splitting long lines (preferably up to 100 characters) can make it easier to - provide feedback on small chunks of text. -- Insert an empty line for new paragraphs. -- Insert an empty line between different markups (for example, after every - paragraph, header, list, and so on). Example: - - ```markdown - ## Header - - Paragraph. - - - List item 1 - - List item 2 - ``` - -### Emphasis - -- Use double asterisks (`**`) to mark a word or text in bold (`**bold**`). -- Use underscore (`_`) for text in italics (`_italic_`). -- Use greater than (`>`) for blockquotes. - -### Punctuation - -Review the general punctuation rules for the GitLab documentation in the -following table. Check specific punctuation rules for [lists](#lists) below. -Additional examples are available in the [Pajamas guide for punctuation](https://design.gitlab.com/content/punctuation/). - -| Rule | Example | -|------------------------------------------------------------------|--------------------------------------------------------| -| Always end full sentences with a period. | _For a complete overview, read through this document._ | -| Always add a space after a period when beginning a new sentence. | _For a complete overview, check this doc. For other references, check out this guide._ | -| Do not use double spaces. (Tested in [`SentenceSpacing.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/SentenceSpacing.yml).) | --- | -| Do not use tabs for indentation. Use spaces instead. You can configure your code editor to output spaces instead of tabs when pressing the tab key. | --- | -| Use serial commas ("Oxford commas") before the final 'and/or' in a list. (Tested in [`OxfordComma.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/OxfordComma.yml).) | _You can create new issues, merge requests, and milestones._ | -| Always add a space before and after dashes when using it in a sentence (for replacing a comma, for example). | _You should try this - or not._ | -| Always use lowercase after a colon. | _Related Issues: a way to create a relationship between issues._ | - -### Placeholder text - -Often in examples, a writer will provide a command or configuration that -uses values specific to the reader. - -In these cases, use [`<` and `>`](https://en.wikipedia.org/wiki/Usage_message#Pattern) -to call out where a reader must replace text with their own value. - -For example: - -```shell -cp <your_source_directory> <your_destination_directory> -``` - -### Keyboard commands - -Use the HTML `<kbd>` tag when referring to keystroke presses. For example: - -```plaintext -To stop the command, press <kbd>Ctrl</kbd>+<kbd>C</kbd>. -``` - -When the docs are generated, the output is: - -To stop the command, press <kbd>Ctrl</kbd>+<kbd>C</kbd>. - -## Lists - -- Always start list items with a capital letter, unless they are parameters or - commands that are in backticks, or similar. -- Always leave a blank line before and after a list. -- Begin a line with spaces (not tabs) to denote a [nested sub-item](#nesting-inside-a-list-item). - -### Ordered vs. unordered lists - -Only use ordered lists when their items describe a sequence of steps to follow. - -Do: - -```markdown -These are the steps to do something: - -1. First, do the first step. -1. Then, do the next step. -1. Finally, do the last step. -``` - -Don't: - -```markdown -This is a list of available features: - -1. Feature 1 -1. Feature 2 -1. Feature 3 -``` - -### Markup - -- Use dashes (`-`) for unordered lists instead of asterisks (`*`). -- Prefix `1.` to every item in an ordered list. When rendered, the list items - will appear with sequential numbering automatically. - -### Punctuation - -- Do not add commas (`,`) or semicolons (`;`) to the end of list items. -- Only add periods to the end of a list item if the item consists of a complete - sentence. The [definition of full sentence](https://www2.le.ac.uk/offices/ld/all-resources/writing/grammar/grammar-guides/sentence) - is: _"a complete sentence always contains a verb, expresses a complete idea, and makes sense standing alone"_. -- Be consistent throughout the list: if the majority of the items do not end in - a period, do not end any of the items in a period, even if they consist of a - complete sentence. The opposite is also valid: if the majority of the items - end with a period, end all with a period. -- Separate list items from explanatory text with a colon (`:`). For example: - - ```markdown - The list is as follows: - - - First item: this explains the first item. - - Second item: this explains the second item. - ``` - -**Examples:** - -Do: - -- First list item -- Second list item -- Third list item - -Don't: - -- First list item -- Second list item -- Third list item. - -Do: - -- Let's say this is a complete sentence. -- Let's say this is also a complete sentence. -- Not a complete sentence. - -Don't (vary use of periods; majority rules): - -- Let's say this is a complete sentence. -- Let's say this is also a complete sentence. -- Not a complete sentence - -### Nesting inside a list item - -It's possible to nest items under a list item, so that they render with the same -indentation as the list item. This can be done with: - -- [Code blocks](#code-blocks) -- [Blockquotes](#blockquotes) -- [Alert boxes](#alert-boxes) -- [Images](#images) - -Items nested in lists should always align with the first character of the list -item. In unordered lists (using `-`), this means two spaces for each level of -indentation: - -````markdown -- Unordered list item 1 - - A line nested using 2 spaces to align with the `U` above. - -- Unordered list item 2 - - > A quote block that will nest - > inside list item 2. - -- Unordered list item 3 - - ```plaintext - a codeblock that will next inside list item 3 - ``` - -- Unordered list item 4 - -  -```` - -For ordered lists, use three spaces for each level of indentation: - -````markdown -1. Ordered list item 1 - - A line nested using 3 spaces to align with the `O` above. - -1. Ordered list item 2 - - > A quote block that will nest - > inside list item 2. - -1. Ordered list item 3 - - ```plaintext - a codeblock that will next inside list item 3 - ``` - -1. Ordered list item 4 - -  -```` - -You can nest full lists inside other lists using the same rules as above. If you -want to mix types, that is also possible, as long as you don't mix items at the -same level: - -```markdown -1. Ordered list item one. -1. Ordered list item two. - - Nested unordered list item one. - - Nested unordered list item two. -1. Ordered list item three. - -- Unordered list item one. -- Unordered list item two. - 1. Nested ordered list item one. - 1. Nested ordered list item two. -- Unordered list item three. -``` - -## Tables - -Tables should be used to describe complex information in a straightforward -manner. Note that in many cases, an unordered list is sufficient to describe a -list of items with a single, simple description per item. But, if you have data -that is best described by a matrix, tables are the best choice for use. - -### Creation guidelines - -Due to accessibility and scannability requirements, tables should not have any -empty cells. If there is no otherwise meaningful value for a cell, consider entering -*N/A* (for 'not applicable') or *none*. - -To help tables be easier to maintain, consider adding additional spaces to the -column widths to make them consistent. For example: - -```markdown -| App name | Description | Requirements | -|:---------|:---------------------|:---------------| -| App 1 | Description text 1. | Requirements 1 | -| App 2 | Description text 2. | None | -``` - -Consider installing a plugin or extension in your editor for formatting tables: - -- [Markdown Table Prettifier](https://marketplace.visualstudio.com/items?itemName=darkriszty.markdown-table-prettify) for Visual Studio Code -- [Markdown Table Formatter](https://packagecontrol.io/packages/Markdown%20Table%20Formatter) for Sublime Text -- [Markdown Table Formatter](https://atom.io/packages/markdown-table-formatter) for Atom - -### Feature tables - -When creating tables of lists of features (such as whether or not features are -available to certain roles on the [Permissions](../../user/permissions.md#project-members-permissions) -page), use the following phrases (based on the SVG icons): - -| Option | Markdown | Displayed result | -|--------|--------------------------|------------------------| -| No | `**{dotted-circle}** No` | **{dotted-circle}** No | -| Yes | `**{check-circle}** Yes` | **{check-circle}** Yes | - -## Quotes - -Valid for Markdown content only, not for front matter entries: - -- Standard quotes: double quotes (`"`). Example: "This is wrapped in double - quotes". -- Quote within a quote: double quotes (`"`) wrap single quotes (`'`). Example: - "I am 'quoting' something within a quote". - -For other punctuation rules, please refer to the -[GitLab UX guide](https://design.gitlab.com/content/punctuation/). - -## Headings - -- Add **only one H1** in each document, by adding `#` at the beginning of - it (when using Markdown). The `h1` will be the document `<title>`. -- Start with an `h2` (`##`), and respect the order `h2` > `h3` > `h4` > `h5` > `h6`. - Never skip the hierarchy level, such as `h2` > `h4` -- Avoid putting numbers in headings. Numbers shift, hence documentation anchor - links shift too, which eventually leads to dead links. If you think it is - compelling to add numbers in headings, make sure to at least discuss it with - someone in the Merge Request. -- [Avoid using symbols and special characters](https://gitlab.com/gitlab-org/gitlab-docs/-/issues/84) - in headers. Whenever possible, they should be plain and short text. -- When possible, avoid including words that might change in the future. Changing - a heading changes its anchor URL, which affects other linked pages. -- When introducing a new document, be careful for the headings to be - grammatically and syntactically correct. Mention an [assigned technical writer (TW)](https://about.gitlab.com/handbook/product/product-categories/) - for review. - This is to ensure that no document with wrong heading is going live without an - audit, thus preventing dead links and redirection issues when corrected. -- Leave exactly one blank line before and after a heading. -- Do not use links in headings. -- Add the corresponding [product badge](#product-badges) according to the tier the - feature belongs. -- Our documentation site search engine prioritizes words used in headings and - subheadings. Make you subheading titles clear, descriptive, and complete to help - users find the right example, as shown in the section on [heading titles](#heading-titles). -- See [Capitalization](#capitalization) for guidelines on capitalizing headings. - -### Heading titles - -Keep heading titles clear and direct. Make every word count. To accommodate -search engine optimization (SEO), use the imperative, where possible. - -| Do | Don't | -|:--------------------------------------|:------------------------------------------------------------| -| Configure GDK | Configuring GDK | -| GitLab Release and Maintenance Policy | This section covers GitLab's Release and Maintenance Policy | -| Backport to older releases | Backporting to older releases | -| GitLab Pages examples | Examples | - -For guidelines on capitalizing headings, see the section on [capitalization](#capitalization). - -NOTE: **Note:** -If you change an existing title, be careful. These changes might affect not -only [links](#anchor-links) within the page, but might also affect links to the -GitLab documentation from both the GitLab application and external sites. - -### Anchor links - -Headings generate anchor links automatically when rendered. `## This is an example` -generates the anchor `#this-is-an-example`. - -NOTE: **Note:** -[Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/39717) in GitLab 13.4, [product badges](#product-badges) used in headings aren't included in the -generated anchor links. For example, when you link to -`## This is an example **(CORE)**`, use the anchor `#this-is-an-example`. - -Keep in mind that the GitLab user interface links to many documentation pages -and anchor links to take the user to the right spot. Therefore, when you change -a heading, search `doc/*`, `app/views/*`, and `ee/app/views/*` for the old -anchor to make sure you're not breaking an anchor linked from other -documentation nor from the GitLab user interface. If you find the old anchor, be -sure to replace it with the new one. - -Important: - -- Avoid crosslinking documentation to headings unless you need to link to a - specific section of the document. This will avoid breaking anchors in the - future in case the heading is changed. -- If possible, avoid changing headings since they're not only linked internally. - There are various links to GitLab documentation on the internet, such as - tutorials, presentations, StackOverflow posts, and other sources. -- Do not link to `h1` headings. - -Note that, with Kramdown, it is possible to add a custom ID to an HTML element -with Markdown markup, but they **do not** work in GitLab's `/help`. Therefore, -do not use this option until further notice. - -## Links - -Links are important in GitLab documentation. They allow you to [link instead of -summarizing](#link-instead-of-summarize) to help preserve a [single source of truth](#why-a-single-source-of-truth) -within GitLab documentation. - -We include guidance for links in the following categories: - -- How to set up [anchor links](#anchor-links) for headings. -- How to set up [criteria](#basic-link-criteria) for configuring a link. -- What to set up when [linking to a `help`](../documentation/index.md#linking-to-help) - page. -- How to set up [links to internal documentation](#links-to-internal-documentation) - for cross-references. -- How to set up [links to external documentation](#links-to-external-documentation) - for authoritative sources. -- When to use [links requiring permissions](#links-requiring-permissions). -- How to set up a [link to a video](#link-to-video). -- How to [include links with version text](#text-for-documentation-requiring-version-text). -- How to [link to specific lines of code](#link-to-specific-lines-of-code) - -### Basic link criteria - -- Use inline link Markdown markup `[Text](https://example.com)`. - It's easier to read, review, and maintain. *Do not* use `[Text][identifier]`. - -- Use [meaningful anchor texts](https://www.futurehosting.com/blog/links-should-have-meaningful-anchor-text-heres-why/). - For example, instead of writing something like `Read more about GitLab Issue Boards [here](LINK)`, - write `Read more about [GitLab Issue Boards](LINK)`. - -### Links to internal documentation - -NOTE: **Note:** -_Internal_ refers to documentation in the same project. When linking to -documentation in separate projects (for example, linking to Omnibus documentation -from GitLab documentation), you must use absolute URLs. - -Do not use absolute URLs like `https://docs.gitlab.com/ee/index.html` to -crosslink to other documentation within the same project. Use relative links to -the file, like `../index.md`. (These are converted to HTML when the site is -rendered.) - -Relative linking enables crosslinks to work: - -- in Review Apps, local previews, and `/help`. -- when working on the documentation locally, so you can verify that they work as - early as possible in the process. -- within the GitLab user interface when browsing doc files in their respective - repositories. For example, the links displayed at - `https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/README.md`. - -To link to internal documentation: - -- Use relative links to Markdown files in the same repository. -- Do not use absolute URLs or URLs from `docs.gitlab.com`. -- Use `../` to navigate to higher-level directories. -- Do not link relative to root. For example, `/ee/user/gitlab_com/index.md`. - - Don't: - - - `https://docs.gitlab.com/ee/administration/geo/replication/troubleshooting.html` - - `/ee/administration/geo/replication/troubleshooting.md` - - Do: `../../geo/replication/troubleshooting.md` - -- Always add the file name `file.md` at the end of the link with the `.md` - extension, not `.html`. - - Don't: - - - `../../merge_requests/` - - `../../issues/tags.html` - - `../../issues/tags.html#stages` - - Do: - - - `../../merge_requests/index.md` - - `../../issues/tags.md` - - `../../issues/tags.md#stages` - -NOTE: **Note:** -Using the Markdown extension is necessary for the [`/help`](index.md#gitlab-help) -section of GitLab. - -### Links to external documentation - -When describing interactions with external software, it's often helpful to -include links to external documentation. When possible, make sure that you're -linking to an [**authoritative** source](#authoritative-sources). For example, -if you're describing a feature in Microsoft's Active Directory, include a link -to official Microsoft documentation. - -### Authoritative sources - -When citing external information, use sources that are written by the people who -created the item or product in question. These sources are the most likely to be -accurate and remain up to date. - -Examples of authoritative sources include: - -- Specifications, such as a [Request for Comments](https://www.ietf.org/standards/rfcs/) - document from the Internet Engineering Task Force. -- Official documentation for a product. For example, if you're setting up an - interface with the Google OAuth 2 authorization server, include a link to - Google's documentation. -- Official documentation for a project. For example, if you're citing NodeJS - functionality, refer directly to [NodeJS documentation](https://nodejs.org/en/docs/). -- Books from an authoritative publisher. - -Examples of sources to avoid include: - -- Personal blog posts. -- Wikipedia. -- Non-trustworthy articles. -- Discussions on forums such as Stack Overflow. -- Documentation from a company that describes another company's product. - -While many of these sources to avoid can help you learn skills and or features, -they can become obsolete quickly. Nobody is obliged to maintain any of these -sites. Therefore, we should avoid using them as reference literature. - -NOTE: **Note:** -Non-authoritative sources are acceptable only if there is no equivalent -authoritative source. Even then, focus on non-authoritative sources that are -extensively cited or peer-reviewed. - -### Links requiring permissions - -Don't link directly to: - -- [Confidential issues](../../user/project/issues/confidential_issues.md). -- Project features that require [special permissions](../../user/permissions.md) - to view. - -These will fail for: - -- Those without sufficient permissions. -- Automated link checkers. - -Instead: - -- To reduce confusion, mention in the text that the information is either: - - Contained in a confidential issue. - - Requires special permission to a project to view. -- Provide a link in back ticks (`` ` ``) so that those with access to the issue - can easily navigate to it. - -Example: - -```markdown -For more information, see the [confidential issue](../../user/project/issues/confidential_issues.md) `https://gitlab.com/gitlab-org/gitlab-foss/-/issues/<issue_number>`. -``` - -### Link to specific lines of code - -When linking to specific lines within a file, link to a commit instead of to the -branch. Lines of code change through time, therefore, linking to a line by using -the commit link ensures the user lands on the line you're referring to. The -**Permalink** button, which is available when viewing a file within a project, -makes it easy to generate a link to the most recent commit of the given file. - -- *Do:* `[link to line 3](https://gitlab.com/gitlab-org/gitlab/-/blob/11f17c56d8b7f0b752562d78a4298a3a95b5ce66/.gitlab/issue_templates/Feature%20proposal.md#L3)` -- *Don't:* `[link to line 3](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/Feature%20proposal.md#L3).` - -If that linked expression is no longer in that line of the file due to additional -commits, you can still search the file for that query. In this case, update the -document to ensure it links to the most recent version of the file. - -## Navigation - -When documenting navigation through the user interface: - -- Use the exact wording as shown in the UI, including any capital letters as-is. -- Use bold text for navigation items and the char "greater than" (`>`) as a - separator. For example: `Navigate to your project's **Settings > CI/CD**`. -- If there are any expandable menus, make sure to mention that the user needs to - expand the tab to find the settings you're referring to. For example: - `Navigate to your project's **Settings > CI/CD** and expand **General pipelines**`. - -### Navigational elements - -Use the following terms when referring to the main GitLab user interface -elements: - -- **Top menu**: This is the top menu that spans the width of the user interface. - It includes the GitLab logo, search field, counters, and the user's avatar. -- **Left sidebar**: This is the navigation sidebar on the left of the user - interface, specific to the project or group. -- **Right sidebar**: This is the navigation sidebar on the right of the user - interface, specific to the open issue, merge request, or epic. - -## Images - -Images, including screenshots, can help a reader better understand a concept. -However, they can be hard to maintain, and should be used sparingly. - -Before including an image in the documentation, ensure it provides value to the -reader. - -### Capture the image - -Use images to help the reader understand where they are in a process, or how -they need to interact with the application. - -When you take screenshots: - -- *Capture the most relevant area of the page.* Don't include unnecessary white - space or areas of the page that don't help illustrate the point. The left - sidebar of the GitLab user interface can change, so don't include the sidebar - if it's not necessary. -- *Keep it small.* If you don't need to show the full width of the screen, don't. - A value of 1000 pixels is a good maximum width for your screenshot image. -- *Be consistent.* Coordinate screenshots with the other screenshots already on - a documentation page. For example, if other screenshots include the left - sidebar, include the sidebar in all screenshots. - -### Save the image - -- Save the image with a lowercase file name that is descriptive of the feature - or concept in the image. If the image is of the GitLab interface, append the - GitLab version to the file name, based on the following format: - `image_name_vX_Y.png`. For example, for a screenshot taken from the pipelines - page of GitLab 11.1, a valid name is `pipelines_v11_1.png`. If you're adding an - illustration that doesn't include parts of the user interface, add the release - number corresponding to the release the image was added to; for an MR added to - 11.1's milestone, a valid name for an illustration is `devops_diagram_v11_1.png`. -- Place images in a separate directory named `img/` in the same directory where - the `.md` document that you're working on is located. -- Consider using PNG images instead of JPEG. -- [Compress all PNG images](#compress-images). -- Compress gifs with <https://ezgif.com/optimize> or similar tool. -- Images should be used (only when necessary) to _illustrate_ the description - of a process, not to _replace_ it. -- Max image size: 100KB (gifs included). -- See also how to link and embed [videos](#videos) to illustrate the - documentation. - -### Add the image link to content - -The Markdown code for including an image in a document is: -`` - -The image description is the alt text for the rendered image on the -documentation site. For accessibility and SEO, use [descriptions](https://webaim.org/techniques/alttext/) -that: - -- Are accurate, succinct, and unique. -- Don't use *image of …* or *graphic of…* to describe the image. - -### Remove image shadow - -All images displayed on the [GitLab documentation site](https://docs.gitlab.com) -have a box shadow by default. To remove the box shadow, use the image class -`.image-noshadow` applied directly to an HTML `img` tag: - -```html -<img src="path/to/image.jpg" alt="Alt text (required)" class="image-noshadow"> -``` - -### Compress images - -You should always compress any new images you add to the documentation. One -known tool is [`pngquant`](https://pngquant.org/), which is cross-platform and -open source. Install it by visiting the official website and following the -instructions for your OS. - -GitLab has a [Rake task](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/tasks/pngquant.rake) -that you can use to automate the process. In the root directory of your local -copy of `https://gitlab.com/gitlab-org/gitlab`, run in a terminal: - -- Before compressing, if you want, check that all documentation PNG images have - been compressed: - - ```shell - bundle exec rake pngquant:lint - ``` - -- Compress all documentation PNG images using `pngquant`: - - ```shell - bundle exec rake pngquant:compress - ``` - -The only caveat is that the task runs on all images under `doc/`, not only the -ones you might have included in a merge request. In that case, you can run the -compress task and only commit the images that are relevant to your merge -request. - -## Videos - -Adding GitLab's existing YouTube video tutorials to the documentation is highly -encouraged, unless the video is outdated. Videos should not replace -documentation, but complement or illustrate it. If content in a video is -fundamental to a feature and its key use cases, but this is not adequately -covered in the documentation, add this detail to the documentation text or -create an issue to review the video and do so. - -Do not upload videos to the product repositories. [Link](#link-to-video) or -[embed](#embed-videos) them instead. - -### Link to video - -To link out to a video, include a YouTube icon so that readers can scan the page -for videos before reading: - -```markdown -<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -For an overview, see [Video Title](link-to-video). -``` - -You can link any up-to-date video that is useful to the GitLab user. - -### Embed videos - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-docs/-/merge_requests/472) in GitLab 12.1. - -The [GitLab documentation site](https://docs.gitlab.com) supports embedded -videos. - -You can only embed videos from [GitLab's official YouTube account](https://www.youtube.com/channel/UCnMGQ8QHMAnVIsI3xJrihhg). -For videos from other sources, [link](#link-to-video) them instead. - -In most cases, it is better to [link to video](#link-to-video) instead, because -an embed takes up a lot of space on the page and can be distracting to readers. - -To embed a video, follow the instructions below and make sure you have your MR -reviewed and approved by a technical writer. - -1. Copy the code below and paste it into your Markdown file. Leave a blank line - above and below it. Do *not* edit the code (don't remove or add any spaces). -1. In YouTube, visit the video URL you want to display. Copy the regular URL - from your browser (`https://www.youtube.com/watch?v=VIDEO-ID`) and replace - the video title and link in the line under `<div class="video-fallback">`. -1. In YouTube, select **Share**, and then select **Embed**. -1. Copy the `<iframe>` source (`src`) **URL only** - (`https://www.youtube.com/embed/VIDEO-ID`), - and paste it, replacing the content of the `src` field in the - `iframe` tag. - -```html -leave a blank line here -<div class="video-fallback"> - See the video: <a href="https://www.youtube.com/watch?v=MqL6BMOySIQ">Video title</a>. -</div> -<figure class="video-container"> - <iframe src="https://www.youtube.com/embed/MqL6BMOySIQ" frameborder="0" allowfullscreen="true"> </iframe> -</figure> -leave a blank line here -``` - -This is how it renders on the GitLab documentation site: - -<div class="video-fallback"> - See the video: <a href="https://www.youtube.com/watch?v=enMumwvLAug">What is GitLab</a>. -</div> -<figure class="video-container"> - <iframe src="https://www.youtube.com/embed/MqL6BMOySIQ" frameborder="0" allowfullscreen="true"> </iframe> -</figure> - -> Notes: -> -> - The `figure` tag is required for semantic SEO and the `video_container` -class is necessary to make sure the video is responsive and displays on -different mobile devices. -> - The `<div class="video-fallback">` is a fallback necessary for GitLab's -`/help`, as GitLab's Markdown processor does not support iframes. It's hidden on -the documentation site, but will be displayed on GitLab's `/help`. - -## Code blocks - -- Always wrap code added to a sentence in inline code blocks (`` ` ``). - For example, `.gitlab-ci.yml`, `git add .`, `CODEOWNERS`, or `only: [master]`. - File names, commands, entries, and anything that refers to code should be - added to code blocks. To make things easier for the user, always add a full - code block for things that can be useful to copy and paste, as they can easily - do it with the button on code blocks. -- HTTP methods (`HTTP POST`) and HTTP status codes, both full (`404 File Not Found`) - and abbreviated (`404`), should be wrapped in inline code blocks when used in sentences. - For example: Send a `DELETE` request to delete the runner. Send a `POST` request to create one. -- Add a blank line above and below code blocks. -- When providing a shell command and its output, prefix the shell command with `$` - and leave a blank line between the command and the output. -- When providing a command without output, don't prefix the shell command with `$`. -- If you need to include triple backticks inside a code block, use four backticks - for the codeblock fences instead of three. -- For regular fenced code blocks, always use a highlighting class corresponding to - the language for better readability. Examples: - - ````markdown - ```ruby - Ruby code - ``` - - ```javascript - JavaScript code - ``` - - ```markdown - [Markdown code example](example.md) - ``` - - ```plaintext - Code or text for which no specific highlighting class is available. - ``` - ```` - -Syntax highlighting is required for fenced code blocks added to the GitLab -documentation. Refer to the following table for the most common language classes, -or check the [complete list](https://github.com/rouge-ruby/rouge/wiki/List-of-supported-languages-and-lexers) -of available language classes: - -| Preferred language tags | Language aliases and notes | -|-------------------------|------------------------------------------------------------------------------| -| `asciidoc` | | -| `dockerfile` | Alias: `docker`. | -| `elixir` | | -| `erb` | | -| `golang` | Alias: `go`. | -| `graphql` | | -| `haml` | | -| `html` | | -| `ini` | For some simple config files that are not in TOML format. | -| `javascript` | Alias `js`. | -| `json` | | -| `markdown` | Alias: `md`. | -| `mermaid` | | -| `nginx` | | -| `perl` | | -| `php` | | -| `plaintext` | Examples with no defined language, such as output from shell commands or API calls. If a codeblock has no language, it defaults to `plaintext`. Alias: `text`. | -| `prometheus` | Prometheus configuration examples. | -| `python` | | -| `ruby` | Alias: `rb`. | -| `shell` | Aliases: `bash` or `sh`. | -| `sql` | | -| `toml` | Runner configuration examples, and other TOML-formatted configuration files. | -| `typescript` | Alias: `ts`. | -| `xml` | | -| `yaml` | Alias: `yml`. | - -For a complete reference on code blocks, see the [Kramdown guide](https://about.gitlab.com/handbook/markdown-guide/#code-blocks). - -## GitLab SVG icons - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-docs/-/issues/384) in GitLab 12.7. - -You can use icons from the [GitLab SVG library](https://gitlab-org.gitlab.io/gitlab-svgs/) -directly in the documentation. - -This way, you can achieve a consistent look when writing about interacting with -GitLab user interface elements. - -Usage examples: - -- Icon with default size (16px): `**{icon-name}**` - - Example: `**{tanuki}**` renders as: **{tanuki}**. -- Icon with custom size: `**{icon-name, size}**` - - Available sizes (in px): 8, 10, 12, 14, 16, 18, 24, 32, 48, and 72 - - Example: `**{tanuki, 24}**` renders as: **{tanuki, 24}**. -- Icon with custom size and class: `**{icon-name, size, class-name}**`. - - You can access any class available to this element in GitLab documentation CSS. - - Example with `float-right`, a - [Bootstrap utility class](https://getbootstrap.com/docs/4.4/utilities/float/): - `**{tanuki, 32, float-right}**` renders as: **{tanuki, 32, float-right}** - -### When to use icons - -Icons should be used sparingly, and only in ways that aid and do not hinder the -readability of the text. - -For example, the following adds little to the accompanying text: - -```markdown -1. Go to **{home}** **Project overview > Details** -``` - -1. Go to **{home}** **Project overview > Details** - -However, the following might help the reader connect the text to the user -interface: - -```markdown -| Section | Description | -|:-------------------------|:----------------------------------------------------------------------------------------------------------------------------| -| **{overview}** Overview | View your GitLab Dashboard, and administer projects, users, groups, jobs, runners, and Gitaly servers. | -| **{monitor}** Monitoring | View GitLab system information, and information on background jobs, logs, health checks, requests profiles, and audit logs. | -| **{messages}** Messages | Send and manage broadcast messages for your users. | -``` - -| Section | Description | -|:-------------------------|:----------------------------------------------------------------------------------------------------------------------------| -| **{overview}** Overview | View your GitLab Dashboard, and administer projects, users, groups, jobs, runners, and Gitaly servers. | -| **{monitor}** Monitoring | View GitLab system information, and information on background jobs, logs, health checks, requests profiles, and audit logs. | -| **{messages}** Messages | Send and manage broadcast messages for your users. | - -Use an icon when you find yourself having to describe an interface element. For -example: - -- Do: Select the Admin Area icon ( **{admin}** ). -- Don't: Select the Admin Area icon (the wrench icon). - -## Alert boxes - -When you need to call special attention to particular sentences, use the -following markup to create highlighted alert boxes. - -Alert boxes work for one paragraph only. Multiple paragraphs, lists, and headers -won't render correctly. For multiple lines, use [blockquotes](#blockquotes) -instead. - -Alert boxes render only on the GitLab documentation site (<https://docs.gitlab.com>). -Within GitLab itself, they will appear as plain Markdown text (like the examples -above the rendered versions, below). - -These alert boxes are used in the GitLab documentation. These aren't strict -guidelines, but for consistency you should try to use these values: - -| Color | Markup | Default keyword | Alternative keywords | -|--------|------------|-----------------|----------------------------------------------------------------------| -| Blue | `NOTE:` | `**Note:**` | | -| Yellow | `CAUTION:` | `**Caution:**` | `**Warning:**`, `**Important:**` | -| Red | `DANGER:` | `**Danger:**` | `**Warning:**`, `**Important:**`, `**Deprecated:**`, `**Required:**` | -| Green | `TIP:` | `**Tip:**` | | - -### Note - -Notes indicate additional information that is of special use to the reader. -Notes are most effective when used _sparingly_. - -Try to avoid them. Too many notes can impact the scannability of a topic and -create an overly busy page. - -Instead of adding a note, try one of these alternatives: - -- Re-write the sentence as part of the most-relevant paragraph. -- Put the information into its own standalone paragraph. -- Put the content under a new subheading that introduces the topic, which makes - it more visible. - -If you must use a note, use the following formatting: - -```markdown -NOTE: **Note:** -This is something to note. -``` - -How it renders on the GitLab documentation site: - -NOTE: **Note:** -This is something to note. - -### Tip - -```markdown -TIP: **Tip:** -This is a tip. -``` - -How it renders on the GitLab documentation site: - -TIP: **Tip:** -This is a tip. - -### Caution - -```markdown -CAUTION: **Caution:** -This is something to be cautious about. -``` - -How it renders on the GitLab documentation site: - -CAUTION: **Caution:** -This is something to be cautious about. - -### Danger - -```markdown -DANGER: **Danger:** -This is a breaking change, a bug, or something very important to note. -``` - -How it renders on the GitLab documentation site: - -DANGER: **Danger:** -This is a breaking change, a bug, or something very important to note. - -## Blockquotes - -For highlighting a text within a blue blockquote, use this format: - -```markdown -> This is a blockquote. -``` - -which renders on the [GitLab documentation site](https://docs.gitlab.com) as: - -> This is a blockquote. - -If the text spans across multiple lines it's OK to split the line. - -For multiple paragraphs, use the symbol `>` before every line: - -```markdown -> This is the first paragraph. -> -> This is the second paragraph. -> -> - This is a list item -> - Second item in the list -``` - -Which renders to: - -> This is the first paragraph. -> -> This is the second paragraph. -> -> - This is a list item -> - Second item in the list - -## Terms - -To maintain consistency through GitLab documentation, the following guides -documentation authors on agreed styles and usage of terms. - -### Merge requests (MRs) - -Merge requests allow you to exchange changes you made to source code and -collaborate with other people on the same project. You'll see this term used in -the following ways: - -- Use lowercase *merge requests* regardless of whether referring to the feature - or individual merge requests. - -As noted in the GitLab [Writing Style Guidelines](https://about.gitlab.com/handbook/communication/#writing-style-guidelines), -if you use the **MR** acronym, expand it at least once per document page. -Typically, the first use would be phrased as _merge request (MR)_ with subsequent -instances being _MR_. - -Examples: - -- "We prefer GitLab merge requests". -- "Open a merge request to fix a broken link". -- "After you open a merge request (MR), submit your MR for review and approval". - -### Describe UI elements - -The following are styles to follow when describing user interface elements in an -application: - -- For elements with a visible label, use that label in bold with matching case. - For example, `the **Cancel** button`. -- For elements with a tooltip or hover label, use that label in bold with - matching case. For example, `the **Add status emoji** button`. - -### Verbs for UI elements - -The following are recommended verbs for specific uses with user interface -elements: - -| Recommended | Used for | Replaces | -|:--------------------|:--------------------------------------|:---------------------------| -| *select* | buttons, links, menu items, dropdowns | "click, "press," "hit" | -| *select* or *clear* | checkboxes | "enable", "click", "press" | -| *expand* | expandable sections | "open" | - -### Other Verbs - -| Recommended | Used for | Replaces | -|:------------|:--------------------------------|:----------------------| -| *go to* | making a browser go to location | "navigate to", "open" | - -## GitLab versions and tiers - -Tagged and released versions of GitLab documentation are available: - -- In the [documentation archives](https://docs.gitlab.com/archives/). -- At the `/help` URL for any GitLab installation. - -The version introducing a new feature is added to the top of the topic in the -documentation to provide a link back to how the feature was developed. - -TIP: **Tip:** -Whenever you have documentation related to the `gitlab.rb` file, you're working -with a self-managed installation. The section or page is therefore likely to -apply only to self-managed instances. If so, the relevant "`TIER` ONLY" -[Product badge](#product-badges) should be included at the highest applicable -heading level. - -### Text for documentation requiring version text - -When a feature is new or updated, you can add version information as a bulleted -item in the **Version history**, or as an inline reference with related text. - -#### Version text in the **Version History** - -If all content in a section is related, add version text in a bulleted list -following the heading for the section. To render correctly, it must be on its -own line and surrounded by blank lines. - -- For features that need to declare the GitLab version that the feature was - introduced. Text similar to the following should be added immediately below - the heading as a blockquote: - - `> Introduced in GitLab 11.3.`. - -- Whenever possible, version text should have a link to the _completed_ issue, - merge request, or epic that introduced the feature. An issue is preferred over - a merge request, and a merge request is preferred over an epic. For example: - - `> [Introduced](<link-to-issue>) in GitLab 11.3.`. - -- If the feature is only available in GitLab Enterprise Edition, mention - the [paid tier](https://about.gitlab.com/handbook/marketing/product-marketing/#tiers) - the feature is available in: - - `> [Introduced](<link-to-issue>) in [GitLab Starter](https://about.gitlab.com/pricing/) 11.3.`. - -- If listing information for multiple version as a feature evolves, add the - information to a block-quoted bullet list. For example: - - ```markdown - > - [Introduced](<link-to-issue>) in GitLab 11.3. - > - Enabled by default in GitLab 11.4. - ``` - -- If a feature is moved to another tier: - - ```markdown - > - [Introduced](<link-to-issue>) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.5. - > - [Moved](<link-to-issue>) to [GitLab Starter](https://about.gitlab.com/pricing/) in 11.8. - > - [Moved](<link-to-issue>) to GitLab Core in 12.0. - ``` - -- If a feature is deprecated, include a link to a replacement (when available): - - ```markdown - > - [Deprecated](<link-to-issue>) in GitLab 11.3. Replaced by [meaningful text](<link-to-appropriate-documentation>). - ``` - - It's also acceptable to describe the replacement in surrounding text, if - available. - - If the deprecation is not obvious in existing text, you may want to include a - warning such as: - - ```markdown - DANGER: **Deprecated:** - This feature was [deprecated](link-to-issue) in GitLab 12.3 - and replaced by [Feature name](link-to-feature-documentation). - ``` - -#### Inline version text - -If you're adding content to an existing topic, you can add version information -inline with the existing text. - -In this case, add `([introduced/deprecated](<link-to-issue>) in GitLab X.X)`. -If applicable, include the paid tier: `([introduced/deprecated](<link-to-issue>) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.4)` - -Including the issue link is encouraged, but isn't a requirement. For example: - -```markdown -The voting strategy (introduced in GitLab 13.4) requires -the primary and secondary voters to agree. -``` - -### Versions in the past or future - -When describing functionality available in past or future versions, use: - -- *Earlier*, and not *older* or *before*. -- *Later*, and not *newer* or *after*. - -For example: - -- Available in GitLab 12.3 and earlier. -- Available in GitLab 12.4 and later. -- If using GitLab 11.4 or earlier, ... -- If using GitLab 10.6 or later, ... - -### Importance of referencing GitLab versions and tiers - -Mentioning GitLab versions and tiers is important to all users and contributors -to quickly have access to the issue or merge request that introduced the change -for reference. Also, they can easily understand what features they have in their -GitLab instance and version, given that the note has some key information. - -`[Introduced](link-to-issue) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.7` -links to the issue that introduced the feature, says which GitLab tier it -belongs to, says the GitLab version that it became available in, and links to -the pricing page in case the user wants to upgrade to a paid tier to use that -feature. - -For example, if you're a regular user and you're looking at the documentation -for a feature you haven't used before, you can immediately see if that feature -is available to you or not. Alternatively, if you've been using a certain -feature for a long time and it changed in some way, it's important to be able to -determine when it changed and what's new in that feature. - -This is even more important as we don't have a perfect process for shipping -documentation. Unfortunately, we still see features without documentation, and -documentation without features. So, for now, we cannot rely 100% on the -documentation site versions. - -Over time, version text will reference a progressively older version of GitLab. -In cases where version text refers to versions of GitLab four or more major -versions back, you can consider removing the text if it's irrelevant or confusing. - -For example, if the current major version is 12.x, version text referencing -versions of GitLab 8.x and older are candidates for removal if necessary for -clearer or cleaner documentation. - -## Products and features - -Refer to the information in this section when describing products and features -within the GitLab product documentation. - -### Avoid line breaks in names - -When entering a product or feature name that includes a space (such as -GitLab Community Edition) or even other companies' products (such as -Amazon Web Services), be sure to not split the product or feature name across -lines with an inserted line break. Splitting product or feature names across -lines makes searching for these items more difficult, and can cause problems if -names change. - -For example, the following Markdown content is *not* formatted correctly: - -```markdown -When entering a product or feature name that includes a space (such as GitLab -Community Edition), don't split the product or feature name across lines. -``` - -Instead, it should appear similar to the following: - -```markdown -When entering a product or feature name that includes a space (such as -GitLab Community Edition), don't split the product or feature name across lines. -``` - -### Product badges - -When a feature is available in paid tiers, add the corresponding tier to the -header or other page element according to the feature's availability: - -| Tier in which feature is available | Tier markup | -|:-----------------------------------------------------------------------|:----------------------| -| GitLab Core and GitLab.com Free, and their higher tiers | `**(CORE)**` | -| GitLab Starter and GitLab.com Bronze, and their higher tiers | `**(STARTER)**` | -| GitLab Premium and GitLab.com Silver, and their higher tiers | `**(PREMIUM)**` | -| GitLab Ultimate and GitLab.com Gold | `**(ULTIMATE)**` | -| *Only* GitLab Core and higher tiers (no GitLab.com-based tiers) | `**(CORE ONLY)**` | -| *Only* GitLab Starter and higher tiers (no GitLab.com-based tiers) | `**(STARTER ONLY)**` | -| *Only* GitLab Premium and higher tiers (no GitLab.com-based tiers) | `**(PREMIUM ONLY)**` | -| *Only* GitLab Ultimate (no GitLab.com-based tiers) | `**(ULTIMATE ONLY)**` | -| *Only* GitLab.com Free and higher tiers (no self-managed instances) | `**(FREE ONLY)**` | -| *Only* GitLab.com Bronze and higher tiers (no self-managed instances) | `**(BRONZE ONLY)**` | -| *Only* GitLab.com Silver and higher tiers (no self-managed instances) | `**(SILVER ONLY)**` | -| *Only* GitLab.com Gold (no self-managed instances) | `**(GOLD ONLY)**` | - -For clarity, all page title headers (H1s) must be have a tier markup for the -lowest tier that has information on the documentation page. - -If sections of a page apply to higher tier levels, they can be separately -labeled with their own tier markup. - -#### Product badge display behavior - -When using the tier markup with headers, the documentation page will display the -full tier badge with the header line. - -You can also use the tier markup with paragraphs, list items, and table cells. -For these cases, the tier mention will be represented by an orange info icon -**{information}** that will display the tiers when visitors point to the icon. -For example: - -- `**(STARTER)**` displays as **(STARTER)** -- `**(STARTER ONLY)**` displays as **(STARTER ONLY)** -- `**(SILVER ONLY)**` displays as **(SILVER ONLY)** - -#### How it works - -Introduced by [!244](https://gitlab.com/gitlab-org/gitlab-docs/-/merge_requests/244), -the special markup `**(STARTER)**` will generate a `span` element to trigger the -badges and tooltips (`<span class="badge-trigger starter">`). When the keyword -*only* is added, the corresponding GitLab.com badge will not be displayed. - -## Specific sections - -Certain styles should be applied to specific sections. Styles for specific -sections are outlined below. - -### GitLab restart - -There are many cases that a restart/reconfigure of GitLab is required. To avoid -duplication, link to the special document that can be found in -[`doc/administration/restart_gitlab.md`](../../administration/restart_gitlab.md). -Usually the text will read like: - -```markdown -Save the file and [reconfigure GitLab](../../administration/restart_gitlab.md) -for the changes to take effect. -``` - -If the document you are editing resides in a place other than the GitLab CE/EE -`doc/` directory, instead of the relative link, use the full path: -`https://docs.gitlab.com/ee/administration/restart_gitlab.html`. Replace -`reconfigure` with `restart` where appropriate. - -### Installation guide - -**Ruby:** -In [step 2 of the installation guide](../../install/installation.md#2-ruby), -we install Ruby from source. Whenever there is a new version that needs to -be updated, remember to change it throughout the codeblock and also replace -the sha256sum (it can be found in the [downloads page](https://www.ruby-lang.org/en/downloads/) -of the Ruby website). - -### Configuration documentation for source and Omnibus installations - -GitLab currently officially supports two installation methods: installations -from source and Omnibus packages installations. - -Whenever there is a setting that is configurable for both installation methods, -prefer to document it in the CE documentation to avoid duplication. - -Configuration settings include: - -1. Settings that touch configuration files in `config/`. -1. NGINX settings and settings in `lib/support/` in general. - -When there is a list of steps to perform, usually that entails editing the -configuration file and reconfiguring/restarting GitLab. In such case, follow -the style below as a guide: - -````markdown -**For Omnibus installations** - -1. Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - external_url "https://gitlab.example.com" - ``` - -1. Save the file and [reconfigure](path/to/administration/restart_gitlab.md#omnibus-gitlab-reconfigure) - GitLab for the changes to take effect. - ---- - -**For installations from source** - -1. Edit `config/gitlab.yml`: - - ```yaml - gitlab: - host: "gitlab.example.com" - ``` - -1. Save the file and [restart](path/to/administration/restart_gitlab.md#installations-from-source) - GitLab for the changes to take effect. -```` - -In this case: - -- Before each step list the installation method is declared in bold. -- Three dashes (`---`) are used to create a horizontal line and separate the two - methods. -- The code blocks are indented one or more spaces under the list item to render - correctly. -- Different highlighting languages are used for each config in the code block. -- The [GitLab Restart](#gitlab-restart) section is used to explain a required - restart or reconfigure of GitLab. - -### Troubleshooting - -For troubleshooting sections, you should provide as much context as possible so -users can identify the problem they are facing and resolve it on their own. You -can facilitate this by making sure the troubleshooting content addresses: - -1. The problem the user needs to solve. -1. How the user can confirm they have the problem. -1. Steps the user can take towards resolution of the problem. - -If the contents of each category can be summarized in one line and a list of -steps aren't required, consider setting up a [table](#tables) with headers of -*Problem* \| *Cause* \| *Solution* (or *Workaround* if the fix is temporary), or -*Error message* \| *Solution*. - -## Feature flags - -Learn how to [document features deployed behind flags](feature_flags.md). For -guidance on developing GitLab with feature flags, see [Feature flags in development of GitLab](../feature_flags/index.md). - -## GraphQL API - -GraphQL APIs are different from [RESTful APIs](restful_api_styleguide.md). Reference -information is generated in our [GraphQL reference](../../api/graphql/reference/index.md). - -However, it's helpful to include examples on how to use GraphQL for different -*use cases*, with samples that readers can use directly in the -[GraphiQL explorer](../api_graphql_styleguide.md#graphiql). - -This section describes the steps required to add your GraphQL examples to -GitLab documentation. - -### Add a dedicated GraphQL page - -To create a dedicated GraphQL page, create a new `.md` file in the -`doc/api/graphql/` directory. Give that file a functional name, such as -`import_from_specific_location.md`. - -### Start the page with an explanation - -Include a page title that describes the GraphQL functionality in a few words, -such as: - -```markdown -# Search for [substitute kind of data] -``` - -Describe the search. One sentence may be all you need. More information may -help readers learn how to use the example for their GitLab deployments. - -### Include a procedure using the GraphiQL explorer - -The GraphiQL explorer can help readers test queries with working deployments. -Set up the section with the following: - -- Use the following title: - - ```markdown - ## Set up the GraphiQL explorer - ``` - -- Include a code block with the query that anyone can include in their - instance of the GraphiQL explorer: - - ````markdown - ```graphql - query { - <insert queries here> - } - ``` - ```` - -- Tell the user what to do: - - ```markdown - 1. Open the GraphiQL explorer tool in the following URL: `https://gitlab.com/-/graphql-explorer`. - 1. Paste the `query` listed above into the left window of your GraphiQL explorer tool. - 1. Select Play to get the result shown here: - ``` - -- Include a screenshot of the result in the GraphiQL explorer. Follow the naming - convention described in the [Save the image](#save-the-image) section. -- Follow up with an example of what you can do with the output. Make sure the - example is something that readers can do on their own deployments. -- Include a link to the [GraphQL API resources](../../api/graphql/reference/index.md). - -### Add the GraphQL example to the Table of Contents - -You'll need to open a second MR, against the [GitLab documentation repository](https://gitlab.com/gitlab-org/gitlab-docs/). - -We store our Table of Contents in the `default-nav.yaml` file, in the -`content/_data` subdirectory. You can find the GraphQL section under the -following line: - -```yaml -- category_title: GraphQL -``` - -Be aware that CI tests for that second MR will fail with a bad link until the -main MR that adds the new GraphQL page is merged. - -And that's all you need! +This document was moved to [another location](styleguide/index.md). diff --git a/doc/development/documentation/styleguide/index.md b/doc/development/documentation/styleguide/index.md new file mode 100644 index 00000000000..41e38266a58 --- /dev/null +++ b/doc/development/documentation/styleguide/index.md @@ -0,0 +1,1999 @@ +--- +stage: none +group: Style Guide +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +description: 'Writing styles, markup, formatting, and other standards for GitLab Documentation.' +--- + +# Documentation Style Guide + +This document defines the standards for GitLab's documentation content and +files. + +For broader information about the documentation, see the [Documentation guidelines](index.md). + +For guidelines specific to text in the GitLab interface, see the Pajamas [Content](https://design.gitlab.com/content/error-messages/) section. + +For information on how to validate styles locally or by using GitLab CI/CD, see [Testing](../testing.md). + +Use this guide for standards on grammar, formatting, word usage, and more. + +You can also view a list of [recent updates to this guide](https://gitlab.com/dashboard/merge_requests?scope=all&utf8=%E2%9C%93&state=merged&label_name[]=tw-style¬[label_name][]=docs%3A%3Afix). + +If you can't find what you need: + +- View the GitLab Handbook for [writing style guidelines](https://about.gitlab.com/handbook/communication/#writing-style-guidelines) that apply to all GitLab content. +- Refer to one of the following: + + - [Microsoft Style Guide](https://docs.microsoft.com/en-us/style-guide/welcome/). + - [Google Developer Documentation Style Guide](https://developers.google.com/style). + +If you have questions about style, mention `@tw-style` in an issue or merge request, or, if you have access to the GitLab Slack workspace, use `#docs-process`. + +## Documentation is the single source of truth (SSOT) + +### Why a single source of truth + +The documentation of GitLab products and features is the SSOT for all +information related to implementation, usage, and troubleshooting. It evolves +continuously, in keeping with new products and features, and with improvements +for clarity, accuracy, and completeness. + +This policy prevents information silos, making it easier to find information +about GitLab products. + +It also informs decisions about the kinds of content we include in our +documentation. + +### All information + +Include problem-solving actions that may address rare cases or be considered +_risky_, so long as proper context is provided in the form of fully detailed +warnings and caveats. This kind of content should be included as it could be +helpful to others and, when properly explained, its benefits outweigh the risks. +If you think you have found an exception to this rule, contact the +Technical Writing team. + +We will add all troubleshooting information to the documentation, no matter how +unlikely a user is to encounter a situation. For the [Troubleshooting sections](#troubleshooting), +people in GitLab Support can merge additions themselves. + +### All media types + +Include any media types/sources if the content is relevant to readers. You can +freely include or link presentations, diagrams, videos, and so on; no matter who +it was originally composed for, if it is helpful to any of our audiences, we can +include it. + +- If you use an image that has a separate source file (for example, a vector or + diagram format), link the image to the source file so that it may be reused or + updated by anyone. +- Do not copy and paste content from other sources unless it is a limited + quotation with the source cited. Typically it is better to either rephrase + relevant information in your own words or link out to the other source. + +### No special types + +In the software industry, it is a best practice to organize documentation in +different types. For example, [Divio recommends](https://www.divio.com/blog/documentation/): + +- Tutorials +- How-to guides +- Explanation +- Reference (for example, a glossary) + +At GitLab, we have so many product changes in our monthly releases that we can't +afford to continuously update multiple types of information. If we have multiple +types, the information will become outdated. Therefore, we have a +[single template](../structure.md) for documentation. + +We currently do not distinguish specific document types, although we are open to +reconsidering this policy after the documentation has reached a future stage of +maturity and quality. If you are reading this, then despite our continuous +improvement efforts, that point hasn't been reached. + +### Link instead of summarize + +There is a temptation to summarize the information on another page. This will +cause the information to live in two places. Instead, link to the single source +of truth and explain why it is important to consume the information. + +### Organize by topic, not by type + +Beyond top-level audience-type folders (for example, `administration`), we +organize content by topic, not by type, so it can be located in the +single-source-of-truth (SSOT) section for the subject matter. + +For example, do not create groupings of similar media types. For example: + +- Glossaries. +- FAQs. +- Sets of all articles or videos. + +Such grouping of content by type makes it difficult to browse for the information +you need and difficult to maintain up-to-date content. Instead, organize content +by its subject (for example, everything related to CI goes together) and +cross-link between any related content. + +### Docs-first methodology + +We employ a _documentation-first methodology_ to help ensure the documentation +remains a complete and trusted resource, and to make communicating about the use +of GitLab more efficient. + +- If the answer to a question exists in documentation, share the link to the + documentation instead of rephrasing the information. +- When you encounter new information not available in GitLab’s documentation (for + example, when working on a support case or testing a feature), your first step + should be to create a merge request (MR) to add this information to the + documentation. You can then share the MR to communicate this information. + +New information that would be useful toward the future usage or troubleshooting +of GitLab should not be written directly in a forum or other messaging system, +but added to a documentation MR and then referenced, as described above. Note +that among any other documentation changes, you can either: + +- Add a [Troubleshooting section](#troubleshooting) to a doc if none exists. +- Un-comment and use the placeholder Troubleshooting section included as part of + our [documentation template](../structure.md#template-for-new-docs), if present. + +The more we reflexively add useful information to the documentation, the more +(and more successfully) the documentation will be used to efficiently accomplish +tasks and solve problems. + +If you have questions when considering, authoring, or editing documentation, ask +the Technical Writing team on Slack in `#docs` or in GitLab by mentioning the +writer for the applicable [DevOps stage](https://about.gitlab.com/handbook/product/product-categories/#devops-stages). +Otherwise, forge ahead with your best effort. It does not need to be perfect; +the team is happy to review and improve upon your content. Review the +[Documentation guidelines](index.md) before you begin your first documentation MR. + +Having a knowledge base in any form that's separate from the documentation would +be against the documentation-first methodology, because the content would overlap with +the documentation. + +## Markdown + +All GitLab documentation is written using [Markdown](https://en.wikipedia.org/wiki/Markdown). + +The [documentation website](https://docs.gitlab.com) uses GitLab Kramdown as its +Markdown rendering engine. For a complete Kramdown reference, see the +[GitLab Markdown Kramdown Guide](https://about.gitlab.com/handbook/markdown-guide/). + +The [`gitlab-kramdown`](https://gitlab.com/gitlab-org/gitlab_kramdown) Ruby gem +will support all [GFM markup](../../../user/markdown.md) in the future, which is +all markup supported for display in the GitLab application itself. For now, use +regular Markdown markup, following the rules in the linked style guide. + +Note that Kramdown-specific markup (for example, `{:.class}`) doesn't render +properly on GitLab instances under [`/help`](../index.md#gitlab-help). + +### HTML in Markdown + +Hard-coded HTML is valid, although it's discouraged from being used while we +have `/help`. HTML is permitted if: + +- There's no equivalent markup in Markdown. +- Advanced tables are necessary. +- Special styling is required. +- Reviewed and approved by a technical writer. + +### Markdown Rules + +GitLab ensures that the Markdown used across all documentation is consistent, as +well as easy to review and maintain, by [testing documentation changes](../testing.md) +with [markdownlint](../testing.md#markdownlint). This lint test fails when any +document has an issue with Markdown formatting that may cause the page to render +incorrectly within GitLab. It will also fail when a document is using +non-standard Markdown (which may render correctly, but is not the current +standard for GitLab documentation). + +#### Markdown rule `MD044/proper-names` (capitalization) + +A rule that could cause confusion is `MD044/proper-names`, as it might not be +immediately clear what caused markdownlint to fail, or how to correct the +failure. This rule checks a list of known words, listed in the `.markdownlint.json` +file in each project, to verify proper use of capitalization and backticks. +Words in backticks will be ignored by markdownlint. + +In general, product names should follow the exact capitalization of the official +names of the products, protocols, and so on. See [`.markdownlint.json`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.markdownlint.json) +for the words tested for proper capitalization in GitLab documentation. + +Some examples fail if incorrect capitalization is used: + +- MinIO (needs capital `IO`) +- NGINX (needs all capitals) +- runit (needs lowercase `r`) + +Additionally, commands, parameters, values, filenames, and so on must be +included in backticks. For example: + +- "Change the `needs` keyword in your `.gitlab.yml`..." + - `needs` is a parameter, and `.gitlab.yml` is a file, so both need backticks. + Additionally, `.gitlab.yml` will fail markdownlint without backticks as it + does not have capital G or L. +- "Run `git clone` to clone a Git repository..." + - `git clone` is a command, so it must be lowercase, while Git is the product, + so it must have a capital G. + +## Structure + +Because we want documentation to be a SSOT, we should [organize by topic, not by +type](#organize-by-topic-not-by-type). + +### Folder structure overview + +The documentation is separated by top-level audience folders [`user`](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/doc/user), +[`administration`](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/doc/administration), +and [`development`](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/doc/development) +(contributing) folders. + +Beyond that, we primarily follow the structure of the GitLab user interface or +API. + +Our goal is to have a clear hierarchical structure with meaningful URLs like +`docs.gitlab.com/user/project/merge_requests/`. With this pattern, you can +immediately tell that you are navigating to user-related documentation about +Project features; specifically about Merge Requests. Our site's paths match +those of our repository, so the clear structure also makes documentation easier +to update. + +Put files for a specific product area into the related folder: + +| Directory | What belongs here | +|:----------------------|:----------------------------------------------------------------------------------------------------------------------------------------| +| `doc/user/` | User related documentation. Anything that can be done within the GitLab user interface goes here, including usage of the `/admin` interface. | +| `doc/administration/` | Documentation that requires the user to have access to the server where GitLab is installed. The admin settings that can be accessed by using GitLab's interface exist under `doc/user/admin_area/`. | +| `doc/api/` | API related documentation. | +| `doc/development/` | Documentation related to the development of GitLab, whether contributing code or documentation. Related process and style guides should go here. | +| `doc/legal/` | Legal documents about contributing to GitLab. | +| `doc/install/` | Contains instructions for installing GitLab. | +| `doc/update/` | Contains instructions for updating GitLab. | +| `doc/topics/` | Indexes per topic (`doc/topics/topic_name/index.md`): all resources for that topic. | + +### Work with directories and files + +Refer to the following items when working with directories and files: + +1. When you create a new directory, always start with an `index.md` file. + Don't use another file name and _do not_ create `README.md` files. +1. _Do not_ use special characters and spaces, or capital letters in file + names, directory names, branch names, and anything that generates a path. +1. When creating or renaming a file or directory and it has more than one word + in its name, use underscores (`_`) instead of spaces or dashes. For example, + proper naming would be `import_project/import_from_github.md`. This applies + to both image files and Markdown files. +1. For image files, do not exceed 100KB. +1. Do not upload video files to the product repositories. + [Link or embed videos](#videos) instead. +1. There are four main directories: `user`, `administration`, `api`, and + `development`. +1. The `doc/user/` directory has five main subdirectories: `project/`, `group/`, + `profile/`, `dashboard/` and `admin_area/`. + - `doc/user/project/` should contain all project related documentation. + - `doc/user/group/` should contain all group related documentation. + - `doc/user/profile/` should contain all profile related documentation. + Every page you would navigate under `/profile` should have its own document, + for example, `account.md`, `applications.md`, or `emails.md`. + - `doc/user/dashboard/` should contain all dashboard related documentation. + - `doc/user/admin_area/` should contain all admin related documentation + describing what can be achieved by accessing GitLab's admin interface + (_not to be confused with `doc/administration` where server access is + required_). + - Every category under `/admin/application_settings/` should have its + own document located at `doc/user/admin_area/settings/`. For example, + the **Visibility and Access Controls** category should have a document + located at `doc/user/admin_area/settings/visibility_and_access_controls.md`. +1. The `doc/topics/` directory holds topic-related technical content. Create + `doc/topics/topic_name/subtopic_name/index.md` when subtopics become necessary. + General user- and admin- related documentation, should be placed accordingly. +1. The `/university/` directory is *deprecated* and the majority of its documentation + has been moved. + +If you're unsure where to place a document or a content addition, this shouldn't +stop you from authoring and contributing. Use your best judgment, and then ask +the reviewer of your MR to confirm your decision, or ask a technical writer at +any stage in the process. The technical writing team will review all +documentation changes, regardless, and can move content if there is a better +place for it. + +### Avoid duplication + +Do not include the same information in multiple places. +[Link to a single source of truth instead.](#link-instead-of-summarize) + +### References across documents + +- Give each folder an `index.md` page that introduces the topic, introduces the + pages within, and links to the pages within (including to the index pages of + any next-level subpaths). +- To ensure discoverability, ensure each new or renamed doc is linked from its + higher-level index page and other related pages. +- When making reference to other GitLab products and features, link to their + respective documentation, at least on first mention. +- When making reference to third-party products or technologies, link out to + their external sites, documentation, and resources. + +### Structure within documents + +- Include any and all applicable subsections as described on the + [structure and template](../structure.md) page. +- Structure content in alphabetical order in tables, lists, and so on, unless + there's a logical reason not to (for example, when mirroring the user + interface or an otherwise ordered sequence). + +## Language + +GitLab documentation should be clear and easy to understand. + +- Be clear, concise, and stick to the goal of the documentation. +- Write in US English with US grammar. (Tested in [`British.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/British.yml).) +- Use [inclusive language](#inclusive-language). + +### Point of view + +In most cases, it’s appropriate to use the second-person (you, yours) point of +view, because it’s friendly and easy to understand. (Tested in +[`FirstPerson.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/FirstPerson.yml).) + +### Capitalization + +#### Headings + +Use sentence case. For example: + +- `# Use variables to configure pipelines` +- `## Use the To-Do List` + +#### UI text + +When referring to specific user interface text, like a button label or menu +item, use the same capitalization that's displayed in the user interface. +Standards for this content are listed in the [Pajamas Design System Content section](https://design.gitlab.com/content/punctuation/) +and typically match what's called for in this Documentation Style Guide. + +If you think there's a mistake in the way the user interface text is styled, +create an issue or an MR to propose a change to the user interface text. + +#### Feature names + +- *Feature names are typically lowercase*, like those describing actions and + types of objects in GitLab. For example: + - epics + - issues + - issue weights + - merge requests + - milestones + - reorder issues + - runner, runners, shared runners + - a to-do item (tested in [`ToDo.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/ToDo.yml)) +- *Some features are capitalized*, typically nouns naming GitLab-specific + capabilities or tools. For example: + - GitLab CI/CD + - Repository Mirroring + - Value Stream Analytics + - the To-Do List + - the Web IDE + - Geo + - GitLab Runner (see [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/233529) for details) + +Document any exceptions in this style guide. If you're not sure, ask a GitLab +Technical Writer so that they can help decide and document the result. + +Do not match the capitalization of terms or phrases on the [Features page](https://about.gitlab.com/features/) +or [features.yml](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/features.yml) +by default. + +#### Other terms + +Capitalize names of: + +- GitLab [product tiers](https://about.gitlab.com/pricing/). For example, + GitLab Core and GitLab Ultimate. (Tested in [`BadgeCapitalization.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/BadgeCapitalization.yml).) +- Third-party organizations, software, and products. For example, Prometheus, + Kubernetes, Git, and The Linux Foundation. +- Methods or methodologies. For example, Continuous Integration, + Continuous Deployment, Scrum, and Agile. (Tested in [`.markdownlint.json`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.markdownlint.json).) + +Follow the capitalization style listed at the [authoritative source](#links-to-external-documentation) +for the entity, which may use non-standard case styles. For example: GitLab and +npm. + +Use forms of *sign in*, instead of *log in* or *login*. For example: + +- Sign in to GitLab. +- Open the sign-in page. + +Exceptions to this rule include the concept of *single sign-on* and +references to user interface elements. For example: + +- To sign in to product X, enter your credentials, and then select **Log in**. + +### Inclusive language + +We strive to create documentation that's inclusive. This section includes +guidance and examples for the following categories: + +- [Gender-specific wording](#avoid-gender-specific-wording). + (Tested in [`InclusionGender.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/InclusionGender.yml).) +- [Ableist language](#avoid-ableist-language). + (Tested in [`InclusionAbleism.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/InclusionAbleism.yml).) +- [Cultural sensitivity](#culturally-sensitive-language). + (Tested in [`InclusionCultural.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/InclusionCultural.yml).) + +We write our developer documentation with inclusivity and diversity in mind. This +page is not an exhaustive reference, but describes some general guidelines and +examples that illustrate some best practices to follow. + +#### Avoid gender-specific wording + +When possible, use gender-neutral pronouns. For example, you can use a singular +[they](https://developers.google.com/style/pronouns#gender-neutral-pronouns) as +a gender-neutral pronoun. + +Avoid the use of gender-specific pronouns, unless referring to a specific person. + +<!-- vale gitlab.InclusionGender = NO --> + +| Use | Avoid | +|-----------------------------------|---------------------------------| +| People, humanity | Mankind | +| GitLab Team Members | Manpower | +| You can install; They can install | He can install; She can install | + +<!-- vale gitlab.InclusionGender = YES --> + +If you need to set up [Fake user information](#fake-user-information), use +diverse or non-gendered names with common surnames. + +#### Avoid ableist language + +Avoid terms that are also used in negative stereotypes for different groups. + +<!-- vale gitlab.InclusionAbleism = NO --> + +| Use | Avoid | +|------------------------|----------------------| +| Check for completeness | Sanity check | +| Uncertain outliers | Crazy outliers | +| Slows the service | Cripples the service | +| Placeholder variable | Dummy variable | +| Active/Inactive | Enabled/Disabled | +| On/Off | Enabled/Disabled | + +<!-- vale gitlab.InclusionAbleism = YES --> + +Credit: [Avoid ableist language](https://developers.google.com/style/inclusive-documentation#ableist-language) +in the Google Developer Style Guide. + +#### Culturally sensitive language + +Avoid terms that reflect negative cultural stereotypes and history. In most +cases, you can replace terms such as `master` and `slave` with terms that are +more precise and functional, such as `primary` and `secondary`. + +<!-- vale gitlab.InclusionCultural = NO --> + +| Use | Avoid | +|----------------------|-----------------------| +| Primary / secondary | Master / slave | +| Allowlist / denylist | Blacklist / whitelist | + +<!-- vale gitlab.InclusionCultural = YES --> + +For more information see the following [Internet Draft specification](https://tools.ietf.org/html/draft-knodel-terminology-02). + +### Fake user information + +You may need to include user information in entries such as a REST call or user profile. +_Do not_ use real user information or email addresses in GitLab documentation. For email +addresses and names, do use: + +- _Email addresses_: Use an email address ending in `example.com`. +- _Names_: Use strings like `example_username`. Alternatively, use diverse or + non-gendered names with common surnames, such as `Sidney Jones`, `Zhang Wei`, + or `Alex Garcia`. + +### Fake URLs + +When including sample URLs in the documentation, use: + +- `example.com` when the domain name is generic. +- `gitlab.example.com` when referring to self-managed instances of GitLab. + +### Fake tokens + +There may be times where a token is needed to demonstrate an API call using +cURL or a variable used in CI. It is strongly advised not to use real tokens in +documentation even if the probability of a token being exploited is low. + +You can use the following fake tokens as examples: + +| Token type | Token value | +|:----------------------|:-------------------------------------------------------------------| +| Private user token | `<your_access_token>` | +| Personal access token | `n671WNGecHugsdEDPsyo` | +| Application ID | `2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6` | +| Application secret | `04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df` | +| CI/CD variable | `Li8j-mLUVA3eZYjPfd_H` | +| Specific runner token | `yrnZW46BrtBFqM7xDzE7dddd` | +| Shared runner token | `6Vk7ZsosqQyfreAxXTZr` | +| Trigger token | `be20d8dcc028677c931e04f3871a9b` | +| Webhook secret token | `6XhDroRcYPM5by_h-HLY` | +| Health check token | `Tu7BgjR9qeZTEyRzGG2P` | +| Request profile token | `7VgpS4Ax5utVD2esNstz` | + +### Language to avoid + +When creating documentation, limit or avoid the use of the following verb +tenses, words, and phrases: + +- Avoid jargon when possible, and when not possible, define the term or + [link to a definition](#links-to-external-documentation). +- Avoid uncommon words when a more-common alternative is possible, ensuring that + content is accessible to more readers. +- Don't write in the first person singular. + (Tested in [`FirstPerson.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/FirstPerson.yml).) + - Instead of _I_ or _me_, use _we_, _you_, _us_, or _one_. + - When possible, stay user focused by writing in the second person (_you_ or + the imperative). +- Don't overuse "that". In many cases, you can remove "that" from a sentence + and improve readability. +- Avoid use of the future tense: + - Instead of "after you execute this command, GitLab will display the + result", use "after you execute this command, GitLab displays the result". + - Only use the future tense to convey when the action or result will actually + occur at a future time. +- Don't use slashes to clump different words together or as a replacement for + the word "or": + - Instead of "and/or," consider using "or," or use another sensible + construction. + - Other examples include "clone/fetch," author/assignee," and + "namespace/repository name." Break apart any such instances in an + appropriate way. + - Exceptions to this rule include commonly accepted technical terms, such as + CI/CD and TCP/IP. +<!-- vale gitlab.LatinTerms = NO --> +- We discourage the use of Latin abbreviations and terms, such as _e.g._, + _i.e._, _etc._, or _via_, as even native users of English can misunderstand + those terms. (Tested in [`LatinTerms.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/LatinTerms.yml).) + - Instead of _i.e._, use _that is_. + - Instead of _via_, use _through_. + - Instead of _e.g._, use _for example_, _such as_, _for instance_, or _like_. + - Instead of _etc._, either use _and so on_ or consider editing it out, since + it can be vague. +<!-- vale gitlab.LatinTerms = YES --> +- Avoid using the word *currently* when talking about the product or its + features. The documentation describes the product as it is, and not as it + will be at some indeterminate point in the future. +- Avoid the using the word *scalability* with increasing GitLab's performance + for additional users. Using the words *scale* or *scaling* in other cases is + acceptable, but references to increasing GitLab's performance for additional + users should direct readers to the GitLab + [reference architectures](../../../administration/reference_architectures/index.md) + page. +- Avoid all forms of the phrases *high availability* and *HA*, and instead + direct readers to the GitLab [reference architectures](../../../administration/reference_architectures/index.md) + for information about configuring GitLab to have the performance needed for + additional users over time. +- Don't use profanity or obscenities. Doing so may negatively affect other + users and contributors, which is contrary to GitLab's value of + [Diversity, Inclusion, and Belonging](https://about.gitlab.com/handbook/values/#diversity-inclusion). +- Avoid the use of [racially-insensitive terminology or phrases](https://www.marketplace.org/2020/06/17/tech-companies-update-language-to-avoid-offensive-terms/). For example: + - Use *primary* and *secondary* for database and server relationships. + - Use *allowlist* and *denylist* to describe access control lists. +- Avoid the word _please_. For details, see the [Microsoft style guide](https://docs.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/p/please). +- Avoid words like _easily_, _simply_, _handy_, and _useful._ If the user + doesn't find the process to be these things, we lose their trust. + +### Word usage clarifications + +- Don't use "may" and "might" interchangeably: + - Use "might" to indicate the probability of something occurring. "If you + skip this step, the import process might fail." + - Use "may" to indicate giving permission for someone to do something, or + consider using "can" instead. "You may select either option on this + screen." Or, "You can select either option on this screen." + +### Contractions + +Contractions are encouraged, and can create a friendly and informal tone, +especially in tutorials, instructional documentation, and +[user interfaces](https://design.gitlab.com/content/punctuation/#contractions). + +Some contractions, however, should be avoided: + +- Do not use contractions with a proper noun and a verb. For example: + + | Do | Don't | + |------------------------------------------|-----------------------------------------| + | GitLab is creating X. | GitLab's creating X. | + +- Do not use contractions when you need to emphasize a negative. For example: + + | Do | Don't | + |------------------------------------------|-----------------------------------------| + | Do *not* install X with Y. | *Don't* install X with Y. | + +- Do not use contractions in reference documentation. For example: + + | Do | Don't | + |------------------------------------------|-----------------------------------------| + | Do *not* set a limit greater than 1000. | *Don't* set a limit greater than 1000. | + | For `parameter1`, the default is 10. | For `parameter1`, the default's 10. | + +- Avoid contractions in error messages. Examples: + + | Do | Don't | + |------------------------------------------|-----------------------------------------| + | Requests to localhost are not allowed. | Requests to localhost aren't allowed. | + | Specified URL cannot be used. | Specified URL can't be used. | + +## Text + +- [Write in Markdown](#markdown). +- Splitting long lines (preferably up to 100 characters) can make it easier to + provide feedback on small chunks of text. +- Insert an empty line for new paragraphs. +- Insert an empty line between different markups (for example, after every + paragraph, header, list, and so on). Example: + + ```markdown + ## Header + + Paragraph. + + - List item 1 + - List item 2 + ``` + +### Emphasis + +- Use double asterisks (`**`) to mark a word or text in bold (`**bold**`). +- Use underscore (`_`) for text in italics (`_italic_`). +- Use greater than (`>`) for blockquotes. + +### Punctuation + +Follow these guidelines for punctuation: + +<!-- vale gitlab.Repetition = NO --> + +| Rule | Example | +|------------------------------------------------------------------|--------------------------------------------------------| +| Always end full sentences with a period. | _For a complete overview, read through this document._ | +| Always add a space after a period when beginning a new sentence. | _For a complete overview, check this doc. For other references, check out this guide._ | +| Do not use double spaces. (Tested in [`SentenceSpacing.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/SentenceSpacing.yml).) | --- | +| Do not use tabs for indentation. Use spaces instead. You can configure your code editor to output spaces instead of tabs when pressing the tab key. | --- | +| Use serial commas (_Oxford commas_) before the final _and_ or _or_ in a list of three or more items. (Tested in [`OxfordComma.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/OxfordComma.yml).) | _You can create new issues, merge requests, and milestones._ | +| Always add a space before and after dashes when using it in a sentence (for replacing a comma, for example). | _You should try this - or not._ | +| Always use lowercase after a colon. | _Related Issues: a way to create a relationship between issues._ | + +<!-- vale gitlab.Repetition = YES --> + +### Placeholder text + +Often in examples, a writer will provide a command or configuration that +uses values specific to the reader. + +In these cases, use [`<` and `>`](https://en.wikipedia.org/wiki/Usage_message#Pattern) +to call out where a reader must replace text with their own value. + +For example: + +```shell +cp <your_source_directory> <your_destination_directory> +``` + +### Keyboard commands + +Use the HTML `<kbd>` tag when referring to keystroke presses. For example: + +```plaintext +To stop the command, press <kbd>Ctrl</kbd>+<kbd>C</kbd>. +``` + +When the docs are generated, the output is: + +To stop the command, press <kbd>Ctrl</kbd>+<kbd>C</kbd>. + +## Lists + +- Always start list items with a capital letter, unless they're parameters or + commands that are in backticks, or similar. +- Always leave a blank line before and after a list. +- Begin a line with spaces (not tabs) to denote a [nested sub-item](#nesting-inside-a-list-item). + +### Ordered vs. unordered lists + +Only use ordered lists when their items describe a sequence of steps to follow. + +Do: + +```markdown +These are the steps to do something: + +1. First, do the first step. +1. Then, do the next step. +1. Finally, do the last step. +``` + +Don't: + +```markdown +This is a list of available features: + +1. Feature 1 +1. Feature 2 +1. Feature 3 +``` + +### Markup + +- Use dashes (`-`) for unordered lists instead of asterisks (`*`). +- Prefix `1.` to every item in an ordered list. When rendered, the list items + will appear with sequential numbering. + +### Punctuation + +- Don't add commas (`,`) or semicolons (`;`) to the ends of list items. +- Only add periods to the end of a list item if the item consists of a complete + sentence (with a subject and a verb). +- Be consistent throughout the list: if the majority of the items do not end in + a period, do not end any of the items in a period, even if they consist of a + complete sentence. The opposite is also valid: if the majority of the items + end with a period, end all with a period. +- Separate list items from explanatory text with a colon (`:`). For example: + + ```markdown + The list is as follows: + + - First item: this explains the first item. + - Second item: this explains the second item. + ``` + +**Examples:** + +Do: + +- First list item +- Second list item +- Third list item + +Don't: + +- First list item +- Second list item +- Third list item. + +Do: + +- Let's say this is a complete sentence. +- Let's say this is also a complete sentence. +- Not a complete sentence. + +Don't (vary use of periods; majority rules): + +- Let's say this is a complete sentence. +- Let's say this is also a complete sentence. +- Not a complete sentence + +### Nesting inside a list item + +It's possible to nest items under a list item, so that they render with the same +indentation as the list item. This can be done with: + +- [Code blocks](#code-blocks) +- [Blockquotes](#blockquotes) +- [Alert boxes](#alert-boxes) +- [Images](#images) + +Items nested in lists should always align with the first character of the list +item. In unordered lists (using `-`), this means two spaces for each level of +indentation: + +````markdown +- Unordered list item 1 + + A line nested using 2 spaces to align with the `U` above. + +- Unordered list item 2 + + > A quote block that will nest + > inside list item 2. + +- Unordered list item 3 + + ```plaintext + a codeblock that will next inside list item 3 + ``` + +- Unordered list item 4 + +  +```` + +For ordered lists, use three spaces for each level of indentation: + +````markdown +1. Ordered list item 1 + + A line nested using 3 spaces to align with the `O` above. + +1. Ordered list item 2 + + > A quote block that will nest + > inside list item 2. + +1. Ordered list item 3 + + ```plaintext + a codeblock that will next inside list item 3 + ``` + +1. Ordered list item 4 + +  +```` + +You can nest full lists inside other lists using the same rules as above. If you +want to mix types, that's also possible, if you don't mix items at the same +level: + +```markdown +1. Ordered list item one. +1. Ordered list item two. + - Nested unordered list item one. + - Nested unordered list item two. +1. Ordered list item three. + +- Unordered list item one. +- Unordered list item two. + 1. Nested ordered list item one. + 1. Nested ordered list item two. +- Unordered list item three. +``` + +## Tables + +Tables should be used to describe complex information in a straightforward +manner. Note that in many cases, an unordered list is sufficient to describe a +list of items with a single, simple description per item. But, if you have data +that's best described by a matrix, tables are the best choice. + +### Creation guidelines + +Due to accessibility and scannability requirements, tables should not have any +empty cells. If there is no otherwise meaningful value for a cell, consider entering +*N/A* (for 'not applicable') or *none*. + +To help tables be easier to maintain, consider adding additional spaces to the +column widths to make them consistent. For example: + +```markdown +| App name | Description | Requirements | +|:---------|:---------------------|:---------------| +| App 1 | Description text 1. | Requirements 1 | +| App 2 | Description text 2. | None | +``` + +Consider installing a plugin or extension in your editor for formatting tables: + +- [Markdown Table Prettifier](https://marketplace.visualstudio.com/items?itemName=darkriszty.markdown-table-prettify) for Visual Studio Code +- [Markdown Table Formatter](https://packagecontrol.io/packages/Markdown%20Table%20Formatter) for Sublime Text +- [Markdown Table Formatter](https://atom.io/packages/markdown-table-formatter) for Atom + +### Feature tables + +When creating tables of lists of features (such as whether or not features are +available to certain roles on the [Permissions](../../../user/permissions.md#project-members-permissions) +page), use the following phrases (based on the SVG icons): + +| Option | Markdown | Displayed result | +|--------|--------------------------|------------------------| +| No | `**{dotted-circle}** No` | **{dotted-circle}** No | +| Yes | `**{check-circle}** Yes` | **{check-circle}** Yes | + +## Quotes + +Valid for Markdown content only, not for front matter entries: + +- Standard quotes: double quotes (`"`). Example: "This is wrapped in double + quotes". +- Quote within a quote: double quotes (`"`) wrap single quotes (`'`). Example: + "I am 'quoting' something within a quote". + +For other punctuation rules, refer to the +[GitLab UX guide](https://design.gitlab.com/content/punctuation/). + +## Headings + +- Add _only one H1_ in each document, by adding `#` at the beginning of + it (when using Markdown). The `h1` will be the document `<title>`. +- Start with an `h2` (`##`), and respect the order `h2` > `h3` > `h4` > `h5` > `h6`. + Never skip the hierarchy level, such as `h2` > `h4` +- Avoid putting numbers in headings. Numbers shift, hence documentation anchor + links shift too, which eventually leads to dead links. If you think it is + compelling to add numbers in headings, make sure to at least discuss it with + someone in the Merge Request. +- [Avoid using symbols and special characters](https://gitlab.com/gitlab-org/gitlab-docs/-/issues/84) + in headers. Whenever possible, they should be plain and short text. +- When possible, avoid including words that might change in the future. Changing + a heading changes its anchor URL, which affects other linked pages. +- When introducing a new document, be careful for the headings to be + grammatically and syntactically correct. Mention an [assigned technical writer (TW)](https://about.gitlab.com/handbook/product/product-categories/) + for review. + This is to ensure that no document with wrong heading is going live without an + audit, thus preventing dead links and redirection issues when corrected. +- Leave exactly one blank line before and after a heading. +- Do not use links in headings. +- Add the corresponding [product badge](#product-badges) according to the tier the + feature belongs. +- Our documentation site search engine prioritizes words used in headings and + subheadings. Make you subheading titles clear, descriptive, and complete to help + users find the right example, as shown in the section on [heading titles](#heading-titles). +- See [Capitalization](#capitalization) for guidelines on capitalizing headings. + +### Heading titles + +Keep heading titles clear and direct. Make every word count. To accommodate +search engine optimization (SEO), use the imperative, where possible. + +| Do | Don't | +|:--------------------------------------|:------------------------------------------------------------| +| Configure GDK | Configuring GDK | +| GitLab Release and Maintenance Policy | This section covers GitLab's Release and Maintenance Policy | +| Backport to older releases | Backporting to older releases | +| GitLab Pages examples | Examples | + +For guidelines on capitalizing headings, see the section on [capitalization](#capitalization). + +NOTE: **Note:** +If you change an existing title, be careful. These changes might affect not +only [links](#anchor-links) within the page, but might also affect links to the +GitLab documentation from both the GitLab application and external sites. + +### Anchor links + +Headings generate anchor links when rendered. `## This is an example` generates +the anchor `#this-is-an-example`. + +NOTE: **Note:** +[Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/39717) in +GitLab 13.4, [product badges](#product-badges) used in headings aren't included +in the generated anchor links. For example, when you link to +`## This is an example **(CORE)**`, use the anchor `#this-is-an-example`. + +Keep in mind that the GitLab user interface links to many documentation pages +and anchor links to take the user to the right spot. Therefore, when you change +a heading, search `doc/*`, `app/views/*`, and `ee/app/views/*` for the old +anchor to make sure you're not breaking an anchor linked from other +documentation nor from the GitLab user interface. If you find the old anchor, be +sure to replace it with the new one. + +Important: + +- Avoid crosslinking documentation to headings unless you need to link to a + specific section of the document. This will avoid breaking anchors in the + future in case the heading is changed. +- If possible, avoid changing headings since they're not only linked internally. + There are various links to GitLab documentation on the internet, such as + tutorials, presentations, StackOverflow posts, and other sources. +- Do not link to `h1` headings. + +Note that, with Kramdown, it is possible to add a custom ID to an HTML element +with Markdown markup, but they _do not_ work in GitLab's `/help`. Therefore, +do not use this option until further notice. + +## Links + +Links are important in GitLab documentation. They allow you to [link instead of +summarizing](#link-instead-of-summarize) to help preserve a [single source of truth](#why-a-single-source-of-truth) +within GitLab documentation. + +We include guidance for links in the following categories: + +- How to set up [anchor links](#anchor-links) for headings. +- How to set up [criteria](#basic-link-criteria) for configuring a link. +- What to set up when [linking to a `help`](../../documentation/index.md#linking-to-help) + page. +- How to set up [links to internal documentation](#links-to-internal-documentation) + for cross-references. +- How to set up [links to external documentation](#links-to-external-documentation) + for authoritative sources. +- When to use [links requiring permissions](#links-requiring-permissions). +- How to set up a [link to a video](#link-to-video). +- How to [include links with version text](#text-for-documentation-requiring-version-text). +- How to [link to specific lines of code](#link-to-specific-lines-of-code) + +### Basic link criteria + +- Use inline link Markdown markup `[Text](https://example.com)`. + It's easier to read, review, and maintain. _Do not_ use `[Text][identifier]`. + +- Use [meaningful anchor texts](https://www.futurehosting.com/blog/links-should-have-meaningful-anchor-text-heres-why/). + For example, instead of writing something like `Read more about GitLab Issue Boards [here](LINK)`, + write `Read more about [GitLab Issue Boards](LINK)`. + +### Links to internal documentation + +NOTE: **Note:** +_Internal_ refers to documentation in the same project. When linking to +documentation in separate projects (for example, linking to Omnibus documentation +from GitLab documentation), you must use absolute URLs. + +Do not use absolute URLs like `https://docs.gitlab.com/ee/index.html` to +crosslink to other documentation within the same project. Use relative links to +the file, like `../index.md`. (These are converted to HTML when the site is +rendered.) + +Relative linking enables crosslinks to work: + +- in Review Apps, local previews, and `/help`. +- when working on the documentation locally, so you can verify that they work as + early as possible in the process. +- within the GitLab user interface when browsing doc files in their respective + repositories. For example, the links displayed at + `https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/README.md`. + +To link to internal documentation: + +- Use relative links to Markdown files in the same repository. +- Do not use absolute URLs or URLs from `docs.gitlab.com`. +- Use `../` to navigate to higher-level directories. +- Don't prepend `./` to links to files or directories. +- Don't link relative to root. For example, `/ee/user/gitlab_com/index.md`. + + Don't: + + - `https://docs.gitlab.com/ee/administration/geo/replication/troubleshooting.html` + - `/ee/administration/geo/replication/troubleshooting.md` + - `./troubleshooting.md` + + Do: `../../geo/replication/troubleshooting.md` + +- Always add the file name `file.md` at the end of the link with the `.md` + extension, not `.html`. + + Don't: + + - `../../merge_requests/` + - `../../issues/tags.html` + - `../../issues/tags.html#stages` + + Do: + + - `../../merge_requests/index.md` + - `../../issues/tags.md` + - `../../issues/tags.md#stages` + +NOTE: **Note:** +Using the Markdown extension is necessary for the [`/help`](../index.md#gitlab-help) +section of GitLab. + +### Links to external documentation + +When describing interactions with external software, it's often helpful to +include links to external documentation. When possible, make sure that you're +linking to an [**authoritative** source](#authoritative-sources). For example, +if you're describing a feature in Microsoft's Active Directory, include a link +to official Microsoft documentation. + +### Authoritative sources + +When citing external information, use sources that are written by the people who +created the item or product in question. These sources are the most likely to be +accurate and remain up to date. + +Examples of authoritative sources include: + +- Specifications, such as a [Request for Comments](https://www.ietf.org/standards/rfcs/) + document from the Internet Engineering Task Force. +- Official documentation for a product. For example, if you're setting up an + interface with the Google OAuth 2 authorization server, include a link to + Google's documentation. +- Official documentation for a project. For example, if you're citing NodeJS + functionality, refer directly to [NodeJS documentation](https://nodejs.org/en/docs/). +- Books from an authoritative publisher. + +Examples of sources to avoid include: + +- Personal blog posts. +- Wikipedia. +- Non-trustworthy articles. +- Discussions on forums such as Stack Overflow. +- Documentation from a company that describes another company's product. + +While many of these sources to avoid can help you learn skills and or features, +they can become obsolete quickly. Nobody is obliged to maintain any of these +sites. Therefore, we should avoid using them as reference literature. + +NOTE: **Note:** +Non-authoritative sources are acceptable only if there is no equivalent +authoritative source. Even then, focus on non-authoritative sources that are +extensively cited or peer-reviewed. + +### Links requiring permissions + +Don't link directly to: + +- [Confidential issues](../../../user/project/issues/confidential_issues.md). +- Project features that require [special permissions](../../../user/permissions.md) + to view. + +These will fail for: + +- Those without sufficient permissions. +- Automated link checkers. + +Instead: + +- To reduce confusion, mention in the text that the information is either: + - Contained in a confidential issue. + - Requires special permission to a project to view. +- Provide a link in back ticks (`` ` ``) so that those with access to the issue + can navigate to it. + +Example: + +```markdown +For more information, see the [confidential issue](../../../user/project/issues/confidential_issues.md) `https://gitlab.com/gitlab-org/gitlab-foss/-/issues/<issue_number>`. +``` + +### Link to specific lines of code + +When linking to specific lines within a file, link to a commit instead of to the +branch. Lines of code change through time, therefore, linking to a line by using +the commit link ensures the user lands on the line you're referring to. The +**Permalink** button, which is available when viewing a file within a project, +makes it easy to generate a link to the most recent commit of the given file. + +- _Do_: `[link to line 3](https://gitlab.com/gitlab-org/gitlab/-/blob/11f17c56d8b7f0b752562d78a4298a3a95b5ce66/.gitlab/issue_templates/Feature%20proposal.md#L3)` +- _Don't_: `[link to line 3](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/Feature%20proposal.md#L3).` + +If that linked expression is no longer in that line of the file due to additional +commits, you can still search the file for that query. In this case, update the +document to ensure it links to the most recent version of the file. + +## Navigation + +When documenting navigation through the user interface: + +- Use the exact wording as shown in the UI, including any capital letters as-is. +- Use bold text for navigation items and the char "greater than" (`>`) as a + separator. For example: `Navigate to your project's **Settings > CI/CD**`. +- If there are any expandable menus, make sure to mention that the user needs to + expand the tab to find the settings you're referring to. For example: + `Navigate to your project's **Settings > CI/CD** and expand **General pipelines**`. + +### Navigational elements + +Use the following terms when referring to the main GitLab user interface +elements: + +- **Top menu**: This is the top menu that spans the width of the user interface. + It includes the GitLab logo, search field, counters, and the user's avatar. +- **Left sidebar**: This is the navigation sidebar on the left of the user + interface, specific to the project or group. +- **Right sidebar**: This is the navigation sidebar on the right of the user + interface, specific to the open issue, merge request, or epic. + +## Images + +Images, including screenshots, can help a reader better understand a concept. +However, they can be hard to maintain, and should be used sparingly. + +Before including an image in the documentation, ensure it provides value to the +reader. + +### Capture the image + +Use images to help the reader understand where they are in a process, or how +they need to interact with the application. + +When you take screenshots: + +- _Capture the most relevant area of the page._ Don't include unnecessary white + space or areas of the page that don't help illustrate the point. The left + sidebar of the GitLab user interface can change, so don't include the sidebar + if it's not necessary. +- _Keep it small._ If you don't need to show the full width of the screen, don't. + A value of 1000 pixels is a good maximum width for your screenshot image. +- _Be consistent._ Coordinate screenshots with the other screenshots already on + a documentation page. For example, if other screenshots include the left + sidebar, include the sidebar in all screenshots. + +### Save the image + +- Save the image with a lowercase file name that's descriptive of the feature + or concept in the image. If the image is of the GitLab interface, append the + GitLab version to the file name, based on the following format: + `image_name_vX_Y.png`. For example, for a screenshot taken from the pipelines + page of GitLab 11.1, a valid name is `pipelines_v11_1.png`. If you're adding an + illustration that doesn't include parts of the user interface, add the release + number corresponding to the release the image was added to; for an MR added to + 11.1's milestone, a valid name for an illustration is `devops_diagram_v11_1.png`. +- Place images in a separate directory named `img/` in the same directory where + the `.md` document that you're working on is located. +- Consider using PNG images instead of JPEG. +- [Compress all PNG images](#compress-images). +- Compress gifs with <https://ezgif.com/optimize> or similar tool. +- Images should be used (only when necessary) to _illustrate_ the description + of a process, not to _replace_ it. +- Max image size: 100KB (gifs included). +- See also how to link and embed [videos](#videos) to illustrate the + documentation. + +### Add the image link to content + +The Markdown code for including an image in a document is: +`` + +The image description is the alt text for the rendered image on the +documentation site. For accessibility and SEO, use [descriptions](https://webaim.org/techniques/alttext/) +that: + +- Are accurate, succinct, and unique. +- Don't use _image of…_ or _graphic of…_ to describe the image. + +### Compress images + +You should always compress any new images you add to the documentation. One +known tool is [`pngquant`](https://pngquant.org/), which is cross-platform and +open source. Install it by visiting the official website and following the +instructions for your OS. + +GitLab has a [Rake task](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/tasks/pngquant.rake) +that you can use to automate the process. In the root directory of your local +copy of `https://gitlab.com/gitlab-org/gitlab`, run in a terminal: + +- Before compressing, if you want, check that all documentation PNG images have + been compressed: + + ```shell + bundle exec rake pngquant:lint + ``` + +- Compress all documentation PNG images using `pngquant`: + + ```shell + bundle exec rake pngquant:compress + ``` + +The only caveat is that the task runs on all images under `doc/`, not only the +ones you might have included in a merge request. In that case, you can run the +compress task and only commit the images that are relevant to your merge +request. + +## Videos + +Adding GitLab's existing YouTube video tutorials to the documentation is highly +encouraged, unless the video is outdated. Videos should not replace +documentation, but complement or illustrate it. If content in a video is +fundamental to a feature and its key use cases, but this is not adequately +covered in the documentation, add this detail to the documentation text or +create an issue to review the video and do so. + +Do not upload videos to the product repositories. [Link](#link-to-video) or +[embed](#embed-videos) them instead. + +### Link to video + +To link out to a video, include a YouTube icon so that readers can scan the page +for videos before reading: + +```markdown +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +For an overview, see [Video Title](link-to-video). +``` + +You can link any up-to-date video that's useful to the GitLab user. + +### Embed videos + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-docs/-/merge_requests/472) in GitLab 12.1. + +The [GitLab documentation site](https://docs.gitlab.com) supports embedded +videos. + +You can only embed videos from [GitLab's official YouTube account](https://www.youtube.com/channel/UCnMGQ8QHMAnVIsI3xJrihhg). +For videos from other sources, [link](#link-to-video) them instead. + +In most cases, it is better to [link to video](#link-to-video) instead, because +an embed takes up a lot of space on the page and can be distracting to readers. + +To embed a video: + +1. Copy the code from this procedure and paste it into your Markdown file. Leave a + blank line above and below it. Do _not_ edit the code (don't remove or add any spaces). +1. In YouTube, visit the video URL you want to display. Copy the regular URL + from your browser (`https://www.youtube.com/watch?v=VIDEO-ID`) and replace + the video title and link in the line under `<div class="video-fallback">`. +1. In YouTube, select **Share**, and then select **Embed**. +1. Copy the `<iframe>` source (`src`) **URL only** + (`https://www.youtube.com/embed/VIDEO-ID`), + and paste it, replacing the content of the `src` field in the + `iframe` tag. + +```html +leave a blank line here +<div class="video-fallback"> + See the video: <a href="https://www.youtube.com/watch?v=MqL6BMOySIQ">Video title</a>. +</div> +<figure class="video-container"> + <iframe src="https://www.youtube.com/embed/MqL6BMOySIQ" frameborder="0" allowfullscreen="true"> </iframe> +</figure> +leave a blank line here +``` + +This is how it renders on the GitLab documentation site: + +<div class="video-fallback"> + See the video: <a href="https://www.youtube.com/watch?v=enMumwvLAug">What is GitLab</a>. +</div> +<figure class="video-container"> + <iframe src="https://www.youtube.com/embed/MqL6BMOySIQ" frameborder="0" allowfullscreen="true"> </iframe> +</figure> + +> Notes: +> +> - The `figure` tag is required for semantic SEO and the `video_container` +class is necessary to make sure the video is responsive and displays on +different mobile devices. +> - The `<div class="video-fallback">` is a fallback necessary for GitLab's +`/help`, as GitLab's Markdown processor does not support iframes. It's hidden on +the documentation site, but will be displayed on GitLab's `/help`. + +## Code blocks + +- Always wrap code added to a sentence in inline code blocks (`` ` ``). + For example, `.gitlab-ci.yml`, `git add .`, `CODEOWNERS`, or `only: [master]`. + File names, commands, entries, and anything that refers to code should be + added to code blocks. To make things easier for the user, always add a full + code block for things that can be useful to copy and paste, as they can do it + with the button on code blocks. +- HTTP methods (`HTTP POST`) and HTTP status codes, both full (`404 File Not Found`) + and abbreviated (`404`), should be wrapped in inline code blocks when used in sentences. + For example: Send a `DELETE` request to delete the runner. Send a `POST` request to create one. +- Add a blank line above and below code blocks. +- When providing a shell command and its output, prefix the shell command with `$` + and leave a blank line between the command and the output. +- When providing a command without output, don't prefix the shell command with `$`. +- If you need to include triple backticks inside a code block, use four backticks + for the codeblock fences instead of three. +- For regular fenced code blocks, always use a highlighting class corresponding to + the language for better readability. Examples: + + ````markdown + ```ruby + Ruby code + ``` + + ```javascript + JavaScript code + ``` + + ```markdown + [Markdown code example](example.md) + ``` + + ```plaintext + Code or text for which no specific highlighting class is available. + ``` + ```` + +Syntax highlighting is required for fenced code blocks added to the GitLab +documentation. Refer to the following table for the most common language classes, +or check the [complete list](https://github.com/rouge-ruby/rouge/wiki/List-of-supported-languages-and-lexers) +of available language classes: + +| Preferred language tags | Language aliases and notes | +|-------------------------|------------------------------------------------------------------------------| +| `asciidoc` | | +| `dockerfile` | Alias: `docker`. | +| `elixir` | | +| `erb` | | +| `golang` | Alias: `go`. | +| `graphql` | | +| `haml` | | +| `html` | | +| `ini` | For some simple config files that are not in TOML format. | +| `javascript` | Alias `js`. | +| `json` | | +| `markdown` | Alias: `md`. | +| `mermaid` | | +| `nginx` | | +| `perl` | | +| `php` | | +| `plaintext` | Examples with no defined language, such as output from shell commands or API calls. If a codeblock has no language, it defaults to `plaintext`. Alias: `text`. | +| `prometheus` | Prometheus configuration examples. | +| `python` | | +| `ruby` | Alias: `rb`. | +| `shell` | Aliases: `bash` or `sh`. | +| `sql` | | +| `toml` | Runner configuration examples, and other TOML-formatted configuration files. | +| `typescript` | Alias: `ts`. | +| `xml` | | +| `yaml` | Alias: `yml`. | + +For a complete reference on code blocks, see the [Kramdown guide](https://about.gitlab.com/handbook/markdown-guide/#code-blocks). + +## GitLab SVG icons + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-docs/-/issues/384) in GitLab 12.7. + +You can use icons from the [GitLab SVG library](https://gitlab-org.gitlab.io/gitlab-svgs/) +directly in the documentation. + +This way, you can achieve a consistent look when writing about interacting with +GitLab user interface elements. + +Usage examples: + +- Icon with default size (16px): `**{icon-name}**` + + Example: `**{tanuki}**` renders as: **{tanuki}**. +- Icon with custom size: `**{icon-name, size}**` + + Available sizes (in px): 8, 10, 12, 14, 16, 18, 24, 32, 48, and 72 + + Example: `**{tanuki, 24}**` renders as: **{tanuki, 24}**. +- Icon with custom size and class: `**{icon-name, size, class-name}**`. + + You can access any class available to this element in GitLab documentation CSS. + + Example with `float-right`, a + [Bootstrap utility class](https://getbootstrap.com/docs/4.4/utilities/float/): + `**{tanuki, 32, float-right}**` renders as: **{tanuki, 32, float-right}** + +### When to use icons + +Icons should be used sparingly, and only in ways that aid and do not hinder the +readability of the text. + +For example, the following adds little to the accompanying text: + +```markdown +1. Go to **{home}** **Project overview > Details** +``` + +1. Go to **{home}** **Project overview > Details** + +However, the following might help the reader connect the text to the user +interface: + +```markdown +| Section | Description | +|:-------------------------|:----------------------------------------------------------------------------------------------------------------------------| +| **{overview}** Overview | View your GitLab Dashboard, and administer projects, users, groups, jobs, runners, and Gitaly servers. | +| **{monitor}** Monitoring | View GitLab system information, and information on background jobs, logs, health checks, requests profiles, and audit logs. | +| **{messages}** Messages | Send and manage broadcast messages for your users. | +``` + +| Section | Description | +|:-------------------------|:----------------------------------------------------------------------------------------------------------------------------| +| **{overview}** Overview | View your GitLab Dashboard, and administer projects, users, groups, jobs, runners, and Gitaly servers. | +| **{monitor}** Monitoring | View GitLab system information, and information on background jobs, logs, health checks, requests profiles, and audit logs. | +| **{messages}** Messages | Send and manage broadcast messages for your users. | + +Use an icon when you find yourself having to describe an interface element. For +example: + +- Do: Select the Admin Area icon ( **{admin}** ). +- Don't: Select the Admin Area icon (the wrench icon). + +## Alert boxes + +When you need to call special attention to particular sentences, use the +following markup to create highlighted alert boxes. + +Alert boxes work for one paragraph only. Multiple paragraphs, lists, and headers +won't render correctly. For multiple lines, use [blockquotes](#blockquotes) +instead. + +Alert boxes render only on the GitLab documentation site (<https://docs.gitlab.com>). +In the GitLab product help, alert boxes appear as plain Markdown text. + +These alert boxes are used in the GitLab documentation. These aren't strict +guidelines, but for consistency you should try to use these values: + +| Color | Markup | Default keyword | Alternative keywords | +|--------|------------|-----------------|----------------------------------------------------------------------| +| Blue | `NOTE:` | `**Note:**` | | +| Yellow | `CAUTION:` | `**Caution:**` | `**Warning:**`, `**Important:**` | +| Red | `DANGER:` | `**Danger:**` | `**Warning:**`, `**Important:**`, `**Deprecated:**`, `**Required:**` | +| Green | `TIP:` | `**Tip:**` | | + +### Note + +Notes indicate additional information that's of special use to the reader. +Notes are most effective when used _sparingly_. + +Try to avoid them. Too many notes can impact the scannability of a topic and +create an overly busy page. + +Instead of adding a note, try one of these alternatives: + +- Re-write the sentence as part of the most-relevant paragraph. +- Put the information into its own standalone paragraph. +- Put the content under a new subheading that introduces the topic, which makes + it more visible. + +If you must use a note, use the following formatting: + +```markdown +NOTE: **Note:** +This is something to note. +``` + +How it renders on the GitLab documentation site: + +NOTE: **Note:** +This is something to note. + +### Tip + +```markdown +TIP: **Tip:** +This is a tip. +``` + +How it renders on the GitLab documentation site: + +TIP: **Tip:** +This is a tip. + +### Caution + +```markdown +CAUTION: **Caution:** +This is something to be cautious about. +``` + +How it renders on the GitLab documentation site: + +CAUTION: **Caution:** +This is something to be cautious about. + +### Danger + +```markdown +DANGER: **Warning:** +This is a breaking change, a bug, or something very important to note. +``` + +How it renders on the GitLab documentation site: + +DANGER: **Warning:** +This is a breaking change, a bug, or something very important to note. + +## Blockquotes + +For highlighting a text within a blue blockquote, use this format: + +```markdown +> This is a blockquote. +``` + +which renders on the [GitLab documentation site](https://docs.gitlab.com) as: + +> This is a blockquote. + +If the text spans across multiple lines it's OK to split the line. + +For multiple paragraphs, use the symbol `>` before every line: + +```markdown +> This is the first paragraph. +> +> This is the second paragraph. +> +> - This is a list item +> - Second item in the list +``` + +Which renders to: + +> This is the first paragraph. +> +> This is the second paragraph. +> +> - This is a list item +> - Second item in the list + +## Terms + +To maintain consistency through GitLab documentation, the following guides +documentation authors on agreed styles and usage of terms. + +### Merge requests (MRs) + +Merge requests allow you to exchange changes you made to source code and +collaborate with other people on the same project. You'll see this term used in +the following ways: + +- Use lowercase _merge requests_ regardless of whether referring to the feature + or individual merge requests. + +As noted in the GitLab [Writing Style Guidelines](https://about.gitlab.com/handbook/communication/#writing-style-guidelines), +if you use the _MR_ acronym, expand it at least once per document page. +Typically, the first use would be phrased as _merge request (MR)_ with subsequent +instances being _MR_. + +Examples: + +- "We prefer GitLab merge requests". +- "Open a merge request to fix a broken link". +- "After you open a merge request (MR), submit your MR for review and approval". + +### Describe UI elements + +The following are styles to follow when describing user interface elements in an +application: + +- For elements with a visible label, use that label in bold with matching case. + For example, `the **Cancel** button`. +- For elements with a tooltip or hover label, use that label in bold with + matching case. For example, `the **Add status emoji** button`. + +### Verbs for UI elements + +The following are recommended verbs for specific uses with user interface +elements: + +| Recommended | Used for | Replaces | +|:--------------------|:--------------------------------------|:---------------------------| +| _select_ | buttons, links, menu items, dropdowns | "click, "press," "hit" | +| _select_ or _clear_ | checkboxes | "enable", "click", "press" | +| _expand_ | expandable sections | "open" | + +### Other Verbs + +| Recommended | Used for | Replaces | +|:------------|:--------------------------------|:----------------------| +| _go to_ | making a browser go to location | "navigate to", "open" | + +## GitLab versions and tiers + +Tagged and released versions of GitLab documentation are available: + +- In the [documentation archives](https://docs.gitlab.com/archives/). +- At the `/help` URL for any GitLab installation. + +The version introducing a new feature is added to the top of the topic in the +documentation to provide a link back to how the feature was developed. + +TIP: **Tip:** +Whenever you have documentation related to the `gitlab.rb` file, you're working +with a self-managed installation. The section or page is therefore likely to +apply only to self-managed instances. If so, the relevant "`TIER` ONLY" +[Product badge](#product-badges) should be included at the highest applicable +heading level. + +### Text for documentation requiring version text + +When a feature is new or updated, you can add version information as a bulleted +item in the **Version history**, or as an inline reference with related text. + +#### Version text in the **Version History** + +If all content in a section is related, add version text following the header for +the section. Each entry should be on a single line. To render correctly, it must be on its +own line and surrounded by blank lines. + +Features should declare the GitLab version that introduced a feature in a blockquote +following the header: + +```markdown +## Feature name + +> Introduced in GitLab 11.3. + +This feature does something. +``` + +Whenever possible, version text should have a link to the _completed_ issue, +merge request, or epic that introduced the feature. An issue is preferred over +a merge request, and a merge request is preferred over an epic. For example: + +```markdown +> [Introduced](<link-to-issue>) in GitLab 11.3. +``` + +If the feature is only available in GitLab Enterprise Edition, mention +the [paid tier](https://about.gitlab.com/handbook/marketing/strategic-marketing/tiers/) +the feature is available in: + +```markdown +> [Introduced](<link-to-issue>) in [GitLab Starter](https://about.gitlab.com/pricing/) 11.3. +``` + +If listing information for multiple version as a feature evolves, add the +information to a block-quoted bullet list. For example: + +```markdown +> - [Introduced](<link-to-issue>) in GitLab 11.3. +> - Enabled by default in GitLab 11.4. +``` + +If a feature is moved to another tier: + +```markdown +> - [Introduced](<link-to-issue>) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.5. +> - [Moved](<link-to-issue>) to [GitLab Starter](https://about.gitlab.com/pricing/) in 11.8. +> - [Moved](<link-to-issue>) to GitLab Core in 12.0. +``` + +If a feature is deprecated, include a link to a replacement (when available): + +```markdown +> - [Deprecated](<link-to-issue>) in GitLab 11.3. Replaced by [meaningful text](<link-to-appropriate-documentation>). +``` + +You can also describe the replacement in surrounding text, if available. + +If the deprecation is not obvious in existing text, you may want to include a +warning such as: + +```markdown +DANGER: **Deprecated:** +This feature was [deprecated](link-to-issue) in GitLab 12.3 +and replaced by [Feature name](link-to-feature-documentation). +``` + +#### Inline version text + +If you're adding content to an existing topic, you can add version information +inline with the existing text. + +In this case, add `([introduced/deprecated](<link-to-issue>) in GitLab X.X)`. +If applicable, include the paid tier: `([introduced/deprecated](<link-to-issue>) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.4)` + +Including the issue link is encouraged, but isn't a requirement. For example: + +```markdown +The voting strategy (in GitLab 13.4 and later) requires +the primary and secondary voters to agree. +``` + +#### End-of-life for features or products + +Whenever a feature or product enters the end-of-life process, indicate its +status by using the `Danger` [alert](#alert-boxes) with the `**Important**` +keyword directly below the feature or product's header (which can include H1 +page titles). Link to the deprecation and removal issues, if possible. + +For example: + +```markdown +DANGER: **Important:** +This feature is in its end-of-life process. It is [deprecated](link-to-issue) +for use in GitLab X.X, and is planned for [removal](link-to-issue) in GitLab X.X. +``` + +### Versions in the past or future + +When describing functionality available in past or future versions, use: + +- _Earlier_, and not _older_ or _before_. +- _Later_, and not _newer_ or _after_. + +For example: + +- Available in GitLab 12.3 and earlier. +- Available in GitLab 12.4 and later. +- In GitLab 11.4 and earlier, ... +- In GitLab 10.6 and later, ... + +### Importance of referencing GitLab versions and tiers + +Mentioning GitLab versions and tiers is important to all users and contributors +to quickly have access to the issue or merge request that introduced the change. +Also, they can know what features they have in their GitLab +instance and version, given that the note has some key information. + +`[Introduced](link-to-issue) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.7` +links to the issue that introduced the feature, says which GitLab tier it +belongs to, says the GitLab version that it became available in, and links to +the pricing page in case the user wants to upgrade to a paid tier to use that +feature. + +For example, if you're a regular user and you're looking at the documentation +for a feature you haven't used before, you can immediately see if that feature +is available to you or not. Alternatively, if you've been using a certain +feature for a long time and it changed in some way, it's important to be able to +determine when it changed and what's new in that feature. + +This is even more important as we don't have a perfect process for shipping +documentation. Unfortunately, we still see features without documentation, and +documentation without features. So, for now, we cannot rely 100% on the +documentation site versions. + +Over time, version text will reference a progressively older version of GitLab. +In cases where version text refers to versions of GitLab four or more major +versions back, you can consider removing the text if it's irrelevant or confusing. + +For example, if the current major version is 12.x, version text referencing +versions of GitLab 8.x and older are candidates for removal if necessary for +clearer or cleaner documentation. + +## Products and features + +Refer to the information in this section when describing products and features +within the GitLab product documentation. + +### Avoid line breaks in names + +When entering a product or feature name that includes a space (such as +GitLab Community Edition) or even other companies' products (such as +Amazon Web Services), be sure to not split the product or feature name across +lines with an inserted line break. Splitting product or feature names across +lines makes searching for these items more difficult, and can cause problems if +names change. + +For example, the following Markdown content is _not_ formatted correctly: + +```markdown +When entering a product or feature name that includes a space (such as GitLab +Community Edition), don't split the product or feature name across lines. +``` + +Instead, it should appear similar to the following: + +```markdown +When entering a product or feature name that includes a space (such as +GitLab Community Edition), don't split the product or feature name across lines. +``` + +### Product badges + +When a feature is available in paid tiers, add the corresponding tier to the +header or other page element according to the feature's availability: + +| Tier in which feature is available | Tier markup | +|:-----------------------------------------------------------------------|:----------------------| +| GitLab Core and GitLab.com Free, and their higher tiers | `**(CORE)**` | +| GitLab Starter and GitLab.com Bronze, and their higher tiers | `**(STARTER)**` | +| GitLab Premium and GitLab.com Silver, and their higher tiers | `**(PREMIUM)**` | +| GitLab Ultimate and GitLab.com Gold | `**(ULTIMATE)**` | +| _Only_ GitLab Core and higher tiers (no GitLab.com-based tiers) | `**(CORE ONLY)**` | +| _Only_ GitLab Starter and higher tiers (no GitLab.com-based tiers) | `**(STARTER ONLY)**` | +| _Only_ GitLab Premium and higher tiers (no GitLab.com-based tiers) | `**(PREMIUM ONLY)**` | +| _Only_ GitLab Ultimate (no GitLab.com-based tiers) | `**(ULTIMATE ONLY)**` | +| _Only_ GitLab.com Free and higher tiers (no self-managed instances) | `**(FREE ONLY)**` | +| _Only_ GitLab.com Bronze and higher tiers (no self-managed instances) | `**(BRONZE ONLY)**` | +| _Only_ GitLab.com Silver and higher tiers (no self-managed instances) | `**(SILVER ONLY)**` | +| _Only_ GitLab.com Gold (no self-managed instances) | `**(GOLD ONLY)**` | + +For clarity, all page title headers (H1s) must be have a tier markup for the +lowest tier that has information on the documentation page. + +If sections of a page apply to higher tier levels, they can be separately +labeled with their own tier markup. + +#### Product badge display behavior + +When using the tier markup with headers, the documentation page will display the +full tier badge with the header line. + +You can also use the tier markup with paragraphs, list items, and table cells. +For these cases, the tier mention will be represented by an orange info icon +**{information}** that will display the tiers when visitors point to the icon. +For example: + +- `**(STARTER)**` displays as **(STARTER)** +- `**(STARTER ONLY)**` displays as **(STARTER ONLY)** +- `**(SILVER ONLY)**` displays as **(SILVER ONLY)** + +#### How it works + +Introduced by [!244](https://gitlab.com/gitlab-org/gitlab-docs/-/merge_requests/244), +the special markup `**(STARTER)**` will generate a `span` element to trigger the +badges and tooltips (`<span class="badge-trigger starter">`). When the keyword +_only_ is added, the corresponding GitLab.com badge will not be displayed. + +## Specific sections + +Certain styles should be applied to specific sections. Styles for specific +sections are outlined in this section. + +### GitLab restart + +There are many cases that a restart/reconfigure of GitLab is required. To avoid +duplication, link to the special document that can be found in +[`doc/administration/restart_gitlab.md`](../../../administration/restart_gitlab.md). +Usually the text will read like: + +```markdown +Save the file and [reconfigure GitLab](../../../administration/restart_gitlab.md) +for the changes to take effect. +``` + +If the document you are editing resides in a place other than the GitLab CE/EE +`doc/` directory, instead of the relative link, use the full path: +`https://docs.gitlab.com/ee/administration/restart_gitlab.html`. Replace +`reconfigure` with `restart` where appropriate. + +### Installation guide + +**Ruby:** +In [step 2 of the installation guide](../../../install/installation.md#2-ruby), +we install Ruby from source. Whenever there is a new version that needs to +be updated, remember to change it throughout the codeblock and also replace +the sha256sum (it can be found in the [downloads page](https://www.ruby-lang.org/en/downloads/) +of the Ruby website). + +### Configuration documentation for source and Omnibus installations + +GitLab currently officially supports two installation methods: installations +from source and Omnibus packages installations. + +Whenever there's a setting that's configurable for both installation methods, +the preference is to document it in the CE documentation to avoid duplication. + +Configuration settings include: + +- Settings that touch configuration files in `config/`. +- NGINX settings and settings in `lib/support/` in general. + +When you document a list of steps, it may entail editing the configuration file +and reconfiguring or restarting GitLab. In that case, use these styles: + +````markdown +**For Omnibus installations** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + external_url "https://gitlab.example.com" + ``` + +1. Save the file and [reconfigure](path/to/administration/restart_gitlab.md#omnibus-gitlab-reconfigure) + GitLab for the changes to take effect. + +--- + +**For installations from source** + +1. Edit `config/gitlab.yml`: + + ```yaml + gitlab: + host: "gitlab.example.com" + ``` + +1. Save the file and [restart](path/to/administration/restart_gitlab.md#installations-from-source) + GitLab for the changes to take effect. +```` + +In this case: + +- Before each step list the installation method is declared in bold. +- Three dashes (`---`) are used to create a horizontal line and separate the two + methods. +- The code blocks are indented one or more spaces under the list item to render + correctly. +- Different highlighting languages are used for each config in the code block. +- The [GitLab Restart](#gitlab-restart) section is used to explain a required + restart or reconfigure of GitLab. + +### Troubleshooting + +For troubleshooting sections, you should provide as much context as possible so +users can identify the problem they are facing and resolve it on their own. You +can facilitate this by making sure the troubleshooting content addresses: + +1. The problem the user needs to solve. +1. How the user can confirm they have the problem. +1. Steps the user can take towards resolution of the problem. + +If the contents of each category can be summarized in one line and a list of +steps aren't required, consider setting up a [table](#tables) with headers of +_Problem_ \| _Cause_ \| _Solution_ (or _Workaround_ if the fix is temporary), or +_Error message_ \| _Solution_. + +## Feature flags + +Learn how to [document features deployed behind flags](../feature_flags.md). For +guidance on developing GitLab with feature flags, see [Feature flags in development of GitLab](../../feature_flags/index.md). diff --git a/doc/development/documentation/testing.md b/doc/development/documentation/testing.md new file mode 100644 index 00000000000..043da38d207 --- /dev/null +++ b/doc/development/documentation/testing.md @@ -0,0 +1,292 @@ +--- +stage: none +group: Documentation Guidelines +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +description: Learn how to contribute to GitLab Documentation. +--- + +# Documentation testing + +We treat documentation as code, and so use tests in our CI pipeline to maintain the +standards and quality of the docs. The current tests, which run in CI jobs when a +merge request with new or changed docs is submitted, are: + +- [`docs lint`](https://gitlab.com/gitlab-org/gitlab/-/blob/0b562014f7b71f98540e682c8d662275f0011f2f/.gitlab/ci/docs.gitlab-ci.yml#L41): + Runs several tests on the content of the docs themselves: + - [`lint-doc.sh` script](https://gitlab.com/gitlab-org/gitlab/blob/master/scripts/lint-doc.sh) + runs the following checks and linters: + - All cURL examples use the long flags (ex: `--header`, not `-H`). + - The `CHANGELOG.md` does not contain duplicate versions. + - No files in `doc/` are executable. + - No new `README.md` was added. + - [markdownlint](#markdownlint). + - [Vale](#vale). + - Nanoc tests: + - [`internal_links`](https://gitlab.com/gitlab-org/gitlab/-/blob/0b562014f7b71f98540e682c8d662275f0011f2f/.gitlab/ci/docs.gitlab-ci.yml#L58) + checks that all internal links (ex: `[link](../index.md)`) are valid. + - [`internal_anchors`](https://gitlab.com/gitlab-org/gitlab/-/blob/0b562014f7b71f98540e682c8d662275f0011f2f/.gitlab/ci/docs.gitlab-ci.yml#L60) + checks that all internal anchors (ex: `[link](../index.md#internal_anchor)`) + are valid. + - [`ui-docs-links lint`](https://gitlab.com/gitlab-org/gitlab/-/blob/0b562014f7b71f98540e682c8d662275f0011f2f/.gitlab/ci/docs.gitlab-ci.yml#L62) + checks that all links to docs from UI elements (`app/views` files, for example) + are linking to valid docs and anchors. + +## Run tests locally + +Apart from [previewing your changes locally](index.md#previewing-the-changes-live), you can also run all lint checks +and Nanoc tests locally. + +### Lint checks + +Lint checks are performed by the [`lint-doc.sh`](https://gitlab.com/gitlab-org/gitlab/blob/master/scripts/lint-doc.sh) +script and can be executed as follows: + +1. Navigate to the `gitlab` directory. +1. Run: + + ```shell + MD_DOC_PATH=path/to/my_doc.md scripts/lint-doc.sh + ``` + +Where `MD_DOC_PATH` points to the file or directory you would like to run lint checks for. +If you omit it completely, it defaults to the `doc/` directory. +The output should be similar to: + +```plaintext +=> Linting documents at path /path/to/gitlab as <user>... +=> Checking for cURL short options... +=> Checking for CHANGELOG.md duplicate entries... +=> Checking /path/to/gitlab/doc for executable permissions... +=> Checking for new README.md files... +=> Linting markdown style... +=> Linting prose... +✔ 0 errors, 0 warnings and 0 suggestions in 1 file. +✔ Linting passed +``` + +This requires you to either: + +- Have the required lint tools installed on your machine. +- A working Docker installation, in which case an image with these tools pre-installed is used. + +### Nanoc tests + +To execute Nanoc tests locally: + +1. Navigate to the [`gitlab-docs`](https://gitlab.com/gitlab-org/gitlab-docs) directory. +1. Run: + + ```shell + # Check for broken internal links + bundle exec nanoc check internal_links + + # Check for broken external links (might take a lot of time to complete). + # This test is set to be allowed to fail and is run only in the gitlab-docs project CI + bundle exec nanoc check internal_anchors + ``` + +### `ui-docs-links` test + +The `ui-docs-links lint` job uses `haml-lint` to test that all links to docs from +UI elements (`app/views` files, for example) are linking to valid docs and anchors. + +To run the `ui-docs-links` test locally: + +1. Open the `gitlab` directory in a terminal window. +1. Run: + + ```shell + bundle exec haml-lint -i DocumentationLinks + ``` + +If you receive an error the first time you run this test, run `bundle install`, which +installs GitLab's dependencies, and try again. + +If you don't want to install all of GitLab's dependencies to test the links, you can: + +1. Open the `gitlab` directory in a terminal window. +1. Install `haml-lint`: + + ```shell + gem install haml_lint + ``` + +1. Run: + + ```shell + haml-lint -i DocumentationLinks + ``` + +If you manually install `haml-lint` with this process, it does not update automatically +and you should make sure your version matches the version used by GitLab. + +## Local linters + +To help adhere to the [documentation style guidelines](styleguide/index.md), and improve the content +added to documentation, [install documentation linters](#install-linters) and +[integrate them with your code editor](#configure-editors). + +At GitLab, we mostly use: + +- [markdownlint](#markdownlint) +- [Vale](#vale) + +### markdownlint + +[markdownlint](https://github.com/DavidAnson/markdownlint) checks that Markdown syntax follows +[certain rules](https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#rules), and is +used by the `docs-lint` test. + +Our [Documentation Style Guide](styleguide/index.md#markdown) and +[Markdown Guide](https://about.gitlab.com/handbook/markdown-guide/) elaborate on which choices must +be made when selecting Markdown syntax for GitLab documentation. This tool helps catch deviations +from those guidelines. + +markdownlint configuration is found in the following projects: + +- [`gitlab`](https://gitlab.com/gitlab-org/gitlab/blob/master/.markdownlint.json) +- [`gitlab-runner`](https://gitlab.com/gitlab-org/gitlab-runner/blob/master/.markdownlint.json) +- [`omnibus-gitlab`](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/.markdownlint.json) +- [`charts`](https://gitlab.com/gitlab-org/charts/gitlab/-/blob/master/.markdownlint.json) +- [`gitlab-development-kit`](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/master/.markdownlint.json) + +This configuration is also used within build pipelines. + +You can use markdownlint: + +- [On the command line](https://github.com/igorshubovych/markdownlint-cli#markdownlint-cli--). +- [Within a code editor](#configure-editors). +- [In a `pre-push` hook](#configure-pre-push-hooks). + +### Vale + +[Vale](https://errata-ai.gitbook.io/vale/) is a grammar, style, and word usage linter for the +English language. Vale's configuration is stored in the +[`.vale.ini`](https://gitlab.com/gitlab-org/gitlab/blob/master/.vale.ini) file located in the root +directory of projects. + +Vale supports creating [custom tests](https://errata-ai.github.io/vale/styles/) that extend any of +several types of checks, which we store in the `.linting/vale/styles/gitlab` directory within the +documentation directory of projects. + +Vale configuration is found in the following projects: + +- [`gitlab`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/doc/.vale/gitlab) +- [`gitlab-runner`](https://gitlab.com/gitlab-org/gitlab-runner/-/tree/master/docs/.vale/gitlab) +- [`omnibus-gitlab`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/tree/master/doc/.vale/gitlab) +- [`charts`](https://gitlab.com/gitlab-org/charts/gitlab/-/tree/master/doc/.vale/gitlab) +- [`gitlab-development-kit`](https://gitlab.com/gitlab-org/gitlab-development-kit/-/tree/master/doc/.vale/gitlab) + +This configuration is also used within build pipelines. + +You can use Vale: + +- [On the command line](https://errata-ai.gitbook.io/vale/getting-started/usage). +- [Within a code editor](#configure-editors). +- [In a Git hook](#configure-pre-push-hooks). Vale only reports errors in the Git hook (the same + configuration as the CI/CD pipelines), and does not report suggestions or warnings. + +### Install linters + +At a minimum, install [markdownlint](#markdownlint) and [Vale](#vale) to match the checks run in +build pipelines: + +1. Install `markdownlint-cli`, using either: + + - `npm`: + + ```shell + npm install -g markdownlint-cli + ``` + + - `yarn`: + + ```shell + yarn global add markdownlint-cli + ``` + + We recommend installing the version of `markdownlint-cli` currently used in the documentation + linting [Docker image](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/master/.gitlab-ci.yml#L420). + +1. Install [`vale`](https://github.com/errata-ai/vale/releases). For example, to install using + `brew` for macOS, run: + + ```shell + brew install vale + ``` + + We recommend installing the version of Vale currently used in the documentation linting + [Docker image](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/master/.gitlab-ci.yml#L419). + +In addition to using markdownlint and Vale at the command line, these tools can be +[integrated with your code editor](#configure-editors). + +### Configure editors + +To configure markdownlint within your editor, install one of the following as appropriate: + +- [Sublime Text](https://packagecontrol.io/packages/SublimeLinter-contrib-markdownlint) +- [Visual Studio Code](https://marketplace.visualstudio.com/items?itemName=DavidAnson.vscode-markdownlint) +- [Atom](https://atom.io/packages/linter-node-markdownlint) +- [Vim](https://github.com/dense-analysis/ale) + +To configure Vale within your editor, install one of the following as appropriate: + +- The Sublime Text [`SublimeLinter-contrib-vale` plugin](https://packagecontrol.io/packages/SublimeLinter-contrib-vale). +- The Visual Studio Code [`errata-ai.vale-server` extension](https://marketplace.visualstudio.com/items?itemName=errata-ai.vale-server). + You don't need Vale Server to use the plugin. You can configure the plugin to + [display only a subset of alerts](#show-subset-of-vale-alerts). +- [Vim](https://github.com/dense-analysis/ale). + +We don't use [Vale Server](https://errata-ai.github.io/vale/#using-vale-with-a-text-editor-or-another-third-party-application). + +### Configure pre-push hooks + +Git [pre-push hooks](https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks) allow Git users to: + +- Run tests or other processes before pushing a branch. +- Avoid pushing a branch if failures occur with these tests. + +[`lefthook`](https://github.com/Arkweid/lefthook) is a Git hooks manager, making configuring, +installing, and removing Git hooks easy. + +Configuration for `lefthook` is available in the [`lefthook.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lefthook.yml) +file for the [`gitlab`](https://gitlab.com/gitlab-org/gitlab) project. + +To set up `lefthook` for documentation linting, see +[Pre-push static analysis](../contributing/style_guides.md#pre-push-static-analysis). + +### Show subset of Vale alerts + +You can set Visual Studio Code to display only a subset of Vale alerts when viewing files: + +1. Go to **Preferences > Settings > Extensions > Vale**. +1. In **Vale CLI: Min Alert Level**, select the minimum alert level you want displayed in files. + +To display only a subset of Vale alerts when running Vale from the command line, use +the `--minAlertLevel` flag, which accepts `error`, `warning`, or `suggestion`. Combine it with `--config` +to point to the configuration file within the project, if needed: + +```shell +vale --config .vale.ini --minAlertLevel error doc/**/*.md +``` + +Omit the flag to display all alerts, including `suggestion` level alerts. + +### Disable Vale tests + +You can disable a specific Vale linting rule or all Vale linting rules for any portion of a +document: + +- To disable a specific rule, add a `<!-- vale gitlab.rulename = NO -->` tag before the text, and a + `<!-- vale gitlab.rulename = YES -->` tag after the text, replacing `rulename` with the filename + of a test in the + [GitLab styles](https://gitlab.com/gitlab-org/gitlab/-/tree/master/doc/.linting/vale/styles/gitlab) + directory. +- To disable all Vale linting rules, add a `<!-- vale off -->` tag before the text, and a + `<!-- vale on -->` tag after the text. + +Whenever possible, exclude only the problematic rule and line(s). + +For more information, see +[Vale's documentation](https://errata-ai.gitbook.io/vale/getting-started/markup#markup-based-configuration). diff --git a/doc/development/documentation/workflow.md b/doc/development/documentation/workflow.md index 488c71a6328..9bc80116d25 100644 --- a/doc/development/documentation/workflow.md +++ b/doc/development/documentation/workflow.md @@ -1,10 +1,15 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Documentation process The process for creating and maintaining GitLab product documentation allows anyone to contribute a merge request or create an issue for GitLab's documentation. -NOTE: **Note:** Documentation updates relating to new features or feature enhancements must use the [feature workflow process](https://about.gitlab.com/handbook/engineering/ux/technical-writing/workflow/#for-a-product-change) described in the GitLab Handbook. @@ -53,7 +58,7 @@ To update GitLab documentation: [GitLab Documentation guidelines](index.md) page. 1. Follow the described standards and processes listed on the page, including: - The [Structure and template](structure.md) page. - - The [Style Guide](styleguide.md). + - The [Style Guide](styleguide/index.md). - The [Markdown Guide](https://about.gitlab.com/handbook/markdown-guide/). 1. Follow GitLab's [Merge Request Guidelines](../contributing/merge_request_workflow.md#merge-request-guidelines). @@ -82,7 +87,7 @@ Anyone with Maintainer access to the relevant GitLab project can merge documenta Maintainers must make a good-faith effort to ensure that the content: - Is clear and sufficiently easy for the intended audience to navigate and understand. -- Meets the [Documentation Guidelines](index.md) and [Style Guide](styleguide.md). +- Meets the [Documentation Guidelines](index.md) and [Style Guide](styleguide/index.md). If the author or reviewer has any questions, they can mention the writer who is assigned to the relevant [DevOps stage group](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments). @@ -149,16 +154,15 @@ Remember: Ensure the following if skipping an initial Technical Writer review: -- That [product badges](styleguide.md#product-badges) are applied. -- That the GitLab [version](styleguide.md#text-for-documentation-requiring-version-text) that +- That [product badges](styleguide/index.md#product-badges) are applied. +- That the GitLab [version](styleguide/index.md#text-for-documentation-requiring-version-text) that introduced the feature has been included. - That changes to headings don't affect in-app hyperlinks. - Specific [user permissions](../../user/permissions.md) are documented. - That new documents are linked from higher-level indexes, for discoverability. - Style guide is followed: - - For [directories and files](styleguide.md#work-with-directories-and-files). - - For [images](styleguide.md#images). + - For [directories and files](styleguide/index.md#work-with-directories-and-files). + - For [images](styleguide/index.md#images). -NOTE: **Note:** Merge requests that change the location of documentation must always be reviewed by a Technical Writer prior to merging. diff --git a/doc/development/ee_features.md b/doc/development/ee_features.md index 01f9d9b16fb..26a1e9ec3aa 100644 --- a/doc/development/ee_features.md +++ b/doc/development/ee_features.md @@ -1,9 +1,15 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Guidelines for implementing Enterprise Edition features - **Write the code and the tests.**: As with any code, EE features should have good test coverage to prevent regressions. - **Write documentation.**: Add documentation to the `doc/` directory. Describe - the feature and include screenshots, if applicable. Indicate [what editions](documentation/styleguide.md#product-badges) + the feature and include screenshots, if applicable. Indicate [what editions](documentation/styleguide/index.md#product-badges) the feature applies to. - **Submit a MR to the `www-gitlab-com` project.**: Add the new feature to the [EE features list](https://about.gitlab.com/features/). @@ -426,6 +432,38 @@ module EE end ``` +### Code in `app/graphql/` + +EE-specific mutations, resolvers, and types should be added to +`ee/app/graphql/{mutations,resolvers,types}`. + +To override a CE mutation, resolver, or type, create the file in +`ee/app/graphql/ee/{mutations,resolvers,types}` and add new code to a +`prepended` block. + +For example, if CE has a mutation called `Mutations::Tanukis::Create` and you +wanted to add a new argument, place the EE override in +`ee/app/graphql/ee/mutations/tanukis/create.rb`: + +```ruby +module EE + module Mutations + module Tanukis + module Create + extend ActiveSupport::Concern + + prepended do + argument :name, + GraphQL::STRING_TYPE, + required: false, + description: 'Tanuki name' + end + end + end + end +end +``` + #### Using `render_if_exists` Instead of using regular `render`, we should use `render_if_exists`, which @@ -535,7 +573,7 @@ constants. #### EE parameters -We can define `params` and utilize `use` in another `params` definition to +We can define `params` and use `use` in another `params` definition to include parameters defined in EE. However, we need to define the "interface" first in CE in order for EE to override it. We don't have to do this in other places due to `prepend_if_ee`, but Grape is complex internally and we couldn't easily diff --git a/doc/development/elasticsearch.md b/doc/development/elasticsearch.md index 639759e5014..cde221aaf16 100644 --- a/doc/development/elasticsearch.md +++ b/doc/development/elasticsearch.md @@ -184,6 +184,38 @@ If the current version is `v12p1`, and we need to create a new version for `v12p 1. Change the namespace for files under `v12p1` folder from `Latest` to `V12p1` 1. Make changes to files under the `latest` folder as needed +## Creating a new Global Search migration + +> This functionality was introduced by [#234046](https://gitlab.com/gitlab-org/gitlab/-/issues/234046). + +NOTE: **Note:** +This only supported for indices created with GitLab 13.0 or greater. + +Migrations are stored in the [`ee/elastic/migrate/`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/elastic/migrate) folder with `YYYYMMDDHHMMSS_migration_name.rb` +file name format, which is similar to Rails database migrations: + +```ruby +# frozen_string_literal: true + +class MigrationName < Elastic::Migration + # Important: Any update to the Elastic index mappings should be replicated in Elastic::Latest::Config + + def migrate + end + + # Check if the migration has completed + # Return true if completed, otherwise return false + def completed? + end +end +``` + +Applied migrations are stored in `gitlab-#{RAILS_ENV}-migrations` index. All unexecuted migrations +are applied by the [`Elastic::MigrationWorker`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/app/workers/elastic/migration_worker.rb) +cron worker sequentially. + +Any update to the Elastic index mappings should be replicated in [`Elastic::Latest::Config`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/elastic/latest/config.rb). + ## Performance Monitoring ### Prometheus @@ -234,7 +266,7 @@ which is useful to diagnose why a search might be slow. ### Correlation ID and `X-Opaque-Id` Our [correlation -ID](./distributed_tracing.md#developer-guidelines-for-working-with-correlation-ids) +ID](distributed_tracing.md#developer-guidelines-for-working-with-correlation-ids) is forwarded by all requests from Rails to Elasticsearch as the [`X-Opaque-Id`](https://www.elastic.co/guide/en/elasticsearch/reference/current/tasks.html#_identifying_running_tasks) header which allows us to track any diff --git a/doc/development/emails.md b/doc/development/emails.md index de9607fef64..4b43bff0e02 100644 --- a/doc/development/emails.md +++ b/doc/development/emails.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Dealing with email in development ## Ensuring compatibility with mailer Sidekiq jobs diff --git a/doc/development/experiment_guide/index.md b/doc/development/experiment_guide/index.md index 18b606450c2..3b2f1d21463 100644 --- a/doc/development/experiment_guide/index.md +++ b/doc/development/experiment_guide/index.md @@ -1,3 +1,9 @@ +--- +stage: Growth +group: Activation +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Experiment Guide Experiments can be conducted by any GitLab team, most often the teams from the [Growth Sub-department](https://about.gitlab.com/handbook/engineering/development/growth/). Experiments are not tied to releases because they will primarily target GitLab.com. @@ -32,7 +38,7 @@ addressed. ## How to create an A/B test -### Implementation +### Implement the experiment 1. Add the experiment to the `Gitlab::Experimentation::EXPERIMENTS` hash in [`experimentation.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib%2Fgitlab%2Fexperimentation.rb): @@ -44,7 +50,7 @@ addressed. # Add your experiment here: signup_flow: { environment: ::Gitlab.dev_env_or_com?, # Target environment, defaults to enabled for development and GitLab.com - tracking_category: 'Growth::Acquisition::Experiment::SignUpFlow' # Used for providing the category when setting up tracking data + tracking_category: 'Growth::Activation::Experiment::SignUpFlow' # Used for providing the category when setting up tracking data } }.freeze ``` @@ -105,8 +111,131 @@ addressed. end ``` -1. Track necessary events. See the [product analytics guide](../product_analytics/index.md) for details. -1. After the merge request is merged, use [`chatops`](../../ci/chatops/README.md) in the +### Implement the tracking events + +To determine whether the experiment is a success or not, we must implement tracking events +to acquire data for analyzing. We can send events to Snowplow via either the backend or frontend. +Read the [product analytics guide](https://about.gitlab.com/handbook/product/product-analytics-guide/) for more details. + +#### Track backend events + +The framework provides the following helper method that is available in controllers: + +```ruby +before_action do + track_experiment_event(:signup_flow, 'action', 'value') +end +``` + +Which can be tested as follows: + +```ruby +context 'when the experiment is active and the user is in the experimental group' do + before do + stub_experiment(signup_flow: true) + stub_experiment_for_user(signup_flow: true) + end + + it 'tracks an event', :snowplow do + subject + + expect_snowplow_event( + category: 'Growth::Activation::Experiment::SignUpFlow', + action: 'action', + value: 'value', + label: 'experimentation_subject_id', + property: 'experimental_group' + ) + end +end +``` + +#### Track frontend events + +The framework provides the following helper method that is available in controllers: + +```ruby +before_action do + push_frontend_experiment(:signup_flow) + frontend_experimentation_tracking_data(:signup_flow, 'action', 'value') +end +``` + +This pushes tracking data to `gon.experiments` and `gon.tracking_data`. + +```ruby +expect(Gon.experiments['signupFlow']).to eq(true) + +expect(Gon.tracking_data).to eq( + { + category: 'Growth::Activation::Experiment::SignUpFlow', + action: 'action', + value: 'value', + label: 'experimentation_subject_id', + property: 'experimental_group' + } +) +``` + +Which can then be used for tracking as follows: + +```javascript +import { isExperimentEnabled } from '~/lib/utils/experimentation'; +import Tracking from '~/tracking'; + +document.addEventListener('DOMContentLoaded', () => { + const signupFlowExperimentEnabled = isExperimentEnabled('signupFlow'); + + if (signupFlowExperimentEnabled && gon.tracking_data) { + const { category, action, ...data } = gon.tracking_data; + + Tracking.event(category, action, data); + } +} +``` + +Which can be tested in Jest as follows: + +```javascript +import { withGonExperiment } from 'helpers/experimentation_helper'; +import Tracking from '~/tracking'; + +describe('event tracking', () => { + describe('with tracking data', () => { + withGonExperiment('signupFlow'); + + beforeEach(() => { + jest.spyOn(Tracking, 'event').mockImplementation(() => {}); + + gon.tracking_data = { + category: 'Growth::Activation::Experiment::SignUpFlow', + action: 'action', + value: 'value', + label: 'experimentation_subject_id', + property: 'experimental_group' + }; + }); + + it('should track data', () => { + performAction() + + expect(Tracking.event).toHaveBeenCalledWith( + 'Growth::Activation::Experiment::SignUpFlow', + 'action', + { + value: 'value', + label: 'experimentation_subject_id', + property: 'experimental_group' + }, + ); + }); + }); +}); +``` + +### Enable the experiment + +After all merge requests have been merged, use [`chatops`](../../ci/chatops/README.md) in the [appropriate channel](../feature_flags/controls.md#communicate-the-change) to start the experiment for 10% of the users. The feature flag should have the name of the experiment with the `_experiment_percentage` suffix appended. For visibility, please also share any commands run against production in the `#s_growth` channel: @@ -121,9 +250,39 @@ For visibility, please also share any commands run against production in the `#s /chatops run feature delete signup_flow_experiment_percentage ``` -### Tests and test helpers +### Testing and test helpers + +#### RSpec + +Use the following in RSpec to mock the experiment: + +```ruby +context 'when the experiment is active' do + before do + stub_experiment(signup_flow: true) + end + + context 'when the user is in the experimental group' do + before do + stub_experiment_for_user(signup_flow: true) + end + + it { is_expected.to do_experimental_thing } + end + + context 'when the user is in the control group' do + before do + stub_experiment_for_user(signup_flow: false) + end + + it { is_expected.to do_control_thing } + end +end +``` + +#### Jest -Use the following in Jest to test the experiment is enabled. +Use the following in Jest to mock the experiment: ```javascript import { withGonExperiment } from 'helpers/experimentation_helper'; diff --git a/doc/development/fe_guide/accessibility.md b/doc/development/fe_guide/accessibility.md index 669c93eb251..8cb61019556 100644 --- a/doc/development/fe_guide/accessibility.md +++ b/doc/development/fe_guide/accessibility.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Accessibility & Readability ## Resources diff --git a/doc/development/fe_guide/architecture.md b/doc/development/fe_guide/architecture.md index c7e1ba59f23..4b45fb97c76 100644 --- a/doc/development/fe_guide/architecture.md +++ b/doc/development/fe_guide/architecture.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Architecture When you are developing a new feature that requires architectural design, or if diff --git a/doc/development/fe_guide/axios.md b/doc/development/fe_guide/axios.md index 38a8c8f1086..856b03e4b47 100644 --- a/doc/development/fe_guide/axios.md +++ b/doc/development/fe_guide/axios.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Axios We use [Axios](https://github.com/axios/axios) to communicate with the server in Vue applications and most new code. diff --git a/doc/development/fe_guide/dependencies.md b/doc/development/fe_guide/dependencies.md index ba97e7e39be..bbe4cdc285d 100644 --- a/doc/development/fe_guide/dependencies.md +++ b/doc/development/fe_guide/dependencies.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Frontend dependencies ## Package manager diff --git a/doc/development/fe_guide/design_patterns.md b/doc/development/fe_guide/design_patterns.md index 5b7b16a9a8a..ed4d91f34bd 100644 --- a/doc/development/fe_guide/design_patterns.md +++ b/doc/development/fe_guide/design_patterns.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Design Patterns ## Singletons diff --git a/doc/development/fe_guide/development_process.md b/doc/development/fe_guide/development_process.md index 2b64534e7c9..915dab06ac2 100644 --- a/doc/development/fe_guide/development_process.md +++ b/doc/development/fe_guide/development_process.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Frontend Development Process You can find more about the organization of the frontend team in the [handbook](https://about.gitlab.com/handbook/engineering/frontend/). diff --git a/doc/development/fe_guide/dropdowns.md b/doc/development/fe_guide/dropdowns.md index 019bd36573d..bd2dae13c5b 100644 --- a/doc/development/fe_guide/dropdowns.md +++ b/doc/development/fe_guide/dropdowns.md @@ -1,3 +1,3 @@ --- -redirect_to: 'https://design.gitlab.com/components/dropdowns/' +redirect_to: 'https://design.gitlab.com/components/dropdown/' --- diff --git a/doc/development/fe_guide/droplab/droplab.md b/doc/development/fe_guide/droplab/droplab.md index 83bc4216403..fe0f07b3019 100644 --- a/doc/development/fe_guide/droplab/droplab.md +++ b/doc/development/fe_guide/droplab/droplab.md @@ -1,22 +1,29 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # DropLab A generic dropdown for all of your custom dropdown needs. ## Usage -DropLab can be used by simply adding a `data-dropdown-trigger` HTML attribute. -This attribute allows us to find the "trigger" _(toggle)_ for the dropdown, -whether that is a button, link or input. +DropLab can be used by adding a `data-dropdown-trigger` HTML attribute. This +attribute allows us to find the "trigger" _(toggle)_ for the dropdown, whether +it's a button, link or input. -The value of the `data-dropdown-trigger` should be a CSS selector that -DropLab can use to find the trigger's dropdown list. +The value of the `data-dropdown-trigger` should be a CSS selector that DropLab +can use to find the trigger's dropdown list. You should also add the `data-dropdown` attribute to declare the dropdown list. The value is irrelevant. -The DropLab class has no side effects, so you must always call `.init` when -the DOM is ready. `DropLab.prototype.init` takes the same arguments as `DropLab.prototype.addHook`. -If you do not provide any arguments, it will globally query and instantiate all droplab compatible dropdowns. +The DropLab class has no side effects, so you must always call `.init` when the +DOM is ready. `DropLab.prototype.init` takes the same arguments as `DropLab.prototype.addHook`. +If you don't provide any arguments, it will globally query and instantiate all +DropLab-compatible dropdowns. ```html <a href="#" data-dropdown-trigger="#list">Toggle</a> @@ -31,8 +38,8 @@ const droplab = new DropLab(); droplab.init(); ``` -As you can see, we have a "Toggle" link, that is declared as a trigger. -It provides a selector to find the dropdown list it should control. +As noted, we have a "Toggle" link that's declared as a trigger. It provides a +selector to find the dropdown list it should control. ### Static data @@ -54,8 +61,8 @@ droplab.init(); ### Explicit instantiation -You can pass the trigger and list elements as constructor arguments to return -a non-global instance of DropLab using the `DropLab.prototype.init` method. +You can pass the trigger and list elements as constructor arguments to return a +non-global instance of DropLab using the `DropLab.prototype.init` method. ```html <a href="#" id="trigger" data-dropdown-trigger="#list">Toggle</a> @@ -96,12 +103,13 @@ droplab.addHook(trigger, list); ### Dynamic data -Adding `data-dynamic` to your dropdown element will enable dynamic list rendering. +Adding `data-dynamic` to your dropdown element will enable dynamic list +rendering. -You can template a list item using the keys of the data object provided. -Use the handlebars syntax `{{ value }}` to HTML escape the value. -Use the `<%= value %>` syntax to simply interpolate the value. -Use the `<%= value %>` syntax to evaluate the value. +You can template a list item using the keys of the data object provided. Use the +handlebars syntax `{{ value }}` to HTML escape the value. Use the `<%= value %>` +syntax to interpolate the value. Use the `<%= value %>` syntax to evaluate the +value. Passing an array of objects to `DropLab.prototype.addData` will render that data for all `data-dynamic` dropdown lists tracked by that DropLab instance. @@ -126,8 +134,9 @@ droplab.init().addData([{ }]); ``` -Alternatively, you can specify a specific dropdown to add this data to but passing -the data as the second argument and the `id` of the trigger element as the first argument. +Alternatively, you can specify a specific dropdown to add this data to by +passing the data as the second argument and the `id` of the trigger element as +the first argument. ```html <a href="#" data-dropdown-trigger="#list" id="trigger">Toggle</a> @@ -149,7 +158,7 @@ droplab.init().addData('trigger', [{ }]); ``` -This allows you to mix static and dynamic content with ease, even with one trigger. +This allows you to mix static and dynamic content, even with one trigger. Note the use of scoping regarding the `data-dropdown` attribute to capture both dropdown lists, one of which is dynamic. @@ -185,12 +194,13 @@ DropLab adds some CSS classes to help lower the barrier to integration. For example: -- The `droplab-item-selected` CSS class is added to items that have been selected - either by a mouse click or by enter key selection. +- The `droplab-item-selected` CSS class is added to items that have been + selected either by a mouse click or by enter key selection. - The `droplab-item-active` CSS class is added to items that have been selected using arrow key navigation. -- You can add the `droplab-item-ignore` CSS class to any item that you do not want to be selectable. For example, - an `<li class="divider"></li>` list divider element that should not be interactive. +- You can add the `droplab-item-ignore` CSS class to any item that you don't + want to be selectable. For example, an `<li class="divider"></li>` list + divider element that shouldn't be interactive. ## Internal events @@ -198,26 +208,33 @@ DropLab uses some custom events to help lower the barrier to integration. For example: -- The `click.dl` event is fired when an `li` list item has been clicked. It is also - fired when a list item has been selected with the keyboard. It is also fired when a - `HookButton` button is clicked (a registered `button` tag or `a` tag trigger). -- The `input.dl` event is fired when a `HookInput` (a registered `input` tag trigger) triggers an `input` event. -- The `mousedown.dl` event is fired when a `HookInput` triggers a `mousedown` event. +- The `click.dl` event is fired when an `li` list item has been clicked. It's + also fired when a list item has been selected with the keyboard. It's also + fired when a `HookButton` button is clicked (a registered `button` tag or `a` + tag trigger). +- The `input.dl` event is fired when a `HookInput` (a registered `input` tag + trigger) triggers an `input` event. +- The `mousedown.dl` event is fired when a `HookInput` triggers a `mousedown` + event. - The `keyup.dl` event is fired when a `HookInput` triggers a `keyup` event. - The `keydown.dl` event is fired when a `HookInput` triggers a `keydown` event. -These custom events add a `detail` object to the vanilla `Event` object that provides some potentially useful data. +These custom events add a `detail` object to the vanilla `Event` object that +provides some potentially useful data. ## Plugins -Plugins are objects that are registered to be executed when a hook is added (when a droplab trigger and dropdown are instantiated). +Plugins are objects that are registered to be executed when a hook is added (when +a DropLab trigger and dropdown are instantiated). -If no modules API is detected, the library will fall back as it does with `window.DropLab` and will add `window.DropLab.plugins.PluginName`. +If no modules API is detected, the library will fall back as it does with +`window.DropLab` and will add `window.DropLab.plugins.PluginName`. ### Usage -To use plugins, you can pass them in an array as the third argument of `DropLab.prototype.init` or `DropLab.prototype.addHook`. -Some plugins require configuration values, the config object can be passed as the fourth argument. +To use plugins, you can pass them in an array as the third argument of +`DropLab.prototype.init` or `DropLab.prototype.addHook`. Some plugins require +configuration values; the config object can be passed as the fourth argument. ```html <a href="#" id="trigger" data-dropdown-trigger="#list">Toggle</a> @@ -240,14 +257,13 @@ droplab.init(trigger, list, [droplabAjax], { ### Documentation -- [Ajax plugin](plugins/ajax.md) -- [Filter plugin](plugins/filter.md) -- [InputSetter plugin](plugins/input_setter.md) +Refer to the list of available [DropLab plugins](plugins/index.md) for +information about their use. ### Development -When plugins are initialised for a droplab trigger+dropdown, DropLab will -call the plugins `init` function, so this must be implemented in the plugin. +When plugins are initialised for a DropLab trigger+dropdown, DropLab calls the +plugins' `init` function, so this must be implemented in the plugin. ```javascript class MyPlugin { diff --git a/doc/development/fe_guide/droplab/plugins/ajax.md b/doc/development/fe_guide/droplab/plugins/ajax.md index f22d95064dd..3ac876ad062 100644 --- a/doc/development/fe_guide/droplab/plugins/ajax.md +++ b/doc/development/fe_guide/droplab/plugins/ajax.md @@ -1,17 +1,25 @@ -# Ajax +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- -`Ajax` is a droplab plugin that allows for retrieving and rendering list data from a server. +# Ajax plugin + +`Ajax` is a DropLab plugin that allows for retrieving and rendering list data +from a server. ## Usage -Add the `Ajax` object to the plugins array of a `DropLab.prototype.init` or `DropLab.prototype.addHook` call. +Add the `Ajax` object to the plugins array of a `DropLab.prototype.init` or +`DropLab.prototype.addHook` call. -`Ajax` requires 2 configuration values, the `endpoint` and `method`. +`Ajax` requires 2 configuration values: the `endpoint` and `method`. -- `endpoint` should be a URL to the request endpoint. -- `method` should be `setData` or `addData`. -- `setData` completely replaces the dropdown with the response data. -- `addData` appends the response data to the current dropdown list. +- `endpoint`: Should be a URL to the request endpoint. +- `method`: Should be `setData` or `addData`. +- `setData`: Completely replaces the dropdown with the response data. +- `addData`: Appends the response data to the current dropdown list. ```html <a href="#" id="trigger" data-dropdown-trigger="#list">Toggle</a> @@ -32,7 +40,7 @@ droplab.addHook(trigger, list, [Ajax], { }); ``` -Optionally you can set `loadingTemplate` to a HTML string. This HTML string will -replace the dropdown list while the request is pending. +Optionally, you can set `loadingTemplate` to a HTML string. This HTML string +replaces the dropdown list while the request is pending. Additionally, you can set `onError` to a function to catch any XHR errors. diff --git a/doc/development/fe_guide/droplab/plugins/filter.md b/doc/development/fe_guide/droplab/plugins/filter.md index e8194e45a41..9ab4946d21d 100644 --- a/doc/development/fe_guide/droplab/plugins/filter.md +++ b/doc/development/fe_guide/droplab/plugins/filter.md @@ -1,15 +1,22 @@ -# Filter +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- -`Filter` is a plugin that allows for filtering data that has been added +# Filter plugin + +`Filter` is a DropLab plugin that allows for filtering data that has been added to the dropdown using a simple fuzzy string search of an input value. ## Usage -Add the `Filter` object to the plugins array of a `DropLab.prototype.init` or `DropLab.prototype.addHook` call. +Add the `Filter` object to the plugins array of a `DropLab.prototype.init` or +`DropLab.prototype.addHook` call. -- `Filter` requires a configuration value for `template`. -- `template` should be the key of the objects within your data array that you want to compare - to the user input string, for filtering. +- `Filter`: Requires a configuration value for `template`. +- `template`: Should be the key of the objects within your data array that you + want to compare to the user input string, for filtering. ```html <input href="#" id="trigger" data-dropdown-trigger="#list"> @@ -39,8 +46,10 @@ droplab.addData('trigger', [{ }]); ``` -Above, the input string will be compared against the `test` key of the passed data objects. +In the previous code, the input string is compared against the `test` key of the +passed data objects. -Optionally you can set `filterFunction` to a function. This function will be used instead -of `Filter`'s built in string search. `filterFunction` is passed 2 arguments, the first -is one of the data objects, the second is the current input value. +Optionally you can set `filterFunction` to a function. This function will be +used instead of `Filter`'s built-in string search. `filterFunction` is passed +two arguments: the first is one of the data objects, and the second is the +current input value. diff --git a/doc/development/fe_guide/droplab/plugins/index.md b/doc/development/fe_guide/droplab/plugins/index.md new file mode 100644 index 00000000000..d44670bfb3c --- /dev/null +++ b/doc/development/fe_guide/droplab/plugins/index.md @@ -0,0 +1,14 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +description: A list of DropLab plugins. +--- + +# DropLab plugins + +The following plugins are available for use with [DropLab](../droplab.md): + +- [Ajax plugin](ajax.md) +- [Filter plugin](filter.md) +- [InputSetter plugin](input_setter.md) diff --git a/doc/development/fe_guide/droplab/plugins/input_setter.md b/doc/development/fe_guide/droplab/plugins/input_setter.md index b873b7a14ee..ab8a5cebd08 100644 --- a/doc/development/fe_guide/droplab/plugins/input_setter.md +++ b/doc/development/fe_guide/droplab/plugins/input_setter.md @@ -1,17 +1,26 @@ -# InputSetter +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- -`InputSetter` is a plugin that allows for updating DOM out of the scope of droplab when a list item is clicked. +# InputSetter plugin + +`InputSetter` is a DropLab plugin that allows for updating DOM out of the scope +of DropLab when a list item is clicked. ## Usage -Add the `InputSetter` object to the plugins array of a `DropLab.prototype.init` or `DropLab.prototype.addHook` call. +Add the `InputSetter` object to the plugins array of a `DropLab.prototype.init` +or `DropLab.prototype.addHook` call. -- `InputSetter` requires a configuration value for `input` and `valueAttribute`. -- `input` should be the DOM element that you want to manipulate. -- `valueAttribute` should be a string that is the name of an attribute on your list items that is used to get the value - to update the `input` element with. +- `InputSetter`: Requires a configuration value for `input` and `valueAttribute`. +- `input`: The DOM element that you want to manipulate. +- `valueAttribute`: A string that's the name of an attribute on your list items + that's used to get the value to update the `input` element with. -You can also set the `InputSetter` configuration to an array of objects, which will allow you to update multiple elements. +You can also set the `InputSetter` configuration to an array of objects, which +allows you to update multiple elements. ```html <input id="input" value=""> @@ -52,9 +61,12 @@ droplab.addData('trigger', [{ }]); ``` -Above, if the second list item was clicked, it would update the `#input` element -to have a `value` of `1`, it would also update the `#div` element's `data-selected-id` to `1`. +In the previous code, if the second list item was clicked, it would update the +`#input` element to have a `value` of `1`, it would also update the `#div` +element's `data-selected-id` to `1`. -Optionally you can set `inputAttribute` to a string that is the name of an attribute on your `input` element that you want to update. -If you do not provide an `inputAttribute`, `InputSetter` will update the `value` of the `input` element if it is an `INPUT` element, -or the `textContent` of the `input` element if it is not an `INPUT` element. +Optionally, you can set `inputAttribute` to a string that's the name of an +attribute on your `input` element that you want to update. If you don't provide +an `inputAttribute`, `InputSetter` will update the `value` of the `input` +element if it's an `INPUT` element, or the `textContent` of the `input` element +if it isn't an `INPUT` element. diff --git a/doc/development/fe_guide/editor_lite.md b/doc/development/fe_guide/editor_lite.md new file mode 100644 index 00000000000..eb5852d258d --- /dev/null +++ b/doc/development/fe_guide/editor_lite.md @@ -0,0 +1,224 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + +# Editor Lite + +## Background + +**Editor Lite** is a technological product driving [Web Editor](../../user/project/repository/web_editor.md), [Snippets](../../user/snippets.md), [CI Linter](../../ci/lint.md), etc. Editor Lite is the driving technology for any single-file editing experience across the product. + +Editor Lite is a thin wrapper around [the Monaco editor](https://microsoft.github.io/monaco-editor/index.html) that provides the necessary helpers and abstractions and extends Monaco using extensions. + +## How to use Editor Lite + +Editor Lite is framework-agnostic and can be used in any application, whether it's Rails or Vue. For the convenience of integration, we have [the dedicated `<editor-lite>` Vue component](#vue-component), but in general, the integration of Editor Lite is pretty straightforward: + +1. Import Editor Lite: + +```javascript +import EditorLite from '~/editor/editor_lite'; +``` + +1. Initialize global editor for the view: + +```javascript +const editor = new EditorLite({ + // Editor Options. + // The list of all accepted options can be found at + // https://microsoft.github.io/monaco-editor/api/enums/monaco.editor.editoroption.html +}); +``` + +1. Create an editor's instance: + +```javascript +editor.createInstance({ + // Editor Lite configuration options. +}) +``` + +An instance of Editor Lite accepts the following configuration options: + +| Option | Required? | Description | +| ---- | ---- | ---- | +| `el` | `true` | `HTML Node`: element on which to render the editor | +| `blobPath` | `false` | `String`: the name of a file to render in the editor. It is used to identify the correct syntax highlighter to use with that or another file type. Can accept wildcard as in `*.js` when the actual filename isn't known or doesn't play any role | +| `blobContent` | `false` | `String`: the initial content to be rendered in the editor | +| `extensions` | `false` | `Array`: extensions to use in this instance | +| `blobGlobalId` | `false` | `String`: auto-generated property.<br>**Note:** this prop might go away in the future. Do not pass `blobGlobalId` unless you know what you're doing.| +| [Editor Options](https://microsoft.github.io/monaco-editor/api/enums/monaco.editor.editoroption.html) | `false` | `Object(s)`: any prop outside of the list above is treated as an Editor Option for this particular instance. This way, one can override global Editor Options on the instance level. | + +## API + +The editor follows the same public API as [provided by Monaco editor](https://microsoft.github.io/monaco-editor/api/interfaces/monaco.editor.istandalonecodeeditor.html) with just a few additional functions on the instance level: + +| Function | Arguments | Description +| ----- | ----- | ----- | +| `updateModelLanguage` | `path`: String | Updates the instance's syntax highlighting to follow the extension of the passed `path`. Available only on _instance_ level| +| `use` | Array of objects | Array of **extensions** to apply to the instance. Accepts only the array of _objects_, which means that the extensions' ES6 modules should be fetched and resolved in your views/components before being passed to `use`. This prop is available on _instance_ (applies extension to this particular instance) and _global edtor_ (applies the same extension to all instances) levels. | +| Monaco Editor options | See [documentation](https://microsoft.github.io/monaco-editor/api/interfaces/monaco.editor.istandalonecodeeditor.html) | Default Monaco editor options | + +## Tips + +1. Editor's loading state. + +Editor Lite comes with the loading state built-in, making spinners and loaders rarely needed in HTML. To benefit the built-in loading state, set the `data-editor-loading` property on the HTML element that is supposed to contain the editor. Editor Lite will show the loader automatically while it's bootstrapping. + + +1. Update syntax highlighting if the file name changes. + +```javascript +// fileNameEl here is the HTML input element that contains the file name +fileNameEl.addEventListener('change', () => { + this.editor.updateModelLanguage(fileNameEl.value); +}); +``` + +1. Get the editor's content. + +We might set up listeners on the editor for every change but it rapidly can become an expensive operation. Instead , we can get editor's content when it's needed. For example on a form's submission: + +```javascript +form.addEventListener('submit', () => { + my_content_variable = this.editor.getValue(); +}); +``` + +1. Performance + +Even though Editor Lite itself is extremely slim, it still depends on Monaco editor. Monaco is not an easily tree-shakeable module. Hence, every time you add Editor Lite to a view, the JavaScript bundle's size significantly increases, affecting your view's loading performance. To avoid that, it is recommended to import the editor on demand on those views where it is not 100% certain that the editor will be used. Or if the editor is a secondary element of the view. Loading Editor Lite on demand is no different from loading any other module: + +```javascript +someActionFunction() { + import(/* webpackChunkName: 'EditorLite' */ '~/editor/editor_lite'). + then(({ default: EditorLite }) => { + const editor = new EditorLite(); + ... + }); + ... +} +``` + +## Extensions + +Editor Lite has been built to provide a universal, extensible editing tool to the whole product, which would not depend on any particular group. Even though the Editor Lite's core is owned by [Create::Editor FE Team](https://about.gitlab.com/handbook/engineering/development/dev/create-editor-fe/), the main functional elements — extensions — can be owned by any group. Editor Lite extensions' main idea is that the core of the editor remains very slim and stable. At the same time, whatever new functionality is needed can be added as an extension to this core, without touching the core itself. It allows any group to build and own any new editing functionality without being afraid of it being broken or overridden with the Editor Lite changes. + +Structurally, the complete implementation of Editor Lite could be presented as the following diagram: + +```mermaid +graph TD; + B[Extension 1]---A[Editor Lite] + C[Extension 2]---A[Editor Lite] + D[Extension 3]---A[Editor Lite] + E[...]---A[Editor Lite] + F[Extension N]---A[Editor Lite] + A[Editor Lite]---Z[Monaco] +``` + +Technically, an extension is just an ES6 module that exports a JavaScript object: + +```javascript +import { Position } from 'monaco-editor'; + +export default { + navigateFileStart() { + this.setPosition(new Position(1, 1)); + }, +}; + +``` + +Important things to note here: + +- We can depend on other modules in our extensions. This organization helps keep the size of Editor Lite's core at bay by importing dependencies only when needed. +- `this` in extension's functions refers to the current Editor Lite instance. Using `this`, you get access to the complete instance's API, such as the `setPosition()` method in this particular case. + +### Using an existing extension + +Adding an extension to Editor Lite's instance is simple: + +```javascript +import EditorLite from '~/editor/editor_lite'; +import MyExtension from '~/my_extension'; + +const editor = new EditorLite().createInstance({ + ... +}); +editor.use(MyExtension); +``` + +### Creating an extension + +Let's create our first Editor Lite extension. As aforementioned, extensions are ES6 modules exporting the simple `Object` that is used to extend Editor Lite's functionality. As the most straightforward test, let's create an extension that extends Editor Lite with a new function that, when called, will output editor's content in `alert`. + +`~/my_folder/my_fancy_extension.js:` + +```javascript +export default { + throwContentAtMe() { + alert(this.getValue()); + }, +}; +``` + +And that's it with our extension! Note that we're using `this` as a reference to the instance. And through it, we get access to the complete underlying [Monaco editor API](https://microsoft.github.io/monaco-editor/api/interfaces/monaco.editor.istandalonecodeeditor.html) like `getValue()` in this case. + +Now let's use our extension: + +`~/my_folder/component_bundle.js`: + +```javascript +import EditorLite from '~/editor/editor_lite'; +import MyFancyExtension from './my_fancy_extension'; + +const editor = new EditorLite().createInstance({ + ... +}); +editor.use(MyFancyExtension); +... +someButton.addEventListener('click', () => { + editor.throwContentAtMe(); +}); +``` + +First of all, we import Editor Lite and our new extension. Then we create the editor and its instance. By default Editor Lite has no `throwContentAtMe` method. But the `editor.use(MyFancyExtension)` line brings that method to our instance. After that, we can use it any time we need it. In this case, we call it when some theoretical button has been clicked. + +This script would result in an alert containing the editor's content when `someButton` is clicked. + + + +### Tips + +1. Performance + +Just like Editor Lite itself, any extension can be loaded on demand to not harm loading performance of the views: + +```javascript +const EditorPromise = import( + /* webpackChunkName: 'EditorLite' */ '~/editor/editor_lite' +); +const MarkdownExtensionPromise = import('~/editor/editor_markdown_ext'); + +Promise.all([EditorPromise, MarkdownExtensionPromise]) + .then(([{ default: EditorLite }, { default: MarkdownExtension }]) => { + const editor = new EditorLite().createInstance({ + ... + }); + editor.use(MarkdownExtension); + }); +``` + +1. Using multiple extensions + +Just pass the array of extensions to your `use` method: + +```javascript +editor.use([FileTemplateExtension, MyFancyExtension]); +``` + +## <a id="vue-component"></a>`<editor-lite>` Vue component + +TBD diff --git a/doc/development/fe_guide/emojis.md b/doc/development/fe_guide/emojis.md index 3cd14c0dfd3..9b1abaa14ae 100644 --- a/doc/development/fe_guide/emojis.md +++ b/doc/development/fe_guide/emojis.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Emojis GitLab supports native Unicode emojis and falls back to image-based emojis selectively diff --git a/doc/development/fe_guide/frontend_faq.md b/doc/development/fe_guide/frontend_faq.md index 0a26fdba934..22a48c8f9f9 100644 --- a/doc/development/fe_guide/frontend_faq.md +++ b/doc/development/fe_guide/frontend_faq.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Frontend FAQ ## Rules of Frontend FAQ @@ -63,7 +69,7 @@ banner on top of the component examples indicates that: > component. For example, at the time of writing, this type of warning can be observed for -[all form components](https://design.gitlab.com/components/forms/). It, however, +[all form components](https://design.gitlab.com/components/form/). It, however, doesn't imply that the component should not be used. GitLab always asks to use `<gl-*>` components whenever a suitable component exists. @@ -174,10 +180,9 @@ which adds the appropriate `core-js` polyfills once for each JavaScript feature we're using that our target browsers don't support. You don't need to add `core-js` polyfills manually. -NOTE: **Note:** -GitLab still manually adds non-`core-js` polyfills for extending browser features -(such as GitLab's SVG polyfill) that allow us reference SVGs by using `<use xlink:href>`. -These polyfills should be added to `app/assets/javascripts/commons/polyfills.js`. +GitLab adds non-`core-js` polyfills for extending browser features (such as +GitLab's SVG polyfill), which allow us to reference SVGs by using `<use xlink:href>`. +Be sure to add these polyfills to `app/assets/javascripts/commons/polyfills.js`. To see what polyfills are being used: diff --git a/doc/development/fe_guide/graphql.md b/doc/development/fe_guide/graphql.md index ad3958d4496..cae2435e4ff 100644 --- a/doc/development/fe_guide/graphql.md +++ b/doc/development/fe_guide/graphql.md @@ -664,7 +664,9 @@ it('calls mutation on submitting form ', () => { ### Testing with mocked Apollo Client -To test the logic of Apollo cache updates, we might want to mock an Apollo Client in our unit tests. To separate tests with mocked client from 'usual' unit tests, it's recommended to create an additional component factory. This way we only create Apollo Client instance when it's necessary: +To test the logic of Apollo cache updates, we might want to mock an Apollo Client in our unit tests. We use [`mock-apollo-client`](https://www.npmjs.com/package/mock-apollo-client) library to mock Apollo client and [`createMockApollo` helper](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/frontend/helpers/mock_apollo_helper.js) we created on top of it. + +To separate tests with mocked client from 'usual' unit tests, it's recommended to create an additional component factory. This way we only create Apollo Client instance when it's necessary: ```javascript function createComponent() {...} @@ -672,12 +674,6 @@ function createComponent() {...} function createComponentWithApollo() {...} ``` -We use [`mock-apollo-client`](https://www.npmjs.com/package/mock-apollo-client) library to mock Apollo client in tests. - -```javascript -import { createMockClient } from 'mock-apollo-client'; -``` - Then we need to inject `VueApollo` to Vue local instance (`localVue.use()` can also be called within `createComponentWithApollo()`) ```javascript @@ -741,8 +737,8 @@ describe('Some component with Apollo mock', () => { }) ``` -NOTE: **Note:** -When mocking resolved values, make sure the structure of the response is the same as actual API response: i.e. root property should be `data` for example +When mocking resolved values, ensure the structure of the response is the same +as the actual API response. For example, root property should be `data`. When testing queries, please keep in mind they are promises, so they need to be _resolved_ to render a result. Without resolving, we can check the `loading` state of the query: @@ -830,6 +826,53 @@ it('calls a mutation with correct parameters and reorders designs', async () => }); ``` +#### Testing `@client` queries + +If your application contains `@client` queries, most probably you will have an Apollo Client warning saying that you have a local query but no resolvers are defined. In order to fix it, you need to pass resolvers to the mocked client with a second parameter (bare minimum is an empty object): + +```javascript +import createMockApollo from 'jest/helpers/mock_apollo_helper'; +... +fakeApollo = createMockApollo(requestHandlers, {}); +``` + +Sometimes we want to test a `result` hook of the local query. In order to have it triggered, we need to populate a cache with correct data to be fetched with this query: + +```javascript +query fetchLocalUser { + fetchLocalUser @client { + name + } +} +``` + +```javascript +import fetchLocalUserQuery from '~/design_management/graphql/queries/fetch_local_user.query.graphql'; + +function createComponentWithApollo() { + const requestHandlers = [ + [getDesignListQuery, jest.fn().mockResolvedValue(designListQueryResponse)], + [permissionsQuery, jest.fn().mockResolvedValue(permissionsQueryResponse)], + ]; + + fakeApollo = createMockApollo(requestHandlers, {}); + fakeApollo.clients.defaultClient.cache.writeQuery({ + query: fetchLocalUserQuery, + data: { + fetchLocalUser: { + __typename: 'User', + name: 'Test', + }, + }, + }) + + wrapper = shallowMount(Index, { + localVue, + apolloProvider: fakeApollo, + }); +} +``` + ## Handling errors GitLab's GraphQL mutations currently have two distinct error modes: [Top-level](#top-level-errors) and [errors-as-data](#errors-as-data). @@ -910,3 +953,99 @@ const defaultClient = createDefaultClient( }, ); ``` + +## Making initial queries early with GraphQL startup calls + +To improve performance, sometimes we want to make initial GraphQL queries early. In order to do this, we can add them to **startup calls** with the following steps: + +- Move all the queries you need initially in your application to `app/graphql/queries`; +- Add `__typename` property to every nested query level: + + ```javascript + query getPermissions($projectPath: ID!) { + project(fullPath: $projectPath) { + __typename + userPermissions { + __typename + pushCode + forkProject + createMergeRequestIn + } + } + } + ``` + +- If queries contain fragments, you need to move fragments to the query file directly instead of importing them: + + ```javascript + fragment PageInfo on PageInfo { + __typename + hasNextPage + hasPreviousPage + startCursor + endCursor + } + + query getFiles( + $projectPath: ID! + $path: String + $ref: String! + ) { + project(fullPath: $projectPath) { + __typename + repository { + __typename + tree(path: $path, ref: $ref) { + __typename + pageInfo { + ...PageInfo + } + } + } + } + } + } + ``` + +- If the fragment is used only once, we can also remove the fragment altogether: + + ```javascript + query getFiles( + $projectPath: ID! + $path: String + $ref: String! + ) { + project(fullPath: $projectPath) { + __typename + repository { + __typename + tree(path: $path, ref: $ref) { + __typename + pageInfo { + __typename + hasNextPage + hasPreviousPage + startCursor + endCursor + } + } + } + } + } + } + ``` + +- Add startup call(s) with correct variables to the HAML file that serves as a view +for your application. To add GraphQL startup calls, we use +`add_page_startup_graphql_call` helper where the first parameter is a path to the +query, the second one is an object containing query variables. Path to the query is +relative to `app/graphql/queries` folder: for example, if we need a +`app/graphql/queries/repository/files.query.graphql` query, the path will be +`repository/files`. + + ```yaml + - current_route_path = request.fullpath.match(/-\/tree\/[^\/]+\/(.+$)/).to_a[1] + - add_page_startup_graphql_call('repository/path_last_commit', { projectPath: @project.full_path, ref: current_ref, path: current_route_path || "" }) + - add_page_startup_graphql_call('repository/permissions', { projectPath: @project.full_path }) + - add_page_startup_graphql_call('repository/files', { nextPageCursor: "", pageSize: 100, projectPath: @project.full_path, ref: current_ref, path: current_route_path || "/"}) + ``` diff --git a/doc/development/fe_guide/icons.md b/doc/development/fe_guide/icons.md index 67add5709d9..994a3aa36fc 100644 --- a/doc/development/fe_guide/icons.md +++ b/doc/development/fe_guide/icons.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Icons and SVG Illustrations We manage our own icon and illustration library in the [`gitlab-svgs`](https://gitlab.com/gitlab-org/gitlab-svgs) @@ -84,7 +90,7 @@ Please use the following function inside JS to render an icon: ### Usage in HAML/Rails -DANGER: **Danger:** +DANGER: **Warning:** Do not use the `spinner` or `icon('spinner spin')` rails helpers to insert loading icons. These helpers rely on the Font Awesome icon library which is deprecated. diff --git a/doc/development/fe_guide/img/editor_lite_create_ext.png b/doc/development/fe_guide/img/editor_lite_create_ext.png Binary files differnew file mode 100644 index 00000000000..73941cf5d62 --- /dev/null +++ b/doc/development/fe_guide/img/editor_lite_create_ext.png diff --git a/doc/development/fe_guide/img/editor_lite_loading.png b/doc/development/fe_guide/img/editor_lite_loading.png Binary files differnew file mode 100644 index 00000000000..f2c242da155 --- /dev/null +++ b/doc/development/fe_guide/img/editor_lite_loading.png diff --git a/doc/development/fe_guide/img/gl-modal.png b/doc/development/fe_guide/img/gl-modal.png Binary files differdeleted file mode 100644 index b2d2d637e57..00000000000 --- a/doc/development/fe_guide/img/gl-modal.png +++ /dev/null diff --git a/doc/development/fe_guide/index.md b/doc/development/fe_guide/index.md index f909866d44e..9f98876bc8e 100644 --- a/doc/development/fe_guide/index.md +++ b/doc/development/fe_guide/index.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Frontend Development Guidelines This document describes various guidelines to ensure consistency and quality diff --git a/doc/development/fe_guide/keyboard_shortcuts.md b/doc/development/fe_guide/keyboard_shortcuts.md index da9b3702a8d..227b5cfd22c 100644 --- a/doc/development/fe_guide/keyboard_shortcuts.md +++ b/doc/development/fe_guide/keyboard_shortcuts.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Implementing keyboard shortcuts We use [Mousetrap](https://craig.is/killing/mice) to implement keyboard diff --git a/doc/development/fe_guide/performance.md b/doc/development/fe_guide/performance.md index 5d2b699c40d..de9a9f5cb14 100644 --- a/doc/development/fe_guide/performance.md +++ b/doc/development/fe_guide/performance.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Performance ## Best Practices @@ -71,9 +77,8 @@ controller with the `index` action. If a corresponding file exists at `pages/projects/issues/index/index.js`, it will be compiled into a webpack bundle and included on the page. -NOTE: **Note:** -Previously we had encouraged the use of -`content_for :page_specific_javascripts` within haml files, along with +Previously, GitLab encouraged the use of +`content_for :page_specific_javascripts` within HAML files, along with manually generated webpack bundles. However under this new system you should not ever need to manually add an entry point to the `webpack.config.js` file. @@ -91,17 +96,55 @@ browser's developer console while on any page within GitLab. modules outside of the entry point script. Just import, read the DOM, instantiate, and nothing else. -- **Entry Points May Be Asynchronous:** - _DO NOT ASSUME_ that the DOM has been fully loaded and available when an - entry point script is run. If you require that some code be run after the - DOM has loaded, you should attach an event handler to the `DOMContentLoaded` - event with: +- **`DOMContentLoaded` should not be used:** + All of GitLab's JavaScript files are added with the `defer` attribute. + According to the [Mozilla documentation](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/script#attr-defer), + this implies that "the script is meant to be executed after the document has + been parsed, but before firing `DOMContentLoaded`". Since the document is already + parsed, `DOMContentLoaded` is not needed to bootstrap applications because all + the DOM nodes are already at our disposal. + +- **JavaScript that relies on CSS for calculations should use [`waitForCSSLoaded()`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/javascripts/helpers/startup_css_helper.js#L34):** + GitLab uses [Startup.css](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38052) + to improve page performance. This can cause issues if JavaScript relies on CSS + for calculations. To fix this the JavaScript can be wrapped in the + [`waitForCSSLoaded()`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/javascripts/helpers/startup_css_helper.js#L34) + helper function. ```javascript import initMyWidget from './my_widget'; + import { waitForCSSLoaded } from '~/helpers/startup_css_helper'; - document.addEventListener('DOMContentLoaded', () => { - initMyWidget(); + waitForCSSLoaded(initMyWidget); + ``` + + Note that `waitForCSSLoaded()` methods supports receiving the action in different ways: + + - With a callback: + + ```javascript + waitForCSSLoaded(action) + ``` + + - With `then()`: + + ```javascript + waitForCSSLoaded().then(action); + ``` + + - With `await` followed by `action`: + + ```javascript + await waitForCSSLoaded; + action(); + ``` + + For example, see how we use this in [app/assets/javascripts/pages/projects/graphs/charts/index.js](https://gitlab.com/gitlab-org/gitlab/-/commit/5e90885d6afd4497002df55bf015b338efcfc3c5#02e81de37f5b1716a3ef3222fa7f7edf22c40969_9_8): + + ```javascript + waitForCSSLoaded(() => { + const languagesContainer = document.getElementById('js-languages-chart'); + //... }); ``` diff --git a/doc/development/fe_guide/principles.md b/doc/development/fe_guide/principles.md index 2bef48fddcf..75954a5f988 100644 --- a/doc/development/fe_guide/principles.md +++ b/doc/development/fe_guide/principles.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Principles These principles will ensure that your frontend contribution starts off in the right direction. diff --git a/doc/development/fe_guide/security.md b/doc/development/fe_guide/security.md index a001dd83ab7..a82c315032f 100644 --- a/doc/development/fe_guide/security.md +++ b/doc/development/fe_guide/security.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Security ## Resources @@ -77,3 +83,25 @@ inject scripts into the web app. Inline styles should be avoided in almost all cases, they should only be used when no alternatives can be found. This allows reusability of styles as well as readability. + +### Sanitize HTML output + +If you need to output raw HTML, you should sanitize it. + +If you are using Vue, you can use the[`v-safe-html` directive](https://gitlab-org.gitlab.io/gitlab-ui/?path=/story/directives-safe-html-directive--default) from GitLab UI. + +For other use cases, wrap a preconfigured version of [`dompurify`](https://www.npmjs.com/package/dompurify) +that also allows the icons to be rendered: + +```javascript +import { sanitize } from '~/lib/dompurify'; + +const unsafeHtml = '<some unsafe content ... >'; + +// ... + +element.appendChild(sanitize(unsafeHtml)); +``` + +This `sanitize` function takes the same configuration as the +original. diff --git a/doc/development/fe_guide/style/html.md b/doc/development/fe_guide/style/html.md index 1445da3f0e1..61a724cf3c7 100644 --- a/doc/development/fe_guide/style/html.md +++ b/doc/development/fe_guide/style/html.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # HTML style guide ## Buttons diff --git a/doc/development/fe_guide/style/index.md b/doc/development/fe_guide/style/index.md index 4ca409664de..0d73b16b5d3 100644 --- a/doc/development/fe_guide/style/index.md +++ b/doc/development/fe_guide/style/index.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # GitLab development style guides See below for the relevant style guides, guidelines, linting, and other information for developing GitLab. diff --git a/doc/development/fe_guide/style/javascript.md b/doc/development/fe_guide/style/javascript.md index 2a06a473878..b8e7429eb2c 100644 --- a/doc/development/fe_guide/style/javascript.md +++ b/doc/development/fe_guide/style/javascript.md @@ -1,4 +1,7 @@ --- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers disqus_identifier: 'https://docs.gitlab.com/ee/development/fe_guide/style_guide_js.html' --- @@ -138,7 +141,7 @@ module.exports = SomeClass; export default SomeClass; ``` -_Note:_ We still use `require` in `scripts/` and `config/` files. +We still use `require` in `scripts/` and `config/` files. ## Absolute vs relative paths for modules diff --git a/doc/development/fe_guide/style/scss.md b/doc/development/fe_guide/style/scss.md index dba39eeb98c..c6ee1e64a80 100644 --- a/doc/development/fe_guide/style/scss.md +++ b/doc/development/fe_guide/style/scss.md @@ -1,4 +1,7 @@ --- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers disqus_identifier: 'https://docs.gitlab.com/ee/development/fe_guide/style_guide_scss.html' --- @@ -46,11 +49,6 @@ We recommend a "utility-first" approach. This encourages an organic growth of component classes and prevents the creation of one-off unreusable classes. Also, the kind of classes that emerge from "utility-first" tend to be design-centered (e.g. `.button`, `.alert`, `.card`) rather than domain-centered (e.g. `.security-report-widget`, `.commit-header-icon`). -Examples of component classes that were created using "utility-first" include: - -- [`.circle-icon-container`](https://gitlab.com/gitlab-org/gitlab/blob/579fa8b8ec7eb38d40c96521f517c9dab8c3b97a/app/assets/stylesheets/framework/icons.scss#L85) -- [`.d-flex-center`](https://gitlab.com/gitlab-org/gitlab/blob/900083d89cd6af391d26ab7922b3f64fa2839bef/app/assets/stylesheets/framework/common.scss#L425) - Inspiration: - <https://tailwindcss.com/docs/utility-first> diff --git a/doc/development/fe_guide/style/vue.md b/doc/development/fe_guide/style/vue.md index 1a64db443bc..6b6ca9a6c71 100644 --- a/doc/development/fe_guide/style/vue.md +++ b/doc/development/fe_guide/style/vue.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Vue.js style guide ## Linting @@ -51,6 +57,66 @@ Please check this [rules](https://github.com/vuejs/eslint-plugin-vue#bulb-rules) 1. Use `.vue` for Vue templates. Do not use `%template` in HAML. +1. Explicitly define data being passed into the Vue app + + ```javascript + // bad + return new Vue({ + el: '#element', + components: { + componentName + }, + provide: { + ...someDataset + }, + props: { + ...anotherDataset + }, + render: createElement => createElement('component-name'), + })); + + // good + const { foobar, barfoo } = someDataset; + const { foo, bar } = anotherDataset; + + return new Vue({ + el: '#element', + components: { + componentName + }, + provide: { + foobar, + barfoo + }, + props: { + foo, + bar + }, + render: createElement => createElement('component-name'), + })); + ``` + + We discourage the use of the spread operator in this specific case in + order to keep our codebase explicit, discoverable, and searchable. + This applies in any place where we'll benefit from the above, such as + when [initializing Vuex state](../vuex.md#why-not-just-spread-the-initial-state). + The pattern above also enables us to easily parse non scalar values during + instantiation. + + ```javascript + return new Vue({ + el: '#element', + components: { + componentName + }, + props: { + foo, + bar: parseBoolean(bar) + }, + render: createElement => createElement('component-name'), + })); + ``` + ## Naming 1. **Extensions**: Use `.vue` extension for Vue components. Do not use `.js` as file extension ([#34371](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/34371)). @@ -184,7 +250,7 @@ Please check this [rules](https://github.com/vuejs/eslint-plugin-vue#bulb-rules) ``` 1. Default key should be provided if the prop is not required. - _Note:_ There are some scenarios where we need to check for the existence of the property. + There are some scenarios where we need to check for the existence of the property. On those a default key should not be provided. ```javascript @@ -400,6 +466,199 @@ Useful links: $('span').tooltip('_fixTitle'); ``` +## Vue testing + +Over time, a number of programming patterns and style preferences have emerged in our efforts to effectively test Vue components. +The following guide describes some of these. **These are not strict guidelines**, but rather a collection of suggestions and +good practices that aim to provide insight into how we write Vue tests at GitLab. + +### Mounting a component + +Typically, when testing a Vue component, the component should be "re-mounted" in every test block. + +To achieve this: + +1. Create a mutable `wrapper` variable inside the top-level `describe` block. +1. Mount the component using [`mount`](https://vue-test-utils.vuejs.org/api/#mount)/[`shallowMount`](https://vue-test-utils.vuejs.org/api/#shallowMount). +1. Reassign the resulting [`Wrapper`](https://vue-test-utils.vuejs.org/api/wrapper/#wrapper) instance to our `wrapper` variable. + +Creating a global, mutable wrapper provides a number of advantages, including the ability to: + +- Define common functions for finding components/DOM elements: + + ```javascript + import MyComponent from '~/path/to/my_component.vue'; + describe('MyComponent', () => { + let wrapper; + + // this can now be reused across tests + const findMyComponent = wrapper.find(MyComponent); + // ... + }) + ``` + +- Use a `beforeEach` block to mount the component (see [the `createComponent` factory](#the-createcomponent-factory) for more information). +- Use an `afterEach` block to destroy the component, for example, `wrapper.destroy()`. + +#### The `createComponent` factory + +To avoid duplicating our mounting logic, it's useful to define a `createComponent` factory function +that we can reuse in each test block. This is a closure which should reassign our `wrapper` variable +to the result of [`mount`](https://vue-test-utils.vuejs.org/api/#mount) and [`shallowMount`](https://vue-test-utils.vuejs.org/api/#shallowMount): + +```javascript +import MyComponent from '~/path/to/my_component.vue'; +import { shallowMount } from '@vue/test-utils'; + +describe('MyComponent', () => { + // Initiate the "global" wrapper variable. This will be used throughout our test: + let wrapper; + + // Define our `createComponent` factory: + function createComponent() { + // Mount component and reassign `wrapper`: + wrapper = shallowMount(MyComponent); + } + + it('mounts', () => { + createComponent(); + + expect(wrapper.exists()).toBe(true); + }); + + it('`isLoading` prop defaults to `false`', () => { + createComponent(); + + expect(wrapper.props('isLoading')).toBe(false); + }); +}) +``` + +Similarly, we could further de-duplicate our test by calling `createComponent` in a `beforeEach` block: + +```javascript +import MyComponent from '~/path/to/my_component.vue'; +import { shallowMount } from '@vue/test-utils'; + +describe('MyComponent', () => { + // Initiate the "global" wrapper variable. This will be used throughout our test + let wrapper; + + // define our `createComponent` factory + function createComponent() { + // mount component and reassign `wrapper` + wrapper = shallowMount(MyComponent); + } + + beforeEach(() => { + createComponent(); + }); + + it('mounts', () => { + expect(wrapper.exists()).toBe(true); + }); + + it('`isLoading` prop defaults to `false`', () => { + expect(wrapper.props('isLoading')).toBe(false); + }); +}) +``` + +#### `createComponent` best practices + +1. Consider using a single (or a limited number of) object arguments over many arguments. + Defining single parameters for common data like `props` is okay, + but keep in mind our [JavaScript style guide](javascript.md#limit-number-of-parameters) and stay within the parameter number limit: + + ```javascript + // bad + function createComponent(data, props, methods, isLoading, mountFn) { } + + // good + function createComponent({ data, props, methods, stubs, isLoading } = {}) { } + + // good + function createComponent(props = {}, { data, methods, stubs, isLoading } = {}) { } + ``` + +1. If you require both `mount` _and_ `shallowMount` within the same set of tests, it +can be useful define a `mountFn` parameter for the `createComponent` factory that accepts +the mounting function (`mount` or `shallowMount`) to be used to mount the component: + + ```javascript + import { shallowMount } from '@vue/test-utils'; + + function createComponent({ mountFn = shallowMount } = {}) { } + ``` + +### Setting component state + +1. Avoid using [`setProps`](https://vue-test-utils.vuejs.org/api/wrapper/#setprops) to set +component state wherever possible. Instead, set the component's +[`propsData`](https://vue-test-utils.vuejs.org/api/options.html#propsdata) when mounting the component: + + ```javascript + // bad + wrapper = shallowMount(MyComponent); + wrapper.setProps({ + myProp: 'my cool prop' + }); + + // good + wrapper = shallowMount({ propsData: { myProp: 'my cool prop' } }); + ``` + + The exception here is when you wish to test component reactivity in some way. + For example, you may want to test the output of a component when after a particular watcher has executed. + Using `setProps` to test such behavior is okay. + +### Accessing component state + +1. When accessing props or attributes, prefer the `wrapper.props('myProp')` syntax over `wrapper.props().myProp`: + + ```javascript + // good + expect(wrapper.props().myProp).toBe(true); + expect(wrapper.attributes().myAttr).toBe(true); + + // better + expect(wrapper.props('myProp').toBe(true); + expect(wrapper.attributes('myAttr')).toBe(true); + ``` + +1. When asserting multiple props, check the deep equality of the `props()` object with [`toEqual`](https://jestjs.io/docs/en/expect#toequalvalue): + + ```javascript + // good + expect(wrapper.props('propA')).toBe('valueA'); + expect(wrapper.props('propB')).toBe('valueB'); + expect(wrapper.props('propC')).toBe('valueC'); + + // better + expect(wrapper.props()).toEqual({ + propA: 'valueA', + propB: 'valueB', + propC: 'valueC', + }); + ``` + +1. If you are only interested in some of the props, you can use [`toMatchObject`](https://jestjs.io/docs/en/expect#tomatchobjectobject). +Prefer `toMatchObject` over [`expect.objectContaining`](https://jestjs.io/docs/en/expect#expectobjectcontainingobject): + + ```javascript + // good + expect(wrapper.props()).toEqual(expect.objectContaining({ + propA: 'valueA', + propB: 'valueB', + })); + + // better + expect(wrapper.props()).toMatchObject({ + propA: 'valueA', + propB: 'valueB', + }); + ``` + ## The JavaScript/Vue Accord The goal of this accord is to make sure we are all on the same page. diff --git a/doc/development/fe_guide/tooling.md b/doc/development/fe_guide/tooling.md index b93c0a9736b..809e05ea61f 100644 --- a/doc/development/fe_guide/tooling.md +++ b/doc/development/fe_guide/tooling.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Tooling ## ESLint @@ -14,7 +20,7 @@ To check all currently staged files (based on `git diff`) with ESLint, run the f yarn eslint-staged ``` -_A list of problems found will be logged to the console._ +A list of problems found will be logged to the console. To apply automatic ESLint fixes to all currently staged files (based on `git diff`), run the following script: @@ -22,7 +28,7 @@ To apply automatic ESLint fixes to all currently staged files (based on `git dif yarn eslint-staged-fix ``` -_If manual changes are required, a list of changes will be sent to the console._ +If manual changes are required, a list of changes will be sent to the console. To check **all** files in the repository with ESLint, run the following script: @@ -30,7 +36,7 @@ To check **all** files in the repository with ESLint, run the following script: yarn eslint ``` -_A list of problems found will be logged to the console._ +A list of problems found will be logged to the console. To apply automatic ESLint fixes to **all** files in the repository, run the following script: @@ -38,7 +44,7 @@ To apply automatic ESLint fixes to **all** files in the repository, run the foll yarn eslint-fix ``` -_If manual changes are required, a list of changes will be sent to the console._ +If manual changes are required, a list of changes will be sent to the console. CAUTION: **Caution:** Limit use to global rule updates. Otherwise, the changes can lead to huge Merge Requests. @@ -54,9 +60,8 @@ rules only if you are invoking/instantiating existing code modules. - [`no-new`](https://eslint.org/docs/rules/no-new) - [`class-method-use-this`](https://eslint.org/docs/rules/class-methods-use-this) -NOTE: **Note:** -Disable these rules on a per-line basis. This makes it easier to refactor -in the future. E.g. use `eslint-disable-next-line` or `eslint-disable-line`. +Disable these rules on a per-line basis. This makes it easier to refactor in the +future. For example, use `eslint-disable-next-line` or `eslint-disable-line`. ### Disabling ESLint for a single violation diff --git a/doc/development/fe_guide/vue.md b/doc/development/fe_guide/vue.md index 58a8332589d..77bdadfe8da 100644 --- a/doc/development/fe_guide/vue.md +++ b/doc/development/fe_guide/vue.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Vue To get started with Vue, read through [their documentation](https://vuejs.org/v2/guide/). @@ -47,7 +53,7 @@ of the new feature should be. The Store and the Service should be imported and initialized in this file and provided as a prop to the main component. -Be sure to read about [page-specific JavaScript](./performance.md#page-specific-javascript). +Be sure to read about [page-specific JavaScript](performance.md#page-specific-javascript). ### Bootstrapping Gotchas @@ -56,33 +62,36 @@ Be sure to read about [page-specific JavaScript](./performance.md#page-specific- While mounting a Vue application, you might need to provide data from Rails to JavaScript. To do that, you can use the `data` attributes in the HTML element and query them while mounting the application. -_Note:_ You should only do this while initializing the application, because the mounted element will be replaced with Vue-generated DOM. +You should only do this while initializing the application, because the mounted element will be replaced with Vue-generated DOM. The advantage of providing data from the DOM to the Vue instance through `props` in the `render` function instead of querying the DOM inside the main Vue component is avoiding the need to create a fixture or an HTML element in the unit test, -which will make the tests easier. See the following example: +which will make the tests easier. + +See the following example, also, please refer to our [Vue style guide](style/vue.md#basic-rules) for additional +information on why we explicitly declare the data being passed into the Vue app; ```javascript // haml #js-vue-app{ data: { endpoint: 'foo' }} // index.js -document.addEventListener('DOMContentLoaded', () => new Vue({ - el: '#js-vue-app', - data() { - const dataset = this.$options.el.dataset; - return { - endpoint: dataset.endpoint, - }; - }, +const el = document.getElementById('js-vue-app'); + +if (!el) return false; + +const { endpoint } = el.dataset; + +return new Vue({ + el, render(createElement) { return createElement('my-component', { props: { - endpoint: this.endpoint, + endpoint }, }); }, -})); +} ``` > When adding an `id` attribute to mount a Vue application, please make sure this `id` is unique across the codebase @@ -94,7 +103,7 @@ By following this practice, we can avoid the need to mock the `gl` object, which It should be done while initializing our Vue instance, and the data should be provided as `props` to the main component: ```javascript -document.addEventListener('DOMContentLoaded', () => new Vue({ +return new Vue({ el: '.js-vue-app', render(createElement) { return createElement('my-component', { @@ -103,7 +112,7 @@ document.addEventListener('DOMContentLoaded', () => new Vue({ }, }); }, -})); +}); ``` #### Accessing feature flags @@ -111,7 +120,7 @@ document.addEventListener('DOMContentLoaded', () => new Vue({ Use Vue's [provide/inject](https://vuejs.org/v2/api/#provide-inject) mechanism to make feature flags available to any descendant components in a Vue application. The `glFeatures` object is already provided in `commons/vue.js`, so -only the mixin is required to utilize the flags: +only the mixin is required to use the flags: ```javascript // An arbitrary descendant component @@ -179,13 +188,40 @@ Check this [page](vuex.md) for more details. - It is acceptable for Vue to listen to existing jQuery events using jQuery event listeners. - It is not recommended to add new jQuery events for Vue to interact with jQuery. +### Mixing Vue and JavaScript classes (in the data function) + +In the [Vue documentation](https://vuejs.org/v2/api/#Options-Data) the Data function/object is defined as follows: + +> The data object for the Vue instance. Vue will recursively convert its properties into getter/setters to make it “reactive”. The object must be plain: native objects such as browser API objects and prototype properties are ignored. A rule of thumb is that data should just be data - it is not recommended to observe objects with their own stateful behavior. + +Based on the Vue guidance: + +- **Do not** use or create a JavaScript class in your [data function](https://vuejs.org/v2/api/#data), such as `user: new User()`. +- **Do not** add new JavaScript class implementations. +- **Do** use [GraphQL](../api_graphql_styleguide.md), [Vuex](vuex.md) or a set of components if cannot use simple primitives or objects. +- **Do** maintain existing implementations using such approaches. +- **Do** Migrate components to a pure object model when there are substantial changes to it. +- **Do** add business logic to helpers or utils, so you can test them separately from your component. + +#### Why + +There are additional reasons why having a JavaScript class presents maintainability issues on a huge codebase: + +- Once a class is created, it is easy to extend it in a way that can infringe Vue reactivity and best practices. +- A class adds a layer of abstraction, which makes the component API and its inner workings less clear. +- It makes it harder to test. Since the class is instantiated by the component data function, it is harder to 'manage' component and class separately. +- Adding OOP to a functional codebase adds yet another way of writing code, reducing consistency and clarity. + ## Style guide Please refer to the Vue section of our [style guide](style/vue.md) -for best practices while writing your Vue components and templates. +for best practices while writing and testing your Vue components and templates. ## Testing Vue Components +Please refer to the [Vue testing style guide](style/vue.md#vue-testing) +for guidelines and best practices for testing your Vue components. + Each Vue component has a unique output. This output is always present in the render function. Although we can test each method of a Vue component individually, our goal must be to test the output @@ -226,7 +262,7 @@ describe('~/todos/app.vue', () => { mock.restore(); }); - // NOTE: It is very helpful to separate setting up the component from + // It is very helpful to separate setting up the component from // its collaborators (i.e. Vuex, axios, etc.) const createWrapper = (props = {}) => { wrapper = shallowMount(App, { @@ -236,7 +272,7 @@ describe('~/todos/app.vue', () => { }, }); }; - // NOTE: Helper methods greatly help test maintainability and readability. + // Helper methods greatly help test maintainability and readability. const findLoader = () => wrapper.find(GlLoadingIcon); const findAddButton = () => wrapper.find('[data-testid="add-button"]'); const findTextInput = () => wrapper.find('[data-testid="text-input"]'); diff --git a/doc/development/fe_guide/vue3_migration.md b/doc/development/fe_guide/vue3_migration.md index afdb2690c02..46437f39dbe 100644 --- a/doc/development/fe_guide/vue3_migration.md +++ b/doc/development/fe_guide/vue3_migration.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Migration to Vue 3 In order to prepare for the eventual migration to Vue 3.x, we should be wary about adding the following features to the codebase: @@ -76,7 +82,6 @@ const FunctionalComp = (props, slots) => { } ``` -NOTE: **Note:** It is not recommended to replace stateful components with functional components unless you absolutely need a performance improvement right now. In Vue 3, performance gains for functional components will be negligible. ## Old slots syntax with `slot` attribute diff --git a/doc/development/fe_guide/vuex.md b/doc/development/fe_guide/vuex.md index 528dcb3b7f4..7765ba04d40 100644 --- a/doc/development/fe_guide/vuex.md +++ b/doc/development/fe_guide/vuex.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Vuex When there's a clear benefit to separating state management from components (e.g. due to state complexity) we recommend using [Vuex](https://vuex.vuejs.org) over any other Flux pattern. Otherwise, feel free to manage state within the components. @@ -9,14 +15,14 @@ Vuex should be strongly considered when: - There are complex interactions with Backend, e.g. multiple API calls - The app involves interacting with backend via both traditional REST API and GraphQL (especially when moving the REST API over to GraphQL is a pending backend task) -_Note:_ All of the below is explained in more detail in the official [Vuex documentation](https://vuex.vuejs.org). +The information included in this page is explained in more detail in the +official [Vuex documentation](https://vuex.vuejs.org). ## Separation of concerns Vuex is composed of State, Getters, Mutations, Actions, and Modules. -When a user clicks on an action, we need to `dispatch` it. This action will `commit` a mutation that will change the state. -_Note:_ The action itself will not update the state, only a mutation should update the state. +When a user clicks on an action, we need to `dispatch` it. This action will `commit` a mutation that will change the state. The action itself will not update the state; only a mutation should update the state. ## File structure @@ -60,7 +66,7 @@ export const createStore = () => The first thing you should do before writing any code is to design the state. -Often we need to provide data from haml to our Vue application. Let's store it in the state for better access. +Often we need to provide data from HAML to our Vue application. Let's store it in the state for better access. ```javascript export default () => ({ @@ -354,8 +360,8 @@ export default initialState => ({ ``` We've made the conscious decision to avoid this pattern to aid in the -discoverability and searchability of our frontend codebase. The reasoning for -this is described in [this +discoverability and searchability of our frontend codebase. The same applies +when [providing data to a Vue app](vue.md#providing-data-from-haml-to-javascript). The reasoning for this is described in [this discussion](https://gitlab.com/gitlab-org/frontend/rfcs/-/issues/56#note_302514865): > Consider a `someStateKey` is being used in the store state. You _may_ not be diff --git a/doc/development/feature_categorization/index.md b/doc/development/feature_categorization/index.md index 57e0ad8b772..66ed2952250 100644 --- a/doc/development/feature_categorization/index.md +++ b/doc/development/feature_categorization/index.md @@ -1,8 +1,14 @@ +--- +stage: Enablement +group: Infrastructure +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Feature Categorization > [Introduced](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/269) in GitLab 13.2. -Each Sidekiq worker, controller action, or (eventually) API endpoint +Each Sidekiq worker, controller action, or API endpoint must declare a `feature_category` attribute. This attribute maps each of these to a [feature category](https://about.gitlab.com/handbook/product/product-categories/). This @@ -112,3 +118,42 @@ assigned to all actions. The spec also validates if the used feature categories are known. And if the actions used in configuration still exist as routes. + +## API endpoints + +Grape API endpoints can use the `feature_category` class method, like +[Rails controllers](#rails-controllers) do: + +```ruby +module API + class Issues < ::API::Base + feature_category :issue_tracking + end +end +``` + +The second argument can be used to specify feature categories for +specific routes: + +```ruby +module API + class Users < ::API::Base + feature_category :users, ['/users/:id/custom_attributes', '/users/:id/custom_attributes/:key'] + end +end +``` + +Or the feature category can be specified in the action itself: + +```ruby +module API + class Users < ::API::Base + get ':id', feature_category: :users do + end + end +end +``` + +As with Rails controllers, an API class must specify the category for +every single action unless the same category is used for every action +within that class. diff --git a/doc/development/feature_flags/controls.md b/doc/development/feature_flags/controls.md index ef38a85bec0..df737912c00 100644 --- a/doc/development/feature_flags/controls.md +++ b/doc/development/feature_flags/controls.md @@ -92,9 +92,7 @@ Guidelines: 1. If the feature meets the requirements for creating a [Change Management](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/#feature-flags-and-the-change-management-process) issue, create a Change Management issue per [criticality guidelines](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/#change-request-workflows). 1. For simple, low-risk, easily reverted features, proceed and [enable the feature in `#production`](#process). -1. For features that impact the user experience, consider notifying - `#support_gitlab-com` first. - `#support_gitlab-com` beforehand. +1. For features that impact the user experience, consider notifying `#support_gitlab-com` beforehand. #### Process @@ -215,7 +213,6 @@ actors. Feature.enabled?(:some_feature, group) ``` -NOTE: **Note:** **Percentage of time** rollout is not a good idea if what you want is to make sure a feature is always on or off to the users. In that case, **Percentage of actors** rollout is a better method. diff --git a/doc/development/feature_flags/development.md b/doc/development/feature_flags/development.md index 067e480f6f8..2855662e1db 100644 --- a/doc/development/feature_flags/development.md +++ b/doc/development/feature_flags/development.md @@ -35,7 +35,6 @@ used so that unfinished code can be deployed in production. A `development` feature flag should have a rollout issue, ideally created using the [Feature Flag Roll Out template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/Feature%20Flag%20Roll%20Out.md). -NOTE: **Note:** This is the default type used when calling `Feature.enabled?`. ### `ops` type @@ -61,30 +60,6 @@ Feature.disabled?(:my_ops_flag, project, type: :ops) push_frontend_feature_flag(:my_ops_flag, project, type: :ops) ``` -### `licensed` type - -`licensed` feature flags are used to temporarily disable licensed features. There -should be a one-to-one mapping of `licensed` feature flags to licensed features. - -`licensed` feature flags must be `default_enabled: true`, because that's the only -supported option in the current implementation. This is under development as per -the [related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/218667). - -The `licensed` type has a dedicated set of functions to check if a licensed -feature is available for a project or namespace. This check validates -if the license is assigned to the namespace and feature flag itself. -The `licensed` feature flag has the same name as a licensed feature name: - -```ruby -# Good: checks if feature flag is enabled -project.feature_available?(:my_licensed_feature) -namespace.feature_available?(:my_licensed_feature) - -# Bad: licensed flag must be accessed via `feature_available?` -Feature.enabled?(:my_licensed_feature, type: :licensed) -push_frontend_feature_flag(:my_licensed_feature, type: :licensed) -``` - ## Feature flag definition and validation > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229161) in GitLab 13.3. @@ -128,7 +103,7 @@ a YAML definition in `config/feature_flags` or `ee/config/feature_flags`. Only feature flags that have a YAML definition file can be used when running the development or testing environments. ```shell -$ bin/feature-flag my-feature-flag +$ bin/feature-flag my_feature_flag >> Specify the group introducing the feature flag, like `group::apm`: ?> group::memory @@ -140,9 +115,9 @@ https://gitlab.com/gitlab-org/gitlab/-/issues/new?issue%5Btitle%5D=%5BFeature+fl >> URL of the rollout issue (enter to skip): ?> https://gitlab.com/gitlab-org/gitlab/-/issues/232533 -create config/feature_flags/development/test-flag.yml +create config/feature_flags/development/my_feature_flag.yml --- -name: test-flag +name: my_feature_flag introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38602 rollout_issue_url: https://gitlab.com/gitlab-org/gitlab/-/issues/232533 group: group::memory @@ -212,6 +187,30 @@ if Feature.disabled?(:my_feature_flag, project, type: :ops) end ``` +DANGER: **Warning:** +Don't use feature flags at application load time. For example, using the `Feature` class in +`config/initializers/*` or at the class level could cause an unexpected error. This error occurs +because a database that a feature flag adapter might depend on doesn't exist at load time +(especially for fresh installations). Checking for the database's existence at the caller isn't +recommended, as some adapters don't require a database at all (for example, the HTTP adapter). The +feature flag setup check must be abstracted in the `Feature` namespace. This approach also requires +application reload when the feature flag changes. You must therefore ask SREs to reload the +Web/API/Sidekiq fleet on production, which takes time to fully rollout/rollback the changes. For +these reasons, use environment variables (for example, `ENV['YOUR_FEATURE_NAME']`) or `gitlab.yml` +instead. + +Here's an example of a pattern that you should avoid: + +```ruby +class MyClass + if Feature.enabled?(:...) + new_process + else + legacy_process + end +end +``` + ### Frontend Use the `push_frontend_feature_flag` method for frontend code, which is @@ -309,57 +308,16 @@ used as an actor for `Feature.enabled?`. ### Feature flags for licensed features -If a feature is license-gated, there's no need to add an additional -explicit feature flag check since the flag is checked as part of the -`License.feature_available?` call. Similarly, there's no need to "clean up" a -feature flag once the feature has reached general availability. - -The [`Project#feature_available?`](https://gitlab.com/gitlab-org/gitlab/blob/4cc1c62918aa4c31750cb21dfb1a6c3492d71080/app/models/project_feature.rb#L63-68), -[`Namespace#feature_available?`](https://gitlab.com/gitlab-org/gitlab/blob/4cc1c62918aa4c31750cb21dfb1a6c3492d71080/ee/app/models/ee/namespace.rb#L71-85) (EE), and -[`License.feature_available?`](https://gitlab.com/gitlab-org/gitlab/blob/4cc1c62918aa4c31750cb21dfb1a6c3492d71080/ee/app/models/license.rb#L293-300) (EE) methods all implicitly check for -a by default enabled feature flag with the same name as the provided argument. - -**An important side-effect of the implicit feature flags mentioned above is that -unless the feature is explicitly disabled or limited to a percentage of users, -the feature flag check defaults to `true`.** - -NOTE: **Note:** -Due to limitations with `feature_available?`, the YAML definition for `licensed` feature -flags accepts only `default_enabled: true`. This is under development as per the -[related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/218667). - -#### Alpha/beta licensed feature flags - -This is relevant when developing the feature using -[several smaller merge requests](https://about.gitlab.com/handbook/values/#make-small-merge-requests), or when the feature is considered to be an -[alpha or beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha-beta-ga), and -should not be available by default. - -As an example, if you were to ship the frontend half of a feature without the -backend, you'd want to disable the feature entirely until the backend half is -also ready to be shipped. To make sure this feature is disabled for both -GitLab.com and self-managed instances, you should use the -[`Namespace#alpha_feature_available?`](https://gitlab.com/gitlab-org/gitlab/blob/458749872f4a8f27abe8add930dbb958044cb926/ee/app/models/ee/namespace.rb#L113) or -[`Namespace#beta_feature_available?`](https://gitlab.com/gitlab-org/gitlab/blob/458749872f4a8f27abe8add930dbb958044cb926/ee/app/models/ee/namespace.rb#L100-112) -method, according to our [definitions](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha-beta-ga). This ensures the feature is disabled unless the feature flag is -_explicitly_ enabled. - -CAUTION: **Caution:** -If `alpha_feature_available?` or `beta_feature_available?` is used, the YAML definition -for the feature flag must use `default_enabled: [false, true]`, because the usage -of the feature flag is undefined. These methods may change, as per the -[related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/218667). +You can't use a feature flag with the same name as a licensed feature name, because +it would cause a naming collision. This was [widely discussed and removed](https://gitlab.com/gitlab-org/gitlab/-/issues/259611) +because it is confusing. -The resulting YAML should be similar to: +To check for licensed features, add a dedicated feature flag under a different name +and check it explicitly, for example: -```yaml -name: scoped_labels -group: group::memory -type: licensed -# The `default_enabled:` is undefined -# as `feature_available?` uses `default_enabled: true` -# as `beta_feature_available?` uses `default_enabled: false` -default_enabled: [false, true] +```ruby +Feature.enabled?(:licensed_feature_feature_flag, project) && + project.feature_available?(:licensed_feature) ``` ### Feature groups @@ -397,7 +355,6 @@ Introducing a feature flag into the codebase creates an additional code path tha It is strongly advised to test all code affected by a feature flag, both when **enabled** and **disabled** to ensure the feature works properly. -NOTE: **Note:** When using the testing environment, all feature flags are enabled by default. To disable a feature flag in a test, use the `stub_feature_flags` diff --git a/doc/development/feature_flags/index.md b/doc/development/feature_flags/index.md index 2643571aec3..a867bcf792a 100644 --- a/doc/development/feature_flags/index.md +++ b/doc/development/feature_flags/index.md @@ -1,5 +1,4 @@ --- -type: index, dev stage: none group: Development info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" diff --git a/doc/development/feature_flags/process.md b/doc/development/feature_flags/process.md index b327eec58e8..b282424f59a 100644 --- a/doc/development/feature_flags/process.md +++ b/doc/development/feature_flags/process.md @@ -80,7 +80,7 @@ In order to build a final release and present the feature for self-managed users, the feature flag should be at least defaulted to **on**. If the feature is deemed stable and there is confidence that removing the feature flag is safe, consider removing the feature flag altogether. It's _strongly_ recommended that -the feature flag is [enabled **globally** on **production**](./controls.md#enabling-a-feature-for-gitlabcom) for **at least one day** +the feature flag is [enabled **globally** on **production**](controls.md#enabling-a-feature-for-gitlabcom) for **at least one day** before making this decision. Unexpected bugs are sometimes discovered during this period. The process for enabling features that are disabled by default can take 5-6 days diff --git a/doc/development/features_inside_dot_gitlab.md b/doc/development/features_inside_dot_gitlab.md index cb883471adf..e8918c76d04 100644 --- a/doc/development/features_inside_dot_gitlab.md +++ b/doc/development/features_inside_dot_gitlab.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Features inside the `.gitlab/` directory We have implemented standard features that depend on configuration files in the `.gitlab/` directory. You can find `.gitlab/` in various GitLab repositories. diff --git a/doc/development/file_storage.md b/doc/development/file_storage.md index e8ae5a11d48..aa91e105513 100644 --- a/doc/development/file_storage.md +++ b/doc/development/file_storage.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # File Storage in GitLab We use the [CarrierWave](https://github.com/carrierwaveuploader/carrierwave) gem to handle file upload, store and retrieval. @@ -50,10 +56,10 @@ In the case of Issues/MR/Notes Markdown attachments, there is a different approa instead of basing the path into a mutable variable `:project_path_with_namespace`, it's possible to use the hash of the project ID instead, if project migrates to the new approach (introduced in 10.2). -> Note: We provide an [all-in-one Rake task](../administration/raketasks/uploads/migrate.md) to migrate all uploads to object -> storage in one go. If a new Uploader class or model type is introduced, make -> sure you add a Rake task invocation corresponding to it to the -> [category list](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/tasks/gitlab/uploads/migrate.rake). +We provide an [all-in-one Rake task](../administration/raketasks/uploads/migrate.md) +to migrate all uploads to object storage in one go. If a new Uploader class or model +type is introduced, make sure you add a Rake task invocation corresponding to it to the +[category list](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/tasks/gitlab/uploads/migrate.rake). ### Path segments @@ -101,7 +107,7 @@ The `CarrierWave::Uploader#store_dir` is overridden to ### Using `ObjectStorage::Extension::RecordsUploads` -> Note: this concern will automatically include `RecordsUploads::Concern` if not already included. +This concern will automatically include `RecordsUploads::Concern` if not already included. The `ObjectStorage::Concern` uploader will search for the matching `Upload` to select the correct object store. The `Upload` is mapped using `#store_dirs + identifier` for each store (LOCAL/REMOTE). diff --git a/doc/development/foreign_keys.md b/doc/development/foreign_keys.md index 8a81dc158a7..ff8c03ce76b 100644 --- a/doc/development/foreign_keys.md +++ b/doc/development/foreign_keys.md @@ -1,3 +1,9 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Foreign Keys & Associations When adding an association to a model you must also add a foreign key. For diff --git a/doc/development/gemfile.md b/doc/development/gemfile.md index 8d93c52e7bc..1daa3ad5cba 100644 --- a/doc/development/gemfile.md +++ b/doc/development/gemfile.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # `Gemfile` guidelines When adding a new entry to `Gemfile` or upgrading an existing dependency pay diff --git a/doc/development/geo/framework.md b/doc/development/geo/framework.md index 55f4be07bb4..e440e324c4a 100644 --- a/doc/development/geo/framework.md +++ b/doc/development/geo/framework.md @@ -4,17 +4,12 @@ group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Geo self-service framework (alpha) +# Geo self-service framework NOTE: **Note:** -This document might be subjected to change. It's a -proposal we're working on and once the implementation is complete this -documentation will be updated. Follow progress in the -[epic](https://gitlab.com/groups/gitlab-org/-/epics/2161). - -NOTE: **Note:** -The Geo self-service framework is currently in -alpha. If you need to replicate a new data type, reach out to the Geo +This document is subject to change as we continue to implement and iterate on the framework. +Follow the progress in the [epic](https://gitlab.com/groups/gitlab-org/-/epics/2161). +If you need to replicate a new data type, reach out to the Geo team to discuss the options. You can contact them in `#g_geo` on Slack or mention `@geo-team` in the issue or merge request. @@ -26,38 +21,40 @@ minimal effort of the engineer who created a data type. ## Nomenclature Before digging into the API, developers need to know some Geo-specific -naming conventions. +naming conventions: -Model -: A model is an Active Model, which is how it is known in the entire +- **Model**: + A model is an Active Model, which is how it is known in the entire Rails codebase. It usually is tied to a database table. From Geo perspective, a model can have one or more resources. -Resource -: A resource is a piece of data that belongs to a model and is +- **Resource**: + A resource is a piece of data that belongs to a model and is produced by a GitLab feature. It is persisted using a storage - mechanism. By default, a resource is not a replicable. + mechanism. By default, a resource is not a Geo replicable. -Data type -: Data type is how a resource is stored. Each resource should +- **Data type**: + Data type is how a resource is stored. Each resource should fit in one of the data types Geo supports: -:- Git repository -:- Blob -:- Database -: For more detail, see [Data types](../../administration/geo/replication/datatypes.md). + - Git repository + - Blob + - Database + + For more detail, see [Data types](../../administration/geo/replication/datatypes.md). -Geo Replicable -: A Replicable is a resource Geo wants to sync across Geo nodes. There +- **Geo Replicable**: + A Replicable is a resource Geo wants to sync across Geo nodes. There is a limited set of supported data types of replicables. The effort required to implement replication of a resource that belongs to one of the known data types is minimal. -Geo Replicator -: A Geo Replicator is the object that knows how to replicate a +- **Geo Replicator**: + A Geo Replicator is the object that knows how to replicate a replicable. It's responsible for: -:- Firing events (producer) -:- Consuming events (consumer) -: It's tied to the Geo Replicable data type. All replicators have a + - Firing events (producer) + - Consuming events (consumer) + + It's tied to the Geo Replicable data type. All replicators have a common interface that can be used to process (that is, produce and consume) events. It takes care of the communication between the primary node (where events are produced) and the secondary node @@ -65,8 +62,8 @@ Geo Replicator Geo in their feature will use the API of replicators to make this happen. -Geo Domain-Specific Language -: The syntactic sugar that allows engineers to easily specify which +- **Geo Domain-Specific Language**: + The syntactic sugar that allows engineers to easily specify which resources should be replicated and how. ## Geo Domain-Specific Language @@ -144,7 +141,7 @@ replicator.model_record ``` The replicator can be used to generate events, for example in -ActiveRecord hooks: +`ActiveRecord` hooks: ```ruby after_create_commit -> { replicator.publish_created_event } @@ -207,9 +204,12 @@ For example, to add support for files referenced by a `Widget` model with a end ``` + If there is a common constraint for records to be available for replication, + make sure to also overwrite the `available_replicables` scope. + 1. Create `ee/app/replicators/geo/widget_replicator.rb`. Implement the - `#carrierwave_uploader` method which should return a `CarrierWave::Uploader`. - And implement the class method `.model` to return the `Widget` class. + `#carrierwave_uploader` method which should return a `CarrierWave::Uploader`, + and implement the class method `.model` to return the `Widget` class: ```ruby # frozen_string_literal: true @@ -236,7 +236,7 @@ For example, to add support for files referenced by a `Widget` model with a ``` 1. Add this replicator class to the method `replicator_classes` in -`ee/lib/gitlab/geo.rb`: + `ee/lib/gitlab/geo.rb`: ```ruby REPLICATOR_CLASSES = [ @@ -247,8 +247,8 @@ For example, to add support for files referenced by a `Widget` model with a ``` 1. Create `ee/spec/replicators/geo/widget_replicator_spec.rb` and perform - the setup necessary to define the `model_record` variable for the shared - examples. + the necessary setup to define the `model_record` variable for the shared + examples: ```ruby # frozen_string_literal: true @@ -319,9 +319,7 @@ For example, to add support for files referenced by a `Widget` model with a ``` 1. Update `REGISTRY_CLASSES` in `ee/app/workers/geo/secondary/registry_consistency_worker.rb`. - 1. Add `widget_registry` to `ActiveSupport::Inflector.inflections` in `config/initializers_before_autoloader/000_inflections.rb`. - 1. Create `ee/spec/factories/geo/widget_registry.rb`: ```ruby @@ -375,17 +373,17 @@ For example, to add support for files referenced by a `Widget` model with a end ``` -Widgets should now be replicated by Geo! +Widgets should now be replicated by Geo. #### Verification There are two ways to add verification related fields so that the Geo primary -can track verification state: +can track verification state. ##### Option 1: Add verification state fields to the existing `widgets` table itself 1. Add a migration to add columns ordered according to [our guidelines](../ordering_table_columns.md) -for verification state to the widgets table: + for verification state to the widgets table: ```ruby # frozen_string_literal: true @@ -462,7 +460,7 @@ for verification state to the widgets table: 1. Create a `widget_states` table and add a partial index on `verification_failure` and `verification_checksum` to ensure re-verification can be performed efficiently. Order - the columns according to [our guidelines](../ordering_table_columns.md): + the columns according to [the guidelines](../ordering_table_columns.md): ```ruby # frozen_string_literal: true @@ -520,12 +518,12 @@ for verification state to the widgets table: To do: Add verification on secondaries. This should be done as part of [Geo: Self Service Framework - First Implementation for Package File verification](https://gitlab.com/groups/gitlab-org/-/epics/1817) -Widgets should now be verified by Geo! +Widgets should now be verified by Geo. #### Metrics Metrics are gathered by `Geo::MetricsUpdateWorker`, persisted in -`GeoNodeStatus` for display in the UI, and sent to Prometheus. +`GeoNodeStatus` for display in the UI, and sent to Prometheus: 1. Add fields `widgets_count`, `widgets_checksummed_count`, `widgets_checksum_failed_count`, `widgets_synced_count`, @@ -560,8 +558,12 @@ Metrics are gathered by `Geo::MetricsUpdateWorker`, persisted in end ``` +1. Make sure the factory also allows setting a `project` attribute. If the model + does not have a direct relation to a project, you can use a `transient` + attribute. Check out `spec/factories/merge_request_diffs.rb` for an example. + Widget replication and verification metrics should now be available in the API, -the Admin Area UI, and Prometheus! +the Admin Area UI, and Prometheus. #### GraphQL API @@ -578,7 +580,6 @@ the Admin Area UI, and Prometheus! 1. Add the new `widget_registries` field name to the `expected_fields` array in `ee/spec/graphql/types/geo/geo_node_type_spec.rb`. - 1. Create `ee/app/graphql/resolvers/geo/widget_registries_resolver.rb`: ```ruby @@ -687,14 +688,14 @@ the Admin Area UI, and Prometheus! ``` Individual widget synchronization and verification data should now be available -via the GraphQL API! +via the GraphQL API. -1. Take care of replicating "update" events. Geo Framework does not currently support - replicating "update" events because all entities added to the framework, by this time, - are immutable. If this is the case - for the entity you're going to add, please follow <https://gitlab.com/gitlab-org/gitlab/-/issues/118743> - and <https://gitlab.com/gitlab-org/gitlab/-/issues/118745> as examples to add the new event type. - Please also remove this notice when you've added it. +Make sure to replicate the "update" events. Geo Framework does not currently support +replicating "update" events because all entities added to the framework, by this time, +are immutable. If this is the case +for the entity you're going to add, follow <https://gitlab.com/gitlab-org/gitlab/-/issues/118743> +and <https://gitlab.com/gitlab-org/gitlab/-/issues/118745> as examples to add the new event type. +Also, remove this notice when you've added it. #### Admin UI @@ -702,7 +703,7 @@ To do: This should be done as part of [Geo: Implement frontend for Self-Service Framework replicables](https://gitlab.com/groups/gitlab-org/-/epics/2525) Widget sync and verification data (aggregate and individual) should now be -available in the Admin UI! +available in the Admin UI. #### Releasing the feature diff --git a/doc/development/git_object_deduplication.md b/doc/development/git_object_deduplication.md index 06af34d9232..4f1afed24ba 100644 --- a/doc/development/git_object_deduplication.md +++ b/doc/development/git_object_deduplication.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # How Git object deduplication works in GitLab When a GitLab user [forks a project](../user/project/repository/forking_workflow.md), @@ -27,7 +33,7 @@ configuration. Objects in A that are not in B will remain in A. For this to work, it is of course critical that **no objects ever get deleted from B** because A might need them. -DANGER: **Danger:** +DANGER: **Warning:** Do not run `git prune` or `git gc` in pool repositories! This can cause data loss in "real" repositories that depend on the pool in question. @@ -156,7 +162,7 @@ repository and a pool. ### Pool existence -If GitLab thinks a pool repository exists (i.e. it exists according to +If GitLab thinks a pool repository exists (i.e. it exists according to SQL), but it does not on the Gitaly server, then it will be created on the fly by Gitaly. diff --git a/doc/development/github_importer.md b/doc/development/github_importer.md index 5a490737f37..9fa55d2662a 100644 --- a/doc/development/github_importer.md +++ b/doc/development/github_importer.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Working with the GitHub importer In GitLab 10.2 a new version of the GitHub importer was introduced. This new diff --git a/doc/development/go_guide/dependencies.md b/doc/development/go_guide/dependencies.md index b85344635c6..461ee394533 100644 --- a/doc/development/go_guide/dependencies.md +++ b/doc/development/go_guide/dependencies.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Dependency Management in Go Go takes an unusual approach to dependency management, in that it is diff --git a/doc/development/go_guide/index.md b/doc/development/go_guide/index.md index 15d25d2d1ed..4077cf2a2c2 100644 --- a/doc/development/go_guide/index.md +++ b/doc/development/go_guide/index.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Go standards and style guidelines This document describes various guidelines and best practices for GitLab diff --git a/doc/development/gotchas.md b/doc/development/gotchas.md index cc3db267d53..691027f385f 100644 --- a/doc/development/gotchas.md +++ b/doc/development/gotchas.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Gotchas The purpose of this guide is to document potential "gotchas" that contributors @@ -157,7 +163,7 @@ allow_next_found_instance_of(Project) do |project| end ``` -_**Note:** Since Active Record is not calling the `.new` method on model classes to instantiate the objects, +Since Active Record is not calling the `.new` method on model classes to instantiate the objects, you should use `expect_next_found_instance_of` or `allow_next_found_instance_of` mock helpers to setup mock on objects returned by Active Record query & finder methods._ If we also want to initialize the instance with some particular arguments, we @@ -182,7 +188,7 @@ refresh_service.execute(oldrev, newrev, ref) See ["Why is it bad style to `rescue Exception => e` in Ruby?"](https://stackoverflow.com/questions/10048173/why-is-it-bad-style-to-rescue-exception-e-in-ruby). -_**Note:** This rule is [enforced automatically by +This rule is [enforced automatically by RuboCop](https://gitlab.com/gitlab-org/gitlab-foss/blob/8-4-stable/.rubocop.yml#L911-914)._ ## Do not use inline JavaScript in views @@ -190,8 +196,8 @@ RuboCop](https://gitlab.com/gitlab-org/gitlab-foss/blob/8-4-stable/.rubocop.yml# Using the inline `:javascript` Haml filters comes with a performance overhead. Using inline JavaScript is not a good way to structure your code and should be avoided. -_**Note:** We've [removed these two filters](https://gitlab.com/gitlab-org/gitlab/blob/master/config/initializers/hamlit.rb) -in an initializer._ +We've [removed these two filters](https://gitlab.com/gitlab-org/gitlab/blob/master/config/initializers/hamlit.rb) +in an initializer. ### Further reading diff --git a/doc/development/graphql_guide/batchloader.md b/doc/development/graphql_guide/batchloader.md new file mode 100644 index 00000000000..6d529358499 --- /dev/null +++ b/doc/development/graphql_guide/batchloader.md @@ -0,0 +1,121 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + +# GraphQL BatchLoader + +GitLab uses the [batch-loader](https://github.com/exAspArk/batch-loader) Ruby gem to optimize and avoid N+1 SQL queries. + +It is the properties of the GraphQL query tree that create opportunities for batching like this - disconnected nodes might need the same data, but cannot know about themselves. + +## When should you use it? + +We should try to batch DB requests as much as possible during GraphQL **query** execution. There is no need to batch loading during **mutations** because they are executed serially. If you need to make a database query, and it is possible to combine two similar (but not identical) queries, then consider using the batch-loader. + +When implementing a new endpoint we should aim to minimise the number of SQL queries. For stability and scalability we must also ensure that our queries do not suffer from N+1 performance issues. + +## Implementation + +Batch loading is useful when a series of queries for inputs `Qα, Qβ, ... Qω` can be combined to a single query for `Q[α, β, ... ω]`. An example of this is lookups by ID, where we can find two users by usernames as cheaply as one, but real-world examples can be more complex. + +Batchloading is not suitable when the result sets have different sort-orders, grouping, aggregation or other non-composable features. + +There are two ways to use the batch-loader in your code. For simple ID lookups, use `::Gitlab::Graphql::Loaders::BatchModelLoader.new(model, id).find`. For more complex cases, you can use the batch API directly. + +For example, to load a `User` by `username`, we can add batching as follows: + +```ruby +class UserResolver < BaseResolver + type UserType, null: true + argument :username, ::GraphQL::STRING_TYPE, required: true + + def resolve(**args) + BatchLoader::GraphQL.for(username).batch do |usernames, loader| + User.by_username(usernames).each do |user| + loader.call(user.username, user) + end + end + end +end +``` + +- `project_id` is the `ID` of the current project being queried +- `loader.call` is used to map the result back to the input key (here a project ID) +- `BatchLoader::GraphQL` returns a lazy object (suspended promise to fetch the data) + +Here an [example MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/46549) illustrating how to use our `BatchLoading` mechanism. + +## How does it work exactly? + +Each lazy object knows which data it needs to load and how to batch the query. When we need to use the lazy objects (which we announce by calling `#sync`), they will be loaded along with all other similar objects in the current batch. + +Inside the block we execute a batch query for our items (`User`). After that, all we have to do is to call loader by passing an item which was used in `BatchLoader::GraphQL.for` method (`usernames`) and the loaded object itself (`user`): + +```ruby +BatchLoader::GraphQL.for(username).batch do |usernames, loader| + User.by_username(usernames).each do |user| + loader.call(user.username, user) + end +end +``` + +### What does lazy mean? + +It is important to avoid syncing batches too early. In the example below we can see how calling sync too early can eliminate opportunities for batching: + +```ruby +x = find_lazy(1) +y = find_lazy(2) + +# calling .sync will flush the current batch and will inhibit maximum laziness +x.sync + +z = find_lazy(3) + +y.sync +z.sync + +# => will run 2 queries +``` + +```ruby +x = find_lazy(1) +y = find_lazy(2) +z = find_lazy(3) + +x.sync +y.sync +z.sync + +# => will run 1 query +``` + +## Testing + +Any GraphQL field that supports `BatchLoading` should be tested using the `batch_sync` method available in [GraphQLHelpers](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/support/helpers/graphql_helpers.rb). + +```ruby +it 'returns data as a batch' do + results = batch_sync(max_queries: 1) do + [{ id: 1 }, { id: 2 }].map { |args| resolve(args) } + end + + expect(results).to eq(expected_results) +end + +def resolve(args = {}, context = { current_user: current_user }) + resolve(described_class, obj: obj, args: args, ctx: context) +end +``` + +We can also use [QueryRecorder](../query_recorder.md) to make sure we are performing only **one SQL query** per call. + +```ruby +it 'executes only 1 SQL query' do + query_count = ActiveRecord::QueryRecorder.new { subject }.count + + expect(query_count).to eq(1) +end +``` diff --git a/doc/development/graphql_guide/index.md b/doc/development/graphql_guide/index.md index 9d7fb5ba0a8..12b4f9796c7 100644 --- a/doc/development/graphql_guide/index.md +++ b/doc/development/graphql_guide/index.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # GraphQL development guidelines This guide contains all the information to successfully contribute to GitLab's diff --git a/doc/development/graphql_guide/pagination.md b/doc/development/graphql_guide/pagination.md index bf9eaa99158..d5140363396 100644 --- a/doc/development/graphql_guide/pagination.md +++ b/doc/development/graphql_guide/pagination.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # GraphQL pagination ## Types of pagination @@ -59,13 +65,13 @@ Some of the benefits and tradeoffs of keyset pagination are - Performance is much better. -- Data stability is greater since you're not going to miss records due to +- More data stability for end-users since records are not missing from lists due to deletions or insertions. - It's the best way to do infinite scrolling. - It's more difficult to program and maintain. Easy for `updated_at` and - `sort_order`, complicated (or impossible) for complex sorting scenarios. + `sort_order`, complicated (or impossible) for [complex sorting scenarios](#limitations-of-query-complexity). ## Implementation @@ -80,12 +86,171 @@ However, there are some cases where we have to use the offset pagination connection, `OffsetActiveRecordRelationConnection`, such as when sorting by label priority in issues, due to the complexity of the sort. -<!-- ### Keyset pagination --> +### Keyset pagination + +The keyset pagination implementation is a subclass of `GraphQL::Pagination::ActiveRecordRelationConnection`, +which is a part of the `graphql` gem. This is installed as the default for all `ActiveRecord::Relation`. +However, instead of using a cursor based on an offset (which is the default), GitLab uses a more specialized cursor. + +The cursor is created by encoding a JSON object which contains the relevant ordering fields. For example: + +```ruby +ordering = {"id"=>"72410125", "created_at"=>"2020-10-08 18:05:21.953398000 UTC"} +json = ordering.to_json +cursor = Base64Bp.urlsafe_encode64(json, padding: false) + +"eyJpZCI6IjcyNDEwMTI1IiwiY3JlYXRlZF9hdCI6IjIwMjAtMTAtMDggMTg6MDU6MjEuOTUzMzk4MDAwIFVUQyJ9" + +json = Base64Bp.urlsafe_decode64(cursor) +Gitlab::Json.parse(json) + +{"id"=>"72410125", "created_at"=>"2020-10-08 18:05:21.953398000 UTC"} +``` + +The benefits of storing the order attribute values in the cursor: + +- If only the ID of the object were stored, the object and its attributes could be queried. + That would require an additional query, and if the object is no longer there, then the needed + attributes are not available. +- If an attribute is `NULL`, then one SQL query can be used. If it's not `NULL`, then a + different SQL query can be used. + +Based on whether the main attribute field being sorted on is `NULL` in the cursor, the proper query +condition is built. The last ordering field is considered to be unique (a primary key), meaning the +column never contains `NULL` values. + +#### Limitations of query complexity + +We only support two ordering fields, and one of those fields needs to be the primary key. + +Here are two examples of pseudocode for the query: + +- **Two-condition query.** `X` represents the values from the cursor. `C` represents + the columns in the database, sorted in ascending order, using an `:after` cursor, and with `NULL` + values sorted last. + + ```plaintext + X1 IS NOT NULL + AND + (C1 > X1) + OR + (C1 IS NULL) + OR + (C1 = X1 + AND + C2 > X2) + + X1 IS NULL + AND + (C1 IS NULL + AND + C2 > X2) + ``` + + Below is an example based on the relation `Issue.order(relative_position: :asc).order(id: :asc)` + with an after cursor of `relative_position: 1500, id: 500`: + + ```plaintext + when cursor[relative_position] is not NULL + + ("issues"."relative_position" > 1500) + OR ( + "issues"."relative_position" = 1500 + AND + "issues"."id" > 500 + ) + OR ("issues"."relative_position" IS NULL) + + when cursor[relative_position] is NULL + + "issues"."relative_position" IS NULL + AND + "issues"."id" > 500 + ``` + +- **Three-condition query.** The example below is not complete, but shows the + complexity of adding one more condition. `X` represents the values from the cursor. `C` represents + the columns in the database, sorted in ascending order, using an `:after` cursor, and with `NULL` + values sorted last. + + ```plaintext + X1 IS NOT NULL + AND + (C1 > X1) + OR + (C1 IS NULL) + OR + (C1 = X1 AND C2 > X2) + OR + (C1 = X1 + AND + X2 IS NOT NULL + AND + ((C2 > X2) + OR + (C2 IS NULL) + OR + (C2 = X2 AND C3 > X3) + OR + X2 IS NULL..... + ``` + +By using +[`Gitlab::Graphql::Pagination::Keyset::QueryBuilder`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/graphql/pagination/keyset/query_builder.rb), +we're able to build the necessary SQL conditions and apply them to the Active Record relation. + +Complex queries can be difficult or impossible to use. For example, +in [`issuable.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/concerns/issuable.rb), +the `order_due_date_and_labels_priority` method creates a very complex query. + +These types of queries are not supported. In these instances, you can use offset pagination. + +### Offset pagination + +There are times when the [complexity of sorting](#limitations-of-query-complexity) +is more than our keyset pagination can handle. + +For example, in [`IssuesResolver`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/graphql/resolvers/issues_resolver.rb), +when sorting by `priority_asc`, we can't use keyset pagination as the ordering is much +too complex. For more information, read [`issuable.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/concerns/issuable.rb). -<!-- ### Offset pagination --> +In cases like this, we can fall back to regular offset pagination by returning a +[`Gitlab::Graphql::Pagination::OffsetActiveRecordRelationConnection`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/graphql/pagination/offset_active_record_relation_connection.rb) +instead of an `ActiveRecord::Relation`: + +```ruby + def resolve(parent, finder, **args) + issues = apply_lookahead(Gitlab::Graphql::Loaders::IssuableLoader.new(parent, finder).batching_find_all) + + if non_stable_cursor_sort?(args[:sort]) + # Certain complex sorts are not supported by the stable cursor pagination yet. + # In these cases, we use offset pagination, so we return the correct connection. + Gitlab::Graphql::Pagination::OffsetActiveRecordRelationConnection.new(issues) + else + issues + end + end +``` <!-- ### External pagination --> +### External pagination + +There may be times where you need to return data through the GitLab API that is stored in +another system. In these cases you may have to paginate a third-party's API. + +An example of this is with our [Error Tracking](../../operations/error_tracking.md) implementation, +where we proxy [Sentry errors](../../operations/error_tracking.md#sentry-error-tracking) through +the GitLab API. We do this by calling the Sentry API which enforces its own pagination rules. +This means we cannot access the collection within GitLab to perform our own custom pagination. + +For consistency, we manually set the pagination cursors based on values returned by the external API, using `Gitlab::Graphql::ExternallyPaginatedArray.new(previous_cursor, next_cursor, *items)`. + +You can see an example implementation in the following files: + +- [`types/error__tracking/sentry_error_collection_type.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/graphql/types/error_tracking/sentry_error_collection_type.rb) which adds an extension to `field :errors`. +- [`resolvers/error_tracking/sentry_errors_resolver.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/app/graphql/resolvers/error_tracking/sentry_errors_resolver.rb) which returns the data from the resolver. + ## Testing Any GraphQL field that supports pagination and sorting should be tested diff --git a/doc/development/hash_indexes.md b/doc/development/hash_indexes.md index 1ed76e35f69..cc0b0b3a736 100644 --- a/doc/development/hash_indexes.md +++ b/doc/development/hash_indexes.md @@ -1,3 +1,9 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Hash Indexes PostgreSQL supports hash indexes besides the regular B-tree diff --git a/doc/development/i18n/externalization.md b/doc/development/i18n/externalization.md index 59399e54c3e..65a1d83a8fc 100644 --- a/doc/development/i18n/externalization.md +++ b/doc/development/i18n/externalization.md @@ -1,3 +1,9 @@ +--- +stage: Manage +group: Import +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Internationalization for GitLab > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10669) in GitLab 9.2. @@ -15,7 +21,7 @@ All `rake` commands described on this page must be run on a GitLab instance, usu In order to be able to work on the [GitLab Community Edition](https://gitlab.com/gitlab-org/gitlab-foss) project you must download and configure it through [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/set-up-gdk.md). -Once you have the GitLab project ready, you can start working on the translation. +After you have the GitLab project ready, you can start working on the translation. ## Tools @@ -98,9 +104,8 @@ Active Record's `:message` option accepts a `Proc`, so we can do this instead: validates :group_id, uniqueness: { scope: [:project_id], message: -> (object, data) { _("already shared with this group") } } ``` -NOTE: **Note:** Messages in the API (`lib/api/` or `app/graphql`) do -not need to be externalised. +not need to be externalized. ### HAML files @@ -379,8 +384,8 @@ Namespaces should be PascalCase. s__('OpenedNDaysAgo|Opened') ``` -Note: The namespace should be removed from the translation. See the [translation -guidelines for more details](translation.md#namespaced-strings). +The namespace should be removed from the translation. See the +[translation guidelines for more details](translation.md#namespaced-strings). ### HTML @@ -475,6 +480,21 @@ This makes use of [`Intl.DateTimeFormat`](https://developer.mozilla.org/en-US/do ## Best practices +### Minimize translation updates + +Updates can result in the loss of the translations for this string. To minimize risks, +avoid changes to strings, unless they: + +- Add value to the user. +- Include extra context for translators. + +For example, we should avoid changes like this: + +```diff +- _('Number of things: %{count}') % { count: 10 } ++ n_('Number of things: %d', 10) +``` + ### Keep translations dynamic There are cases when it makes sense to keep translations together within an array or a hash. diff --git a/doc/development/i18n/index.md b/doc/development/i18n/index.md index 2d84fe4536f..13f2b43b788 100644 --- a/doc/development/i18n/index.md +++ b/doc/development/i18n/index.md @@ -1,3 +1,9 @@ +--- +stage: Manage +group: Import +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Translate GitLab to your language The text in GitLab's user interface is in American English by default. diff --git a/doc/development/i18n/merging_translations.md b/doc/development/i18n/merging_translations.md index 8b3357c41d3..c4a709f84d3 100644 --- a/doc/development/i18n/merging_translations.md +++ b/doc/development/i18n/merging_translations.md @@ -1,3 +1,9 @@ +--- +stage: Manage +group: Import +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Merging translations from CrowdIn CrowdIn automatically syncs the `gitlab.pot` file with the CrowdIn service, presenting diff --git a/doc/development/i18n/proofreader.md b/doc/development/i18n/proofreader.md index 1916f96801d..56a4f835b3f 100644 --- a/doc/development/i18n/proofreader.md +++ b/doc/development/i18n/proofreader.md @@ -1,3 +1,9 @@ +--- +stage: Manage +group: Import +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Proofread Translations Most translations are contributed, reviewed, and accepted by the community. We @@ -115,10 +121,8 @@ are very appreciative of the work done by translators and proofreaders! ## Become a proofreader -NOTE: **Note:** -Before requesting Proofreader permissions in CrowdIn please make -sure that you have a history of contributing translations to the GitLab -project. +Before requesting Proofreader permissions in CrowdIn, be sure you have a history +of contributing translations to the GitLab project. 1. Contribute translations to GitLab. See instructions for [translating GitLab](translation.md). @@ -140,8 +144,8 @@ project. - link to your GitLab profile - link to your CrowdIn profile - In the merge request description, please include links to any projects you - have previously translated. + In the merge request description, include links to any projects you have + previously translated. 1. Your request to become a proofreader will be considered on the merits of your previous translations by [GitLab team members](https://about.gitlab.com/company/team/) diff --git a/doc/development/i18n/translation.md b/doc/development/i18n/translation.md index ed205c20c0d..128bacfda97 100644 --- a/doc/development/i18n/translation.md +++ b/doc/development/i18n/translation.md @@ -1,3 +1,9 @@ +--- +stage: Manage +group: Import +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Translating GitLab For managing the translation process we use [CrowdIn](https://crowdin.com). @@ -94,7 +100,7 @@ To propose additions to the glossary please ### Inclusive language in French <!-- vale gitlab.Spelling = NO --> -In French, the "écriture inclusive" is now over (see on [Legifrance](https://www.legifrance.gouv.fr/affichTexte.do?cidTexte=JORFTEXT000036068906&categorieLien=id)). +In French, the "écriture inclusive" is now over (see on [Legifrance](https://www.legifrance.gouv.fr/affichTexte.do?cidTexte=JORFTEXT000036068906&categorieLien=id)). So, to include both genders, write “Utilisateurs et utilisatrices” instead of “Utilisateur·rice·s”. When space is missing, the male gender should be used alone. <!-- vale gitlab.Spelling = YES --> diff --git a/doc/development/image_scaling.md b/doc/development/image_scaling.md new file mode 100644 index 00000000000..05f44209bc4 --- /dev/null +++ b/doc/development/image_scaling.md @@ -0,0 +1,96 @@ +--- +stage: Enablement +group: Memory +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + +# Image scaling guide + +This section contains a brief overview of GitLab's image scaler and how to work with it. + +For a general introduction to the history of image scaling at GitLab, you might be interested in +[this Unfiltered blog post](https://about.gitlab.com/blog/2020/11/02/scaling-down-how-we-prototyped-an-image-scaler-at-gitlab/). + +## Why image scaling? + +Since version 13.6, GitLab scales down images on demand in order to reduce the page data footprint. +This both reduces the amount of data "on the wire", but also helps with rendering performance, +since the browser has less work to do. + +## When do we scale images? + +Generally, the image scaler is triggered whenever a client requests an image resource by adding +the `width` parameter to the query string. However, we only scale images of certain kinds and formats. +Whether we allow an image to be rescaled or not is decided by combination of hard-coded rules and configuration settings. + +The hard-coded rules only permit: + +- [Project, group and user avatars](https://gitlab.com/gitlab-org/gitlab/-/blob/fd08748862a5fe5c25b919079858146ea85843ae/app/controllers/concerns/send_file_upload.rb#L65-67) +- [PNGs or JPEGs](https://gitlab.com/gitlab-org/gitlab/-/blob/5dff8fa3814f2a683d8884f468cba1ec06a60972/lib/gitlab/file_type_detection.rb#L23) +- [Specific dimensions](https://gitlab.com/gitlab-org/gitlab/-/blob/5dff8fa3814f2a683d8884f468cba1ec06a60972/app/models/concerns/avatarable.rb#L6) + +Furthermore, configuration in Workhorse can lead to the image scaler rejecting a request if: + +- The image file is too large (controlled by [`max_filesize`](- we only rescale images that do not exceed a configured size in bytes (see [`max_filesize`](https://gitlab.com/gitlab-org/gitlab-workhorse/-/blob/67ab3a2985d2097392f93523ae1cffe0dbf01b31/config.toml.example#L17)))). +- Too many image scalers are already running (controlled by [`max_scaler_procs`](https://gitlab.com/gitlab-org/gitlab-workhorse/-/blob/67ab3a2985d2097392f93523ae1cffe0dbf01b31/config.toml.example#L16)). + +For instance, here are two different URLs that serve the GitLab project avatar both in its +original size and scaled down to 64 pixels. Only the second request will trigger the image scaler: + +- [`/uploads/-/system/project/avatar/278964/logo-extra-whitespace.png`](https://assets.gitlab-static.net/uploads/-/system/project/avatar/278964/logo-extra-whitespace.png) +- [`/uploads/-/system/project/avatar/278964/logo-extra-whitespace.png?width=64`](https://assets.gitlab-static.net/uploads/-/system/project/avatar/278964/logo-extra-whitespace.png?width=64) + +## Where do we scale images? + +Rails and Workhorse currently collaborate to rescale images. This is a common implementation and performance +pattern in GitLab: important business logic such as request authentication and validation +happens in Rails, whereas the "heavy lifting", scaling and serving the binary data, happens in Workhorse. + +The overall request flow is as follows: + +```mermaid +sequenceDiagram + Client->>+Workhorse: GET /uploads/-/system/project/avatar/278964/logo-extra-whitespace.png?width=64 + Workhorse->>+Rails: forward request + Rails->>+Rails: validate request + Rails->>+Rails: resolve image location + Rails-->>-Workhorse: Gitlab-Workhorse-Send-Data: send-scaled-image + Workhorse->>+Workhorse: invoke image scaler + Workhorse-->>-Client: 200 OK +``` + +### Rails + +Currently, image scaling is limited to `Upload` entities, specifically avatars as mentioned above. +Therefore, all image scaling related logic in Rails is currently found in the +[`send_file_upload`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/concerns/send_file_upload.rb) +controller mixin. Upon receiving a request coming from a client through Workhorse, we check whether +it should trigger the image scaler as per the criteria mentioned above, and if so, render a special response +header field (`Gitlab-Workhorse-Send-Data`) with the necessary parameters for Workhorse to carry +out the scaling request. If Rails decides the request does not constitute a valid image scaling request, +we simply follow the path we take to serve any ordinary upload. + +### Workhorse + +Assuming Rails decided the request to be valid, Workhorse will take over. Upon receiving the `send-scaled-image` +instruction through the Rails response, a [special response injecter](https://gitlab.com/gitlab-org/gitlab-workhorse/-/blob/master/internal/imageresizer/image_resizer.go) +will be invoked that knows how to rescale images. The only inputs it requires are the location of the image +(a path if the image resides in block storage, or a URL to remote storage otherwise) and the desired width. +Workhorse will handle the location transparently so Rails does not need to be concerned with where the image +actually resides. + +Additionally, to request validation in Rails, Workhorse will run several pre-condition checks to ensure that +we can actually rescale the image, such as making sure we wouldn't outgrow our scaler process budget but also +if the file meets the configured maximum allowed size constraint (to keep memory consumption in check). + +To actually scale the image, Workhorse will finally fork into a child process that performs the actual +scaling work, and stream the result back to the client. + +#### Caching rescaled images + +We currently do not store rescaled images anywhere; the scaler runs every time a smaller version is requested. +However, Workhorse implements standard conditional HTTP request strategies that allow us to skip the scaler +if the image in the client cache is up-to-date. +To that end we transmit a `Last-Modified` header field carrying the UTC +timestamp of the original image file and match it against the `If-Modified-Since` header field in client requests. +Only if the original image has changed and rescaling becomes necessary do we run the scaler again. diff --git a/doc/development/img/architecture_simplified.png b/doc/development/img/architecture_simplified.png Binary files differindex 72d00b91129..bd731758ddd 100644 --- a/doc/development/img/architecture_simplified.png +++ b/doc/development/img/architecture_simplified.png diff --git a/doc/development/img/bullet_v13_0.png b/doc/development/img/bullet_v13_0.png Binary files differindex e185bdef76d..2a3248e380c 100644 --- a/doc/development/img/bullet_v13_0.png +++ b/doc/development/img/bullet_v13_0.png diff --git a/doc/development/img/distributed_tracing_jaeger_ui.png b/doc/development/img/distributed_tracing_jaeger_ui.png Binary files differindex dcd18b1ec9f..5452294caa3 100644 --- a/doc/development/img/distributed_tracing_jaeger_ui.png +++ b/doc/development/img/distributed_tracing_jaeger_ui.png diff --git a/doc/development/img/memory_ruby_heap_fragmentation.png b/doc/development/img/memory_ruby_heap_fragmentation.png Binary files differindex 4703da7491d..204130f8d87 100644 --- a/doc/development/img/memory_ruby_heap_fragmentation.png +++ b/doc/development/img/memory_ruby_heap_fragmentation.png diff --git a/doc/development/img/merge_ref_head_options_v13_6.png b/doc/development/img/merge_ref_head_options_v13_6.png Binary files differnew file mode 100644 index 00000000000..3134092cc92 --- /dev/null +++ b/doc/development/img/merge_ref_head_options_v13_6.png diff --git a/doc/development/img/performance_bar_cached_queries.png b/doc/development/img/performance_bar_cached_queries.png Binary files differnew file mode 100644 index 00000000000..f5bdc6ffd84 --- /dev/null +++ b/doc/development/img/performance_bar_cached_queries.png diff --git a/doc/development/img/performance_bar_fixed_cached_queries.png b/doc/development/img/performance_bar_fixed_cached_queries.png Binary files differnew file mode 100644 index 00000000000..3f4232fb88f --- /dev/null +++ b/doc/development/img/performance_bar_fixed_cached_queries.png diff --git a/doc/development/img/performance_bar_members_page.png b/doc/development/img/performance_bar_members_page.png Binary files differnew file mode 100644 index 00000000000..bc81cd645e5 --- /dev/null +++ b/doc/development/img/performance_bar_members_page.png diff --git a/doc/development/img/telemetry_system_overview.png b/doc/development/img/telemetry_system_overview.png Binary files differdeleted file mode 100644 index f2e6b300e94..00000000000 --- a/doc/development/img/telemetry_system_overview.png +++ /dev/null diff --git a/doc/development/import_export.md b/doc/development/import_export.md index 8d2be1baf24..a73144abce6 100644 --- a/doc/development/import_export.md +++ b/doc/development/import_export.md @@ -1,3 +1,9 @@ +--- +stage: Manage +group: Import +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Import/Export development documentation Troubleshooting and general development guidelines and tips for the [Import/Export feature](../user/project/settings/import_export.md). @@ -354,28 +360,28 @@ The NDJSON tree will look like this: ```shell tree ├── project -│ ├── auto_devops.ndjson -│ ├── boards.ndjson -│ ├── ci_cd_settings.ndjson -│ ├── ci_pipelines.ndjson -│ ├── container_expiration_policy.ndjson -│ ├── custom_attributes.ndjson -│ ├── error_tracking_setting.ndjson -│ ├── external_pull_requests.ndjson -│ ├── issues.ndjson -│ ├── labels.ndjson -│ ├── merge_requests.ndjson -│ ├── milestones.ndjson -│ ├── pipeline_schedules.ndjson -│ ├── project_badges.ndjson -│ ├── project_feature.ndjson -│ ├── project_members.ndjson -│ ├── protected_branches.ndjson -│ ├── protected_tags.ndjson -│ ├── releases.ndjson -│ ├── services.ndjson -│ ├── snippets.ndjson -│ └── triggers.ndjson +│ ├── auto_devops.ndjson +│ ├── boards.ndjson +│ ├── ci_cd_settings.ndjson +│ ├── ci_pipelines.ndjson +│ ├── container_expiration_policy.ndjson +│ ├── custom_attributes.ndjson +│ ├── error_tracking_setting.ndjson +│ ├── external_pull_requests.ndjson +│ ├── issues.ndjson +│ ├── labels.ndjson +│ ├── merge_requests.ndjson +│ ├── milestones.ndjson +│ ├── pipeline_schedules.ndjson +│ ├── project_badges.ndjson +│ ├── project_feature.ndjson +│ ├── project_members.ndjson +│ ├── protected_branches.ndjson +│ ├── protected_tags.ndjson +│ ├── releases.ndjson +│ ├── services.ndjson +│ ├── snippets.ndjson +│ └── triggers.ndjson └── project.json ``` @@ -389,19 +395,19 @@ The NDJSON tree will look like this: tree └── groups ├── 4351 - │ ├── badges.ndjson - │ ├── boards.ndjson - │ ├── epics.ndjson - │ ├── labels.ndjson - │ ├── members.ndjson - │ └── milestones.ndjson + │ ├── badges.ndjson + │ ├── boards.ndjson + │ ├── epics.ndjson + │ ├── labels.ndjson + │ ├── members.ndjson + │ └── milestones.ndjson ├── 4352 - │ ├── badges.ndjson - │ ├── boards.ndjson - │ ├── epics.ndjson - │ ├── labels.ndjson - │ ├── members.ndjson - │ └── milestones.ndjson + │ ├── badges.ndjson + │ ├── boards.ndjson + │ ├── epics.ndjson + │ ├── labels.ndjson + │ ├── members.ndjson + │ └── milestones.ndjson ├── _all.ndjson ├── 4351.json └── 4352.json diff --git a/doc/development/import_project.md b/doc/development/import_project.md index 9e2f5af6738..8b27b7d28a5 100644 --- a/doc/development/import_project.md +++ b/doc/development/import_project.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Test Import Project For testing, we can import our own [GitLab CE](https://gitlab.com/gitlab-org/gitlab-foss/) project (named `gitlabhq` in this case) under a group named `qa-perf-testing`. Project tarballs that can be used for testing can be found over on the [performance-data](https://gitlab.com/gitlab-org/quality/performance-data) project. A different project could be used if required. @@ -17,7 +23,6 @@ The first option is to simply [import the Project tarball file via the GitLab UI It should take up to 15 minutes for the project to fully import. You can head to the project's main page for the current status. -NOTE: **Note:** This method ignores all the errors silently (including the ones related to `GITALY_DISABLE_REQUEST_LIMITS`) and is used by GitLab's users. For development and testing, check the other methods below. ### Importing via the `import-project` script diff --git a/doc/development/insert_into_tables_in_batches.md b/doc/development/insert_into_tables_in_batches.md index f65d2478d2e..5fe37a95b59 100644 --- a/doc/development/insert_into_tables_in_batches.md +++ b/doc/development/insert_into_tables_in_batches.md @@ -1,4 +1,7 @@ --- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers description: "Sometimes it is necessary to store large amounts of records at once, which can be inefficient when iterating collections and performing individual `save`s. With the arrival of `insert_all` in Rails 6, which operates at the row level (that is, using `Hash`es), GitLab has added a set diff --git a/doc/development/integrations/jenkins.md b/doc/development/integrations/jenkins.md index 5d3c869d922..d81dee94f17 100644 --- a/doc/development/integrations/jenkins.md +++ b/doc/development/integrations/jenkins.md @@ -1,3 +1,9 @@ +--- +stage: Create +group: Ecosystem +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # How to run Jenkins in development environment (on macOS) **(STARTER)** This is a step by step guide on how to set up [Jenkins](https://www.jenkins.io/) on your local machine and connect to it from your GitLab instance. GitLab triggers webhooks on Jenkins, and Jenkins connects to GitLab using the API. By running both applications on the same machine, we can make sure they are able to access each other. diff --git a/doc/development/integrations/jira_connect.md b/doc/development/integrations/jira_connect.md index 1f9b03075f0..66a93f8c947 100644 --- a/doc/development/integrations/jira_connect.md +++ b/doc/development/integrations/jira_connect.md @@ -1,31 +1,37 @@ -# Setting up a development environment +--- +stage: Create +group: Ecosystem +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- -The following are required to install and test the app: - -1. A Jira Cloud instance +# Set up a development environment - Atlassian provides free instances for development and testing. [Click here to sign up](https://developer.atlassian.com/platform/marketplace/getting-started/#free-developer-instances-to-build-and-test-your-app). +The following are required to install and test the app: -1. A GitLab instance available over the internet +- A Jira Cloud instance. Atlassian provides [free instances for development and testing](https://developer.atlassian.com/platform/marketplace/getting-started/#free-developer-instances-to-build-and-test-your-app). +- A GitLab instance available over the internet. For the app to work, Jira Cloud should + be able to connect to the GitLab instance through the internet. To easily expose your + local development environment, you can use tools like: + - [serveo](https://medium.com/automationmaster/how-to-forward-my-local-port-to-public-using-serveo-4979f352a3bf) + - [ngrok](https://ngrok.com). - For the app to work, Jira Cloud should be able to connect to the GitLab instance through the internet. + These also take care of SSL for you because Jira requires all connections to the app + host to be over SSL. - To easily expose your local development environment, you can use tools like - [serveo](https://medium.com/automationmaster/how-to-forward-my-local-port-to-public-using-serveo-4979f352a3bf) - or [ngrok](https://ngrok.com). These also take care of SSL for you because Jira - requires all connections to the app host to be over SSL. +## Install the app in Jira -## Installing the app in Jira +To install the app in Jira: -1. Enable Jira development mode to install apps that are not from the Atlassian Marketplace +1. Enable Jira development mode to install apps that are not from the Atlassian + Marketplace: - 1. Navigate to **Jira settings** (cog icon) > **Apps** > **Manage apps**. + 1. In Jira, navigate to **Jira settings > Apps > Manage apps**. 1. Scroll to the bottom of the **Manage apps** page and click **Settings**. 1. Select **Enable development mode** and click **Apply**. -1. Install the app +1. Install the app: - 1. Navigate to Jira, then choose **Jira settings** (cog icon) > **Apps** > **Manage apps**. + 1. In Jira, navigate to **Jira settings > Apps > Manage apps**. 1. Click **Upload app**. 1. In the **From this URL** field, provide a link to the app descriptor. The host and port must point to your GitLab instance. diff --git a/doc/development/integrations/secure.md b/doc/development/integrations/secure.md index 1094074cab6..9bb92709d54 100644 --- a/doc/development/integrations/secure.md +++ b/doc/development/integrations/secure.md @@ -1,7 +1,13 @@ +--- +stage: Protect +group: Container Security +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Security scanner integration Integrating a security scanner into GitLab consists of providing end users -with a [CI job definition](../../ci/yaml/README.md#introduction) +with a [CI job definition](../../ci/yaml/README.md) they can add to their CI configuration files to scan their GitLab projects. This CI job should then output its results in a GitLab-specified format. These results are then automatically presented in various places in GitLab, such as the Pipeline view, Merge Request @@ -40,12 +46,12 @@ Because the `script` entry can't be left empty, it must be set to the command th It is not possible to rely on the predefined `ENTRYPOINT` and `CMD` of the Docker image to perform the scan automatically, without passing any command. -The [`before_script`](../../ci/yaml/README.md#before_script-and-after_script) +The [`before_script`](../../ci/yaml/README.md#before_script) should not be used in the job definition because users may rely on this to prepare their projects before performing the scan. For instance, it is common practice to use `before_script` to install system libraries a particular project needs before performing SAST or Dependency Scanning. -Similarly, [`after_script`](../../ci/yaml/README.md#before_script-and-after_script) +Similarly, [`after_script`](../../ci/yaml/README.md#after_script) should not be used in the job definition, because it may be overridden by users. ### Stage @@ -175,7 +181,9 @@ SAST and Dependency Scanning scanners must scan the files in the project directo In order to be consistent with the official Container Scanning for GitLab, scanners must scan the Docker image whose name and tag are given by -`CI_APPLICATION_REPOSITORY` and `CI_APPLICATION_TAG`, respectively. +`CI_APPLICATION_REPOSITORY` and `CI_APPLICATION_TAG`, respectively. If the `DOCKER_IMAGE` +variable is provided, then the `CI_APPLICATION_REPOSITORY` and `CI_APPLICATION_TAG` variables +are ignored, and the image specified in the `DOCKER_IMAGE` variable is scanned instead. If not provided, `CI_APPLICATION_REPOSITORY` should default to `$CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG`, which is a combination of predefined CI variables. @@ -248,6 +256,11 @@ It is recommended to use the `debug` level for verbose logging that could be useful when debugging. The default value for `SECURE_LOG_LEVEL` should be set to `info`. +When executing command lines, scanners should use the `debug` level to log the command line and its output. +For instance, the [bundler-audit](https://gitlab.com/gitlab-org/security-products/analyzers/bundler-audit) scanner +uses the `debug` level to log the command line `bundle audit check --quiet`, +and what `bundle audit` writes to the standard output. + #### common logutil package If you are using [go](https://golang.org/) and @@ -278,8 +291,7 @@ You can find the schemas for these scanners here: ### Version -This field specifies the version of the report schema you are using. Please reference individual scanner -pages for the specific versions to use. +This field specifies the version of the [Security Report Schemas](https://gitlab.com/gitlab-org/security-products/security-report-schemas) you are using. Please refer to the [releases](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/releases) of the schemas for the specific versions to use. ### Vulnerabilities @@ -287,7 +299,7 @@ The `vulnerabilities` field of the report is an array of vulnerability objects. #### ID -The `id` field is the unique identifier of the vulnerability. +The `id` field is the unique identifier of the vulnerability. It is used to reference a fixed vulnerability from a [remediation objects](#remediations). We recommend that you generate a UUID and use it as the `id` field's value. @@ -532,7 +544,7 @@ of the available SAST Analyzers and what data is currently available. The `remediations` field of the report is an array of remediation objects. Each remediation describes a patch that can be applied to -[automatically fix](../../user/application_security/#solutions-for-vulnerabilities-auto-remediation) +[automatically fix](../../user/application_security/#automatic-remediation-for-vulnerabilities) a set of vulnerabilities. Here is an example of a report that contains remediations. diff --git a/doc/development/integrations/secure_partner_integration.md b/doc/development/integrations/secure_partner_integration.md index 19fd86f4bf6..52d10f0bd3c 100644 --- a/doc/development/integrations/secure_partner_integration.md +++ b/doc/development/integrations/secure_partner_integration.md @@ -1,3 +1,9 @@ +--- +stage: Secure +group: Static Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Secure Partner Integration - Onboarding Process If you want to integrate your product with the [Secure Stage](https://about.gitlab.com/direction/secure/), @@ -95,7 +101,7 @@ and complete an integration with the Secure stage. - Users can interact with the findings from your artifact within their workflow. They can dismiss the findings or accept them and create a backlog issue. - To automatically create issues without user interaction, use the [issue API](../../api/issues.md). This will be replaced by [Standalone Vulnerabilities](https://gitlab.com/groups/gitlab-org/-/epics/634) in the future. 1. Optional: Provide auto-remediation steps: - - If you specified `remediations` in your artifact, it is proposed through our [auto-remediation](../../user/application_security/index.md#solutions-for-vulnerabilities-auto-remediation) + - If you specified `remediations` in your artifact, it is proposed through our [auto-remediation](../../user/application_security/index.md#automatic-remediation-for-vulnerabilities) interface. 1. Demo the integration to GitLab: - After you have tested and are ready to demo your integration please diff --git a/doc/development/interacting_components.md b/doc/development/interacting_components.md index 697c64986b1..87bbe30d9fd 100644 --- a/doc/development/interacting_components.md +++ b/doc/development/interacting_components.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Developing against interacting components or features It's not uncommon that a single code change can reflect and interact with multiple parts of GitLab diff --git a/doc/development/internal_users.md b/doc/development/internal_users.md new file mode 100644 index 00000000000..9f1428dce80 --- /dev/null +++ b/doc/development/internal_users.md @@ -0,0 +1,44 @@ +--- +description: "Internal users documentation." +type: concepts, reference, dev +stage: none +group: Development +info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +--- + +# Internal users + +GitLab uses internal users (sometimes referred to as "bots") to perform +actions or functions that cannot be attributed to a regular user. + +These users are created programatically throughout the codebase itself when +necessary, and do not count towards a license limit. + +They are used when a traditional user account would not be applicable, for +example when generating alerts or automatic review feedback. + +Technically, an internal user is a type of user, but they have reduced access +and a very specific purpose. They cannot be used for regular user actions, +such as authentication or API requests. + +They have email addresses and names which can be attributed to any actions +they perform. + +For example, when we [migrated](https://gitlab.com/gitlab-org/gitlab/-/issues/216120) +GitLab Snippets to [Versioned Snippets](../user/snippets.md#versioned-snippets) +in GitLab 13.0, we used an internal user to attribute the authorship of +snippets to itself when a snippet's author wasn't available for creating +repository commits, such as when the user has been disabled, so the Migration +Bot was used instead. + +For this bot: + +- The name was set to `GitLab Migration Bot`. +- The email was set to `noreply+gitlab-migration-bot@{instance host}`. + +Other examples of internal users: + +- [Alert Bot](../operations/metrics/alerts.md#trigger-actions-from-alerts) +- [Ghost User](../user/profile/account/delete_account.md#associated-records) +- [Support Bot](../user/project/service_desk.md#support-bot-user) +- Visual Review Bot diff --git a/doc/development/iterating_tables_in_batches.md b/doc/development/iterating_tables_in_batches.md index 56cbb3a0e9e..062b755de38 100644 --- a/doc/development/iterating_tables_in_batches.md +++ b/doc/development/iterating_tables_in_batches.md @@ -1,3 +1,9 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Iterating Tables In Batches Rails provides a method called `in_batches` that can be used to iterate over diff --git a/doc/development/lfs.md b/doc/development/lfs.md index 3ba81e6a140..64bc709ff9a 100644 --- a/doc/development/lfs.md +++ b/doc/development/lfs.md @@ -1,3 +1,9 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Git LFS ## Deep Dive diff --git a/doc/development/licensed_feature_availability.md b/doc/development/licensed_feature_availability.md index 4350c7fb4d4..f83a57fe1ca 100644 --- a/doc/development/licensed_feature_availability.md +++ b/doc/development/licensed_feature_availability.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Licensed feature availability **(STARTER)** As of GitLab 9.4, we've been supporting a simplified version of licensed diff --git a/doc/development/licensing.md b/doc/development/licensing.md index 8cda3d5f361..d1808822ba6 100644 --- a/doc/development/licensing.md +++ b/doc/development/licensing.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # GitLab Licensing and Compatibility [GitLab Community Edition](https://gitlab.com/gitlab-org/gitlab-foss/) (CE) is licensed [under the terms of the MIT License](https://gitlab.com/gitlab-org/gitlab-foss/blob/master/LICENSE). [GitLab Enterprise Edition](https://gitlab.com/gitlab-org/gitlab/) (EE) is licensed under "[The GitLab Enterprise Edition (EE) license](https://gitlab.com/gitlab-org/gitlab/blob/master/LICENSE)" wherein there are more restrictions. @@ -12,7 +18,8 @@ Some gems may not include their license information in their `gemspec` file, and ### License Finder commands -> Note: License Finder currently uses GitLab misused terms of `whitelist` and `blacklist`. As a result, the commands below reference those terms. We've created an [issue on their project](https://github.com/pivotal/LicenseFinder/issues/745) to propose that they rename their commands. +NOTE: **Note:** +License Finder currently uses GitLab misused terms of `whitelist` and `blacklist`. As a result, the commands below reference those terms. We've created an [issue on their project](https://github.com/pivotal/LicenseFinder/issues/745) to propose that they rename their commands. There are a few basic commands License Finder provides that you'll need in order to manage license detection. diff --git a/doc/development/mass_insert.md b/doc/development/mass_insert.md index c19850ca67e..dfffab564bc 100644 --- a/doc/development/mass_insert.md +++ b/doc/development/mass_insert.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Mass inserting Rails models Setting the environment variable [`MASS_INSERT=1`](rake_tasks.md#environment-variables) diff --git a/doc/development/merge_request_performance_guidelines.md b/doc/development/merge_request_performance_guidelines.md index 2f084937cc9..a5d9a653472 100644 --- a/doc/development/merge_request_performance_guidelines.md +++ b/doc/development/merge_request_performance_guidelines.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Merge Request Performance Guidelines Each new introduced merge request **should be performant by default**. @@ -119,10 +125,10 @@ read this section on [how to prepare the merge request for a database review](da ## Query Counts -**Summary:** a merge request **should not** increase the number of executed SQL +**Summary:** a merge request **should not** increase the total number of executed SQL queries unless absolutely necessary. -The number of queries executed by the code modified or added by a merge request +The total number of queries executed by the code modified or added by a merge request must not increase unless absolutely necessary. When building features it's entirely possible you will need some extra queries, but you should try to keep this at a minimum. @@ -141,7 +147,7 @@ end This will end up running one query for every object to update. This code can easily overload a database given enough rows to update or many instances of this code running in parallel. This particular problem is known as the -["N+1 query problem"](https://guides.rubyonrails.org/active_record_querying.html#eager-loading-associations). You can write a test with [QueryRecoder](query_recorder.md) to detect this and prevent regressions. +["N+1 query problem"](https://guides.rubyonrails.org/active_record_querying.html#eager-loading-associations). You can write a test with [QueryRecorder](query_recorder.md) to detect this and prevent regressions. In this particular case the workaround is fairly easy: @@ -152,6 +158,63 @@ objects_to_update.update_all(some_field: some_value) This uses ActiveRecord's `update_all` method to update all rows in a single query. This in turn makes it much harder for this code to overload a database. +## Cached Queries + +**Summary:** a merge request **should not** execute duplicated cached queries. + +Rails provides an [SQL Query Cache](cached_queries.md#cached-queries-guidelines), +used to cache the results of database queries for the duration of the request. + +See [why cached queries are considered bad](cached_queries.md#why-cached-queries-are-considered-bad) and +[how to detect them](cached_queries.md#how-to-detect). + +The code introduced by a merge request, should not execute multiple duplicated cached queries. + +The total number of the queries (including cached ones) executed by the code modified or added by a merge request +should not increase unless absolutely necessary. +The number of executed queries (including cached queries) should not depend on +collection size. +You can write a test by passing the `skip_cached` variable to [QueryRecorder](query_recorder.md) to detect this and prevent regressions. + +As an example, say you have a CI pipeline. All pipeline builds belong to the same pipeline, +thus they also belong to the same project (`pipeline.project`): + +```ruby +pipeline_project = pipeline.project +# Project Load (0.6ms) SELECT "projects".* FROM "projects" WHERE "projects"."id" = $1 LIMIT $2 +build = pipeline.builds.first + +build.project == pipeline_project +# CACHE Project Load (0.0ms) SELECT "projects".* FROM "projects" WHERE "projects"."id" = $1 LIMIT $2 +# => true +``` + +When we call `build.project`, it will not hit the database, it will use the cached result, but it will re-instantiate +same pipeline project object. It turns out that associated objects do not point to the same in-memory object. + +If we try to serialize each build: + +```ruby +pipeline.builds.each do |build| + build.to_json(only: [:name], include: [project: { only: [:name]}]) +end +``` + +It will re-instantiate project object for each build, instead of using the same in-memory object. + +In this particular case the workaround is fairly easy: + +```ruby +pipeline.builds.each do |build| + build.project = pipeline.project + build.to_json(only: [:name], include: [project: { only: [:name]}]) +end +``` + +We can assign `pipeline.project` to each `build.project`, since we know it should point to the same project. +This will allow us that each build point to the same in-memory project, +avoiding the cached SQL query and re-instantiation of the project object for each build. + ## Executing Queries in Loops **Summary:** SQL queries **must not** be executed in a loop unless absolutely @@ -464,7 +527,7 @@ end The usage of shared temporary storage is required if your intent is to persistent file for a disk-based storage, and not Object Storage. -[Workhorse direct_upload](./uploads.md#direct-upload) when accepting file +[Workhorse direct_upload](uploads.md#direct-upload) when accepting file can write it to shared storage, and later GitLab Rails can perform a move operation. The move operation on the same destination is instantaneous. The system instead of performing `copy` operation just re-attaches file into a new place. @@ -488,7 +551,7 @@ that implements a seamless support for Shared and Object Storage-based persisten #### Data access Each feature that accepts data uploads or allows to download them needs to use -[Workhorse direct_upload](./uploads.md#direct-upload). It means that uploads needs to be +[Workhorse direct_upload](uploads.md#direct-upload). It means that uploads needs to be saved directly to Object Storage by Workhorse, and all downloads needs to be served by Workhorse. @@ -500,5 +563,5 @@ can time out, which is especially problematic for slow clients. If clients take to upload/download the processing slot might be killed due to request processing timeout (usually between 30s-60s). -For the above reasons it is required that [Workhorse direct_upload](./uploads.md#direct-upload) is implemented +For the above reasons it is required that [Workhorse direct_upload](uploads.md#direct-upload) is implemented for all file uploads and downloads. diff --git a/doc/development/migration_style_guide.md b/doc/development/migration_style_guide.md index 71191d1d871..84679a78545 100644 --- a/doc/development/migration_style_guide.md +++ b/doc/development/migration_style_guide.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Migration Style Guide When writing migrations for GitLab, you have to take into account that @@ -376,8 +382,7 @@ Example changes: - `change_column_default` - `create_table` / `drop_table` -NOTE: **Note:** -`with_lock_retries` method **cannot** be used within the `change` method, you must manually define the `up` and `down` methods to make the migration reversible. +The `with_lock_retries` method **cannot** be used within the `change` method, you must manually define the `up` and `down` methods to make the migration reversible. ### How the helper method works @@ -437,7 +442,6 @@ the `with_multiple_threads` block, instead of re-using the global connection pool. This ensures each thread has its own connection object, and won't time out when trying to obtain one. -NOTE: **Note:** PostgreSQL has a maximum amount of connections that it allows. This limit can vary from installation to installation. As a result, it's recommended you do not use more than 32 threads in a single migration. Usually, 4-8 threads @@ -474,6 +478,12 @@ For a small table (such as an empty one or one with less than `1,000` records), it is recommended to use `remove_index` in a single-transaction migration, combining it with other operations that don't require `disable_ddl_transaction!`. +### Disabling an index + +There are certain situations in which you might want to disable an index before removing it. +See the [maintenance operations guide](database/maintenance_operations.md#disabling-an-index) +for more details. + ## Adding indexes Before adding an index, consider if this one is necessary. There are situations in which an index @@ -612,7 +622,6 @@ Before PostgreSQL 11, adding a column with a default was problematic as it would have caused a full table rewrite. The corresponding helper `add_column_with_default` has been deprecated and will be removed in a later release. -NOTE: **Note:** If a backport adding a column with a default value is needed for %12.9 or earlier versions, it should use `add_column_with_default` helper. If a [large table](https://gitlab.com/gitlab-org/gitlab/-/blob/master/rubocop/rubocop-migrations.yml#L3) is involved, backporting to %12.9 is contraindicated. @@ -958,7 +967,6 @@ in a previous migration. ### Example: Add a column `my_column` to the users table -NOTE: **Note:** It is important not to leave out the `User.reset_column_information` command, in order to ensure that the old schema is dropped from the cache and ActiveRecord loads the updated schema information. ```ruby diff --git a/doc/development/module_with_instance_variables.md b/doc/development/module_with_instance_variables.md index 2229c99c99b..80e926f800c 100644 --- a/doc/development/module_with_instance_variables.md +++ b/doc/development/module_with_instance_variables.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Modules with instance variables could be considered harmful ## Background diff --git a/doc/development/multi_version_compatibility.md b/doc/development/multi_version_compatibility.md index 714be296b40..2501f8a169f 100644 --- a/doc/development/multi_version_compatibility.md +++ b/doc/development/multi_version_compatibility.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Compatibility with multiple versions of the application running at the same time When adding or changing features, we must be aware that there may be multiple versions of the application running diff --git a/doc/development/namespaces_storage_statistics.md b/doc/development/namespaces_storage_statistics.md index 5207276ba73..b4a7c8c3132 100644 --- a/doc/development/namespaces_storage_statistics.md +++ b/doc/development/namespaces_storage_statistics.md @@ -1,17 +1,23 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Database case study: Namespaces storage statistics -## Introduction +## Introduction On [Storage and limits management for groups](https://gitlab.com/groups/gitlab-org/-/epics/886), we want to facilitate a method for easily viewing the amount of storage consumed by a group, and allow easy management. -## Proposal +## Proposal 1. Create a new ActiveRecord model to hold the namespaces' statistics in an aggregated form (only for root namespaces). 1. Refresh the statistics in this model every time a project belonging to this namespace is changed. -## Problem +## Problem In GitLab, we update the project storage statistics through a [callback](https://gitlab.com/gitlab-org/gitlab/blob/4ab54c2233e91f60a80e5b6fa2181e6899fdcc3e/app/models/project.rb#L97) @@ -36,7 +42,7 @@ alternative method. ## Attempts -### Attempt A: PostgreSQL materialized view +### Attempt A: PostgreSQL materialized view Model can be updated through a refresh strategy based on a project routes SQL and a [materialized view](https://www.postgresql.org/docs/11/rules-materializedviews.html): @@ -65,7 +71,7 @@ While this implied a single query update (and probably a fast one), it has some - Materialized views syntax varies from PostgreSQL and MySQL. While this feature was worked on, MySQL was still supported by GitLab. - Rails does not have native support for materialized views. We'd need to use a specialized gem to take care of the management of the database views, which implies additional work. -### Attempt B: An update through a CTE +### Attempt B: An update through a CTE Similar to Attempt A: Model update done through a refresh strategy with a [Common Table Expression](https://www.postgresql.org/docs/9.1/queries-with.html) @@ -134,7 +140,7 @@ Even though this approach would make aggregating much easier, it has some major - We'd have to migrate **all namespaces** by adding and filling a new column. Because of the size of the table, dealing with time/cost will not be great. The background migration will take approximately `153h`, see <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/29772>. - Background migration has to be shipped one release before, delaying the functionality by another milestone. -### Attempt E (final): Update the namespace storage statistics in async way +### Attempt E (final): Update the namespace storage statistics in async way This approach consists of keep using the incremental statistics updates we currently already have, but we refresh them through Sidekiq jobs and in different transactions: @@ -164,7 +170,7 @@ The only downside of this approach is that namespaces' statistics are updated up which means there's a time window in which the statistics are inaccurate. Because we're still not [enforcing storage limits](https://gitlab.com/gitlab-org/gitlab/-/issues/17664), this is not a major problem. -## Conclusion +## Conclusion Updating the storage statistics asynchronously, was the less problematic and performant approach of aggregating the root namespaces. diff --git a/doc/development/new_fe_guide/dependencies.md b/doc/development/new_fe_guide/dependencies.md index afdf6e27b37..fad004f9df5 100644 --- a/doc/development/new_fe_guide/dependencies.md +++ b/doc/development/new_fe_guide/dependencies.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Dependencies ## Adding Dependencies diff --git a/doc/development/new_fe_guide/development/accessibility.md b/doc/development/new_fe_guide/development/accessibility.md index 9c63ccad6e1..1189dd1137b 100644 --- a/doc/development/new_fe_guide/development/accessibility.md +++ b/doc/development/new_fe_guide/development/accessibility.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Accessibility Using semantic HTML plays a key role when it comes to accessibility. @@ -6,7 +12,7 @@ Using semantic HTML plays a key role when it comes to accessibility. WAI-ARIA (the Accessible Rich Internet Applications specification) defines a way to make Web content and Web applications more accessible to people with disabilities. -> Note: It is [recommended](https://www.w3.org/TR/using-aria/#notes2) to use semantic elements as the primary method to achieve accessibility rather than adding aria attributes. Adding aria attributes should be seen as a secondary method for creating accessible elements. +The W3C recommends [using semantic elements](https://www.w3.org/TR/using-aria/#notes2) as the primary method to achieve accessibility rather than adding aria attributes. Adding aria attributes should be seen as a secondary method for creating accessible elements. ### Role diff --git a/doc/development/new_fe_guide/development/components.md b/doc/development/new_fe_guide/development/components.md index b7233f5d7c0..1597d4e62e7 100644 --- a/doc/development/new_fe_guide/development/components.md +++ b/doc/development/new_fe_guide/development/components.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Components ## Graphs diff --git a/doc/development/new_fe_guide/development/index.md b/doc/development/new_fe_guide/development/index.md index 119dbc58012..52435ca719d 100644 --- a/doc/development/new_fe_guide/development/index.md +++ b/doc/development/new_fe_guide/development/index.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Development ## [Components](components.md) diff --git a/doc/development/new_fe_guide/development/performance.md b/doc/development/new_fe_guide/development/performance.md index ad6fdc0b85c..44f5bccde95 100644 --- a/doc/development/new_fe_guide/development/performance.md +++ b/doc/development/new_fe_guide/development/performance.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Performance ## Monitoring diff --git a/doc/development/new_fe_guide/index.md b/doc/development/new_fe_guide/index.md index 9e9c367807f..c9b655c2274 100644 --- a/doc/development/new_fe_guide/index.md +++ b/doc/development/new_fe_guide/index.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Frontend Development Guidelines This guide contains all the information to successfully contribute to GitLab's frontend. diff --git a/doc/development/new_fe_guide/modules/dirty_submit.md b/doc/development/new_fe_guide/modules/dirty_submit.md index dd336ad3a90..17cdab3f240 100644 --- a/doc/development/new_fe_guide/modules/dirty_submit.md +++ b/doc/development/new_fe_guide/modules/dirty_submit.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Dirty Submit > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/21115) in GitLab 11.3. diff --git a/doc/development/new_fe_guide/modules/index.md b/doc/development/new_fe_guide/modules/index.md index a7820442df0..18c5b05432c 100644 --- a/doc/development/new_fe_guide/modules/index.md +++ b/doc/development/new_fe_guide/modules/index.md @@ -1,5 +1,15 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Modules - [DirtySubmit](dirty_submit.md) Disable form submits until there are unsaved changes. + +- [Merge Request widget extensions](widget_extensions.md) + + Easily add extensions into the merge request widget diff --git a/doc/development/new_fe_guide/modules/widget_extensions.md b/doc/development/new_fe_guide/modules/widget_extensions.md new file mode 100644 index 00000000000..3844fedb126 --- /dev/null +++ b/doc/development/new_fe_guide/modules/widget_extensions.md @@ -0,0 +1,56 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + +# Merge request widget extensions + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/44616) in GitLab 13.6. + +## Summary + +Extensions in the merge request widget allow for others team to quickly and easily add new features +into the widget that will match the existing design and interaction as other extensions. + +## Usage + +To use extensions you need to first create a new extension object that will be used to fetch the +data that will be rendered in the extension. See the example file in +app/assets/javascripts/vue_merge_request_widget/extensions/issues.js for a working example. + +The basic object structure is as below: + +```javascript +export default { + name: '', + props: [], + computed: { + summary() {}, + statusIcon() {}, + }, + methods: { + fetchCollapsedData() {}, + fetchFullData() {}, + }, +}; +``` + +Following the same data structure allows each extension to follow the same registering structure +but allows for each extension to manage where it gets its own data from. + +After creating this structure you need to register it. Registering the extension can happen at any +point _after_ the widget has been created. + +To register a extension the following can be done: + +```javascript +// Import the register method +import { registerExtension } from '~/vue_merge_request_widget/components/extensions'; + +// Import the new extension +import issueExtension from '~/vue_merge_request_widget/extensions/issues'; + +// Register the imported extension +registerExtension(issueExtension); +``` diff --git a/doc/development/new_fe_guide/tips.md b/doc/development/new_fe_guide/tips.md index c65266a3f25..d7e9c440335 100644 --- a/doc/development/new_fe_guide/tips.md +++ b/doc/development/new_fe_guide/tips.md @@ -1,3 +1,9 @@ +--- +stage: none +group: Development +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Tips ## Clearing production compiled assets diff --git a/doc/development/newlines_styleguide.md b/doc/development/newlines_styleguide.md index e298ba1d935..f123482fa5a 100644 --- a/doc/development/newlines_styleguide.md +++ b/doc/development/newlines_styleguide.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Newlines style guide This style guide recommends best practices for newlines in Ruby code. diff --git a/doc/development/omnibus.md b/doc/development/omnibus.md index deaf72d2ecf..84c395e2e57 100644 --- a/doc/development/omnibus.md +++ b/doc/development/omnibus.md @@ -1,3 +1,9 @@ +--- +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # What you should know about Omnibus packages Most users install GitLab using our Omnibus packages. As a developer it can be diff --git a/doc/development/ordering_table_columns.md b/doc/development/ordering_table_columns.md index 18788d0b86e..124f82ac2c8 100644 --- a/doc/development/ordering_table_columns.md +++ b/doc/development/ordering_table_columns.md @@ -1,3 +1,9 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Ordering Table Columns in PostgreSQL For GitLab we require that columns of new tables are ordered to use the @@ -45,7 +51,6 @@ In these examples, the `id` and `user_id` columns are packed together, which means we only need 8 bytes to store _both_ of them. This in turn means each row will require 8 bytes less space. -Note: **NOTE:** Since Ruby on Rails 5.1, the default data type for IDs is `bigint`, which uses 8 bytes. We are using `integer` in the examples to showcase a more realistic reordering scenario. diff --git a/doc/development/packages.md b/doc/development/packages.md index 9eae99ff890..de6cac2ce73 100644 --- a/doc/development/packages.md +++ b/doc/development/packages.md @@ -13,12 +13,13 @@ See already supported package types in [Packages documentation](../administratio Since GitLab packages' UI is pretty generic, it is possible to add basic new package system support with solely backend changes. This guide is superficial and does not cover the way the code should be written. However, you can find a good example -by looking at merge requests with Maven and NPM support: +by looking at the following merge requests: - [NPM registry support](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/8673). -- [Conan repository](https://gitlab.com/gitlab-org/gitlab/-/issues/8248). - [Maven repository](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/6607). -- [Instance level endpoint for Maven repository](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/8757) +- [Composer repository for PHP dependencies](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/22415). +- [Terraform modules registry](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/18834). +- [Instance-level endpoint for Maven repository](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/8757). ## General information diff --git a/doc/development/performance.md b/doc/development/performance.md index 3b59393bae6..e20633a05f6 100644 --- a/doc/development/performance.md +++ b/doc/development/performance.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Performance Guidelines This document describes various guidelines to follow to ensure good and @@ -334,15 +340,49 @@ These results can also be placed into a PostgreSQL database by setting the `RSPEC_PROFILING_POSTGRES_URL` variable. This is used to profile the test suite when running in the CI environment. -We store these results also when running CI jobs on the default branch on -`gitlab.com`. Statistics of these profiling data are [available -online](https://gitlab-org.gitlab.io/rspec_profiling_stats/). For example, -you can find which tests take longest to run or which execute the most +We store these results also when running nightly scheduled CI jobs on the +default branch on `gitlab.com`. Statistics of these profiling data are +[available online](https://gitlab-org.gitlab.io/rspec_profiling_stats/). For +example, you can find which tests take longest to run or which execute the most queries. This can be handy for optimizing our tests or identifying performance issues in our code. ## Memory profiling +We can use two approaches, often in combination, to track down memory issues: + +- Leaving the code intact and wrapping a profiler around it. +- Monitor memory usage of the process while disabling/enabling different parts of the code we suspect could be problematic. + +### Using Memory Profiler + +We can use `memory_profiler` for profiling. + +The [`memory_profiler`](https://github.com/SamSaffron/memory_profiler) gem is already present in GitLab's `Gemfile`, +you just need to require it: + +```ruby +require 'sidekiq/testing' + +report = MemoryProfiler.report do + # Code you want to profile +end + +output = File.open('/tmp/profile.txt','w') +report.pretty_print(output) +``` + +The report breaks down 2 key concepts: + +- Retained: long lived memory use and object count retained due to the execution of the code block. +- Allocated: all object allocation and memory allocation during code block. + +As a general rule, **retained** will always be smaller than or equal to allocated. + +The actual RSS cost will always be slightly higher as MRI heaps are not squashed to size and memory fragments. + +### Rbtrace + One of the reasons of the increased memory footprint could be Ruby memory fragmentation. To diagnose it, you can visualize Ruby heap as described in [this post by Aaron Patterson](https://tenderlovemaking.com/2017/09/27/visualizing-your-ruby-heap.html). diff --git a/doc/development/permissions.md b/doc/development/permissions.md index e930345caec..859c838280e 100644 --- a/doc/development/permissions.md +++ b/doc/development/permissions.md @@ -1,3 +1,9 @@ +--- +stage: Manage +group: Access +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # GitLab permissions guide There are multiple types of permissions across GitLab, and when implementing diff --git a/doc/development/pipelines.md b/doc/development/pipelines.md index d12220d8c95..a8ef5701d50 100644 --- a/doc/development/pipelines.md +++ b/doc/development/pipelines.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Pipelines for the GitLab project Pipelines for <https://gitlab.com/gitlab-org/gitlab> and <https://gitlab.com/gitlab-org/gitlab-foss> (as well as the @@ -618,7 +624,7 @@ each pipeline includes default variables defined in Most of the jobs [extend from a few CI definitions](../ci/yaml/README.md#extends) defined in [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/global.gitlab-ci.yml) -that are scoped to a single [configuration parameter](../ci/yaml/README.md#configuration-parameters). +that are scoped to a single [configuration keyword](../ci/yaml/README.md#job-keywords). | Job definitions | Description | |------------------|-------------| diff --git a/doc/development/policies.md b/doc/development/policies.md index 8dfd4763551..b44367f7075 100644 --- a/doc/development/policies.md +++ b/doc/development/policies.md @@ -1,3 +1,9 @@ +--- +stage: Manage +group: Access +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # `DeclarativePolicy` framework The DeclarativePolicy framework is designed to assist in performance of policy checks, and to enable ease of extension for EE. The DSL code in `app/policies` is what `Ability.allowed?` uses to check whether a particular action is allowed on a subject. diff --git a/doc/development/polling.md b/doc/development/polling.md index 47cfc32d934..b06507787b0 100644 --- a/doc/development/polling.md +++ b/doc/development/polling.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Polling with ETag caching Polling for changes (repeatedly asking server if there are any new changes) diff --git a/doc/development/polymorphic_associations.md b/doc/development/polymorphic_associations.md index b6567704d8e..66ea974063a 100644 --- a/doc/development/polymorphic_associations.md +++ b/doc/development/polymorphic_associations.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Polymorphic Associations **Summary:** always use separate tables instead of polymorphic associations. diff --git a/doc/development/post_deployment_migrations.md b/doc/development/post_deployment_migrations.md index 4d523178a21..2550f9547df 100644 --- a/doc/development/post_deployment_migrations.md +++ b/doc/development/post_deployment_migrations.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Post Deployment Migrations Post deployment migrations are regular Rails migrations that can optionally be diff --git a/doc/development/product_analytics/event_dictionary.md b/doc/development/product_analytics/event_dictionary.md index b049db21c30..88cb75fdb83 100644 --- a/doc/development/product_analytics/event_dictionary.md +++ b/doc/development/product_analytics/event_dictionary.md @@ -1,32 +1,5 @@ --- -stage: Growth -group: Product Analytics -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +redirect_to: 'https://about.gitlab.com/handbook/product/product-analytics-guide/' --- -# Event Dictionary - -**Note: We've temporarily moved the Event Dictionary to a [Google Sheet](https://docs.google.com/spreadsheets/d/1VzE8R72Px_Y_LlE3Z05LxUlG_dumWe3vl-HeUo70TPw/edit?usp=sharing)**. The previous Markdown table exceeded 600 rows making it difficult to manage. In the future, our intention is to move this back into our docs using a [YAML file](https://gitlab.com/gitlab-org/gitlab-docs/-/issues/823). - -The event dictionary is a single source of truth for the metrics and events we collect for product usage data. The Event Dictionary lists all the metrics and events we track, why we're tracking them, and where they are tracked. - -This is a living document that is updated any time a new event is planned or implemented. It includes the following information. - -- Section, stage, or group -- Description -- Implementation status -- Availability by plan type -- Code path - -We're currently focusing our Event Dictionary on [Usage Ping](usage_ping.md). In the future, we will also include [Snowplow](snowplow.md). We currently have an initiative across the entire product organization to complete the [Event Dictionary for Usage Ping](https://gitlab.com/groups/gitlab-org/-/epics/4174). - -## Instructions - -1. Open the Event Dictionary and fill in all the **PM to edit** columns highlighted in yellow. -1. Check that all the metrics and events are assigned to the correct section, stage, or group. If a metric is used across many groups, assign it to the stage. If a metric is used across many stages, assign it to the section. If a metric is incorrectly assigned to another section, stage, or group, let the PM know you have reassigned it. If your group has no assigned metrics and events, check that your metrics and events are not incorrectly assigned to another PM. -1. Add descriptions of what your metrics and events are tracking. Work with your Engineering team or the Product Analytics team if you need help understanding this. -1. Add what plans this metric is available on. Work with your Engineering team or the Product Analytics team if you need help understanding this. - -## Planned metrics and events - -For future metrics and events you plan to track, please add them to the Event Dictionary and note the status as `Planned`, `In Progress`, or `Implemented`. Once you have confirmed the metric has been implemented and have confirmed the metric data is in our data warehouse, change the status to **Data Available**. +This document was moved to [another location](https://about.gitlab.com/handbook/product/product-analytics-guide/). diff --git a/doc/development/product_analytics/index.md b/doc/development/product_analytics/index.md index ab76d6f0561..88cb75fdb83 100644 --- a/doc/development/product_analytics/index.md +++ b/doc/development/product_analytics/index.md @@ -1,182 +1,5 @@ --- -stage: Growth -group: Product Analytics -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +redirect_to: 'https://about.gitlab.com/handbook/product/product-analytics-guide/' --- -# Product Analytics Guide - -At GitLab, we collect product usage data for the purpose of helping us build a better product. Data helps GitLab understand which parts of the product need improvement and which features we should build next. Product usage data also helps our team better understand the reasons why people use GitLab. With this knowledge we are able to make better product decisions. - -We encourage users to enable tracking, and we embrace full transparency with our tracking approach so it can be easily understood and trusted. - -By enabling tracking, users can: - -- Contribute back to the wider community. -- Help GitLab improve on the product. - -## Our tracking tools - -We use three methods to gather product usage data: - -- [Snowplow](#snowplow) -- [Usage Ping](#usage-ping) -- [Database import](#database-import) - -### Snowplow - -Snowplow is an enterprise-grade marketing and product analytics platform which helps track the way -users engage with our website and application. - -Snowplow consists of two components: - -- [Snowplow JS](https://github.com/snowplow/snowplow/wiki/javascript-tracker) tracks client-side - events. -- [Snowplow Ruby](https://github.com/snowplow/snowplow/wiki/ruby-tracker) tracks server-side events. - -For more details, read the [Snowplow](snowplow.md) guide. - -### Usage Ping - -Usage Ping is a method for GitLab Inc to collect usage data on a GitLab instance. Usage Ping is primarily composed of row counts for different tables in the instance’s database. By comparing these counts month over month (or week over week), we can get a rough sense for how an instance is using the different features within the product. This high-level data is used to help our product, support, and sales teams. - -For more details, read the [Usage Ping](usage_ping.md) guide. - -### Database import - -Database imports are full imports of data into GitLab's data warehouse. For GitLab.com, the PostgreSQL database is loaded into Snowflake data warehouse every 6 hours. For more details, see the [data team handbook](https://about.gitlab.com/handbook/business-ops/data-team/platform/#extract-and-load). - -## What data can be tracked - -Our different tracking tools allows us to track different types of events. The event types and examples of what data can be tracked are outlined below. - -The availability of event types and their tracking tools varies by segment. For example, on Self-Managed Users, we only have reporting using Database records via Usage Ping. - -| Event Types | SaaS Instance | SaaS Plan | SaaS Group | SaaS Session | SaaS User | SM Instance | SM Plan | SM Group | SM Session | SM User | -|----------------------------------------|---------------|-----------|------------|--------------|-----------|-------------|---------|----------|------------|---------| -| Snowplow (JS Pageview events) | ✅ | 📅 | 📅 | ✅ | 📅 | 📅 | 📅 | 📅 | 📅 | 📅 | -| Snowplow (JS UI events) | ✅ | 📅 | 📅 | ✅ | 📅 | 📅 | 📅 | 📅 | 📅 | 📅 | -| Snowplow (Ruby Pageview events) | ✅ | 📅 | 📅 | ✅ | 📅 | 📅 | 📅 | 📅 | 📅 | 📅 | -| Snowplow (Ruby CRUD / API events) | ✅ | 📅 | 📅 | ✅ | 📅 | 📅 | 📅 | 📅 | 📅 | 📅 | -| Usage Ping (Redis UI counters) | 🔄 | 🔄 | 🔄 | ✖️ | 🔄 | 🔄 | 🔄 | 🔄 | ✖️ | 🔄 | -| Usage Ping (Redis Pageview counters) | 🔄 | 🔄 | 🔄 | ✖️ | 🔄 | 🔄 | 🔄 | 🔄 | ✖️ | 🔄 | -| Usage Ping (Redis CRUD / API counters) | 🔄 | 🔄 | 🔄 | ✖️ | 🔄 | 🔄 | 🔄 | 🔄 | ✖️ | 🔄 | -| Usage Ping (Database counters) | ✅ | 🔄 | 📅 | ✖️ | ✅ | ✅ | ✅ | ✅ | ✖️ | ✅ | -| Usage Ping (Instance settings) | ✅ | 🔄 | 📅 | ✖️ | ✅ | ✅ | ✅ | ✅ | ✖️ | ✅ | -| Usage Ping (Integration settings) | ✅ | 🔄 | 📅 | ✖️ | ✅ | ✅ | ✅ | ✅ | ✖️ | ✅ | -| Database import (Database records) | ✅ | ✅ | ✅ | ✖️ | ✅ | ✖️ | ✖️ | ✖️ | ✖️ | ✖️ | - -[Source file](https://docs.google.com/spreadsheets/d/1e8Afo41Ar8x3JxAXJF3nL83UxVZ3hPIyXdt243VnNuE/edit?usp=sharing) - -**Legend** - -✅ Available, 🔄 In Progress, 📅 Planned, ✖️ Not Possible - -SaaS = GitLab.com. SM = Self-Managed instance - -### Pageview events - -- Number of sessions that visited the /dashboard/groups page - -### UI events - -- Number of sessions that clicked on a button or link -- Number of sessions that closed a modal - -UI events are any interface-driven actions from the browser including click data. - -### CRUD or API events - -- Number of Git pushes -- Number of GraphQL queries -- Number of requests to a Rails action or controller - -These are backend events that include the creation, read, update, deletion of records, and other events that might be triggered from layers other than those available in the interface. - -### Database records - -These are raw database records which can be explored using business intelligence tools like Sisense. The full list of available tables can be found in [structure.sql](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/structure.sql). - -### Instance settings - -These are settings of your instance such as the instance's Git version and if certain features are enabled such as `container_registry_enabled`. - -### Integration settings - -These are integrations your GitLab instance interacts with such as an [external storage provider](../../administration/static_objects_external_storage.md) or an [external container registry](../../administration/packages/container_registry.md#use-an-external-container-registry-with-gitlab-as-an-auth-endpoint). These services must be able to send data back into a GitLab instance for data to be tracked. - -## Reporting level - -Our reporting levels of aggregate or individual reporting varies by segment. For example, on Self-Managed Users, we can report at an aggregate user level using Usage Ping but not on an Individual user level. - -| Aggregated Reporting | SaaS Instance | SaaS Plan | SaaS Group | SaaS Session | SaaS User | SM Instance | SM Plan | SM Group | SM Session | SM User | -|----------------------|---------------|-----------|------------|--------------|-----------|-------------|---------|----------|------------|---------| -| Snowplow | ✅ | 📅 | 📅 | ✅ | 📅 | ✅ | 📅 | 📅 | ✅ | 📅 | -| Usage Ping | ✅ | 🔄 | 📅 | 📅 | ✅ | ✅ | ✅ | ✅ | 📅 | ✅ | -| Database import | ✅ | ✅ | ✅ | ✖️ | ✅ | ✖️ | ✖️ | ✖️ | ✖️ | ✖️ | - -| Identifiable Reporting | SaaS Instance | SaaS Plan | SaaS Group | SaaS Session | SaaS User | SM Instance | SM Plan | SM Group | SM Session | SM User | -|------------------------|---------------|-----------|------------|--------------|-----------|-------------|---------|----------|------------|---------| -| Snowplow | ✅ | 📅 | 📅 | ✅ | 📅 | ✖️ | ✖️ | ✖️ | ✖️ | ✖️ | -| Usage Ping | ✅ | 🔄 | 📅 | ✖️ | ✖️ | ✅ | ✅ | ✖️ | ✖️ | ✖️ | -| Database import | ✅ | ✅ | ✅ | ✖️ | ✅ | ✖️ | ✖️ | ✖️ | ✖️ | ✖️ | - -**Legend** - -✅ Available, 🔄 In Progress, 📅 Planned, ✖️ Not Possible - -SaaS = GitLab.com. SM = Self-Managed instance - -## Reporting time period - -Our reporting time periods varies by segment. For example, on Self-Managed Users, we can report all time counts and 28 day counts in Usage Ping. - -| Reporting Time Period | All Time | 28 Days | 7 Days | Daily | -|-----------------------|----------|---------|--------|-------| -| Snowplow | ✅ | ✅ | ✅ | ✅ | -| Usage Ping | ✅ | ✅ | 📅 | ✖️ | -| Database import | ✅ | ✅ | ✅ | ✅ | - -**Legend** - -✅ Available, 🔄 In Progress, 📅 Planned, ✖️ Not Possible - -## Systems overview - -The systems overview is a simplified diagram showing the interactions between GitLab Inc and self-managed instances. - - - -[Source file](https://app.diagrams.net/#G13DVpN-XnhWGz9tqReIj8pp1UE4ehk_EC) - -### GitLab Inc - -For Product Analytics purposes, GitLab Inc has three major components: - -1. [Data Infrastructure](https://about.gitlab.com/handbook/business-ops/data-team/platform/infrastructure/): This contains everything managed by our data team including Sisense Dashboards for visualization, Snowflake for Data Warehousing, incoming data sources such as PostgreSQL Pipeline and S3 Bucket, and lastly our data collectors [GitLab.com's Snowplow Collector](https://gitlab.com/gitlab-com/gl-infra/readiness/-/tree/master/library/snowplow/) and GitLab's Versions Application. -1. GitLab.com: This is the production GitLab application which is made up of a Client and Server. On the Client or browser side, a Snowplow JS Tracker (Frontend) is used to track client-side events. On the Server or application side, a Snowplow Ruby Tracker (Backend) is used to track server-side events. The server also contains Usage Ping which leverages a PostgreSQL database and a Redis in-memory data store to report on usage data. Lastly, the server also contains System Logs which are generated from running the GitLab application. -1. [Monitoring infrastructure](https://about.gitlab.com/handbook/engineering/monitoring/): This is the infrastructure used to ensure GitLab.com is operating smoothly. System Logs are sent from GitLab.com to our monitoring infrastructure and collected by a FluentD collector. From FluentD, logs are either sent to long term Google Cloud Services cold storage via Stackdriver, or, they are sent to our Elastic Cluster via Cloud Pub/Sub which can be explored in real-time using Kibana. - -### Self-managed - -For Product Analytics purposes, self-managed instances have two major components: - -1. Data infrastructure: Having a data infrastructure setup is optional on self-managed instances. If you'd like to collect Snowplow tracking events for your self-managed instance, you can setup your own self-managed Snowplow collector and configure your Snowplow events to point to your own collector. -1. GitLab: A self-managed GitLab instance contains all of the same components as GitLab.com mentioned above. - -### Differences between GitLab Inc and Self-managed - -As shown by the orange lines, on GitLab.com Snowplow JS, Snowplow Ruby, Usage Ping, and PostgreSQL database imports all flow into GitLab Inc's data infrastructure. However, on self-managed, only Usage Ping flows into GitLab Inc's data infrastructure. - -As shown by the green lines, on GitLab.com system logs flow into GitLab Inc's monitoring infrastructure. On self-managed, there are no logs sent to GitLab Inc's monitoring infrastructure. - -Note (1): Snowplow JS and Snowplow Ruby are available on self-managed, however, the Snowplow Collector endpoint is set to a self-managed Snowplow Collector which GitLab Inc does not have access to. - -## Additional information - -More useful links: - -- [Product Analytics Direction](https://about.gitlab.com/direction/product-analytics/) -- [Data Analysis Process](https://about.gitlab.com/handbook/business-ops/data-team/#data-analysis-process/) -- [Data for Product Managers](https://about.gitlab.com/handbook/business-ops/data-team/programs/data-for-product-managers/) -- [Data Infrastructure](https://about.gitlab.com/handbook/business-ops/data-team/platform/infrastructure/) +This document was moved to [another location](https://about.gitlab.com/handbook/product/product-analytics-guide/). diff --git a/doc/development/product_analytics/snowplow.md b/doc/development/product_analytics/snowplow.md index 21d92566ffd..c5f48994d5c 100644 --- a/doc/development/product_analytics/snowplow.md +++ b/doc/development/product_analytics/snowplow.md @@ -10,7 +10,7 @@ This guide provides an overview of how Snowplow works, and implementation detail For more information about Product Analytics, see: -- [Product Analytics Guide](index.md) +- [Product Analytics Guide](https://about.gitlab.com/handbook/product/product-analytics-guide/) - [Usage Ping Guide](usage_ping.md) More useful links: @@ -52,7 +52,7 @@ Tracking can be enabled at: - The instance level, which enables tracking on both the frontend and backend layers. - User level, though user tracking can be disabled on a per-user basis. GitLab tracking respects the [Do Not Track](https://www.eff.org/issues/do-not-track) standard, so any user who has enabled the Do Not Track option in their browser is not tracked at a user level. -We utilize Snowplow for the majority of our tracking strategy and it is enabled on GitLab.com. On a self-managed instance, Snowplow can be enabled by navigating to: +We use Snowplow for the majority of our tracking strategy and it is enabled on GitLab.com. On a self-managed instance, Snowplow can be enabled by navigating to: - **Admin Area > Settings > General** in the UI. - `admin/application_settings/integrations` in your browser. @@ -112,7 +112,7 @@ The current method provides several attributes that are sent on each click event ## Implementing Snowplow JS (Frontend) tracking -GitLab provides `Tracking`, an interface that wraps the [Snowplow JavaScript Tracker](https://github.com/snowplow/snowplow/wiki/javascript-tracker) for tracking custom events. There are a few ways to utilize tracking, but each generally requires at minimum, a `category` and an `action`. Additional data can be provided that adheres to our [Structured event taxonomy](#structured-event-taxonomy). +GitLab provides `Tracking`, an interface that wraps the [Snowplow JavaScript Tracker](https://github.com/snowplow/snowplow/wiki/javascript-tracker) for tracking custom events. There are a few ways to use tracking, but each generally requires at minimum, a `category` and an `action`. Additional data can be provided that adheres to our [Structured event taxonomy](#structured-event-taxonomy). | field | type | default value | description | |:-----------|:-------|:---------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| @@ -294,7 +294,7 @@ Custom event tracking and instrumentation can be added by directly calling the ` | `action` | string | 'generic' | The action being taken, which can be anything from a controller action like `create` to something like an Active Record callback. | | `data` | object | {} | Additional data such as `label`, `property`, `value`, and `context` as described in [Structured event taxonomy](#structured-event-taxonomy). These are set as empty strings if you don't provide them. | -Tracking can be viewed as either tracking user behavior, or can be utilized for instrumentation to monitor and visualize performance over time in an area or aspect of code. +Tracking can be viewed as either tracking user behavior, or can be used for instrumentation to monitor and visualize performance over time in an area or aspect of code. For example: diff --git a/doc/development/product_analytics/usage_ping.md b/doc/development/product_analytics/usage_ping.md index d482af77d8a..fa785d934cb 100644 --- a/doc/development/product_analytics/usage_ping.md +++ b/doc/development/product_analytics/usage_ping.md @@ -15,7 +15,7 @@ This guide describes Usage Ping's purpose and how it's implemented. For more information about Product Analytics, see: -- [Product Analytics Guide](index.md) +- [Product Analytics Guide](https://about.gitlab.com/handbook/product/product-analytics-guide/) - [Snowplow Guide](snowplow.md) More useful links: @@ -270,7 +270,7 @@ Implemented using Redis methods [PFADD](https://redis.io/commands/pfadd) and [PF ##### Adding new events -1. Define events in [`known_events.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events.yml). +1. Define events in [`known_events`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events/). Example event: @@ -312,6 +312,7 @@ Implemented using Redis methods [PFADD](https://redis.io/commands/pfadd) and [PF - `aggregation`: aggregation `:daily` or `:weekly`. The argument defines how we build the Redis keys for data storage. For `daily` we keep a key for metric per day of the year, for `weekly` we keep a key for metric per week of the year. + - `feature_flag`: optional. For details, see our [GitLab internal Feature flags](../feature_flags/) documentation. 1. Track event in controller using `RedisTracking` module with `track_redis_hll_event(*controller_actions, name:, feature:, feature_default_enabled: false)`. @@ -402,7 +403,7 @@ Implemented using Redis methods [PFADD](https://redis.io/commands/pfadd) and [PF | `event` | string | yes | The event name it should be tracked | Response -w + Return 200 if tracking failed for any reason. - `200` if event was tracked or any errors @@ -412,7 +413,7 @@ w 1. Track events using JavaScript/Vue API helper which calls the API above - Example usage for an existing event already defined in [known events](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events.yml): + Example usage for an existing event already defined in [known events](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events/): Note that `usage_data_api` and `usage_data_#{event_name}` should be enabled in order to be able to track events @@ -422,20 +423,46 @@ w api.trackRedisHllUserEvent('my_already_defined_event_name'), ``` -1. Track event using base module `Gitlab::UsageDataCounters::HLLRedisCounter.track_event(entity_id, event_name)`. +1. Track event using base module `Gitlab::UsageDataCounters::HLLRedisCounter.track_event(values, event_name)`. + + Arguments: + + - `values`: One value or array of values we count. For example: user_id, visitor_id, user_ids. + - `event_name`: event name. + +1. Track event on context level using base module `Gitlab::UsageDataCounters::HLLRedisCounter.track_event_in_context(entity_id, event_name, context)`. Arguments: - `entity_id`: value we count. For example: user_id, visitor_id. - `event_name`: event name. + - `context`: context value. Allowed values are `default`, `free`, `bronze`, `silver`, `gold`, `starter`, `premium`, `ultimate` -1. Get event data using `Gitlab::UsageDataCounters::HLLRedisCounter.unique_events(event_names:, start_date:, end_date)`. +1. Get event data using `Gitlab::UsageDataCounters::HLLRedisCounter.unique_events(event_names:, start_date:, end_date:, context: '')`. Arguments: - `event_names`: the list of event names. - `start_date`: start date of the period for which we want to get event data. - `end_date`: end date of the period for which we want to get event data. + - `context`: context of the event. Allowed values are `default`, `free`, `bronze`, `silver`, `gold`, `starter`, `premium`, `ultimate`. + +1. Testing tracking and getting unique events + +Trigger events in rails console by using `track_event` method + + ```ruby + Gitlab::UsageDataCounters::HLLRedisCounter.track_event(1, 'g_compliance_audit_events') + Gitlab::UsageDataCounters::HLLRedisCounter.track_event(2, 'g_compliance_audit_events') + ``` + +Next, get the unique events for the current week. + + ```ruby + # Get unique events for metric for current_week + Gitlab::UsageDataCounters::HLLRedisCounter.unique_events(event_names: 'g_compliance_audit_events', + start_date: Date.current.beginning_of_week, end_date: Date.current.end_of_week) + ``` Recommendations: @@ -445,9 +472,23 @@ Recommendations: - Use a [feature flag](../../operations/feature_flags.md) to have a control over the impact when adding new metrics. +##### Enable/Disable Redis HLL tracking + +Events are tracked behind [feature flags](../feature_flags/index.md) due to concerns for Redis performance and scalability. + +For a full list of events and coresponding feature flags see, [known_events](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events/) files. + +To enable or disable tracking for specific event within <https://gitlab.com> or <https://staging.gitlab.com>, run commands such as the following to +[enable or disable the corresponding feature](../feature_flags/index.md). + +```shell +/chatops run feature set <feature_name> true +/chatops run feature set <feature_name> false +``` + ##### Known events in usage data payload -All events added in [`known_events.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events.yml) are automatically added to usage data generation under the `redis_hll_counters` key. This column is stored in [version-app as a JSON](https://gitlab.com/gitlab-services/version-gitlab-com/-/blob/master/db/schema.rb#L209). +All events added in [`known_events/common.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events/common.yml) are automatically added to usage data generation under the `redis_hll_counters` key. This column is stored in [version-app as a JSON](https://gitlab.com/gitlab-services/version-gitlab-com/-/blob/master/db/schema.rb#L209). For each event we add metrics for the weekly and monthly time frames, and totals for each where applicable: - `#{event_name}_weekly`: Data for 7 days for daily [aggregation](#adding-new-events) events and data for the last complete week for weekly [aggregation](#adding-new-events) events. @@ -493,7 +534,7 @@ Example usage: redis_usage_data(Gitlab::UsageDataCounters::WikiPageCounter) redis_usage_data { ::Gitlab::UsageCounters::PodLogs.usage_totals[:total] } -# Define events in known_events.yml https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events.yml +# Define events in common.yml https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events/common.yml # Tracking events Gitlab::UsageDataCounters::HLLRedisCounter.track_event(visitor_id, 'expand_vulnerabilities') @@ -548,7 +589,17 @@ for how to use its API to query for data. ## Developing and testing Usage Ping -### 1. Use your Rails console to manually test counters +### 1. Naming and placing the metrics + +Add the metric in one of the top level keys + +- `license`: for license related metrics. +- `settings`: for settings related metrics. +- `counts_weekly`: for counters that have data for the most recent 7 days. +- `counts_monthly`: for counters that have data for the most recent 28 days. +- `counts`: for counters that have data for all time. + +### 2. Use your Rails console to manually test counters ```ruby # count @@ -560,7 +611,7 @@ Gitlab::UsageData.distinct_count(::Project, :creator_id) Gitlab::UsageData.distinct_count(::Note.with_suggestions.where(time_period), :author_id, start: ::User.minimum(:id), finish: ::User.maximum(:id)) ``` -### 2. Generate the SQL query +### 3. Generate the SQL query Your Rails console will return the generated SQL queries. @@ -574,7 +625,7 @@ pry(main)> Gitlab::UsageData.count(User.active) (1.9ms) SELECT COUNT("users"."id") FROM "users" WHERE ("users"."state" IN ('active')) AND ("users"."user_type" IS NULL OR "users"."user_type" IN (6, 4)) AND "users"."id" BETWEEN 1 AND 100000 ``` -### 3. Optimize queries with #database-lab +### 4. Optimize queries with #database-lab Paste the SQL query into `#database-lab` to see how the query performs at scale. @@ -601,27 +652,27 @@ We also use `#database-lab` and [explain.depesz.com](https://explain.depesz.com/ - Avoid joins and write the queries as simply as possible, [example](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/36316). - Set a custom `batch_size` for `distinct_count`, [example](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38000). -### 4. Add the metric definition +### 5. Add the metric definition -When adding, changing, or updating metrics, please update the [Event Dictionary's **Usage Ping** table](event_dictionary.md). +When adding, changing, or updating metrics, please update the [Event Dictionary's **Usage Ping** table](https://about.gitlab.com/handbook/product/product-analytics-guide#event-dictionary). -### 5. Add new metric to Versions Application +### 6. Add new metric to Versions Application Check if new metrics need to be added to the Versions Application. See `usage_data` [schema](https://gitlab.com/gitlab-services/version-gitlab-com/-/blob/master/db/schema.rb#L147) and usage data [parameters accepted](https://gitlab.com/gitlab-services/version-gitlab-com/-/blob/master/app/services/usage_ping.rb). Any metrics added under the `counts` key are saved in the `stats` column. -### 6. Add the feature label +### 7. Add the feature label Add the `feature` label to the Merge Request for new Usage Ping metrics. These are user-facing changes and are part of expanding the Usage Ping feature. -### 7. Add a changelog file +### 8. Add a changelog file Ensure you comply with the [Changelog entries guide](../changelog.md). -### 8. Ask for a Product Analytics Review +### 9. Ask for a Product Analytics Review On GitLab.com, we have DangerBot setup to monitor Product Analytics related files and DangerBot will recommend a Product Analytics review. Mention `@gitlab-org/growth/product_analytics/engineers` in your MR for a review. -### 9. Verify your metric +### 10. Verify your metric On GitLab.com, the Product Analytics team regularly monitors Usage Ping. They may alert you that your metrics need further optimization to run quicker and with greater success. You may also use the [Usage Ping QA dashboard](https://app.periscopedata.com/app/gitlab/632033/Usage-Ping-QA) to check how well your metric performs. The dashboard allows filtering by GitLab version, by "Self-managed" & "Saas" and shows you how many failures have occurred for each metric. Whenever you notice a high failure rate, you may re-optimize your metric. @@ -671,6 +722,59 @@ with any of the other services that are running. That is not how node metrics ar always runs as a process alongside other GitLab components on any given node. From Usage Ping's perspective none of the node data would therefore appear to be associated to any of the services running, since they all appear to be running on different hosts. To alleviate this problem, the `node_exporter` in GCK was arbitrarily "assigned" to the `web` service, meaning only for this service `node_*` metrics will appear in Usage Ping. +## Aggregated metrics + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/45979) in GitLab 13.6. +> - It's [deployed behind a feature flag](../../user/feature_flags.md), disabled by default. +> - It's enabled on GitLab.com. + +CAUTION: **Warning:** +This feature is intended solely for internal GitLab use. + +In order to add data for aggregated metrics into Usage Ping payload you should add corresponding definition into [`aggregated_metrics.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/aggregated_metrics.yml) file. Each aggregate definition includes following parts: + +- name: unique name under which aggregate metric will be added to Usage Ping payload +- operator: operator that defines how aggregated metric data will be counted. Available operators are: + - `OR`: removes duplicates and counts all entries that triggered any of listed events + - `AND`: removes duplicates and counts all elements that were observed triggering all of following events +- events: list of events names (from [`known_events.yml`](#known-events-in-usage-data-payload)) to aggregate into metric. All events in this list must have the same `redis_slot` and `aggregation` attributes. +- feature_flag: name of [development feature flag](../feature_flags/development.md#development-type) that will be checked before +metrics aggregation is performed. Corresponding feature flag should have `default_enabled` attribute set to `false`. +`feature_flag` attribute is **OPTIONAL** and can be omitted, when `feature_flag` is missing no feature flag will be checked. + +Example aggregated metric entries: + +```yaml +- name: product_analytics_test_metrics_union + operator: OR + events: ['i_search_total', 'i_search_advanced', 'i_search_paid'] +- name: product_analytics_test_metrics_intersection_with_feautre_flag + operator: AND + events: ['i_search_total', 'i_search_advanced', 'i_search_paid'] + feature_flag: example_aggregated_metric +``` + +Aggregated metrics will be added under `aggregated_metrics` key in both `counts_weekly` and `counts_monthly` top level keys in Usage Ping payload. + +```ruby +{ + :counts_monthly => { + :deployments => 1003, + :successful_deployments => 78, + :failed_deployments => 275, + :packages => 155, + :personal_snippets => 2106, + :project_snippets => 407, + :promoted_issues => 719, + :aggregated_metrics => { + :product_analytics_test_metrics_union => 7, + :product_analytics_test_metrics_intersection_with_feautre_flag => 2 + }, + :snippets => 2513 + } +} +``` + ## Example Usage Ping payload The following is example content of the Usage Ping payload. @@ -935,3 +1039,7 @@ bin/rake gitlab:usage_data:dump_sql_in_json # You may pipe the output into a file bin/rake gitlab:usage_data:dump_sql_in_yaml > ~/Desktop/usage-metrics-2020-09-02.yaml ``` + +## Generating and troubleshooting usage ping + +To get a usage ping, or to troubleshoot caching issues on your GitLab instance, please follow [instructions to generate usage ping](../../administration/troubleshooting/gitlab_rails_cheat_sheet.md#generate-usage-ping). diff --git a/doc/development/profiling.md b/doc/development/profiling.md index f5a4d1edb92..6e1b81e659d 100644 --- a/doc/development/profiling.md +++ b/doc/development/profiling.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Profiling To make it easier to track down performance problems GitLab comes with a set of @@ -10,7 +16,6 @@ There is a `Gitlab::Profiler.profile` method, and corresponding `bin/profile-url` script, that enable profiling a GET or POST request to a specific URL, either as an anonymous user (the default) or as a specific user. -NOTE: **Note:** The first argument to the profiler is either a full URL (including the instance hostname) or an absolute path, including the leading slash. diff --git a/doc/development/projections.md b/doc/development/projections.md index 9d5702da530..b8f476c53e9 100644 --- a/doc/development/projections.md +++ b/doc/development/projections.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Projections Projections are a way to define relations between files. Every file can have a diff --git a/doc/development/prometheus.md b/doc/development/prometheus.md index 3e7acaf6d94..fc1f2303d0a 100644 --- a/doc/development/prometheus.md +++ b/doc/development/prometheus.md @@ -1,59 +1,5 @@ --- -stage: Monitor -group: Health -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +redirect_to: '../user/project/integrations/prometheus.md' --- -# Working with Prometheus - -For more information on working with [Prometheus metrics](prometheus_metrics.md), see -the documentation. - -## Access the UI of a Prometheus managed application in Kubernetes - -You can connect directly to Prometheus, and view the Prometheus user interface, when -using a Prometheus managed application in Kubernetes: - -1. Find the name of the Prometheus pod in the user interface of your Kubernetes - provider, such as GKE, or by running the following `kubectl` command in your - terminal: - - ```shell - kubectl get pods -n gitlab-managed-apps | grep 'prometheus-prometheus-server' - ``` - - The command should return a result like the following example, where - `prometheus-prometheus-server-55b4bd64c9-dpc6b` is the name of the Prometheus pod: - - ```plaintext - gitlab-managed-apps prometheus-prometheus-server-55b4bd64c9-dpc6b 2/2 Running 0 71d - ``` - -1. Run a `kubectl port-forward` command. In the following example, `9090` is the - Prometheus server's listening port: - - ```shell - kubectl port-forward prometheus-prometheus-server-55b4bd64c9-dpc6b 9090:9090 -n gitlab-managed-apps - ``` - - The `port-forward` command forwards all requests sent to your system's `9090` port - to the `9090` port of the Prometheus pod. If the `9090` port on your system is used - by another application, you can change the port number before the colon to your - desired port. For example, to forward port `8080` of your local system, change the - command to: - - ```shell - kubectl port-forward prometheus-prometheus-server-55b4bd64c9-dpc6b 8080:9090 -n gitlab-managed-apps - ``` - -1. Open `localhost:9090` in your browser to display the Prometheus user interface. - -## Script access to Prometheus - -You can script the access to Prometheus, extracting the name of the pod automatically like this: - -```shell -POD_INFORMATION=$(kubectl get pods -n gitlab-managed-apps | grep 'prometheus-prometheus-server') -POD_NAME=$(echo $POD_INFORMATION | awk '{print $1;}') -kubectl port-forward $POD_NAME 9090:9090 -n gitlab-managed-apps -``` +This document was moved to [another location](../user/project/integrations/prometheus.md). diff --git a/doc/development/pry_debugging.md b/doc/development/pry_debugging.md index 0558a0a515a..7f9a49d4d50 100644 --- a/doc/development/pry_debugging.md +++ b/doc/development/pry_debugging.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Pry debugging ## Invoking pry debugging diff --git a/doc/development/python_guide/index.md b/doc/development/python_guide/index.md index fae2e495417..22a01c3b877 100644 --- a/doc/development/python_guide/index.md +++ b/doc/development/python_guide/index.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Python Development Guidelines GitLab requires Python as a dependency for [reStructuredText](https://docutils.sourceforge.io/rst.html) diff --git a/doc/development/query_count_limits.md b/doc/development/query_count_limits.md index b3ecaf30d8a..07dcb6c7d90 100644 --- a/doc/development/query_count_limits.md +++ b/doc/development/query_count_limits.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Query Count Limits Each controller or API endpoint is allowed to execute up to 100 SQL queries and diff --git a/doc/development/query_recorder.md b/doc/development/query_recorder.md index ef9a3c657aa..4a02af3348c 100644 --- a/doc/development/query_recorder.md +++ b/doc/development/query_recorder.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # QueryRecorder QueryRecorder is a tool for detecting the [N+1 queries problem](https://guides.rubyonrails.org/active_record_querying.html#eager-loading-associations) from tests. @@ -20,17 +26,17 @@ end As an example you might create 5 issues in between counts, which would cause the query count to increase by 5 if an N+1 problem exists. -NOTE: **Note:** In some cases the query count might change slightly between runs for unrelated reasons. In this case you might need to test `exceed_query_limit(control_count + acceptable_change)`, but this should be avoided if possible. ## Cached queries -By default, QueryRecorder will ignore cached queries in the count. However, it may be better to count -all queries to avoid introducing an N+1 query that may be masked by the statement cache. To do this, -pass the `skip_cached` variable to `QueryRecorder` and use the `exceed_all_query_limit` matcher: +By default, QueryRecorder will ignore [cached queries](merge_request_performance_guidelines.md#cached-queries) in the count. However, it may be better to count +all queries to avoid introducing an N+1 query that may be masked by the statement cache. +To do this, this requires the `:use_sql_query_cache` flag to be set. +You should pass the `skip_cached` variable to `QueryRecorder` and use the `exceed_all_query_limit` matcher: ```ruby -it "avoids N+1 database queries" do +it "avoids N+1 database queries", :use_sql_query_cache do control_count = ActiveRecord::QueryRecorder.new(skip_cached: false) { visit_some_page }.count create_list(:issue, 5) expect { visit_some_page }.not_to exceed_all_query_limit(control_count) @@ -118,4 +124,5 @@ There are multiple ways to find the source of queries. - [Bullet](profiling.md#bullet) For finding `N+1` query problems - [Performance guidelines](performance.md) -- [Merge request performance guidelines](merge_request_performance_guidelines.md#query-counts) +- [Merge request performance guidelines - Query counts](merge_request_performance_guidelines.md#query-counts) +- [Merge request performance guidelines - Cached queries](merge_request_performance_guidelines.md#cached-queries) diff --git a/doc/development/rails_initializers.md b/doc/development/rails_initializers.md index 6473baf58d4..bc0ac963f3f 100644 --- a/doc/development/rails_initializers.md +++ b/doc/development/rails_initializers.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Rails initializers By default, Rails loads Zeitwerk after the initializers in `config/initializers` are loaded. diff --git a/doc/development/rake_tasks.md b/doc/development/rake_tasks.md index fd5dee69fc3..2dca747425c 100644 --- a/doc/development/rake_tasks.md +++ b/doc/development/rake_tasks.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Rake tasks for developers Rake tasks are available for developers and others contributing to GitLab. @@ -12,7 +18,7 @@ bundle exec rake setup The `setup` task is an alias for `gitlab:setup`. This tasks calls `db:reset` to create the database, and calls `db:seed_fu` to seed the database. -Note: `db:setup` calls `db:seed` but this does nothing. +`db:setup` calls `db:seed` but this does nothing. ### Environment variables diff --git a/doc/development/reactive_caching.md b/doc/development/reactive_caching.md index cf125c46565..106a9b5de31 100644 --- a/doc/development/reactive_caching.md +++ b/doc/development/reactive_caching.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # `ReactiveCaching` > This doc refers to <https://gitlab.com/gitlab-org/gitlab/blob/master/app/models/concerns/reactive_caching.rb>. diff --git a/doc/development/redis.md b/doc/development/redis.md index a0ae84beb8d..7ee8afda36a 100644 --- a/doc/development/redis.md +++ b/doc/development/redis.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Redis guidelines GitLab uses [Redis](https://redis.io) for the following distinct purposes: diff --git a/doc/development/refactoring_guide/index.md b/doc/development/refactoring_guide/index.md index a9ff9556aed..17deffb4cb2 100644 --- a/doc/development/refactoring_guide/index.md +++ b/doc/development/refactoring_guide/index.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Refactoring guide This document is a collection of techniques and best practices to consider while performing a refactor. diff --git a/doc/development/reference_processing.md b/doc/development/reference_processing.md index cf587043cae..07833c0d302 100644 --- a/doc/development/reference_processing.md +++ b/doc/development/reference_processing.md @@ -1,4 +1,7 @@ --- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers description: 'An introduction to reference parsers and reference filters, and a guide to their implementation.' --- @@ -10,7 +13,6 @@ abstractions in the `Banzai` pipeline: `ReferenceFilter` and `ReferenceParser`. This page explains what these are, how they are used, and how you would implement a new filter/parser pair. -NOTE: **Note:** Each `ReferenceFilter` must have a corresponding `ReferenceParser`. It is possible to share reference parsers between filters - if two filters find @@ -39,7 +41,7 @@ For example, the class is responsible for handling references to issues, such as `gitlab-org/gitlab#123` and `https://gitlab.com/gitlab-org/gitlab/-/issues/200048`. -All reference filters are instances of [`HTML::Pipeline::Filter`](https://www.rubydoc.info/github/jch/html-pipeline/v1.11.0/HTML/Pipeline/Filter), +All reference filters are instances of [`HTML::Pipeline::Filter`](https://www.rubydoc.info/github/jch/html-pipeline/HTML/Pipeline/Filter), and inherit (often indirectly) from [`Banzai::Filter::ReferenceFilter`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/banzai/filter/reference_filter.rb). `HTML::Pipeline::Filter` has a simple interface consisting of `#call`, a void @@ -196,6 +198,5 @@ In practice, all reference parsers inherit from [`BaseParser`](https://gitlab.co - `#references_relation` an active record relation for objects by ID. - `#nodes_user_can_reference(user, nodes)` to filter nodes directly. -NOTE: **Note:** A failure to implement this class for each reference type means that the application will raise exceptions during Markdown processing. diff --git a/doc/development/renaming_features.md b/doc/development/renaming_features.md index daf437027db..02d1851dbfe 100644 --- a/doc/development/renaming_features.md +++ b/doc/development/renaming_features.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Renaming features Sometimes the business asks to change the name of a feature. Broadly speaking, there are 2 approaches to that task. They basically trade between immediate effort and future complexity/bug risk: diff --git a/doc/development/repository_mirroring.md b/doc/development/repository_mirroring.md index fe6db987471..02d1499ca35 100644 --- a/doc/development/repository_mirroring.md +++ b/doc/development/repository_mirroring.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Repository mirroring ## Deep Dive diff --git a/doc/development/reusing_abstractions.md b/doc/development/reusing_abstractions.md index 8711bac69e0..77534eea076 100644 --- a/doc/development/reusing_abstractions.md +++ b/doc/development/reusing_abstractions.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Guidelines for reusing abstractions As GitLab has grown, different patterns emerged across the codebase. Service diff --git a/doc/development/rolling_out_changes_using_feature_flags.md b/doc/development/rolling_out_changes_using_feature_flags.md index 6bad91d6287..cff88388ba0 100644 --- a/doc/development/rolling_out_changes_using_feature_flags.md +++ b/doc/development/rolling_out_changes_using_feature_flags.md @@ -1 +1,5 @@ +--- +redirect_to: 'feature_flags/index.md' +--- + This document was moved to [another location](feature_flags/index.md). diff --git a/doc/development/routing.md b/doc/development/routing.md index e164431853f..cbae2b38427 100644 --- a/doc/development/routing.md +++ b/doc/development/routing.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Routing The GitLab backend is written primarily with Rails so it uses [Rails diff --git a/doc/development/scalability.md b/doc/development/scalability.md index 0fb54d89913..73f7c5e0915 100644 --- a/doc/development/scalability.md +++ b/doc/development/scalability.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # GitLab scalability This section describes the current architecture of GitLab as it relates to diff --git a/doc/development/secure_coding_guidelines.md b/doc/development/secure_coding_guidelines.md index e35bda82aaa..ebab0e59cc3 100644 --- a/doc/development/secure_coding_guidelines.md +++ b/doc/development/secure_coding_guidelines.md @@ -219,11 +219,11 @@ the mitigations for a new feature. - [More details](https://dev.gitlab.org/gitlab/gitlabhq/-/merge_requests/2530/diffs) -#### Feature-specific Mitigations +#### Feature-specific mitigations For situations in which an allowlist or GitLab:HTTP cannot be used, it will be necessary to implement mitigations directly in the feature. It is best to validate the destination IP addresses themselves, not just domain names, as DNS can be controlled by the attacker. Below are a list of mitigations that should be implemented. -**Important Note:** There are many tricks to bypass common SSRF validations. If feature-specific mitigations are necessary, they should be reviewed by the AppSec team, or a developer who has worked on SSRF mitigations previously. +There are many tricks to bypass common SSRF validations. If feature-specific mitigations are necessary, they should be reviewed by the AppSec team, or a developer who has worked on SSRF mitigations previously. - Block connections to all localhost addresses - `127.0.0.1/8` (IPv4 - note the subnet mask) @@ -269,7 +269,7 @@ When user submitted data is included in responses to end users, which is just ab ### Mitigation -In most situations, a two-step solution can be utilized: input validation and output encoding in the appropriate context. +In most situations, a two-step solution can be used: input validation and output encoding in the appropriate context. #### Input validation @@ -311,6 +311,7 @@ Specifically, the following options are dangerous because they mark strings as t |----------------------|-------------------------------| | HAML templates | `html_safe`, `raw`, `!=` | | Embedded Ruby (ERB) | `html_safe`, `raw`, `<%== %>` | + In case you want to sanitize user-controlled values against XSS vulnerabilities, you can use [`ActionView::Helpers::SanitizeHelper`](https://api.rubyonrails.org/classes/ActionView/Helpers/SanitizeHelper.html). Calling `link_to` and `redirect_to` with user-controlled parameters can also lead to cross-site scripting. @@ -328,6 +329,7 @@ References: - When updating the content of an HTML element using JavaScript, mark user-controlled values as `textContent` or `nodeValue` instead of `innerHTML`. - Avoid using `v-html` with user-controlled data, use [`v-safe-html`](https://gitlab-org.gitlab.io/gitlab-ui/?path=/story/directives-safe-html-directive--default) instead. +- Render unsafe or unsanitized content using [`dompurify`](fe_guide/security.md#sanitize-html-output). - Consider using [`gl-sprintf`](../../ee/development/i18n/externalization.md#interpolation) to interpolate translated strings securely. - Avoid `__()` with translations that contain user-controlled values. - When working with `postMessage`, ensure the `origin` of the message is allowlisted. @@ -428,6 +430,132 @@ The Path Traversal check can also be used to forbid any absolute path: requires :file_path, type: String, file_path: true ``` -NOTE: **Note:** Absolute paths are not allowed by default. If allowing an absolute path is required, you -need to provide an array of paths to the parameter `allowlist`. +need to provide an array of paths to the parameter `allowlist`. + +## OS command injection guidelines + +Command injection is an issue in which an attacker is able to execute arbitrary commands on the host +operating system through a vulnerable application. Such attacks don't always provide feedback to a +user, but the attacker can use simple commands like `curl` to obtain an answer. + +### Impact + +The impact of command injection greatly depends on the user context running the commands, as well as +how data is validated and sanitized. It can vary from low impact because the user running the +injected commands has limited rights, to critical impact if running as the root user. + +Potential impacts include: + +- Execution of arbitrary commands on the host machine. +- Unauthorized access to sensitive data, including passwords and tokens in secrets or configuration + files. +- Exposure of sensitive system files on the host machine, such as `/etc/passwd/` or `/etc/shadow`. +- Compromise of related systems and services gained through access to the host machine. + +You should be aware of and take steps to prevent command injection when working with user-controlled +data that are used to run OS commands. + +### Mitigation and prevention + +To prevent OS command injections, user-supplied data shouldn't be used within OS commands. In cases +where you can't avoid this: + +- Validate user-supplied data against an allowlist. +- Ensure that user-supplied data only contains alphanumeric characters (and no syntax or whitespace + characters, for example). +- Always use `--` to separate options from arguments. + +### Ruby + +Consider using `system("command", "arg0", "arg1", ...)` whenever you can. This prevents an attacker +from concatenating commands. + +For more examples on how to use shell commands securely, consult +[Guidelines for shell commands in the GitLab codebase](shell_commands.md). +It contains various examples on how to securely call OS commands. + +### Go + +Go has built-in protections that usually prevent an attacker from successfully injecting OS commands. + +Consider the following example: + +```golang +package main + +import ( + "fmt" + "os/exec" +) + +func main() { + cmd := exec.Command("echo", "1; cat /etc/passwd") + out, _ := cmd.Output() + fmt.Printf("%s", out) +} +``` + +This echoes `"1; cat /etc/passwd"`. + +**Do not** use `sh`, as it bypasses internal protections: + +```golang +out, _ = exec.Command("sh", "-c", "echo 1 | cat /etc/passwd").Output() +``` + +This outputs `1` followed by the content of `/etc/passwd`. + +## GitLab Internal Authorization + +### Introduction + +There are some cases where `users` passed in the code is actually referring to a `DeployToken`/`DeployKey` entity instead of a real `User`, because of the code below in **`/lib/api/api_guard.rb`** + +```ruby + def find_user_from_sources + strong_memoize(:find_user_from_sources) do + deploy_token_from_request || + find_user_from_bearer_token || + find_user_from_job_token || + user_from_warden + end + end +``` + +### Past Vulnerable Code + +In some scenarios such as [this one](https://gitlab.com/gitlab-org/gitlab/-/issues/237795), user impersonation is possible because a `DeployToken` ID can be used in place of a `User` ID. This happened because there was no check on the line with `Gitlab::Auth::CurrentUserMode.bypass_session!(user.id)`. In this case, the `id` is actually a `DeployToken` ID instead of a `User` ID. + +```ruby + def find_current_user! + user = find_user_from_sources + return unless user + + # Sessions are enforced to be unavailable for API calls, so ignore them for admin mode + Gitlab::Auth::CurrentUserMode.bypass_session!(user.id) if Feature.enabled?(:user_mode_in_session) + + unless api_access_allowed?(user) + forbidden!(api_access_denied_message(user)) + end +``` + +### Best Practices + +In order to prevent this from happening, it is recommended to use the method `user.is_a?(User)` to make sure it returns `true` when we are expecting to deal with a `User` object. This could prevent the ID confusion from the method `find_user_from_sources` mentioned above. Below code snippet shows the fixed code after applying the best practice to the vulnerable code above. + +```ruby + def find_current_user! + user = find_user_from_sources + return unless user + + if user.is_a?(User) && Feature.enabled?(:user_mode_in_session) + # Sessions are enforced to be unavailable for API calls, so ignore them for admin mode + Gitlab::Auth::CurrentUserMode.bypass_session!(user.id) + end + + unless api_access_allowed?(user) + forbidden!(api_access_denied_message(user)) + end +``` +
\ No newline at end of file diff --git a/doc/development/serializing_data.md b/doc/development/serializing_data.md index af791f5a4e0..83a75eef556 100644 --- a/doc/development/serializing_data.md +++ b/doc/development/serializing_data.md @@ -1,3 +1,9 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Serializing Data **Summary:** don't store serialized data in the database, use separate columns diff --git a/doc/development/service_measurement.md b/doc/development/service_measurement.md index 24b16aac95b..571bbddb494 100644 --- a/doc/development/service_measurement.md +++ b/doc/development/service_measurement.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # GitLab Developers Guide to service measurement You can enable service measurement in order to debug any slow service's execution time, number of SQL calls, garbage collection stats, memory usage, etc. @@ -16,7 +22,6 @@ The measuring module is a tool that allows to measure a service's execution, and The measuring module will log these measurements into a structured log called [`service_measurement.log`](../administration/logs.md#service_measurementlog), as a single entry for each service execution. -NOTE: **Note:** For GitLab.com, `service_measurement.log` is ingested in Elasticsearch and Kibana as part of our monitoring solution. ## How to use it @@ -64,9 +69,8 @@ def extra_attributes_for_measurement end ``` -NOTE: **Note:** -Once the measurement module is injected in the service, it will be behind generic feature flag. -In order to actually use it, you need to enable measuring for the desired service by enabling the feature flag. +After the measurement module is injected in the service, it will be behind a generic feature flag. +To actually use it, you need to enable measuring for the desired service by enabling the feature flag. ### Enabling measurement using feature flags diff --git a/doc/development/session.md b/doc/development/session.md index 971795d8816..a7f69f570c3 100644 --- a/doc/development/session.md +++ b/doc/development/session.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Accessing session data Session data in GitLab is stored in Redis and can be accessed in a variety of ways. diff --git a/doc/development/sha1_as_binary.md b/doc/development/sha1_as_binary.md index 6c4252ec634..e69ba149cde 100644 --- a/doc/development/sha1_as_binary.md +++ b/doc/development/sha1_as_binary.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Storing SHA1 Hashes As Binary Storing SHA1 hashes as strings is not very space efficient. A SHA1 as a string diff --git a/doc/development/shared_files.md b/doc/development/shared_files.md index fcd905b54a4..2c6b1222cfb 100644 --- a/doc/development/shared_files.md +++ b/doc/development/shared_files.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Shared files Historically, GitLab has been storing shared files in many different diff --git a/doc/development/shell_commands.md b/doc/development/shell_commands.md index b71fcbdc2ab..25113b4ac29 100644 --- a/doc/development/shell_commands.md +++ b/doc/development/shell_commands.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Guidelines for shell commands in the GitLab codebase This document contains guidelines for working with processes and files in the GitLab codebase. diff --git a/doc/development/shell_scripting_guide/index.md b/doc/development/shell_scripting_guide/index.md index 704b46c7efa..f447a0e0e72 100644 --- a/doc/development/shell_scripting_guide/index.md +++ b/doc/development/shell_scripting_guide/index.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Shell scripting standards and style guidelines GitLab consists of many various services and sub-projects. The majority of diff --git a/doc/development/sidekiq_debugging.md b/doc/development/sidekiq_debugging.md index b6a11dd813d..e0f74b39c9a 100644 --- a/doc/development/sidekiq_debugging.md +++ b/doc/development/sidekiq_debugging.md @@ -1,6 +1,5 @@ -# Sidekiq debugging +--- +redirect_to: '../administration/troubleshooting/sidekiq.md' +--- -## Log arguments to Sidekiq jobs - -This content has been moved to the -[Troubleshooting Sidekiq docs](../administration/troubleshooting/sidekiq.md). +This document was moved to [another location](../administration/troubleshooting/sidekiq.md). diff --git a/doc/development/sidekiq_style_guide.md b/doc/development/sidekiq_style_guide.md index 24570cfc07b..13ae39997bc 100644 --- a/doc/development/sidekiq_style_guide.md +++ b/doc/development/sidekiq_style_guide.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Sidekiq Style Guide This document outlines various guidelines that should be followed when adding or @@ -21,7 +27,10 @@ After adding a new queue, run `bin/rake gitlab:sidekiq:all_queues_yml:generate` to regenerate `app/workers/all_queues.yml` or `ee/app/workers/all_queues.yml` so that it can be picked up by -[`sidekiq-cluster`](../administration/operations/extra_sidekiq_processes.md). +[`sidekiq-cluster`](../administration/operations/extra_sidekiq_processes.md). +Additionally, run +`bin/rake gitlab:sidekiq:sidekiq_queues_yml:generate` to regenerate +`config/sidekiq_queues.yml`. ## Queue Namespaces @@ -112,7 +121,6 @@ As a general rule, a worker can be considered idempotent if: A good example of that would be a cache expiration worker. -NOTE: **Note:** A job scheduled for an idempotent worker will automatically be [deduplicated](#deduplication) when an unstarted job with the same arguments is already in the queue. @@ -152,7 +160,6 @@ end It's encouraged to only have the `idempotent!` call in the top-most worker class, even if the `perform` method is defined in another class or module. -NOTE: **Note:** If the worker class is not marked as idempotent, a cop will fail. Consider skipping the cop if you're not confident your job can safely run multiple times. @@ -165,6 +172,22 @@ job. The work is skipped because the same work would be done by the job that was scheduled first; by the time the second job executed, the first job would do nothing. +#### Strategies + +GitLab supports two deduplication strategies: + +- `until_executing` +- `until_executed` + +More [deduplication strategies have been +suggested](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/195). If +you are implementing a worker that could benefit from a different +strategy, please comment in the issue. + +##### Until Executing + +This strategy takes a lock when a job is added to the queue, and removes that lock before the job starts. + For example, `AuthorizedProjectsWorker` takes a user ID. When the worker runs, it recalculates a user's authorizations. GitLab schedules this job each time an action potentially changes a user's @@ -173,10 +196,47 @@ same time, the second job can be skipped if the first job hasn't begun, because when the first job runs, it creates the authorizations for both projects. +```ruby +module AuthorizedProjectUpdate + class UserRefreshOverUserRangeWorker + include ApplicationWorker + + deduplicate :until_executing + idempotent! + + # ... + end +end +``` + +##### Until Executed + +This strategy takes a lock when a job is added to the queue, and removes that lock after the job finishes. +It can be used to prevent jobs from running simultaneously multiple times. + +```ruby +module Ci + class BuildTraceChunkFlushWorker + include ApplicationWorker + + deduplicate :until_executed + idempotent! + + # ... + end +end +``` + +#### Scheduling jobs in the future + GitLab doesn't skip jobs scheduled in the future, as we assume that the state will have changed by the time the job is scheduled to -execute. If you do want to deduplicate jobs scheduled in the future -this can be specified on the worker as follows: +execute. Deduplication of jobs scheduled in the feature is possible +for both `until_executed` and `until_executing` strategies. + +If you do want to deduplicate jobs scheduled in the future, +this can be specified on the worker by passing `including_scheduled: true` argument +when defining deduplication strategy: ```ruby module AuthorizedProjectUpdate @@ -191,11 +251,7 @@ module AuthorizedProjectUpdate end ``` -This strategy is called `until_executing`. More [deduplication -strategies have been -suggested](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/195). If -you are implementing a worker that could benefit from a different -strategy, please comment in the issue. +#### Troubleshooting If the automatic deduplication were to cause issues in certain queues. This can be temporarily disabled by enabling a feature flag @@ -429,9 +485,7 @@ class ExternalDependencyWorker end ``` -NOTE: **Note:** -Note that a job cannot be both high urgency and have -external dependencies. +A job cannot be both high urgency and have external dependencies. ## CPU-bound and Memory-bound Workers @@ -645,8 +699,8 @@ blocks: ## Arguments logging -When [`SIDEKIQ_LOG_ARGUMENTS`](../administration/troubleshooting/sidekiq.md#log-arguments-to-sidekiq-jobs) -is enabled, Sidekiq job arguments will be logged. +As of GitLab 13.6, Sidekiq job arguments will be logged by default, unless [`SIDEKIQ_LOG_ARGUMENTS`](../administration/troubleshooting/sidekiq.md#log-arguments-to-sidekiq-jobs) +is disabled. By default, the only arguments logged are numeric arguments, because arguments of other types could contain sensitive information. To @@ -767,7 +821,7 @@ This approach requires multiple releases. ##### Parameter hash This approach will not require multiple releases if an existing worker already -utilizes a parameter hash. +uses a parameter hash. 1. Use a parameter hash in the worker to allow future flexibility. diff --git a/doc/development/single_table_inheritance.md b/doc/development/single_table_inheritance.md index 27c3c4f3199..51268092225 100644 --- a/doc/development/single_table_inheritance.md +++ b/doc/development/single_table_inheritance.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Single Table Inheritance **Summary:** don't use Single Table Inheritance (STI), use separate tables diff --git a/doc/development/sql.md b/doc/development/sql.md index 55a8192578b..bf405eab688 100644 --- a/doc/development/sql.md +++ b/doc/development/sql.md @@ -1,3 +1,9 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # SQL Query Guidelines This document describes various guidelines to follow when writing SQL queries, diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index 2c1d70a005e..dabb18c1f75 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -361,7 +361,7 @@ Finished in 34.51 seconds (files took 0.76702 seconds to load) 1 example, 0 failures ``` -Note: `live_debug` only works on JavaScript enabled specs. +`live_debug` only works on JavaScript enabled specs. #### Run `:js` spec in a visible browser @@ -584,9 +584,9 @@ this trait should be either fixed to not rely on Sidekiq processing jobs, or the `:sidekiq_might_not_need_inline` trait should be updated to `:sidekiq_inline` if the processing of background jobs is needed/expected. -NOTE: **Note:** -The usage of `perform_enqueued_jobs` is only useful for testing delayed mail -deliveries since our Sidekiq workers aren't inheriting from `ApplicationJob` / `ActiveJob::Base`. +The usage of `perform_enqueued_jobs` is useful only for testing delayed mail +deliveries, because our Sidekiq workers aren't inheriting from `ApplicationJob` +/ `ActiveJob::Base`. #### DNS diff --git a/doc/development/testing_guide/ci.md b/doc/development/testing_guide/ci.md index 8091142410c..618f9010b4d 100644 --- a/doc/development/testing_guide/ci.md +++ b/doc/development/testing_guide/ci.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # GitLab tests in the Continuous Integration (CI) context ## Test suite parallelization on the CI diff --git a/doc/development/testing_guide/end_to_end/beginners_guide.md b/doc/development/testing_guide/end_to_end/beginners_guide.md index a1883f44170..ef0bd9902e1 100644 --- a/doc/development/testing_guide/end_to_end/beginners_guide.md +++ b/doc/development/testing_guide/end_to_end/beginners_guide.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Beginner's guide to writing end-to-end tests In this tutorial, you will learn about the creation of end-to-end (_e2e_) tests @@ -52,7 +58,6 @@ Check both [GitLab Community Edition](https://gitlab-org.gitlab.io/gitlab-foss/c for previously-written tests for this feature. For analyzing the code coverage, you must understand which application files implement specific features. -NOTE: **Note:** In this tutorial we're writing a login end-to-end test, even though it has been sufficiently covered by lower-level testing, because it's the first step for most end-to-end flows, and is easiest to understand. @@ -68,7 +73,6 @@ under the stage.  -NOTE: **Note:** If the test is Enterprise Edition only, the test will be created in the `features/ee` directory, but follow the same DevOps lifecycle format. @@ -204,7 +208,6 @@ end 1. Check if the user avatar appears in the top navigation. 1. Check if the user avatar *does not* appear in the top navigation. -NOTE: **Note:** Behind the scenes, `be_signed_in` is a [predicate matcher](https://relishapp.com/rspec/rspec-expectations/v/3-8/docs/built-in-matchers/predicate-matchers) that [implements checking the user avatar](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/page/main/menu.rb#L74). diff --git a/doc/development/testing_guide/end_to_end/best_practices.md b/doc/development/testing_guide/end_to_end/best_practices.md index 58bae749dc5..4d12a0f79cb 100644 --- a/doc/development/testing_guide/end_to_end/best_practices.md +++ b/doc/development/testing_guide/end_to_end/best_practices.md @@ -1,6 +1,11 @@ +--- +stage: none +group: Development +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # End-to-end testing Best Practices -NOTE: **Note:** This is a tailored extension of the Best Practices [found in the testing guide](../best_practices.md). ## Link a test to its test-case issue @@ -263,7 +268,7 @@ We don't run tests that require Administrator access against our Production envi When you add a new test that requires Administrator access, apply the RSpec metadata `:requires_admin` so that the test will not be included in the test suites executed against Production and other environments on which we don't want to run those tests. -Note: When running tests locally or configuring a pipeline, the environment variable `QA_CAN_TEST_ADMIN_FEATURES` can be set to `false` to skip tests that have the `:requires_admin` tag. +When running tests locally or configuring a pipeline, the environment variable `QA_CAN_TEST_ADMIN_FEATURES` can be set to `false` to skip tests that have the `:requires_admin` tag. ## Prefer `Commit` resource over `ProjectPush` @@ -288,7 +293,6 @@ Resource::Repository::ProjectPush.fabricate! do |push| end ``` -NOTE: **Note:** A few exceptions for using a `ProjectPush` would be when your test calls for testing SSH integration or using the Git CLI. @@ -321,39 +325,69 @@ In general, we use an `expect` statement to check that something _is_ as we expe ```ruby Page::Project::Pipeline::Show.perform do |pipeline| - expect(pipeline).to have_job("a_job") + expect(pipeline).to have_job('a_job') end ``` -### Ensure `expect` checks for negation efficiently +### Create negatable matchers to speed `expect` checks However, sometimes we want to check that something is _not_ as we _don't_ want it to be. In other -words, we want to make sure something is absent. In such a case we should use an appropriate -predicate method that returns quickly, rather than waiting for a state that won't appear. +words, we want to make sure something is absent. For unit tests and feature specs, +we commonly use `not_to` +because RSpec's built-in matchers are negatable, as are Capybara's, which means the following two statements are +equivalent. + +```ruby +except(page).not_to have_text('hidden') +except(page).to have_no_text('hidden') +``` + +Unfortunately, that's not automatically the case for the predicate methods that we add to our +[page objects](page_objects.md). We need to [create our own negatable matchers](https://relishapp.com/rspec/rspec-expectations/v/3-9/docs/custom-matchers/define-a-custom-matcher#matcher-with-separate-logic-for-expect().to-and-expect().not-to). + +The initial example uses the `have_job` matcher which is derived from the [`has_job?` predicate +method of the `Page::Project::Pipeline::Show` page object](https://gitlab.com/gitlab-org/gitlab/-/blob/87864b3047c23b4308f59c27a3757045944af447/qa/qa/page/project/pipeline/show.rb#L53). +To create a negatable matcher, we use `has_no_job?` for the negative case: + +```ruby +RSpec::Matchers.define :have_job do |job_name| + match do |page_object| + page_object.has_job?(job_name) + end + + match_when_negated do |page_object| + page_object.has_no_job?(job_name) + end +end +``` -It's most efficient to use a predicate method that returns immediately when there is no job, or waits -until it disappears: +And then the two `expect` statements in the following example are equivalent: ```ruby -# Good Page::Project::Pipeline::Show.perform do |pipeline| - expect(pipeline).to have_no_job("a_job") + expect(pipeline).not_to have_job('a_job') + expect(pipeline).to have_no_job('a_job') end ``` -### Problematic alternatives +[See this merge request for a real example of adding a custom matcher](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/46302). -Alternatively, if we want to check that a job doesn't exist it might be tempting to use `not_to`: +NOTE: **Note:** +We need to create custom negatable matchers only for the predicate methods we've added to the test framework, and only if we're using `not_to`. If we use `to have_no_*` a negatable matcher is not necessary. + +### Why we need negatable matchers + +Consider the following code, but assume that we _don't_ have a custom negatable matcher for `have_job`. ```ruby # Bad Page::Project::Pipeline::Show.perform do |pipeline| - expect(pipeline).not_to have_job("a_job") + expect(pipeline).not_to have_job('a_job') end ``` -For this statement to pass, `have_job("a_job")` has to return `false` so that `not_to` can negate it. -The problem is that `have_job("a_job")` waits up to ten seconds for `"a job"` to appear before +For this statement to pass, `have_job('a_job')` has to return `false` so that `not_to` can negate it. +The problem is that `have_job('a_job')` waits up to ten seconds for `'a job'` to appear before returning `false`. Under the expected condition this test will take ten seconds longer than it needs to. Instead, we could force no wait: @@ -361,9 +395,13 @@ Instead, we could force no wait: ```ruby # Not as bad but potentially flaky Page::Project::Pipeline::Show.perform do |pipeline| - expect(pipeline).not_to have_job("a_job", wait: 0) + expect(pipeline).not_to have_job('a_job', wait: 0) end ``` -The problem is that if `"a_job"` is present and we're waiting for it to disappear, this statement -will fail. +The problem is that if `'a_job'` is present and we're waiting for it to disappear, this statement will fail. + +Neither problem is present if we create a custom negatable matcher because the `has_no_job?` predicate method +would be used, which would wait only as long as necessary for the job to disappear. + +Lastly, negatable matchers are preferred over using matchers of the form `have_no_*` because it's a common and familiar practice to negate matchers using `not_to`. If we facilitate that practice by adding negatable matchers, we make it easier for subsequent test authors to write efficient tests. diff --git a/doc/development/testing_guide/end_to_end/dynamic_element_validation.md b/doc/development/testing_guide/end_to_end/dynamic_element_validation.md index 32b1c304a9a..871b3f80c18 100644 --- a/doc/development/testing_guide/end_to_end/dynamic_element_validation.md +++ b/doc/development/testing_guide/end_to_end/dynamic_element_validation.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Dynamic Element Validation We devised a solution to solve common test automation problems such as the dreaded `NoSuchElementException`. diff --git a/doc/development/testing_guide/end_to_end/environment_selection.md b/doc/development/testing_guide/end_to_end/environment_selection.md index 325f251b280..f5e3e99b79e 100644 --- a/doc/development/testing_guide/end_to_end/environment_selection.md +++ b/doc/development/testing_guide/end_to_end/environment_selection.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Environment selection Some tests are designed to be run against specific environments or [pipelines](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#scheduled-qa-test-pipelines). @@ -42,14 +48,13 @@ RSpec.describe 'Area' do it 'runs in dev environment', only: { tld: '.org', domain: 'gitlab', subdomain: 'dev' } do; end it 'runs in prod and staging environments', only: { subdomain: /(staging.)?/, domain: 'gitlab' } {} - + it 'runs only in nightly pipeline', only: { pipeline: :nightly } do; end - + it 'runs in nightly and canary pipelines', only: { pipeline: [:nightly, :canary] } do; end end ``` -NOTE: **Note:** If the test has a `before` or `after`, you must add the `only` metadata to the outer `RSpec.describe`. diff --git a/doc/development/testing_guide/end_to_end/feature_flags.md b/doc/development/testing_guide/end_to_end/feature_flags.md index e571774167d..2ff1c9f6dc3 100644 --- a/doc/development/testing_guide/end_to_end/feature_flags.md +++ b/doc/development/testing_guide/end_to_end/feature_flags.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Testing with feature flags To run a specific test with a feature flag enabled you can use the `QA::Runtime::Feature` class to diff --git a/doc/development/testing_guide/end_to_end/flows.md b/doc/development/testing_guide/end_to_end/flows.md index fb1d82914aa..291d8bd5319 100644 --- a/doc/development/testing_guide/end_to_end/flows.md +++ b/doc/development/testing_guide/end_to_end/flows.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Flows in GitLab QA Flows are frequently used sequences of actions. They are a higher level diff --git a/doc/development/testing_guide/end_to_end/index.md b/doc/development/testing_guide/end_to_end/index.md index f61eab5c8f3..1d144359ed8 100644 --- a/doc/development/testing_guide/end_to_end/index.md +++ b/doc/development/testing_guide/end_to_end/index.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # End-to-end Testing ## What is end-to-end testing? diff --git a/doc/development/testing_guide/end_to_end/page_objects.md b/doc/development/testing_guide/end_to_end/page_objects.md index 6ce44b2d359..7eacaf4b08a 100644 --- a/doc/development/testing_guide/end_to_end/page_objects.md +++ b/doc/development/testing_guide/end_to_end/page_objects.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Page objects in GitLab QA In GitLab QA we are using a known pattern, called _Page Objects_. diff --git a/doc/development/testing_guide/end_to_end/resources.md b/doc/development/testing_guide/end_to_end/resources.md index b8a093c54c6..d73bae331d5 100644 --- a/doc/development/testing_guide/end_to_end/resources.md +++ b/doc/development/testing_guide/end_to_end/resources.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Resource class in GitLab QA Resources are primarily created using Browser UI steps, but can also diff --git a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md index 3a1303d9c0c..8e99cf18ea0 100644 --- a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md +++ b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # RSpec metadata for end-to-end tests This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec-core/docs/metadata/user-defined-metadata) @@ -10,7 +16,7 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec | `:elasticsearch` | The test requires an Elasticsearch service. It is used by the [instance-level scenario](https://gitlab.com/gitlab-org/gitlab-qa#definitions) [`Test::Integration::Elasticsearch`](https://gitlab.com/gitlab-org/gitlab/-/blob/72b62b51bdf513e2936301cb6c7c91ec27c35b4d/qa/qa/ee/scenario/test/integration/elasticsearch.rb) to include only tests that require Elasticsearch. | | `:gitaly_cluster` | The test will run against a GitLab instance where repositories are stored on redundant Gitaly nodes behind a Praefect node. All nodes are [separate containers](../../../administration/gitaly/praefect.md#requirements-for-configuring-a-gitaly-cluster). Tests that use this tag have a longer setup time since there are three additional containers that need to be started. | | `:jira` | The test requires a Jira Server. [GitLab-QA](https://gitlab.com/gitlab-org/gitlab-qa) will provision the Jira Server in a Docker container when the `Test::Integration::Jira` test scenario is run. -| `:kubernetes` | The test includes a GitLab instance that is configured to be run behind an SSH tunnel, allowing a TLS-accessible GitLab. This test will also include provisioning of at least one Kubernetes cluster to test against. *This tag is often be paired with `:orchestrated`.* | +| `:kubernetes` | The test includes a GitLab instance that is configured to be run behind an SSH tunnel, allowing a TLS-accessible GitLab. This test will also include provisioning of at least one Kubernetes cluster to test against. _This tag is often be paired with `:orchestrated`._ | | `:only` | The test is only to be run against specific environments or pipelines. See [Environment selection](environment_selection.md) for more information. | | `:orchestrated` | The GitLab instance under test may be [configured by `gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#orchestrated-tests) to be different to the default GitLab configuration, or `gitlab-qa` may launch additional services in separate Docker containers, or both. Tests tagged with `:orchestrated` are excluded when testing environments where we can't dynamically modify GitLab's configuration (for example, Staging). | | `:quarantine` | The test has been [quarantined](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests), will run in a separate job that only includes quarantined tests, and is allowed to fail. The test will be skipped in its regular job so that if it fails it will not hold up the pipeline. Note that you can also [quarantine a test only when it runs against specific environment](environment_selection.md#quarantining-a-test-for-a-specific-environment). | @@ -19,3 +25,20 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec | `:runner` | The test depends on and will set up a GitLab Runner instance, typically to run a pipeline. | | `:skip_live_env` | The test will be excluded when run against live deployed environments such as Staging, Canary, and Production. | | `:testcase` | The link to the test case issue in the [Quality Testcases project](https://gitlab.com/gitlab-org/quality/testcases/). | +| `:mattermost` | The test requires a GitLab Mattermost service on the GitLab instance. | +| `:ldap_no_server` | The test requires a GitLab instance to be configured to use LDAP. To be used with the `:orchestrated` tag. It does not spin up an LDAP server at orchestration time. Instead, it creates the LDAP server at runtime. | +| `:ldap_no_tls` | The test requires a GitLab instance to be configured to use an external LDAP server with TLS not enabled. | +| `:ldap_tls` | The test requires a GitLab instance to be configured to use an external LDAP server with TLS enabled. | +| `:object_storage` | The test requires a GitLab instance to be configured to use multiple [object storage types](../../../administration/object_storage.md). Uses MinIO as the object storage server. | +| `:smtp` | The test requires a GitLab instance to be configured to use an SMTP server. Tests SMTP notification email delivery from GitLab by using MailHog. | +| `:group_saml` | The test requires a GitLab instance that has SAML SSO enabled at the group level. Interacts with an external SAML identity provider. Paired with the `:orchestrated` tag. | +| `:instance_saml` | The test requires a GitLab instance that has SAML SSO enabled at the instance level. Interacts with an external SAML identity provider. Paired with the `:orchestrated` tag. | +| `:skip_signup_disabled` | The test uses UI to sign up a new user and will be skipped in any environment that does not allow new user registration via the UI. | +| `:smoke` | The test belongs to the test suite which verifies basic functionality of a GitLab instance.| +| `:github` | The test requires a GitHub personal access token. | +| `:repository_storage` | The test requires a GitLab instance to be configured to use multiple [repository storage paths](../../../administration/repository_storage_paths.md). Paired with the `:orchestrated` tag. | +| `:geo` | The test requires two GitLab Geo instances - a primary and a secondary - to be spun up. | +| `:relative_url` | The test requires a GitLab instance to be installed under a [relative URL](../../../install/relative_url.md). | +| `:requires_git_protocol_v2` | The test requires that Git protocol version 2 is enabled on the server. It's assumed to be enabled by default but if not the test can be skipped by setting `QA_CAN_TEST_GIT_PROTOCOL_V2` to `false`. | +| `:requires_praefect` | The test requires that the GitLab instance uses [Gitaly Cluster](../../../administration/gitaly/praefect.md) (a.k.a. Praefect) as the repository storage . It's assumed to be used by default but if not the test can be skipped by setting `QA_CAN_TEST_PRAEFECT` to `false`. | +| `:packages` | The test requires a GitLab instance that has the [Package Registry](../../../administration/packages/#gitlab-package-registry-administration) enabled. | diff --git a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md index 658839fcf96..df4b833526c 100644 --- a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md +++ b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Running tests that require special setup ## Jenkins spec @@ -232,7 +238,11 @@ run: ./services/tunnel_gitlab: (pid 2650) 10875s, normally down; run: log: (pid run: ./services/tunnel_registry: (pid 2651) 10875s, normally down; run: log: (pid 65155) 177993s ``` -Also, restarting Docker and then, on the Terminal, issue the command `docker login https://[YOUR-REGISTRY-PORT].qa-tunnel.gitlab.info:443` and use the GDK credentials to login. Note that the Registry port and GDK port are not the same. When configuring Auto DevOps in GDK, the `gdk reconfigure` command outputs the port of the Registry: +Also, restarting Docker and then, on the Terminal, issue the command +`docker login https://[YOUR-REGISTRY-PORT].qa-tunnel.gitlab.info:443` and use +the GDK credentials to sign in. Note that the Registry port and GDK port aren't +the same. When configuring Auto DevOps in GDK, the `gdk reconfigure` command +outputs the port of the Registry: ```shell ********************************************* diff --git a/doc/development/testing_guide/end_to_end/style_guide.md b/doc/development/testing_guide/end_to_end/style_guide.md index 645c2633831..e6c96bca93e 100644 --- a/doc/development/testing_guide/end_to_end/style_guide.md +++ b/doc/development/testing_guide/end_to_end/style_guide.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Style guide for writing end-to-end tests This document describes the conventions used at GitLab for writing End-to-end (E2E) tests using the GitLab QA project. diff --git a/doc/development/testing_guide/flaky_tests.md b/doc/development/testing_guide/flaky_tests.md index 7aed908c4f6..47ed11d76a2 100644 --- a/doc/development/testing_guide/flaky_tests.md +++ b/doc/development/testing_guide/flaky_tests.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Flaky tests ## What's a flaky test? @@ -12,7 +18,13 @@ When a test frequently fails in `master`, should be created. If the test cannot be fixed in a timely fashion, there is an impact on the productivity of all the developers, so it should be placed in quarantine by -assigning the `:quarantine` metadata. +assigning the `:quarantine` metadata with the issue URL. + +```ruby +it 'should succeed', quarantine: 'https://gitlab.com/gitlab-org/gitlab/-/issues/12345' do + expect(response).to have_gitlab_http_status(:ok) +end +``` This means it will be skipped unless run with `--tag quarantine`: diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md index 730f8d5ad7d..28fe63f1fb4 100644 --- a/doc/development/testing_guide/frontend_testing.md +++ b/doc/development/testing_guide/frontend_testing.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Frontend testing standards and style guidelines There are two types of test suites you'll encounter while developing frontend code @@ -24,7 +30,6 @@ We have started to migrate frontend tests to the [Jest](https://jestjs.io) testi Jest tests can be found in `/spec/frontend` and `/ee/spec/frontend` in EE. -NOTE: **Note:** Most examples have a Jest and Karma example. See the Karma examples only as explanation to what's going on in the code, should you stumble over some use cases during your discovery. The Jest examples are the one you should follow. ## Karma test suite @@ -110,6 +115,37 @@ describe('Component', () => { Remember that the performance of each test depends on the environment. +### Timout error due to async components + +If your component is fetching some other components asynchroneously based on some conditions, it might happen so that your Jest suite for this component will become flaky timing out from time to time. + +```javascript +// ide.vue +export default { + components: { + 'error-message': () => import('./error_message.vue'), + 'gl-button': () => import('@gitlab/ui/src/components/base/button/button.vue'), + ... +}; +``` + +To address this issue, you can "help" Jest by stubbing the async components so that Jest would not need to fetch those asynchroneously at the run-time. + +```javascript +// ide_spec.js +import { GlButton } from '@gitlab/ui'; +import ErrorMessage from '~/ide/components/error_message.vue'; +... +return shallowMount(ide, { + ... + stubs: { + ErrorMessage, + GlButton, + ... + }, +}) +``` + ## What and how to test Before jumping into more gritty details about Jest-specific workflows like mocks and spies, we should briefly cover what to test with Jest. @@ -198,47 +234,33 @@ Following you'll find some general common practices you will find as part of our When it comes to querying DOM elements in your tests, it is best to uniquely and semantically target the element. -Preferentially, this is done by targeting text the user actually sees using [DOM Testing Library](https://testing-library.com/docs/dom-testing-library/intro). +Preferentially, this is done by targeting what the user actually sees using [DOM Testing Library](https://testing-library.com/docs/dom-testing-library/intro). When selecting by text it is best to use [`getByRole` or `findByRole`](https://testing-library.com/docs/dom-testing-library/api-queries#byrole) as these enforce accessibility best practices as well. The examples below demonstrate the order of preference. -Sometimes this cannot be done feasibly. In these cases, adding test attributes to simplify the -selectors might be the best option. +When writing Vue component unit tests, it can be wise to query children by component, so that the unit test can focus on comprehensive value coverage +rather than dealing with the complexity of a child component's behavior. + +Sometimes, neither of the above are feasible. In these cases, adding test attributes to simplify the selectors might be the best option. A list of +possible selectors include: - A semantic attribute like `name` (also verifies that `name` was setup properly) - A `data-testid` attribute ([recommended by maintainers of `@vue/test-utils`](https://github.com/vuejs/vue-test-utils/issues/1498#issuecomment-610133465)) - a Vue `ref` (if using `@vue/test-utils`) ```javascript -import { mount, shallowMount } from '@vue/test-utils' import { getByRole, getByText } from '@testing-library/dom' -let wrapper -let el - -const createComponent = (mountFn = shallowMount) => { - wrapper = mountFn(Component) - el = wrapper.vm.$el // reference to the container element -} - -beforeEach(() => { - createComponent() -}) - - +// In this example, `wrapper` is a `@vue/test-utils` wrapper returned from `mount` or `shallowMount`. it('exists', () => { - // Best - - // NOTE: both mount and shallowMount work as long as a DOM element is available - // Finds a properly formatted link with an accessible name of "Click Me" - getByRole(el, 'link', { name: /Click Me/i }) - getByRole(el, 'link', { name: 'Click Me' }) - // Finds any element with the text "Click Me" - getByText(el, 'Click Me') - // Regex is also available - getByText(el, /Click Me/i) - - // Good + // Best (especially for integration tests) + getByRole(wrapper.element, 'link', { name: /Click Me/i }) + getByRole(wrapper.element, 'link', { name: 'Click Me' }) + getByText(wrapper.element, 'Click Me') + getByText(wrapper.element, /Click Me/i) + + // Good (especially for unit tests) + wrapper.find(FooComponent); wrapper.find('input[name=foo]'); wrapper.find('[data-testid="foo"]'); wrapper.find({ ref: 'foo'}); @@ -249,14 +271,6 @@ it('exists', () => { wrapper.find('.qa-foo-component'); wrapper.find('[data-qa-selector="foo"]'); }); - -// Good -it('exists', () => { - wrapper.find(FooComponent); - wrapper.find('input[name=foo]'); - wrapper.find('[data-testid="foo"]'); - wrapper.find({ ref: 'foo'}); -}); ``` It is not recommended that you add `.js-*` classes just for testing purposes. Only do this if there are no other feasible options available. @@ -327,7 +341,6 @@ it('tests a promise rejection', async () => { You can also simply return a promise from the test function. -NOTE: **Note:** Using the `done` and `done.fail` callbacks is discouraged when working with promises. They should only be used when testing callback-based code. @@ -777,7 +790,7 @@ While you work on a test suite, you may want to run these specs in watch mode, s # Watch and rerun all specs matching the name icon yarn jest --watch icon -# Watch and rerun one specifc file +# Watch and rerun one specific file yarn jest --watch path/to/spec/file.spec.js ``` @@ -917,6 +930,32 @@ it.each([ ); ``` +**Note**: only use template literal block if pretty print is **not** needed for spec output. For example, empty strings, nested objects etc. + +For example, when testing the difference between an empty search string and a non-empty search string, the use of the array block syntax with the pretty print option would be preferred. That way the differences between an empty string e.g. `''` and a non-empty string e.g. `'search string'` would be visible in the spec output. Whereas with a template literal block, the empty string would be shown as a space, which could lead to a confusing developer experience + +```javascript +// bad +it.each` + searchTerm | expected + ${''} | ${{ issue: { users: { nodes: [] } } }} + ${'search term'} | ${{ issue: { other: { nested: [] } } }} +`('when search term is $searchTerm, it returns $expected', ({ searchTerm, expected }) => { + expect(search(searchTerm)).toEqual(expected) +}); + +// good +it.each([ + ['', { issue: { users: { nodes: [] } } }], + ['search term', { issue: { other: { nested: [] } } }], +])('when search term is %p, expect to return %p', + (searchTerm, expected) => { + expect(search(searchTerm)).toEqual(expected) + } +); + +``` + ```javascript // test suite with tagged template literal block describe.each` diff --git a/doc/development/testing_guide/index.md b/doc/development/testing_guide/index.md index a61a700594c..840c8c9206c 100644 --- a/doc/development/testing_guide/index.md +++ b/doc/development/testing_guide/index.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Testing standards and style guidelines This document describes various guidelines and best practices for automated diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md index 61d3299cabf..5dcc7e7091e 100644 --- a/doc/development/testing_guide/review_apps.md +++ b/doc/development/testing_guide/review_apps.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Review Apps Review Apps are automatically deployed by [the diff --git a/doc/development/testing_guide/smoke.md b/doc/development/testing_guide/smoke.md index 5a144b43478..76484dd193b 100644 --- a/doc/development/testing_guide/smoke.md +++ b/doc/development/testing_guide/smoke.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Smoke Tests It is imperative in any testing suite that we have Smoke Tests. In short, smoke diff --git a/doc/development/testing_guide/testing_levels.md b/doc/development/testing_guide/testing_levels.md index 88d7cf9ca08..d9e7edfa0c8 100644 --- a/doc/development/testing_guide/testing_levels.md +++ b/doc/development/testing_guide/testing_levels.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Testing levels  diff --git a/doc/development/testing_guide/testing_migrations_guide.md b/doc/development/testing_guide/testing_migrations_guide.md index a5bcb651d71..b944c7ed406 100644 --- a/doc/development/testing_guide/testing_migrations_guide.md +++ b/doc/development/testing_guide/testing_migrations_guide.md @@ -1,4 +1,7 @@ --- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- @@ -236,6 +239,5 @@ describe Gitlab::BackgroundMigration::ArchiveLegacyTraces, schema: 2018052915262 end ``` -NOTE: **Note:** These tests do not run within a database transaction, as we use a deletion database cleanup strategy. Do not depend on a transaction being present. diff --git a/doc/development/testing_guide/testing_rake_tasks.md b/doc/development/testing_guide/testing_rake_tasks.md index db8ca87e9f8..384739d1adb 100644 --- a/doc/development/testing_guide/testing_rake_tasks.md +++ b/doc/development/testing_guide/testing_rake_tasks.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Testing Rake tasks To make testing Rake tasks a little easier, there is a helper that can be included diff --git a/doc/development/understanding_explain_plans.md b/doc/development/understanding_explain_plans.md index 797dfc676eb..896b34a4ab4 100644 --- a/doc/development/understanding_explain_plans.md +++ b/doc/development/understanding_explain_plans.md @@ -1,3 +1,9 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Understanding EXPLAIN plans PostgreSQL allows you to obtain query plans using the `EXPLAIN` command. This @@ -428,6 +434,17 @@ If there aren't any, check if you can perhaps slightly change an existing one to fit both the existing and new queries. Only add a new index if none of the existing indexes can be used in any way. +When comparing execution plans, don't take timing as the only important metric. +Good timing is the main goal of any optimization, but it can be too volatile to +be used for comparison (for example, it depends a lot on the state of cache). +When optimizing a query, we usually need to reduce the amount of data we're +dealing with. Indexes are the way to work with fewer pages (buffers) to get the +result, so, during optimization, look at the number of buffers used (read and hit), +and work on reducing these numbers. Reduced timing will be the consequence of reduced +buffer numbers. [#database-lab](#database-lab) guarantees that the plan is structurally +identical to production (and overall number of buffers is the same as on production), +but difference in cache state and I/O speed may lead to different timings. + ## Queries that can't be optimised Now that we have seen how to optimise a query, let's look at another query that diff --git a/doc/development/uploads.md b/doc/development/uploads.md index ee94553c200..0307ab76cd1 100644 --- a/doc/development/uploads.md +++ b/doc/development/uploads.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Uploads development documentation [GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) has special rules for handling uploads. @@ -214,7 +220,6 @@ In this setup, an extra Rails route must be implemented in order to handle autho and [its routes](https://gitlab.com/gitlab-org/gitlab/blob/cc723071ad337573e0360a879cbf99bc4fb7adb9/config/routes/git_http.rb#L31-32). - [API endpoints for uploading packages](packages.md#file-uploads). -Note: **Note:** This will fallback to _disk buffered upload_ when `direct_upload` is disabled inside the [object storage setting](../administration/uploads.md#object-storage-settings). The answer to the `/authorize` call will only contain a file system path. @@ -273,8 +278,10 @@ Uploads routes belong to one of these categories: 1. Rails controllers: uploads handled by Rails controllers. 1. Grape API: uploads handled by a Grape API endpoint. -1. GraphQL API: uploads handled by a GraphQL resolve function. In these cases, there is nothing else - to do apart from implementing the actual upload. +1. GraphQL API: uploads handled by a GraphQL resolve function. + +CAUTION: **Warning:** +GraphQL uploads do not support [direct upload](#direct-upload) yet. Depending on the use case, the feature may not work on installations without NFS (like GitLab.com or Kubernetes installations). Uploading to object storage inside the GraphQL resolve function may result in timeout errors. For more details please follow [issue #280819](https://gitlab.com/gitlab-org/gitlab/-/issues/280819). ### Update Workhorse for the new route diff --git a/doc/development/utilities.md b/doc/development/utilities.md index 52151d37038..aca14507fcd 100644 --- a/doc/development/utilities.md +++ b/doc/development/utilities.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # GitLab utilities We have developed a number of utilities to help ease development: diff --git a/doc/development/value_stream_analytics.md b/doc/development/value_stream_analytics.md index 70b16cc1739..4560a115562 100644 --- a/doc/development/value_stream_analytics.md +++ b/doc/development/value_stream_analytics.md @@ -1,3 +1,9 @@ +--- +stage: Manage +group: Optimize +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Value Stream Analytics development guide Value stream analytics calculates the time between two arbitrary events recorded on domain objects and provides aggregated statistics about the duration. @@ -53,8 +59,8 @@ def timestamp_projection end ``` -NOTE: **Note:** -More complex expressions are also possible (e.g. using `COALESCE`). Look at the existing event classes for examples. +More complex expressions are also possible (for example, using `COALESCE`). +Review the existing event classes for examples. In some cases, defining the `timestamp_projection` method is not enough. The calculation query should know which table contains the timestamp expression. Each `Event` class is responsible for making modifications to the calculation query to make the `timestamp_projection` work. This usually means joining an additional table. diff --git a/doc/development/verifying_database_capabilities.md b/doc/development/verifying_database_capabilities.md index f6c78e51299..fe60ec11bce 100644 --- a/doc/development/verifying_database_capabilities.md +++ b/doc/development/verifying_database_capabilities.md @@ -1,3 +1,9 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Verifying Database Capabilities Sometimes certain bits of code may only work on a certain database diff --git a/doc/development/what_requires_downtime.md b/doc/development/what_requires_downtime.md index 9063fb867e2..18a2e17967a 100644 --- a/doc/development/what_requires_downtime.md +++ b/doc/development/what_requires_downtime.md @@ -1,3 +1,9 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # What requires downtime? When working with a database certain operations can be performed without taking @@ -88,6 +94,8 @@ renaming. For example class RenameUsersUpdatedAtToUpdatedAtTimestamp < ActiveRecord::Migration[4.2] include Gitlab::Database::MigrationHelpers + DOWNTIME = false + disable_ddl_transaction! def up @@ -100,13 +108,12 @@ class RenameUsersUpdatedAtToUpdatedAtTimestamp < ActiveRecord::Migration[4.2] end ``` -This will take care of renaming the column, ensuring data stays in sync, copying -over indexes and foreign keys, etc. +This will take care of renaming the column, ensuring data stays in sync, and +copying over indexes and foreign keys. -NOTE: **Note:** -If a column contains 1 or more indexes that do not contain the name of -the original column, the above procedure will fail. In this case you will first -need to rename these indexes. +If a column contains one or more indexes that don't contain the name of the +original column, the previously described procedure will fail. In that case, +you'll first need to rename these indexes. ### Step 2: Add A Post-Deployment Migration @@ -131,7 +138,6 @@ class CleanupUsersUpdatedAtRename < ActiveRecord::Migration[4.2] end ``` -NOTE: **Note:** If you're renaming a [large table](https://gitlab.com/gitlab-org/gitlab/-/blob/master/rubocop/rubocop-migrations.yml#L3), please carefully consider the state when the first migration has run but the second cleanup migration hasn't been run yet. With [Canary](https://gitlab.com/gitlab-com/gl-infra/readiness/-/tree/master/library/canary/) it is possible that the system runs in this state for a significant amount of time. @@ -142,7 +148,7 @@ done without requiring downtime. However, this does require that any application changes are deployed _first_. Thus, changing the constraints of a column should happen in a post-deployment migration. -NOTE: Avoid using `change_column` as it produces an inefficient query because it re-defines +Avoid using `change_column` as it produces an inefficient query because it re-defines the whole column type. You can check the following guides for each specific use case: diff --git a/doc/development/wikis.md b/doc/development/wikis.md index 9a436762645..f8bcaac2ec0 100644 --- a/doc/development/wikis.md +++ b/doc/development/wikis.md @@ -1,8 +1,8 @@ --- type: reference, dev -stage: none -group: Development -info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +stage: Create +group: Knowledge +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers description: "GitLab's development guidelines for Wikis" --- diff --git a/doc/development/windows.md b/doc/development/windows.md index 3301e4f7c8f..2ca99508746 100644 --- a/doc/development/windows.md +++ b/doc/development/windows.md @@ -1,4 +1,7 @@ --- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference, howto --- |
