diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2020-09-19 01:45:44 +0000 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2020-09-19 01:45:44 +0000 |
| commit | 85dc423f7090da0a52c73eb66faf22ddb20efff9 (patch) | |
| tree | 9160f299afd8c80c038f08e1545be119f5e3f1e1 /doc/development | |
| parent | 15c2c8c66dbe422588e5411eee7e68f1fa440bb8 (diff) | |
| download | gitlab-ce-85dc423f7090da0a52c73eb66faf22ddb20efff9.tar.gz | |
Add latest changes from gitlab-org/gitlab@13-4-stable-ee
Diffstat (limited to 'doc/development')
76 files changed, 3729 insertions, 1170 deletions
diff --git a/doc/development/README.md b/doc/development/README.md index 74068db726c..abdd5c662f3 100644 --- a/doc/development/README.md +++ b/doc/development/README.md @@ -18,7 +18,7 @@ For information on how to install, configure, update, and upgrade your own GitLa ## Get started -- Set up GitLab's development environment with [GitLab Development Kit (GDK)](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/README.md) +- Set up GitLab's development environment with [GitLab Development Kit (GDK)](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/master/README.md) - [GitLab contributing guide](contributing/index.md) - [Issues workflow](contributing/issue_workflow.md) for more information on: - Issue tracker guidelines. @@ -57,6 +57,7 @@ Complementary reads: - [Generate a changelog entry with `bin/changelog`](changelog.md) - [Requesting access to Chatops on GitLab.com](chatops_on_gitlabcom.md#requesting-access) (for GitLab team members) - [Patch release process for developers](https://gitlab.com/gitlab-org/release/docs/blob/master/general/patch/process.md#process-for-developers) +- [Adding a new service component to GitLab](adding_service_component.md) ## UX and Frontend guides @@ -197,6 +198,7 @@ See [database guidelines](database/index.md). - [Defining relations between files using projections](projections.md) - [Reference processing](./reference_processing.md) - [Compatibility with multiple versions of the application running at the same time](multi_version_compatibility.md) +- [Features inside `.gitlab/`](./features_inside_dot_gitlab.md) ## Other GitLab Development Kit (GDK) guides diff --git a/doc/development/adding_database_indexes.md b/doc/development/adding_database_indexes.md index 7fe047b380b..03557491e68 100644 --- a/doc/development/adding_database_indexes.md +++ b/doc/development/adding_database_indexes.md @@ -121,3 +121,71 @@ may be affected by factors such as (but not limited to): In other words, this data is only reliable for a frequently used database with plenty of data and with as many GitLab features enabled (and being used) as possible. + +## Requirements for naming indexes + +Indexes with complex definitions need to be explicitly named rather than +relying on the implicit naming behavior of migration methods. In short, +that means you **must** provide an explicit name argument for an index +created with one or more of the following options: + +- `where` +- `using` +- `order` +- `length` +- `type` +- `opclass` + +### Considerations for index names + +Index names don't have any significance in the database, so they should +attempt to communicate intent to others. The most important rule to +remember is that generic names are more likely to conflict or be duplicated, +and should not be used. Some other points to consider: + +- For general indexes, use a template, like: `index_{table}_{column}_{options}`. +- For indexes added to solve a very specific problem, it may make sense + for the name to reflect their use. +- Identifiers in PostgreSQL have a maximum length of 63 bytes. +- Check `db/structure.sql` for conflicts and ideas. + +### Why explicit names are required + +As Rails is database agnostic, it generates an index name only +from the required options of all indexes: table name and column name(s). +For example, imagine the following two indexes are created in a migration: + +```ruby +def up + add_index :my_table, :my_column + + add_index :my_table, :my_column, where: 'my_column IS NOT NULL' +end +``` + +Creation of the second index would fail, because Rails would generate +the same name for both indexes. + +This is further complicated by the behavior of the `index_exists?` method. +It considers only the table name, column name(s) and uniqueness specification +of the index when making a comparison. Consider: + +```ruby +def up + unless index_exists?(:my_table, :my_column, where: 'my_column IS NOT NULL') + add_index :my_table, :my_column, where: 'my_column IS NOT NULL' + end +end +``` + +The call to `index_exists?` will return true if **any** index exists on +`:my_table` and `:my_column`, and index creation will be bypassed. + +The `add_concurrent_index` helper is a requirement for creating indexes +on populated tables. Since it cannot be used inside a transactional +migration, it has a built-in check that detects if the index already +exists. In the event a match is found, index creation is skipped. +Without an explicit name argument, Rails can return a false positive +for `index_exists?`, causing a required index to not be created +properly. By always requiring a name for certain types of indexes, the +chance of error is greatly reduced. diff --git a/doc/development/adding_service_component.md b/doc/development/adding_service_component.md new file mode 100644 index 00000000000..2801e27145d --- /dev/null +++ b/doc/development/adding_service_component.md @@ -0,0 +1,89 @@ +# Adding a new Service Component to GitLab + +The GitLab product is made up of several service components that run as independent system processes in communication with each other. These services can be run on the same instance, or spread across different instances. A list of the existing components can be found in the [GitLab architecture overview](architecture.md). + +## Integration phases + +The following outline re-uses the [maturity metric](https://about.gitlab.com/direction/maturity) naming as an example of the various phases of integrating a component. These are only loosely coupled to a components actual maturity, and are intended as a guide for implementation order (for example, a component does not need to be enabled by default to be Lovable, and being enabled by default does not on its own cause a component to be Lovable). + +- Proposed + - [Proposing a new component](#proposing-a-new-component) +- Minimal + - [Integrating a new service with GitLab](#integrating-a-new-service-with-gitlab) + - [Handling service dependencies](#handling-service-dependencies) +- Viable + - [Bundled with GitLab installations](#bundling-a-service-with-gitlab) + - [End-to-end testing in GitLab QA](testing_guide/end_to_end/beginners_guide.md) + - [Release management](#release-management) + - [Enabled on GitLab.com](feature_flags/controls.md#enabling-a-feature-for-gitlabcom) +- Complete + - [Configurable by the GitLab orchestrator](https://gitlab.com/gitlab-org/gitlab-orchestrator) +- Lovable + - Enabled by default for the majority of users + +## Proposing a new component + +The initial step for integrating a new component with GitLab starts with creating a [Feature proposal in the issue tracker](https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Feature%20proposal). + +Identify the [product category](https://about.gitlab.com/handbook/product/categories/) the component falls under and assign the Engineering Manager and Product Manager responsible for that category. + +The general steps for getting any GitLab feature from proposal to release can be found in the [Product development flow](https://about.gitlab.com/handbook/product-development-flow/). + +## Integrating a new service with GitLab + +Adding a new service follows the same [merge request workflow](contributing/merge_request_workflow.md) as other contributions, and must meet the same [completion criteria](contributing/merge_request_workflow.md#definition-of-done) and in addition needs to cover the following: + +- The [architecture component list](architecture.md#component-list) has been updated to include the service. +- Features provided by the component have been accepted into the [GitLab Product Direction](https://about.gitlab.com/direction/). +- Documentation is available and the support team has been made aware of the new component. + +**For services that can operate completely separate from GitLab:** + +The first iteration should be to add the ability to connect and use the service as an externally installed component. Often this involves providing settings in GitLab to connect to the service, or allow connections from it. And then shipping documentation on how to install and configure the service with GitLab. + +TIP: **Tip:** +[Elasticsearch](../integration/elasticsearch.md#installing-elasticsearch) is an example of a service that has been integrated this way. And many of the other services, including internal projects like Gitaly, started off as separately installed alternatives. + +**For services that depend on the existing GitLab codebase:** + +The first iteration should be opt-in, either through the `gitlab.yml` configuration or through [feature flags](feature_flags.md). For these types of services it is often necessary to [bundle the service and its dependencies with GitLab](#bundling-a-service-with-gitlab) as part of the initial integration. + +TIP: **Tip:** +[ActionCable](https://docs.gitlab.com/omnibus/settings/actioncable.html) is an example of a service that has been added this way. + +## Bundling a service with GitLab + +NOTE: **Note:** +Code shipped with GitLab needs to use a license approved by the Legal team. See the list of [existing approved licenses](https://about.gitlab.com/handbook/engineering/open-source/#using-open-source-libraries). + +NOTE: **Note:** +Notify the [Distribution team](https://about.gitlab.com/handbook/engineering/development/enablement/distribution/) when adding a new dependency that must be compiled. We must be able to compile the dependency on all supported platforms. + +New services to be bundled with GitLab need to be available in the following environments. + +**Dev environment** + +The first step of bundling a new service is to provide it in the development environment to engage in collaboration and feedback. + +- [Include in the GDK](https://gitlab.com/gitlab-org/gitlab-development-kit) +- [Include in the source install instructions](../install/installation.md) + +**Standard install methods** + +In order for a service to be bundled for end-users or GitLab.com, it needs to be included in the standard install methods: + +- [Included in the Omnibus package](https://gitlab.com/gitlab-org/omnibus-gitlab) +- [Included in the GitLab Helm charts](https://gitlab.com/gitlab-org/charts/gitlab) + +## Handling service dependencies + +Dependencies should be kept up to date and be tracked for security updates. For the Rails codebase, the JavaScript and Ruby dependencies are +scanned for vulnerabilities using GitLab [dependency scanning](../user/application_security/dependency_scanning/index.md). + +In addition, any system dependencies used in Omnibus packages or the Cloud Native images should be added to the [dependency update automation](https://about.gitlab.com/handbook/engineering/development/enablement/distribution/maintenance/dependencies.io.html#adding-new-dependencies). + +## Release management + +If the service component needs to be updated or released with the monthly GitLab release, then the component should be added to the [release tools automation](https://gitlab.com/gitlab-org/release-tools). This project is maintained by the [Delivery team](https://about.gitlab.com/handbook/engineering/infrastructure/team/delivery/). A list of the projects managed this way can be found in the [release tools project directory](https://about.gitlab.com/handbook/engineering/infrastructure/team/delivery/). + +For example, during the monthly GitLab release, the desired version of Gitaly, GitLab Workhorse, GitLab Shell, etc., need to synchronized through the various release pipelines. diff --git a/doc/development/api_graphql_styleguide.md b/doc/development/api_graphql_styleguide.md index bf2d6400f56..18fc0fb7d33 100644 --- a/doc/development/api_graphql_styleguide.md +++ b/doc/development/api_graphql_styleguide.md @@ -14,7 +14,7 @@ which is exposed as an API endpoint at `/api/graphql`. In March 2019, Nick Thomas hosted a Deep Dive (GitLab team members only: `https://gitlab.com/gitlab-org/create-stage/issues/1`) on GitLab's [GraphQL API](../api/graphql/index.md) to share his domain specific knowledge -with anyone who may work in this part of the code base in the future. You can find the +with anyone who may work in this part of the codebase in the future. You can find the [recording on YouTube](https://www.youtube.com/watch?v=-9L_1MWrjkg), and the slides on [Google Slides](https://docs.google.com/presentation/d/1qOTxpkTdHIp1CRjuTvO-aXg0_rUtzE3ETfLUdnBB5uQ/edit) and in [PDF](https://gitlab.com/gitlab-org/create-stage/uploads/8e78ea7f326b2ef649e7d7d569c26d56/GraphQL_Deep_Dive__Create_.pdf). @@ -33,7 +33,7 @@ Authentication happens through the `GraphqlController`, right now this uses the same authentication as the Rails application. So the session can be shared. -It is also possible to add a `private_token` to the querystring, or +It's also possible to add a `private_token` to the query string, or add a `HTTP_PRIVATE_TOKEN` header. ## Global IDs @@ -41,7 +41,7 @@ add a `HTTP_PRIVATE_TOKEN` header. GitLab's GraphQL API uses Global IDs (i.e: `"gid://gitlab/MyObject/123"`) and never database primary key IDs. -Global ID is [a standard](https://graphql.org/learn/global-object-identification/) +Global ID is [a convention](https://graphql.org/learn/global-object-identification/) used for caching and fetching in client-side libraries. See also: @@ -75,7 +75,7 @@ The `iid`, `title` and `description` are _scalar_ GraphQL types. When exposing a model through the GraphQL API, we do so by creating a new type in `app/graphql/types`. You can also declare custom GraphQL data types -for scalar data types (e.g. `TimeType`). +for scalar data types (for example `TimeType`). When exposing properties in a type, make sure to keep the logic inside the definition as minimal as possible. Instead, consider moving any @@ -142,7 +142,10 @@ def reply_id end ``` -### Connection Types +### Connection types + +TIP: **Tip:** +For specifics on implementation, see [Pagination implementation](#pagination-implementation). GraphQL uses [cursor based pagination](https://graphql.org/learn/pagination/#pagination-and-edges) @@ -363,16 +366,16 @@ def foo end ``` -## Deprecating fields +## Deprecating fields and enum values GitLab's GraphQL API is versionless, which means we maintain backwards compatibility with older versions of the API with every change. Rather -than removing a field, we need to _deprecate_ the field instead. In -future, GitLab -[may remove deprecated fields](https://gitlab.com/gitlab-org/gitlab/-/issues/32292). +than removing a field or [enum value](#enums), we need to _deprecate_ it instead. +In future, GitLab +[may remove deprecated parts of the schema](https://gitlab.com/gitlab-org/gitlab/-/issues/32292). -Fields are deprecated using the `deprecated` property. The value -of the property is a `Hash` of: +Fields and enum values are deprecated using the `deprecated` property. +The value of the property is a `Hash` of: - `reason` - Reason for the deprecation. - `milestone` - Milestone that the field was deprecated. @@ -385,13 +388,14 @@ field :token, GraphQL::STRING_TYPE, null: true, description: 'Token for login' ``` -The original `description:` of the field should be maintained, and should -_not_ be updated to mention the deprecation. +The original `description` of the things being deprecated should be maintained, +and should _not_ be updated to mention the deprecation. Instead, the `reason` will +be appended to the `description`. ### Deprecation reason style guide -Where the reason for deprecation is due to the field being replaced -with another field, the `reason` must be: +Where the reason for deprecation is due to the field or enum value being +replaced, the `reason` must be: ```plaintext Use `otherFieldName` @@ -405,9 +409,22 @@ field :designs, ::Types::DesignManagement::DesignCollectionType, null: true, description: 'The designs associated with this issue', ``` +```ruby +module Types + class TodoStateEnum < BaseEnum + value 'pending', deprecated: { reason: 'Use PENDING', milestone: '10.0' } + value 'done', deprecated: { reason: 'Use DONE', milestone: '10.0' } + value 'PENDING', value: 'pending' + value 'DONE', value: 'done' + end +end +``` + If the field is not being replaced by another field, a descriptive deprecation `reason` should be given. +See also [Aliasing and deprecating mutations](#aliasing-and-deprecating-mutations). + ## Enums GitLab GraphQL enums are defined in `app/graphql/types`. When defining new enums, the @@ -452,6 +469,9 @@ module Types end ``` +Enum values can be deprecated using the +[`deprecated` keyword](#deprecating-fields-and-enum-values). + ## JSON When data to be returned by GraphQL is stored as @@ -760,6 +780,44 @@ to advertise the need for lookahead: For an example of real world use, please see [`ResolvesMergeRequests`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/graphql/resolvers/concerns/resolves_merge_requests.rb). +## Pass a parent object into a child Presenter + +Sometimes you need to access the resolved query parent in a child context to compute fields. Usually the parent is only +available in the `Resolver` class as `parent`. + +To find the parent object in your `Presenter` class: + +1. Add the parent object to the GraphQL `context` from within your resolver's `resolve` method: + + ```ruby + def resolve(**args) + context[:parent_object] = parent + end + ``` + +1. Declare that your fields require the `parent` field context. For example: + + ```ruby + # in ChildType + field :computed_field, SomeType, null: true, + method: :my_computing_method, + extras: [:parent], # Necessary + description: 'My field description' + ``` + +1. Declare your field's method in your Presenter class and have it accept the `parent` keyword argument. +This argument contains the parent **GraphQL context**, so you have to access the parent object with +`parent[:parent_object]` or whatever key you used in your `Resolver`: + + ```ruby + # in ChildPresenter + def my_computing_method(parent:) + # do something with `parent[:parent_object]` here + end + ``` + +For an example of real-world use, check [this MR that added `scopedPath` and `scopedUrl` to `IterationPresenter`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/39543) + ## Mutations Mutations are used to change any stored values, or to trigger @@ -1114,7 +1172,8 @@ mount_aliased_mutation 'BarMutation', Mutations::FooMutation ``` This allows us to rename a mutation and continue to support the old name, -when coupled with the [`deprecated`](#deprecating-fields) argument. +when coupled with the [`deprecated`](#deprecating-fields-and-enum-values) +argument. Example: @@ -1130,6 +1189,10 @@ tested for within the unit test of `Types::MutationType`. The merge request can be referred to as an example of this, including the method of testing deprecated aliased mutations. +## Pagination implementation + +To learn more, visit [GraphQL pagination](graphql_guide/pagination.md). + ## Validating arguments For validations of single arguments, use the @@ -1199,6 +1262,9 @@ Using the `GraphqlHelpers#all_graphql_fields_for`-helper, a query including all available fields can be constructed. This makes it easy to add a test rendering all possible fields for a query. +If you're adding a field to a query that supports pagination and sorting, +visit [Testing](graphql_guide/pagination.md#testing) for details. + To test GraphQL mutation requests, `GraphqlHelpers` provides 2 helpers: `graphql_mutation` which takes the name of the mutation, and a hash with the input for the mutation. This will return a struct with @@ -1285,7 +1351,7 @@ end More about complexity: [GraphQL Ruby documentation](https://graphql-ruby.org/queries/complexity_and_depth.html). -## Documentation and Schema +## Documentation and schema Our schema is located at `app/graphql/gitlab_schema.rb`. See the [schema reference](../api/graphql/reference/index.md) for details. diff --git a/doc/development/api_styleguide.md b/doc/development/api_styleguide.md index 06b05f49b12..400752c69e9 100644 --- a/doc/development/api_styleguide.md +++ b/doc/development/api_styleguide.md @@ -13,10 +13,12 @@ Always use an [Entity](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/api/ ## Documentation -API endpoints must come with [documentation](documentation/styleguide.md#restful-api), unless it is internal or behind a feature flag. +Each new or updated API endpoint must come with documentation, unless it is internal or behind a feature flag. The docs should be in the same merge request, or, if strictly necessary, in a follow-up with the same milestone as the original merge request. +See the [Documentation Style Guide RESTful API section](documentation/styleguide.md#restful-api) for details on documenting API resources in Markdown as well as in OpenAPI definition files. + ## Methods and parameters description Every method must be described using the [Grape DSL](https://github.com/ruby-grape/grape#describing-methods) @@ -173,7 +175,7 @@ guide on how you can add a new custom validator. validates the parameter value for different cases. Mainly, it checks whether a path is relative and does it contain `../../` relative traversal using `File::Separator` or not, and whether the path is absolute, for example - `/etc/passwd/`. By default, absolute paths are not allowed. However, you can optionally pass in an allowlist for allowed absolute paths in the following way: + `/etc/passwd/`. By default, absolute paths are not allowed. However, you can optionally pass in an allowlist for allowed absolute paths in the following way: `requires :file_path, type: String, file_path: { allowlist: ['/foo/bar/', '/home/foo/', '/app/home'] }` - `Git SHA`: @@ -247,7 +249,7 @@ most basic entity, with successive entities building upon that scope. The `with_api_entity_associations` scope will also [automatically preload data](https://gitlab.com/gitlab-org/gitlab/blob/19f74903240e209736c7668132e6a5a735954e7c/app%2Fmodels%2Ftodo.rb#L34) -for `Todo` _targets_ when returned in the Todos API. +for `Todo` _targets_ when returned in the [to-dos API](../api/todos.md). For more context and discussion about preloading see [this merge request](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/25711) diff --git a/doc/development/application_limits.md b/doc/development/application_limits.md index 4d296451add..f96ed2e7f57 100644 --- a/doc/development/application_limits.md +++ b/doc/development/application_limits.md @@ -17,50 +17,50 @@ limits](https://about.gitlab.com/handbook/product/product-processes/#introducing ### Insert database plan limits -In the `plan_limits` table, you have to create a new column and insert the -limit values. It's recommended to create separate migration script files. - -1. Add new column to the `plan_limits` table with non-null default value - that represents desired limit, such as: - - ```ruby - add_column(:plan_limits, :project_hooks, :integer, default: 100, null: false) - ``` - - NOTE: **Note:** - Plan limits entries set to `0` mean that limits are not - enabled. You should use this setting only in special and documented circumstances. - -1. (Optionally) Create the database migration that fine-tunes each level with - a desired limit using `create_or_update_plan_limit` migration helper, such as: - - ```ruby - class InsertProjectHooksPlanLimits < ActiveRecord::Migration[5.2] - include Gitlab::Database::MigrationHelpers - - DOWNTIME = false - - def up - create_or_update_plan_limit('project_hooks', 'default', 0) - create_or_update_plan_limit('project_hooks', 'free', 10) - create_or_update_plan_limit('project_hooks', 'bronze', 20) - create_or_update_plan_limit('project_hooks', 'silver', 30) - create_or_update_plan_limit('project_hooks', 'gold', 100) - end - - def down - create_or_update_plan_limit('project_hooks', 'default', 0) - create_or_update_plan_limit('project_hooks', 'free', 0) - create_or_update_plan_limit('project_hooks', 'bronze', 0) - create_or_update_plan_limit('project_hooks', 'silver', 0) - create_or_update_plan_limit('project_hooks', 'gold', 0) - end - end - ``` - -NOTE: **Note:** -Some plans exist only on GitLab.com. This will be no-op -for plans that do not exist. +In the `plan_limits` table, create a new column and insert the limit values. +It's recommended to create two separate migration script files. + +1. Add a new column to the `plan_limits` table with non-null default value that + represents desired limit, such as: + + ```ruby + add_column(:plan_limits, :project_hooks, :integer, default: 100, null: false) + ``` + + NOTE: **Note:** + Plan limits entries set to `0` mean that limits are not enabled. You should + use this setting only in special and documented circumstances. + +1. (Optionally) Create the database migration that fine-tunes each level with a + desired limit using `create_or_update_plan_limit` migration helper, such as: + + ```ruby + class InsertProjectHooksPlanLimits < ActiveRecord::Migration[5.2] + include Gitlab::Database::MigrationHelpers + + DOWNTIME = false + + def up + create_or_update_plan_limit('project_hooks', 'default', 0) + create_or_update_plan_limit('project_hooks', 'free', 10) + create_or_update_plan_limit('project_hooks', 'bronze', 20) + create_or_update_plan_limit('project_hooks', 'silver', 30) + create_or_update_plan_limit('project_hooks', 'gold', 100) + end + + def down + create_or_update_plan_limit('project_hooks', 'default', 0) + create_or_update_plan_limit('project_hooks', 'free', 0) + create_or_update_plan_limit('project_hooks', 'bronze', 0) + create_or_update_plan_limit('project_hooks', 'silver', 0) + create_or_update_plan_limit('project_hooks', 'gold', 0) + end + end + ``` + + NOTE: **Note:** + Some plans exist only on GitLab.com. This will be a no-op for plans + that do not exist. ### Plan limits validation diff --git a/doc/development/approval_rules.md b/doc/development/approval_rules.md index 65df82721de..f295c20a36f 100644 --- a/doc/development/approval_rules.md +++ b/doc/development/approval_rules.md @@ -81,7 +81,7 @@ The `ApprovalState` model get these records when approval rules are not overwritten. The `protected_branches` attribute is set and used when a rule is scoped to -protected branches. See [Scoped to Protected Branch doc](../user/project/merge_requests/merge_request_approvals.md#scoped-to-protected-branch-premium) +protected branches. See [Scoped to Protected Branch doc](../user/project/merge_requests/merge_request_approvals.md#scoped-to-protected-branch) for more information about the feature. ### `ApprovalMergeRequestRule` diff --git a/doc/development/architecture.md b/doc/development/architecture.md index 963e1e618a1..f6b1c8cd914 100644 --- a/doc/development/architecture.md +++ b/doc/development/architecture.md @@ -173,7 +173,7 @@ Table description links: | [Gitaly](#gitaly) | Git RPC service for handling all Git calls made by GitLab | ✅ | ✅ | ✅ | ✅ | ⚙ | ✅ | CE & EE | | [GitLab Exporter](#gitlab-exporter) | Generates a variety of GitLab metrics | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | CE & EE | | [GitLab Geo Node](#gitlab-geo) | Geographically distributed GitLab nodes | ⚙ | ⚙ | ❌ | ✅ | ❌ | ⚙ | EE Only | -| [GitLab Managed Apps](#gitlab-managed-apps) | Deploy Helm, Ingress, Cert-Manager, Prometheus, a Runner, JupyterHub, or Knative to a cluster | ⤓ | ⤓ | ⤓ | ⤓ | ⤓ | ⤓ | CE & EE | +| [GitLab Managed Apps](#gitlab-managed-apps) | Deploy Helm, Ingress, Cert-Manager, Prometheus, GitLab Runner, JupyterHub, or Knative to a cluster | ⤓ | ⤓ | ⤓ | ⤓ | ⤓ | ⤓ | CE & EE | | [GitLab Pages](#gitlab-pages) | Hosts static websites | ⚙ | ❌ | ❌ | ✅ | ⚙ | ⚙ | CE & EE | | [GitLab self-monitoring: Alertmanager](#alertmanager) | Deduplicates, groups, and routes alerts from Prometheus | ⚙ | ✅ | ⚙ | ✅ | ❌ | ❌ | CE & EE | | [GitLab self-monitoring: Grafana](#grafana) | Metrics dashboard | ✅ | ⚙ | ⤓ | ✅ | ❌ | ❌ | CE & EE | @@ -259,7 +259,7 @@ Consul is a tool for service discovery and configuration. Consul is distributed, - Configuration: - [Omnibus](https://docs.gitlab.com/omnibus/settings/database.html#disabling-automatic-database-migration) - [Charts](https://docs.gitlab.com/charts/charts/gitlab/migrations/) - - [Source](../update/upgrading_from_source.md#13-install-libraries-migrations-etc) + - [Source](../update/upgrading_from_source.md#14-install-libraries-migrations-etc) - Layer: Core Service (Data) #### Elasticsearch @@ -304,7 +304,7 @@ repository updates to secondary nodes. #### GitLab Geo - Configuration: - - [Omnibus](../administration/geo/replication/index.md#setup-instructions) + - [Omnibus](../administration/geo/setup/index.md) - [Charts](https://docs.gitlab.com/charts/advanced/geo/) - [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/geo.md) - Layer: Core Service (Processor) @@ -555,7 +555,7 @@ Redis is packaged to provide a place to store: - [Project page](https://github.com/docker/distribution/blob/master/README.md) - Configuration: - - [Omnibus](../update/upgrading_from_source.md#13-install-libraries-migrations-etc) + - [Omnibus](../update/upgrading_from_source.md#14-install-libraries-migrations-etc) - [Charts](https://docs.gitlab.com/charts/charts/registry/) - [Source](../administration/packages/container_registry.md#enable-the-container-registry) - [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/registry.md) @@ -665,7 +665,7 @@ You can install them after you create a cluster. This includes: - [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) - [Cert-Manager](https://cert-manager.io/docs/) - [Prometheus](https://prometheus.io/docs/introduction/overview/) -- a [Runner](https://docs.gitlab.com/runner/) +- [GitLab Runner](https://docs.gitlab.com/runner/) - [JupyterHub](https://jupyter.org) - [Knative](https://cloud.google.com/knative/) diff --git a/doc/development/auto_devops.md b/doc/development/auto_devops.md index 6bdc77fff63..bf259e47cb1 100644 --- a/doc/development/auto_devops.md +++ b/doc/development/auto_devops.md @@ -38,8 +38,7 @@ Some jobs use images that are built from external projects: in which the jobs defined in this template use an image that is built using the [`auto-deploy-image`](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image) project. By default, the Helm chart defined in - [`auto-deploy-app`](https://gitlab.com/gitlab-org/charts/auto-deploy-app) - is used to deploy. + [`auto-deploy-app`](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image/-/tree/master/assets/auto-deploy-app) is used to deploy. There are extra variables that get passed to the CI jobs when Auto DevOps is enabled that are not present in a normal CI job. These can be diff --git a/doc/development/changelog.md b/doc/development/changelog.md index 8aaf4056384..f57e666540c 100644 --- a/doc/development/changelog.md +++ b/doc/development/changelog.md @@ -30,7 +30,8 @@ the `author` field. GitLab team members **should not**. ## What warrants a changelog entry? - Any change that introduces a database migration, whether it's regular, post, - or data migration, **must** have a changelog entry. + or data migration, **must** have a changelog entry, even if it is behind a + disabled feature flag. - [Security fixes](https://gitlab.com/gitlab-org/release/docs/blob/master/general/security/developer.md) **must** have a changelog entry, without `merge_request` value and with `type` set to `security`. @@ -113,6 +114,11 @@ the `--ee` option: bin/changelog --ee 'Hey DZ, I added a feature to GitLab!' ``` +NOTE: **Note:** +All entries in the `CHANGELOG.md` file apply to all editions of GitLab. +Changelog updates are based on a common [GitLab codebase](https://gitlab.com/gitlab-org/gitlab/), +and are mirrored without proprietary code to [GitLab FOSS](https://gitlab.com/gitlab-org/gitlab-foss/) (also known as GitLab Community Edition). + At this point the script would ask you to select the category of the change (mapped to the `type` field in the entry): ```plaintext diff --git a/doc/development/chatops_on_gitlabcom.md b/doc/development/chatops_on_gitlabcom.md index 0dd916c37fd..3c1c7750842 100644 --- a/doc/development/chatops_on_gitlabcom.md +++ b/doc/development/chatops_on_gitlabcom.md @@ -21,7 +21,7 @@ To request access to Chatops on GitLab.com: 1. Log into <https://ops.gitlab.net/users/sign_in> **using the same username** as for GitLab.com (you may have to rename it). 1. You could also use the "Sign in with" Google button to sign in, with your GitLab.com email address. -1. Ask in the [#production](https://gitlab.slack.com/messages/production) channel for an existing member to add you to the `chatops` project in Ops. They can do it by running `/chatops run member add <username> gitlab-com/chatops --ops` command in that channel. +1. Ask one of your team members to add you to the `chatops` project in Ops. They can do it by running `/chatops run member add <username> gitlab-com/chatops --ops` command in the `#chat-ops-test` Slack channel. NOTE: **Note:** If you had to change your username for GitLab.com on the first step, make sure [to reflect this information](https://gitlab.com/gitlab-com/www-gitlab-com#adding-yourself-to-the-team-page) on [the team page](https://about.gitlab.com/company/team/). diff --git a/doc/development/cicd/index.md b/doc/development/cicd/index.md index 5b598a19a6e..30ccc52ec5e 100644 --- a/doc/development/cicd/index.md +++ b/doc/development/cicd/index.md @@ -25,7 +25,7 @@ On the left side we have the events that can trigger a pipeline based on various - The [Web API](../../api/pipelines.md#create-a-new-pipeline). - A user clicking the "Run Pipeline" button in the UI. - When a [merge request is created or updated](../../ci/merge_request_pipelines/index.md#pipelines-for-merge-requests). -- When an MR is added to a [Merge Train](../../ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md#merge-trains-premium). +- When an MR is added to a [Merge Train](../../ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md#merge-trains). - A [scheduled pipeline](../../ci/pipelines/schedules.md#pipeline-schedules). - When project is [subscribed to an upstream project](../../ci/multi_project_pipelines.md#trigger-a-pipeline-when-an-upstream-project-is-rebuilt). - When [Auto DevOps](../../topics/autodevops/index.md) is enabled. @@ -53,28 +53,28 @@ The component that processes a pipeline is [`ProcessPipelineService`](https://gi which is responsible for moving all the pipeline's jobs to a completed state. When a pipeline is created, all its jobs are initially in `created` state. This services looks at what jobs in `created` stage are eligible to be processed based on the pipeline structure. Then it moves them into the `pending` state, which means -they can now [be picked up by a Runner](#job-scheduling). After a job has been executed it can complete +they can now [be picked up by a runner](#job-scheduling). After a job has been executed it can complete successfully or fail. Each status transition for job within a pipeline triggers this service again, which looks for the next jobs to be transitioned towards completion. While doing that, `ProcessPipelineService` updates the status of jobs, stages and the overall pipeline. -On the right side of the diagram we have a list of [Runners](../../ci/runners/README.md#configuring-gitlab-runners) -connected to the GitLab instance. These can be Shared Runners, Group Runners or Project-specific Runners. -The communication between Runners and the Rails server occurs through a set of API endpoints, grouped as +On the right side of the diagram we have a list of [runners](../../ci/runners/README.md) +connected to the GitLab instance. These can be shared runners, group runners, or project-specific runners. +The communication between runners and the Rails server occurs through a set of API endpoints, grouped as the `Runner API Gateway`. -We can register, delete and verify Runners, which also causes read/write queries to the database. After a Runner is connected, +We can register, delete, and verify runners, which also causes read/write queries to the database. After a runner is connected, it keeps asking for the next job to execute. This invokes the [`RegisterJobService`](https://gitlab.com/gitlab-org/gitlab/blob/master/app/services/ci/register_job_service.rb) -which will pick the next job and assign it to the Runner. At this point the job will transition to a +which will pick the next job and assign it to the runner. At this point the job will transition to a `running` state, which again triggers `ProcessPipelineService` due to the status change. For more details read [Job scheduling](#job-scheduling)). -While a job is being executed, the Runner sends logs back to the server as well any possible artifacts +While a job is being executed, the runner sends logs back to the server as well any possible artifacts that need to be stored. Also, a job may depend on artifacts from previous jobs in order to run. In this -case the Runner will download them using a dedicated API endpoint. +case the runner will download them using a dedicated API endpoint. Artifacts are stored in object storage, while metadata is kept in the database. An important example of artifacts -is reports (JUnit, SAST, DAST, etc.) which are parsed and rendered in the merge request. +are reports (JUnit, SAST, DAST, etc.) which are parsed and rendered in the merge request. Job status transitions are not all automated. A user may run [manual jobs](../../ci/yaml/README.md#whenmanual), cancel a pipeline, retry specific failed jobs or the entire pipeline. Anything that @@ -90,25 +90,25 @@ from the `CreatePipelineService` every time a downstream pipeline is triggered. When a Pipeline is created all its jobs are created at once for all stages, with an initial state of `created`. This makes it possible to visualize the full content of a pipeline. -A job with the `created` state won't be seen by the Runner yet. To make it possible to assign a job to a Runner, the job must transition first into the `pending` state, which can happen if: +A job with the `created` state won't be seen by the runner yet. To make it possible to assign a job to a runner, the job must transition first into the `pending` state, which can happen if: 1. The job is created in the very first stage of the pipeline. 1. The job required a manual start and it has been triggered. 1. All jobs from the previous stage have completed successfully. In this case we transition all jobs from the next stage to `pending`. 1. The job specifies DAG dependencies using `needs:` and all the dependent jobs are completed. -When the Runner is connected, it requests the next `pending` job to run by polling the server continuously. +When the runner is connected, it requests the next `pending` job to run by polling the server continuously. NOTE: **Note:** -API endpoints used by the Runner to interact with GitLab are defined in [`lib/api/runner.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/api/runner.rb) +API endpoints used by the runner to interact with GitLab are defined in [`lib/api/ci/runner.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/api/ci/runner.rb) -After the server receives the request it selects a `pending` job based on the [`Ci::RegisterJobService` algorithm](#ciregisterjobservice), then assigns and sends the job to the Runner. +After the server receives the request it selects a `pending` job based on the [`Ci::RegisterJobService` algorithm](#ciregisterjobservice), then assigns and sends the job to the runner. -Once all jobs are completed for the current stage, the server "unlocks" all the jobs from the next stage by changing their state to `pending`. These can now be picked by the scheduling algorithm when the Runner requests new jobs, and continues like this until all stages are completed. +Once all jobs are completed for the current stage, the server "unlocks" all the jobs from the next stage by changing their state to `pending`. These can now be picked by the scheduling algorithm when the runner requests new jobs, and continues like this until all stages are completed. -### Communication between Runner and GitLab server +### Communication between runner and GitLab server -Once the Runner is [registered](https://docs.gitlab.com/runner/register/) using the registration token, the server knows what type of jobs it can execute. This depends on: +Once the runner is [registered](https://docs.gitlab.com/runner/register/) using the registration token, the server knows what type of jobs it can execute. This depends on: - The type of runner it is registered as: - a shared runner @@ -116,30 +116,30 @@ Once the Runner is [registered](https://docs.gitlab.com/runner/register/) using - a project specific runner - Any associated tags. -The Runner initiates the communication by requesting jobs to execute with `POST /api/v4/jobs/request`. Although this polling generally happens every few seconds we leverage caching via HTTP headers to reduce the server-side work load if the job queue doesn't change. +The runner initiates the communication by requesting jobs to execute with `POST /api/v4/jobs/request`. Although this polling generally happens every few seconds we leverage caching via HTTP headers to reduce the server-side work load if the job queue doesn't change. This API endpoint runs [`Ci::RegisterJobService`](https://gitlab.com/gitlab-org/gitlab/blob/master/app/services/ci/register_job_service.rb), which: 1. Picks the next job to run from the pool of `pending` jobs -1. Assigns it to the Runner -1. Presents it to the Runner via the API response +1. Assigns it to the runner +1. Presents it to the runner via the API response ### `Ci::RegisterJobService` -There are 3 top level queries that this service uses to gather the majority of the jobs and they are selected based on the level where the Runner is registered to: +There are 3 top level queries that this service uses to gather the majority of the jobs and they are selected based on the level where the runner is registered to: -- Select jobs for shared Runner (instance level) -- Select jobs for group level Runner -- Select jobs for project Runner +- Select jobs for shared runner (instance level) +- Select jobs for group runner +- Select jobs for project runner -This list of jobs is then filtered further by matching tags between job and Runner tags. +This list of jobs is then filtered further by matching tags between job and runner tags. NOTE: **Note:** -If a job contains tags, the Runner will not pick the job if it does not match **all** the tags. -The Runner may have more tags than defined for the job, but not vice-versa. +If a job contains tags, the runner will not pick the job if it does not match **all** the tags. +The runner may have more tags than defined for the job, but not vice-versa. -Finally if the Runner can only pick jobs that are tagged, all untagged jobs are filtered out. +Finally if the runner can only pick jobs that are tagged, all untagged jobs are filtered out. -At this point we loop through remaining `pending` jobs and we try to assign the first job that the Runner "can pick" based on additional policies. For example, Runners marked as `protected` can only pick jobs that run against protected branches (such as production deployments). +At this point we loop through remaining `pending` jobs and we try to assign the first job that the runner "can pick" based on additional policies. For example, runners marked as `protected` can only pick jobs that run against protected branches (such as production deployments). -As we increase the number of Runners in the pool we also increase the chances of conflicts which would arise if assigning the same job to different Runners. To prevent that we gracefully rescue conflict errors and assign the next job in the list. +As we increase the number of runners in the pool we also increase the chances of conflicts which would arise if assigning the same job to different runners. To prevent that we gracefully rescue conflict errors and assign the next job in the list. diff --git a/doc/development/cicd/templates.md b/doc/development/cicd/templates.md index 0169ca42ac6..77cedc9814e 100644 --- a/doc/development/cicd/templates.md +++ b/doc/development/cicd/templates.md @@ -13,15 +13,15 @@ This document explains how to develop [GitLab CI/CD templates](../../ci/examples All template files reside in the `lib/gitlab/ci/templates` directory, and are categorized by the following sub-directories: -| Sub-directroy | Content | [Selectable in UI](#make-sure-the-new-template-can-be-selected-in-ui) | -|---------------|--------------------------------------------------------------|-----------------------------------------------------------------------| -| `/AWS/*` | Cloud Deployment (AWS) related jobs | No | -| `/Jobs/*` | Auto DevOps related jobs | Yes | -| `/Pages/*` | Static site generators for GitLab Pages (for example Jekyll) | Yes | -| `/Security/*` | Security related jobs | Yes | -| `/Verify/*` | Verify/testing related jobs | Yes | -| `/Worklows/*` | Common uses of the `workflow:` keyword | No | -| `/*` (root) | General templates | Yes | +| Sub-directory | Content | [Selectable in UI](#make-sure-the-new-template-can-be-selected-in-ui) | +|----------------|--------------------------------------------------------------|-----------------------------------------------------------------------| +| `/AWS/*` | Cloud Deployment (AWS) related jobs | No | +| `/Jobs/*` | Auto DevOps related jobs | No | +| `/Pages/*` | Static site generators for GitLab Pages (for example Jekyll) | Yes | +| `/Security/*` | Security related jobs | Yes | +| `/Verify/*` | Verify/testing related jobs | Yes | +| `/Workflows/*` | Common uses of the `workflow:` keyword | No | +| `/*` (root) | General templates | Yes | ## Criteria @@ -64,6 +64,67 @@ users have to fix their `.gitlab-ci.yml` that could annoy their workflow. Please read [versioning](#versioning) section for introducing breaking change safely. +## Versioning + +Versioning allows you to introduce a new template without modifying the existing +one. This process is useful when we need to introduce a breaking change, +but don't want to affect the existing projects that depends on the current template. + +### Stable version + +A stable CI/CD template is a template that only introduces breaking changes in major +release milestones. Name the stable version of a template as `<template-name>.gitlab-ci.yml`, +for example `Jobs/Deploy.gitlab-ci.yml`. + +You can make a new stable template by copying [the latest template](#latest-version) +available in a major milestone release of GitLab like `13.0`. All breaking changes +must be announced in a blog post before the official release, for example +[GitLab.com is moving to 13.0, with narrow breaking changes](https://about.gitlab.com/releases/2020/05/06/gitlab-com-13-0-breaking-changes/) + +You can change a stable template version in a minor GitLab release like `13.1` if: + +- The change is not a [breaking change](#backward-compatibility). +- The change is ported to [the latest template](#latest-version), if one exists. + +### Latest version + +Templates marked as `latest` can be updated in any release, even with +[breaking changes](#backward-compatibility). Add `.latest` to the template name if +it's considered the latest version, for example `Jobs/Deploy.latest.gitlab-ci.yml`. + +When you introduce [a breaking change](#backward-compatibility), +you **must** test and document [the upgrade path](#verify-breaking-changes). +In general, we should not promote the latest template as the best option, as it could surprise users with unexpected problems. + +If the `latest` template does not exist yet, you can copy [the stable template](#stable-version). + +### How to include an older stable template + +Users may want to use an older [stable template](#stable-version) that is not bundled +in the current GitLab package. For example, the stable templates in GitLab v13.0 and +GitLab v14.0 could be so different that a user will want to continue using the v13.0 template even +after upgrading to GitLab 14.0. + +You can add a note in the template or in documentation explaining how to use `include:remote` +to include older template versions. If other templates are included with `include: template`, +they can be combined with the `include: remote`: + +```yaml +# To use the v13 stable template, which is not included in v14, fetch the specifc +# template from the remote template repository with the `include:remote:` keyword. +# If you fetch from the GitLab canonical project, use the following URL format: +# https://gitlab.com/gitlab-org/gitlab/-/raw/<version>/lib/gitlab/ci/templates/<template-name> +include: + - template: Auto-DevOps.gitlab-ci.yml + - remote: https://gitlab.com/gitlab-org/gitlab/-/raw/v13.0.1-ee/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml +``` + +### Further reading + +There is an [open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/17716) about +introducing versioning concepts in GitLab CI Templates. You can check that issue to +follow the progress. + ## Testing Each CI/CD template must be tested in order to make sure that it's safe to be published. @@ -95,18 +156,20 @@ You should write an RSpec test to make sure that pipeline jobs will be generated 1. Add a test file at `spec/lib/gitlab/ci/templates/<template-category>/<template-name>_spec.rb` 1. Test that pipeline jobs are properly created via `Ci::CreatePipelineService`. +### Verify breaking changes + +When you introduce a breaking change to [a `latest` template](#latest-version), +you must: + +1. Test the upgrade path from [the stable template](#stable-version). +1. Verify what kind of errors users will encounter. +1. Document it as a troubleshooting guide. + +This information will be important for users when [a stable template](#stable-version) +is updated in a major version GitLab release. + ## Security A template could contain malicious code. For example, a template that contains the `export` shell command in a job might accidentally expose project secret variables in a job log. If you're unsure if it's secure or not, you need to ask security experts for cross-validation. - -## Versioning - -Versioning allows you to introduce a new template without modifying the existing -one. This is useful process especially when we need to introduce a breaking change, -but don't want to affect the existing projects that depends on the current template. - -There is an [open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/17716) for -introducing versioning concept in GitLab Ci Template. Please follow the issue for -checking the progress. diff --git a/doc/development/code_review.md b/doc/development/code_review.md index 2159f7a9ed5..2e319efa5f3 100644 --- a/doc/development/code_review.md +++ b/doc/development/code_review.md @@ -99,6 +99,7 @@ with [domain expertise](#domain-experts). 1. If your merge request includes end-to-end **and** non-end-to-end changes (*3*), it must be **approved by a [Software Engineer in Test](https://about.gitlab.com/handbook/engineering/quality/#individual-contributors)**. 1. If your merge request only includes end-to-end changes (*3*) **or** if the MR author is a [Software Engineer in Test](https://about.gitlab.com/handbook/engineering/quality/#individual-contributors), it must be **approved by a [Quality maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_maintainers_qa)** +1. If your merge request includes a new or updated [application limit](https://about.gitlab.com/handbook/product/product-processes/#introducing-application-limits), it must be **approved by a [product manager](https://about.gitlab.com/company/team/)**. - (*1*): Please note that specs other than JavaScript specs are considered backend code. - (*2*): We encourage you to seek guidance from a database maintainer if your merge @@ -138,7 +139,7 @@ up confusion or verify that the end result matches what they had in mind, to database specialists to get input on the data model or specific queries, or to any other developer to get an in-depth review of the solution. -If an author is unsure if a merge request needs a [domain experts's](#domain-experts) opinion, that's +If an author is unsure if a merge request needs a [domain expert's](#domain-experts) opinion, that's usually a pretty good sign that it does, since without it the required level of confidence in their solution will not have been reached. @@ -358,12 +359,13 @@ When ready to merge: messy commit history that is intended to be squashed. - **Start a new merge request pipeline with the `Run Pipeline` button in the merge request's "Pipelines" tab, and enable "Merge When Pipeline Succeeds" (MWPS).** Note that: - - If the **latest [Pipeline for Merged Results](../ci/merge_request_pipelines/pipelines_for_merged_results/#pipelines-for-merged-results-premium)** finished less than 2 hours ago, you + - If the **latest [Pipeline for Merged Results](../ci/merge_request_pipelines/pipelines_for_merged_results/#pipelines-for-merged-results)** finished less than 2 hours ago, you might merge without starting a new pipeline as the merge request is close enough to `master`. - - If the **merge request is from a fork**, we can't use [Pipelines for Merged Results](../ci/merge_request_pipelines/pipelines_for_merged_results/index.md#prerequisites), therefore, they're more prone to breaking `master`. - Check how far behind `master` the source branch is. If it's more than 100 commits behind, ask the author to - rebase it before merging. + - If the **merge request is from a fork**, we can use [Pipelines for Merged Results from a forked project](../ci/merge_request_pipelines/index.md#run-pipelines-in-the-parent-project-for-merge-requests-from-a-forked-project) with caution. + Before triggering the pipeline, review all changes for **malicious code**. + If you cannot trigger the pipeline, review the status of the fork relative to `master`. + If it's more than 100 commits behind, ask the author to rebase it before merging. - If [master is broken](https://about.gitlab.com/handbook/engineering/workflow/#broken-master), in addition to the two above rules, check that any failure also happens in `master` and post a link to the ~"master:broken" issue before clicking the diff --git a/doc/development/contributing/index.md b/doc/development/contributing/index.md index cea9043a333..7550fe69546 100644 --- a/doc/development/contributing/index.md +++ b/doc/development/contributing/index.md @@ -67,6 +67,11 @@ we credit the original author by adding a changelog entry crediting the author and optionally include the original author on at least one of the commits within the MR. +## Closing policy for inactive bugs + +GitLab values the time spent by contributors on reporting bugs. However, if a bug remains inactive for a very long period, +it will qualify for auto-closure. Please refer to the [auto-close inactive bugs](https://about.gitlab.com/handbook/engineering/quality/triage-operations/#auto-close-inactive-bugs) section in our handbook to understand the complete workflow. + ## Helping others Help other GitLab users when you can. @@ -81,6 +86,11 @@ If you would like to contribute to GitLab: - Issues with the [`~Accepting merge requests` label](issue_workflow.md#label-for-community-contributors) are a great place to start. +- Optimizing our tests is another great opportunity to contribute. You can use + [RSpec profiling statistics](https://gitlab-org.gitlab.io/rspec_profiling_stats/) to identify + slowest tests. These tests are good candidates for improving and checking if any of + [best practices](../testing_guide/best_practices.md) + could speed them up. - Consult the [Contribution Flow](#contribution-flow) section to learn the process. If you have any questions or need help visit [Getting Help](https://about.gitlab.com/get-help/) to diff --git a/doc/development/contributing/issue_workflow.md b/doc/development/contributing/issue_workflow.md index 76175cb7b66..bb7b4713a5e 100644 --- a/doc/development/contributing/issue_workflow.md +++ b/doc/development/contributing/issue_workflow.md @@ -36,21 +36,21 @@ project. To allow for asynchronous issue handling, we use [milestones](https://gitlab.com/groups/gitlab-org/-/milestones) and [labels](https://gitlab.com/gitlab-org/gitlab/-/labels). Leads and product managers handle most of the -scheduling into milestones. Labeling is a task for everyone. +scheduling into milestones. Labeling is a task for everyone. (For some projects, labels can be set only by GitLab team members and not by community contributors). Most issues will have labels for at least one of the following: -- Type: `~feature`, `~bug`, `~backstage`, `~documentation`, etc. +- Type: `~feature`, `~bug`, `~tooling`, `~documentation`, etc. - Stage: `~"devops::plan"`, `~"devops::create"`, etc. - Group: `~"group::source code"`, `~"group::knowledge"`, `~"group::editor"`, etc. -- Category: `~"Category:Code Analytics"`, `~"Category:DevOps Score"`, `~"Category:Templates"`, etc. +- Category: `~"Category:Code Analytics"`, `~"Category:DevOps Reports"`, `~"Category:Templates"`, etc. - Feature: `~wiki`, `~ldap`, `~api`, `~issues`, `~"merge requests"`, etc. - Department: `~UX`, `~Quality` - Team: `~"Technical Writing"`, `~Delivery` - Specialization: `~frontend`, `~backend`, `~documentation` - Release Scoping: `~Deliverable`, `~Stretch`, `~"Next Patch Release"` -- Priority: `~P::1`, `~P::2`, `~P::3`, `~P::4` -- Severity: ~`S::1`, `~S::2`, `~S::3`, `~S::4` +- Priority: `~"priority::1"`, `~"priority::2"`, `~"priority::3"`, `~"priority::4"` +- Severity: ~`"severity::1"`, `~"severity::2"`, `~"severity::3"`, `~"severity::4"` All labels, their meaning and priority are defined on the [labels page](https://gitlab.com/gitlab-org/gitlab/-/labels). @@ -67,7 +67,7 @@ The current type labels are: - ~feature - ~bug -- ~backstage +- ~tooling - ~"support request" - ~meta - ~documentation @@ -93,9 +93,9 @@ Following is a non-exhaustive list of facet labels: - ~enhancement: This label can refine an issue that has the ~feature label. - ~"master:broken": This label can refine an issue that has the ~bug label. - ~"failure::flaky-test": This label can refine an issue that has the ~bug label. -- ~"technical debt": This label can refine an issue that has the ~backstage label. -- ~"static analysis": This label can refine an issue that has the ~backstage label. -- ~"ci-build": This label can refine an issue that has the ~backstage label. +- ~"technical debt": This label can refine an issue that has the ~tooling label. +- ~"static analysis": This label can refine an issue that has the ~tooling label. +- ~"ci-build": This label can refine an issue that has the ~tooling label. - ~performance: A performance issue could describe a ~bug or a ~feature. - ~security: A security issue could describe a ~bug or a ~feature. - ~database: A database issue could describe a ~bug or a ~feature. @@ -118,7 +118,7 @@ the `gitlab-org` group since its key under `stages` is `manage`. The current stage labels can be found by [searching the labels list for `devops::`](https://gitlab.com/groups/gitlab-org/-/labels?search=devops::). -These labels are [scoped labels](../../user/project/labels.md#scoped-labels-premium) +These labels are [scoped labels](../../user/project/labels.md#scoped-labels) and thus are mutually exclusive. The Stage labels are used to generate the [direction pages](https://about.gitlab.com/direction/) automatically. @@ -145,7 +145,7 @@ under `stages.manage.groups` is `continuous_integration`. The current group labels can be found by [searching the labels list for `group::`](https://gitlab.com/groups/gitlab-org/-/labels?search=group::). -These labels are [scoped labels](../../user/project/labels.md#scoped-labels-premium) +These labels are [scoped labels](../../user/project/labels.md#scoped-labels) and thus are mutually exclusive. You can find the groups listed in the [Product Stages, Groups, and Categories](https://about.gitlab.com/handbook/product/product-categories/) page. @@ -188,9 +188,9 @@ their color is `#428BCA`. `<Category Name>` is the category name as it is in the single source of truth for categories at <https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/categories.yml>. -For instance, the "DevOps Score" category is represented by the -~"Category:DevOps Score" label in the `gitlab-org` group since its -`devops_score.name` value is "DevOps Score". +For instance, the "DevOps Report" category is represented by the +~"Category:DevOps Reports" label in the `gitlab-org` group since its +`devops_reports.name` value is "DevOps Reports". If a category's label doesn't respect this naming convention, it should be specified with [the `label` attribute](https://about.gitlab.com/handbook/marketing/website/#category-attributes) @@ -275,10 +275,10 @@ or ~"Stretch". Any open issue for a previous milestone should be labeled We have the following priority labels: -- ~P::1 -- ~P::2 -- ~P::3 -- ~P::4 +- ~"priority::1" +- ~"priority::2" +- ~"priority::3" +- ~"priority::4" Please refer to the issue triage [priority label](https://about.gitlab.com/handbook/engineering/quality/issue-triage/#priority) section in our handbook to see how it's used. @@ -286,10 +286,10 @@ Please refer to the issue triage [priority label](https://about.gitlab.com/handb We have the following severity labels: -- ~S::1 -- ~S::2 -- ~S::3 -- ~S::4 +- ~"severity::1" +- ~"severity::2" +- ~"severity::3" +- ~"severity::4" Please refer to the issue triage [severity label](https://about.gitlab.com/handbook/engineering/quality/issue-triage/#severity) section in our handbook to see how it's used. diff --git a/doc/development/contributing/merge_request_workflow.md b/doc/development/contributing/merge_request_workflow.md index e5a8bdad7b0..d88b159b666 100644 --- a/doc/development/contributing/merge_request_workflow.md +++ b/doc/development/contributing/merge_request_workflow.md @@ -176,7 +176,7 @@ the contribution acceptance criteria below: exposing a bug in existing code). Every new class should have corresponding unit tests, even if the class is exercised at a higher level, such as a feature test. - If a failing CI build seems to be unrelated to your contribution, you can try - restarting the failing CI job, rebasing from master to bring in updates that + restarting the failing CI job, rebasing from `master` to bring in updates that may resolve the failure, or if it has not been fixed yet, ask a developer to help you fix the test. 1. The MR initially contains a few logically organized commits. @@ -242,6 +242,7 @@ request: 1. The [GitLab Development Kit](https://gitlab.com/gitlab-org/gitlab-development-kit). 1. The [CI environment preparation](https://gitlab.com/gitlab-org/gitlab/blob/master/scripts/prepare_build.sh). 1. The [Omnibus package creator](https://gitlab.com/gitlab-org/omnibus-gitlab). +1. The [Cloud Native GitLab Dockerfiles](https://gitlab.com/gitlab-org/build/CNG) ## Incremental improvements diff --git a/doc/development/creating_enums.md b/doc/development/creating_enums.md index 3833f771bb5..af9bf919b2b 100644 --- a/doc/development/creating_enums.md +++ b/doc/development/creating_enums.md @@ -33,28 +33,32 @@ tempted to organize the `enum` as the following: ```ruby # Define `failure_reason` enum in `Pipeline` model: class Pipeline < ApplicationRecord - enum failure_reason: ::PipelineEnums.failure_reasons + enum failure_reason: Enums::Pipeline.failure_reasons end ``` ```ruby # Define key/value pairs that used in FOSS and EE: -module PipelineEnums - def self.failure_reasons - { unknown_failure: 0, config_error: 1 } +module Enums + module Pipeline + def self.failure_reasons + { unknown_failure: 0, config_error: 1 } + end end end -PipelineEnums.prepend_if_ee('EE::PipelineEnums') +Enums::Pipeline.prepend_if_ee('EE::Enums::Pipeline') ``` ```ruby # Define key/value pairs that used in EE only: module EE - module PipelineEnums - override :failure_reasons - def failure_reasons - super.merge(activity_limit_exceeded: 2) + module Enums + module Pipeline + override :failure_reasons + def failure_reasons + super.merge(activity_limit_exceeded: 2) + end end end end @@ -63,7 +67,7 @@ end This works as-is, however, it has a couple of downside that: - Someone could define a key/value pair in EE that is **conflicted** with a value defined in FOSS. - e.g. Define `activity_limit_exceeded: 1` in `EE::PipelineEnums`. + e.g. Define `activity_limit_exceeded: 1` in `EE::Enums::Pipeline`. - When it happens, the feature works totally different. e.g. We cannot figure out `failure_reason` is either `config_error` or `activity_limit_exceeded`. - When it happens, we have to ship a database migration to fix the data integrity, @@ -74,10 +78,12 @@ For example, this example sets `1000` as the offset: ```ruby module EE - module PipelineEnums - override :failure_reasons - def failure_reasons - super.merge(activity_limit_exceeded: 1_000, size_limit_exceeded: 1_001) + module Enums + module Pipeline + override :failure_reasons + def failure_reasons + super.merge(activity_limit_exceeded: 1_000, size_limit_exceeded: 1_001) + end end end end diff --git a/doc/development/database/database_reviewer_guidelines.md b/doc/development/database/database_reviewer_guidelines.md index 894b1ea15f0..6cb061f9959 100644 --- a/doc/development/database/database_reviewer_guidelines.md +++ b/doc/development/database/database_reviewer_guidelines.md @@ -23,9 +23,9 @@ For more information on the database review process, check the [database review Team members are encouraged to self-identify as database domain experts and add it to their [team profile](https://gitlab.com/gitlab-com/www-gitlab-com/-/blob/master/data/team.yml) ```yaml - projects: - gitlab: - - reviewer database +projects: + gitlab: + - reviewer database ``` Assign the MR which adds your expertise to the `team.yml` file to a database maintainer @@ -70,9 +70,9 @@ they can update their [team profile](https://gitlab.com/gitlab-com/www-gitlab-co to a `trainee_maintainer database`: ```yaml - projects: - gitlab: - - trainee_maintainer database +projects: + gitlab: + - trainee_maintainer database ``` The first step is to a create a [Trainee Database Maintainer Issue](https://gitlab.com/gitlab-com/www-gitlab-com/-/issues/new?issuable_template=trainee-database-maintainer). diff --git a/doc/development/database/strings_and_the_text_data_type.md b/doc/development/database/strings_and_the_text_data_type.md index 0e77e3972e0..b73dfa859fb 100644 --- a/doc/development/database/strings_and_the_text_data_type.md +++ b/doc/development/database/strings_and_the_text_data_type.md @@ -38,6 +38,8 @@ For example, consider a migration that creates a table with two text columns, ```ruby class CreateDbGuides < ActiveRecord::Migration[6.0] + include Gitlab::Database::MigrationHelpers + DOWNTIME = false disable_ddl_transaction! @@ -179,6 +181,7 @@ in a post-deployment migration, ```ruby class AddTextLimitMigration < ActiveRecord::Migration[6.0] include Gitlab::Database::MigrationHelpers + DOWNTIME = false disable_ddl_transaction! diff --git a/doc/development/database_debugging.md b/doc/development/database_debugging.md index 25b62e0d693..61e8ac60bfe 100644 --- a/doc/development/database_debugging.md +++ b/doc/development/database_debugging.md @@ -72,7 +72,7 @@ Use these instructions for exploring the GitLab database while developing with t 1. **Port number to connect to**: `5432` (default). 1. <!-- vale gitlab.Spelling = NO --> **Use an ssl connection?** - <!-- vale gitlab.rulename = NO --> This depends on your installation. Options are: + <!-- vale gitlab.Spelling = YES --> This depends on your installation. Options are: - **Use Secure Connection** - **Standard Connection** (default) 1. **(Optional) The database to connect to**: `gitlabhq_development`. diff --git a/doc/development/distributed_tracing.md b/doc/development/distributed_tracing.md index cbbeae47a41..9a4c15c5c19 100644 --- a/doc/development/distributed_tracing.md +++ b/doc/development/distributed_tracing.md @@ -6,6 +6,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Distributed Tracing - development guidelines +NOTE: **Note:** +Distributed Tracing in GitLab is currently considered **experimental**, as it has not yet been tested at scale on GitLab.com. + GitLab is instrumented for distributed tracing. According to [Open Tracing](https://opentracing.io/docs/overview/what-is-tracing/): diff --git a/doc/development/documentation/feature_flags.md b/doc/development/documentation/feature_flags.md index e2fbf25eb8a..a4a6ee2fa0f 100644 --- a/doc/development/documentation/feature_flags.md +++ b/doc/development/documentation/feature_flags.md @@ -1,5 +1,8 @@ --- -type: reference +type: reference, dev +stage: none +group: Development +info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" description: "GitLab development - how to document features deployed behind feature flags" --- @@ -24,6 +27,7 @@ See how to document them below, according to the state of the flag: - [Features disabled by default](#features-disabled-by-default). - [Features that became enabled by default](#features-that-became-enabled-by-default). - [Features directly enabled by default](#features-directly-enabled-by-default). +- [Features that can be enabled or disabled for a single project](#features-enabled-by-project). - [Features with the feature flag removed](#features-with-flag-removed). NOTE: **Note:** @@ -37,105 +41,120 @@ therefore, it indicates that it cannot be done by regular users of GitLab.com. For features disabled by default, if they cannot be used yet due to lack of completeness, or if they're still under internal evaluation (for example, for performance implications) do **not document them**: add (or merge) the docs -only when the feature is safe and ready to use and test by end users. +only when the feature is safe and ready to use and test by end-users. For feature flags disabled by default, if they can be used by end users: - Say that it's disabled by default. - Say whether it's enabled on GitLab.com. -- Say whether it can be enabled or disabled per-project. +- If the feature can be enabled/disabled for a single project, add the + [by-project information](#features-enabled-by-project). Otherwise, + do not say anything about it. - Say whether it's recommended for production use. - Document how to enable and disable it. +- Add a warning to the user saying that the feature is disabled. -For example, for a feature disabled by default, disabled on GitLab.com, can be enabled or disabled per-project, and -not ready for production use: +For example, for a feature disabled by default, disabled on GitLab.com, cannot +be enabled for a single project, and is not ready for production use: ````markdown # Feature Name > - [Introduced](link-to-issue) in GitLab 12.0. -> - It's deployed behind a feature flag, disabled by default. +> - It's [deployed behind a feature flag](<replace with path to>/user/feature_flags.md), disabled by default. > - It's disabled on GitLab.com. -> - It's able to be enabled or disabled per-project. > - It's not recommended for production use. > - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#anchor-to-section). **(CORE ONLY)** -(...) +CAUTION: **Warning:** +This feature might not be available to you. Check the **version history** note above for details. + +(...Regular content goes here...) + +<!-- Add this at the end of the file --> ### Enable or disable <Feature Name> **(CORE ONLY)** <Feature Name> is under development and not ready for production use. It is deployed behind a feature flag that is **disabled by default**. -[GitLab administrators with access to the GitLab Rails console](../path/to/administration/feature_flags.md) -can enable it for your instance. <Feature Name> can be enabled or disabled per-project. +[GitLab administrators with access to the GitLab Rails console](<replace with path to>/administration/feature_flags.md) +can enable it. To enable it: ```ruby -# Instance-wide Feature.enable(:<feature flag>) -# or by project -Feature.enable(:<feature flag>, Project.find(<project id>)) ``` To disable it: ```ruby -# Instance-wide Feature.disable(:<feature flag>) -# or by project -Feature.disable(:<feature flag>, Project.find(<project id>)) ``` ```` Adjust the blurb according to the state of the feature you're documenting. +Replace `<Feature name>`, `**(CORE ONLY)**`, `<feature flag>`, and +`<replace with path to>`, and `#anchor-to-section` accordingly. ### Features that became enabled by default -For features that became enabled by default: +For features that were released disabled by default but became enabled by +default: - Say that it became enabled by default. - Say whether it's enabled on GitLab.com. -- Say whether it can be enabled or disabled per-project. +- If the feature can be enabled/disabled for a single project, add the + [by-project information](#features-enabled-by-project). Otherwise, + do not say anything about it. - Say whether it's recommended for production use. - Document how to disable and enable it. +- Add a warning to the user saying that the feature might be disabled. -For example, for a feature initially deployed disabled by default, that became enabled by default, that is enabled on GitLab.com, that cannot be enabled or disabled per-project, and ready for production use: +For example, for a feature initially deployed disabled by default, that became +enabled by default, that is enabled on GitLab.com, and is ready for production +use: ````markdown # Feature Name > - [Introduced](link-to-issue) in GitLab 12.0. -> - It was deployed behind a feature flag, disabled by default. +> - It was [deployed behind a feature flag](<replace with path to>/user/feature_flags.md), disabled by default. > - [Became enabled by default](link-to-issue) on GitLab 12.1. > - It's enabled on GitLab.com. -> - It's not able to be enabled or disabled per-project. > - It's recommended for production use. > - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#anchor-to-section). **(CORE ONLY)** -(...) +CAUTION: **Warning:** +This feature might not be available to you. Check the **version history** note above for details. + +(...Regular content goes here...) + +<!-- Add this at the end of the file --> ### Enable or disable <Feature Name> **(CORE ONLY)** <Feature Name> is under development but ready for production use. It is deployed behind a feature flag that is **enabled by default**. -[GitLab administrators with access to the GitLab Rails console](..path/to/administration/feature_flags.md) -can opt to disable it for your instance it cannot be enabled or disabled per-project. +[GitLab administrators with access to the GitLab Rails console](<replace with path to>/administration/feature_flags.md) +can opt to disable it. -To disable it: +To enable it: ```ruby -Feature.disable(:<feature flag>) +Feature.enable(:<feature flag>) ``` -To enable it: +To disable it: ```ruby -Feature.enable(:<feature flag>) +Feature.disable(:<feature flag>) ``` ```` Adjust the blurb according to the state of the feature you're documenting. +Replace `<Feature name>`, `**(CORE ONLY)**`, `<feature flag>`, +`<replace with path to>`, and `#anchor-to-section` accordingly. ### Features directly enabled by default @@ -143,45 +162,134 @@ For features enabled by default: - Say it's enabled by default. - Say whether it's enabled on GitLab.com. -- Say whether it can be enabled or disabled per-project. +- If the feature can be enabled/disabled for a single project, add the + [by-project information](#features-enabled-by-project). Otherwise, + do not say anything about it. - Say whether it's recommended for production use. - Document how to disable and enable it. +- Add a warning to the user saying that the feature might be disabled. -For example, for a feature enabled by default, enabled on GitLab.com, cannot be enabled or disabled per-project, and ready for production use: +For example, for a feature enabled by default, enabled on GitLab.com, that +cannot be enabled for a single project, and is ready for production use: ````markdown # Feature Name > - [Introduced](link-to-issue) in GitLab 12.0. -> - It's deployed behind a feature flag, enabled by default. +> - It's [deployed behind a feature flag](<replace with path to>/user/feature_flags.md), enabled by default. > - It's enabled on GitLab.com. -> - It's not able to be enabled or disabled per-project. > - It's recommended for production use. > - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#anchor-to-section). **(CORE ONLY)** -(...) +CAUTION: **Warning:** +This feature might not be available to you. Check the **version history** note above for details. + +(...Regular content goes here...) + +<!-- Add this at the end of the file --> ### Enable or disable <Feature Name> **(CORE ONLY)** <Feature Name> is under development but ready for production use. It is deployed behind a feature flag that is **enabled by default**. -[GitLab administrators with access to the GitLab Rails console](..path/to/administration/feature_flags.md) -can opt to disable it for your instance. +[GitLab administrators with access to the GitLab Rails console](<replace with path to>/administration/feature_flags.md) +can opt to disable it. + +To enable it: + +```ruby +Feature.enable(:<feature flag>) +``` To disable it: ```ruby Feature.disable(:<feature flag>) ``` +```` -To enable it: +Adjust the blurb according to the state of the feature you're documenting. +Replace `<Feature name>`, `**(CORE ONLY)**`, `<feature flag>`, +`<replace with path to>`, and `#anchor-to-section` accordingly. + +### Features enabled by project + +If the feature can be enabled/disabled for a single project, include in the +version history note: + +```markdown +> - It can be enabled or disabled for a single project. +``` + +Then add the by-project code to the code blocks: + +Enable code: + +```ruby +# For the instance +Feature.enable(:<feature flag>) +# For a single project +Feature.enable(:<feature flag>, Project.find(<project id>)) +``` + +Disable code: + +```ruby +# For the instance +Feature.disable(:<feature flag>) +# For a single project +Feature.disable(:<feature flag>, Project.find(<project id>)) +``` + +For example, for a feature enabled by default, enabled on GitLab.com, that can +be enabled by project, and is ready for production use: + +````markdown +# Feature Name + +> - [Introduced](link-to-issue) in GitLab 12.0. +> - It's [deployed behind a feature flag](<replace with path to>/user/feature_flags.md), enabled by default. +> - It's enabled on GitLab.com. +> - It can be enabled or disable for a single project. +> - It's recommended for production use. +> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#anchor-to-section). **(CORE ONLY)** + +CAUTION: **Warning:** +This feature might not be available to you. Check the **version history** note above for details. + +(...Regular content goes here...) + +<!-- Add this at the end of the file --> + +### Enable or disable <Feature Name> **(CORE ONLY)** + +<Feature Name> is under development but ready for production use. +It is deployed behind a feature flag that is **enabled by default**. +[GitLab administrators with access to the GitLab Rails console](<replace with path to>/administration/feature_flags.md) +can opt to disable it. + +To enabled it: ```ruby +# For the instance Feature.enable(:<feature flag>) +# For a single project +Feature.enable(:<feature flag>, Project.find(<project id>)) +``` + +To disable it: + +```ruby +# For the instance +Feature.disable(:<feature flag>) +# For a single project +Feature.disable(:<feature flag>, Project.find(<project id>)) ``` ```` Adjust the blurb according to the state of the feature you're documenting. +Replace `<Feature name>`, `**(CORE ONLY)**`, `<feature flag>`, +`<replace with path to>`, and `#anchor-to-section` accordingly. ### Features with flag removed @@ -195,6 +303,6 @@ mentions the flag in the version history notes: > - [Introduced](link-to-issue) in GitLab 12.0. > - [Feature flag removed](link-to-issue) in GitLab 12.2. -(...) +(...Regular content...) ```` diff --git a/doc/development/documentation/index.md b/doc/development/documentation/index.md index 283060ba8d4..d6f24d6374d 100644 --- a/doc/development/documentation/index.md +++ b/doc/development/documentation/index.md @@ -105,22 +105,22 @@ info: To determine the technical writer assigned to the Stage/Group associated w --- ``` -### Page type metadata +### Document type metadata Originally discussed in [this epic](https://gitlab.com/groups/gitlab-org/-/epics/1280), -each page should have a `type` metadata. It can be one or more of the following: - -- `index`: Index/overview pages. They serve as a list to other pages. Doesn't - necessarily mean the page should be named `index.md`. [Example page](../../install/README.md). -- `concepts`: What you need to know before using product. Informational, not - instructional. For example, abstract ideas, explain meaning or benefit, support - understanding of tasks. They are read for background information, for example - "Why X is important". [Example page](../../topics/autodevops/index.md). -- `howto`: Specific use case instructions. [Example page](../../ssh/README.md). -- `tutorial`: Learn a process/concept by doing. [Example page](../../gitlab-basics/start-using-git.md). -- `reference`: Covers what things are/do. Things like specific settings, facts - without too much explanation that are read for detailed information. - [Example page](../../ci/yaml/README.md). +each page should have a metadata tag called `type`. It can be one or more of the +following: + +- `index`: It consists mostly of a list of links to other pages. + [Example page](../../README.md). +- `concepts`: The background or context of a subject. + [Example page](../../topics/autodevops/index.md). +- `howto`: Specific use case instructions. + [Example page](../../ssh/README.md). +- `tutorial`: Learn a process/concept by doing. + [Example page](../../gitlab-basics/start-using-git.md). +- `reference`: A collection of information used as a reference to use a feature + or a functionality. [Example page](../../ci/yaml/README.md). ### Redirection metadata @@ -147,17 +147,10 @@ comments: false ### Additional page metadata -Each page can have additional (optional) metadata (set in the +Each page can have additional, optional metadata (set in the [default.html](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/fc3577921343173d589dfa43d837b4307e4e620f/layouts/default.html#L30-52) -Nanoc layout), which will be shown to the top of the page if defined: - -- `author`: The name of the author of a page, usually a tutorial. It requires `author_gitlab` in order to be shown. -- `author_gitlab`: The username of the author on GitLab.com. It requires `author` in order to be shown. -- `date`: The date the page was created, usually for tutorials. -- `article_type`: The type of article. Can be either `tutorial` or `user guide`. -- `level`: The level of complexity of a how-to or tutorial. Can be either `beginner`, - `advanced`, or `intermediate`. -- `last_updated`: The date in ISO format when the page was last updated. For example `2020-02-14`. +Nanoc layout), which will be displayed at the top of the page if defined: + - `reading_time`: If you want to add an indication of the approximate reading time of a page, you can set `reading_time` to `true`. This uses a simple [algorithm](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/master/lib/helpers/reading_time.rb) @@ -335,12 +328,12 @@ You can combine one or more of the following: = link_to 'Help page', help_page_path('user/permissions'), target: '_blank' ``` -1. **Linking to a circle icon.** Usually used in settings where a long +1. **Using a question icon.** Usually used in settings where a long description cannot be used, like near checkboxes. You can basically use - any font awesome icon, but prefer the `question-circle`: + any GitLab SVG icon, but prefer the `question-o`: ```haml - = link_to icon('question-circle'), help_page_path('user/permissions') + = link_to sprite_icon('question-o'), help_page_path('user/permissions') ``` 1. **Using a button link.** Useful in places where text would be out of context @@ -462,7 +455,7 @@ If you want to know the in-depth details, here's what's really happening: [skips the test jobs](https://gitlab.com/gitlab-org/gitlab-docs/blob/8d5d5c750c602a835614b02f9db42ead1c4b2f5e/.gitlab-ci.yml#L50-55) to lower the build time. 1. Once the docs site is built, the HTML files are uploaded as artifacts. -1. A specific Runner tied only to the docs project, runs the Review App job +1. A specific runner tied only to the docs project, runs the Review App job that downloads the artifacts and uses `rsync` to transfer the files over to a location where NGINX serves them. @@ -472,7 +465,7 @@ The following GitLab features are used among others: - [Multi project pipelines](../../ci/multi_project_pipeline_graphs.md) - [Review Apps](../../ci/review_apps/index.md) - [Artifacts](../../ci/yaml/README.md#artifacts) -- [Specific Runner](../../ci/runners/README.md#prevent-a-specific-runner-from-being-enabled-for-other-projects) +- [Specific runner](../../ci/runners/README.md#prevent-a-specific-runner-from-being-enabled-for-other-projects) - [Pipelines for merge requests](../../ci/merge_request_pipelines/index.md) ## Testing @@ -701,9 +694,9 @@ To configure markdownlint within your editor, install one of the following as ap To configure Vale within your editor, install one of the following as appropriate: -- The Sublime Text [`SublimeLinter-contrib-vale` plugin](https://packagecontrol.io/packages/SublimeLinter-contrib-vale) -- The Visual Studio Code [`testthedocs.vale` extension](https://marketplace.visualstudio.com/items?itemName=testthedocs.vale) -- [Vim](https://github.com/dense-analysis/ale) +- The Sublime Text [`SublimeLinter-contrib-vale` plugin](https://packagecontrol.io/packages/SublimeLinter-contrib-vale). +- The Visual Studio Code [`errata-ai.vale-server` extension](https://marketplace.visualstudio.com/items?itemName=errata-ai.vale-server). You don't need Vale Server to use the plugin. +- [Vim](https://github.com/dense-analysis/ale). We don't use [Vale Server](https://errata-ai.github.io/vale/#using-vale-with-a-text-editor-or-another-third-party-application). @@ -736,9 +729,7 @@ document: - To disable all Vale linting rules, add a `<!-- vale off -->` tag before the text, and a `<!-- vale on -->` tag after the text. -Whenever possible, exclude only the problematic rule and line(s). In some cases, such as list items, -you may need to disable linting for the entire list until -[Vale issue #175](https://github.com/errata-ai/vale/issues/175) is resolved. +Whenever possible, exclude only the problematic rule and line(s). For more information, see [Vale's documentation](https://errata-ai.gitbook.io/vale/getting-started/markup#markup-based-configuration). @@ -750,3 +741,64 @@ code review. For docs changes in merge requests, whenever a change to files unde is made, Danger Bot leaves a comment with further instructions about the documentation process. This is configured in the `Dangerfile` in the GitLab repository under [/danger/documentation/](https://gitlab.com/gitlab-org/gitlab/tree/master/danger/documentation). + +## Automatic screenshot generator + +You can now set up an automatic screenshot generator to take and compress screenshots, with the +help of a configuration file known as **screenshot generator**. + +### Use the tool + +To run the tool on an existing screenshot generator, take the following steps: + +1. Set up the [GitLab Development Kit (GDK)](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/gitlab_docs.md). +1. Navigate to the subdirectory with your cloned GitLab repository, typically `gdk/gitlab`. +1. Make sure that your GDK database is fully migrated: `bin/rake db:migrate RAILS_ENV=development`. +1. Install pngquant, see the tool website for more info: [`pngquant`](https://pngquant.org/) +1. Run `scripts/docs_screenshots.rb spec/docs_screenshots/<name_of_screenshot_generator>.rb <milestone-version>`. +1. Identify the location of the screenshots, based on the `gitlab/doc` location defined by the `it` parameter in your script. +1. Commit the newly created screenshots. + +### Extending the tool + +To add an additional **screenshot generator**, take the following steps: + +- Locate the `spec/docs_screenshots` directory. +- Add a new file with a `_docs.rb` extension. +- Be sure to include the following bits in the file: + +```ruby +require 'spec_helper' + +RSpec.describe '<What I am taking screenshots of>', :js do + include DocsScreenshotHelpers # Helper that enables the screenshots taking mechanism + + before do + page.driver.browser.manage.window.resize_to(1366, 1024) # length and width of the page + end +``` + +- In addition, every `it` block must include the path where the screenshot is saved + +```ruby + it 'user/packages/container_registry/img/project_image_repositories_list' +``` + +#### Full page screenshots + +To take a full page screenshot simply `visit the page` and perform any expectation on real content (to have capybara wait till the page is ready and not take a white screenshot). + +#### Element screenshot + +To have the screenshot focuses few more steps are needed: + +- **find the area**: `screenshot_area = find('#js-registry-policies')` +- **scroll the area in focus**: `scroll_to screenshot_area` +- **wait for the content**: `expect(screenshot_area).to have_content 'Expiration interval'` +- **set the crop area**: `set_crop_data(screenshot_area, 20)` + +In particular `set_crop_data` accepts as arguments: a `DOM` element and a padding, the padding will be added around the element enlarging the screenshot area. + +#### Live example + +Please use `spec/docs_screenshots/container_registry_docs.rb` as a guide and as an example to create your own scripts. diff --git a/doc/development/documentation/site_architecture/index.md b/doc/development/documentation/site_architecture/index.md index 63cd9959985..5d3af6721d1 100644 --- a/doc/development/documentation/site_architecture/index.md +++ b/doc/development/documentation/site_architecture/index.md @@ -227,11 +227,12 @@ for its search function. This is how it works: there's a JavaScript snippet which initiates DocSearch by using an API key and an index name (`gitlab`) that are needed for Algolia to show the results. -NOTE: **For GitLab employees:** -The credentials to access the Algolia dashboard are stored in 1Password. If you -want to receive weekly reports of the search usage, search the Google doc with +NOTE: **For GitLab Team Members:** +If you’re a GitLab Team Member, find credentials for the Algolia dashboard +in the shared [GitLab 1Password account](https://about.gitlab.com/handbook/security/#1password-for-teams). +To receive weekly reports of the search usage, search the Google doc with title `Email, Slack, and GitLab Groups and Aliases`, search for `docsearch`, -and add a comment with your email to be added to the alias that gets the weekly +and add a comment with your email. You'll be added to the alias that gets the weekly reports. ## Monthly release process (versions) diff --git a/doc/development/documentation/site_architecture/release_process.md b/doc/development/documentation/site_architecture/release_process.md index d04d34ff786..98bb116aba6 100644 --- a/doc/development/documentation/site_architecture/release_process.md +++ b/doc/development/documentation/site_architecture/release_process.md @@ -121,11 +121,10 @@ versions (stable branches `X.Y` of the `gitlab-docs` project): pipelines succeed: NOTE: **Note:** - The `release-X-Y` branch needs to be present locally, - and you need to have switched to it, otherwise the Rake task will fail. + The `release-X-Y` branch needs to be present locally, otherwise the Rake + task will abort. ```shell - git checkout release-X-Y ./bin/rake release:dropdowns ``` diff --git a/doc/development/documentation/structure.md b/doc/development/documentation/structure.md index e13b2f4d031..e454a401a9d 100644 --- a/doc/development/documentation/structure.md +++ b/doc/development/documentation/structure.md @@ -1,45 +1,46 @@ --- +stage: none +group: Style Guide +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers description: What to include in GitLab documentation pages. --- # Documentation structure and template -This document will help you determine how to structure a page within GitLab's -documentation and what content to include. These standards help ensure consistency -and completeness throughout the documentation, and they make it easier to contribute. +Use these standards to contribute content to the GitLab documentation. Before getting started, familiarize yourself with [GitLab's Documentation guidelines](index.md) -and the section on Content in the [Style Guide](styleguide.md). +and the [Documentation Style Guide](styleguide.md). ## Components of a documentation page -Most pages will be dedicated to a specific GitLab feature or to a use case that involves -one or more features, potentially in conjunction with third-party tools. - -Every feature or use case document should include the following content in the following sequence, -with exceptions and details noted below and in the template included on this page. - -- **Title**: Top-level heading with the feature name, or a use case name, which would start with - a verb, like "Configure", "Enable", and so on. -- **Introduction**: A couple sentences about the subject matter and what's to be found -on this page. Describe what the feature or topic is, what it does, and in what context it should -be used. There is no need to add a title called "Introduction" or "Overview," because people rarely - search for these terms. Just put this information after the title. -- **Use cases**: describes real use case scenarios for that feature/configuration. -- **Requirements**: describes what software, configuration, account, or knowledge is required. -- **Instructions**: one or more sets of detailed instructions to follow. -- **Troubleshooting** guide (recommended but not required). - -For additional details on each, see the [template for new docs](#template-for-new-docs), -below. - -Note that you can include additional subsections, as appropriate, such as 'How it Works', 'Architecture', -and other logical divisions such as pre-deployment and post-deployment steps. +Most pages are dedicated to a specific GitLab feature or to a use case that +involves one or more features, potentially in conjunction with third-party tools. + +In general, each topic should include the following content, in this sequence: + +- *Metadata*: Information about the stage, group, and how to find the technical + writer for the topic. This information isn't visible in the published help. +- *Title*: A top-level heading with the feature or use case name. Choose a term + that defines the functionality and use the same term in all the resources + where the feature is mentioned. +- *Introduction*: In a few sentences beneath the title, describe what the + feature or topic is, what it does, and in what context it should be used. +- *Use cases*: Describe real user scenarios. +- *Prerequisites*: Describe the software, configuration, account, permissions, + or knowledge required to use this functionality. +- *Tasks*: Present detailed step-by-step instructions on how to use the feature. +- *Troubleshooting*: List errors and how to address them. Recommended but not + required. + +You can include additional subsections, as appropriate, such as *How it Works*, +or *Architecture*. You can also include other logical divisions, such as +pre-deployment and post-deployment tasks. ## Template for new docs -To start a new document, respect the file tree and file name guidelines, -as well as the style guidelines. Use the following template: +Follow the [folder structure and file name guidelines](styleguide.md#folder-structure-overview) +and create a new topic by using this template: ```markdown <!--Follow the Style Guide when working on this document. @@ -47,94 +48,87 @@ https://docs.gitlab.com/ee/development/documentation/styleguide.html When done, remove all of this commented-out text, except a commented-out Troubleshooting section, which, if empty, can be left in place to encourage future use.--> --- -description: "Short document description." # Up to ~200 chars long. They will be displayed +description: "Short document description." # Up to ~200 chars long. This information is displayed in Google Search snippets. It may help to write the page intro first, and then reuse it here. -stage: "Add the stage name here, and remove the quotation marks" -group: "Add the group name here, and remove the quotation marks" +stage: Add the stage name here +group: Add the group name here info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Feature Name or Use Case Name **[TIER]** (1) -<!--If writing about a use case, drop the tier, and start with a verb, +# Feature or Use Case Name **[TIER]** (1) +<!--If you are writing about a use case, start with a verb, for example, "Configure", "Implement", + the goal/scenario--> <!--For pages on newly-introduced features, add the following line. If only some aspects of the feature have been introduced, specify which parts of the feature.--> > [Introduced](link_to_issue_or_mr) in GitLab (Tier) X.Y (2). -An introduction -- without its own additional header -- goes here. -Offer a description of the feature or use case, and what to expect on this page. -(You can reuse this content, or part of it, for the front matter's `description` at the top -of this file). - -The introduction should answer the following questions: +Write a description of the feature or use case. This introduction should answer +these questions: - What is this feature or use case? - Who is it for? -- What is the context in which it is used and are there any prerequisites/requirements? -- What can the audience do with this? (Be sure to consider all applicable audiences, like - GitLab admin and developer-user.) -- What are the benefits to using this over any alternatives? +- What is the context in which it is used and are there any prerequisites or + requirements? +- What can the audience do with this? (Be sure to consider all applicable + audiences, such as GitLab admin and developer-user.) +- What are the benefits of using this over any existing alternatives? + +You can reuse this content, or part of it, for the front matter's `description` +at the top of this file. ## Use cases -Describe some use cases, typically in bulleted form. Include real-life examples for each. +Describe common use cases, typically in bulleted form. Include real-life examples +for each. -If the page itself is dedicated to a use case, this section can usually include more specific -scenarios for use (for example, variations on the main use case), but if that's not applicable, -the section can be omitted. +If the page itself is dedicated to a use case, this section usually includes more +specific scenarios for use (for example, variations on the main use case), but if +that's not applicable, you can omit this section. Examples of use cases on feature pages: + - CE and EE: [Issues](../../user/project/issues/index.md#use-cases) - CE and EE: [Merge Requests](../../user/project/merge_requests/index.md) -- EE-only: [Geo](../../administration/geo/replication/index.md) +- EE-only: [Geo](../../administration/geo/index.md) - EE-only: [Jenkins integration](../../integration/jenkins.md) -## Requirements +## Prerequisites -State any requirements for using the feature and/or following along with the instructions. +State any prerequisites for using the feature. These might include: -These can include both: -- technical requirements (for example, an account on a third party service, an amount of storage space, - prior configuration of another feature) -- prerequisite knowledge (for example, familiarity with certain GitLab features, cloud technologies) +- Technical prereqs (for example, an account on a third-party service, an amount + of storage space, or prior configuration of another feature) +- Prerequisite knowledge (for example, familiarity with certain GitLab features + or other products and technologies). Link each one to an appropriate place for more information. -## Instructions +## Tasks -This is the part of the document where you can include one or more sets of instructions. Each topic should help users accomplish a specific task. -Headers should describe the task the reader will achieve by following the instructions within, -typically starting with a verb. For example, `Create a package` or `Configure a pipeline`. +The heading should: + +- Describe the task and start with a verb. For example, `Create a package` or + `Configure a pipeline`. +- Be short and descriptive (up to ~50 chars). +- Start from an `h2` (`##`), then go over `h3`, `h4`, `h5`, and `h6` as needed. + Never skip a hierarchy level (like `h2` > `h4`). It breaks the table of + contents and can affect the breadcrumbs. -Larger instruction sets may have subsections covering specific phases of the process. -Where appropriate, provide examples of code or configuration files to better clarify -intended usage. +Bigger tasks can have subsections that explain specific phases of the process. -- Write a step-by-step guide, with no gaps between the steps. -- Include example code or configurations as part of the relevant step. - Use appropriate Markdown to wrap code blocks with - [syntax highlighting](../../user/markdown.md#colored-code-and-syntax-highlighting). -- Start with an h2 (`##`), break complex steps into small steps using - subheadings h3 > h4 > h5 > h6. _Never skip a hierarchy level, such - as h2 > h4_, as it will break the TOC and may affect the breadcrumbs. -- Use short and descriptive headings (up to ~50 chars). You can use one - single heading like `## Configure X` for instructions when the feature - is simple and the document is short. +Include example code or configurations when needed. Use Markdown to wrap code +blocks with [syntax highlighting](../../user/markdown.md#colored-code-and-syntax-highlighting). Example topic: ## Create a teddy bear -Start by writing a sentence or two about _why_ someone would want to perform this task. -It's not always possible, but is a good practice. For example: - -Create a teddy bear when you need something to hug. - -Follow this information with the task steps. +Create a teddy bear when you need something to hug. (Include the reason why you +might do the task.) To create a teddy bear: @@ -142,40 +136,40 @@ To create a teddy bear: 1. Expand **This** and click **This**. 1. Do another step. -After the numbered list, add a sentence with the expected result, if it -is not obvious, and any next steps. For example: - -The teddy bear is now in the kitchen, in the cupboard above the sink. +The teddy bear is now in the kitchen, in the cupboard above the sink. _(This is the result.)_ -You can retrieve the teddy bear and put it on the couch with the other animals. +You can retrieve the teddy bear and put it on the couch with the other animals. _(These are next steps.)_ -Screenshots are not necessary. They are difficult to keep up-to-date and can clutter the page. +Screenshots are not necessary. They are difficult to keep up-to-date and can +clutter the page. <!-- ## Troubleshooting -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. +Include any troubleshooting steps that you can foresee. If you know beforehand +what issues one might have when setting this up, or when something is changed, +or on upgrading, it's important to describe those, too. Think of things that may +go wrong and include them here. This is important to minimize requests for +Support, and to avoid documentation comments with questions that you know +someone might ask. Each scenario can be a third-level heading, for example, `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> +If you have none to add when creating a doc, leave this section in place but +commented out to help encourage others to add to it in the future. --> --- Notes: -- (1): Apply the [tier badges](styleguide.md#product-badges) accordingly +- (1): Apply the [tier badges](styleguide.md#product-badges) accordingly. - (2): Apply the correct format for the - [GitLab version that introduces the feature](styleguide.md#gitlab-versions-and-tiers) + [GitLab version that introduces the feature](styleguide.md#gitlab-versions-and-tiers). ``` ## Help and feedback section -The "help and feedback" section (introduced by [!319](https://gitlab.com/gitlab-org/gitlab-docs/-/merge_requests/319)) displayed at the end of each document -can be omitted from the doc by adding a key into the its front matter: +This section ([introduced](https://gitlab.com/gitlab-org/gitlab-docs/-/merge_requests/319) in GitLab 11.4) +is displayed at the end of each document and can be omitted by adding a key into +the front matter: ```yaml --- @@ -183,8 +177,8 @@ feedback: false --- ``` -The default is to leave it there. If you want to omit it from a document, -you must check with a technical writer before doing so. +The default is to leave it there. If you want to omit it from a document, you +must check with a technical writer before doing so. ### Disqus @@ -192,8 +186,8 @@ We also have integrated the docs site with Disqus (introduced by [!151](https://gitlab.com/gitlab-org/gitlab-docs/-/merge_requests/151)), allowing our users to post comments. -To omit only the comments from the feedback section, use the following -key on the front matter: +To omit only the comments from the feedback section, use the following key in +the front matter: ```yaml --- @@ -201,36 +195,42 @@ comments: false --- ``` -We are only hiding comments in main index pages, such as [the main documentation index](../../README.md), since its content is too broad to comment on. Before omitting Disqus, -you must check with a technical writer. +We're hiding comments only in main index pages, such as [the main documentation index](../../README.md), +since its content is too broad to comment on. Before omitting Disqus, you must +check with a technical writer. -Note that once `feedback: false` is added to the front matter, it will automatically omit +Note that after adding `feedback: false` to the front matter, it will omit Disqus, therefore, don't add both keys to the same document. -The click events in the feedback section are tracked with Google Tag Manager. The -conversions can be viewed on Google Analytics by navigating to **Behavior > Events > Top events > docs**. +The click events in the feedback section are tracked with Google Tag Manager. +The conversions can be viewed on Google Analytics by navigating to +**Behavior > Events > Top events > docs**. ## Guidelines for good practices > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/36576/) in GitLab 13.2 as GitLab Development documentation. -"Good practice" examples demonstrate encouraged ways of writing code while comparing with examples of practices to avoid. -These examples are labeled as "Bad" or "Good". -In GitLab development guidelines, when presenting the cases, it is recommended -to follow a **first-bad-then-good** strategy. First demonstrate the "Bad" practice (how things _could_ be done, which is often still working code), -and then how things _should_ be done better, using a "Good" example. This is typically an improved example of the same code. +*Good practice* examples demonstrate encouraged ways of writing code while +comparing with examples of practices to avoid. These examples are labeled as +*Bad* or *Good*. In GitLab development guidelines, when presenting the cases, +it's recommended to follow a *first-bad-then-good* strategy. First demonstrate +the *Bad* practice (how things *could* be done, which is often still working +code), and then how things *should* be done better, using a *Good* example. This +is typically an improved example of the same code. Consider the following guidelines when offering examples: -- First, offer the "Bad" example, then the "Good" one. +- First, offer the *Bad* example, and then the *Good* one. - When only one bad case and one good case is given, use the same code block. -- When more than one bad case or one good case is offered, use separated code blocks for each. -With many examples being presented, a clear separation helps the reader to go directly to the good part. -Consider offering an explanation (for example, a comment, a link to a resource, etc.) on why something is bad practice. +- When more than one bad case or one good case is offered, use separated code + blocks for each. With many examples being presented, a clear separation helps + the reader to go directly to the good part. Consider offering an explanation + (for example, a comment, or a link to a resource) on why something is bad + practice. - Better and best cases can be considered part of the good case(s) code block. -In the same code block, precede each with comments: `# Better` and `# Best`. + In the same code block, precede each with comments: `# Better` and `# Best`. NOTE: **Note:** -While the bad-then-good approach is acceptable for the GitLab development guidelines, do not use it -for user documentation. For user documentation, use "Do" and "Don't." For example, see the -[Pajamas Design System](https://design.gitlab.com/content/punctuation/). +Although the bad-then-good approach is acceptable for the GitLab development +guidelines, do not use it for user documentation. For user documentation, use +*Do* and *Don't*. For examples, see the [Pajamas Design System](https://design.gitlab.com/content/punctuation/). diff --git a/doc/development/documentation/styleguide.md b/doc/development/documentation/styleguide.md index c252f6425d0..984c64b9e9e 100644 --- a/doc/development/documentation/styleguide.md +++ b/doc/development/documentation/styleguide.md @@ -1,69 +1,109 @@ --- +stage: none +group: Style Guide +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers description: 'Writing styles, markup, formatting, and other standards for GitLab Documentation.' --- # Documentation Style Guide -This document defines the standards for GitLab's documentation content and files. +This document defines the standards for GitLab's documentation content and +files. For broader information about the documentation, see the [Documentation guidelines](index.md). -For programmatic help adhering to the guidelines, see [Testing](index.md#testing). +For guidelines specific to text in the GitLab interface, see the Pajamas [Content](https://design.gitlab.com/content/error-messages) section. -See the GitLab handbook for further [writing style guidelines](https://about.gitlab.com/handbook/communication/#writing-style-guidelines) -that apply to all GitLab content, not just documentation. +For information on how to validate styles locally or by using GitLab CI/CD, see [Testing](index.md#testing). -View [a list of recent style guide updates](https://gitlab.com/dashboard/merge_requests?scope=all&utf8=%E2%9C%93&state=merged&label_name[]=tw-style¬[label_name][]=docs%3A%3Afix). +Use this guide for standards on grammar, formatting, word usage, and more. + +You can also view a list of [recent updates to this guide](https://gitlab.com/dashboard/merge_requests?scope=all&utf8=%E2%9C%93&state=merged&label_name[]=tw-style¬[label_name][]=docs%3A%3Afix). + +If you can't find what you need: + +- View the GitLab Handbook for [writing style guidelines](https://about.gitlab.com/handbook/communication/#writing-style-guidelines) that apply to all GitLab content. +- Refer to one of the following: + + - [Microsoft Style Guide](https://docs.microsoft.com/en-us/style-guide/). + - [Google Developer Documentation Style Guide](https://developers.google.com/style). + +If you have questions about style, mention `@tw-style` in an issue or merge request, or, if you have access to the GitLab Slack workspace, use `#docs-process`. ## Documentation is the single source of truth (SSOT) ### Why a single source of truth -The documentation of GitLab products and features is the SSOT for all information related to implementation, usage, and troubleshooting. It evolves continuously, in keeping with new products and features, and with improvements for clarity, accuracy, and completeness. +The documentation of GitLab products and features is the SSOT for all +information related to implementation, usage, and troubleshooting. It evolves +continuously, in keeping with new products and features, and with improvements +for clarity, accuracy, and completeness. -This policy prevents information silos, making it easier to find information about GitLab products. +This policy prevents information silos, making it easier to find information +about GitLab products. -It also informs decisions about the kinds of content we include in our documentation. +It also informs decisions about the kinds of content we include in our +documentation. ### All information -Include problem-solving actions that may address rare cases or be considered 'risky', so long as proper context is provided in the form of fully detailed warnings and caveats. This kind of content should be included as it could be helpful to others and, when properly explained, its benefits outweigh the risks. If you think you have found an exception to this rule, contact the Technical Writing team. +Include problem-solving actions that may address rare cases or be considered +*risky*, so long as proper context is provided in the form of fully detailed +warnings and caveats. This kind of content should be included as it could be +helpful to others and, when properly explained, its benefits outweigh the risks. +If you think you have found an exception to this rule, contact the +Technical Writing team. -We will add all troubleshooting information to the documentation, no matter how unlikely a user is to encounter a situation. -For the [Troubleshooting sections](#troubleshooting), people in GitLab Support can merge additions themselves. +We will add all troubleshooting information to the documentation, no matter how +unlikely a user is to encounter a situation. For the [Troubleshooting sections](#troubleshooting), +people in GitLab Support can merge additions themselves. ### All media types -Include any media types/sources if the content is relevant to readers. You can freely include or link presentations, diagrams, videos, and so on; no matter who it was originally composed for, if it is helpful to any of our audiences, we can include it. +Include any media types/sources if the content is relevant to readers. You can +freely include or link presentations, diagrams, videos, and so on; no matter who +it was originally composed for, if it is helpful to any of our audiences, we can +include it. -- If you use an image that has a separate source file (for example, a vector or diagram format), link the image to the source file so that it may be reused or updated by anyone. -- Do not copy and paste content from other sources unless it is a limited quotation with the source cited. Typically it is better to either rephrase relevant information in your own words or link out to the other source. +- If you use an image that has a separate source file (for example, a vector or + diagram format), link the image to the source file so that it may be reused or + updated by anyone. +- Do not copy and paste content from other sources unless it is a limited + quotation with the source cited. Typically it is better to either rephrase + relevant information in your own words or link out to the other source. ### No special types -In the software industry, it is a best practice to organize documentation in different types. For example, [Divio recommends](https://www.divio.com/blog/documentation/): +In the software industry, it is a best practice to organize documentation in +different types. For example, [Divio recommends](https://www.divio.com/blog/documentation/): -1. Tutorials -1. How-to guides -1. Explanation -1. Reference (for example, a glossary) +- Tutorials +- How-to guides +- Explanation +- Reference (for example, a glossary) -At GitLab, we have so many product changes in our monthly releases that we can't afford to continuously update multiple types of information. -If we have multiple types, the information will become outdated. Therefore, we have a [single template](structure.md) for documentation. +At GitLab, we have so many product changes in our monthly releases that we can't +afford to continuously update multiple types of information. If we have multiple +types, the information will become outdated. Therefore, we have a +[single template](structure.md) for documentation. -We currently do not distinguish specific document types, although we are open to reconsidering this policy -once the documentation has reached a future stage of maturity and quality. If you are reading this, then despite our -continuous improvement efforts, that point hasn't been reached. +We currently do not distinguish specific document types, although we are open to +reconsidering this policy after the documentation has reached a future stage of +maturity and quality. If you are reading this, then despite our continuous +improvement efforts, that point hasn't been reached. ### Link instead of summarize -There is a temptation to summarize the information on another page. -This will cause the information to live in two places. -Instead, link to the SSOT and explain why it is important to consume the information. +There is a temptation to summarize the information on another page. This will +cause the information to live in two places. Instead, link to the single source +of truth and explain why it is important to consume the information. ### Organize by topic, not by type -Beyond top-level audience-type folders (for example, `administration`), we organize content by topic, not by type, so it can be located as easily as possible within the single-source-of-truth (SSOT) section for the subject matter. +Beyond top-level audience-type folders (for example, `administration`), we +organize content by topic, not by type, so it can be located as easily as +possible within the single-source-of-truth (SSOT) section for the subject +matter. For example, do not create groupings of similar media types. For example: @@ -71,46 +111,69 @@ For example, do not create groupings of similar media types. For example: - FAQs. - Sets of all articles or videos. -Such grouping of content by type makes -it difficult to browse for the information you need and difficult to maintain up-to-date content. -Instead, organize content by its subject (for example, everything related to CI goes together) -and cross-link between any related content. +Such grouping of content by type makes it difficult to browse for the information +you need and difficult to maintain up-to-date content. Instead, organize content +by its subject (for example, everything related to CI goes together) and +cross-link between any related content. ### Docs-first methodology -We employ a **docs-first methodology** to help ensure the docs remain a complete and trusted resource, and to make communicating about the use of GitLab more efficient. +We employ a *documentation-first methodology* to help ensure the documentation +remains a complete and trusted resource, and to make communicating about the use +of GitLab more efficient. -- If the answer to a question exists in documentation, share the link to the docs instead of rephrasing the information. -- When you encounter new information not available in GitLab’s documentation (for example, when working on a support case or testing a feature), your first step should be to create a merge request (MR) to add this information to the docs. You can then share the MR in order to communicate this information. +- If the answer to a question exists in documentation, share the link to the + documentation instead of rephrasing the information. +- When you encounter new information not available in GitLab’s documentation (for + example, when working on a support case or testing a feature), your first step + should be to create a merge request (MR) to add this information to the + documentation. You can then share the MR in order to communicate this + information. -New information that would be useful toward the future usage or troubleshooting of GitLab should not be written directly in a forum or other messaging system, but added to a docs MR and then referenced, as described above. Note that among any other doc changes, you can either: +New information that would be useful toward the future usage or troubleshooting +of GitLab should not be written directly in a forum or other messaging system, +but added to a documentation MR and then referenced, as described above. Note +that among any other documentation changes, you can either: - Add a [Troubleshooting section](#troubleshooting) to a doc if none exists. -- Un-comment and use the placeholder Troubleshooting section included as part of our [doc template](structure.md#template-for-new-docs), if present. +- Un-comment and use the placeholder Troubleshooting section included as part of + our [documentation template](structure.md#template-for-new-docs), if present. -The more we reflexively add useful information to the docs, the more (and more successfully) the docs will be used to efficiently accomplish tasks and solve problems. +The more we reflexively add useful information to the documentation, the more +(and more successfully) the documentation will be used to efficiently accomplish +tasks and solve problems. -If you have questions when considering, authoring, or editing docs, ask the Technical Writing team on Slack in `#docs` or in GitLab by mentioning the writer for the applicable [DevOps stage](https://about.gitlab.com/handbook/product/product-categories/#devops-stages). Otherwise, forge ahead with your best effort. It does not need to be perfect; the team is happy to review and improve upon your content. Please review the [Documentation guidelines](index.md) before you begin your first documentation MR. +If you have questions when considering, authoring, or editing documentation, ask +the Technical Writing team on Slack in `#docs` or in GitLab by mentioning the +writer for the applicable [DevOps stage](https://about.gitlab.com/handbook/product/product-categories/#devops-stages). +Otherwise, forge ahead with your best effort. It does not need to be perfect; +the team is happy to review and improve upon your content. Please review the +[Documentation guidelines](index.md) before you begin your first documentation MR. -Having a knowledge base in any form that is separate from the documentation would be against the docs-first methodology because the content would overlap with the documentation. +Having a knowledge base in any form that is separate from the documentation would +be against the documentation-first methodology because the content would overlap with +the documentation. ## Markdown All GitLab documentation is written using [Markdown](https://en.wikipedia.org/wiki/Markdown). -The [documentation website](https://docs.gitlab.com) uses GitLab Kramdown as its Markdown rendering engine. For a complete Kramdown reference, see the [GitLab Markdown Kramdown Guide](https://about.gitlab.com/handbook/markdown-guide/). +The [documentation website](https://docs.gitlab.com) uses GitLab Kramdown as its +Markdown rendering engine. For a complete Kramdown reference, see the +[GitLab Markdown Kramdown Guide](https://about.gitlab.com/handbook/markdown-guide/). -The [`gitlab-kramdown`](https://gitlab.com/gitlab-org/gitlab_kramdown) -Ruby gem will support all [GFM markup](../../user/markdown.md) in the future. That is, -all markup supported for display in the GitLab application itself. For now, -use regular Markdown markup, following the rules in the linked style guide. +The [`gitlab-kramdown`](https://gitlab.com/gitlab-org/gitlab_kramdown) Ruby gem +will support all [GFM markup](../../user/markdown.md) in the future. That is, +all markup supported for display in the GitLab application itself. For now, use +regular Markdown markup, following the rules in the linked style guide. -Note that Kramdown-specific markup (for example, `{:.class}`) will not render properly on GitLab instances under [`/help`](index.md#gitlab-help). +Note that Kramdown-specific markup (for example, `{:.class}`) will not render +properly on GitLab instances under [`/help`](index.md#gitlab-help). ### HTML in Markdown -Hard-coded HTML is valid, although it's discouraged from being used while we have `/help`. -HTML is permitted as long as: +Hard-coded HTML is valid, although it's discouraged from being used while we +have `/help`. HTML is permitted as long as: - There's no equivalent markup in Markdown. - Advanced tables are necessary. @@ -120,23 +183,23 @@ HTML is permitted as long as: ### Markdown Rules GitLab ensures that the Markdown used across all documentation is consistent, as -well as easy to review and maintain, by [testing documentation changes](index.md#testing) with -[markdownlint](index.md#markdownlint). This lint test fails when any document has an issue -with Markdown formatting that may cause the page to render incorrectly within GitLab. -It will also fail when a document is using non-standard Markdown (which may render -correctly, but is not the current standard for GitLab documentation). +well as easy to review and maintain, by [testing documentation changes](index.md#testing) +with [markdownlint](index.md#markdownlint). This lint test fails when any +document has an issue with Markdown formatting that may cause the page to render +incorrectly within GitLab. It will also fail when a document is using +non-standard Markdown (which may render correctly, but is not the current +standard for GitLab documentation). #### Markdown rule `MD044/proper-names` (capitalization) -A rule that could cause confusion is `MD044/proper-names`, as it might not be immediately -clear what caused markdownlint to fail, or how to correct the failure. This rule -checks a list of known words, listed in the `.markdownlint.json` file in each project, -to verify proper use of capitalization and backticks. Words in backticks will -be ignored by markdownlint. +A rule that could cause confusion is `MD044/proper-names`, as it might not be +immediately clear what caused markdownlint to fail, or how to correct the +failure. This rule checks a list of known words, listed in the `.markdownlint.json` +file in each project, to verify proper use of capitalization and backticks. +Words in backticks will be ignored by markdownlint. -In general, product names should follow the exact capitalization of the official names -of the products, protocols, and so on. -See [`.markdownlint.json`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.markdownlint.json) +In general, product names should follow the exact capitalization of the official +names of the products, protocols, and so on. See [`.markdownlint.json`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.markdownlint.json) for the words tested for proper capitalization in GitLab documentation. Some examples fail if incorrect capitalization is used: @@ -145,60 +208,66 @@ Some examples fail if incorrect capitalization is used: - NGINX (needs all capitals) - runit (needs lowercase `r`) -Additionally, commands, parameters, values, filenames, and so on must be included in backticks. -For example: +Additionally, commands, parameters, values, filenames, and so on must be +included in backticks. For example: - "Change the `needs` keyword in your `.gitlab.yml`..." - - `needs` is a parameter, and `.gitlab.yml` is a file, so both need backticks. Additionally, - `.gitlab.yml` will fail markdownlint without backticks as it does not have capital G or L. + - `needs` is a parameter, and `.gitlab.yml` is a file, so both need backticks. + Additionally, `.gitlab.yml` will fail markdownlint without backticks as it + does not have capital G or L. - "Run `git clone` to clone a Git repository..." - - `git clone` is a command, so it must be lowercase, while Git is the product, so - it must have a capital G. + - `git clone` is a command, so it must be lowercase, while Git is the product, + so it must have a capital G. ## Structure -Because we want documentation to be a SSOT, we should [organize by topic, not by type](#organize-by-topic-not-by-type). +Because we want documentation to be a SSOT, we should [organize by topic, not by +type](#organize-by-topic-not-by-type). ### Folder structure overview The documentation is separated by top-level audience folders [`user`](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/doc/user), -[`administration`](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/doc/administration), and [`development`](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/doc/development) (contributing) folders. +[`administration`](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/doc/administration), +and [`development`](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/doc/development) +(contributing) folders. -Beyond that, we primarily follow the structure of the GitLab user interface or API. +Beyond that, we primarily follow the structure of the GitLab user interface or +API. -Our goal is to have a clear hierarchical structure with meaningful URLs -like `docs.gitlab.com/user/project/merge_requests/`. With this pattern, -you can immediately tell that you are navigating to user-related documentation -about Project features; specifically about Merge Requests. Our site's paths match +Our goal is to have a clear hierarchical structure with meaningful URLs like +`docs.gitlab.com/user/project/merge_requests/`. With this pattern, you can +immediately tell that you are navigating to user-related documentation about +Project features; specifically about Merge Requests. Our site's paths match those of our repository, so the clear structure also makes documentation easier to update. The table below shows what kind of documentation goes where. -| Directory | What belongs here | -|:----------------------|:---------------------------------------------------------------------------------------------------------------------------------| -| `doc/user/` | User related documentation. Anything that can be done within the GitLab UI goes here, including usage of the `/admin` interface. | +| Directory | What belongs here | +|:----------------------|:----------------------------------------------------------------------------------------------------------------------------------------| +| `doc/user/` | User related documentation. Anything that can be done within the GitLab user interface goes here, including usage of the `/admin` interface. | | `doc/administration/` | Documentation that requires the user to have access to the server where GitLab is installed. The admin settings that can be accessed via GitLab's interface exist under `doc/user/admin_area/`. | -| `doc/api/` | API related documentation. | -| `doc/development/` | Documentation related to the development of GitLab, whether contributing code or docs. Related process and style guides should go here. | -| `doc/legal/` | Legal documents about contributing to GitLab. | -| `doc/install/` | Contains instructions for installing GitLab. | -| `doc/update/` | Contains instructions for updating GitLab. | -| `doc/topics/` | Indexes per topic (`doc/topics/topic-name/index.md`): all resources for that topic. | +| `doc/api/` | API related documentation. | +| `doc/development/` | Documentation related to the development of GitLab, whether contributing code or documentation. Related process and style guides should go here. | +| `doc/legal/` | Legal documents about contributing to GitLab. | +| `doc/install/` | Contains instructions for installing GitLab. | +| `doc/update/` | Contains instructions for updating GitLab. | +| `doc/topics/` | Indexes per topic (`doc/topics/topic-name/index.md`): all resources for that topic. | ### Work with directories and files 1. When you create a new directory, always start with an `index.md` file. - Do not use another file name and **do not** create `README.md` files. -1. **Do not** use special characters and spaces, or capital letters in file names, - directory names, branch names, and anything that generates a path. -1. When creating a new document and it has more than one word in its name, - make sure to use underscores instead of spaces or dashes (`-`). For example, - a proper naming would be `import_projects_from_github.md`. The same rule - applies to images. + Do not use another file name and *do not* create `README.md` files. +1. *Do not* use special characters and spaces, or capital letters in file + names, directory names, branch names, and anything that generates a path. +1. When creating or renaming a file or directory and it has more than one word + in its name, use underscores (`_`) instead of spaces or dashes. For example, + proper naming would be `import_project/import_from_github.md`. This applies + to both image files and Markdown files. 1. For image files, do not exceed 100KB. 1. Do not upload video files to the product repositories. [Link or embed videos](#videos) instead. -1. There are four main directories, `user`, `administration`, `api` and `development`. +1. There are four main directories: `user`, `administration`, `api`, and + `development`. 1. The `doc/user/` directory has five main subdirectories: `project/`, `group/`, `profile/`, `dashboard/` and `admin_area/`. 1. `doc/user/project/` should contain all project related documentation. @@ -216,34 +285,43 @@ The table below shows what kind of documentation goes where. the **Visibility and Access Controls** category should have a document located at `doc/user/admin_area/settings/visibility_and_access_controls.md`. 1. The `doc/topics/` directory holds topic-related technical content. Create - `doc/topics/topic-name/subtopic-name/index.md` when subtopics become necessary. + `doc/topics/topic_name/subtopic_name/index.md` when subtopics become necessary. General user- and admin- related documentation, should be placed accordingly. -1. The directories `/workflow/`, `/university/`, and `/articles/` have - been **deprecated** and the majority their docs have been moved to their correct location - in small iterations. +1. The directories `/workflow/`, `/university/`, and `/articles/` have been + *deprecated* and the majority their documentation has been moved to their + correct location in small iterations. -If you are unsure where a document or a content addition should live, this should +If you are unsure where to place a document or a content addition, this should not stop you from authoring and contributing. You can use your best judgment and -then ask the reviewer of your MR to confirm your decision, and/or ask a technical writer -at any stage in the process. The technical writing team will review all documentation -changes, regardless, and can move content if there is a better place for it. +then ask the reviewer of your MR to confirm your decision, and/or ask a +technical writer at any stage in the process. The technical writing team will +review all documentation changes, regardless, and can move content if there is a +better place for it. ### Avoid duplication -Do not include the same information in multiple places. [Link to a SSOT instead.](#link-instead-of-summarize) +Do not include the same information in multiple places. +[Link to a single source of truth instead.](#link-instead-of-summarize) ### References across documents -- Give each folder an index.md page that introduces the topic, introduces the pages within, and links to the pages within (including to the index pages of any next-level subpaths). -- To ensure discoverability, ensure each new or renamed doc is linked from its higher-level index page and other related pages. -- When making reference to other GitLab products and features, link to their respective docs, at least on first mention. -- When making reference to third-party products or technologies, link out to their external sites, documentation, and resources. +- Give each folder an `index.md` page that introduces the topic, introduces the + pages within, and links to the pages within (including to the index pages of + any next-level subpaths). +- To ensure discoverability, ensure each new or renamed doc is linked from its + higher-level index page and other related pages. +- When making reference to other GitLab products and features, link to their + respective documentation, at least on first mention. +- When making reference to third-party products or technologies, link out to + their external sites, documentation, and resources. ### Structure within documents -- Include any and all applicable subsections as described on the [structure and template](structure.md) page. -- Structure content in alphabetical order in tables, lists, and so on, unless there is - a logical reason not to (for example, when mirroring the UI or an otherwise ordered sequence). +- Include any and all applicable subsections as described on the + [structure and template](structure.md) page. +- Structure content in alphabetical order in tables, lists, and so on, unless + there's a logical reason not to (for example, when mirroring the user + interface or an otherwise ordered sequence). ## Language @@ -255,9 +333,9 @@ GitLab documentation should be clear and easy to understand. ### Point of view -In most cases, it’s appropriate to use the second-person (you, yours) point of view, -because it’s friendly and easy to understand. -(Tested in [`FirstPerson.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/FirstPerson.yml).) +In most cases, it’s appropriate to use the second-person (you, yours) point of +view, because it’s friendly and easy to understand. (Tested in +[`FirstPerson.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/FirstPerson.yml).) <!-- How do we harmonize the second person in Pajamas with our first person plural in our doc guide? --> @@ -272,16 +350,18 @@ Use sentence case. For example: #### UI text -When referring to specific user interface text, like a button label or menu item, use the same capitalization that is displayed in the UI. -Standards for this content are listed in the [Pajamas Design System Content section](https://design.gitlab.com/content/punctuation/) and typically -match what is called for in this Documentation Style Guide. +When referring to specific user interface text, like a button label or menu +item, use the same capitalization that is displayed in the user interface. +Standards for this content are listed in the [Pajamas Design System Content section](https://design.gitlab.com/content/punctuation/) +and typically match what is called for in this Documentation Style Guide. -If you think there is a mistake in the way the UI text is styled, -create an issue or an MR to propose a change to the UI text. +If you think there is a mistake in the way the user interface text is styled, +create an issue or an MR to propose a change to the user interface text. #### Feature names -- **Feature names are typically lowercase**, like those describing actions and types of objects in GitLab. For example: +- *Feature names are typically lowercase*, like those describing actions and + types of objects in GitLab. For example: - epics - issues - issue weights @@ -289,7 +369,9 @@ create an issue or an MR to propose a change to the UI text. - milestones - reorder issues - runner, runners, shared runners -- **Some features are capitalized**, typically nouns naming GitLab-specific capabilities or tools. For example: + - a to-do, to-dos +- *Some features are capitalized*, typically nouns naming GitLab-specific + capabilities or tools. For example: - GitLab CI/CD - Repository Mirroring - Value Stream Analytics @@ -298,48 +380,67 @@ create an issue or an MR to propose a change to the UI text. - Geo - GitLab Runner (see [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/233529) for details) -Document any exceptions in this style guide. If you're not sure, ask a GitLab Technical Writer so that they can help decide and document the result. +Document any exceptions in this style guide. If you're not sure, ask a GitLab +Technical Writer so that they can help decide and document the result. -Do not match the capitalization of terms or phrases on the [Features page](https://about.gitlab.com/features/) or [features.yml](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/features.yml) by default. +Do not match the capitalization of terms or phrases on the [Features page](https://about.gitlab.com/features/) +or [features.yml](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/features.yml) +by default. #### Other terms Capitalize names of: -- GitLab [product tiers](https://about.gitlab.com/pricing/). For example, GitLab Core - and GitLab Ultimate. (Tested in [`BadgeCapitalization.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/BadgeCapitalization.yml).) -- Third-party organizations, software, and products. For example, Prometheus, Kubernetes, Git, and The Linux Foundation. -- Methods or methodologies. For example, Continuous Integration, Continuous Deployment, Scrum, and Agile. (Tested in [`.markdownlint.json`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.markdownlint.json).) +- GitLab [product tiers](https://about.gitlab.com/pricing/). For example, + GitLab Core and GitLab Ultimate. (Tested in [`BadgeCapitalization.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/BadgeCapitalization.yml).) +- Third-party organizations, software, and products. For example, Prometheus, + Kubernetes, Git, and The Linux Foundation. +- Methods or methodologies. For example, Continuous Integration, + Continuous Deployment, Scrum, and Agile. (Tested in [`.markdownlint.json`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.markdownlint.json).) + +Follow the capitalization style listed at the [authoritative source](#links-to-external-documentation) +for the entity, which may use non-standard case styles. For example: GitLab and +npm. -Follow the capitalization style listed at the [authoritative source](#links-to-external-documentation) for the entity, which may use non-standard case styles. For example: GitLab and npm. +Use forms of *sign in*, instead of *log in* or *login*. For example: + +- Sign in to GitLab. +- Open the sign-in page. + +Exceptions to this rule include the concept of *single sign-on* and +references to user interface elements. For example: + +- To sign in to product X, enter your credentials, and then click **Log in**. ### Inclusive language -We strive to create documentation that is inclusive. This section includes guidance and examples in the -following categories: +We strive to create documentation that is inclusive. This section includes +guidance and examples in the following categories: - [Gender-specific wording](#avoid-gender-specific-wording). - [Ableist language](#avoid-ableist-language). - [Cultural sensitivity](#culturally-sensitive-language). -We write our developer documentation with inclusivity and diversity in mind. This page is not an exhaustive reference, but describes some general guidelines and examples that illustrate some best practices to follow. +We write our developer documentation with inclusivity and diversity in mind. This +page is not an exhaustive reference, but describes some general guidelines and +examples that illustrate some best practices to follow. #### Avoid gender-specific wording When possible, use gender-neutral pronouns. For example, you can use a singular -[they](https://developers.google.com/style/pronouns#gender-neutral-pronouns) as a gender-neutral -pronoun. +[they](https://developers.google.com/style/pronouns#gender-neutral-pronouns) as +a gender-neutral pronoun. Avoid the use of gender-specific pronouns, unless referring to a specific person. -| Use | Avoid | -|-----------------------------------|-----------------| -| People, humanity | Mankind | -| GitLab Team Members | Manpower | +| Use | Avoid | +|-----------------------------------|---------------------------------| +| People, humanity | Mankind | +| GitLab Team Members | Manpower | | You can install; They can install | He can install; She can install | -If you need to set up [Fake user information](#fake-user-information), use diverse or non-gendered -names with common surnames. +If you need to set up [Fake user information](#fake-user-information), use +diverse or non-gendered names with common surnames. #### Avoid ableist language @@ -354,11 +455,14 @@ Avoid terms that are also used in negative stereotypes for different groups. | Active/Inactive | Enabled/Disabled | | On/Off | Enabled/Disabled | -Credit: [Avoid ableist language](https://developers.google.com/style/inclusive-documentation#ableist-language) in the Google Developer Style Guide. +Credit: [Avoid ableist language](https://developers.google.com/style/inclusive-documentation#ableist-language) +in the Google Developer Style Guide. #### Culturally sensitive language -Avoid terms that reflect negative cultural stereotypes and history. In most cases, you can replace terms such as `master` and `slave` with terms that are more precise and functional, such as `primary` and `secondary`. +Avoid terms that reflect negative cultural stereotypes and history. In most +cases, you can replace terms such as `master` and `slave` with terms that are +more precise and functional, such as `primary` and `secondary`. | Use | Avoid | |----------------------|-----------------------| @@ -372,8 +476,10 @@ For more information see the following [Internet Draft specification](https://to When creating documentation, limit or avoid the use of the following verb tenses, words, and phrases: -- Avoid jargon when possible, and when not possible, define the term or [link to a definition](#links-to-external-documentation). -- Avoid uncommon words when a more-common alternative is possible, ensuring that content is accessible to more readers. +- Avoid jargon when possible, and when not possible, define the term or + [link to a definition](#links-to-external-documentation). +- Avoid uncommon words when a more-common alternative is possible, ensuring that + content is accessible to more readers. - Don't write in the first person singular. (Tested in [`FirstPerson.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/FirstPerson.yml).) - Instead of "I" or "me," use "we," "you," "us," or "one." @@ -403,7 +509,7 @@ tenses, words, and phrases: - Instead of "e.g.," use "for example," "such as," "for instance," or "like." - Instead of "etc.," either use "and so on" or consider editing it out, since it can be vague. - <!-- vale gitlab.rulename = NO --> + <!-- vale gitlab.LatinTerms = YES --> - Avoid using the word *currently* when talking about the product or its features. The documentation describes the product as it is, and not as it will be at some indeterminate point in the future. @@ -435,64 +541,46 @@ tenses, words, and phrases: ### Contractions -- Use common contractions when it helps create a friendly and informal tone, especially in tutorials, instructional documentation, and [UIs](https://design.gitlab.com/content/punctuation/#contractions). (Tested in [`Contractions.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/Contractions.yml).) - - | Do | Don't | - |----------|-----------| - | it's | it is | - | can't | cannot | - | wouldn't | would not | - | you're | you are | - | you've | you have | - | haven't | have not | - | don't | do not | - | we're | we are | - | that's | that is | - | won't | will not | - -- Avoid less common contractions: - - | Do | Don't | - |--------------|-------------| - | he would | he'd | - | it will | it'll | - | should have | should've | - | there would | there'd | +Contractions are encouraged, and can create a friendly and informal tone, +especially in tutorials, instructional documentation, and +[user interfaces](https://design.gitlab.com/content/punctuation/#contractions). + +Some contractions, however, should be avoided: - Do not use contractions with a proper noun and a verb. For example: - | Do | Don't | - |----------------------|---------------------| - | GitLab is creating X | GitLab's creating X | + | Do | Don't | + |------------------------------------------|-----------------------------------------| + | GitLab is creating X. | GitLab's creating X. | - Do not use contractions when you need to emphasize a negative. For example: - | Do | Don't | - |-----------------------------|----------------------------| - | Do **not** install X with Y | **Don't** install X with Y | + | Do | Don't | + |------------------------------------------|-----------------------------------------| + | Do *not* install X with Y. | *Don't* install X with Y. | - Do not use contractions in reference documentation. For example: - | Do | Don't | - |------------------------------------------|----------------------------| - | Do **not** set a limit greater than 1000 | **Don't** set a limit greater than 1000 | - | For `parameter1`, the default is 10 | For `parameter1`, the default's 10 | + | Do | Don't | + |------------------------------------------|-----------------------------------------| + | Do *not* set a limit greater than 1000. | *Don't* set a limit greater than 1000. | + | For `parameter1`, the default is 10. | For `parameter1`, the default's 10. | - Avoid contractions in error messages. Examples: - | Do | Don't | - |------------------------------------------|----------------------------| - | Requests to localhost are not allowed | Requests to localhost aren't allowed | - | Specified URL cannot be used | Specified URL can't be used | - -<!-- vale on --> + | Do | Don't | + |------------------------------------------|-----------------------------------------| + | Requests to localhost are not allowed. | Requests to localhost aren't allowed. | + | Specified URL cannot be used. | Specified URL can't be used. | ## Text - [Write in Markdown](#markdown). -- Splitting long lines (preferably up to 100 characters) can make it easier to provide feedback on small chunks of text. +- Splitting long lines (preferably up to 100 characters) can make it easier to + provide feedback on small chunks of text. - Insert an empty line for new paragraphs. -- Insert an empty line between different markups (for example, after every paragraph, header, list, and so on). Example: +- Insert an empty line between different markups (for example, after every + paragraph, header, list, and so on). Example: ```markdown ## Header @@ -511,27 +599,27 @@ tenses, words, and phrases: ### Punctuation -Check the general punctuation rules for the GitLab documentation on the table below. -Check specific punctuation rules for [lists](#lists) below. +Review the general punctuation rules for the GitLab documentation in the +following table. Check specific punctuation rules for [lists](#lists) below. Additional examples are available in the [Pajamas guide for punctuation](https://design.gitlab.com/content/punctuation/). -| Rule | Example | -| ---- | ------- | -| Always end full sentences with a period. | _For a complete overview, read through this document._| +| Rule | Example | +|------------------------------------------------------------------|--------------------------------------------------------| +| Always end full sentences with a period. | _For a complete overview, read through this document._ | | Always add a space after a period when beginning a new sentence. | _For a complete overview, check this doc. For other references, check out this guide._ | | Do not use double spaces. (Tested in [`SentenceSpacing.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/SentenceSpacing.yml).) | --- | | Do not use tabs for indentation. Use spaces instead. You can configure your code editor to output spaces instead of tabs when pressing the tab key. | --- | | Use serial commas ("Oxford commas") before the final 'and/or' in a list. (Tested in [`OxfordComma.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/OxfordComma.yml).) | _You can create new issues, merge requests, and milestones._ | | Always add a space before and after dashes when using it in a sentence (for replacing a comma, for example). | _You should try this - or not._ | -| Always use lowercase after a colon. | _Related Issues: a way to create a relationship between issues._ | +| Always use lowercase after a colon. | _Related Issues: a way to create a relationship between issues._ | ### Placeholder text -Often in examples, a writer will provide a command or configuration that is complete apart from -a value specific to the reader. +Often in examples, a writer will provide a command or configuration that +uses values specific to the reader. -In these cases, use [`<` and `>`](https://en.wikipedia.org/wiki/Usage_message#Pattern) to call out -where a reader must replace text with their own value. +In these cases, use [`<` and `>`](https://en.wikipedia.org/wiki/Usage_message#Pattern) +to call out where a reader must replace text with their own value. For example: @@ -539,10 +627,22 @@ For example: cp <your_source_directory> <your_destination_directory> ``` +### Keyboard commands + +Use the HTML `<kbd>` tag when referring to keystroke presses. For example: + +```plaintext +To stop the command, press <kbd>Ctrl</kbd>+<kbd>C</kbd>. +``` + +When the docs are generated, the output is: + +To stop the command, press <kbd>Ctrl</kbd>+<kbd>C</kbd>. + ## Lists -- Always start list items with a capital letter, unless they are parameters or commands - that are in backticks, or similar. +- Always start list items with a capital letter, unless they are parameters or + commands that are in backticks, or similar. - Always leave a blank line before and after a list. - Begin a line with spaces (not tabs) to denote a [nested sub-item](#nesting-inside-a-list-item). @@ -573,19 +673,19 @@ This is a list of available features: ### Markup - Use dashes (`-`) for unordered lists instead of asterisks (`*`). -- Prefix `1.` to every item in an ordered list. - When rendered, the list items will appear with sequential numbering automatically. +- Prefix `1.` to every item in an ordered list. When rendered, the list items + will appear with sequential numbering automatically. ### Punctuation - Do not add commas (`,`) or semicolons (`;`) to the end of list items. -- Only add periods to the end of a list item if the item consists of a complete sentence. - The [definition of full sentence](https://www2.le.ac.uk/offices/ld/all-resources/writing/grammar/grammar-guides/sentence) +- Only add periods to the end of a list item if the item consists of a complete + sentence. The [definition of full sentence](https://www2.le.ac.uk/offices/ld/all-resources/writing/grammar/grammar-guides/sentence) is: _"a complete sentence always contains a verb, expresses a complete idea, and makes sense standing alone"_. -- Be consistent throughout the list: if the majority of the items do not end in a period, - do not end any of the items in a period, even if they consist of a complete sentence. - The opposite is also valid: if the majority of the items end with a period, end - all with a period. +- Be consistent throughout the list: if the majority of the items do not end in + a period, do not end any of the items in a period, even if they consist of a + complete sentence. The opposite is also valid: if the majority of the items + end with a period, end all with a period. - Separate list items from explanatory text with a colon (`:`). For example: ```markdown @@ -623,16 +723,17 @@ Don't (vary use of periods; majority rules): ### Nesting inside a list item -It is possible to nest items under a list item, so that they render with the same indentation -as the list item. This can be done with: +It's possible to nest items under a list item, so that they render with the same +indentation as the list item. This can be done with: - [Code blocks](#code-blocks) - [Blockquotes](#blockquotes) - [Alert boxes](#alert-boxes) - [Images](#images) -Items nested in lists should always align with the first character of the list item. -In unordered lists (using `-`), this means two spaces for each level of indentation: +Items nested in lists should always align with the first character of the list +item. In unordered lists (using `-`), this means two spaces for each level of +indentation: ````markdown - Unordered list item 1 @@ -678,8 +779,9 @@ For ordered lists, use three spaces for each level of indentation:  ```` -You can nest full lists inside other lists using the same rules as above. If you wish -to mix types, that is also possible, as long as you don't mix items at the same level: +You can nest full lists inside other lists using the same rules as above. If you +want to mix types, that is also possible, as long as you don't mix items at the +same level: ```markdown 1. Ordered list item one. @@ -737,8 +839,10 @@ page), use the following phrases (based on the SVG icons): Valid for Markdown content only, not for front matter entries: -- Standard quotes: double quotes (`"`). Example: "This is wrapped in double quotes". -- Quote within a quote: double quotes (`"`) wrap single quotes (`'`). Example: "I am 'quoting' something within a quote". +- Standard quotes: double quotes (`"`). Example: "This is wrapped in double + quotes". +- Quote within a quote: double quotes (`"`) wrap single quotes (`'`). Example: + "I am 'quoting' something within a quote". For other punctuation rules, please refer to the [GitLab UX guide](https://design.gitlab.com/content/punctuation/). @@ -755,58 +859,66 @@ For other punctuation rules, please refer to the someone in the Merge Request. - [Avoid using symbols and special characters](https://gitlab.com/gitlab-org/gitlab-docs/-/issues/84) in headers. Whenever possible, they should be plain and short text. -- Avoid adding things that show ephemeral statuses. For example, if a feature is - considered beta or experimental, put this information in a note, not in the heading. +- When possible, avoid including words that might change in the future. Changing + a heading changes its anchor URL, which affects other linked pages. - When introducing a new document, be careful for the headings to be grammatically and syntactically correct. Mention an [assigned technical writer (TW)](https://about.gitlab.com/handbook/product/product-categories/) for review. - This is to ensure that no document with wrong heading is going - live without an audit, thus preventing dead links and redirection issues when - corrected. + This is to ensure that no document with wrong heading is going live without an + audit, thus preventing dead links and redirection issues when corrected. - Leave exactly one blank line before and after a heading. - Do not use links in headings. -- Add the corresponding [product badge](#product-badges) according to the tier the feature belongs. -- Our docs site search engine prioritizes words used in headings and subheadings. - Make you subheading titles clear, descriptive, and complete to help users find the - right example, as shown in the section on [heading titles](#heading-titles). +- Add the corresponding [product badge](#product-badges) according to the tier the + feature belongs. +- Our documentation site search engine prioritizes words used in headings and + subheadings. Make you subheading titles clear, descriptive, and complete to help + users find the right example, as shown in the section on [heading titles](#heading-titles). - See [Capitalization](#capitalization) for guidelines on capitalizing headings. ### Heading titles -Keep heading titles clear and direct. Make every word count. To accommodate search engine optimization (SEO), use the imperative, where possible. +Keep heading titles clear and direct. Make every word count. To accommodate +search engine optimization (SEO), use the imperative, where possible. -| Do | Don't | -|:-----|:--------| -| Configure GDK | Configuring GDK | +| Do | Don't | +|:--------------------------------------|:------------------------------------------------------------| +| Configure GDK | Configuring GDK | | GitLab Release and Maintenance Policy | This section covers GitLab's Release and Maintenance Policy | -| Backport to older releases | Backporting to older releases | -| GitLab Pages examples | Examples | +| Backport to older releases | Backporting to older releases | +| GitLab Pages examples | Examples | For guidelines on capitalizing headings, see the section on [capitalization](#capitalization). NOTE: **Note:** -If you change an existing title, be careful. Any such changes may affect not only [links](#anchor-links) -within the page, but may also affect links from GitLab itself, as well as external links, to GitLab documentation. +If you change an existing title, be careful. Any such changes may affect not +only [links](#anchor-links) within the page, but may also affect links to the +GitLab documentation from both the GitLab application and external sites. ### Anchor links Headings generate anchor links automatically when rendered. `## This is an example` generates the anchor `#this-is-an-example`. -Keep in mind that the GitLab UI links to a large number of docs and respective -anchor links to take the user to the right spot. Therefore, when you change a -heading, search `doc/*`, `app/views/*`, and `ee/app/views/*` for the old anchor -to make sure you're not breaking an anchor linked from other docs nor from the -GitLab UI. If you find the old anchor, make sure to replace it with the new one. +NOTE: **Note:** +[Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/39717) in GitLab 13.4, [product badges](#product-badges) used in headings aren't included in the +generated anchor links. For example, when you link to +`## This is an example **(CORE)**`, use the anchor `#this-is-an-example`. + +Keep in mind that the GitLab user interface links to many documentation pages +and anchor links to take the user to the right spot. Therefore, when you change +a heading, search `doc/*`, `app/views/*`, and `ee/app/views/*` for the old +anchor to make sure you're not breaking an anchor linked from other +documentation nor from the GitLab user interface. If you find the old anchor, be +sure to replace it with the new one. Important: -- Avoid crosslinking docs to headings unless you need to link to a specific section - of the document. This will avoid breaking anchors in the future in case the heading - is changed. +- Avoid crosslinking documentation to headings unless you need to link to a + specific section of the document. This will avoid breaking anchors in the + future in case the heading is changed. - If possible, avoid changing headings since they're not only linked internally. - There are various links to GitLab documentation on the internet, such as tutorials, - presentations, StackOverflow posts, and other sources. + There are various links to GitLab documentation on the internet, such as + tutorials, presentations, StackOverflow posts, and other sources. - Do not link to `h1` headings. Note that, with Kramdown, it is possible to add a custom ID to an HTML element @@ -815,16 +927,20 @@ do not use this option until further notice. ## Links -Links are important in GitLab documentation. They allow you to [link instead of summarizing](#link-instead-of-summarize) -to help preserve an [SSoT](#why-a-single-source-of-truth) within GitLab documentation. +Links are important in GitLab documentation. They allow you to [link instead of +summarizing](#link-instead-of-summarize) to help preserve a [single source of truth](#why-a-single-source-of-truth) +within GitLab documentation. We include guidance for links in the following categories: - How to set up [anchor links](#anchor-links) for headings. - How to set up [criteria](#basic-link-criteria) for configuring a link. -- What to set up when [linking to a `help`](../documentation/index.md#linking-to-help) page. -- How to set up [links to internal documentation](#links-to-internal-documentation) for cross-references. -- How to set up [links to external documentation](#links-to-external-documentation) for authoritative sources. +- What to set up when [linking to a `help`](../documentation/index.md#linking-to-help) + page. +- How to set up [links to internal documentation](#links-to-internal-documentation) + for cross-references. +- How to set up [links to external documentation](#links-to-external-documentation) + for authoritative sources. - When to use [links requiring permissions](#links-requiring-permissions). - How to set up a [link to a video](#link-to-video). - How to [include links with version text](#text-for-documentation-requiring-version-text). @@ -833,7 +949,7 @@ We include guidance for links in the following categories: ### Basic link criteria - Use inline link Markdown markup `[Text](https://example.com)`. - It's easier to read, review, and maintain. **Do not** use `[Text][identifier]`. + It's easier to read, review, and maintain. *Do not* use `[Text][identifier]`. - Use [meaningful anchor texts](https://www.futurehosting.com/blog/links-should-have-meaningful-anchor-text-heres-why/). For example, instead of writing something like `Read more about GitLab Issue Boards [here](LINK)`, @@ -842,18 +958,23 @@ We include guidance for links in the following categories: ### Links to internal documentation NOTE: **Note:** -_Internal_ refers to documentation in the same project. When linking to documentation in -separate projects (for example, linking to Omnibus docs from GitLab docs), you must use absolute -URLs. +_Internal_ refers to documentation in the same project. When linking to +documentation in separate projects (for example, linking to Omnibus documentation +from GitLab documentation), you must use absolute URLs. -Do not use absolute URLs like `https://docs.gitlab.com/ee/index.html` to crosslink -to other docs within the same project. Use relative links to the file, like `../index.md`. (These are converted to HTML when the site is rendered.) +Do not use absolute URLs like `https://docs.gitlab.com/ee/index.html` to +crosslink to other documentation within the same project. Use relative links to +the file, like `../index.md`. (These are converted to HTML when the site is +rendered.) Relative linking enables crosslinks to work: - in Review Apps, local previews, and `/help`. -- when working on the docs locally, so you can verify that they work as early as possible in the process. -- within the GitLab UI when browsing doc files in their respective repositories. For example, the links displayed at `https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/README.md`. +- when working on the documentation locally, so you can verify that they work as + early as possible in the process. +- within the GitLab user interface when browsing doc files in their respective + repositories. For example, the links displayed at + `https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/README.md`. To link to internal documentation: @@ -869,7 +990,8 @@ To link to internal documentation: Do: `../../geo/replication/troubleshooting.md` -- Always add the file name `file.md` at the end of the link with the `.md` extension, not `.html`. +- Always add the file name `file.md` at the end of the link with the `.md` + extension, not `.html`. Don't: @@ -884,28 +1006,32 @@ To link to internal documentation: - `../../issues/tags.md#stages` NOTE: **Note:** -Using the Markdown extension is necessary for the [`/help`](index.md#gitlab-help) section of GitLab. +Using the Markdown extension is necessary for the [`/help`](index.md#gitlab-help) +section of GitLab. ### Links to external documentation -When describing interactions with external software, it's often helpful to include links to external -documentation. When possible, make sure that you're linking to an [**authoritative** source](#authoritative-sources). -For example, if you're describing a feature in Microsoft's Active Directory, include a link to official Microsoft documentation. +When describing interactions with external software, it's often helpful to +include links to external documentation. When possible, make sure that you're +linking to an [**authoritative** source](#authoritative-sources). For example, +if you're describing a feature in Microsoft's Active Directory, include a link +to official Microsoft documentation. ### Authoritative sources -When citing external information, use sources that are written by the people who created -the item or product in question. These sources are the most likely to -be accurate and remain up to date. +When citing external information, use sources that are written by the people who +created the item or product in question. These sources are the most likely to be +accurate and remain up to date. Examples of authoritative sources include: -- Specifications, such as a [Request for Comments](https://www.ietf.org/standards/rfcs/) document -from the Internet Engineering Task Force. -- Official documentation for a product. For example, if you're setting up an interface with the -Google OAuth 2 authorization server, include a link to Google's documentation. -- Official documentation for a project. For example, if you're citing NodeJS functionality, -refer directly to [NodeJS documentation](https://nodejs.org/en/docs/). +- Specifications, such as a [Request for Comments](https://www.ietf.org/standards/rfcs/) + document from the Internet Engineering Task Force. +- Official documentation for a product. For example, if you're setting up an + interface with the Google OAuth 2 authorization server, include a link to + Google's documentation. +- Official documentation for a project. For example, if you're citing NodeJS + functionality, refer directly to [NodeJS documentation](https://nodejs.org/en/docs/). - Books from an authoritative publisher. Examples of sources to avoid include: @@ -916,19 +1042,22 @@ Examples of sources to avoid include: - Discussions on forums such as Stack Overflow. - Documentation from a company that describes another company's product. -While many of these sources to avoid can help you learn skills and or features, they can become -obsolete quickly. Nobody is obliged to maintain any of these sites. Therefore, we should avoid using them as reference literature. +While many of these sources to avoid can help you learn skills and or features, +they can become obsolete quickly. Nobody is obliged to maintain any of these +sites. Therefore, we should avoid using them as reference literature. NOTE: **Note:** -Non-authoritative sources are acceptable only if there is no equivalent authoritative source. -Even then, focus on non-authoritative sources that are extensively cited or peer-reviewed. +Non-authoritative sources are acceptable only if there is no equivalent +authoritative source. Even then, focus on non-authoritative sources that are +extensively cited or peer-reviewed. ### Links requiring permissions Don't link directly to: - [Confidential issues](../../user/project/issues/confidential_issues.md). -- Project features that require [special permissions](../../user/permissions.md) to view. +- Project features that require [special permissions](../../user/permissions.md) + to view. These will fail for: @@ -940,7 +1069,8 @@ Instead: - To reduce confusion, mention in the text that the information is either: - Contained in a confidential issue. - Requires special permission to a project to view. -- Provide a link in back ticks (`` ` ``) so that those with access to the issue can easily navigate to it. +- Provide a link in back ticks (`` ` ``) so that those with access to the issue + can easily navigate to it. Example: @@ -950,54 +1080,54 @@ For more information, see the [confidential issue](../../user/project/issues/con ### Link to specific lines of code -When linking to specific lines within a file, link to a commit instead of to the branch. -Lines of code change through time, therefore, linking to a line by using the commit link -ensures the user lands on the line you're referring to. The **Permalink** button, which is -available when viewing a file within a project, makes it easy to generate a link to the -most recent commit of the given file. +When linking to specific lines within a file, link to a commit instead of to the +branch. Lines of code change through time, therefore, linking to a line by using +the commit link ensures the user lands on the line you're referring to. The +**Permalink** button, which is available when viewing a file within a project, +makes it easy to generate a link to the most recent commit of the given file. -- **Do:** `[link to line 3](https://gitlab.com/gitlab-org/gitlab/-/blob/11f17c56d8b7f0b752562d78a4298a3a95b5ce66/.gitlab/issue_templates/Feature%20proposal.md#L3)` -- **Don't:** `[link to line 3](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/Feature%20proposal.md#L3).` +- *Do:* `[link to line 3](https://gitlab.com/gitlab-org/gitlab/-/blob/11f17c56d8b7f0b752562d78a4298a3a95b5ce66/.gitlab/issue_templates/Feature%20proposal.md#L3)` +- *Don't:* `[link to line 3](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/Feature%20proposal.md#L3).` -If that linked expression is no longer in that line of the file due to further commits, you -can still search the file for that query. In this case, update the document to ensure it -links to the most recent version of the file. +If that linked expression is no longer in that line of the file due to additional +commits, you can still search the file for that query. In this case, update the +document to ensure it links to the most recent version of the file. ## Navigation -To indicate the steps of navigation through the UI: +To indicate the steps of navigation through the user interface: - Use the exact word as shown in the UI, including any capital letters as-is. -- Use bold text for navigation items and the char "greater than" (`>`) as separator - (for example, `Navigate to your project's **Settings > CI/CD**` ). -- If there are any expandable menus, make sure to mention that the user - needs to expand the tab to find the settings you're referring to (for example, `Navigate to your project's **Settings > CI/CD** and expand **General pipelines**`). +- Use bold text for navigation items and the char "greater than" (`>`) as a + separator (for example, `Navigate to your project's **Settings > CI/CD**` ). +- If there are any expandable menus, make sure to mention that the user needs to + expand the tab to find the settings you're referring to (for example, + `Navigate to your project's **Settings > CI/CD** and expand **General pipelines**`). ## Images Images, including screenshots, can help a reader better understand a concept. However, they can be hard to maintain, and should be used sparingly. -Before including an image in the documentation, ensure it provides value to the reader. +Before including an image in the documentation, ensure it provides value to the +reader. ### Capture the image -Use images to help the reader understand where they are in a process, or how they need to -interact with the application. +Use images to help the reader understand where they are in a process, or how +they need to interact with the application. When you take screenshots: - *Capture the most relevant area of the page.* Don't include unnecessary white - space or areas of the page that don't help illustrate your point. Also, don't - include the entire page if you don't have to, but also ensure the image - contains enough information to allow the user to determine where things are. -- *Be consistent.* Find a browser window size that works for you that also - displays all areas of the product, including the left navigation (usually > - 1200px wide). For consistency, use this browser window size for your - screenshots by installing a browser extension for setting a window to a - specific size (for example, - [Window Resizer](https://chrome.google.com/webstore/detail/window-resizer/kkelicaakdanhinjdeammmilcgefonfh/related?hl=en) - for Google Chrome). + space or areas of the page that don't help illustrate the point. The left + sidebar of the GitLab user interface can change, so don't include the sidebar + if it's not necessary. +- *Keep it small.* If you don't need to show the full width of the screen, don't. + A value of 1000 pixels is a good maximum width for your screenshot image. +- *Be consistent.* Coordinate screenshots with the other screenshots already on + a documentation page. For example, if other screenshots include the left + sidebar, include the sidebar in all screenshots. ### Save the image @@ -1017,15 +1147,16 @@ When you take screenshots: - Images should be used (only when necessary) to _illustrate_ the description of a process, not to _replace_ it. - Max image size: 100KB (gifs included). -- See also how to link and embed [videos](#videos) to illustrate the docs. +- See also how to link and embed [videos](#videos) to illustrate the + documentation. ### Add the image link to content The Markdown code for including an image in a document is: `` -The image description is the alt text for the rendered image on the docs site. -For accessibility and SEO, use [descriptions](https://webaim.org/techniques/alttext/) +The image description is the alt text for the rendered image on the +documentation site. For accessibility and SEO, use [descriptions](https://webaim.org/techniques/alttext/) that: - Are accurate, succinct, and unique. @@ -1036,9 +1167,9 @@ Also, if a heading immediately follows an image, be sure to add three dashes ### Remove image shadow -All images displayed on the [GitLab Docs site](https://docs.gitlab.com) have a box shadow by default. -To remove the box shadow, use the image class `.image-noshadow` applied -directly to an HTML `img` tag: +All images displayed on the [GitLab documentation site](https://docs.gitlab.com) +have a box shadow by default. To remove the box shadow, use the image class +`.image-noshadow` applied directly to an HTML `img` tag: ```html <img src="path/to/image.jpg" alt="Alt text (required)" class="image-noshadow"> @@ -1075,18 +1206,20 @@ request. ## Videos -Adding GitLab's existing YouTube video tutorials to the documentation is -highly encouraged, unless the video is outdated. Videos should not -replace documentation, but complement or illustrate it. If content in a video is -fundamental to a feature and its key use cases, but this is not adequately covered in the documentation, -add this detail to the documentation text or create an issue to review the video and do so. +Adding GitLab's existing YouTube video tutorials to the documentation is highly +encouraged, unless the video is outdated. Videos should not replace +documentation, but complement or illustrate it. If content in a video is +fundamental to a feature and its key use cases, but this is not adequately +covered in the documentation, add this detail to the documentation text or +create an issue to review the video and do so. -Do not upload videos to the product repositories. [Link](#link-to-video) or [embed](#embed-videos) them instead. +Do not upload videos to the product repositories. [Link](#link-to-video) or +[embed](#embed-videos) them instead. ### Link to video -To link out to a video, include a YouTube icon so that readers can -quickly and easily scan the page for videos before reading: +To link out to a video, include a YouTube icon so that readers can scan the page +for videos before reading: ```markdown <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> @@ -1099,26 +1232,24 @@ You can link any up-to-date video that is useful to the GitLab user. > [Introduced](https://gitlab.com/gitlab-org/gitlab-docs/-/merge_requests/472) in GitLab 12.1. -The [GitLab Docs site](https://docs.gitlab.com) supports embedded videos. +The [GitLab documentation site](https://docs.gitlab.com) supports embedded +videos. -You can only embed videos from -[GitLab's official YouTube account](https://www.youtube.com/channel/UCnMGQ8QHMAnVIsI3xJrihhg). +You can only embed videos from [GitLab's official YouTube account](https://www.youtube.com/channel/UCnMGQ8QHMAnVIsI3xJrihhg). For videos from other sources, [link](#link-to-video) them instead. -In most cases, it is better to [link to video](#link-to-video) instead, -because an embed takes up a lot of space on the page and can be distracting -to readers. +In most cases, it is better to [link to video](#link-to-video) instead, because +an embed takes up a lot of space on the page and can be distracting to readers. -To embed a video, follow the instructions below and make sure -you have your MR reviewed and approved by a technical writer. +To embed a video, follow the instructions below and make sure you have your MR +reviewed and approved by a technical writer. -1. Copy the code below and paste it into your Markdown file. - Leave a blank line above and below it. Do NOT edit the code - (don't remove or add any spaces). -1. On YouTube, visit the video URL you want to display. Copy - the regular URL from your browser (`https://www.youtube.com/watch?v=VIDEO-ID`) - and replace the video title and link in the line under `<div class="video-fallback">`. -1. On YouTube, click **Share**, then **Embed**. +1. Copy the code below and paste it into your Markdown file. Leave a blank line + above and below it. Do *not* edit the code (don't remove or add any spaces). +1. In YouTube, visit the video URL you want to display. Copy the regular URL + from your browser (`https://www.youtube.com/watch?v=VIDEO-ID`) and replace + the video title and link in the line under `<div class="video-fallback">`. +1. In YouTube, click **Share**, and then click **Embed**. 1. Copy the `<iframe>` source (`src`) **URL only** (`https://www.youtube.com/embed/VIDEO-ID`), and paste it, replacing the content of the `src` field in the @@ -1135,7 +1266,7 @@ leave a blank line here leave a blank line here ``` -This is how it renders on the GitLab Docs site: +This is how it renders on the GitLab documentation site: <div class="video-fallback"> See the video: <a href="https://www.youtube.com/watch?v=enMumwvLAug">What is GitLab</a>. @@ -1147,26 +1278,28 @@ This is how it renders on the GitLab Docs site: > Notes: > > - The `figure` tag is required for semantic SEO and the `video_container` -class is necessary to make sure the video is responsive and displays -nicely on different mobile devices. +class is necessary to make sure the video is responsive and displays on +different mobile devices. > - The `<div class="video-fallback">` is a fallback necessary for GitLab's -`/help`, as GitLab's Markdown processor does not support iframes. It's hidden on the docs site but will be displayed on GitLab's `/help`. +`/help`, as GitLab's Markdown processor does not support iframes. It's hidden on +the documentation site, but will be displayed on GitLab's `/help`. ## Code blocks - Always wrap code added to a sentence in inline code blocks (`` ` ``). For example, `.gitlab-ci.yml`, `git add .`, `CODEOWNERS`, or `only: [master]`. - File names, commands, entries, and anything that refers to code should be added to code blocks. - To make things easier for the user, always add a full code block for things that can be - useful to copy and paste, as they can easily do it with the button on code blocks. + File names, commands, entries, and anything that refers to code should be + added to code blocks. To make things easier for the user, always add a full + code block for things that can be useful to copy and paste, as they can easily + do it with the button on code blocks. - Add a blank line above and below code blocks. -- When providing a shell command and its output, prefix the shell command with `$` and - leave a blank line between the command and the output. +- When providing a shell command and its output, prefix the shell command with `$` + and leave a blank line between the command and the output. - When providing a command without output, don't prefix the shell command with `$`. - If you need to include triple backticks inside a code block, use four backticks for the codeblock fences instead of three. -- For regular fenced code blocks, always use a highlighting class corresponding to the - language for better readability. Examples: +- For regular fenced code blocks, always use a highlighting class corresponding to + the language for better readability. Examples: ````markdown ```ruby @@ -1186,10 +1319,10 @@ nicely on different mobile devices. ``` ```` -Syntax highlighting is required for fenced code blocks added to the GitLab documentation. -Refer to the table below for the most common language classes, or check the -[complete list](https://github.com/rouge-ruby/rouge/wiki/List-of-supported-languages-and-lexers) -of language classes available. +Syntax highlighting is required for fenced code blocks added to the GitLab +documentation. Refer to the following table for the most common language classes, +or check the [complete list](https://github.com/rouge-ruby/rouge/wiki/List-of-supported-languages-and-lexers) +of available language classes: | Preferred language tags | Language aliases and notes | |-------------------------|------------------------------------------------------------------------------| @@ -1220,16 +1353,17 @@ of language classes available. | `xml` | | | `yaml` | Alias: `yml`. | -For a complete reference on code blocks, check the [Kramdown guide](https://about.gitlab.com/handbook/markdown-guide/#code-blocks). +For a complete reference on code blocks, see the [Kramdown guide](https://about.gitlab.com/handbook/markdown-guide/#code-blocks). ## GitLab SVG icons > [Introduced](https://gitlab.com/gitlab-org/gitlab-docs/-/issues/384) in GitLab 12.7. -You can use icons from the [GitLab SVG library](https://gitlab-org.gitlab.io/gitlab-svgs/) directly -in the documentation. +You can use icons from the [GitLab SVG library](https://gitlab-org.gitlab.io/gitlab-svgs/) +directly in the documentation. -This way, you can achieve a consistent look when writing about interacting with GitLab UI elements. +This way, you can achieve a consistent look when writing about interacting with +GitLab user interface elements. Usage examples: @@ -1243,7 +1377,7 @@ Usage examples: Example: `**{tanuki, 24}**` renders as: **{tanuki, 24}**. - Icon with custom size and class: `**{icon-name, size, class-name}**`. - You can access any class available to this element in GitLab docs CSS. + You can access any class available to this element in GitLab documentation CSS. Example with `float-right`, a [Bootstrap utility class](https://getbootstrap.com/docs/4.4/utilities/float/): @@ -1251,8 +1385,8 @@ Usage examples: ### When to use icons -Icons should be used sparingly, and only in ways that aid and do not hinder the readability of the -text. +Icons should be used sparingly, and only in ways that aid and do not hinder the +readability of the text. For example, the following adds little to the accompanying text: @@ -1262,54 +1396,68 @@ For example, the following adds little to the accompanying text: 1. Go to **{home}** **Project overview > Details** -However, the following might help the reader connect the text to the user interface: +However, the following might help the reader connect the text to the user +interface: ```markdown | Section | Description | |:-------------------------|:----------------------------------------------------------------------------------------------------------------------------| -| **{overview}** Overview | View your GitLab Dashboard, and administer projects, users, groups, jobs, Runners, and Gitaly servers. | +| **{overview}** Overview | View your GitLab Dashboard, and administer projects, users, groups, jobs, runners, and Gitaly servers. | | **{monitor}** Monitoring | View GitLab system information, and information on background jobs, logs, health checks, requests profiles, and audit logs. | | **{messages}** Messages | Send and manage broadcast messages for your users. | ``` | Section | Description | |:-------------------------|:----------------------------------------------------------------------------------------------------------------------------| -| **{overview}** Overview | View your GitLab Dashboard, and administer projects, users, groups, jobs, Runners, and Gitaly servers. | +| **{overview}** Overview | View your GitLab Dashboard, and administer projects, users, groups, jobs, runners, and Gitaly servers. | | **{monitor}** Monitoring | View GitLab system information, and information on background jobs, logs, health checks, requests profiles, and audit logs. | | **{messages}** Messages | Send and manage broadcast messages for your users. | -Use an icon when you find youself having to describe an interface element. For example: +Use an icon when you find yourself having to describe an interface element. For +example: - Do: Click the Admin Area icon ( **{admin}** ). - Don't: Click the Admin Area icon (the wrench icon). ## Alert boxes -Whenever you need to call special attention to particular sentences, -use the following markup for highlighting. +When you need to call special attention to particular sentences, use the +following markup to create highlighted alert boxes. Note that the alert boxes only work for one paragraph only. Multiple paragraphs, -lists, headers and so on, will not render correctly. For multiple lines, use [blockquotes](#blockquotes) instead. +lists, headers and so on, will not render correctly. For multiple lines, use +[blockquotes](#blockquotes) instead. -Alert boxes only render on the GitLab Docs site (<https://docs.gitlab.com>). +Alert boxes render only on the GitLab documentation site (<https://docs.gitlab.com>). Within GitLab itself, they will appear as plain Markdown text (like the examples above the rendered versions, below). +These alert boxes are used in the GitLab documentation. These aren't strict +guidelines, but for consistency you should try to use these values: + +| Color | Markup | Default keyword | Alternative keywords | +|--------|------------|-----------------|----------------------------------------------------------------------| +| Blue | `NOTE:` | `**Note:**` | | +| Yellow | `CAUTION:` | `**Caution:**` | `**Warning:**`, `**Important:**` | +| Red | `DANGER:` | `**Danger:**` | `**Warning:**`, `**Important:**`, `**Deprecated:**`, `**Required:**` | +| Green | `TIP:` | `**Tip:**` | | + ### Note Notes catch the eye of most readers, and therefore should be used very sparingly. In most cases, content considered for a note should be included: -- As just another sentence in the previous paragraph or the most-relevant paragraph. +- As just another sentence in the previous paragraph or the most-relevant + paragraph. - As its own standalone paragraph. -- As content under a new subheading that introduces the topic, making it more visible/findable. +- As content under a new subheading that introduces the topic, making it more + visible or findable. #### When to use Use a note when there is a reason that most or all readers who browse the -section should see the content. That is, if missed, it’s likely to cause -major trouble for a minority of users or significant trouble for a majority -of users. +section should see the content. That is, if missed, it’s likely to cause major +trouble for a minority of users or significant trouble for a majority of users. Weigh the costs of distracting users to whom the content is not relevant against the cost of users missing the content if it were not expressed as a note. @@ -1319,7 +1467,7 @@ NOTE: **Note:** This is something to note. ``` -How it renders on the GitLab Docs site: +How it renders on the GitLab documentation site: NOTE: **Note:** This is something to note. @@ -1331,7 +1479,7 @@ TIP: **Tip:** This is a tip. ``` -How it renders on the GitLab Docs site: +How it renders on the GitLab documentation site: TIP: **Tip:** This is a tip. @@ -1343,7 +1491,7 @@ CAUTION: **Caution:** This is something to be cautious about. ``` -How it renders on the GitLab Docs site: +How it renders on the GitLab documentation site: CAUTION: **Caution:** This is something to be cautious about. @@ -1355,7 +1503,7 @@ DANGER: **Danger:** This is a breaking change, a bug, or something very important to note. ``` -How it renders on the GitLab Docs site: +How it renders on the GitLab documentation site: DANGER: **Danger:** This is a breaking change, a bug, or something very important to note. @@ -1368,7 +1516,7 @@ For highlighting a text within a blue blockquote, use this format: > This is a blockquote. ``` -which renders on the [GitLab Docs site](https://docs.gitlab.com) as: +which renders on the [GitLab documentation site](https://docs.gitlab.com) as: > This is a blockquote. @@ -1396,19 +1544,22 @@ Which renders to: ## Terms -To maintain consistency through GitLab documentation, the following guides documentation authors -on agreed styles and usage of terms. +To maintain consistency through GitLab documentation, the following guides +documentation authors on agreed styles and usage of terms. ### Merge requests (MRs) -Merge requests allow you to exchange changes you made to source code and collaborate -with other people on the same project. You'll see this term used in the following ways: +Merge requests allow you to exchange changes you made to source code and +collaborate with other people on the same project. You'll see this term used in +the following ways: -- Use lowercase **merge requests** regardless of whether referring to the feature or individual merge requests. +- Use lowercase *merge requests* regardless of whether referring to the feature + or individual merge requests. As noted in the GitLab [Writing Style Guidelines](https://about.gitlab.com/handbook/communication/#writing-style-guidelines), if you use the **MR** acronym, expand it at least once per document page. -Typically, the first use would be phrased as _merge request (MR)_ with subsequent instances being _MR_. +Typically, the first use would be phrased as _merge request (MR)_ with subsequent +instances being _MR_. Examples: @@ -1418,14 +1569,18 @@ Examples: ### Describe UI elements -The following are styles to follow when describing UI elements on a screen: +The following are styles to follow when describing user interface elements in an +application: -- For elements with a visible label, use that label in bold with matching case. For example, `the **Cancel** button`. -- For elements with a tooltip or hover label, use that label in bold with matching case. For example, `the **Add status emoji** button`. +- For elements with a visible label, use that label in bold with matching case. + For example, `the **Cancel** button`. +- For elements with a tooltip or hover label, use that label in bold with + matching case. For example, `the **Add status emoji** button`. ### Verbs for UI elements -The following are recommended verbs for specific uses with UI elements: +The following are recommended verbs for specific uses with user interface +elements: | Recommended | Used for | Replaces | |:--------------------|:---------------------------|:---------------------------| @@ -1447,22 +1602,26 @@ Tagged and released versions of GitLab documentation are available: - In the [documentation archives](https://docs.gitlab.com/archives/). - At the `/help` URL for any GitLab installation. -The version introducing a new feature is added to the top of the topic in the documentation to provide -a helpful link back to how the feature was developed. +The version introducing a new feature is added to the top of the topic in the +documentation to provide a link back to how the feature was developed. TIP: **Tip:** -Whenever you have documentation related to the `gitlab.rb` file, you're working with a self-managed installation. -The section or page is therefore likely to apply only to self-managed instances. -If so, the relevant "`TIER` ONLY" [Product badge](#product-badges) should be included at the highest applicable heading level. +Whenever you have documentation related to the `gitlab.rb` file, you're working +with a self-managed installation. The section or page is therefore likely to +apply only to self-managed instances. If so, the relevant "`TIER` ONLY" +[Product badge](#product-badges) should be included at the highest applicable +heading level. ### Text for documentation requiring version text -- For features that need to declare the GitLab version that the feature was introduced. Text similar - to the following should be added immediately below the heading as a blockquote: +- For features that need to declare the GitLab version that the feature was + introduced. Text similar to the following should be added immediately below + the heading as a blockquote: - `> Introduced in GitLab 11.3.`. -- Whenever possible, version text should have a link to the _completed_ issue, merge request, or epic that introduced the feature. - An issue is preferred over a merge request, and a merge request is preferred over an epic. For example: +- Whenever possible, version text should have a link to the _completed_ issue, + merge request, or epic that introduced the feature. An issue is preferred over + a merge request, and a merge request is preferred over an epic. For example: - `> [Introduced](<link-to-issue>) in GitLab 11.3.`. - If the feature is only available in GitLab Enterprise Edition, mention @@ -1470,8 +1629,8 @@ If so, the relevant "`TIER` ONLY" [Product badge](#product-badges) should be inc the feature is available in: - `> [Introduced](<link-to-issue>) in [GitLab Starter](https://about.gitlab.com/pricing/) 11.3.`. -- If listing information for multiple version as a feature evolves, add the information to a - block-quoted bullet list. For example: +- If listing information for multiple version as a feature evolves, add the + information to a block-quoted bullet list. For example: ```markdown > - [Introduced](<link-to-issue>) in GitLab 11.3. @@ -1492,25 +1651,28 @@ If so, the relevant "`TIER` ONLY" [Product badge](#product-badges) should be inc > - [Deprecated](<link-to-issue>) in GitLab 11.3. Replaced by [meaningful text](<link-to-appropriate-documentation>). ``` - It's also acceptable to describe the replacement in surrounding text, if available. + It's also acceptable to describe the replacement in surrounding text, if + available. - If the deprecation is not obvious in existing text, you may want to include a warning such as: + If the deprecation is not obvious in existing text, you may want to include a + warning such as: ```markdown - CAUTION: **Warning:** + DANGER: **Deprecated:** This feature was [deprecated](link-to-issue) in GitLab 12.3 and replaced by [Feature name](link-to-feature-documentation). ``` NOTE: **Note:** -Version text must be on its own line and surrounded by blank lines to render correctly. +Version text must be on its own line and surrounded by blank lines to render +correctly. ### Versions in the past or future When describing functionality available in past or future versions, use: -- **Earlier**, and not **older** or **before**. -- **Later**, and not **newer** or **after**. +- *Earlier*, and not *older* or *before*. +- *Later*, and not *newer* or *after*. For example: @@ -1522,33 +1684,34 @@ For example: ### Importance of referencing GitLab versions and tiers Mentioning GitLab versions and tiers is important to all users and contributors -to quickly have access to the issue or merge request that -introduced the change for reference. Also, they can easily understand what -features they have in their GitLab instance and version, given that the note has -some key information. +to quickly have access to the issue or merge request that introduced the change +for reference. Also, they can easily understand what features they have in their +GitLab instance and version, given that the note has some key information. `[Introduced](link-to-issue) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.7` links to the issue that introduced the feature, says which GitLab tier it belongs to, says the GitLab version that it became available in, and links to -the pricing page in case the user wants to upgrade to a paid tier -to use that feature. +the pricing page in case the user wants to upgrade to a paid tier to use that +feature. -For example, if you're a regular user and you're looking at the docs for a feature you haven't used before, -you can immediately see if that feature is available to you or not. Alternatively, -if you've been using a certain feature for a long time and it changed in some way, -it's important -to be able to spot when it changed and what's new in that feature. +For example, if you're a regular user and you're looking at the documentation +for a feature you haven't used before, you can immediately see if that feature +is available to you or not. Alternatively, if you've been using a certain +feature for a long time and it changed in some way, it's important to be able to +determine when it changed and what's new in that feature. -This is even more important as we don't have a perfect process for shipping docs. -Unfortunately, we still see features without docs and docs without -features. So, for now, we cannot rely 100% on the docs site versions. +This is even more important as we don't have a perfect process for shipping +documentation. Unfortunately, we still see features without documentation, and +documentation without features. So, for now, we cannot rely 100% on the +documentation site versions. Over time, version text will reference a progressively older version of GitLab. In cases where version text refers to versions of GitLab four or more major versions back, you can consider removing the text if it's irrelevant or confusing. -For example, if the current major version is 12.x, version text referencing versions of GitLab 8.x -and older are candidates for removal if necessary for clearer or cleaner docs. +For example, if the current major version is 12.x, version text referencing +versions of GitLab 8.x and older are candidates for removal if necessary for +clearer or cleaner documentation. ## Products and features @@ -1598,21 +1761,21 @@ header or other page element according to the feature's availability: | *Only* GitLab.com Silver and higher tiers (no self-managed instances) | `**(SILVER ONLY)**` | | *Only* GitLab.com Gold (no self-managed instances) | `**(GOLD ONLY)**` | -For clarity, all page title headers (H1s) must be have a tier markup for -the lowest tier that has information on the documentation page. +For clarity, all page title headers (H1s) must be have a tier markup for the +lowest tier that has information on the documentation page. If sections of a page apply to higher tier levels, they can be separately labeled with their own tier markup. #### Product badge display behavior -When using the tier markup with headers, the documentation page will -display the full tier badge with the header line. +When using the tier markup with headers, the documentation page will display the +full tier badge with the header line. -You can also use the tier markup with paragraphs, list items, -and table cells. For these cases, the tier mention will be represented by an -orange info icon **{information}** that will display the tiers when visitors -point to the icon. For example: +You can also use the tier markup with paragraphs, list items, and table cells. +For these cases, the tier mention will be represented by an orange info icon +**{information}** that will display the tiers when visitors point to the icon. +For example: - `**(STARTER)**` displays as **(STARTER)** - `**(STARTER ONLY)**` displays as **(STARTER ONLY)** @@ -1623,16 +1786,17 @@ point to the icon. For example: Introduced by [!244](https://gitlab.com/gitlab-org/gitlab-docs/-/merge_requests/244), the special markup `**(STARTER)**` will generate a `span` element to trigger the badges and tooltips (`<span class="badge-trigger starter">`). When the keyword -"only" is added, the corresponding GitLab.com badge will not be displayed. +*only* is added, the corresponding GitLab.com badge will not be displayed. ## Specific sections -Certain styles should be applied to specific sections. Styles for specific sections are outlined below. +Certain styles should be applied to specific sections. Styles for specific +sections are outlined below. ### GitLab restart -There are many cases that a restart/reconfigure of GitLab is required. To -avoid duplication, link to the special document that can be found in +There are many cases that a restart/reconfigure of GitLab is required. To avoid +duplication, link to the special document that can be found in [`doc/administration/restart_gitlab.md`](../../administration/restart_gitlab.md). Usually the text will read like: @@ -1643,8 +1807,8 @@ for the changes to take effect. If the document you are editing resides in a place other than the GitLab CE/EE `doc/` directory, instead of the relative link, use the full path: -`https://docs.gitlab.com/ce/administration/restart_gitlab.html`. -Replace `reconfigure` with `restart` where appropriate. +`https://docs.gitlab.com/ce/administration/restart_gitlab.html`. Replace +`reconfigure` with `restart` where appropriate. ### Installation guide @@ -1652,8 +1816,8 @@ Replace `reconfigure` with `restart` where appropriate. In [step 2 of the installation guide](../../install/installation.md#2-ruby), we install Ruby from source. Whenever there is a new version that needs to be updated, remember to change it throughout the codeblock and also replace -the sha256sum (it can be found in the [downloads page](https://www.ruby-lang.org/en/downloads/) of the Ruby -website). +the sha256sum (it can be found in the [downloads page](https://www.ruby-lang.org/en/downloads/) +of the Ruby website). ### Configuration documentation for source and Omnibus installations @@ -1661,7 +1825,7 @@ GitLab currently officially supports two installation methods: installations from source and Omnibus packages installations. Whenever there is a setting that is configurable for both installation methods, -prefer to document it in the CE docs to avoid duplication. +prefer to document it in the CE documentation to avoid duplication. Configuration settings include: @@ -1681,7 +1845,8 @@ the style below as a guide: external_url "https://gitlab.example.com" ``` -1. Save the file and [reconfigure](path/to/administration/restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab for the changes to take effect. +1. Save the file and [reconfigure](path/to/administration/restart_gitlab.md#omnibus-gitlab-reconfigure) + GitLab for the changes to take effect. --- @@ -1694,18 +1859,20 @@ the style below as a guide: host: "gitlab.example.com" ``` -1. Save the file and [restart](path/to/administration/restart_gitlab.md#installations-from-source) GitLab for the changes to take effect. +1. Save the file and [restart](path/to/administration/restart_gitlab.md#installations-from-source) + GitLab for the changes to take effect. ```` In this case: - Before each step list the installation method is declared in bold. -- Three dashes (`---`) are used to create a horizontal line and separate the - two methods. +- Three dashes (`---`) are used to create a horizontal line and separate the two + methods. - The code blocks are indented one or more spaces under the list item to render correctly. - Different highlighting languages are used for each config in the code block. -- The [GitLab Restart](#gitlab-restart) section is used to explain a required restart/reconfigure of GitLab. +- The [GitLab Restart](#gitlab-restart) section is used to explain a required + restart or reconfigure of GitLab. ### Troubleshooting @@ -1717,19 +1884,30 @@ can facilitate this by making sure the troubleshooting content addresses: 1. How the user can confirm they have the problem. 1. Steps the user can take towards resolution of the problem. -If the contents of each category can be summarized in one line and a list of steps aren't required, consider setting up a -[table](#tables) with headers of *Problem* \| *Cause* \| *Solution* (or *Workaround* if the fix is temporary), or *Error message* \| *Solution*. +If the contents of each category can be summarized in one line and a list of +steps aren't required, consider setting up a [table](#tables) with headers of +*Problem* \| *Cause* \| *Solution* (or *Workaround* if the fix is temporary), or +*Error message* \| *Solution*. ## Feature flags -Learn how to [document features deployed behind flags](feature_flags.md). -For guidance on developing GitLab with feature flags, see -[Feature flags in development of GitLab](../feature_flags/index.md). +Learn how to [document features deployed behind flags](feature_flags.md). For +guidance on developing GitLab with feature flags, see [Feature flags in development of GitLab](../feature_flags/index.md). ## RESTful API -Here is a list of must-have items for RESTful API documentation. Use them in the -exact order that appears on this document. Further explanation is given below. +REST API resources are documented in Markdown under +[`/doc/api`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/doc/api). Each +resource has its own Markdown file, which is linked from `api_resources.md`. + +When modifying the Markdown, also update the corresponding +[OpenAPI definition](https://gitlab.com/gitlab-org/gitlab/-/tree/master/doc/api/openapi) +if one exists for the resource. If not, consider creating one. Match the latest +[OpenAPI 3.0.x specification](https://swagger.io/specification/). (For more +information, see the discussion in this +[issue](https://gitlab.com/gitlab-org/gitlab/-/issues/16023#note_370901810).) + +In the Markdown doc for a resource (AKA endpoint): - Every method must have the REST API request. For example: @@ -1737,8 +1915,7 @@ exact order that appears on this document. Further explanation is given below. GET /projects/:id/repository/branches ``` -- Every method must have a detailed - [description of the parameters](#method-description). +- Every method must have a detailed [description of the parameters](#method-description). - Every method must have a cURL example. - Every method must have a response body (in JSON format). @@ -1763,7 +1940,7 @@ METHOD /endpoint Example request: ```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" 'https://gitlab.example.com/api/v4/endpoint?parameters' +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/endpoint?parameters" ``` Example response: @@ -1782,8 +1959,9 @@ You may need to demonstrate an API call or a cURL command that includes the name and email address of a user. Don't use real user information in API calls: - **Email addresses**: Use an email address ending in `example.com`. -- **Names**: Use strings like `Example Username`. Alternatively, use diverse or non-gendered names with - common surnames, such as `Sidney Jones`, `Zhang Wei`. or `Maria Garcia`. +- **Names**: Use strings like `Example Username`. Alternatively, use diverse or + non-gendered names with common surnames, such as `Sidney Jones`, `Zhang Wei`, + or `Maria Garcia`. ### Fake URLs @@ -1795,11 +1973,10 @@ When including sample URLs in the documentation, use: ### Fake tokens There may be times where a token is needed to demonstrate an API call using -cURL or a variable used in CI. It is strongly advised not to use real -tokens in documentation even if the probability of a token being exploited is -low. +cURL or a variable used in CI. It is strongly advised not to use real tokens in +documentation even if the probability of a token being exploited is low. -You can use the following fake tokens as examples. +You can use the following fake tokens as examples: | Token type | Token value | |:----------------------|:-------------------------------------------------------------------| @@ -1808,8 +1985,8 @@ You can use the following fake tokens as examples. | Application ID | `2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6` | | Application secret | `04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df` | | CI/CD variable | `Li8j-mLUVA3eZYjPfd_H` | -| Specific Runner token | `yrnZW46BrtBFqM7xDzE7dddd` | -| Shared Runner token | `6Vk7ZsosqQyfreAxXTZr` | +| Specific runner token | `yrnZW46BrtBFqM7xDzE7dddd` | +| Shared runner token | `6Vk7ZsosqQyfreAxXTZr` | | Trigger token | `be20d8dcc028677c931e04f3871a9b` | | Webhook secret token | `6XhDroRcYPM5by_h-HLY` | | Health check token | `Tu7BgjR9qeZTEyRzGG2P` | @@ -1850,14 +2027,15 @@ Rendered example: ### cURL Examples -Below is a set of [cURL](https://curl.haxx.se) examples that you can use in the API documentation. +The following sections include a set of [cURL](https://curl.haxx.se) examples +you can use in the API documentation. #### Simple cURL command Get the details of a group: ```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/gitlab-org +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/gitlab-org" ``` #### cURL example with parameters passed in the URL @@ -1870,9 +2048,9 @@ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitla #### Post data using cURL's `--data` -Instead of using `--request POST` and appending the parameters to the URI, you can use -cURL's `--data` option. The example below will create a new project `foo` under -the authenticated user's namespace. +Instead of using `--request POST` and appending the parameters to the URI, you +can use cURL's `--data` option. The example below will create a new project +`foo` under the authenticated user's namespace. ```shell curl --data "name=foo" --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects" @@ -1881,11 +2059,11 @@ curl --data "name=foo" --header "PRIVATE-TOKEN: <your_access_token>" "https://gi #### Post data using JSON content NOTE: **Note:** -In this example we create a new group. Watch carefully the single -and double quotes. +In this example we create a new group. Watch carefully the single and double +quotes. ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" --data '{"path": "my-group", "name": "My group"}' https://gitlab.example.com/api/v4/groups +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" --data '{"path": "my-group", "name": "My group"}' "https://gitlab.example.com/api/v4/groups" ``` #### Post data using form-data @@ -1894,7 +2072,7 @@ Instead of using JSON or urlencode you can use multipart/form-data which properly handles data encoding: ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --form "title=ssh-key" --form "key=ssh-rsa AAAAB3NzaC1yc2EA..." https://gitlab.example.com/api/v4/users/25/keys +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --form "title=ssh-key" --form "key=ssh-rsa AAAAB3NzaC1yc2EA..." "https://gitlab.example.com/api/v4/users/25/keys" ``` The above example is run by and administrator and will add an SSH public key @@ -1916,41 +2094,47 @@ Use `%2F` for slashes (`/`). #### Pass arrays to API calls The GitLab API sometimes accepts arrays of strings or integers. For example, to -exclude specific users when requesting a list of users for a project, you would do something like this: +exclude specific users when requesting a list of users for a project, you would +do something like this: ```shell -curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" --data "skip_users[]=<user_id>" --data "skip_users[]=<user_id>" https://gitlab.example.com/api/v4/projects/<project_id>/users +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" --data "skip_users[]=<user_id>" --data "skip_users[]=<user_id>" "https://gitlab.example.com/api/v4/projects/<project_id>/users" ``` ## GraphQL API -GraphQL APIs are different from [RESTful APIs](#restful-api). Reference information is -generated automatically in our [GraphQL reference](../../api/graphql/reference/index.md). +GraphQL APIs are different from [RESTful APIs](#restful-api). Reference +information is generated in our [GraphQL reference](../../api/graphql/reference/index.md). -However, it's helpful to include examples on how to use GraphQL for different "use cases", -with samples that readers can use directly in the [GraphiQL explorer](../api_graphql_styleguide.md#graphiql). +However, it's helpful to include examples on how to use GraphQL for different +*use cases*, with samples that readers can use directly in the +[GraphiQL explorer](../api_graphql_styleguide.md#graphiql). -This section describes the steps required to add your GraphQL examples to GitLab documentation. +This section describes the steps required to add your GraphQL examples to +GitLab documentation. ### Add a dedicated GraphQL page -To create a dedicated GraphQL page, create a new `.md` file in the `doc/api/graphql/` directory. -Give that file a functional name, such as `import_from_specific_location.md`. +To create a dedicated GraphQL page, create a new `.md` file in the +`doc/api/graphql/` directory. Give that file a functional name, such as +`import_from_specific_location.md`. ### Start the page with an explanation -Include a page title that describes the GraphQL functionality in a few words, such as: +Include a page title that describes the GraphQL functionality in a few words, +such as: ```markdown # Search for [substitute kind of data] ``` -Describe the search. One sentence may be all you need. More information may help -readers learn how to use the example for their GitLab deployments. +Describe the search. One sentence may be all you need. More information may +help readers learn how to use the example for their GitLab deployments. ### Include a procedure using the GraphiQL explorer -The GraphiQL explorer can help readers test queries with working deployments. Set up the section with the following: +The GraphiQL explorer can help readers test queries with working deployments. +Set up the section with the following: - Use the following title: @@ -1958,8 +2142,8 @@ The GraphiQL explorer can help readers test queries with working deployments. Se ## Set up the GraphiQL explorer ``` -- Include a code block with the query that anyone can include in their instance of - the GraphiQL explorer: +- Include a code block with the query that anyone can include in their + instance of the GraphiQL explorer: ````markdown ```graphql @@ -1979,22 +2163,23 @@ The GraphiQL explorer can help readers test queries with working deployments. Se - Include a screenshot of the result in the GraphiQL explorer. Follow the naming convention described in the [Save the image](#save-the-image) section. -- Follow up with an example of what you can do with the output. - Make sure the example is something that readers can do on their own deployments. +- Follow up with an example of what you can do with the output. Make sure the + example is something that readers can do on their own deployments. - Include a link to the [GraphQL API resources](../../api/graphql/reference/index.md). ### Add the GraphQL example to the Table of Contents -You'll need to open a second MR, against the [GitLab Docs repository](https://gitlab.com/gitlab-org/gitlab-docs/). +You'll need to open a second MR, against the [GitLab documentation repository](https://gitlab.com/gitlab-org/gitlab-docs/). -We store our Table of Contents in the `default-nav.yaml` file, in the `content/_data` -subdirectory. You can find the GraphQL section under the following line: +We store our Table of Contents in the `default-nav.yaml` file, in the +`content/_data` subdirectory. You can find the GraphQL section under the +following line: ```yaml - - category_title: GraphQL +- category_title: GraphQL ``` -Be aware that CI tests for that second MR will fail with a bad link until the main MR -that adds the new GraphQL page is merged. +Be aware that CI tests for that second MR will fail with a bad link until the +main MR that adds the new GraphQL page is merged. And that's all you need! diff --git a/doc/development/documentation/workflow.md b/doc/development/documentation/workflow.md index c3e15cb1b2b..488c71a6328 100644 --- a/doc/development/documentation/workflow.md +++ b/doc/development/documentation/workflow.md @@ -29,7 +29,7 @@ is added or updated. The following are added by the issue or merge request autho The following are also added by members of the Technical Writing team: -- A documentation [scoped label](../../user/project/labels.md#scoped-labels-premium) with the +- A documentation [scoped label](../../user/project/labels.md#scoped-labels) with the `docs::` prefix. For example, `~docs::improvement`. - The `~Technical Writing` [team label](../contributing/issue_workflow.md#team-labels). diff --git a/doc/development/ee_features.md b/doc/development/ee_features.md index e7954fa910b..01f9d9b16fb 100644 --- a/doc/development/ee_features.md +++ b/doc/development/ee_features.md @@ -26,6 +26,16 @@ setting the [`FOSS_ONLY` environment variable](https://gitlab.com/gitlab-org/git to something that evaluates as `true`. The same works for running tests (for example `FOSS_ONLY=1 yarn jest`). +## CI pipelines in a FOSS context + +By default, merge request pipelines for development run in an EE-context only. If you are +developing features that differ between FOSS and EE, you may wish to run pipelines in a +FOSS context as well. + +To run pipelines in both contexts, include `RUN AS-IF-FOSS` in the merge request title. + +See the [As-if-FOSS jobs](pipelines.md#as-if-foss-jobs) pipelines documentation for more information. + ## Separation of EE code All EE code should be put inside the `ee/` top-level directory. The @@ -929,7 +939,7 @@ export default { - We can use slots and/or scoped slots to achieve the same thing as we did with mixins. If you only need an EE component there is no need to create the CE component. -1. First, we have a CE component that can render a slot incase we need EE template and functionality to be decorated on top of the CE base. +1. First, we have a CE component that can render a slot in case we need EE template and functionality to be decorated on top of the CE base. ```vue // ./ce/my_component.vue @@ -1030,7 +1040,7 @@ separate SCSS file in an appropriate directory within `app/assets/stylesheets`. In some cases, this is not entirely possible or creating dedicated SCSS file is an overkill, e.g. a text style of some component is different for EE. In such cases, -styles are usually kept in stylesheet that is common for both CE and EE, and it is wise +styles are usually kept in a stylesheet that is common for both CE and EE, and it is wise to isolate such ruleset from rest of CE rules (along with adding comment describing the same) to avoid conflicts during CE to EE merge. diff --git a/doc/development/elasticsearch.md b/doc/development/elasticsearch.md index 2f01692e944..e70cf456101 100644 --- a/doc/development/elasticsearch.md +++ b/doc/development/elasticsearch.md @@ -15,6 +15,8 @@ the [Elasticsearch integration documentation](../integration/elasticsearch.md#en In June 2019, Mario de la Ossa hosted a Deep Dive (GitLab team members only: `https://gitlab.com/gitlab-org/create-stage/issues/1`) on GitLab's [Elasticsearch integration](../integration/elasticsearch.md) to share his domain specific knowledge with anyone who may work in this part of the code base in the future. You can find the [recording on YouTube](https://www.youtube.com/watch?v=vrvl-tN2EaA), and the slides on [Google Slides](https://docs.google.com/presentation/d/1H-pCzI_LNrgrL5pJAIQgvLX8Ji0-jIKOg1QeJQzChug/edit) and in [PDF](https://gitlab.com/gitlab-org/create-stage/uploads/c5aa32b6b07476fa8b597004899ec538/Elasticsearch_Deep_Dive.pdf). Everything covered in this deep dive was accurate as of GitLab 12.0, and while specific details may have changed since then, it should still serve as a good introduction. +In August 2020, a second Deep Dive was hosted, focusing on [GitLab's specific architecture for multi-indices support](#zero-downtime-reindexing-with-multiple-indices). The [recording on YouTube](https://www.youtube.com/watch?v=0WdPR9oB2fg) and the [slides](https://lulalala.gitlab.io/gitlab-elasticsearch-deepdive) are available. Everything covered in this deep dive was accurate as of GitLab 13.3. + ## Supported Versions See [Version Requirements](../integration/elasticsearch.md#version-requirements). @@ -218,7 +220,7 @@ logs will also include the time spent on Database and Gitaly requests, which may help to diagnose which part of the search is performing poorly. There are additional logs specific to Elasticsearch that are sent to -[`elasticsearch.log`](../administration/logs.md#elasticsearchlog-starter-only) +[`elasticsearch.log`](../administration/logs.md#elasticsearchlog) that may contain information to help diagnose performance issues. ### Performance Bar @@ -229,7 +231,7 @@ be used both locally in development and on any deployed GitLab instance to diagnose poor search performance. This will show the exact queries being made, which is useful to diagnose why a search might be slow. -### Correlation ID and X-Opaque-Id +### Correlation ID and `X-Opaque-Id` Our [correlation ID](./distributed_tracing.md#developer-guidelines-for-working-with-correlation-ids) diff --git a/doc/development/experiment_guide/index.md b/doc/development/experiment_guide/index.md index b1a15c7749f..07b803603a5 100644 --- a/doc/development/experiment_guide/index.md +++ b/doc/development/experiment_guide/index.md @@ -16,7 +16,7 @@ In either case, an outcome of the experiment should be posted to the issue with ## Code reviews -Since the code of experiments will not be part of the codebase for a long time and we want to iterate fast to retrieve data,the code quality of experiments might sometimes not fulfill our standards but should not negatively impact the availability of GitLab whether the experiment is running or not. +Since the code of experiments will not be part of the codebase for a long time and we want to iterate fast to retrieve data, the code quality of experiments might sometimes not fulfill our standards but should not negatively impact the availability of GitLab whether the experiment is running or not. Initially experiments will only be deployed to a fraction of users but we still want a flawless experience for those users. Therefore, experiments still require tests. For reviewers and maintainers: if you find code that would usually not make it through the review, but is temporarily acceptable, please mention your concerns but note that it's not necessary to change. diff --git a/doc/development/fe_guide/development_process.md b/doc/development/fe_guide/development_process.md index ff36b8b5c6a..2b64534e7c9 100644 --- a/doc/development/fe_guide/development_process.md +++ b/doc/development/fe_guide/development_process.md @@ -4,7 +4,7 @@ You can find more about the organization of the frontend team in the [handbook]( ## Development Checklist -The idea is to remind us about specific topics during the time we build a new feature or start something. This is a common practice in other industries (like pilots) that also use standardised checklists to reduce problems early on. +The idea is to remind us about specific topics during the time we build a new feature or start something. This is a common practice in other industries (like pilots) that also use standardized checklists to reduce problems early on. Copy the content over to your issue or merge request and if something doesn't apply simply remove it from your current list. diff --git a/doc/development/fe_guide/frontend_faq.md b/doc/development/fe_guide/frontend_faq.md index 71436a7c7fb..0a26fdba934 100644 --- a/doc/development/fe_guide/frontend_faq.md +++ b/doc/development/fe_guide/frontend_faq.md @@ -162,7 +162,7 @@ To return to the normal development mode: 1. Open `gitlab.yaml` located in your `gitlab` installation folder, scroll down to the `webpack` section and change back `dev_server` to `enabled: true`. 1. Run `yarn clean` to remove the production assets and free some space (optional). 1. Start webpack again: `gdk start webpack`. -1. Restart GDK: `gdk-restart rails-web`. +1. Restart GDK: `gdk restart rails-web`. ### 8. Babel polyfills diff --git a/doc/development/fe_guide/graphql.md b/doc/development/fe_guide/graphql.md index f5e16d377f1..82cd19dce4f 100644 --- a/doc/development/fe_guide/graphql.md +++ b/doc/development/fe_guide/graphql.md @@ -1,3 +1,10 @@ +--- +type: reference, dev +stage: none +group: Development +info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +--- + # GraphQL ## Getting Started @@ -32,6 +39,9 @@ can help you learn how to integrate Vue Apollo. For other use cases, check out the [Usage outside of Vue](#usage-outside-of-vue) section. +We use [Immer](https://immerjs.github.io/immer/docs/introduction) for immutable cache updates; +see [Immutability and cache updates](#immutability-and-cache-updates) for more information. + ### Tooling - [Apollo Client Devtools](https://github.com/apollographql/apollo-client-devtools) @@ -131,6 +141,56 @@ fragment DesignItem on Design { More about fragments: [GraphQL Docs](https://graphql.org/learn/queries/#fragments) +## Global IDs + +GitLab's GraphQL API expresses `id` fields as Global IDs rather than the PostgreSQL +primary key `id`. Global ID is [a convention](https://graphql.org/learn/global-object-identification/) +used for caching and fetching in client-side libraries. + +To convert a Global ID to the primary key `id`, you can use `getIdFromGraphQLId`: + +```javascript +import { getIdFromGraphQLId } from '~/graphql_shared/utils'; + +const primaryKeyId = getIdFromGraphQLId(data.id); +``` + +## Immutability and cache updates + +From Apollo version 3.0.0 all the cache updates need to be immutable; it needs to be replaced entirely +with a **new and updated** object. + +To facilitate the process of updating the cache and returning the new object we use the library [Immer](https://immerjs.github.io/immer/docs/introduction). +When possible, follow these conventions: + +- The updated cache is named `data`. +- The original cache data is named `sourceData`. + +A typical update process looks like this: + +```javascript +... +const sourceData = client.readQuery({ query }); + +const data = produce(sourceData, draftState => { + draftState.commits.push(newCommit); +}); + +client.writeQuery({ + query, + data, +}); +... +``` + +As shown in the code example by using `produce`, we can perform any kind of direct manipulation of the +`draftState`. Besides, `immer` guarantees that a new state which includes the changes to `draftState` will be generated. + +Finally, to verify whether the immutable cache update is working properly, we need to change +`assumeImmutableResults` to `true` in the `default client config` (see [Apollo Client](#apollo-client) for more info). + +If everything is working properly `assumeImmutableResults` should remain set to `true`. + ## Usage in Vue To use Vue Apollo, import the [Vue Apollo](https://github.com/vuejs/vue-apollo) plugin as well @@ -602,6 +662,174 @@ it('calls mutation on submitting form ', () => { }); ``` +### Testing with mocked Apollo Client + +To test the logic of Apollo cache updates, we might want to mock an Apollo Client in our unit tests. To separate tests with mocked client from 'usual' unit tests, it's recommended to create an additional component factory. This way we only create Apollo Client instance when it's necessary: + +```javascript +function createComponent() {...} + +function createComponentWithApollo() {...} +``` + +We use [`mock-apollo-client`](https://www.npmjs.com/package/mock-apollo-client) library to mock Apollo client in tests. + +```javascript +import { createMockClient } from 'mock-apollo-client'; +``` + +Then we need to inject `VueApollo` to Vue local instance (`localVue.use()` can also be called within `createComponentWithApollo()`) + +```javascript +import VueApollo from 'vue-apollo'; +import { createLocalVue } from '@vue/test-utils'; + +const localVue = createLocalVue(); +localVue.use(VueApollo); +``` + +After this, on the global `describe`, we should create a variable for `fakeApollo`: + +```javascript +describe('Some component with Apollo mock', () => { + let wrapper; + let fakeApollo +}) +``` + +Within component factory, we need to define an array of _handlers_ for every query or mutation: + +```javascript +import getDesignListQuery from '~/design_management/graphql/queries/get_design_list.query.graphql'; +import permissionsQuery from '~/design_management/graphql/queries/design_permissions.query.graphql'; +import moveDesignMutation from '~/design_management/graphql/mutations/move_design.mutation.graphql'; + +describe('Some component with Apollo mock', () => { + let wrapper; + let fakeApollo; + + function createComponentWithApollo() { + const requestHandlers = [ + [getDesignListQuery, jest.fn().mockResolvedValue(designListQueryResponse)], + [permissionsQuery, jest.fn().mockResolvedValue(permissionsQueryResponse)], + ]; + } +}) +``` + +After this, we need to create a mock Apollo Client instance using a helper: + +```javascript +import createMockApollo from 'jest/helpers/mock_apollo_helper'; + +describe('Some component with Apollo mock', () => { + let wrapper; + let fakeApollo; + + function createComponentWithApollo() { + const requestHandlers = [ + [getDesignListQuery, jest.fn().mockResolvedValue(designListQueryResponse)], + [permissionsQuery, jest.fn().mockResolvedValue(permissionsQueryResponse)], + ]; + + fakeApollo = createMockApollo(requestHandlers); + wrapper = shallowMount(Index, { + localVue, + apolloProvider: fakeApollo, + }); + } +}) +``` + +NOTE: **Note:** +When mocking resolved values, make sure the structure of the response is the same as actual API response: i.e. root property should be `data` for example + +When testing queries, please keep in mind they are promises, so they need to be _resolved_ to render a result. Without resolving, we can check the `loading` state of the query: + +```javascript +it('renders a loading state', () => { + createComponentWithApollo(); + + expect(wrapper.find(LoadingSpinner).exists()).toBe(true) +}); + +it('renders designs list', async () => { + createComponentWithApollo(); + + jest.runOnlyPendingTimers(); + await wrapper.vm.$nextTick(); + + expect(findDesigns()).toHaveLength(3); +}); +``` + +If we need to test a query error, we need to mock a rejected value as request handler: + +```javascript +function createComponentWithApollo() { + ... + const requestHandlers = [ + [getDesignListQuery, jest.fn().mockRejectedValue(new Error('GraphQL error')], + ]; + ... +} +... + +it('renders error if query fails', async () => { + createComponent() + + jest.runOnlyPendingTimers(); + await wrapper.vm.$nextTick(); + + expect(wrapper.find('.test-error').exists()).toBe(true) +}) +``` + +Request handlers can also be passed to component factory as a parameter. + +Mutations could be tested the same way with a few additional `nextTick`s to get the updated result: + +```javascript +function createComponentWithApollo({ + moveHandler = jest.fn().mockResolvedValue(moveDesignMutationResponse), +}) { + moveDesignHandler = moveHandler; + + const requestHandlers = [ + [getDesignListQuery, jest.fn().mockResolvedValue(designListQueryResponse)], + [permissionsQuery, jest.fn().mockResolvedValue(permissionsQueryResponse)], + [moveDesignMutation, moveDesignHandler], + ]; + + fakeApollo = createMockApollo(requestHandlers); + wrapper = shallowMount(Index, { + localVue, + apolloProvider: fakeApollo, + }); +} +... +it('calls a mutation with correct parameters and reorders designs', async () => { + createComponentWithApollo({}); + + wrapper.find(VueDraggable).vm.$emit('change', { + moved: { + newIndex: 0, + element: designToMove, + }, + }); + + expect(moveDesignHandler).toHaveBeenCalled(); + + await wrapper.vm.$nextTick(); + + expect( + findDesigns() + .at(0) + .props('id'), + ).toBe('2'); +}); +``` + ## Handling errors GitLab's GraphQL mutations currently have two distinct error modes: [Top-level](#top-level-errors) and [errors-as-data](#errors-as-data). diff --git a/doc/development/fe_guide/style/index.md b/doc/development/fe_guide/style/index.md index 8d70d4db893..4ca409664de 100644 --- a/doc/development/fe_guide/style/index.md +++ b/doc/development/fe_guide/style/index.md @@ -1,11 +1,11 @@ # GitLab development style guides -See below the relevant style guides, guidelines, linting, and other information for developing GitLab. +See below for the relevant style guides, guidelines, linting, and other information for developing GitLab. ## JavaScript style guide We use `eslint` to enforce our [JavaScript style guides](javascript.md). Our guide is based on -the excellent [AirBnB](https://github.com/airbnb/javascript) style guide with a few small +the excellent [Airbnb](https://github.com/airbnb/javascript) style guide with a few small changes. ## SCSS style guide diff --git a/doc/development/fe_guide/vuex.md b/doc/development/fe_guide/vuex.md index 9573dd36e63..4badf3f0845 100644 --- a/doc/development/fe_guide/vuex.md +++ b/doc/development/fe_guide/vuex.md @@ -55,10 +55,6 @@ export const createStore = () => }); ``` -_Note:_ Until this -[RFC](https://gitlab.com/gitlab-org/frontend/rfcs/-/issues/20) is implemented, -the above will need to disable the `import/prefer-default-export` ESLint rule. - ### `state.js` The first thing you should do before writing any code is to design the state. @@ -220,12 +216,15 @@ A mutation written like this is harder to maintain and more error prone. We shou // Good export default { [types.MARK_AS_CLOSED](state, itemId) { - const item = state.items.find(i => i.id == itemId); - Vue.set(item, 'closed', true) + const item = state.items.find(x => x.id === itemId); - state.items.splice(index, 1, item) - } -} + if (!item) { + return; + } + + Vue.set(item, 'closed', true); + }, +}; ``` This approach is better because: diff --git a/doc/development/feature_flags/controls.md b/doc/development/feature_flags/controls.md index 09260be1264..605b5919e0b 100644 --- a/doc/development/feature_flags/controls.md +++ b/doc/development/feature_flags/controls.md @@ -1,3 +1,10 @@ +--- +type: reference, dev +stage: none +group: Development +info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +--- + # Feature flag controls ## Access diff --git a/doc/development/feature_flags/development.md b/doc/development/feature_flags/development.md index 9b30187fcd1..29bd0ca0a7e 100644 --- a/doc/development/feature_flags/development.md +++ b/doc/development/feature_flags/development.md @@ -1,3 +1,10 @@ +--- +type: reference, dev +stage: none +group: Development +info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +--- + # Developing with feature flags This document provides guidelines on how to use feature flags @@ -18,9 +25,7 @@ This document is the subject of continued work as part of an epic to [improve in ## Types of feature flags -Currently, only a single type of feature flag is available. -Additional feature flag types will be provided in the future, -with descriptions for their usage. +Choose a feature flag type that matches the expected usage. ### `development` type @@ -33,6 +38,53 @@ ideally created using the [Feature Flag Roll Out template](https://gitlab.com/gi NOTE: **Note:** This is the default type used when calling `Feature.enabled?`. +### `ops` type + +`ops` feature flags are long-lived feature flags that control operational aspects +of GitLab's behavior. For example, feature flags that disable features that might +have a performance impact, like special Sidekiq worker behavior. + +`ops` feature flags likely do not have rollout issues, as it is hard to +predict when they will be enabled or disabled. + +To use `ops` feature flags, you must append `type: :ops` to `Feature.enabled?` +invocations: + +```ruby +# Check if feature flag is enabled +Feature.enabled?(:my_ops_flag, project, type: ops) + +# Check if feature flag is disabled +Feature.disabled?(:my_ops_flag, project, type: ops) + +# Push feature flag to Frontend +push_frontend_feature_flag(:my_ops_flag, project, type: :ops) +``` + +### `licensed` type + +`licensed` feature flags are used to temporarily disable licensed features. There +should be a one-to-one mapping of `licensed` feature flags to licensed features. + +`licensed` feature flags must be `default_enabled: true`, because that's the only +supported option in the current implementation. This is under development as per +the [related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/218667). + +The `licensed` type has a dedicated set of functions to check if a licensed +feature is available for a project or namespace. This check validates +if the license is assigned to the namespace and feature flag itself. +The `licensed` feature flag has the same name as a licensed feature name: + +```ruby +# Good: checks if feature flag is enabled +project.feature_available?(:my_licensed_feature) +namespace.feature_available?(:my_licensed_feature) + +# Bad: licensed flag must be accessed via `feature_available?` +Feature.enabled?(:my_licensed_feature, type: :licensed) +push_frontend_feature_flag(:my_licensed_feature, type: :licensed) +``` + ## Feature flag definition and validation > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229161) in GitLab 13.3. @@ -58,7 +110,7 @@ Each feature flag is defined in a separate YAML file consisting of a number of f |---------------------|----------|----------------------------------------------------------------| | `name` | yes | Name of the feature flag. | | `type` | yes | Type of feature flag. | -| `default_enabled` | yes | The default state of the feature flag that is strongly validated, with `default_enabled:` passed as an argument. | +| `default_enabled` | yes | The default state of the feature flag that is strictly validated, with `default_enabled:` passed as an argument. | | `introduced_by_url` | no | The URL to the Merge Request that introduced the feature flag. | | `rollout_issue_url` | no | The URL to the Issue covering the feature flag rollout. | | `group` | no | The [group](https://about.gitlab.com/handbook/product/product-categories/#devops-stages) that owns the feature flag. | @@ -77,16 +129,16 @@ Only feature flags that have a YAML definition file can be used when running the ```shell $ bin/feature-flag my-feature-flag ->> Please specify the group introducing feature flag, like `group::apm`: +>> Specify the group introducing the feature flag, like `group::apm`: ?> group::memory ->> If you have MR open, can you paste the URL here? (or enter to skip) +>> URL of the MR introducing the feature flag (enter to skip): ?> https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38602 ->> Open this URL and fill the rest of details: +>> Open this URL and fill in the rest of the details: https://gitlab.com/gitlab-org/gitlab/-/issues/new?issue%5Btitle%5D=%5BFeature+flag%5D+Rollout+of+%60test-flag%60&issuable_template=Feature+Flag+Roll+Out ->> Paste URL of `rollout issue` here, or enter to skip: +>> URL of the rollout issue (enter to skip): ?> https://gitlab.com/gitlab-org/gitlab/-/issues/232533 create config/feature_flags/development/test-flag.yml --- @@ -140,6 +192,21 @@ if Feature.disabled?(:my_feature_flag, project, default_enabled: true) end ``` +If not specified, the default feature flag type for `Feature.enabled?` and `Feature.disabled?` +is `type: development`. For all other feature flag types, you must specify the `type:`: + +```ruby +if Feature.enabled?(:feature_flag, project, type: :ops) + # execute code if ops feature flag is enabled +else + # execute code if ops feature flag is disabled +end + +if Feature.disabled?(:my_feature_flag, project, type: :ops) + # execute code if feature flag is disabled +end +``` + ### Frontend Use the `push_frontend_feature_flag` method for frontend code, which is @@ -185,6 +252,15 @@ before_action do end ``` +If not specified, the default feature flag type for `push_frontend_feature_flag` +is `type: development`. For all other feature flag types, you must specify the `type:`: + +```ruby +before_action do + push_frontend_feature_flag(:vim_bindings, project, type: :ops) +end +``` + ### Feature actors **It is strongly advised to use actors with feature flags.** Actors provide a simple @@ -229,7 +305,7 @@ used as an actor for `Feature.enabled?`. ### Feature flags for licensed features If a feature is license-gated, there's no need to add an additional -explicit feature flag check since the flag will be checked as part of the +explicit feature flag check since the flag is checked as part of the `License.feature_available?` call. Similarly, there's no need to "clean up" a feature flag once the feature has reached general availability. @@ -238,12 +314,16 @@ The [`Project#feature_available?`](https://gitlab.com/gitlab-org/gitlab/blob/4cc [`License.feature_available?`](https://gitlab.com/gitlab-org/gitlab/blob/4cc1c62918aa4c31750cb21dfb1a6c3492d71080/ee/app/models/license.rb#L293-300) (EE) methods all implicitly check for a by default enabled feature flag with the same name as the provided argument. -You'd still want to use an explicit `Feature.enabled?` check if your new feature -isn't gated by a License or Plan. - **An important side-effect of the implicit feature flags mentioned above is that unless the feature is explicitly disabled or limited to a percentage of users, -the feature flag check will default to `true`.** +the feature flag check defaults to `true`.** + +NOTE: **Note:** +Due to limitations with `feature_available?`, the YAML definition for `licensed` feature +flags accepts only `default_enabled: true`. This is under development as per the +[related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/218667). + +#### Alpha/beta licensed feature flags This is relevant when developing the feature using [several smaller merge requests](https://about.gitlab.com/handbook/values/#make-small-merge-requests), or when the feature is considered to be an @@ -259,13 +339,31 @@ GitLab.com and self-managed instances, you should use the method, according to our [definitions](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha-beta-ga). This ensures the feature is disabled unless the feature flag is _explicitly_ enabled. +CAUTION: **Caution:** +If `alpha_feature_available?` or `beta_feature_available?` is used, the YAML definition +for the feature flag must use `default_enabled: [false, true]`, because the usage +of the feature flag is undefined. These methods may change, as per the +[related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/218667). + +The resulting YAML should be similar to: + +```yaml +name: scoped_labels +group: group::memory +type: licensed +# The `default_enabled:` is undefined +# as `feature_available?` uses `default_enabled: true` +# as `beta_feature_available?` uses `default_enabled: false` +default_enabled: [false, true] +``` + ### Feature groups Feature groups must be defined statically in `lib/feature.rb` (in the `.register_feature_groups` method), but their implementation can obviously be -dynamic (querying the DB etc.). +dynamic (querying the DB, for example). -Once defined in `lib/feature.rb`, you will be able to activate a +Once defined in `lib/feature.rb`, you can to activate a feature for a given feature group via the [`feature_group` parameter of the features API](../../api/features.md#set-or-create-a-feature) ### Enabling a feature flag locally (in development) @@ -276,7 +374,7 @@ In the rails console (`rails c`), enter the following command to enable a featur Feature.enable(:feature_flag_name) ``` -Similarly, the following command will disable a feature flag: +Similarly, the following command disables a feature flag: ```ruby Feature.disable(:feature_flag_name) @@ -290,7 +388,7 @@ Feature.enable(:feature_flag_name, Project.find_by_full_path("root/my-project")) ## Feature flags in tests -Introducing a feature flag into the codebase creates an additional codepath that should be tested. +Introducing a feature flag into the codebase creates an additional code path that should be tested. It is strongly advised to test all code affected by a feature flag, both when **enabled** and **disabled** to ensure the feature works properly. @@ -325,10 +423,10 @@ Feature.enabled?(:ci_live_trace, project2) # => false The behavior of FlipperGate is as follows: -1. You can enable an override for a specified actor to be enabled +1. You can enable an override for a specified actor to be enabled. 1. You can disable (remove) an override for a specified actor, - falling back to default state -1. There's no way to model that you explicitly disable a specified actor + falling back to the default state. +1. There's no way to model that you explicitly disabled a specified actor. ```ruby Feature.enable(:my_feature) @@ -369,7 +467,7 @@ Feature.enable_percentage_of_time(:my_feature_3, 50) Feature.enable_percentage_of_actors(:my_feature_4, 50) ``` -Each feature flag that has a defined state will be persisted +Each feature flag that has a defined state is persisted during test execution time: ```ruby diff --git a/doc/development/feature_flags/index.md b/doc/development/feature_flags/index.md index 0c1e34edc6f..2643571aec3 100644 --- a/doc/development/feature_flags/index.md +++ b/doc/development/feature_flags/index.md @@ -1,15 +1,46 @@ +--- +type: index, dev +stage: none +group: Development +info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +--- + # Feature flags in development of GitLab -[Feature Flags](../../operations/feature_flags.md) -can be used to gradually roll out changes, be -it a new feature, or a performance improvement. By using feature flags, we can -comfortably measure the impact of our changes, while still being able to easily -disable those changes, without having to revert an entire release. +Feature flags can be used to gradually deploy changes, regardless of whether +they are new features or performance improvements. By using feature flags, +you can determine the impact of GitLab-directed changes, while still being able +to disable those changes without having to revert an entire release. + +Before using feature flags for GitLab's development, review the following development guides: + +NOTE: **Note:** +The feature flags used by GitLab to deploy its own features **are not** the same +as the [feature flags offered as part of the product](../../operations/feature_flags.md). + +For an overview about starting with feature flags in GitLab's development, +use this [training template](https://gitlab.com/gitlab-com/www-gitlab-com/-/blob/master/.gitlab/issue_templates/feature-flag-training.md). + +Development guides: + +- [Process for using features flags](process.md): When you should use + feature flags in the development of GitLab, what's the cost of using them, + and how to include them in a release. +- [Developing with feature flags](development.md): Learn about the types of + feature flags, their definition and validation, how to create them, frontend and + backend details, and other information. +- [Documenting features deployed behind feature flags](../documentation/feature_flags.md): + How to document features deployed behind feature flags, and how to update the + documentation for features' flags when their states change. +- [Controlling feature flags](controls.md): Learn the process for deploying + a new feature, enabling it on GitLab.com, communicating the change, + logging, and cleaning up. -Before using feature flags for GitLab's development, read through the following: +User guides: -- [Process for using features flags](process.md). -- [Developing with feature flags](development.md). -- [Controlling feature flags](controls.md). -- [Documenting features deployed behind feature flags](../documentation/feature_flags.md). -- [How GitLab administrators can enable and disable features behind flags](../../administration/feature_flags.md). +- [How GitLab administrators can enable and disable features behind flags](../../administration/feature_flags.md): + An explanation for GitLab administrators about how they can + enable or disable GitLab features behind feature flags. +- [What "features deployed behind flags" means to the GitLab user](../../user/feature_flags.md): + An explanation for GitLab users regarding how certain features + might not be available to them until they are enabled. diff --git a/doc/development/feature_flags/process.md b/doc/development/feature_flags/process.md index 5dc3cf44a6e..b327eec58e8 100644 --- a/doc/development/feature_flags/process.md +++ b/doc/development/feature_flags/process.md @@ -1,3 +1,10 @@ +--- +type: reference, dev +stage: none +group: Development +info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +--- + # Feature flags process ## Feature flags for user applications diff --git a/doc/development/features_inside_dot_gitlab.md b/doc/development/features_inside_dot_gitlab.md new file mode 100644 index 00000000000..cb883471adf --- /dev/null +++ b/doc/development/features_inside_dot_gitlab.md @@ -0,0 +1,16 @@ +# Features inside the `.gitlab/` directory + +We have implemented standard features that depend on configuration files in the `.gitlab/` directory. You can find `.gitlab/` in various GitLab repositories. +When implementing new features, please refer to these existing features to avoid conflicts: + +- [Custom Dashboards](../operations/metrics/dashboards/index.md#add-a-new-dashboard-to-your-project): `.gitlab/dashboards/`. +- [Issue Templates](../user/project/description_templates.md#creating-issue-templates): `.gitlab/issue_templates/`. +- [Merge Request Templates](../user/project/description_templates.md#creating-merge-request-templates): `.gitlab/merge_request_templates/`. +- [GitLab Kubernetes Agents](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/configuration_repository.md#layout): `.gitlab/agents/`. +- [CODEOWNERS](../user/project/code_owners.md#how-to-set-up-code-owners): `.gitlab/CODEOWNERS`. +- [Route Maps](../ci/review_apps/#route-maps): `.gitlab/route-map.yml`. +- [Customize Auto DevOps Helm Values](../topics/autodevops/customize.md#customize-values-for-helm-chart): `.gitlab/auto-deploy-values.yaml`. +- [GitLab managed apps CI/CD](../user/clusters/applications.md#usage): `.gitlab/managed-apps/config.yaml`. +- [Insights](../user/project/insights/index.md#configure-your-insights): `.gitlab/insights.yml`. +- [Service Desk Templates](../user/project/service_desk.md#using-customized-email-templates): `.gitlab/service_desk_templates/`. +- [Web IDE](../user/project/web_ide/#web-ide-configuration-file): `.gitlab/.gitlab-webide.yml`. diff --git a/doc/development/geo.md b/doc/development/geo.md index 57959b07e49..5b4af1c9931 100644 --- a/doc/development/geo.md +++ b/doc/development/geo.md @@ -147,7 +147,7 @@ request must also include the SHA256 sum of the file. An example JWT payload looks like: ```yaml -{ "data": { sha256: "31806bb23580caab78040f8c45d329f5016b0115" }, iat: "1234567890" } +{"data": {sha256: "31806bb23580caab78040f8c45d329f5016b0115"}, iat: "1234567890"} ``` If the requested file matches the requested SHA256 sum, then the Geo diff --git a/doc/development/geo/framework.md b/doc/development/geo/framework.md index 64c9030e3dd..b720a6ca47e 100644 --- a/doc/development/geo/framework.md +++ b/doc/development/geo/framework.md @@ -128,7 +128,7 @@ When this is set in place, it's easy to access the replicator through the model: ```ruby -package_file = Packages::PackageFile.find(4) # just a random id as example +package_file = Packages::PackageFile.find(4) # just a random ID as example replicator = package_file.replicator ``` @@ -235,11 +235,10 @@ For example, to add support for files referenced by a `Widget` model with a `ee/lib/gitlab/geo.rb`: ```ruby - def self.replicator_classes - classes = [::Geo::PackageFileReplicator, - ::Geo::WidgetReplicator] - - classes.select(&:enabled?) + REPLICATOR_CLASSES = [ + ::Geo::PackageFileReplicator, + ::Geo::WidgetReplicator + ] end ``` @@ -315,10 +314,6 @@ For example, to add support for files referenced by a `Widget` model with a end ``` - The method `has_create_events?` should return `true` in most of the cases. - However, if the entity you add doesn't have the create event, don't add the - method at all. - 1. Update `REGISTRY_CLASSES` in `ee/app/workers/geo/secondary/registry_consistency_worker.rb`. 1. Add `widget_registry` to `ActiveSupport::Inflector.inflections` in `config/initializers_before_autoloader/000_inflections.rb`. @@ -416,18 +411,26 @@ for verification state to the widgets table: # frozen_string_literal: true class AddVerificationFailureLimitToWidgets < ActiveRecord::Migration[6.0] + include Gitlab::Database::MigrationHelpers + DOWNTIME = false disable_ddl_transaction! - def change - add_text_limit :widgets, :verification_failure, 255 + CONSTRAINT_NAME = 'widget_verification_failure_text_limit' + + def up + add_text_limit :widget, :verification_failure, 255, constraint_name: CONSTRAINT_NAME + end + + def down + remove_check_constraint(:widget, CONSTRAINT_NAME) end end ``` 1. Add a partial index on `verification_failure` and `verification_checksum` to ensure - re-verification can be performed efficiently. Add a migration in `ee/db/geo/migrate/`: + re-verification can be performed efficiently: ```ruby # frozen_string_literal: true @@ -453,9 +456,9 @@ for verification state to the widgets table: ##### Option 2: Create a separate `widget_states` table with verification state fields -1. Add a migration in `ee/db/geo/migrate/` to create a `widget_states` table and add a - partial index on `verification_failure` and `verification_checksum` to ensure - re-verification can be performed efficiently. Order the columns according to [our guidelines](../ordering_table_columns.md): +1. Create a `widget_states` table and add a partial index on `verification_failure` and + `verification_checksum` to ensure re-verification can be performed efficiently. Order + the columns according to [our guidelines](../ordering_table_columns.md): ```ruby # frozen_string_literal: true @@ -520,44 +523,37 @@ Widgets should now be verified by Geo! Metrics are gathered by `Geo::MetricsUpdateWorker`, persisted in `GeoNodeStatus` for display in the UI, and sent to Prometheus. -1. Add fields `widget_count`, `widget_checksummed_count`, - `widget_checksum_failed_count`, `widget_synced_count`, - `widget_failed_count`, and `widget_registry_count` to - `GeoNodeStatus#RESOURCE_STATUS_FIELDS` array in - `ee/app/models/geo_node_status.rb`. -1. Add the same fields to `GeoNodeStatus#PROMETHEUS_METRICS` hash in - `ee/app/models/geo_node_status.rb`. -1. Add the same fields to `Sidekiq metrics` table in - `doc/administration/monitoring/prometheus/gitlab_metrics.md`. -1. Add the same fields to `GET /geo_nodes/status` example response in +1. Add fields `widgets_count`, `widgets_checksummed_count`, + `widgets_checksum_failed_count`, `widgets_synced_count`, + `widgets_failed_count`, and `widgets_registry_count` to + `GET /geo_nodes/status` example response in `doc/api/geo_nodes.md`. -1. Add the same fields to `ee/spec/models/geo_node_status_spec.rb` and - `ee/spec/factories/geo_node_statuses.rb`. -1. Set `widget_count` in `GeoNodeStatus#load_data_from_current_node`: +1. Add the same fields to `GET /geo_nodes/status` example response in + `ee/spec/fixtures/api/schemas/public_api/v4/geo_node_status.json`. +1. Add fields `geo_widgets`, `geo_widgets_checksummed`, + `geo_widgets_checksum_failed`, `geo_widgets_synced`, + `geo_widgets_failed`, and `geo_widgets_registry` to + `Sidekiq metrics` table in + `doc/administration/monitoring/prometheus/gitlab_metrics.md`. +1. Add the following to the parameterized table in + `ee/spec/models/geo_node_status_spec.rb`: ```ruby - self.widget_count = Geo::WidgetReplicator.primary_total_count + Geo::WidgetReplicator | :widget | :geo_widget_registry ``` -1. Add `GeoNodeStatus#load_widgets_data` to set `widget_synced_count`, - `widget_failed_count`, and `widget_registry_count`: +1. Add the following to `spec/factories/widgets.rb`: ```ruby - def load_widget_data - self.widget_synced_count = Geo::WidgetReplicator.synced_count - self.widget_failed_count = Geo::WidgetReplicator.failed_count - self.widget_registry_count = Geo::WidgetReplicator.registry_count + trait(:checksummed) do + with_file + verification_checksum { 'abc' } end - ``` -1. Call `GeoNodeStatus#load_widgets_data` in - `GeoNodeStatus#load_secondary_data`. - -1. Set `widget_checksummed_count` and `widget_checksum_failed_count` in - `GeoNodeStatus#load_verification_data`: - - ```ruby - self.widget_checksummed_count = Geo::WidgetReplicator.checksummed_count self.widget_checksum_failed_count = Geo::WidgetReplicator.checksum_failed_count + trait(:checksum_failure) do + with_file + verification_failure { 'Could not calculate the checksum' } + end ``` Widget replication and verification metrics should now be available in the API, @@ -573,7 +569,7 @@ the Admin Area UI, and Prometheus! null: true, resolver: ::Resolvers::Geo::WidgetRegistriesResolver, description: 'Find widget registries on this Geo node', - feature_flag: :geo_self_service_framework + feature_flag: :geo_widget_replication ``` 1. Add the new `widget_registries` field name to the `expected_fields` array in diff --git a/doc/development/gotchas.md b/doc/development/gotchas.md index 6dff9deb59d..f7b44e74c17 100644 --- a/doc/development/gotchas.md +++ b/doc/development/gotchas.md @@ -106,7 +106,7 @@ end Using `any_instance` to stub a method (elasticsearch_indexing) that has been defined on a prepended module (EE::ApplicationSetting) is not supported. ``` -### Alternative: `expect_next_instance_of` or `allow_next_instance_of` +### Alternative: `expect_next_instance_of`, `allow_next_instance_of`, `expect_next_found_instance_of` or `allow_next_found_instance_of` Instead of writing: @@ -130,8 +130,21 @@ end allow_next_instance_of(Project) do |project| allow(project).to receive(:add_import_job) end + +# Do this: +expect_next_found_instance_of(Project) do |project| + expect(project).to receive(:add_import_job) +end + +# Do this: +allow_next_found_instance_of(Project) do |project| + allow(project).to receive(:add_import_job) +end ``` +_**Note:** Since Active Record is not calling the `.new` method on model classes to instantiate the objects, +you should use `expect_next_found_instance_of` or `allow_next_found_instance_of` mock helpers to setup mock on objects returned by Active Record query & finder methods._ + If we also want to initialize the instance with some particular arguments, we could also pass it like: diff --git a/doc/development/graphql_guide/pagination.md b/doc/development/graphql_guide/pagination.md new file mode 100644 index 00000000000..bf9eaa99158 --- /dev/null +++ b/doc/development/graphql_guide/pagination.md @@ -0,0 +1,142 @@ +# GraphQL pagination + +## Types of pagination + +GitLab uses two primary types of pagination: **offset** and **keyset** +(sometimes called cursor-based) pagination. +The GraphQL API mainly uses keyset pagination, falling back to offset pagination when needed. + +### Offset pagination + +This is the traditional, page-by-page pagination, that is most common, +and used across much of GitLab. You can recognize it by +a list of page numbers near the bottom of a page, which, when clicked, +take you to that page of results. + +For example, when you click **Page 100**, we send `100` to the +backend. For example, if each page has say 20 items, the +backend calculates `20 * 100 = 2000`, +and it queries the database by offsetting (skipping) the first 2000 +records and pulls the next 20. + +```plaintext +page number * page size = where to find my records +``` + +There are a couple of problems with this: + +- Performance. When we query for page 100 (which gives an offset of + 2000), then the database has to scan through the table to that + specific offset, and then pick up the next 20 records. As the offset + increases, the performance degrades quickly. + Read more in + [The SQL I Love <3. Efficient pagination of a table with 100M records](http://allyouneedisbackend.com/blog/2017/09/24/the-sql-i-love-part-1-scanning-large-table/). + +- Data stability. When you get the 20 items for page 100 (at + offset 2000), GitLab shows those 20 items. If someone then + deletes or adds records in page 99 or before, the items at + offset 2000 become a different set of items. You can even get into a + situation where, when paginating, you could skip over items, + because the list keeps changing. + Read more in + [Pagination: You're (Probably) Doing It Wrong](https://coderwall.com/p/lkcaag/pagination-you-re-probably-doing-it-wrong). + +### Keyset pagination + +Given any specific record, if you know how to calculate what comes +after it, you can query the database for those specific records. + +For example, suppose you have a list of issues sorted by creation date. +If you know the first item on a page has a specific date (say Jan 1), you can ask +for all records that were created after that date and take the first 20. +It no longer matters if many are deleted or added, as you always ask for +the ones after that date, and so get the correct items. + +Unfortunately, there is no easy way to know if the issue created +on Jan 1 is on page 20 or page 100. + +Some of the benefits and tradeoffs of keyset pagination are + +- Performance is much better. + +- Data stability is greater since you're not going to miss records due to + deletions or insertions. + +- It's the best way to do infinite scrolling. + +- It's more difficult to program and maintain. Easy for `updated_at` and + `sort_order`, complicated (or impossible) for complex sorting scenarios. + +## Implementation + +When pagination is supported for a query, GitLab defaults to using +keyset pagination. You can see where this is configured in +[`pagination/connections.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/graphql/pagination/connections.rb). +If a query returns `ActiveRecord::Relation`, keyset pagination is automatically used. + +This was a conscious decision to support performance and data stability. + +However, there are some cases where we have to use the offset +pagination connection, `OffsetActiveRecordRelationConnection`, such as when +sorting by label priority in issues, due to the complexity of the sort. + +<!-- ### Keyset pagination --> + +<!-- ### Offset pagination --> + +<!-- ### External pagination --> + +## Testing + +Any GraphQL field that supports pagination and sorting should be tested +using the sorted paginated query shared example found in +[`graphql/sorted_paginated_query_shared_examples.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/support/shared_examples/graphql/sorted_paginated_query_shared_examples.rb). +It helps verify that your sort keys are compatible and that cursors +work properly. + +This is particularly important when using keyset pagination, as some sort keys might not be supported. + +Add a section to your request specs like this: + +```ruby +describe 'sorting and pagination' do + ... +end +``` + +You can then use +[`issues_spec.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/requests/api/graphql/project/issues_spec.rb) +as an example to construct your tests. + +[`graphql/sorted_paginated_query_shared_examples.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/support/shared_examples/graphql/sorted_paginated_query_shared_examples.rb) +also contains some documentation on how to use the shared examples. + +The shared example requires certain `let` variables and methods to be set up: + +```ruby +describe 'sorting and pagination' do + let(:sort_project) { create(:project, :public) } + let(:data_path) { [:project, :issues] } + + def pagination_query(params, page_info) + graphql_query_for( + 'project', + { 'fullPath' => sort_project.full_path }, + query_graphql_field('issues', params, "#{page_info} edges { node { id } }") + ) + end + + def pagination_results_data(data) + data.map { |issue| issue.dig('node', 'iid').to_i } + end + + context 'when sorting by weight' do + ... + context 'when ascending' do + it_behaves_like 'sorted paginated query' do + let(:sort_param) { 'WEIGHT_ASC' } + let(:first_param) { 2 } + let(:expected_results) { [weight_issue3.iid, weight_issue5.iid, weight_issue1.iid, weight_issue4.iid, weight_issue2.iid] } + end + end +``` diff --git a/doc/development/i18n/externalization.md b/doc/development/i18n/externalization.md index 1374ca92256..b5c5a199b1e 100644 --- a/doc/development/i18n/externalization.md +++ b/doc/development/i18n/externalization.md @@ -7,6 +7,9 @@ For working with internationalization (i18n), used tool for this task and there are a lot of applications that will help us to work with it. +TIP: **Tip:** +All `rake` commands described on this page must be run on a GitLab instance, usually GDK. + ## Setting up GitLab Development Kit (GDK) In order to be able to work on the [GitLab Community Edition](https://gitlab.com/gitlab-org/gitlab-foss) @@ -138,7 +141,90 @@ const label = __('Subscribe'); ``` In order to test JavaScript translations you have to change the GitLab -localization to other language than English and you have to generate JSON files +localization to another language than English and you have to generate JSON files +using `bin/rake gettext:po_to_json` or `bin/rake gettext:compile`. + +### Vue files + +In Vue files we make both the `__()` (double underscore parenthesis) function and the `s__()` (namespaced double underscore parenthesis) function available that you can import from the `~/locale` file. For instance: + +```javascript +import { __, s__ } from '~/locale'; +const label = __('Subscribe'); +const nameSpacedlabel = __('Plan|Subscribe'); +``` + +For the static text strings we suggest two patterns for using these translations in Vue files: + +- External constants file: + + ```javascript + javascripts + │ + └───alert_settings + │ │ constants.js + │ └───components + │ │ alert_settings_form.vue + + + // constants.js + + import { s__ } from '~/locale'; + + /* Integration constants */ + + export const I18N_ALERT_SETTINGS_FORM = { + saveBtnLabel: __('Save changes'), + }; + + + // alert_settings_form.vue + + import { + I18N_ALERT_SETTINGS_FORM, + } from '../constants'; + + <script> + export default { + i18n: { + I18N_ALERT_SETTINGS_FORM, + } + } + </script> + + <template> + <gl-button + ref="submitBtn" + variant="success" + type="submit" + > + {{ $options.i18n.I18N_ALERT_SETTINGS_FORM }} + </gl-button> + </template> + ``` + + When possible, you should opt for this pattern, as this allows you to import these strings directly into your component specs for re-use during testing. + +- Internal component `$options` object `: + + ```javascript + <script> + export default { + i18n: { + buttonLabel: s__('Plan|Button Label') + } + }, + </script> + + <template> + <gl-button :aria-label="$options.i18n.buttonLabel"> + {{ $options.i18n.buttonLabel }} + </gl-button> + </template> + ``` + +In order to visually test the Vue translations you have to change the GitLab +localization to another language than English and you have to generate JSON files using `bin/rake gettext:po_to_json` or `bin/rake gettext:compile`. ### Dynamic translations @@ -346,9 +432,9 @@ To avoid this error, use the applicable HTML entity code (`<` or `>`) inst - In JavaScript: ```javascript - import sanitize from 'sanitize-html'; + import { sanitize } from 'dompurify'; - const i18n = { LESS_THAN_ONE_HOUR: sanitize(__('In < 1 hours'), { allowedTags: [] }) }; + const i18n = { LESS_THAN_ONE_HOUR: sanitize(__('In < 1 hour'), { ALLOWED_TAGS: [] }) }; // ... using the string element.innerHTML = i18n.LESS_THAN_ONE_HOUR; diff --git a/doc/development/i18n/proofreader.md b/doc/development/i18n/proofreader.md index 9d2997379c1..1916f96801d 100644 --- a/doc/development/i18n/proofreader.md +++ b/doc/development/i18n/proofreader.md @@ -77,7 +77,7 @@ are very appreciative of the work done by translators and proofreaders! - Mongolian - Proofreaders needed. - Norwegian Bokmal - - Proofreaders needed. + - Imre Kristoffer Eilertsen - [GitLab](https://gitlab.com/DandelionSprout), [CrowdIn](https://crowdin.com/profile/DandelionSprout) - Polish - Filip Mech - [GitLab](https://gitlab.com/mehenz), [CrowdIn](https://crowdin.com/profile/mehenz) - Maksymilian Roman - [GitLab](https://gitlab.com/villaincandle), [CrowdIn](https://crowdin.com/profile/villaincandle) diff --git a/doc/development/import_export.md b/doc/development/import_export.md index 556fa6c7db4..8d2be1baf24 100644 --- a/doc/development/import_export.md +++ b/doc/development/import_export.md @@ -244,7 +244,7 @@ project_tree: - :push_event_payload - issues: - events: - - ... + # ... ``` Only include the following attributes for the models specified: @@ -254,8 +254,7 @@ included_attributes: user: - :id - :email - ... - + # ... ``` Do not include the following attributes for the models specified: diff --git a/doc/development/import_project.md b/doc/development/import_project.md index 1fa6ea5d405..9e2f5af6738 100644 --- a/doc/development/import_project.md +++ b/doc/development/import_project.md @@ -96,6 +96,13 @@ If you want to import it to a new group or subgroup then create it first. The specified project export file in `archive_path` is missing. +##### `Exception: Permission denied @ rb_sysopen - (filename)` + +The specified project export file can not be accessed by the `git` user. + +Setting the file owner to `git:git`, changing the file permissions to `0400`, and moving it to a +public folder (for example `/tmp/`) fixes the issue. + ##### `Name can contain only letters, digits, emojis ...` ```plaintext diff --git a/doc/development/integrations/secure.md b/doc/development/integrations/secure.md index 21076fa681f..dcfd0f40bf0 100644 --- a/doc/development/integrations/secure.md +++ b/doc/development/integrations/secure.md @@ -265,10 +265,16 @@ This documentation gives an overview of the report JSON format, as well as recommendations and examples to help integrators set its fields. The format is extensively described in the documentation of [SAST](../../user/application_security/sast/index.md#reports-json-format), +[DAST](../../user/application_security/dast/#reports), [Dependency Scanning](../../user/application_security/dependency_scanning/index.md#reports-json-format), and [Container Scanning](../../user/application_security/container_scanning/index.md#reports-json-format). -The DAST variant of the report JSON format is not documented at the moment. +You can find the schemas for these scanners here: + +- [SAST](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/dist/sast-report-format.json) +- [DAST](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/dist/dast-report-format.json) +- [Dependency Scanning](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/dist/dependency-scanning-report-format.json) +- [Container Scanning](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/dist/container-scanning-report-format.json) ### Version diff --git a/doc/development/integrations/secure_partner_integration.md b/doc/development/integrations/secure_partner_integration.md index 19a497641f9..830cb84e257 100644 --- a/doc/development/integrations/secure_partner_integration.md +++ b/doc/development/integrations/secure_partner_integration.md @@ -37,14 +37,14 @@ best place to integrate your own product and its results into GitLab. implications for app security, corporate policy, or compliance. When complete, the job reports back on its status and creates a [job artifact](../../user/project/pipelines/job_artifacts.md) as a result. -- The [Merge Request Security Widget](../../user/project/merge_requests/testing_and_reports_in_merge_requests.md#security-reports-ultimate) +- The [Merge Request Security Widget](../../user/project/merge_requests/testing_and_reports_in_merge_requests.md#security-reports) displays the results of the pipeline's security checks and the developer can review them. The developer can review both a summary and a detailed version of the results. - If certain policies (such as [merge request approvals](../../user/project/merge_requests/merge_request_approvals.md)) are in place for a project, developers must resolve specific findings or get an approval from a specific list of people. -- The [security dashboard](../../user/application_security/security_dashboard/index.md#gitlab-security-dashboard-ultimate) +- The [security dashboard](../../user/application_security/security_dashboard/index.md#gitlab-security-dashboard) also shows results which can developers can use to quickly see all the vulnerabilities that need to be addressed in the code. - When the developer reads the details about a vulnerability, they are @@ -88,7 +88,7 @@ and complete an integration with the Secure stage. - If you need a new kind of scan or report, [create an issue](https://gitlab.com/gitlab-org/gitlab/-/issues/new#) and add the label `devops::secure`. - Once the job is completed, the data can be seen: - - In the [Merge Request Security Report](../../user/project/merge_requests/testing_and_reports_in_merge_requests.md#security-reports-ultimate) ([MR Security Report data flow](https://gitlab.com/snippets/1910005#merge-request-view)). + - In the [Merge Request Security Report](../../user/project/merge_requests/testing_and_reports_in_merge_requests.md#security-reports) ([MR Security Report data flow](https://gitlab.com/snippets/1910005#merge-request-view)). - While [browsing a Job Artifact](../../user/project/pipelines/job_artifacts.md). - In the [Security Dashboard](../../user/application_security/security_dashboard/index.md) ([Dashboard data flow](https://gitlab.com/snippets/1910005#project-and-group-dashboards)). 1. Optional: Provide a way to interact with results as Vulnerabilities: diff --git a/doc/development/internal_api.md b/doc/development/internal_api.md index c51bf66be46..4b46787c2c3 100644 --- a/doc/development/internal_api.md +++ b/doc/development/internal_api.md @@ -26,8 +26,8 @@ file, and include the token Base64 encoded in a `secret_token` parameter or in the `Gitlab-Shared-Secret` header. NOTE: **Note:** -The internal API used by GitLab Pages uses a different kind of -authentication. +The internal API used by GitLab Pages, and GitLab Kubernetes Agent Server (kas) uses JSON Web Token (JWT) +authentication, which is different from GitLab Shell. ## Git Authentication @@ -370,3 +370,80 @@ Example response: "reference_counter_decreased": true } ``` + +## Kubernetes agent endpoints + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/41045) in GitLab 13.4. +> - This feature is not deployed on GitLab.com +> - It's not recommended for production use. + +The following endpoints are used by the GitLab Kubernetes Agent Server (kas) +for various purposes. + +These endpoints are all authenticated using JWT. The JWT secret is stored in a file +specified in `config/gitlab.yml`. By default, the location is in the root of the +GitLab Rails app in a file called `.gitlab_kas_secret`. + +CAUTION: **Caution:** +The Kubernetes agent is under development and is not recommended for production use. + +### Kubernetes agent information + +Called from GitLab Kubernetes Agent Server (kas) to retrieve agent +information for the given agent token. This returns the Gitaly connection +information for the agent's project in order for kas to fetch and update +the agent's configuration. + +```plaintext +GET /internal/kubernetes/agent_info +``` + +Example Request: + +```shell +curl --request GET --header "Gitlab-Kas-Api-Request: <JWT token>" --header "Authorization: Bearer <agent token>" "http://localhost:3000/api/v4/internal/kubernetes/agent_info" +``` + +### Kubernetes agent project information + +Called from GitLab Kubernetes Agent Server (kas) to retrieve project +information for the given agent token. This returns the Gitaly +connection for the requested project. GitLab kas uses this to configure +the agent to fetch Kubernetes resources from the project repository to +sync. + +Only public projects are currently supported. For private projects, the ability for the +agent to be authorized is [not yet implemented](https://gitlab.com/gitlab-org/gitlab/-/issues/220912). + +| Attribute | Type | Required | Description | +|:----------|:-------|:---------|:------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](../api/README.md#namespaced-path-encoding) | + +```plaintext +GET /internal/kubernetes/project_info +``` + +Example Request: + +```shell +curl --request GET --header "Gitlab-Kas-Api-Request: <JWT token>" --header "Authorization: Bearer <agent token>" "http://localhost:3000/api/v4/internal/kubernetes/project_info?id=7" +``` + +### Kubernetes agent usage metrics + +Called from GitLab Kubernetes Agent Server (kas) to increase the usage +metric counters. + +| Attribute | Type | Required | Description | +|:----------|:-------|:---------|:------------| +| `gitops_sync_count` | integer| yes | The number to increase the `gitops_sync_count` counter by | + +```plaintext +POST /internal/kubernetes/usage_metrics +``` + +Example Request: + +```shell +curl --request POST --header "Gitlab-Kas-Api-Request: <JWT token>" --header "Content-Type: application/json" --data '{"gitops_sync_count":1}' "http://localhost:3000/api/v4/internal/kubernetes/usage_metrics" +``` diff --git a/doc/development/licensed_feature_availability.md b/doc/development/licensed_feature_availability.md index cf479544eea..4350c7fb4d4 100644 --- a/doc/development/licensed_feature_availability.md +++ b/doc/development/licensed_feature_availability.md @@ -22,7 +22,7 @@ project.feature_available?(:feature_symbol) ## Restricting global features (instance) -However, for features such as [Geo](../administration/geo/replication/index.md) and +However, for features such as [Geo](../administration/geo/index.md) and [Load balancing](../administration/database_load_balancing.md), which cannot be restricted to only a subset of projects or namespaces, the check will be made directly in the instance license. @@ -35,7 +35,3 @@ the instance license. ```ruby License.feature_available?(:feature_symbol) ``` - -## Enabling promo features on GitLab.com - -A paid feature can be made available to everyone on GitLab.com by enabling the feature flag `"promo_#{feature}"`. diff --git a/doc/development/merge_request_performance_guidelines.md b/doc/development/merge_request_performance_guidelines.md index b3829a82d59..2f084937cc9 100644 --- a/doc/development/merge_request_performance_guidelines.md +++ b/doc/development/merge_request_performance_guidelines.md @@ -211,7 +211,7 @@ For keeping transaction as minimal as possible, please consider using `AfterComm module or `after_commit` AR hook. Here is [an example](https://gitlab.com/gitlab-org/gitlab/-/issues/36154#note_247228859) -that one request to Gitaly instance during transaction triggered a P::1 issue. +that one request to Gitaly instance during transaction triggered a ~"priority::1" issue. ## Eager Loading diff --git a/doc/development/migration_style_guide.md b/doc/development/migration_style_guide.md index ebaceac6d17..207dd02d258 100644 --- a/doc/development/migration_style_guide.md +++ b/doc/development/migration_style_guide.md @@ -39,7 +39,7 @@ Changes to the schema should be committed to `db/structure.sql`. This file is automatically generated by Rails, so you normally should not edit this file by hand. If your migration is adding a column to a table, that column will be added at the bottom. Please do not reorder -columns manually for existing tables as this will cause confusing to +columns manually for existing tables as this will cause confusion to other people using `db/structure.sql` generated by Rails. When your local database in your GDK is diverging from the schema from @@ -260,7 +260,9 @@ def up end def down - drop_table :issues + with_lock_retries do + drop_table :issues + end end ``` @@ -298,7 +300,7 @@ include Gitlab::Database::MigrationHelpers def up with_lock_retries do - add_foreign_key :imports, :projects, column: :project_id, on_delete: :cascade + add_foreign_key :imports, :projects, column: :project_id, on_delete: :cascade # rubocop:disable Migration/AddConcurrentForeignKey end end @@ -316,7 +318,7 @@ include Gitlab::Database::MigrationHelpers def up with_lock_retries do - add_foreign_key :imports, :users, column: :user_id, on_delete: :cascade + add_foreign_key :imports, :users, column: :user_id, on_delete: :cascade # rubocop:disable Migration/AddConcurrentForeignKey end end @@ -367,6 +369,7 @@ migration involves one of the high-traffic tables: - `users` - `projects` - `namespaces` +- `gitlab_subscriptions` - `issues` - `merge_requests` - `ci_pipelines` @@ -462,13 +465,17 @@ class MyMigration < ActiveRecord::Migration[6.0] disable_ddl_transaction! def up - remove_concurrent_index :table_name, :column_name + remove_concurrent_index :table_name, :column_name, name: :index_name end end ``` Note that it is not necessary to check if the index exists prior to -removing it. +removing it, however it is required to specify the name of the +index that is being removed. This can be done either by passing the name +as an option to the appropriate form of `remove_index` or `remove_concurrent_index`, +or more simply by using the `remove_concurrent_index_by_name` method. Explicitly +specifying the name is important to ensure the correct index is removed. For a small table (such as an empty one or one with less than `1,000` records), it is recommended to use `remove_index` in a single-transaction migration, @@ -509,11 +516,16 @@ class MyMigration < ActiveRecord::Migration[6.0] end def down - remove_concurrent_index :table, :column + remove_concurrent_index :table, :column, name: index_name end end ``` +You must explicitly name indexes that are created with more complex +definitions beyond table name, column name(s) and uniqueness constraint. +Consult the [Adding Database Indexes](adding_database_indexes.md#requirements-for-naming-indexes) +guide for more details. + If you need to add a unique index, please keep in mind there is the possibility of existing duplicates being present in the database. This means that should always _first_ add a migration that removes any duplicates, before adding the @@ -523,6 +535,42 @@ For a small table (such as an empty one or one with less than `1,000` records), it is recommended to use `add_index` in a single-transaction migration, combining it with other operations that don't require `disable_ddl_transaction!`. +## Testing for existence of indexes + +If a migration requires conditional logic based on the absence or +presence of an index, you must test for existence of that index using +its name. This helps avoids problems with how Rails compares index definitions, +which can lead to unexpected results. For more details, review the +[Adding Database Indexes](adding_database_indexes.md#why-explicit-names-are-required) +guide. + +The easiest way to test for existence of an index by name is to use the +`index_name_exists?` method, but the `index_exists?` method can also +be used with a name option. For example: + +```ruby +class MyMigration < ActiveRecord::Migration[6.0] + include Gitlab::Database::MigrationHelpers + + INDEX_NAME = 'index_name' + + def up + # an index must be conditionally created due to schema inconsistency + unless index_exists?(:table_name, :column_name, name: INDEX_NAME) + add_index :table_name, :column_name, name: INDEX_NAME + end + end + + def down + # no op + end +end +``` + +Keep in mind that concurrent index helpers like `add_concurrent_index`, +`remove_concurrent_index`, and `remove_concurrent_index_by_name` already +perform existence checks internally. + ## Adding foreign-key constraints When adding a foreign-key constraint to either an existing or a new column also diff --git a/doc/development/packages.md b/doc/development/packages.md index e21eff543b4..55e22d4bb5f 100644 --- a/doc/development/packages.md +++ b/doc/development/packages.md @@ -67,7 +67,7 @@ The current state of existing package registries availability is: | Repository Type | Project Level | Group Level | Instance Level | |-----------------|---------------|-------------|----------------| | Maven | Yes | Yes | Yes | -| Conan | No - [open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/11679) | No - [open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/11679) | Yes | +| Conan | Yes | No - [open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/11679) | Yes | | NPM | No - [open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/36853) | Yes | No - [open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/36853) | | NuGet | Yes | No - [open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/36423) | No | | PyPI | Yes | No | No | @@ -87,7 +87,7 @@ Composer package naming scope is Instance Level. To avoid name conflict for instance-level endpoints you will need to define a package naming convention that gives a way to identify the project that the package belongs to. This generally involves using the project ID or full project path in the package name. See -[Conan's naming convention](../user/packages/conan_repository/index.md#package-recipe-naming-convention) as an example. +[Conan's naming convention](../user/packages/conan_repository/index.md#package-recipe-naming-convention-for-instance-level-remote) as an example. For group and project-level endpoints, naming can be less constrained and it will be up to the group and project members to be certain that there is no conflict between two package names. However, the system should prevent diff --git a/doc/development/pipelines.md b/doc/development/pipelines.md index aef14535a96..7756ef376fc 100644 --- a/doc/development/pipelines.md +++ b/doc/development/pipelines.md @@ -56,7 +56,7 @@ graph LR subgraph "No needed jobs"; 1-1["danger-review (3.5 minutes)"]; click 1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8100542&udv=0" - 1-50["docs lint (6.75 minutes)"]; + 1-50["docs lint (9 minutes)"]; click 1-50 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356757&udv=0" end ``` @@ -72,21 +72,21 @@ graph RL; subgraph "No needed jobs"; 1-1["danger-review (3.5 minutes)"]; click 1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8100542&udv=0" - 1-2["build-qa-image (3.4 minutes)"]; + 1-2["build-qa-image (2.4 minutes)"]; click 1-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914325&udv=0" - 1-3["compile-test-assets (9.06 minutes)"]; + 1-3["compile-test-assets (8.5 minutes)"]; click 1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914317&udv=0" 1-4["compile-test-assets as-if-foss (8.35 minutes)"]; click 1-4 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356616&udv=0" - 1-5["compile-production-assets (22 minutes)"]; + 1-5["compile-production-assets (19 minutes)"]; click 1-5 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914312&udv=0" - 1-6["setup-test-env (8.22 minutes)"]; + 1-6["setup-test-env (7.4 minutes)"]; click 1-6 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914315&udv=0" 1-7["review-stop-failed-deployment"]; 1-8["dependency_scanning"]; 1-9["qa:internal, qa:internal-as-if-foss"]; 1-11["qa:selectors, qa:selectors-as-if-foss"]; - 1-14["retrieve-tests-metadata (1.5 minutes)"]; + 1-14["retrieve-tests-metadata (1.9 minutes)"]; click 1-14 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356697&udv=0" 1-15["code_quality"]; 1-16["brakeman-sast"]; @@ -113,11 +113,9 @@ graph RL; 2_1-1 & 2_1-2 & 2_1-3 & 2_1-4 --> 1-6; end - 2_2-2["frontend-fixtures (17.2 minutes)"]; + 2_2-2["frontend-fixtures (16.5 minutes)"]; class 2_2-2 criticalPath; click 2_2-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7910143&udv=0" - 2_2-3["frontend-fixtures-as-if-foss (8.75 minutes)"]; - click 2_2-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7910154&udv=0" 2_2-4["memory-on-boot (7.19 minutes)"]; click 2_2-4 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356727&udv=0" 2_2-5["webpack-dev-server (6.1 minutes)"]; @@ -148,21 +146,15 @@ graph RL; 3_1-1["jest (15 minutes)"]; class 3_1-1 criticalPath; click 3_1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914204&udv=0" - 3_1-2["karma (8 minutes)"]; + 3_1-2["karma (4 minutes)"]; click 3_1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914200&udv=0" - 3_1-3["jest-as-if-foss (19.7 minutes)"]; - click 3_1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914205&udv=0" - 3_1-4["karma-as-if-foss (7.5 minutes)"]; - click 3_1-4 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914203&udv=0" subgraph "Needs `frontend-fixtures`"; 3_1-1 & 3_1-2 --> 2_2-2; - 3_1-3 & 3_1-4 --> 2_2-3; end - 3_2-1["rspec:coverage (6.5 minutes)"]; + 3_2-1["rspec:coverage (7.5 minutes)"]; subgraph "Depends on `rspec` jobs"; 3_2-1 -.->|"(don't use needs because of limitations)"| 2_5-1; - class 3_2-1 criticalPath; click 3_2-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7248745&udv=0" end @@ -185,21 +177,21 @@ graph RL; subgraph "No needed jobs"; 1-1["danger-review (3.5 minutes)"]; click 1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8100542&udv=0" - 1-2["build-qa-image (3.4 minutes)"]; + 1-2["build-qa-image (2.4 minutes)"]; click 1-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914325&udv=0" - 1-3["compile-test-assets (9.06 minutes)"]; + 1-3["compile-test-assets (8.5 minutes)"]; click 1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914317&udv=0" 1-4["compile-test-assets as-if-foss (8.35 minutes)"]; click 1-4 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356616&udv=0" - 1-5["compile-production-assets (22 minutes)"]; + 1-5["compile-production-assets (19 minutes)"]; click 1-5 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914312&udv=0" - 1-6["setup-test-env (8.22 minutes)"]; + 1-6["setup-test-env (7.4 minutes)"]; click 1-6 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914315&udv=0" 1-7["review-stop-failed-deployment"]; 1-8["dependency_scanning"]; 1-9["qa:internal, qa:internal-as-if-foss"]; 1-11["qa:selectors, qa:selectors-as-if-foss"]; - 1-14["retrieve-tests-metadata (1.5 minutes)"]; + 1-14["retrieve-tests-metadata (1.9 minutes)"]; click 1-14 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356697&udv=0" 1-15["code_quality"]; 1-16["brakeman-sast"]; @@ -227,11 +219,9 @@ graph RL; 2_1-1 & 2_1-2 & 2_1-3 & 2_1-4 --> 1-6; end - 2_2-2["frontend-fixtures (17.2 minutes)"]; + 2_2-2["frontend-fixtures (16.5 minutes)"]; class 2_2-2 criticalPath; click 2_2-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7910143&udv=0" - 2_2-3["frontend-fixtures-as-if-foss (8.75 minutes)"]; - click 2_2-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7910154&udv=0" 2_2-4["memory-on-boot (7.19 minutes)"]; click 2_2-4 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356727&udv=0" 2_2-5["webpack-dev-server (6.1 minutes)"]; @@ -270,21 +260,15 @@ graph RL; 3_1-1["jest (15 minutes)"]; class 3_1-1 criticalPath; click 3_1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914204&udv=0" - 3_1-2["karma (8 minutes)"]; + 3_1-2["karma (4 minutes)"]; click 3_1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914200&udv=0" - 3_1-3["jest-as-if-foss (19.7 minutes)"]; - click 3_1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914205&udv=0" - 3_1-4["karma-as-if-foss (7.5 minutes)"]; - click 3_1-4 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914203&udv=0" subgraph "Needs `frontend-fixtures`"; - 3_1-1 & 3_1-3 --> 2_2-2; - 3_1-2 & 3_1-4 --> 2_2-3; + 3_1-1 & 3_1-2 --> 2_2-2; end - 3_2-1["rspec:coverage (6.5 minutes)"]; + 3_2-1["rspec:coverage (7.5 minutes)"]; subgraph "Depends on `rspec` jobs"; 3_2-1 -.->|"(don't use needs because of limitations)"| 2_5-1; - class 3_2-1 criticalPath; click 3_2-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7248745&udv=0" end @@ -325,21 +309,21 @@ graph RL; subgraph "No needed jobs"; 1-1["danger-review (3.5 minutes)"]; click 1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8100542&udv=0" - 1-2["build-qa-image (3.4 minutes)"]; + 1-2["build-qa-image (2.4 minutes)"]; click 1-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914325&udv=0" - 1-3["compile-test-assets (9.06 minutes)"]; + 1-3["compile-test-assets (8.5 minutes)"]; click 1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914317&udv=0" 1-4["compile-test-assets as-if-foss (8.35 minutes)"]; click 1-4 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356616&udv=0" - 1-5["compile-production-assets (22 minutes)"]; + 1-5["compile-production-assets (19 minutes)"]; click 1-5 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914312&udv=0" - 1-6["setup-test-env (8.22 minutes)"]; + 1-6["setup-test-env (7.4 minutes)"]; click 1-6 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914315&udv=0" 1-7["review-stop-failed-deployment"]; 1-8["dependency_scanning"]; 1-9["qa:internal, qa:internal-as-if-foss"]; 1-11["qa:selectors, qa:selectors-as-if-foss"]; - 1-14["retrieve-tests-metadata (1.5 minutes)"]; + 1-14["retrieve-tests-metadata (1.9 minutes)"]; click 1-14 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356697&udv=0" 1-15["code_quality"]; 1-16["brakeman-sast"]; @@ -373,6 +357,65 @@ graph RL; end ``` +### Fail-fast pipeline in Merge Requests + +To provide faster feedback when a Merge Request breaks existing tests, we are experimenting with a +fail-fast mechanism. + +An `rspec fail-fast` job is added in parallel to all other `rspec` jobs in a Merge +Request pipeline. This job runs the tests that are directly related to the changes +in the Merge Request. + +If any of these tests fail, the `rspec fail-fast` job fails, triggering a +`fail-pipeline-early` job to run. The `fail-pipeline-early` job: + +- Cancels the currently running pipeline and all in-progress jobs. +- Sets pipeline to have status `failed`. + +For example: + +```mermaid +graph LR + subgraph "prepare stage"; + A["detect-tests"] + end + + subgraph "test stage"; + B["jest"]; + C["rspec migration"]; + D["rspec unit"]; + E["rspec integration"]; + F["rspec system"]; + G["rspec fail-fast"]; + end + + subgraph "post-test stage"; + Z["fail-pipeline-early"]; + end + + A --"artifact: list of test files"--> G + G --"on failure"--> Z +``` + +A Merge Request author may choose to opt-out of the fail fast mechanism by doing one of the following: + +- Including `[SKIP RSPEC FAIL-FAST]` in the Merge Request title. +- Starting the `dont-interrupt-me` job found in the `sync` stage of a Merge Request pipeline. + +The `rspec fail-fast` is a no-op if there are more than 10 test files related to the +Merge Request. This prevents `rspec fail-fast` duration from exceeding the average +`rspec` job duration and defeating its purpose. + +This number can be overridden by setting a CI variable named `RSPEC_FAIL_FAST_TEST_FILE_COUNT_THRESHOLD`. + +NOTE: **Note:** +This experiment is only enabled when the CI variable `RSPEC_FAIL_FAST_ENABLED=true` is set. + +#### Determining related test files in a Merge Request + +The test files related to the Merge Request are determined using the [`test_file_finder`](https://gitlab.com/gitlab-org/ci-cd/test_file_finder) gem. +We are using a custom mapping between source file to test files, maintained in the `tests.yml` file. + ### PostgreSQL versions testing #### Current versions testing @@ -417,8 +460,8 @@ of the `gitlab-org/gitlab-foss` project. These jobs are only created in the foll - Merge requests which include `RUN AS-IF-FOSS` in their title. - Merge requests that changes the CI configuration. -The `* as-if-foss` jobs have the `FOSS_ONLY='1'` variable set and gets their EE-specific -folders removed before the tests start running. +The `* as-if-foss` jobs are run in addition to the regular EE-context jobs. They have the `FOSS_ONLY='1'` variable +set and get their EE-specific folders removed before the tests start running. The intent is to ensure that a change won't introduce a failure once the `gitlab-org/gitlab` project will be synced to the `gitlab-org/gitlab-foss` project. @@ -514,8 +557,9 @@ overwrites the Git configuration with the appropriate settings to fetch from the GitLab repository. `CI_REPO_CACHE_CREDENTIALS` contains the Google Cloud service account -JSON for uploading to the `gitlab-ci-git-repo-cache` bucket. These -credentials are stored in the 1Password GitLab.com Production vault. +JSON for uploading to the `gitlab-ci-git-repo-cache` bucket. (If you’re a +GitLab Team Member, find credentials in the +[GitLab shared 1Password account](https://about.gitlab.com/handbook/security/#1password-for-teams). Note that this bucket should be located in the same continent as the runner, or [network egress charges will apply](https://cloud.google.com/storage/pricing). @@ -540,8 +584,10 @@ The current stages are: later used by the (Helm) Review App deployment (see [Review Apps](testing_guide/review_apps.md) for details). - `review`: This stage includes jobs that deploy the GitLab and Docs Review Apps. +- `dast`: This stage includes jobs that run a DAST full scan against the Review App +that is deployed in stage `review`. - `qa`: This stage includes jobs that perform QA tasks against the Review App - that is deployed in the previous stage. + that is deployed in stage `review`. - `post-qa`: This stage includes jobs that build reports or gather data from the `qa` stage's jobs (e.g. Review App performance report). - `pages`: This stage includes a job that deploys the various reports as diff --git a/doc/development/redis.md b/doc/development/redis.md index d5d42a3869e..d205082b9c6 100644 --- a/doc/development/redis.md +++ b/doc/development/redis.md @@ -1,10 +1,19 @@ # Redis guidelines -GitLab uses [Redis](https://redis.io) for three distinct purposes: +GitLab uses [Redis](https://redis.io) for the following distinct purposes: -- Caching via `Rails.cache`. +- Caching (mostly via `Rails.cache`). - As a job processing queue with [Sidekiq](sidekiq_style_guide.md). - To manage the shared application state. +- As a Pub/Sub queue backend for ActionCable. + +In most environments (including the GDK), all of these point to the same +Redis instance. + +On GitLab.com, we use [separate Redis +instances](../administration/redis/replication_and_failover.md#running-multiple-redis-clusters). +(We do not currently use [ActionCable on +GitLab.com](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/228)). Every application process is configured to use the same Redis servers, so they can be used for inter-process communication in cases where [PostgreSQL](sql.md) @@ -21,11 +30,11 @@ to key names to avoid collisions. Typically we use colon-separated elements to provide a semblance of structure at application level. An example might be `projects:1:somekey`. -Although we split our Redis usage into three separate purposes, and those may -map to separate Redis servers in a [Highly Available](../administration/high_availability/redis.md) -configuration, the default Omnibus and GDK setups share a single Redis server. -This means that keys should **always** be globally unique across the three -purposes. +Although we split our Redis usage by purpose into distinct categories, and +those may map to separate Redis servers in a Highly Available +configuration like GitLab.com, the default Omnibus and GDK setups share +a single Redis server. This means that keys should **always** be +globally unique across all categories. It is usually better to use immutable identifiers - project ID rather than full path, for instance - in Redis key names. If full path is used, the key will @@ -56,3 +65,127 @@ Currently, we validate this in the development and test environments with the [`RedisClusterValidator`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/instrumentation/redis_cluster_validator.rb), which is enabled for the `cache` and `shared_state` [Redis instances](https://docs.gitlab.com/omnibus/settings/redis.html#running-with-multiple-redis-instances).. + +## Redis in structured logging + +Our [structured logging](logging.md#use-structured-json-logging) for web +requests and Sidekiq jobs contains fields for the duration, call count, +bytes written, and bytes read per Redis instance, along with a total for +all Redis instances. For a particular request, this might look like: + +| Field | Value | +| --- | --- | +| `json.queue_duration_s` | 0.01 | +| `json.redis_cache_calls` | 1 | +| `json.redis_cache_duration_s` | 0 | +| `json.redis_cache_read_bytes` | 109 | +| `json.redis_cache_write_bytes` | 49 | +| `json.redis_calls` | 2 | +| `json.redis_duration_s` | 0.001 | +| `json.redis_read_bytes` | 111 | +| `json.redis_shared_state_calls` | 1 | +| `json.redis_shared_state_duration_s` | 0 | +| `json.redis_shared_state_read_bytes` | 2 | +| `json.redis_shared_state_write_bytes` | 206 | +| `json.redis_write_bytes` | 255 | + +As all of these fields are indexed, it is then straightforward to +investigate Redis usage in production. For instance, to find the +requests that read the most data from the cache, we can just sort by +`redis_cache_read_bytes` in descending order. + +### The slow log + +On GitLab.com, entries from the [Redis +slow log](https://redis.io/commands/slowlog) are available in the +`pubsub-redis-inf-gprd*` index with the [`redis.slowlog` +tag](https://log.gprd.gitlab.net/app/kibana#/discover?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-1d,to:now))&_a=(columns:!(json.type,json.command,json.exec_time),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:AWSQX_Vf93rHTYrsexmk,key:json.tag,negate:!f,params:(query:redis.slowlog),type:phrase),query:(match:(json.tag:(query:redis.slowlog,type:phrase))))),index:AWSQX_Vf93rHTYrsexmk)). +This shows commands that have taken a long time and may be a performance +concern. + +The +[fluent-plugin-redis-slowlog](https://gitlab.com/gitlab-org/fluent-plugin-redis-slowlog) +project is responsible for taking the slowlog entries from Redis and +passing to fluentd (and ultimately Elasticsearch). + +## Analyzing the entire keyspace + +The [Redis Keyspace +Analyzer](https://gitlab.com/gitlab-com/gl-infra/redis-keyspace-analyzer) +project contains tools for dumping the full key list and memory usage of a Redis +instance, and then analyzing those lists while elimating potentially sensitive +data from the results. It can be used to find the most frequent key patterns, or +those that use the most memory. + +Currently this is not run automatically for the GitLab.com Redis instances, but +is run manually on an as-needed basis. + +## Utility classes + +We have some extra classes to help with specific use cases. These are +mostly for fine-grained control of Redis usage, so they wouldn't be used +in combination with the `Rails.cache` wrapper: we'd either use +`Rails.cache` or these classes and literal Redis commands. + +`Rails.cache` or these classes and literal Redis commands. We prefer +using `Rails.cache` so we can reap the benefits of future optimizations +done to Rails. It is worth noting that Ruby objects are +[marshalled](https://github.com/rails/rails/blob/v6.0.3.1/activesupport/lib/active_support/cache/redis_cache_store.rb#L447) +when written to Redis, so we need to pay attention to not to store huge +objects, or untrusted user input. + +Typically we would only use these classes when at least one of the +following is true: + +1. We want to manipulate data on a non-cache Redis instance. +1. `Rails.cache` does not support the operations we want to perform. + +### `Gitlab::Redis::{Cache,SharedState,Queues}` + +These classes wrap the Redis instances (using +[`Gitlab::Redis::Wrapper`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/redis/wrapper.rb)) +to make it convenient to work with them directly. The typical use is to +call `.with` on the class, which takes a block that yields the Redis +connection. For example: + +```ruby +# Get the value of `key` from the shared state (persistent) Redis +Gitlab::Redis::SharedState.with { |redis| redis.get(key) } + +# Check if `value` is a member of the set `key` +Gitlab::Redis::Cache.with { |redis| redis.sismember(key, value) } +``` + +### `Gitlab::Redis::Boolean` + +In Redis, every value is a string. +[`Gitlab::Redis::Boolean`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/redis/boolean.rb) +makes sure that booleans are encoded and decoded consistently. + +### `Gitlab::Redis::HLL` + +The Redis [`PFCOUNT`](https://redis.io/commands/pfcount), +[`PFADD`](https://redis.io/commands/pfadd), and +[`PFMERGE`](https://redis.io/commands/pfmergge) commands operate on +HyperLogLogs, a data structure that allows estimating the number of unique +elements with low memory usage. (In addition to the `PFCOUNT` documentation, +Thoughtbot's article on [HyperLogLogs in +Redis](https://thoughtbot.com/blog/hyperloglogs-in-redis) provides a good +background here.) + +[`Gitlab::Redis::HLL`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/redis/hll.rb) +provides a convenient interface for adding and counting values in HyperLogLogs. + +### `Gitlab::SetCache` + +For cases where we need to efficiently check the whether an item is in a group +of items, we can use a Redis set. +[`Gitlab::SetCache`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/set_cache.rb) +provides an `#include?` method that will use the +[`SISMEMBER`](https://redis.io/commands/sismember) command, as well as `#read` +to fetch all entries in the set. + +This is used by the +[`RepositorySetCache`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/repository_set_cache.rb) +to provide a convenient way to use sets to cache repository data like branch +names. diff --git a/doc/development/repository_mirroring.md b/doc/development/repository_mirroring.md index 1d4dbe88399..fe6db987471 100644 --- a/doc/development/repository_mirroring.md +++ b/doc/development/repository_mirroring.md @@ -3,7 +3,7 @@ ## Deep Dive In December 2018, Tiago Botelho hosted a Deep Dive (GitLab team members only: `https://gitlab.com/gitlab-org/create-stage/issues/1`) -on GitLab's [Pull Repository Mirroring functionality](../user/project/repository/repository_mirroring.md#pulling-from-a-remote-repository-starter) +on GitLab's [Pull Repository Mirroring functionality](../user/project/repository/repository_mirroring.md#pulling-from-a-remote-repository) to share his domain specific knowledge with anyone who may work in this part of the code base in the future. You can find the [recording on YouTube](https://www.youtube.com/watch?v=sSZq0fpdY-Y), and the slides in [PDF](https://gitlab.com/gitlab-org/create-stage/uploads/8693404888a941fd851f8a8ecdec9675/Gitlab_Create_-_Pull_Mirroring_Deep_Dive.pdf). diff --git a/doc/development/secure_coding_guidelines.md b/doc/development/secure_coding_guidelines.md index 65953620ce6..1961d1dcc34 100644 --- a/doc/development/secure_coding_guidelines.md +++ b/doc/development/secure_coding_guidelines.md @@ -84,7 +84,7 @@ This Ruby Regex specialty can have security impact, as often regular expressions GitLab specific examples can be found [here](https://gitlab.com/gitlab-org/gitlab/-/issues/36029#note_251262187) and [there](https://gitlab.com/gitlab-org/gitlab/-/issues/33569). -Another example would be this fictional Ruby On Rails controller: +Another example would be this fictional Ruby on Rails controller: ```ruby class PingController < ApplicationController @@ -127,9 +127,9 @@ class Email < ApplicationRecord DOMAIN_MATCH = Regexp.new('([a-zA-Z0-9]+)+\.com') validates :domain_matches - + private - + def domain_matches errors.add(:email, 'does not match') if email =~ DOMAIN_MATCH end @@ -184,7 +184,7 @@ have been reported to GitLab include: - Reading internal services, including cloud service metadata. - The latter can be a serious problem, because an attacker can obtain keys that allow control of the victim's cloud infrastructure. (This is also a good reason to give only necessary privileges to the token.). [More details](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/51490). -- When combined with CRLF vulnerability, remote code execution. [More details](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/41293) +- When combined with CRLF vulnerability, remote code execution. [More details](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/41293). ### When to Consider @@ -213,7 +213,7 @@ the mitigations for a new feature. #### Feature-specific Mitigations -For situtions in which an allowlist or GitLab:HTTP cannot be used, it will be necessary to implement mitigations directly in the feature. It is best to validate the destination IP addresses themselves, not just domain names, as DNS can be controlled by the attacker. Below are a list of mitigations that should be implemented. +For situations in which an allowlist or GitLab:HTTP cannot be used, it will be necessary to implement mitigations directly in the feature. It is best to validate the destination IP addresses themselves, not just domain names, as DNS can be controlled by the attacker. Below are a list of mitigations that should be implemented. **Important Note:** There are many tricks to bypass common SSRF validations. If feature-specific mitigations are necessary, they should be reviewed by the AppSec team, or a developer who has worked on SSRF mitigations previously. diff --git a/doc/development/shell_scripting_guide/index.md b/doc/development/shell_scripting_guide/index.md index c04a4e90e59..622c90d7a97 100644 --- a/doc/development/shell_scripting_guide/index.md +++ b/doc/development/shell_scripting_guide/index.md @@ -65,7 +65,7 @@ shell check: before_script: - shellcheck --version script: - - shellcheck scripts/**/*.sh # path to your shell scripts + - shellcheck scripts/**/*.sh # path to your shell scripts ``` TIP: **Tip:** @@ -93,7 +93,7 @@ shfmt: before_script: - shfmt -version script: - - shfmt -i 2 -ci -d scripts # path to your shell scripts + - shfmt -i 2 -ci -d scripts # path to your shell scripts ``` TIP: **Tip:** diff --git a/doc/development/sidekiq_style_guide.md b/doc/development/sidekiq_style_guide.md index c5dfc5731e6..fdea27f43ca 100644 --- a/doc/development/sidekiq_style_guide.md +++ b/doc/development/sidekiq_style_guide.md @@ -615,42 +615,51 @@ Jobs need to be backward and forward compatible between consecutive versions of the application. Adding or removing an argument may cause problems during deployment before all Rails and Sidekiq nodes have the updated code. -#### Remove an argument +#### Deprecate and remove an argument -**Do not remove arguments from the `perform` function.**. Instead, use the -following approach: +**Before you remove arguments from the `perform_async` and `perform` methods.**, deprecate them. The +following example deprecates and then removes `arg2` from the `perform_async` method: 1. Provide a default value (usually `nil`) and use a comment to mark the - argument as deprecated -1. Stop using the argument in `perform_async`. -1. Ignore the value in the worker class, but do not remove it until the next - major release. + argument as deprecated in the coming minor release. (Release M) -In the following example, if you want to remove `arg2`, first set a `nil` default value, -and then update locations where `ExampleWorker.perform_async` is called. + ```ruby + class ExampleWorker + # Keep arg2 parameter for backwards compatibility. + def perform(object_id, arg1, arg2 = nil) + # ... + end + end + ``` -```ruby -class ExampleWorker - def perform(object_id, arg1, arg2 = nil) - # ... - end -end -``` +1. One minor release later, stop using the argument in `perform_async`. (Release M+1) + + ```ruby + ExampleWorker.perform_async(object_id, arg1) + ``` + +1. At the next major release, remove the value from the worker class. (Next major release) + + ```ruby + class ExampleWorker + def perform(object_id, arg1) + # ... + end + end + ``` #### Add an argument There are two options for safely adding new arguments to Sidekiq workers: -1. Set up a [multi-step deployment](#multi-step-deployment) in which the new argument is first added to the worker +1. Set up a [multi-step deployment](#multi-step-deployment) in which the new argument is first added to the worker. 1. Use a [parameter hash](#parameter-hash) for additional arguments. This is perhaps the most flexible option. ##### Multi-step deployment -This approach requires multiple merge requests and for the first merge request -to be merged and deployed before additional changes are merged. +This approach requires multiple releases. -1. In an initial merge request, add the argument to the worker with a default - value: +1. Add the argument to the worker with a default value (Release M). ```ruby class ExampleWorker @@ -660,16 +669,28 @@ to be merged and deployed before additional changes are merged. end ``` -1. Merge and deploy the worker with the new argument. -1. In a further merge request, update `ExampleWorker.perform_async` calls to - use the new argument. +1. Add the new argument to all the invocations of the worker (Release M+1). + + ```ruby + ExampleWorker.perform_async(object_id, new_arg) + ``` + +1. Remove the default value (Release M+2). + + ```ruby + class ExampleWorker + def perform(object_id, new_arg) + # ... + end + end + ``` ##### Parameter hash -This approach will not require multiple deployments if an existing worker already +This approach will not require multiple releases if an existing worker already utilizes a parameter hash. -1. Use a parameter hash in the worker to allow for future flexibility: +1. Use a parameter hash in the worker to allow future flexibility. ```ruby class ExampleWorker diff --git a/doc/development/telemetry/snowplow.md b/doc/development/telemetry/snowplow.md index 547ba36464b..f427d7d1488 100644 --- a/doc/development/telemetry/snowplow.md +++ b/doc/development/telemetry/snowplow.md @@ -40,7 +40,7 @@ Snowplow is an enterprise-grade marketing and product analytics platform which h We have many definitions of Snowplow's schema. We have an active issue to [standardize this schema](https://gitlab.com/gitlab-org/gitlab/-/issues/207930) including the following definitions: - Frontend and backend taxonomy as listed below -- [Feature instrumentation taxonomy](https://about.gitlab.com/handbook/product/product-processes/#taxonomy) +- [Feature instrumentation taxonomy](https://about.gitlab.com/handbook/business-ops/data-team/programs/data-for-product-managers/#sts=Taxonomy) - [Self describing events](https://github.com/snowplow/snowplow/wiki/Custom-events#self-describing-events) - [Iglu schema](https://gitlab.com/gitlab-org/iglu/) - [Snowplow authored events](https://github.com/snowplow/snowplow/wiki/Snowplow-authored-events) @@ -98,13 +98,13 @@ sequenceDiagram ## Implementing Snowplow JS (Frontend) tracking -GitLab provides `Tracking`, an interface that wraps the [Snowplow JavaScript Tracker](https://github.com/snowplow/snowplow/wiki/javascript-tracker) for tracking custom events. There are a few ways to utilize tracking, but each generally requires at minimum, a `category` and an `action`. Additional data can be provided that adheres to our [Feature instrumentation taxonomy](https://about.gitlab.com/handbook/product/product-processes/#taxonomy). +GitLab provides `Tracking`, an interface that wraps the [Snowplow JavaScript Tracker](https://github.com/snowplow/snowplow/wiki/javascript-tracker) for tracking custom events. There are a few ways to utilize tracking, but each generally requires at minimum, a `category` and an `action`. Additional data can be provided that adheres to our [Feature instrumentation taxonomy](https://about.gitlab.com/handbook/business-ops/data-team/programs/data-for-product-managers/#sts=Taxonomy). | field | type | default value | description | |:-----------|:-------|:---------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `category` | string | document.body.dataset.page | Page or subsection of a page that events are being captured within. | | `action` | string | 'generic' | Action the user is taking. Clicks should be `click` and activations should be `activate`, so for example, focusing a form field would be `activate_form_input`, and clicking a button would be `click_button`. | -| `data` | object | {} | Additional data such as `label`, `property`, `value`, and `context` as described [in our Feature Instrumentation taxonomy](https://about.gitlab.com/handbook/product/product-processes/#taxonomy). | +| `data` | object | {} | Additional data such as `label`, `property`, `value`, and `context` as described [in our Feature Instrumentation taxonomy](https://about.gitlab.com/handbook/business-ops/data-team/programs/data-for-product-managers/#sts=Taxonomy). | ### Tracking in HAML (or Vue Templates) @@ -131,10 +131,10 @@ Below is a list of supported `data-track-*` attributes: | attribute | required | description | |:----------------------|:---------|:------------| | `data-track-event` | true | Action the user is taking. Clicks must be prepended with `click` and activations must be prepended with `activate`. For example, focusing a form field would be `activate_form_input` and clicking a button would be `click_button`. | -| `data-track-label` | false | The `label` as described [in our Feature Instrumentation taxonomy](https://about.gitlab.com/handbook/product/product-processes/#taxonomy). | -| `data-track-property` | false | The `property` as described [in our Feature Instrumentation taxonomy](https://about.gitlab.com/handbook/product/product-processes/#taxonomy). | -| `data-track-value` | false | The `value` as described [in our Feature Instrumentation taxonomy](https://about.gitlab.com/handbook/product/product-processes/#taxonomy). If omitted, this is the element's `value` property or an empty string. For checkboxes, the default value is the element's checked attribute or `false` when unchecked. | -| `data-track-context` | false | The `context` as described [in our Feature Instrumentation taxonomy](https://about.gitlab.com/handbook/product/product-processes/#taxonomy). | +| `data-track-label` | false | The `label` as described [in our Feature Instrumentation taxonomy](https://about.gitlab.com/handbook/business-ops/data-team/programs/data-for-product-managers/#sts=Taxonomy). | +| `data-track-property` | false | The `property` as described [in our Feature Instrumentation taxonomy](https://about.gitlab.com/handbook/business-ops/data-team/programs/data-for-product-managers/#sts=Taxonomy). | +| `data-track-value` | false | The `value` as described [in our Feature Instrumentation taxonomy](https://about.gitlab.com/handbook/business-ops/data-team/programs/data-for-product-managers/#sts=Taxonomy). If omitted, this is the element's `value` property or an empty string. For checkboxes, the default value is the element's checked attribute or `false` when unchecked. | +| `data-track-context` | false | The `context` as described [in our Feature Instrumentation taxonomy](https://about.gitlab.com/handbook/business-ops/data-team/programs/data-for-product-managers/#sts=Taxonomy). | ### Tracking within Vue components @@ -278,7 +278,7 @@ Custom event tracking and instrumentation can be added by directly calling the ` |:-----------|:-------|:--------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `category` | string | 'application' | Area or aspect of the application. This could be `HealthCheckController` or `Lfs::FileTransformer` for instance. | | `action` | string | 'generic' | The action being taken, which can be anything from a controller action like `create` to something like an Active Record callback. | -| `data` | object | {} | Additional data such as `label`, `property`, `value`, and `context` as described in [Instrumentation at GitLab](https://about.gitlab.com/handbook/product/product-processes/#taxonomy). These are set as empty strings if you don't provide them. | +| `data` | object | {} | Additional data such as `label`, `property`, `value`, and `context` as described in [Instrumentation at GitLab](https://about.gitlab.com/handbook/business-ops/data-team/programs/data-for-product-managers/#sts=Taxonomy). These are set as empty strings if you don't provide them. | Tracking can be viewed as either tracking user behavior, or can be utilized for instrumentation to monitor and visualize performance over time in an area or aspect of code. @@ -316,6 +316,11 @@ There are several tools for developing and testing Snowplow Event **{check-circle}** Available, **{status_preparing}** In progress, **{dotted-circle}** Not Planned +### Preparing your MR for Review + +1. For frontend events, in the MR description section, add a screenshot of the event's relevant section using the [Snowplow Analytics Debugger](https://chrome.google.com/webstore/detail/snowplow-analytics-debugg/jbnlcgeengmijcghameodeaenefieedm) Chrome browser extension. +1. For backend events, please use Snowplow Micro and add the output of the Snowplow Micro good events `GET http://localhost:9090/micro/good`. + ### Snowplow Analytics Debugger Chrome Extension Snowplow Analytics Debugger is a browser extension for testing frontend events. This works on production, staging and local development environments. @@ -393,7 +398,7 @@ Snowplow Micro is a Docker-based solution for testing frontend and backend event 1. Send a test Snowplow event from the Rails console: ```ruby - Gitlab::Tracking.self_describing_event('iglu:com.gitlab/pageview_context/jsonschema/1-0-0', { page_type: ‘MY_TYPE' }, context: nil ) + Gitlab::Tracking.self_describing_event('iglu:com.gitlab/pageview_context/jsonschema/1-0-0', { page_type: 'MY_TYPE' }, context: nil ) ``` ### Snowplow Mini diff --git a/doc/development/telemetry/usage_ping.md b/doc/development/telemetry/usage_ping.md index ea5eb6c389f..ff4e7e0797b 100644 --- a/doc/development/telemetry/usage_ping.md +++ b/doc/development/telemetry/usage_ping.md @@ -37,7 +37,7 @@ More useful links: - The main purpose of Usage Ping is to build a better GitLab. Data about how GitLab is used is collected to better understand feature/stage adoption and usage, which helps us understand how GitLab is adding value and helps our team better understand the reasons why people use GitLab and with this knowledge we're able to make better product decisions. - As a benefit of having the usage ping active, GitLab lets you analyze the users’ activities over time of your GitLab installation. -- As a benefit of having the usage ping active, GitLab provides you with The DevOps Score,which gives you an overview of your entire instance’s adoption of Concurrent DevOps from planning to monitoring. +- As a benefit of having the usage ping active, GitLab provides you with The DevOps Report,which gives you an overview of your entire instance’s adoption of Concurrent DevOps from planning to monitoring. - You will get better, more proactive support. (assuming that our TAMs and support organization used the data to deliver more value) - You will get insight and advice into how to get the most value out of your investment in GitLab. Wouldn't you want to know that a number of features or values are not being adopted in your organization? - You get a report that illustrates how you compare against other similar organizations (anonymized), with specific advice and recommendations on how to improve your DevOps processes. @@ -108,7 +108,7 @@ sequenceDiagram S3 Bucket->>Snowflake DW: Import data Snowflake DW->>Snowflake DW: Transform data using dbt Snowflake DW->>Sisense Dashboards: Data available for querying - Versions Application->>GitLab Instance: DevOps Score (Conversational Development Index) + Versions Application->>GitLab Instance: DevOps Report (Conversational Development Index) ``` ## How Usage Ping works @@ -222,38 +222,239 @@ Examples of implementation: #### Redis HLL Counters -With `Gitlab::Redis::HLL` we have available data structures used to count unique values. +With `Gitlab::UsageDataCounters::HLLRedisCounter` we have available data structures used to count unique values. Implemented using Redis methods [PFADD](https://redis.io/commands/pfadd) and [PFCOUNT](https://redis.io/commands/pfcount). -Recommendations: +##### Adding new events -- Key should expire in 29 days. -- If possible, data granularity should be a week. For example a key could be composed from the metric's name and week of the year, `2020-33-{metric_name}`. -- Use a [feature flag](../../operations/feature_flags.md) in order to have a control over the impact when adding new metrics. -- If possible, data granularity should be week, for example a key could be composed from metric name and week of the year, 2020-33-{metric_name} -- Use a [feature flag](../../operations/feature_flags.md) in order to have a control over the impact when adding new metrics +1. Define events in [`known_events.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events.yml). -Examples of implementation: + Example event: -- [`Gitlab::UsageDataCounters::TrackUniqueActions`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/track_unique_actions.rb) -- [`Gitlab::Analytics::UniqueVisits`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/analytics/unique_visits.rb) + ```yaml + - name: i_compliance_credential_inventory + category: compliance + redis_slot: compliance + expiry: 42 # 6 weeks + aggregation: weekly + ``` -Example of usage: + Keys: + + - `name`: unique event name. + + Name format `<prefix>_<redis_slot>_name`. + + Use one of the following prefixes for the event's name: + + - `g_` for group, as an event which is tracked for group. + - `p_` for project, as an event which is tracked for project. + - `i_` for instance, as an event which is tracked for instance. + - `a_` for events encompassing all `g_`, `p_`, `i_`. + - `o_` for other. + + Consider including in the event's name the Redis slot in order to be able to count totals for a specific category. + + Example names: `i_compliance_credential_inventory`, `g_analytics_contribution`. + + - `category`: event category. Used for getting total counts for events in a category, for easier + access to a group of events. + - `redis_slot`: optional Redis slot; default value: event name. Used if needed to calculate totals + for a group of metrics. Ensure keys are in the same slot. For example: + `i_compliance_credential_inventory` with `redis_slot: 'compliance'` will build Redis key + `i_{compliance}_credential_inventory-2020-34`. If `redis_slot` is not defined the Redis key will + be `{i_compliance_credential_inventory}-2020-34`. + - `expiry`: expiry time in days. Default: 29 days for daily aggregation and 6 weeks for weekly + aggregation. + - `aggregation`: aggregation `:daily` or `:weekly`. The argument defines how we build the Redis + keys for data storage. For `daily` we keep a key for metric per day of the year, for `weekly` we + keep a key for metric per week of the year. + +1. Track event in controller using `RedisTracking` module with `track_redis_hll_event(*controller_actions, name:, feature:, feature_default_enabled: false)`. + + Arguments: + + - `controller_actions`: controller actions we want to track. + - `name`: event name. + - `feature`: feature name, all metrics we track should be under feature flag. + - `feature_default_enabled`: feature flag is disabled by default, set to `true` for it to be enabled by default. + + Example usage: + + ```ruby + # controller + class ProjectsController < Projects::ApplicationController + include RedisTracking + + skip_before_action :authenticate_user!, only: :show + track_redis_hll_event :index, :show, name: 'i_analytics_dev_ops_score', feature: :g_compliance_dashboard_feature, feature_default_enabled: true + + def index + render html: 'index' + end + + def new + render html: 'new' + end + + def show + render html: 'show' + end + end + ``` + +1. Track event in API using `increment_unique_values(event_name, values)` helper method. + + In order to be able to track the event, Usage Ping must be enabled and the event feature `usage_data_<event_name>` must be enabled. + + Arguments: + + - `event_name`: event name. + - `values`: values counted, one value or array of values. + + Example usage: + + ```ruby + get ':id/registry/repositories' do + repositories = ContainerRepositoriesFinder.new( + user: current_user, subject: user_group + ).execute + + increment_unique_values('i_list_repositories', current_user.id) + + present paginate(repositories), with: Entities::ContainerRegistry::Repository, tags: params[:tags], tags_count: params[:tags_count] + end + ``` + +1. Track event using `track_usage_event(event_name, values) in services and graphql + + Increment unique values count using Redis HLL, for given event name. + + Example: + + [Track usage event for incident created in service](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/issues/update_service.rb) + + [Track usage event for incident created in graphql](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/graphql/mutations/alert_management/update_alert_status.rb) + + ```ruby + track_usage_event(:incident_management_incident_created, current_user.id) + ``` + +1. Track event using `UsageData` API + + Increment unique users count using Redis HLL, for given event name. + + Tracking events using the `UsageData` API requires the `usage_data_api` feature flag to be enabled, which is disabled by default. + + API requests are protected by checking for a valid CSRF token. + + In order to be able to increment the values the related feature `usage_data<event_name>` should be enabled. + + ```plaintext + POST /usage_data/increment_unique_users + ``` + + | Attribute | Type | Required | Description | + | :-------- | :--- | :------- | :---------- | + | `event` | string | yes | The event name it should be tracked | + + Response + + Return 200 if tracking failed for any reason. + + - `200` if event was tracked or any errors + - `400 Bad request` if event parameter is missing + - `401 Unauthorized` if user is not authenticated + - `403 Forbidden` for invalid CSRF token provided + +1. Track event using base module `Gitlab::UsageDataCounters::HLLRedisCounter.track_event(entity_id, event_name)`. + + Arguments: + + - `entity_id`: value we count. For example: user_id, visitor_id. + - `event_name`: event name. + +1. Get event data using `Gitlab::UsageDataCounters::HLLRedisCounter.unique_events(event_names:, start_date:, end_date)`. + + Arguments: + + - `event_names`: the list of event names. + - `start_date`: start date of the period for which we want to get event data. + - `end_date`: end date of the period for which we want to get event data. + +Recommendations: + +- Key should expire in 29 days for daily and 42 days for weekly. +- If possible, data granularity should be a week. For example a key could be composed from the + metric's name and week of the year, `2020-33-{metric_name}`. +- Use a [feature flag](../../operations/feature_flags.md) to have a control over the impact when + adding new metrics. + +##### Known events in usage data payload + +All events added in [`known_events.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events.yml) are automatically added to usage data generation under the `redis_hll_counters` key. This column is stored in [version-app as a JSON](https://gitlab.com/gitlab-services/version-gitlab-com/-/blob/master/db/schema.rb#L209). +For each event we add metrics for the weekly and monthly time frames, and totals for each where applicable: + +- `#{event_name}_weekly` data for 7 days for daily [aggregation](#adding-new-events) events and data for last complete week for weekly [aggregation](#adding-new-events) events. +- `#{event_name}_monthly` data for 28 days for daily [aggregation](#adding-new-events) events and data for last 4 complete weeks for weekly [aggregation](#adding-new-events) events. +- `#{category}_total_unique_counts_weekly` total unique counts for events in same category for last 7 days or last complete week, if events are in the same Redis slot and if we have more than one metric. +- `#{event_name}_weekly` - Data for 7 days for daily [aggregation](#adding-new-events) events and data for the last complete week for weekly [aggregation](#adding-new-events) events. +- `#{event_name}_monthly` - Data for 28 days for daily [aggregation](#adding-new-events) events and data for the last 4 complete weeks for weekly [aggregation](#adding-new-events) events. +- `#{category}_total_unique_counts_weekly` - Total unique counts for events in the same category for the last 7 days or the last complete week, if events are in the same Redis slot and we have more than one metric. +- `#{event_name}_weekly`: Data for 7 days for daily [aggregation](#adding-new-events) events and data for last complete week for weekly [aggregation](#adding-new-events) events. +- `#{event_name}_monthly`: Data for 28 days for daily [aggregation](#adding-new-events) events and data for last 4 complete weeks for weekly [aggregation](#adding-new-events) events. +- `#{category}_total_unique_counts_weekly` total unique counts for events in same category for last 7 days or last complete week, if events are in the same Redis slot and if we have more than one metric. +- `#{event_name}_weekly`: Data for 7 days for daily [aggregation](#adding-new-events) events and data for the last complete week for weekly [aggregation](#adding-new-events) events. +- `#{event_name}_monthly`: Data for 28 days for daily [aggregation](#adding-new-events) events and data for the last 4 complete weeks for weekly [aggregation](#adding-new-events) events. +- `#{category}_total_unique_counts_weekly`: Total unique counts for events in the same category for the last 7 days or the last complete week, if events are in the same Redis slot and we have more than one metric. +- `#{category}_total_unique_counts_monthly`: Total unique counts for events in same category for the last 28 days or the last 4 complete weeks, if events are in the same Redis slot and we have more than one metric. + +Example of `redis_hll_counters` data: + +```ruby +{:redis_hll_counters=> + {"compliance"=> + {"g_compliance_dashboard_weekly"=>0, + "g_compliance_dashboard_monthly"=>0, + "g_compliance_audit_events_weekly"=>0, + "g_compliance_audit_events_monthly"=>0, + "compliance_total_unique_counts_weekly"=>0, + "compliance_total_unique_counts_monthly"=>0}, + "analytics"=> + {"g_analytics_contribution_weekly"=>0, + "g_analytics_contribution_monthly"=>0, + "g_analytics_insights_weekly"=>0, + "g_analytics_insights_monthly"=>0, + "analytics_total_unique_counts_weekly"=>0, + "analytics_total_unique_counts_monthly"=>0}, + "ide_edit"=> + {"g_edit_by_web_ide_weekly"=>0, + "g_edit_by_web_ide_monthly"=>0, + "g_edit_by_sfe_weekly"=>0, + "g_edit_by_sfe_monthly"=>0, + "ide_edit_total_unique_counts_weekly"=>0, + "ide_edit_total_unique_counts_monthly"=>0}, + "search"=> + {"i_search_total_weekly"=>0, "i_search_total_monthly"=>0, "i_search_advanced_weekly"=>0, "i_search_advanced_monthly"=>0, "i_search_paid_weekly"=>0, "i_search_paid_monthly"=>0, "search_total_unique_counts_weekly"=>0, "search_total_unique_counts_monthly"=>0}, + "source_code"=>{"wiki_action_weekly"=>0, "wiki_action_monthly"=>0} + } +``` + +Example usage: ```ruby # Redis Counters redis_usage_data(Gitlab::UsageDataCounters::WikiPageCounter) redis_usage_data { ::Gitlab::UsageCounters::PodLogs.usage_totals[:total] } -# Redis HLL counter -counter = Gitlab::UsageDataCounters::TrackUniqueActions -redis_usage_data do - counter.count_unique_events( - event_action: Gitlab::UsageDataCounters::TrackUniqueActions::PUSH_ACTION, - date_from: time_period[:created_at].first, - date_to: time_period[:created_at].last - ) +# Define events in known_events.yml https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events.yml + +# Tracking events +Gitlab::UsageDataCounters::HLLRedisCounter.track_event(visitor_id, 'expand_vulnerabilities') + +# Get unique events for metric +redis_usage_data { Gitlab::UsageDataCounters::HLLRedisCounter.unique_events(event_names: 'expand_vulnerabilities', start_date: 28.days.ago, end_date: Date.current) } ``` ### Alternative Counters @@ -363,8 +564,6 @@ When adding, changing, or updating metrics, please update the [Event Dictionary' Check if new metrics need to be added to the Versions Application. See `usage_data` [schema](https://gitlab.com/gitlab-services/version-gitlab-com/-/blob/master/db/schema.rb#L147) and usage data [parameters accepted](https://gitlab.com/gitlab-services/version-gitlab-com/-/blob/master/app/services/usage_ping.rb). Any metrics added under the `counts` key are saved in the `counts` column. -For further details, see the [Process to add additional instrumentation to the Usage Ping](https://about.gitlab.com/handbook/product/product-processes/#process-to-add-additional-instrumentation-to-the-usage-ping). - ### 6. Add the feature label Add the `feature` label to the Merge Request for new Usage Ping metrics. These are user-facing changes and are part of expanding the Usage Ping feature. @@ -627,6 +826,7 @@ The following is example content of the Usage Ping payload. "topology": { "duration_s": 0.013836685999194742, "application_requests_per_hour": 4224, + "query_apdex_weekly_average": 0.996, "failures": [], "nodes": [ { @@ -664,3 +864,24 @@ The following is example content of the Usage Ping payload. } } ``` + +## Exporting Usage Ping SQL queries and definitions + +Two Rake tasks exist to export Usage Ping definitions. + +- The Rake tasks export the raw SQL queries for `count`, `distinct_count`, `sum`. +- The Rake tasks export the Redis counter class or the line of the Redis block for `redis_usage_data`. +- The Rake tasks calculate the `alt_usage_data` metrics. + +In the home directory of your local GitLab installation run the following Rake tasks for the YAML and JSON versions respectively: + +```shell +# for YAML export +bin/rake gitlab:usage_data:dump_sql_in_yaml + +# for JSON export +bin/rake gitlab:usage_data:dump_sql_in_json + +# You may pipe the output into a file +bin/rake gitlab:usage_data:dump_sql_in_yaml > ~/Desktop/usage-metrics-2020-09-02.yaml +``` diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index b60a26c29b5..6ef9be381b4 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -1,3 +1,11 @@ +--- +type: reference, dev +stage: none +group: Development +info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +description: "GitLab development guidelines - testing best practices." +--- + # Testing best practices ## Test Design @@ -15,21 +23,6 @@ manifest themselves within our code. When designing our tests, take time to revi our test design. We can find some helpful heuristics documented in the Handbook in the [Test Engineering](https://about.gitlab.com/handbook/engineering/quality/test-engineering/#test-heuristics) section. -## Test speed - -GitLab has a massive test suite that, without [parallelization](ci.md#test-suite-parallelization-on-the-ci), can take hours -to run. It's important that we make an effort to write tests that are accurate -and effective _as well as_ fast. - -Here are some things to keep in mind regarding test performance: - -- `instance_double` and `spy` are faster than `FactoryBot.build(...)` -- `FactoryBot.build(...)` and `.build_stubbed` are faster than `.create`. -- Don't `create` an object when `build`, `build_stubbed`, `attributes_for`, - `spy`, or `instance_double` will do. Database persistence is slow! -- Don't mark a feature as requiring JavaScript (through `:js` in RSpec) unless it's _actually_ required for the test - to be valid. Headless browser testing is slow! - ## RSpec To run RSpec tests: @@ -57,13 +50,218 @@ bundle exec guard When using spring and guard together, use `SPRING=1 bundle exec guard` instead to make use of spring. -Use [Factory Doctor](https://test-prof.evilmartians.io/#/profilers/factory_doctor) to find cases on un-necessary database manipulation, which can cause slow tests. +### Test speed + +GitLab has a massive test suite that, without [parallelization](ci.md#test-suite-parallelization-on-the-ci), can take hours +to run. It's important that we make an effort to write tests that are accurate +and effective _as well as_ fast. + +Test performance is important to maintaining quality and velocity, and has a +direct impact on CI build times and thus fixed costs. We want thorough, correct, +and fast tests. Here you can find some information about tools and techniques +available to you to achieve that. + +#### Don't request capabilities you don't need + +We make it easy to add capabilities to our examples by annotating the example or +a parent context. Examples of these are: + +- `:js` in feature specs, which runs a full JavaScript capable headless browser. +- `:clean_gitlab_redis_cache` which provides a clean Redis cache to the examples. +- `:request_store` which provides a request store to the examples. + +Obviously we should reduce test dependencies, and avoiding +capabilities also reduces the amount of set-up needed. + +`:js` is particularly important to avoid. This must only be used if the feature +test requires JavaScript reactivity in the browser, since using a headless +browser is much slower than parsing the HTML response from the app. + +#### Optimize factory usage + +A common cause of slow tests is excessive creation of objects, and thus +computation and DB time. Factories are essential to development, but they can +make inserting data into the DB so easy that we may be able to optimize. + +The two basic techniques to bear in mind here are: + +- **Reduce**: avoid creating objects, and avoid persisting them. +- **Reuse**: shared objects, especially nested ones we do not examine, can generally be shared. + +To avoid creation, it is worth bearing in mind that: + +- `instance_double` and `spy` are faster than `FactoryBot.build(...)`. +- `FactoryBot.build(...)` and `.build_stubbed` are faster than `.create`. +- Don't `create` an object when `build`, `build_stubbed`, `attributes_for`, + `spy`, or `instance_double` will do. Database persistence is slow! + +Use [Factory Doctor](https://test-prof.evilmartians.io/#/profilers/factory_doctor) to find cases where database persistence is not needed in a given test. ```shell # run test for path FDOC=1 bin/rspec spec/[path]/[to]/[spec].rb ``` +A common change is to use `build` or `build_stubbed` instead of `create`: + +```ruby +# Old +let(:project) { create(:project) } + +# New +let(:project) { build(:project) } +``` + +[Factory Profiler](https://test-prof.evilmartians.io/#/profilers/factory_prof) can help to identify repetitive database persistence via factories. + +```shell +# run test for path +FPROF=1 bin/rspec spec/[path]/[to]/[spec].rb + +# to visualize with a flamegraph +FPROF=flamegraph bin/rspec spec/[path]/[to]/[spec].rb +``` + +A common change is to use [`let_it_be`](#common-test-setup): + +```ruby +# Old +let(:project) { create(:project) } + +# New +let_it_be(:project) { create(:project) } +``` + +A common cause of a large number of created factories is [factory cascades](https://github.com/test-prof/test-prof/blob/master/docs/profilers/factory_prof.md#factory-flamegraph), which result when factories create and recreate associations. +They can be identified by a noticeable difference between `total time` and `top-level time` numbers: + +```plaintext + total top-level total time time per call top-level time name + + 208 0 9.5812s 0.0461s 0.0000s namespace + 208 76 37.4214s 0.1799s 13.8749s project +``` + +The table above shows us that we never create any `namespace` objects explicitly +(`top-level == 0`) - they are all created implicitly for us. But we still end up +with 208 of them (one for each project) and this takes 9.5 seconds. + +In order to reuse a single object for all calls to a named factory in implicit parent associations, +[`FactoryDefault`](https://github.com/test-prof/test-prof/blob/master/docs/recipes/factory_default.md) +can be used: + +```ruby + let_it_be(:namespace) { create_default(:namespace) } +``` + +Then every project we create will use this `namespace`, without us having to pass +it as `namespace: namespace`. + +Maybe we don't need to create 208 different projects - we +can create one and reuse it. In addition, we can see that only about 1/3 of the +projects we create are ones we ask for (76/208), so there is benefit in setting +a default value for projects as well: + +```ruby + let_it_be(:project) { create_default(:project) } +``` + +In this case, the `total time` and `top-level time` numbers match more closely: + +```plaintext + total top-level total time time per call top-level time name + + 31 30 4.6378s 0.1496s 4.5366s project + 8 8 0.0477s 0.0477s 0.0477s namespace +``` + +#### Identify slow tests + +Running a spec with profiling is a good way to start optimizing a spec. This can +be done with: + +```shell +bundle exec rspec --profile -- path/to/spec_file.rb +``` + +Which includes information like the following: + +```plaintext +Top 10 slowest examples (10.69 seconds, 7.7% of total time): + Issue behaves like an editable mentionable creates new cross-reference notes when the mentionable text is edited + 1.62 seconds ./spec/support/shared_examples/models/mentionable_shared_examples.rb:164 + Issue relative positioning behaves like a class that supports relative positioning .move_nulls_to_end manages to move nulls to the end, stacking if we cannot create enough space + 1.39 seconds ./spec/support/shared_examples/models/relative_positioning_shared_examples.rb:88 + Issue relative positioning behaves like a class that supports relative positioning .move_nulls_to_start manages to move nulls to the end, stacking if we cannot create enough space + 1.27 seconds ./spec/support/shared_examples/models/relative_positioning_shared_examples.rb:180 + Issue behaves like an editable mentionable behaves like a mentionable extracts references from its reference property + 0.99253 seconds ./spec/support/shared_examples/models/mentionable_shared_examples.rb:69 + Issue behaves like an editable mentionable behaves like a mentionable creates cross-reference notes + 0.94987 seconds ./spec/support/shared_examples/models/mentionable_shared_examples.rb:101 + Issue behaves like an editable mentionable behaves like a mentionable when there are cached markdown fields sends in cached markdown fields when appropriate + 0.94148 seconds ./spec/support/shared_examples/models/mentionable_shared_examples.rb:86 + Issue behaves like an editable mentionable when there are cached markdown fields when the markdown cache is stale persists the refreshed cache so that it does not have to be refreshed every time + 0.92833 seconds ./spec/support/shared_examples/models/mentionable_shared_examples.rb:153 + Issue behaves like an editable mentionable when there are cached markdown fields refreshes markdown cache if necessary + 0.88153 seconds ./spec/support/shared_examples/models/mentionable_shared_examples.rb:130 + Issue behaves like an editable mentionable behaves like a mentionable generates a descriptive back-reference + 0.86914 seconds ./spec/support/shared_examples/models/mentionable_shared_examples.rb:65 + Issue#related_issues returns only authorized related issues for given user + 0.84242 seconds ./spec/models/issue_spec.rb:335 + +Finished in 2 minutes 19 seconds (files took 1 minute 4.42 seconds to load) +277 examples, 0 failures, 1 pending +``` + +From this result, we can see the most expensive examples in our spec, giving us +a place to start. The fact that the most expensive examples here are in +shared examples means that any reductions are likely to have a larger impact as +they are called in multiple places. + +#### Avoid repeating expensive actions + +While isolated examples are very clear, and help serve the purpose of specs as +specification, the following example shows how we can combine expensive +actions: + +```ruby +subject { described_class.new(arg_0, arg_1) } + +it 'creates an event' do + expect { subject.execute }.to change(Event, :count).by(1) +end + +it 'sets the frobulance' do + expect { subject.execute }.to change { arg_0.reset.frobulance }.to('wibble') +end + +it 'schedules a background job' do + expect(BackgroundJob).to receive(:perform_async) + + subject.execute +end +``` + +If the call to `subject.execute` is expensive, then we are repeating the same +action just to make different assertions. We can reduce this repetition by +combining the examples: + +```ruby +it 'performs the expected side-effects' do + expect(BackgroundJob).to receive(:perform_async) + + expect { subject.execute } + .to change(Event, :count).by(1) + .and change { arg_0.frobulance }.to('wibble') +end +``` + +Be careful doing this, as this sacrifices clarity and test independence for +performance gains. + +When combining tests, consider using `:aggregate_failures`, so that the full +results are available, and not just the first failure. + ### General guidelines - Use a single, top-level `RSpec.describe ClassName` block. @@ -229,9 +427,9 @@ spec itself, but the former is preferred. It takes around one second to load tests that are using `fast_spec_helper` instead of 30+ seconds in case of a regular `spec_helper`. -### `let` variables +### `subject` and `let` variables -GitLab's RSpec suite has made extensive use of `let`(along with it strict, non-lazy +GitLab's RSpec suite has made extensive use of `let`(along with its strict, non-lazy version `let!`) variables to reduce duplication. However, this sometimes [comes at the cost of clarity](https://thoughtbot.com/blog/lets-not), so we need to set some guidelines for their use going forward: @@ -250,6 +448,9 @@ so we need to set some guidelines for their use going forward: - `let!` variables should be used only in case if strict evaluation with defined order is required, otherwise `let` will suffice. Remember that `let` is lazy and won't be evaluated until it is referenced. +- Avoid referencing `subject` in examples. Use a named subject `subject(:name)`, or a `let` variable instead, so + the variable has a contextual name. +- If the `subject` is never referenced inside examples, then it's acceptable to define the `subject` without a name. ### Common test setup @@ -468,6 +669,48 @@ for modifications. If you have no other choice, an `around` block similar to the example for global variables, above, can be used, but this should be avoided if at all possible. +#### Test Snowplow events + +CAUTION: **Warning:** +Snowplow performs **runtime type checks** by using the [contracts gem](https://rubygems.org/gems/contracts). +Since Snowplow is **by default disabled in tests and development**, it can be hard to +**catch exceptions** when mocking `Gitlab::Tracking`. + +To catch runtime errors due to type checks, you can enable Snowplow in tests by marking the spec with +`:snowplow` and use the `expect_snowplow_event` helper which will check for +calls to `Gitlab::Tracking#event`. + +```ruby +describe '#show', :snowplow do + it 'tracks snowplow events' do + get :show + + expect_snowplow_event( + category: 'Experiment', + action: 'start', + ) + expect_snowplow_event( + category: 'Experiment', + action: 'sent', + property: 'property', + label: 'label' + ) + end +end +``` + +When you want to ensure that no event got called, you can use `expect_no_snowplow_event`. + +```ruby + describe '#show', :snowplow do + it 'does not track any snowplow events' do + get :show + + expect_no_snowplow_event + end + end +``` + ### Table-based / Parameterized tests This style of testing is used to exercise one piece of code with a comprehensive diff --git a/doc/development/testing_guide/end_to_end/beginners_guide.md b/doc/development/testing_guide/end_to_end/beginners_guide.md index 15a9b4406ab..c552c44c864 100644 --- a/doc/development/testing_guide/end_to_end/beginners_guide.md +++ b/doc/development/testing_guide/end_to_end/beginners_guide.md @@ -84,7 +84,7 @@ See the [`RSpec.describe` outer block](#the-outer-rspecdescribe-block) CAUTION: **Deprecation notice:** The outer `context` [was deprecated](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/550) in `13.2` -in adherance to RSpec 4.0 specifications. Use `RSpec.describe` instead. +in adherence to RSpec 4.0 specifications. Use `RSpec.describe` instead. ### The outer `RSpec.describe` block @@ -287,7 +287,7 @@ Note the following important points: - Our test fabricates only what it needs, when it needs it. - The issue is fabricated through the API to save time. - GitLab prefers `let()` over instance variables. See - [best practices](../best_practices.md#let-variables). + [best practices](../best_practices.md#subject-and-let-variables). - `be_closed` is not implemented in `page/project/issue/show.rb` yet, but will be implemented in the next step. diff --git a/doc/development/testing_guide/end_to_end/best_practices.md b/doc/development/testing_guide/end_to_end/best_practices.md index 3b193721143..36cb49256a6 100644 --- a/doc/development/testing_guide/end_to_end/best_practices.md +++ b/doc/development/testing_guide/end_to_end/best_practices.md @@ -1,7 +1,128 @@ # End-to-end testing Best Practices NOTE: **Note:** -This is an tailored extension of the Best Practices [found in the testing guide](../best_practices.md). +This is a tailored extension of the Best Practices [found in the testing guide](../best_practices.md). + +## Link a test to its test-case issue + +Every test should have a corresponding issue in the [Quality Testcases project](https://gitlab.com/gitlab-org/quality/testcases/). +It's recommended that you reuse the issue created to plan the test. If one does not already exist you +can create the issue yourself. Alternatively, you can run the test in a pipeline that has reporting +enabled and the test-case issue reporter will automatically create a new issue. + +Whether you create a new test-case issue or one is created automatically, you will need to manually add +a `testcase` RSpec metadata tag. In most cases, a single test will be associated with a single test-case +issue ([see below for exceptions](#exceptions)). + +For example: + +```ruby +RSpec.describe 'Stage' do + describe 'General description of the feature under test' do + it 'test name', testcase: 'https://gitlab.com/gitlab-org/quality/testcases/-/issues/:issue_id' do + ... + end + + it 'another test', testcase: 'https://gitlab.com/gitlab-org/quality/testcases/-/issues/:another_issue_id' do + ... + end + end +end +``` + +### Exceptions + +Most tests are defined by a single line of a `spec` file, which is why those tests can be linked to a +single test-case issue via the `testcase` tag. + +However, some tests don't have a one-to-one relationship between a line of a `spec` file and a test-case +issue. This is because some tests are defined in a way that means a single line is associated with +multiple tests, including: + +- Parallelized tests. +- Templated tests. +- Tests in shared examples that include more than one example. + +In those and similar cases we can't assign a single `testcase` tag and so we rely on the test-case +reporter to programmatically determine the correct test-case issue based on the name and description of +the test. In such cases, the test-case reporter will automatically create a test-case issue the first time +the test runs, if no issue exists already. + +In such a case, if you create the issue yourself or want to reuse an existing issue, +you must use this [end-to-end test issue template](https://gitlab.com/gitlab-org/quality/testcases/-/blob/master/.gitlab/issue_templates/End-to-end%20Test.md) +to format the issue description. + +To illustrate, there are two tests in the shared examples in [`qa/specs/features/ee/browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/47b17db82c38ab704a23b5ba5d296ea0c6a732c8/qa/qa/specs/features/ee/browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb): + +```ruby +shared_examples 'only user with access pushes and merges' do + it 'unselected maintainer user fails to push' do + ... + end + + it 'selected developer user pushes and merges' do + ... + end +end +``` + +Consider the following test that includes the shared examples: + +```ruby +RSpec.describe 'Create' do + describe 'Restricted protected branch push and merge' do + context 'when only one user is allowed to merge and push to a protected branch' do + ... + it_behaves_like 'only user with access pushes and merges' + end + end +end +``` + +There would be two associated test-case issues, one for each shared example, with the following content: + +[Test 1](https://gitlab.com/gitlab-org/quality/testcases/-/issues/600): + +````markdown +```markdown +Title: browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb | Create Restricted +protected branch push and merge when only one user is allowed to merge and push to a protected +branch behaves like only user with access pushes and merges selecte... + +Description: +### Full description + +Create Restricted protected branch push and merge when only one user is allowed to merge and push +to a protected branch behaves like only user with access pushes and merges selected developer user +pushes and merges + +### File path + +./qa/specs/features/ee/browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb + +``` +```` + +[Test 2](https://gitlab.com/gitlab-org/quality/testcases/-/issues/602): + +````markdown +```markdown +Title: browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb | Create Restricted +protected branch push and merge when only one user is allowed to merge and push to a protected +branch behaves like only user with access pushes and merges unselec... + +Description: +### Full description + +Create Restricted protected branch push and merge when only one user is allowed to merge and push +to a protected branch behaves like only user with access pushes and merges unselected maintainer +user fails to push + +### File path + +./qa/specs/features/ee/browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb +``` +```` ## Prefer API over UI @@ -166,3 +287,26 @@ end NOTE: **Note:** A few exceptions for using a `ProjectPush` would be when your test calls for testing SSH integration or using the Git CLI. + +## Preferred method to blur elements + +To blur an element, the preferred method is to click another element that does not alter the test state. +If there's a mask that blocks the page elements, such as may occur with some dropdowns, +use WebDriver's native mouse events to simulate a click event on the coordinates of an element. Use the following method: `click_element_coordinates`. + +Avoid clicking the `body` for blurring elements such as inputs and dropdowns because it clicks the center of the viewport. +This action can also unintentionally click other elements, altering the test state and causing it to fail. + +```ruby +# Clicking another element to blur an input +def add_issue_to_epic(issue_url) + find_element(:issue_actions_split_button).find('button', text: 'Add an issue').click + fill_element :add_issue_input, issue_url + # Clicking the title blurs the input + click_element :title + click_element :add_issue_button +end + +# Using native mouse click events in the case of a mask/overlay +click_element_coordinates(:title) +``` diff --git a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md index 4059c1960e2..a9f54b53e5a 100644 --- a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md +++ b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md @@ -8,13 +8,14 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec | Tag | Description | |-----|-------------| | `:elasticsearch` | The test requires an Elasticsearch service. It is used by the [instance-level scenario](https://gitlab.com/gitlab-org/gitlab-qa#definitions) [`Test::Integration::Elasticsearch`](https://gitlab.com/gitlab-org/gitlab/-/blob/72b62b51bdf513e2936301cb6c7c91ec27c35b4d/qa/qa/ee/scenario/test/integration/elasticsearch.rb) to include only tests that require Elasticsearch. | +| `:gitaly_cluster` | The test will run against a GitLab instance where repositories are stored on redundant Gitaly nodes behind a Praefect node. All nodes are [separate containers](../../../administration/gitaly/praefect.md#requirements-for-configuring-a-gitaly-cluster). Tests that use this tag have a longer setup time since there are three additional containers that need to be started. | +| `:jira` | The test requires a Jira Server. [GitLab-QA](https://gitlab.com/gitlab-org/gitlab-qa) will provision the Jira Server in a Docker container when the `Test::Integration::Jira` test scenario is run. | `:kubernetes` | The test includes a GitLab instance that is configured to be run behind an SSH tunnel, allowing a TLS-accessible GitLab. This test will also include provisioning of at least one Kubernetes cluster to test against. *This tag is often be paired with `:orchestrated`.* | +| `:only` | The test is only to be run against specific environments. See [Environment selection](environment_selection.md) for more information. | | `:orchestrated` | The GitLab instance under test may be [configured by `gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#orchestrated-tests) to be different to the default GitLab configuration, or `gitlab-qa` may launch additional services in separate Docker containers, or both. Tests tagged with `:orchestrated` are excluded when testing environments where we can't dynamically modify GitLab's configuration (for example, Staging). | | `:quarantine` | The test has been [quarantined](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests), will run in a separate job that only includes quarantined tests, and is allowed to fail. The test will be skipped in its regular job so that if it fails it will not hold up the pipeline. Note that you can also [quarantine a test only when it runs against specific environment](environment_selection.md#quarantining-a-test-for-a-specific-environment). | | `:reliable` | The test has been [promoted to a reliable test](https://about.gitlab.com/handbook/engineering/quality/guidelines/reliable-tests/#promoting-an-existing-test-to-reliable) meaning it passes consistently in all pipelines, including merge requests. | | `:requires_admin` | The test requires an admin account. Tests with the tag are excluded when run against Canary and Production environments. | | `:runner` | The test depends on and will set up a GitLab Runner instance, typically to run a pipeline. | -| `:gitaly_ha` | The test will run against a GitLab instance where repositories are stored on redundant Gitaly nodes behind a Praefect node. All nodes are [separate containers](../../../administration/gitaly/praefect.md#requirements-for-configuring-a-gitaly-cluster). Tests that use this tag have a longer setup time since there are three additional containers that need to be started. | | `:skip_live_env` | The test will be excluded when run against live deployed environments such as Staging, Canary, and Production. | -| `:jira` | The test requires a Jira Server. [GitLab-QA](https://gitlab.com/gitlab-org/gitlab-qa) will provision the Jira Server in a Docker container when the `Test::Integration::Jira` test scenario is run. -| `:only` | The test is only to be run against specific environments. See [Environment selection](environment_selection.md) for more information. | +| `:testcase` | The link to the test case issue in the [Quality Testcases project](https://gitlab.com/gitlab-org/quality/testcases/). | diff --git a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md index 2cf2bb5b1d0..7ac0a00fcff 100644 --- a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md +++ b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md @@ -134,3 +134,262 @@ Once you have finished testing you can stop and remove the Docker containers: docker stop gitlab-gitaly-ha praefect postgres gitaly3 gitaly2 gitaly1 docker rm gitlab-gitaly-ha praefect postgres gitaly3 gitaly2 gitaly1 ``` + +## Guide to run and debug Monitor tests + +### How to set up + +To run the Monitor tests locally, against the GDK, please follow the preparation steps below: + +1. Complete the [Prerequisites](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/master/doc/howto/auto_devops/index.md#prerequisites-for-gitlab-team-members-only), at least through step 5. Note that the monitor tests do not require permissions to work with GKE because they use [k3s as a Kubernetes cluster provider](https://github.com/rancher/k3s). +1. The test setup deploys the app in a Kubernetes cluster, using the Auto DevOps deployment strategy. +To enable Auto DevOps in GDK, follow the [associated setup](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/master/doc/howto/auto_devops/index.md#setup) instructions. If you have problems, review the [troubleshooting guide](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/master/doc/howto/auto_devops/tips_and_troubleshooting.md) or reach out to the `#gdk` channel in the internal GitLab Slack. +1. Do [secure your GitLab instance](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/master/doc/howto/auto_devops/index.md#secure-your-gitlab-instance) since it is now publicly accessible on `https://[YOUR-PORT].qa-tunnel.gitlab.info`. +1. Install the Kubernetes command line tool known as `kubectl`. Use the [official installation instructions](https://kubernetes.io/docs/tasks/tools/install-kubectl/). + +You might see NGINX issues when you run `gdk start` or `gdk restart`. In that case, run `sft login` to revalidate your credentials and regain access the QA Tunnel. + +### How to run + +Navigate to the folder in `/your-gdk/gitlab/qa` and issue the command: + +```shell +QA_DEBUG=true CHROME_HEADLESS=false GITLAB_ADMIN_USERNAME=rootusername GITLAB_ADMIN_PASSWORD=rootpassword GITLAB_QA_ACCESS_TOKEN=your_token_here GITLAB_QA_ADMIN_ACCESS_TOKEN=your_token_here CLUSTER_API_URL=https://kubernetes.docker.internal:6443 bundle exec bin/qa Test::Instance::All https://[YOUR-PORT].qa-tunnel.gitlab.info/ -- qa/specs/features/browser_ui/8_monitor/all_monitor_core_features_spec.rb --tag kubernetes --tag orchestrated --tag requires_admin +``` + +The following includes more information on the command: + +-`QA_DEBUG` - Set to `true` to verbosely log page object actions. +-`CHROME_HEADLESS` - When running locally, set to `false` to allow Chrome tests to be visible - watch your tests being run. +-`GITLAB_ADMIN_USERNAME` - Admin username to use when adding a license. +-`GITLAB_ADMIN_PASSWORD` - Admin password to use when adding a license. +-`GITLAB_QA_ACCESS_TOKEN` and `GITLAB_QA_ADMIN_ACCESS_TOKEN` - A valid personal access token with the `api` scope. This is used for API access during tests, and is used in the version that staging is currently running. The `ADMIN_ACCESS_TOKEN` is from a user with admin access. Used for API access as an admin during tests. +-`CLUSTER_API_URL` - Use the address `https://kubernetes.docker.internal:6443` . This address is used to enable the cluster to be network accessible while deploying using Auto DevOps. +-`https://[YOUR-PORT].qa-tunnel.gitlab.info/` - The address of your local GDK +-`qa/specs/features/browser_ui/8_monitor/all_monitor_core_features_spec.rb` - The path to the monitor core specs +-`--tag` - the meta-tags used to filter the specs correctly + +At the moment of this writing, there are two specs which run monitor tests: + +-`qa/specs/features/browser_ui/8_monitor/all_monitor_core_features_spec.rb` - has the specs of features in GitLab Core +-`qa/specs/features/ee/browser_ui/8_monitor/all_monitor_features_spec.rb` - has the specs of features for paid GitLab (Enterprise Edition) + +### How to debug + +The monitor tests follow this setup flow: + +1. Creates a k3s cluster on your local machine. +1. Creates a project that has Auto DevOps enabled and uses an Express template (NodeJS) for the app to be deployed. +1. Associates the created cluster to the project and installs GitLab Runner, Prometheus and Ingress which are the needed components for a successful deployment. +1. Creates a CI pipeline with 2 jobs (`build` and `production`) to deploy the app on the Kubernetes cluster. +1. Goes to Operation > Metrics menu to verify data is being received and the app is being monitored successfully. + +The test requires a number of components. The setup requires time to collect the metrics of a real deployment. +The complexity of the setup may lead to problems unrelated to the app. The following sections include common strategies to debug possible issues. + +#### Deployment with Auto DevOps + +When debugging issues in the CI or locally in the CLI, open the Kubernetes job in the pipeline. +In the job log window, click on the top right icon labeled as *"Show complete raw"* to reveal raw job logs. +You can now search through the logs for *Job log*, which matches delimited sections like this one: + +```shell +------- Job log: ------- +``` + +A Job log is a subsection within these logs, related to app deployment. We use two jobs: `build` and `production`. +You can find the root causes of deployment failures in these logs, which can compromise the entire test. +If a `build` job fails, the `production` job won't run, and the test fails. + +The long test setup does not take screenshots of failures, which is a known [issue](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/270). +However, if the spec fails (after a successful deployment) then you should be able to find screenshots which display the feature failure. +To access them in CI, go to the main job log window, look on the left side panel's Job artifacts section, and click Browse. + +#### Common issues + +**Container Registry** + +When enabling Auto DevOps in the GDK, you may see issues with the Container Registry, which stores +images of the app to be deployed. + +You can access the Registry is available by opening an existing project. On the left hand menu, +select `Packages & Registries > Container Registries`. If the Registry is available, this page should load normally. + +Also, the Registry should be running in Docker: + +```shell +$ docker ps + +CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES +f035f339506c registry.gitlab.com/gitlab-org/build/cng/gitlab-container-registry:v2.9.1-gitlab "/bin/sh -c 'exec /b…" 3 hours ago Up 3 hours 0.0.0.0:5000->5000/tcp jovial_proskuriakova +``` + +The `gdk status` command shows if the registry is running: + +```shell +run: ./services/registry: (pid 2662) 10875s, normally down; run: log: (pid 65148) 177993s +run: ./services/tunnel_gitlab: (pid 2650) 10875s, normally down; run: log: (pid 65154) 177993s +run: ./services/tunnel_registry: (pid 2651) 10875s, normally down; run: log: (pid 65155) 177993s +``` + +Also, restarting Docker and then, on the Terminal, issue the command `docker login https://[YOUR-REGISTRY-PORT].qa-tunnel.gitlab.info:443` and use the GDK credentials to login. Note that the Registry port and GDK port are not the same. When configuring Auto DevOps in GDK, the `gdk reconfigure` command outputs the port of the Registry: + +```shell +********************************************* +Tunnel URLs + +GitLab: https://[PORT].qa-tunnel.gitlab.info +Registry: https://[PORT].qa-tunnel.gitlab.info +********************************************* +``` + +These Tunnel URLs are used by the QA SSH Tunnel generated when enabling Auto DevOps on the GDK. + +**Pod Eviction** + +Pod eviction happens when a node in a Kubernetes cluster is running out of memory or disk. After many local deployments this issue can happen. The UI shows that installing Prometheus, GitLab Runner and Ingress failed. How to be sure it is an Eviction? While the test is running, open another Terminal window and debug the current Kubernetes cluster by `kubectl get pods --all-namespaces`. If you observe that Pods have *Evicted status* such as the install-runner here: + +```shell +$ kubectl get pods --all-namespaces + +NAMESPACE NAME READY STATUS RESTARTS AGE +gitlab-managed-apps install-ingress 0/1 Pending 0 25s +gitlab-managed-apps install-prometheus 0/1 Pending 0 12s +gitlab-managed-apps install-runner 0/1 Evicted 0 75s +``` + +You can free some memory with either of the following commands: `docker prune system` or `docker prune volume`. + +## Geo tests + +Geo end-to-end tests can run locally against a [Geo GDK setup](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/master/doc/howto/geo.md) or on Geo spun up in Docker containers. + +### Using Geo GDK + +Run from the [`qa/` directory](https://gitlab.com/gitlab-org/gitlab/-/blob/f7272b77e80215c39d1ffeaed27794c220dbe03f/qa) with both GDK Geo primary and Geo secondary instances running: + +```shell +CHROME_HEADLESS=false bundle exec bin/qa QA::EE::Scenario::Test::Geo --primary-address http://localhost:3001 --secondary-address http://localhost:3002 --without-setup +``` + +### Using Geo in Docker + +You can use [GitLab-QA Orchestrator](https://gitlab.com/gitlab-org/gitlab-qa) to orchestrate two GitLab containers and configure them as a Geo setup. + +Geo requires an EE license. To visit the Geo sites in your browser, you will need a reverse proxy server (for example, [NGINX](https://www.nginx.com/)). + +1. Export your EE license + + ```shell + export EE_LICENSE=$(cat <path/to/your/gitlab_license>) + ``` + +1. (Optional) Pull the GitLab image + + This step is optional because pulling the Docker image is part of the [`Test::Integration::Geo` orchestrated scenario](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/d8c5c40607c2be0eda58bbca1b9f534b00889a0b/lib/gitlab/qa/scenario/test/integration/geo.rb). However, it's easier to monitor the download progress if you pull the image first, and the scenario will skip this step after checking that the image is up to date. + + ```shell + # For the most recent nightly image + docker pull gitlab/gitlab-ee:nightly + + # For a specific release + docker pull gitlab/gitlab-ee:13.0.10-ee.0 + + # For a specific image + docker pull registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:examplesha123456789 + ``` + +1. Run the [`Test::Integration::Geo` orchestrated scenario](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/d8c5c40607c2be0eda58bbca1b9f534b00889a0b/lib/gitlab/qa/scenario/test/integration/geo.rb) with the `--no-teardown` option to build the GitLab containers, configure the Geo setup, and run Geo end-to-end tests. Running the tests after the Geo setup is complete is optional; the containers will keep running after you stop the tests. + + ```shell + # Using the most recent nightly image + gitlab-qa Test::Integration::Geo EE --no-teardown + + # Using a specific GitLab release + gitlab-qa Test::Integration::Geo EE:13.0.10-ee.0 --no-teardown + + # Using a full image address + GITLAB_QA_ACCESS_TOKEN=your-token-here gitlab-qa Test::Integration::Geo registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:examplesha123456789 --no-teardown + ``` + + You can use the `--no-tests` option to build the containers only, and then run the [`EE::Scenario::Test::Geo` scenario](https://gitlab.com/gitlab-org/gitlab/-/blob/f7272b77e80215c39d1ffeaed27794c220dbe03f/qa/qa/ee/scenario/test/geo.rb) from your GDK to complete setup and run tests. However, there might be configuration issues if your GDK and the containers are based on different GitLab versions. With the `--no-teardown` option, GitLab-QA uses the same GitLab version for the GitLab containers and the GitLab QA container used to configure the Geo setup. + +1. To visit the Geo sites in your browser, proxy requests to the hostnames used inside the containers. NGINX is used as the reverse proxy server for this example. + + _Map the hostnames to the local IP in `/etc/hosts` file on your machine:_ + + ```plaintext + 127.0.0.1 gitlab-primary.geo gitlab-secondary.geo + ``` + + _Note the assigned ports:_ + + ```shell + $ docker port gitlab-primary + + 80/tcp -> 0.0.0.0:32768 + + $ docker port gitlab-secondary + + 80/tcp -> 0.0.0.0:32769 + ``` + + _Configure the reverse proxy server with the assigned ports in `nginx.conf` file (usually found in `/usr/local/etc/nginx` on a Mac):_ + + ```plaintext + server { + server_name gitlab-primary.geo; + location / { + proxy_pass http://localhost:32768; # Change port to your assigned port + proxy_set_header Host gitlab-primary.geo; + } + } + + server { + server_name gitlab-secondary.geo; + location / { + proxy_pass http://localhost:32769; # Change port to your assigned port + proxy_set_header Host gitlab-secondary.geo; + } + } + ``` + + _Start or reload the reverse proxy server:_ + + ```shell + sudo nginx + # or + sudo nginx -s reload + ``` + +1. To run end-to-end tests from your local GDK, run the [`EE::Scenario::Test::Geo` scenario](https://gitlab.com/gitlab-org/gitlab/-/blob/f7272b77e80215c39d1ffeaed27794c220dbe03f/qa/qa/ee/scenario/test/geo.rb) from the [`gitlab/qa/` directory](https://gitlab.com/gitlab-org/gitlab/-/blob/f7272b77e80215c39d1ffeaed27794c220dbe03f/qa). Include `--without-setup` to skip the Geo configuration steps. + + ```shell + QA_DEBUG=true GITLAB_QA_ACCESS_TOKEN=[add token here] GITLAB_QA_ADMIN_ACCESS_TOKEN=[add token here] bundle exec bin/qa QA::EE::Scenario::Test::Geo \ + --primary-address http://gitlab-primary.geo \ + --secondary-address http://gitlab-secondary.geo \ + --without-setup + ``` + + If the containers need to be configured first (for example, if you used the `--no-tests` option in the previous step), run the `QA::EE::Scenario::Test::Geo scenario` as shown below to first do the Geo configuration steps, and then run Geo end-to-end tests. Make sure that `EE_LICENSE` is (still) defined in your shell session. + + ```shell + QA_DEBUG=true bundle exec bin/qa QA::EE::Scenario::Test::Geo \ + --primary-address http://gitlab-primary.geo \ + --primary-name gitlab-primary \ + --secondary-address http://gitlab-secondary.geo \ + --secondary-name gitlab-secondary + ``` + +1. Stop and remove containers + + ```shell + docker stop gitlab-primary gitlab-secondary + docker rm gitlab-primary gitlab-secondary + ``` + +#### Notes + +- You can find the full image address from a pipeline by [following these instructions](https://about.gitlab.com/handbook/engineering/quality/guidelines/tips-and-tricks/#running-gitlab-qa-pipeline-against-a-specific-gitlab-release). You might be prompted to set the `GITLAB_QA_ACCESS_TOKEN` variable if you specify the full image address. +- You can increase the wait time for replication by setting `GEO_MAX_FILE_REPLICATION_TIME` and `GEO_MAX_DB_REPLICATION_TIME`. The default is 120 seconds. +- To save time during tests, create a Personal Access Token with API access on the Geo primary node, and pass that value in as `GITLAB_QA_ACCESS_TOKEN` and `GITLAB_QA_ADMIN_ACCESS_TOKEN`. diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md index 83d03097466..30e78766dde 100644 --- a/doc/development/testing_guide/frontend_testing.md +++ b/doc/development/testing_guide/frontend_testing.md @@ -230,7 +230,7 @@ it('exists', () => { // Best // NOTE: both mount and shallowMount work as long as a DOM element is available - // Finds a properly formatted link with an accessable name of "Click Me" + // Finds a properly formatted link with an accessible name of "Click Me" getByRole(el, 'link', { name: /Click Me/i }) getByRole(el, 'link', { name: 'Click Me' }) // Finds any element with the text "Click Me" @@ -321,80 +321,56 @@ it('tests a promise', async () => { }); it('tests a promise rejection', async () => { - expect.assertions(1); - try { - await user.getUserName(1); - } catch (e) { - expect(e).toEqual({ - error: 'User with 1 not found.', - }); - } + await expect(user.getUserName(1)).rejects.toThrow('User with 1 not found.'); }); ``` -You can also work with Promise chains. In this case, you can make use of the `done` callback and `done.fail` in case an error occurred. Following are some examples: +You can also simply return a promise from the test function. + +NOTE: **Note:** +Using the `done` and `done.fail` callbacks is discouraged when working with +promises. They should only be used when testing callback-based code. **Bad**: ```javascript -// missing done callback +// missing return it('tests a promise', () => { promise.then(data => { expect(data).toBe(asExpected); }); }); -// missing catch -it('tests a promise', done => { - promise - .then(data => { - expect(data).toBe(asExpected); - }) - .then(done); -}); - -// use done.fail in asynchronous tests +// uses done/done.fail it('tests a promise', done => { promise .then(data => { expect(data).toBe(asExpected); }) .then(done) - .catch(fail); -}); - -// missing catch -it('tests a promise rejection', done => { - promise - .catch(error => { - expect(error).toBe(expectedError); - }) - .then(done); + .catch(done.fail); }); ``` **Good**: ```javascript -// handling success -it('tests a promise', done => { - promise +// verifying a resolved promise +it('tests a promise', () => { + return promise .then(data => { expect(data).toBe(asExpected); - }) - .then(done) - .catch(done.fail); + }); }); -// failure case -it('tests a promise rejection', done => { - promise - .then(done.fail) - .catch(error => { - expect(error).toBe(expectedError); - }) - .then(done) - .catch(done.fail); +// verifying a resolved promise using Jest's `resolves` matcher +it('tests a promise', () => { + return expect(promise).resolves.toBe(asExpected); +}); + +// verifying a rejected promise using Jest's `rejects` matcher +it('tests a promise rejection', () => { + return expect(promise).rejects.toThrow(expectedError); }); ``` diff --git a/doc/development/testing_guide/index.md b/doc/development/testing_guide/index.md index 0d470e0e737..a61a700594c 100644 --- a/doc/development/testing_guide/index.md +++ b/doc/development/testing_guide/index.md @@ -4,7 +4,7 @@ This document describes various guidelines and best practices for automated testing of the GitLab project. It is meant to be an _extension_ of the [thoughtbot testing -style guide](https://github.com/thoughtbot/guides/tree/master/style/testing). If +style guide](https://github.com/thoughtbot/guides/tree/master/testing-rspec). If this guide defines a rule that contradicts the thoughtbot guide, this guide takes precedence. Some guidelines may be repeated verbatim to stress their importance. diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md index 68816ccfe45..61d3299cabf 100644 --- a/doc/development/testing_guide/review_apps.md +++ b/doc/development/testing_guide/review_apps.md @@ -165,8 +165,11 @@ This will grant you the following permissions for: ### Log into my Review App -The default username is `root` and its password can be found in the 1Password -secure note named `gitlab-{ce,ee} Review App's root password`. +For GitLab Team Members only. If you want to sign in to the review app, review +the GitLab handbook information for the [shared 1Password account](https://about.gitlab.com/handbook/security/#1password-for-teams). + +- The default username is `root`. +- The password can be found in the 1Password secure note named `gitlab-{ce,ee} Review App's root password`. ### Enable a feature flag for my Review App diff --git a/doc/development/uploads.md b/doc/development/uploads.md index 0c8b712a001..ee94553c200 100644 --- a/doc/development/uploads.md +++ b/doc/development/uploads.md @@ -264,3 +264,77 @@ sequenceDiagram deactivate sidekiq end ``` + +## How to add a new upload route + +In this section, we'll describe how to add a new upload route [accelerated](#uploading-technologies) by Workhorse for [body and multipart](#upload-encodings) encoded uploads. + +Uploads routes belong to one of these categories: + +1. Rails controllers: uploads handled by Rails controllers. +1. Grape API: uploads handled by a Grape API endpoint. +1. GraphQL API: uploads handled by a GraphQL resolve function. In these cases, there is nothing else + to do apart from implementing the actual upload. + +### Update Workhorse for the new route + +For both the Rails controller and Grape API uploads, Workhorse has to be updated in order to get the +support for the new upload route. + +1. Open an new issue in the [Workhorse tracker](https://gitlab.com/gitlab-org/gitlab-workhorse/-/issues/new) describing precisely the new upload route: + - The route's URL. + - The [upload encoding](#upload-encodings). + - If possible, provide a dump of the upload request. +1. Implement and get the MR merged for this issue above. +1. Ask the Maintainers of [Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) to create a new release. You can do that in the MR + directly during the maintainer review or ask for it in the `#workhorse` Slack channel. +1. Bump the [Workhorse version file](https://gitlab.com/gitlab-org/gitlab/-/blob/master/GITLAB_WORKHORSE_VERSION) + to the version you have from the previous points, or bump it in the same merge request that contains + the Rails changes (see [Implementing the new route with a Rails controller](#implementing-the-new-route-with-a-rails-controller) or [Implementing the new route with a Grape API endpoint](#implementing-the-new-route-with-a-grape-api-endpoint) below). + +### Implementing the new route with a Rails controller + +For a Rails controller upload, we usually have a [multipart](#upload-encodings) upload and there are a +few things to do: + +1. The upload is available under the parameter name you're using. For example, it could be an `artifact` + or a nested parameter such as `user[avatar]`. Let's say that we have the upload under the + `file` parameter, reading `params[:file]` should get you an [`UploadedFile`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/uploaded_file.rb) instance. +1. Generally speaking, it's a good idea to check if the instance is from the [`UploadedFile`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/uploaded_file.rb) class. For example, see how we checked +[that the parameter is indeed an `UploadedFile`](https://gitlab.com/gitlab-org/gitlab/-/commit/ea30fe8a71bf16ba07f1050ab4820607b5658719#51c0cc7a17b7f12c32bc41cfab3649ff2739b0eb_79_77). + +CAUTION: **Caution:** +**Do not** call `UploadedFile#from_params` directly! Do not build an [`UploadedFile`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/uploaded_file.rb) +instance using `UploadedFile#from_params`! This method can be unsafe to use depending on the `params` +passed. Instead, use the [`UploadedFile`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/uploaded_file.rb) +instance that [`multipart.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/middleware/multipart.rb) +builds automatically for you. + +### Implementing the new route with a Grape API endpoint + +For a Grape API upload, we can have [body or a multipart](#upload-encodings) upload. Things are slightly more complicated: two endpoints are needed. One for the +Workhorse pre-upload authorization and one for accepting the upload metadata from Workhorse: + +1. Implement an endpoint with the URL + `/authorize` suffix that will: + - Check that the request is coming from Workhorse with the `require_gitlab_workhorse!` from the [API helpers](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/api/helpers.rb). + - Check user permissions. + - Set the status to `200` with `status 200`. + - Set the content type with `content_type Gitlab::Workhorse::INTERNAL_API_CONTENT_TYPE`. + - Use your dedicated `Uploader` class (let's say that it's `FileUploader`) to build the response with `FileUploader.workhorse_authorize(params)`. +1. Implement the endpoint for the upload request that will: + - Require all the `UploadedFile` objects as parameters. + - For example, if we expect a single parameter `file` to be an [`UploadedFile`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/uploaded_file.rb) instance, +use `requires :file, type: ::API::Validations::Types::WorkhorseFile`. + - Body upload requests have their upload available under the parameter `file`. + - Check that the request is coming from Workhorse with the `require_gitlab_workhorse!` from the +[API helpers](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/api/helpers.rb). + - Check the user permissions. + - The remaining code of the processing. This is where the code must be reading the parameter (for +our example, it would be `params[:file]`). + +CAUTION: **Caution:** +**Do not** call `UploadedFile#from_params` directly! Do not build an [`UploadedFile`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/uploaded_file.rb) +object using `UploadedFile#from_params`! This method can be unsafe to use depending on the `params` +passed. Instead, use the [`UploadedFile`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/uploaded_file.rb) +object that [`multipart.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/middleware/multipart.rb) +builds automatically for you. diff --git a/doc/development/what_requires_downtime.md b/doc/development/what_requires_downtime.md index e9d4ed8eaf6..7c99bcde413 100644 --- a/doc/development/what_requires_downtime.md +++ b/doc/development/what_requires_downtime.md @@ -30,14 +30,14 @@ places. This can be done by defining the columns to ignore. For example, to igno ```ruby class User < ApplicationRecord include IgnorableColumns - ignore_column :updated_at, remove_with: '12.7', remove_after: '2019-12-22' + ignore_column :updated_at, remove_with: '12.7', remove_after: '2020-01-22' end ``` Multiple columns can be ignored, too: ```ruby -ignore_columns %i[updated_at created_at], remove_with: '12.7', remove_after: '2019-12-22' +ignore_columns %i[updated_at created_at], remove_with: '12.7', remove_after: '2020-01-22' ``` We require indication of when it is safe to remove the column ignore with: @@ -45,7 +45,7 @@ We require indication of when it is safe to remove the column ignore with: - `remove_with`: set to a GitLab release typically two releases (M+2) after adding the column ignore. - `remove_after`: set to a date after which we consider it safe to remove the column - ignore, typically within the development cycle of release M+2. + ignore, typically last date of the development cycle of release M+2 - namely the release date. This information allows us to reason better about column ignores and makes sure we don't remove column ignores too early for both regular releases and deployments to GitLab.com. For diff --git a/doc/development/windows.md b/doc/development/windows.md index c92a468fad3..3301e4f7c8f 100644 --- a/doc/development/windows.md +++ b/doc/development/windows.md @@ -87,7 +87,7 @@ You should now be remoted into a Windows machine with a command prompt. - Start the runner: `gitlab-runner.exe start`. For more information, see [Install GitLab Runner on Windows](https://docs.gitlab.com/runner/install/windows.html) -and [Registering Runners](https://docs.gitlab.com/runner/register/index.html). +and [Registering runners](https://docs.gitlab.com/runner/register/index.html). ## Developer tips |