diff options
Diffstat (limited to 'doc/development')
100 files changed, 1720 insertions, 746 deletions
diff --git a/doc/development/adding_database_indexes.md b/doc/development/adding_database_indexes.md index 571d2f353d4..0e8e8289464 100644 --- a/doc/development/adding_database_indexes.md +++ b/doc/development/adding_database_indexes.md @@ -317,11 +317,11 @@ end You must test the database index changes locally before creating a merge request. -### Verify indexes created asynchronously +### Verify indexes created asynchronously Use the asynchronous index helpers on your local environment to test changes for creating an index: 1. Enable the feature flags by running `Feature.enable(:database_async_index_creation)` and `Feature.enable(:database_reindexing)` in the Rails console. 1. Run `bundle exec rails db:migrate` so that it creates an entry in the `postgres_async_indexes` table. 1. Run `bundle exec rails gitlab:db:reindex` so that the index is created asynchronously. -1. To verify the index, open the PostgreSQL console using the [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/postgresql.md) command `gdk psql` and run the command `\d <index_name>` to check that your newly created index exists. +1. To verify the index, open the PostgreSQL console using the [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/postgresql.md) command `gdk psql` and run the command `\d <index_name>` to check that your newly created index exists. diff --git a/doc/development/api_graphql_styleguide.md b/doc/development/api_graphql_styleguide.md index 9a17ac4c813..afd745533c9 100644 --- a/doc/development/api_graphql_styleguide.md +++ b/doc/development/api_graphql_styleguide.md @@ -152,8 +152,8 @@ The `iid`, `title` and `description` are _scalar_ GraphQL types. `title` and `description` are regular `GraphQL::Types::String` types. Note that the old scalar types `GraphQL:ID`, `GraphQL::INT_TYPE`, `GraphQL::STRING_TYPE`, -and `GraphQL:BOOLEAN_TYPE` are no longer allowed. Please use `GraphQL::Types::ID`, -`GraphQL::Types::Int`, `GraphQL::Types::String`, and `GraphQL::Types::Boolean`. +`GraphQL:BOOLEAN_TYPE`, and `GraphQL::FLOAT_TYPE` are no longer allowed. Please use `GraphQL::Types::ID`, +`GraphQL::Types::Int`, `GraphQL::Types::String`, `GraphQL::Types::Boolean`, and `GraphQL::Types::Float`. When exposing a model through the GraphQL API, we do so by creating a new type in `app/graphql/types`. You can also declare custom GraphQL data types @@ -698,7 +698,10 @@ aware of the support. The documentation will mention that the old Global ID style is now deprecated. -See also [Aliasing and deprecating mutations](#aliasing-and-deprecating-mutations). +See also: + +- [Aliasing and deprecating mutations](#aliasing-and-deprecating-mutations). +- [How to filter Kibana for queries that used deprecated fields](graphql_guide/monitoring.md#queries-that-used-a-deprecated-field). ## Enums @@ -830,7 +833,7 @@ field :id, GraphQL::Types::ID, description: 'ID of the resource.' Descriptions of fields and arguments are viewable to users through: - The [GraphiQL explorer](#graphiql). -- The [static GraphQL API reference](../api/graphql/#reference). +- The [static GraphQL API reference](../api/graphql/reference/index.md). ### Description style guide @@ -2027,3 +2030,7 @@ elimination of laziness, where needed. For dealing with lazy values without forcing them, use `Gitlab::Graphql::Lazy.with_value`. + +## Monitoring GraphQL + +See the [Monitoring GraphQL](graphql_guide/monitoring.md) guide for tips on how to inspect logs of GraphQL requests and monitor the performance of your GraphQL queries. diff --git a/doc/development/application_slis/index.md b/doc/development/application_slis/index.md index 5bc6fffdb48..87c0bcfede5 100644 --- a/doc/development/application_slis/index.md +++ b/doc/development/application_slis/index.md @@ -180,7 +180,7 @@ alerts about the SLI in specified Slack channels. For more information, read the [alert routing documentation](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/uncategorized/alert-routing.md). In [this project](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/614) we are extending this so alerts for SLIs with a `feature_category` -label in the souce metrics can also be routed. +label in the source metrics can also be routed. For any question, please don't hesitate to create an issue in [the Scalability issue tracker](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues) diff --git a/doc/development/architecture.md b/doc/development/architecture.md index 38d0d5d7843..078eb5dd0de 100644 --- a/doc/development/architecture.md +++ b/doc/development/architecture.md @@ -126,7 +126,7 @@ graph LR Geo -- TCP 22 --> SSH Geo -- TCP 5432 --> PostgreSQL Runner{{GitLab Runner}} -- TCP 443 --> HTTP - K8sAgent{{GitLab Kubernetes Agent}} -- TCP 443 --> HTTP + K8sAgent{{GitLab Agent}} -- TCP 443 --> HTTP %% GitLab Application Suite subgraph GitLab @@ -157,7 +157,7 @@ graph LR Puma["Puma (GitLab Rails)"] Puma <--> Registry GitLabWorkhorse[GitLab Workhorse] <--> Puma - GitLabKas[GitLab Kubernetes Agent Server] --> GitLabWorkhorse + GitLabKas[GitLab Agent Server] --> GitLabWorkhorse GitLabPages[GitLab Pages] --> GitLabWorkhorse Mailroom Sidekiq @@ -349,7 +349,7 @@ Component statuses are linked to configuration documentation for each component. | [GitLab Exporter](#gitlab-exporter) | Generates a variety of GitLab metrics | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | CE & EE | | [GitLab Geo Node](#gitlab-geo) | Geographically distributed GitLab nodes | ⚙ | ⚙ | ❌ | ❌ | ✅ | ❌ | ⚙ | EE Only | | [GitLab Pages](#gitlab-pages) | Hosts static websites | ⚙ | ⚙ | ❌ | ❌ | ✅ | ⚙ | ⚙ | CE & EE | -| [GitLab Kubernetes Agent](#gitlab-kubernetes-agent) | Integrate Kubernetes clusters in a cloud-native way | ⚙ | ⚙ | ⚙ | ❌ | ❌ | ⤓ | ⚙ | EE Only | +| [GitLab Agent](#gitlab-agent) | Integrate Kubernetes clusters in a cloud-native way | ⚙ | ⚙ | ⚙ | ❌ | ❌ | ⤓ | ⚙ | EE Only | | [GitLab self-monitoring: Alertmanager](#alertmanager) | Deduplicates, groups, and routes alerts from Prometheus | ⚙ | ⚙ | ✅ | ⚙ | ✅ | ❌ | ❌ | CE & EE | | [GitLab self-monitoring: Grafana](#grafana) | Metrics dashboard | ✅ | ✅ | ⚙ | ⤓ | ✅ | ❌ | ⚙ | CE & EE | | [GitLab self-monitoring: Jaeger](#jaeger) | View traces generated by the GitLab instance | ❌ | ⚙ | ⚙ | ❌ | ❌ | ⤓ | ⚙ | CE & EE | @@ -499,14 +499,14 @@ Geo is a premium feature built to help speed up the development of distributed t GitLab Exporter is a process designed in house that allows us to export metrics about GitLab application internals to Prometheus. You can read more [in the project's README](https://gitlab.com/gitlab-org/gitlab-exporter). -#### GitLab Kubernetes Agent +#### GitLab Agent - [Project page](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent) - Configuration: - [Omnibus](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template) - [Charts](https://docs.gitlab.com/charts/charts/gitlab/kas/index.html) -[GitLab Kubernetes Agent](../user/clusters/agent/index.md) is an active in-cluster +The [GitLab Agent](../user/clusters/agent/index.md) is an active in-cluster component for solving GitLab and Kubernetes integration tasks in a secure and cloud-native way. diff --git a/doc/development/audit_event_guide/index.md b/doc/development/audit_event_guide/index.md index 65a7fa0ae90..ae2f9748178 100644 --- a/doc/development/audit_event_guide/index.md +++ b/doc/development/audit_event_guide/index.md @@ -176,3 +176,14 @@ deactivate B In addition to recording to the database, we also write these events to [a log file](../../administration/logs.md#audit_jsonlog). + +## Event streaming + +All events where the entity is a `Group` or `Project` are recorded in the audit log, and also streamed to one or more +[event streaming destinations](../../administration/audit_event_streaming.md). When the entity is a: + +- `Group`, events are streamed to the group's root ancestor's event streaming destinations. +- `Project`, events are streamed to the project's root ancestor's event streaming destinations. + +This feature is under heavy development. Follow the [parent epic](https://gitlab.com/groups/gitlab-org/-/epics/5925) for updates on feature +development. diff --git a/doc/development/backend/ruby_style_guide.md b/doc/development/backend/ruby_style_guide.md index 461bb9aafde..419db628b0d 100644 --- a/doc/development/backend/ruby_style_guide.md +++ b/doc/development/backend/ruby_style_guide.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w This is a GitLab-specific style guide for Ruby code. -Generally, if a style is not covered by [existing rubocop rules or styleguides](../contributing/style_guides.md#ruby-rails-rspec), it shouldn't be a blocker. +Generally, if a style is not covered by [existing Rubocop rules or style guides](../contributing/style_guides.md#ruby-rails-rspec), it shouldn't be a blocker. Before adding a new cop to enforce a given style, make sure to discuss it with your team. When the style is approved by a backend EM or by a BE staff eng, add a new section to this page to document the new rule. For every new guideline, add it in a new section and link the discussion from the section's diff --git a/doc/development/background_migrations.md b/doc/development/background_migrations.md index c93b5b448f0..4a18b2123da 100644 --- a/doc/development/background_migrations.md +++ b/doc/development/background_migrations.md @@ -165,7 +165,8 @@ big JSON blob) to column `bar` (containing a string). The process for this would roughly be as follows: 1. Release A: - 1. Create a migration class that perform the migration for a row with a given ID. + 1. Create a migration class that performs the migration for a row with a given ID. + You can use [background jobs tracking](#background-jobs-tracking) to simplify cleaning up. 1. Deploy the code for this release, this should include some code that will schedule jobs for newly created data (for example, using an `after_create` hook). 1. Schedule jobs for all existing rows in a post-deployment migration. It's @@ -174,8 +175,10 @@ roughly be as follows: 1. Release B: 1. Deploy code so that the application starts using the new column and stops scheduling jobs for newly created data. - 1. In a post-deployment migration use `finalize_background_migration` from - `BackgroundMigrationHelpers` to ensure no jobs remain. This helper will: + 1. In a post-deployment migration, finalize all jobs that have not succeeded by now. + If you used [background jobs tracking](#background-jobs-tracking) in release A, + you can use `finalize_background_migration` from `BackgroundMigrationHelpers` to ensure no jobs remain. + This helper will: 1. Use `Gitlab::BackgroundMigration.steal` to process any remaining jobs in Sidekiq. 1. Reschedule the migration to be run directly (that is, not through Sidekiq) @@ -510,12 +513,12 @@ See [`db/post_migrate/20210604070207_retry_backfill_traversal_ids.rb`](https://g ### Viewing failure error logs -After running a background migration, if any jobs have failed, you can view the logs in [Kibana](https://log.gprd.gitlab.net/goto/3afc1393447c401d7602c1874793e2f6). +After running a background migration, if any jobs have failed, you can view the logs in [Kibana](https://log.gprd.gitlab.net/goto/5f06a57f768c6025e1c65aefb4075694). View the production Sidekiq log and filter for: - `json.class: BackgroundMigrationWorker` - `json.job_status: fail` -- `json.args: <MyBackgroundMigrationClassName>` +- `json.meta.caller_id: <MyBackgroundMigrationClassName>` Looking at the `json.error_class`, `json.error_message` and `json.error_backtrace` values may be helpful in understanding why the jobs failed. diff --git a/doc/development/cascading_settings.md b/doc/development/cascading_settings.md index a85fc52d303..d04761400ac 100644 --- a/doc/development/cascading_settings.md +++ b/doc/development/cascading_settings.md @@ -194,7 +194,7 @@ This function should be imported and called in the [page-specific JavaScript](fe setting_locked: delayed_project_removal_locked, settings_path_helper: -> (locked_ancestor) { edit_group_path(locked_ancestor, anchor: 'js-permissions-settings') }, help_text: s_('Settings|Projects will be permanently deleted after a 7-day delay. Inherited by subgroups.') do - = s_('Settings|Enable delayed project removal') + = s_('Settings|Enable delayed project deletion') = render 'shared/namespaces/cascading_settings/enforcement_checkbox', attribute: :delayed_project_removal, group: @group, diff --git a/doc/development/changelog.md b/doc/development/changelog.md index 2753257c941..7c4a600e1fa 100644 --- a/doc/development/changelog.md +++ b/doc/development/changelog.md @@ -101,7 +101,7 @@ EE: true - _Any_ contribution from a community member, no matter how small, **may** have a changelog entry regardless of these guidelines if the contributor wants one. - Any [GLEX experiment](experiment_guide/gitlab_experiment.md) changes **should not** have a changelog entry. -- [Removing](feature_flags/#changelog) a feature flag, when the new code is retained. +- [Modifying](feature_flags/#changelog) a feature flag (flag removal, default-on setting). ## Writing good changelog entries diff --git a/doc/development/cicd/cicd_reference_documentation_guide.md b/doc/development/cicd/cicd_reference_documentation_guide.md index aa3888cd866..e937220d208 100644 --- a/doc/development/cicd/cicd_reference_documentation_guide.md +++ b/doc/development/cicd/cicd_reference_documentation_guide.md @@ -6,17 +6,17 @@ info: To determine the technical writer assigned to the Stage/Group associated w # CI/CD YAML reference style guide **(FREE)** -The CI/CD YAML reference uses a standard style to make it easier to use and update. +The [CI/CD YAML reference](../../ci/yaml/index.md) uses a standard style to make it easier to use and update. The reference information should be kept as simple as possible, and expanded details -and examples documented in a separate page. +and examples should be documented on other pages. ## YAML reference structure Every YAML keyword must have its own section in the reference. The sections should be nested so that the keywords follow a logical tree structure. For example: -```plaintext +```markdown ### `artifacts` #### `artifacts:name` #### `artifacts:paths` @@ -27,128 +27,127 @@ be nested so that the keywords follow a logical tree structure. For example: ## YAML reference style -Each keyword entry in the reference should use the following style: +Each keyword entry in the reference: -````markdown -### `keyword-name` - -> Version information - -Keyword description and main details. - -**Keyword type**: - -**Possible inputs**: +- Must have a simple introductory section. The introduction should give the fundamental + information needed to use the keyword. Advanced details and tasks should be in + feature pages, not the reference page. -**Example of `keyword-name`**: +- Must use the keyword name as the title, for example: -(optional) In this example... + ```markdown + ### `artifacts` + ``` -(optional) **Additional details**: +- Should include the following sections: + - [Keyword type](#keyword-type) + - [Possible inputs](#possible-inputs) + - [Example of `keyword-name`](#example-of-keyword-name) +- (Optional) Can also include the following sections when needed: + - [Additional details](#additional-details) + - [Related topics](#related-topics) -- List of extra details. +The keyword name must always be in backticks without a final `:`, like `artifacts`, not `artifacts:`. +If it is a subkey of another keyword, write out all the subkeys to the "parent" key the first time it +is used, like `artifacts:reports:dast`. Afterwards, you can use just the subkey alone, like `dast`. -(optional) **Related topics**: +## Keyword type -- List of links to topics related to the keyword. -```` - -- ``### `keyword-name` ``: The keyword name must always be in backticks. - If it is a subkey of another keyword, write out all the keywords, with each separated - by `:`, for example: `artifacts:reports:dast`. - -- ``> Version information``: The [version history details](../documentation/styleguide/index.md#version-text-in-the-version-history). - If the keyword is feature flagged, see the [feature flag documentation guide](../documentation/feature_flags.md) - as well. +The keyword can be either a job or global keyword. If it can be used in a `default` +section, make not of that as well, for example: -- `Keyword description and main details.`: A simple description of the keyword, and - how to use it. Additional use cases and benefits should be added to a page outside - the reference document. Link to that document in this section. +- `**Keyword type**: Global keyword.` +- `**Keyword type**: Job keyword. You can use it only as part of a job.` +- ``**Keyword type**: Job keyword. You can use it only as part of a job or in the [`default:` section](#default).`` -- `**Keyword type**:`: Most keywords are defined at the job level, like `script`, - or at the pipeline level, like `stages`. Add the appropriate line: +### Possible inputs - - `**Keyword type**: Job keyword. You can use it only as part of a job.` - - `**Keyword type**: Pipeline keyword. You cannot use it as part of a job.` +List all the possible inputs, and any extra details about the inputs, such as defaults +or changes due to different GitLab versions, for example: - If a keyword can be used at both the job and pipeline level, like `variables`, - explain it in detail instead of using the pre-written lines above. +```markdown +**Possible inputs**: -- `**Possible inputs**:`: Explain in detail which inputs the keyword can accept. - You can add the details in a sentence, paragraph, or list. +- `true` (default if not defined) or `false`. +``` -- ``**Example of `keyword-name`**:``: An example configuration that uses the keyword. - Do not add extra keywords that are not required to understand the behavior. +```markdown +**Possible inputs**: -- (optional) `In this example...`: If the example needs extra details, - add the clarification text below the example. +- A single exit code. +- An array of exit codes. +``` -- (optional) `**Additional details**:` If there are any caveats or extra details you - want to document along with the keyword, add each one as a list item here. +```markdown +**Possible inputs**: -- (optional) `**Related topics**:` If there are any other keywords or pages that - relate to this keyword, add these links as list items here. +- A string with the long description. +- The path to a file that contains the description. Introduced in [GitLab 13.7](https://gitlab.com/gitlab-org/release-cli/-/merge_requests/67). + - The file location must be relative to the project directory (`$CI_PROJECT_DIR`). + - If the file is a symbolic link, it must be in the `$CI_PROJECT_DIR`. + - The `./path/to/file` and filename can't contain spaces. +``` -### YAML reference style example +### Example of `keyword-name` -See the [`only:changes` / `except:changes`](../../ci/yaml/index.md#onlychanges--exceptchanges) -documentation for an example of the YAML reference style. The following example is a -shortened version of that documentation's Markdown: +An example of the keyword. Use the minimum number of other keywords necessary +to make the example valid. If the example needs explanation, add it after the example, +for example: ````markdown -#### `only:changes` / `except:changes` +**Example of `dast`**: -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/19232) in GitLab 11.4. - -Use the `changes` keyword with `only` to run a job, or with `except` to skip a job, -when a Git push event modifies a file. +```yaml +stages: + - build + - dast -Use `changes` in pipelines with the following refs: +include: + - template: DAST.gitlab-ci.yml -- `branches` -- `external_pull_requests` -- `merge_requests` (see additional details about [using `only:changes` with pipelines for merge requests](../jobs/job_control.md#use-onlychanges-with-pipelines-for-merge-requests)) +dast: + dast_configuration: + site_profile: "Example Co" + scanner_profile: "Quick Passive Test" +``` -**Keyword type**: Job keyword. You can use it only as part of a job. +In this example, the `dast` job extends the `dast` configuration added with the `include:` keyword +to select a specific site profile and scanner profile. +```` -**Possible inputs**: An array including any number of: +### Additional details -- Paths to files. -- Wildcard paths for single directories, for example `path/to/directory/*`, or a directory - and all its subdirectories, for example `path/to/directory/**/*`. +The additional details should be an unordered list of extra information that is +useful to know, but not important enough to put in the introduction. This information +can include changes introduced in different GitLab versions. For example: -**Example of `only:changes`**: +```markdown +**Additional details**: -```yaml -docker build: - script: docker build -t my-image:$CI_COMMIT_REF_SLUG . - only: - refs: - - branches - changes: - - Dockerfile - - docker/scripts/* - - dockerfiles/**/* +- The expiration time period begins when the artifact is uploaded and stored on GitLab. + If the expiry time is not defined, it defaults to the [instance wide setting](../../user/admin_area/settings/continuous_integration.md#default-artifacts-expiration). +- To override the expiration date and protect artifacts from being automatically deleted: + - Select **Keep** on the job page. + - [In GitLab 13.3 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/22761), set the value of + `expire_in` to `never`. ``` -In this example, `docker build` only runs in branch pipelines, and only if at least one of -these files changed: +### Related topics -- `Dockerfile`. -- Any file in `docker/scripts` -- Any file in `dockerfiles/` or any of its subdirectories. +The related topics should be an unordered list of crosslinks to related pages, including: -**Additional details**: +- Specific tasks that you can accomplish with the keyword. +- Advanced examples of the keyword. +- Other related keywords that can be used together with this keyword. -- If you use refs other than `branches`, `external_pull_requests`, or `merge_requests`, - `changes` can't determine if a given file is new or old and always returns `true`. -- If you use `only: changes` with other refs, jobs ignore the changes and always run. -- If you use `except: changes` with other refs, jobs ignore the changes and never run. +For example: +```markdown **Related topics**: -- [`only: changes` and `except: changes` examples](../jobs/job_control.md#onlychanges--exceptchanges-examples). -- If you use `changes` with [only allow merge requests to be merged if the pipeline succeeds](../../user/project/merge_requests/merge_when_pipeline_succeeds.md#only-allow-merge-requests-to-be-merged-if-the-pipeline-succeeds), - you should [also use `only:merge_requests`](../jobs/job_control.md#use-onlychanges-with-pipelines-for-merge-requests). -- Use `changes` with [scheduled pipelines](../jobs/job_control.md#use-onlychanges-with-scheduled-pipelines). -```` +- You can specify a [fallback cache key](../caching/index.md#use-a-fallback-cache-key) + to use if the specified `cache:key` is not found. +- You can [use multiple cache keys](../caching/index.md#use-multiple-caches) in a single job. +- See the [common `cache` use cases](../caching/index.md#common-use-cases-for-caches) for more + `cache:key` examples. +``` diff --git a/doc/development/cicd/templates.md b/doc/development/cicd/templates.md index 46442aa6106..b1252b86cc0 100644 --- a/doc/development/cicd/templates.md +++ b/doc/development/cicd/templates.md @@ -392,6 +392,6 @@ If you're unsure if it's secure or not, you must ask security experts for cross- After your CI/CD template MR is created and labeled with `ci::templates`, DangerBot suggests one reviewer and one maintainer that can review your code. When your merge -request is ready for review, please [mention](../../user/project/issues/issue_data_and_actions.md#mentions) +request is ready for review, please [mention](../../user/discussions/index.md#mentions) the reviewer and ask them to review your CI/CD template changes. See details in the merge request that added [a DangerBot task for CI/CD template MRs](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/44688). diff --git a/doc/development/code_review.md b/doc/development/code_review.md index 7e797309a26..742d183e15c 100644 --- a/doc/development/code_review.md +++ b/doc/development/code_review.md @@ -381,7 +381,7 @@ first time. ### Requesting a review When you are ready to have your merge request reviewed, -you should request an initial review by assigning it to a reviewer from your group or team. +you should [request an initial review](../user/project/merge_requests/getting_started.md#reviewer) by selecting a reviewer from your group or team. However, you can also assign it to any reviewer. The list of reviewers can be found on [Engineering projects](https://about.gitlab.com/handbook/engineering/projects/) page. You can also use `workflow::ready for review` label. That means that your merge request is ready to be reviewed and any reviewer can pick it. It is recommended to use that label only if there isn't time pressure and make sure the merge request is assigned to a reviewer. diff --git a/doc/development/contributing/index.md b/doc/development/contributing/index.md index fe7dc52d077..ec05f75c709 100644 --- a/doc/development/contributing/index.md +++ b/doc/development/contributing/index.md @@ -129,7 +129,7 @@ The general flow of contributing to GitLab is: 1. In the merge request's description: - Ensure you provide complete and accurate information. - Review the provided checklist. -1. Assign the merge request (if possible) to, or [mention](../../user/project/issues/issue_data_and_actions.md#mentions), +1. Assign the merge request (if possible) to, or [mention](../../user/discussions/index.md#mentions), one of the [code owners](../../user/project/code_owners.md) for the relevant project, and explain that you are ready for review. diff --git a/doc/development/contributing/merge_request_workflow.md b/doc/development/contributing/merge_request_workflow.md index 82fd62d8d79..cc6997e1a20 100644 --- a/doc/development/contributing/merge_request_workflow.md +++ b/doc/development/contributing/merge_request_workflow.md @@ -100,7 +100,7 @@ request is as follows: There isn't a way to know anything about our customers' data on their [self-managed instances](../../subscriptions/self_managed/index.md), so keep that in mind for any data implications with your merge request. - + 1. Merge requests **must** adhere to the [merge request performance guidelines](../merge_request_performance_guidelines.md). 1. For tests that use Capybara, read [how to write reliable, asynchronous integration tests](https://thoughtbot.com/blog/write-reliable-asynchronous-integration-tests-with-capybara). diff --git a/doc/development/contributing/style_guides.md b/doc/development/contributing/style_guides.md index 754e6c7aec6..fdb6e99fdcd 100644 --- a/doc/development/contributing/style_guides.md +++ b/doc/development/contributing/style_guides.md @@ -171,7 +171,8 @@ we should track our progress through the exception list. When auto-generating the `.rubocop_todo.yml` exception list for a particular Cop, and more than 15 files are affected, we should add the exception list to -a different file, `.rubocop_manual_todo.yml`. +a different file in the directory `.rubocop_todo/`. For example, the configuration for the cop +`Gitlab/NamespacedClass` is in `.rubocop_todo/gitlab/namespaced_class.yml`. This ensures that our list isn't mistakenly removed by another auto generation of the `.rubocop_todo.yml`. This also allows us greater visibility into the exceptions @@ -184,19 +185,19 @@ bundle exec rake rubocop:todo:generate ``` You can then move the list from the freshly generated `.rubocop_todo.yml` for the Cop being actively -resolved and place it in the `.rubocop_manual_todo.yml`. In this scenario, do not commit auto generated -changes to the `.rubocop_todo.yml` as an `exclude limit` that is higher than 15 will make the -`.rubocop_todo.yml` hard to parse. +resolved and place it in the directory `.rubocop_todo/`. In this scenario, do not commit +auto-generated changes to the `.rubocop_todo.yml`, as an `exclude limit` that is higher than 15 +makes the `.rubocop_todo.yml` hard to parse. ### Reveal existing RuboCop exceptions To reveal existing RuboCop exceptions in the code that have been excluded via `.rubocop_todo.yml` and -`.rubocop_manual_todo.yml`, set the environment variable `REVEAL_RUBOCOP_TODO` to `1`. +`.rubocop_todo/**/*.yml`, set the environment variable `REVEAL_RUBOCOP_TODO` to `1`. This allows you to reveal existing RuboCop exceptions during your daily work cycle and fix them along the way. NOTE: -Permanent `Exclude`s should be defined in `.rubocop.yml` instead of `.rubocop_manual_todo.yml`. +Define permanent `Exclude`s in `.rubocop.yml` instead of `.rubocop_todo/**/*.yml`. ## Database migrations diff --git a/doc/development/dangerbot.md b/doc/development/dangerbot.md index 829f6af76be..374e4e5de68 100644 --- a/doc/development/dangerbot.md +++ b/doc/development/dangerbot.md @@ -131,7 +131,7 @@ The main `Dangerfile` will then take care of adding the labels to the MR with a #### Shared rules and plugins If the rule or plugin you implement can be useful for other projects, think about -upstreaming them to the [`gitlab-org/gitlab-dangerfiles`](https://gitlab.com/gitlab-org/gitlab-dangerfiles) project. +upstreaming them to the [`gitlab-dangerfiles`](https://gitlab.com/gitlab-org/ruby/gems/gitlab-dangerfiles) project. #### Enable Danger on a project diff --git a/doc/development/database/loose_foreign_keys.md b/doc/development/database/loose_foreign_keys.md index 157c1284512..c60989f225d 100644 --- a/doc/development/database/loose_foreign_keys.md +++ b/doc/development/database/loose_foreign_keys.md @@ -52,25 +52,40 @@ For this procedure to work, we must register which tables to clean up asynchrono ## Example migration and configuration -### Configure the model +### Configure the loose foreign key -First, tell the application that the `projects` table has a new loose foreign key. -You can do this in the `Project` model: +Loose foreign keys are defined in a YAML file. The configuration requires the +following information: -```ruby -class Project < ApplicationRecord - # ... +- Parent table name (`projects`) +- Child table name (`ci_pipelines`) +- The data cleanup method (`async_delete` or `async_nullify`) - include LooseForeignKey +The YAML file is located at `lib/gitlab/database/gitlab_loose_foreign_keys.yml`. The file groups +foreign key definitions by the name of the child table. The child table can have multiple loose +foreign key definitions, therefore we store them as an array. - loose_foreign_key :ci_pipelines, :project_id, on_delete: :async_delete # or async_nullify +Example definition: - # ... -end +```yaml +ci_pipelines: + - table: projects + column: project_id + on_delete: async_delete ``` -This instruction ensures the asynchronous cleanup process knows about the association, and the -how to do the cleanup. In this case, the associated `ci_pipelines` records are deleted. +If the `ci_pipelines` key is already present in the YAML file, then a new entry can be added +to the array: + +```yaml +ci_pipelines: + - table: projects + column: project_id + on_delete: async_delete + - table: another_table + column: another_id + on_delete: :async_nullify +``` ### Track record changes @@ -127,6 +142,19 @@ end At this point, the setup phase is concluded. The deleted `projects` records should be automatically picked up by the scheduled cleanup worker job. +## Testing + +The "`it has loose foreign keys`" shared example can be used to test the presence of the `ON DELETE` trigger and the +loose foreign key definitions. + +Simply add to the model test file: + +```ruby +it_behaves_like 'it has loose foreign keys' do + let(:factory_name) { :project } +end +``` + ## Caveats of loose foreign keys ### Record creation @@ -180,3 +208,90 @@ end NOTE: This example is unlikely in GitLab, because we usually look up the parent models to perform permission checks. + +## A note on `dependent: :destroy` and `dependent: :nullify` + +We considered using these Rails features as an alternative to foreign keys but there are several problems which include: + +1. These run on a different connection in the context of a transaction [which we do not allow](multiple_databases.md#removing-cross-database-transactions). +1. These can lead to severe performance degradation as we load all records from PostgreSQL, loop over them in Ruby, and call individual `DELETE` queries. +1. These can miss data as they only cover the case when the `destroy` method is called directly on the model. There are other cases including `delete_all` and cascading deletes from another parent table that could mean these are missed. + +## Risks of loose foreign keys and possible mitigations + +In general, the loose foreign keys architecture is eventually consistent and +the cleanup latency might lead to problems visible to GitLab users or +operators. We consider the tradeoff as acceptable, but there might be +cases where the problems are too frequent or too severe, and we must +implement a mitigation strategy. A general mitigation strategy might be to have +an "urgent" queue for cleanup of records that have higher impact with a delayed +cleanup. + +Below are some more specific examples of problems that might occur and how we +might mitigate them. In all the listed cases we might still consider the problem +described to be low risk and low impact, and in that case we would choose to not +implement any mitigation. + +### The record should be deleted but it shows up in a view + +This hypothetical example might happen with a foreign key like: + +```sql +ALTER TABLE ONLY vulnerability_occurrence_pipelines + ADD CONSTRAINT fk_rails_6421e35d7d FOREIGN KEY (pipeline_id) REFERENCES ci_pipelines(id) ON DELETE CASCADE; +``` + +In this example we expect to delete all associated `vulnerability_occurrence_pipelines` records +whenever we delete the `ci_pipelines` record associated with them. In this case +you might end up with some vulnerability page in GitLab which shows an occurrence +of a vulnerability. However, when you try to click a link to the pipeline, you get +a 404, because the pipeline is deleted. Then, when you navigate back you might find the +occurrence has disappeared too. + +**Mitigation** + +When rendering the vulnerability occurrences on the vulnerability page we could +try to load the corresponding pipeline and choose to skip displaying that +occurrence if pipeline is not found. + +### The deleted parent record is needed to render a view and causes a `500` error + +This hypothetical example might happen with a foreign key like: + +```sql +ALTER TABLE ONLY vulnerability_occurrence_pipelines + ADD CONSTRAINT fk_rails_6421e35d7d FOREIGN KEY (pipeline_id) REFERENCES ci_pipelines(id) ON DELETE CASCADE; +``` + +In this example we expect to delete all associated `vulnerability_occurrence_pipelines` records +whenever we delete the `ci_pipelines` record associated with them. In this case +you might end up with a vulnerability page in GitLab which shows an "occurrence" +of a vulnerability. However, when rendering the occurrence we try to load, for example, +`occurrence.pipeline.created_at`, which causes a 500 for the user. + +**Mitigation** + +When rendering the vulnerability occurrences on the vulnerability page we could +try to load the corresponding pipeline and choose to skip displaying that +occurrence if pipeline is not found. + +### The deleted parent record is accessed in a Sidekiq worker and causes a failed job + +This hypothetical example might happen with a foreign key like: + +```sql +ALTER TABLE ONLY vulnerability_occurrence_pipelines + ADD CONSTRAINT fk_rails_6421e35d7d FOREIGN KEY (pipeline_id) REFERENCES ci_pipelines(id) ON DELETE CASCADE; +``` + +In this example we expect to delete all associated `vulnerability_occurrence_pipelines` records +whenever we delete the `ci_pipelines` record associated with them. In this case +you might end up with a Sidekiq worker that is responsible for processing a +vulnerability and looping over all occurrences causing a Sidekiq job to fail if +it executes `occurrence.pipeline.created_at`. + +**Mitigation** + +When looping through the vulnerability occurrences in the Sidekiq worker, we +could try to load the corresponding pipeline and choose to skip processing that +occurrence if pipeline is not found. diff --git a/doc/development/database/multiple_databases.md b/doc/development/database/multiple_databases.md index a17ad798305..1338e83070f 100644 --- a/doc/development/database/multiple_databases.md +++ b/doc/development/database/multiple_databases.md @@ -15,10 +15,46 @@ To scale GitLab, the we are database for CI/CD tables was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/64289) in GitLab 14.1. This feature is still under development, and is not ready for production use. +### Development setup + By default, GitLab is configured to use only one main database. To opt-in to use a main database, and CI database, modify the `config/database.yml` file to have a `main` and a `ci` database -configurations. For example, given a `config/database.yml` like below: +configurations. + +You can set this up using [GDK](#gdk-configuration) or by +[manually configuring `config/database.yml`](#manually-set-up-the-cicd-database). + +#### GDK configuration + +If you are using GDK, you can follow the following steps: + +1. On the GDK root directory, run: + + ```shell + gdk config set gitlab.rails.multiple_databases true + ``` + +1. Open your `gdk.yml`, and confirm that it has the following lines: + + ```yaml + gitlab: + rails: + multiple_databases: true + ``` + +1. Reconfigure GDK: + + ```shell + gdk reconfigure + ``` + +1. [Create the new CI/CD database](#create-the-new-database). + +#### Manually set up the CI/CD database + +You can manually edit `config/database.yml` to split the databases. +To do so, consider a `config/database.yml` file like the example below: ```yaml development: @@ -44,7 +80,7 @@ test: &test statement_timeout: 120s ``` -Edit the `config/database.yml` to look like this: +Edit it to split the databases into `main` and `ci`: ```yaml development: @@ -88,6 +124,25 @@ test: &test statement_timeout: 120s ``` +Next, [create the new CI/CD database](#create-the-new-database). + +#### Create the new database + +After configuring GitLab for the two databases, create the new CI/CD database: + +1. Create the new `ci:` database, load the DB schema into the `ci:` database, + and run any pending migrations: + + ```shell + bundle exec rails db:create db:schema:load:ci db:migrate + ``` + +1. Restart GDK: + + ```shell + gdk restart + ``` + <!-- NOTE: The `validate_cross_joins!` method in `spec/support/database/prevent_cross_joins.rb` references the following heading in the code, so if you make a change to this heading, make sure to update @@ -557,11 +612,31 @@ Don't hesitate to reach out to the [sharding group](https://about.gitlab.com/handbook/engineering/development/enablement/sharding/) for advice. +##### Avoid `dependent: :nullify` and `dependent: :destroy` across databases + +There may be cases where we want to use `dependent: :nullify` or `dependent: :destroy` +across databases. This is technically possible, but it's problematic because +these hooks run in the context of an outer transaction from the call to +`#destroy`, which creates a cross-database transaction and we are trying to +avoid that. Cross-database transactions caused this way could lead to confusing +outcomes when we switch to decomposed, because now you have some queries +happening outside the transaction and they may be partially applied while the +outer transaction fails, which could lead to surprising bugs. + +If you need to do some cleanup after a `destroy` you will need to choose +from some of the options above. If all you need to do is cleanup the child +records themselves from PostgreSQL then you could consider using ["loose foreign +keys"](loose_foreign_keys.md). + ## `config/database.yml` -GitLab will support running multiple databases in the future, for example to [separate tables for the continuous integration features](https://gitlab.com/groups/gitlab-org/-/epics/6167) from the main database. In order to prepare for this change, we [validate the structure of the configuration](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67877) in `database.yml` to ensure that only known databases are used. +GitLab is adding support to run multiple databases, for example to +[separate tables for the continuous integration features](https://gitlab.com/groups/gitlab-org/-/epics/6167) +from the main database. In order to prepare for this change, we +[validate the structure of the configuration](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67877) +in `database.yml` to ensure that only known databases are used. -Previously, the `config/database.yml` would look like this: +Previously, the `config/database.yml` looked like this: ```yaml production: @@ -571,15 +646,16 @@ production: ... ``` -With the support for many databases the support for this -syntax is deprecated and will be removed in [15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/338182). +With the support for many databases this +syntax is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/338182) +and will be removed in [15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/338182). The new `config/database.yml` needs to include a database name to define a database configuration. Only `main:` and `ci:` database -names are supported today. The `main:` needs to always be a first +names are supported. The `main:` database must always be a first entry in a hash. This change applies to decomposed and non-decomposed -change. If an invalidate or deprecated syntax is used the error -or warning will be printed during application start. +change. If an invalid or deprecated syntax is used the error +or warning is printed during application start. ```yaml # Non-decomposed database @@ -603,3 +679,15 @@ production: database: gitlabhq_production_ci ... ``` + +## Foreign keys that cross databases + +There are many places where we use foreign keys that reference across the two +databases. This is not possible to do with two separate PostgreSQL +databases, so we need to replicate the behavior we get from PostgreSQL in a +performant way. We can't, and shouldn't, try to replicate the data guarantees +given by PostgreSQL which prevent creating invalid references, but we still need a +way to replace cascading deletes so we don't end up with orphaned data +or records that point to nowhere, which might lead to bugs. As such we created +["loose foreign keys"](loose_foreign_keys.md) which is an asynchronous +process of cleaning up orphaned records. diff --git a/doc/development/database_debugging.md b/doc/development/database_debugging.md index 7c17a39746e..426d355bd82 100644 --- a/doc/development/database_debugging.md +++ b/doc/development/database_debugging.md @@ -115,7 +115,7 @@ Use these instructions for exploring the GitLab database while developing with t 1. **Use an SSL connection?** This depends on your installation. Options are: - **Use Secure Connection** - **Standard Connection** (default) - 1. **(Optional) The database to connect to**: `gitlabhq_development`. + 1. **Optional. The database to connect to**: `gitlabhq_development`. 1. **The display name for the database connection**: `gitlabhq_development`. Your database connection should now be displayed in the PostgreSQL Explorer pane and diff --git a/doc/development/deprecation_guidelines/index.md b/doc/development/deprecation_guidelines/index.md index f8ee29e6904..27c29a1ed7c 100644 --- a/doc/development/deprecation_guidelines/index.md +++ b/doc/development/deprecation_guidelines/index.md @@ -23,7 +23,9 @@ deprecated. A feature can be deprecated at any time, provided there is a viable alternative. -Deprecations should be announced via [release posts](https://about.gitlab.com/handbook/marketing/blog/release-posts/#deprecations). +Deprecations should be announced on the [Deprecated feature removal schedule](../../update/deprecations.md). + +For steps to create a deprecation entry, see [Deprecations](https://about.gitlab.com/handbook/marketing/blog/release-posts/#deprecations). ## When can a feature be removed/changed? diff --git a/doc/development/documentation/feature_flags.md b/doc/development/documentation/feature_flags.md index 8fc6f2e2641..6a618e47211 100644 --- a/doc/development/documentation/feature_flags.md +++ b/doc/development/documentation/feature_flags.md @@ -58,7 +58,7 @@ FLAG: If needed, you can add this sentence: -`You should not use this feature for production environments.` +`The feature is not ready for production use.` ## Add version history text @@ -109,7 +109,7 @@ And, when the feature is done and fully available to all users: ```markdown > - Introduced in GitLab 13.7 [with a flag](../../administration/feature_flags.md) named `forti_token_cloud`. Disabled by default. -> - [Enabled on self-managed](https://gitlab.com/issue/etc) GitLab 13.8. +> - [Enabled on self-managed](https://gitlab.com/issue/etc) in GitLab 13.8. > - [Enabled on GitLab.com](https://gitlab.com/issue/etc) in GitLab 13.9. > - [Generally available](issue-link) in GitLab 14.0. [Feature flag <flag name>](issue-link) removed. ``` diff --git a/doc/development/documentation/img/manual_build_docs_v14_3.png b/doc/development/documentation/img/manual_build_docs_v14_3.png Binary files differdeleted file mode 100644 index e366a2f7ec4..00000000000 --- a/doc/development/documentation/img/manual_build_docs_v14_3.png +++ /dev/null diff --git a/doc/development/documentation/img/manual_build_docs_v14_6.png b/doc/development/documentation/img/manual_build_docs_v14_6.png Binary files differnew file mode 100644 index 00000000000..731bda3dd56 --- /dev/null +++ b/doc/development/documentation/img/manual_build_docs_v14_6.png diff --git a/doc/development/documentation/restful_api_styleguide.md b/doc/development/documentation/restful_api_styleguide.md index 03980a42381..5dc627c93e9 100644 --- a/doc/development/documentation/restful_api_styleguide.md +++ b/doc/development/documentation/restful_api_styleguide.md @@ -1,9 +1,6 @@ --- -type: reference, dev -stage: none -group: Development -info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" -description: "Writing styles, markup, formatting, and other standards for the GitLab RESTful APIs." +info: For assistance with this Style Guide page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-other-projects-and-subjects. +description: 'Writing styles, markup, formatting, and other standards for the GitLab RESTful APIs.' --- # RESTful API @@ -30,6 +27,10 @@ In the Markdown doc for a resource (AKA endpoint): - Every method must have a detailed [description of the parameters](#method-description). - Every method must have a cURL example. - Every method must have a response body (in JSON format). +- If an attribute is available only to higher level tiers than the other + parameters, add the appropriate inline [tier badge](styleguide/index.md#product-tier-badges). + Put the badge in the **Attribute** column, like the + `**(<tier>)**` code in the following template. ## API topic template @@ -49,12 +50,12 @@ METHOD /endpoint Supported attributes: -| Attribute | Type | Required | Description | -| :---------- | :------- | :--------------------- | :-------------------- | -| `attribute` | datatype | **{check-circle}** Yes | Detailed description. | -| `attribute` | datatype | **{dotted-circle}** No | Detailed description. | -| `attribute` | datatype | **{dotted-circle}** No | Detailed description. | -| `attribute` | datatype | **{dotted-circle}** No | Detailed description. | +| Attribute | Type | Required | Description | +|:-------------------------|:---------|:-----------------------|:----------------------| +| `attribute` | datatype | **{check-circle}** Yes | Detailed description. | +| `attribute` **(<tier>)** | datatype | **{dotted-circle}** No | Detailed description. | +| `attribute` | datatype | **{dotted-circle}** No | Detailed description. | +| `attribute` | datatype | **{dotted-circle}** No | Detailed description. | Example request: @@ -83,20 +84,20 @@ always be in code blocks using backticks (`` ` ``). Sort the attributes in the table: first, required, then alphabetically. ```markdown -| Attribute | Type | Required | Description | -| :------------- | :------------ | :--------------------- | :--------------------------------------------------- | -| `user` | string | **{check-circle}** Yes | The GitLab username. | -| `assignee_ids` | integer array | **{dotted-circle}** No | The IDs of the users to assign the issue to. | -| `confidential` | boolean | **{dotted-circle}** No | Set an issue to be confidential. Default is `false`. | +| Attribute | Type | Required | Description | +|:-----------------------------|:--------------|:-----------------------|:-----------------------------------------------------| +| `user` | string | **{check-circle}** Yes | The GitLab username. | +| `assignee_ids` **(PREMIUM)** | integer array | **{dotted-circle}** No | The IDs of the users to assign the issue to. | +| `confidential` | boolean | **{dotted-circle}** No | Set an issue to be confidential. Default is `false`. | ``` Rendered example: -| Attribute | Type | Required | Description | -| :------------- | :------------ | :--------------------- | :--------------------------------------------------- | -| `user` | string | **{check-circle}** Yes | The GitLab username. | -| `assignee_ids` | integer array | **{dotted-circle}** No | The IDs of the users to assign the issue to. | -| `confidential` | boolean | **{dotted-circle}** No | Set an issue to be confidential. Default is `false`. | +| Attribute | Type | Required | Description | +|:-----------------------------|:--------------|:-----------------------|:-----------------------------------------------------| +| `user` | string | **{check-circle}** Yes | The GitLab username. | +| `assignee_ids` **(PREMIUM)** | integer array | **{dotted-circle}** No | The IDs of the users to assign the issue to. | +| `confidential` | boolean | **{dotted-circle}** No | Set an issue to be confidential. Default is `false`. | ## cURL commands @@ -109,7 +110,7 @@ Rendered example: username and password. | Methods | Description | -| :---------------------------------------------- | :----------------------------------------------------- | +|:------------------------------------------------|:-------------------------------------------------------| | `--header "PRIVATE-TOKEN: <your_access_token>"` | Use this method as is, whenever authentication needed. | | `--request POST` | Use this method when creating new objects | | `--request PUT` | Use this method when updating existing objects | diff --git a/doc/development/documentation/review_apps.md b/doc/development/documentation/review_apps.md index a5094ea87f0..4b58778a20c 100644 --- a/doc/development/documentation/review_apps.md +++ b/doc/development/documentation/review_apps.md @@ -26,7 +26,7 @@ to render and preview the documentation locally. If a merge request has documentation changes, use the `review-docs-deploy` manual job to deploy the documentation review app for your merge request. - + The `review-docs-deploy*` job triggers a cross project pipeline and builds the docs site with your changes. When the pipeline finishes, the review app URL diff --git a/doc/development/documentation/site_architecture/deployment_process.md b/doc/development/documentation/site_architecture/deployment_process.md index c038ee96dbf..25bc699c9d4 100644 --- a/doc/development/documentation/site_architecture/deployment_process.md +++ b/doc/development/documentation/site_architecture/deployment_process.md @@ -50,8 +50,8 @@ that stable documentation and deploys it to the registry. For example: - [13.12 merge request pipeline](https://gitlab.com/gitlab-org/gitlab-docs/-/pipelines/395365202). - [12.10 merge request pipeline](https://gitlab.com/gitlab-org/gitlab-docs/-/pipelines/395365405). -In particular, the [`image:docs-single` job](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/4c18963fe0a414ad62f55b9e18f922588b2dd155/.gitlab-ci.yml#L655) in each pipeline -takes what is built, and pushes it to the [container registry](https://gitlab.com/gitlab-org/gitlab-docs/container_registry/631635). +In particular, the [`image:docs-single` job](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/4c18963fe0a414ad62f55b9e18f922588b2dd155/.gitlab-ci.yml#L655) in each pipeline runs automatically. +It takes what is built, and pushes it to the [container registry](https://gitlab.com/gitlab-org/gitlab-docs/container_registry/631635). ```mermaid graph TD @@ -91,6 +91,7 @@ The [`image:docs-latest` job](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/4 - Pulls the latest documentation from the default branches of the relevant upstream projects. - Pulls the Docker images previously built by the `image:docs-single` jobs. +- Must be run manually on a scheduled pipeline. For example, [a pipeline](https://gitlab.com/gitlab-org/gitlab-docs/-/pipelines/399233948) containing the [`image:docs-latest` job](https://gitlab.com/gitlab-org/gitlab-docs/-/jobs/1733948330): @@ -122,7 +123,8 @@ graph TD for it must be deployed to become available. The [`pages`](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/4c18963fe0a414ad62f55b9e18f922588b2dd155/.gitlab-ci.yml#L491) -job runs the necessary commands to combine: +job runs automatically when a pipeline runs on the default branch (`main`). +It runs the necessary commands to combine: - A very up-to-date build of the `gitlab-docs` site code. - The latest docs from the default branches of the upstream projects. diff --git a/doc/development/documentation/structure.md b/doc/development/documentation/structure.md index fac83af89f4..6ecffce01b4 100644 --- a/doc/development/documentation/structure.md +++ b/doc/development/documentation/structure.md @@ -88,7 +88,7 @@ Create an issue when you want to track bugs or future work. Prerequisites: -- You must have at least the Developer role for a project. +- You must have at least the Developer role for the project. To create an issue: diff --git a/doc/development/documentation/styleguide/img/callouts.png b/doc/development/documentation/styleguide/img/callouts.png Binary files differnew file mode 100644 index 00000000000..b84e9e269dc --- /dev/null +++ b/doc/development/documentation/styleguide/img/callouts.png diff --git a/doc/development/documentation/styleguide/index.md b/doc/development/documentation/styleguide/index.md index 1382ec263f2..7f2e5a1ba26 100644 --- a/doc/development/documentation/styleguide/index.md +++ b/doc/development/documentation/styleguide/index.md @@ -58,8 +58,7 @@ it was originally composed for, if it is helpful to any of our audiences, we can include it. - If you use an image that has a separate source file (for example, a vector or - diagram format), link the image to the source file so that it may be reused or - updated by anyone. + diagram format), link the image to the source file so that anyone can update or reuse it. - Do not copy and paste content from other sources unless it is a limited quotation with the source cited. Typically it is better to either rephrase relevant information in your own words or link out to the other source. @@ -113,8 +112,9 @@ The more we reflexively add information to the documentation, the more the documentation helps others efficiently accomplish tasks and solve problems. If you have questions when considering, authoring, or editing documentation, ask -the Technical Writing team. They're available on Slack in `#docs` or in GitLab by mentioning the -writer for the applicable [DevOps stage](https://about.gitlab.com/handbook/product/categories/#devops-stages). +the Technical Writing team. They're available on Slack in `#docs` or in GitLab by +mentioning [the writer for](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments) +the applicable [DevOps stage or group](https://about.gitlab.com/handbook/product/categories/#devops-stages). Otherwise, forge ahead with your best effort. It does not need to be perfect; the team is happy to review and improve upon your content. Review the [Documentation guidelines](index.md) before you begin your first documentation MR. @@ -787,10 +787,13 @@ This is overridden by the [documentation-specific punctuation rules](#punctuatio - When possible, avoid including words that might change in the future. Changing a heading changes its anchor URL, which affects other linked pages. - When introducing a new document, be careful for the headings to be - grammatically and syntactically correct. Mention an [assigned technical writer (TW)](https://about.gitlab.com/handbook/product/categories/) - for review. + grammatically and syntactically correct. Mention an [assigned technical writer (TW)](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments) + for review, based upon the [product category](https://about.gitlab.com/handbook/product/categories/). This is to ensure that no document with wrong heading is going live without an audit, thus preventing dead links and redirection issues when corrected. +- Use the context provided by parent section headings. That is, don't repeat the parent heading's text in each + subsection's heading. +- Use articles and prepositions in headings where it would make sense in regular text. - Leave exactly one blank line before and after a heading. - Do not use links in headings. - Add the corresponding [product badge](#product-tier-badges) according to the tier the @@ -1053,7 +1056,9 @@ Guidance for each individual UI element is in [the word list](word_list.md). ### How to write navigation task steps -To be consistent, use this format when you write navigation steps in a task topic. +To be consistent, use these templates when you write navigation steps in a task topic. + +To open project settings: ```markdown 1. On the top bar, select **Menu > Projects** and find your project. @@ -1061,7 +1066,7 @@ To be consistent, use this format when you write navigation steps in a task topi 1. Expand **General pipelines**. ``` -Another example: +To open group settings: ```markdown 1. On the top bar, select **Menu > Groups** and find your group. @@ -1069,7 +1074,7 @@ Another example: 1. Expand **General pipelines**. ``` -An Admin Area example: +To open the Admin Area: ```markdown 1. On the top bar, select **Menu > Admin**. @@ -1081,6 +1086,15 @@ To select your avatar: 1. On the top bar, in the top right corner, select your avatar. ``` +To save the selection in some dropdown lists: + +```markdown +1. Go to your issue. +1. On the right sidebar, in the **Iteration** section, select **Edit**. +1. From the dropdown list, select the iteration to associate this issue with. +1. Select any area outside the dropdown list. +``` + ### Optional steps If a step is optional, start the step with the word `Optional` followed by a period. @@ -1150,6 +1164,17 @@ review app in the merge request. Make sure the image isn't blurry or overwhelmin - **Be consistent.** Coordinate screenshots with the other screenshots already on a documentation page for a consistent reading experience. +### Add callouts + +If you need to emphasize an area in a screenshot, use an arrow. + +- For color, use `#EE2604`. If you use the Preview application on macOS, this is the default red. +- For the line width, use 3 pt. If you use the Preview application on macOS, this is the third line in the list. +- Use the arrow style shown in the following image. +- If you have multiple arrows, make them parallel when possible. + + + ### Save the image - Resize any wide or tall screenshots if needed, but make sure the screenshot is @@ -1575,7 +1600,7 @@ users be aware of recent improvements or additions. The GitLab Technical Writing team determines which versions of documentation to display on this site based on the GitLab -[Statement of Support](https://about.gitlab.com/support/statement-of-support.html#we-support-the-current-major-version-and-the-two-previous-major-versions). +[Statement of Support](https://about.gitlab.com/support/statement-of-support.html#version-support). ### View older GitLab documentation versions @@ -1628,6 +1653,13 @@ This feature does something. This feature does something else. ``` +If you're documenting elements of a feature, start with the feature name or a gerund: + +```markdown +> - Notifications for expiring tokens [introduced](<link-to-issue>) in GitLab 11.3. +> - Creating an issue from an issue board [introduced](<link-to-issue>) in GitLab 13.1. +``` + If a feature is moved to another tier: ```markdown @@ -1781,7 +1813,9 @@ after the heading text. For example: # Heading title **(FREE)** ``` -Do not add tier badges inline with other text. The single source of truth for a feature should be the heading where the functionality is described. +Do not add tier badges inline with other text, except for [API attributes](../restful_api_styleguide.md). +The single source of truth for a feature should be the heading where the +functionality is described. #### Available product tier badges diff --git a/doc/development/documentation/styleguide/word_list.md b/doc/development/documentation/styleguide/word_list.md index 595dab09bf5..9c375379685 100644 --- a/doc/development/documentation/styleguide/word_list.md +++ b/doc/development/documentation/styleguide/word_list.md @@ -5,9 +5,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w description: 'Writing styles, markup, formatting, and other standards for GitLab Documentation.' --- -# A-Z word list +# Recommended word list -To help ensure consistency in the documentation, follow this guidance. +To help ensure consistency in the documentation, the Technical Writing team +recommends these wording choices. The GitLab handbook also maintains a list of +[top misused terms](https://about.gitlab.com/handbook/communication/top-misused-terms/). For guidance not on this page, we defer to these style guides: @@ -20,7 +22,7 @@ For guidance not on this page, we defer to these style guides: ## `@mention` Try to avoid **`@mention`**. Say **mention** instead, and consider linking to the -[mentions topic](../../../user/project/issues/issue_data_and_actions.md#mentions). +[mentions topic](../../../user/discussions/index.md#mentions). Don't use backticks. ## above @@ -31,9 +33,14 @@ Try to avoid using **above** when referring to an example or table in a document Do not use **above** when referring to versions of the product. Use [**later**](#later) instead. -- Do: In GitLab 14.4 and later... -- Do not: In GitLab 14.4 and above... -- Do not: In GitLab 14.4 and higher... +Use: + +- In GitLab 14.4 and later... + +Instead of: + +- In GitLab 14.4 and above... +- In GitLab 14.4 and higher... ## access level @@ -47,16 +54,21 @@ Capitalize these words when you refer to the UI. Otherwise use lowercase. Use **administrator** instead of **admin** when talking about a user's access level. Use lowercase unless you are referring to the **Admin** access level you select in the UI. -To view the administrator access type, in the GitLab UI, go to the Admin Area and select +To view the administrator access level, in the GitLab UI, go to the Admin Area and select **Users**. Then select **New user**.  An **administrator** is not a [role](#roles) or [permission](#permissions). -- Do: To do this thing, you must be an administrator. -- Do: To do this thing, you must have the administrator access level. -- Do not: To do this thing, you must have the Admin role. +Use: + +- To do this thing, you must be an administrator. +- To do this thing, you must have the administrator access level. + +Instead of: + +- To do this thing, you must have the Admin role. ## Admin Area @@ -65,10 +77,16 @@ This area of the UI says **Admin Area** at the top of the page and on the menu. ## allow, enable -Try to avoid **allow** and **enable**, unless you are talking about security-related features. For example: +Try to avoid **allow** and **enable**, unless you are talking about security-related features. -- Do: Use this feature to create a pipeline. -- Do not: This feature allows you to create a pipeline. +Use: + +- You can add a file to your repository. + +Instead of: + +- This feature allows you to add a file to your repository. +- This feature enables users to add files to their repository. This phrasing is more active and is from the user perspective, rather than the person who implemented the feature. [View details in the Microsoft style guide](https://docs.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/a/allow-allows). @@ -124,8 +142,13 @@ Use **text box** to refer to the UI field. Do not use **field** or **box**. For Don't use a descriptor with **button**. -- Do: Select **Run pipelines**. -- Do not: Select the **Run pipelines** button. +Use: + +- Select **Run pipelines**. + +Instead of: + +- Select the **Run pipelines** button. ## cannot, can not @@ -148,6 +171,13 @@ If you must refer to the checkbox, you can say it is selected or cleared. For ex - Ensure the **Protect environment** checkbox is cleared. - Ensure the **Protect environment** checkbox is selected. +## checkout, check out + +Use **check out** as a verb. For the Git command, use `checkout`. + +- Use `git checkout` to check out a branch locally. +- Check out the files you want to edit. + ## CI/CD CI/CD is always uppercase. No need to spell it out on first use. @@ -185,8 +215,8 @@ When writing about the Developer role: - Do not use the phrase, **if you are a developer** to mean someone who is assigned the Developer role. Instead, write it out. For example, **if you are assigned the Developer role**. - To describe a situation where the Developer role is the minimum required: - - Do: at least the Developer role - - Do not: the Developer role or higher + - Use: at least the Developer role + - Instead of: the Developer role or higher Do not use **Developer permissions**. A user who is assigned the Developer role has a set of associated permissions. @@ -208,8 +238,13 @@ For example: Use **earlier** when talking about version numbers. -- Do: In GitLab 14.1 and earlier. -- Do not: In GitLab 14.1 and lower. +Use: + +- In GitLab 14.1 and earlier. + +Instead of: + +- In GitLab 14.1 and lower. ## easily @@ -245,8 +280,13 @@ Use lowercase for **epic board**. Try to avoid **etc.**. Be as specific as you can. Do not use [**and so on**](#and-so-on) as a replacement. -- Do: You can update objects, like merge requests and issues. -- Do not: You can update objects, like merge requests, issues, etc. +Use: + +- You can update objects, like merge requests and issues. + +Instead of: + +- You can update objects, like merge requests, issues, etc. ## expand @@ -256,8 +296,13 @@ Use **expand** instead of **open** when you are talking about expanding or colla Use **box** instead of **field** or **text box**. -- Do: In the **Variable name** box, enter `my text`. -- Do not: In the **Variable name** field, enter `my text`. +Use: + +- In the **Variable name** box, enter `my text`. + +Instead of: + +- In the **Variable name** field, enter `my text`. However, you can make an exception when you are writing a task and you need to refer to all of the fields at once. For example: @@ -311,8 +356,8 @@ When writing about the Guest role: - Do not use the phrase, **if you are a guest** to mean someone who is assigned the Guest role. Instead, write it out. For example, **if you are assigned the Guest role**. - To describe a situation where the Guest role is the minimum required: - - Do: at least the Guest role - - Do not: the Guest role or higher + - Use: at least the Guest role + - Instead of: the Guest role or higher Do not use **Guest permissions**. A user who is assigned the Guest role has a set of associated permissions. @@ -328,16 +373,26 @@ Do not use **high availability** or **HA**. Instead, direct readers to the GitLa Do not use **higher** when talking about version numbers. -- Do: In GitLab 14.4 and later... -- Do not: In GitLab 14.4 and higher... -- Do not: In GitLab 14.4 and above... +Use: + +- In GitLab 14.4 and later... + +Instead of: + +- In GitLab 14.4 and higher... +- In GitLab 14.4 and above... ## hit Don't use **hit** to mean **press**. -- Do: Press **ENTER**. -- Do not: Hit the **ENTER** button. +Use: + +- Press **ENTER**. + +Instead of: + +- Hit the **ENTER** button. ## I @@ -369,13 +424,31 @@ Do not use **build** to be synonymous with **job**. A job is defined in the `.gi If you want to use **CI** with the word **job**, use **CI/CD job** rather than **CI job**. +## Kubernetes executor + +GitLab Runner can run jobs on a Kubernetes cluster. To do this, GitLab Runner uses the Kubernetes executor. + +When referring to this feature, use: + +- Kubernetes executor for GitLab Runner +- Kubernetes executor + +Do not use: + +- GitLab Runner Kubernetes executor, because this can infringe on the Kubernetes trademark. + ## later Use **later** when talking about version numbers. -- Do: In GitLab 14.1 and later... -- Do not: In GitLab 14.1 and higher... -- Do not: In GitLab 14.1 and above... +Use: + +- In GitLab 14.1 and later... + +Instead of: + +- In GitLab 14.1 and higher... +- In GitLab 14.1 and above... ## list @@ -390,8 +463,13 @@ Do not use **log in** or **log on**. Use [sign in](#sign-in) instead. If the use Do not use **lower** when talking about version numbers. -- Do: In GitLab 14.1 and earlier. -- Do not: In GitLab 14.1 and lower. +Use: + +- In GitLab 14.1 and earlier. + +Instead of: + +- In GitLab 14.1 and lower. ## Maintainer @@ -402,8 +480,8 @@ When writing about the Maintainer role: - Do not use the phrase, **if you are a maintainer** to mean someone who is assigned the Maintainer role. Instead, write it out. For example, **if you are assigned the Maintainer role**. - To describe a situation where the Maintainer role is the minimum required: - - Do: at least the Maintainer role - - Do not: the Maintainer role or higher + - Use: at least the Maintainer role + - Instead of: the Maintainer role or higher Do not use **Maintainer permissions**. A user who is assigned the Maintainer role has a set of associated permissions. @@ -446,8 +524,14 @@ Do not use **navigate**. Use **go** instead. For example: Try to avoid **needs to**, because it's wordy. Avoid **should** when you can be more specific. If something is required, use **must**. -- Do: You must set the variable. Or: Set the variable. -- Do not: You need to set the variable. +Use: + +- You must set the variable. +- Set the variable. + +Instead of: + +- You need to set the variable. **Should** is acceptable for recommended actions or items, or in cases where an event may not happen. For example: @@ -461,22 +545,63 @@ happen. For example: Do not use **note that** because it's wordy. -- Do: You can change the settings. -- Do not: Note that you can change the settings. +Use: + +- You can change the settings. + +Instead of: + +- Note that you can change the settings. ## on When documenting how to select high-level UI elements, use the word **on**. -- Do: `On the left sidebar...` +Use: + +- `On the left sidebar...` + +Instead of: + - Do not: `From the left sidebar...` or `In the left sidebar...` ## once The word **once** means **one time**. Don't use it to mean **after** or **when**. -- Do: When the process is complete... -- Do not: Once the process is complete... +Use: + +- When the process is complete... + +Instead of: + +- Once the process is complete... + +## only + +Put the word **only** next to the word it modifies. + +- You can create only private projects. + +In this example, **only** modifies the noun **projects**. The sentence means you can create one type of project--a private project. + +- You can only create private projects. + +In this example, **only** modifies the verb **create**. This sentence means that you can't perform other actions, +like deleting private projects, or adding users to them. + +## override + +Use **override** to indicate temporary replacement. + +For example, a value might be overridden when a job runs. The +original value does not change. + +## overwrite + +Use **overwrite** to indicate permanent replacement. + +For example, a log file might overwrite a log file of the same name. ## Owner @@ -518,8 +643,8 @@ When writing about the Reporter role: - Do not use the phrase, **if you are a reporter** to mean someone who is assigned the Reporter role. Instead, write it out. For example, **if you are assigned the Reporter role**. - To describe a situation where the Reporter role is the minimum required: - - Do: at least the Reporter role - - Do not: the Reporter role or higher + - Use: at least the Reporter role + - Instead of: the Reporter role or higher Do not use **Reporter permissions**. A user who is assigned the Reporter role has a set of associated permissions. @@ -541,8 +666,13 @@ Use lowercase for **runners**. These are the agents that run CI/CD jobs. See als Do not use **(s)** to make a word optionally plural. It can slow down comprehension. For example: -- Do: Select the jobs you want. -- Do not: Select the job(s) you want. +Use: + +- Select the jobs you want. + +Instead of: + +- Select the job(s) you want. If you can select multiples of something, then write the word as plural. @@ -564,7 +694,12 @@ into separate areas, refer to these areas as sections. We often think of expandable/collapsible areas as **sections**. When you refer to expanding or collapsing a section, don't include the word **section**. -- Do: Expand **Auto DevOps**. +Use: + +- Expand **Auto DevOps**. + +Instead of: + - Do not: Expand the **Auto DevOps** section. ## select @@ -597,8 +732,13 @@ Do not use **simply** or **simple**. If the user doesn't find the process to be The word **since** indicates a timeframe. For example, **Since 1984, Bon Jovi has existed**. Don't use **since** to mean **because**. -- Do: Because you have the Developer role, you can delete the widget. -- Do not: Since you have the Developer role, you can delete the widget. +Use: + +- Because you have the Developer role, you can delete the widget. + +Instead of: + +- Since you have the Developer role, you can delete the widget. ## slashes @@ -616,8 +756,13 @@ Use **subgroup** (no hyphen) instead of **sub-group**. ([Vale](../testing.md#val Do not use **that** when describing a noun. For example: -- Do: The file you save... -- Do not: The file **that** you save... +Use: + +- The file you save... + +Instead of: + +- The file **that** you save... See also [this, these, that, those](#this-these-that-those). @@ -636,8 +781,13 @@ Use **text box** instead of **field** or **box** when referring to the UI elemen Try to avoid **there is** and **there are**. These phrases hide the subject. -- Do: The bucket has holes. -- Do not: There are holes in the bucket. +Use: + +- The bucket has holes. + +Instead of: + +- There are holes in the bucket. ## they @@ -649,17 +799,17 @@ a gender-neutral pronoun. Always follow these words with a noun. For example: -- Do: **This setting** improves performance. -- Do not: **This** improves performance. +- Use: **This setting** improves performance. +- Instead of: **This** improves performance. -- Do: **These pants** are the best. -- Do not: **These** are the best. +- Use: **These pants** are the best. +- Instead of: **These** are the best. -- Do: **That droid** is the one you are looking for. -- Do not: **That** is the one you are looking for. +- Use: **That droid** is the one you are looking for. +- Instead of: **That** is the one you are looking for. -- Do: **Those settings** need to be configured. (Or even better, **Configure those settings.**) -- Do not: **Those** need to be configured. +- Use: **Those settings** need to be configured. (Or even better, **Configure those settings.**) +- Instead of: **Those** need to be configured. ## to-do item @@ -683,6 +833,19 @@ Do not use **type** if you can avoid it. Use **enter** instead. Do not use **useful**. If the user doesn't find the process to be useful, we lose their trust. ([Vale](../testing.md#vale) rule: [`Simplicity.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/Simplicity.yml)) +## user, users + +When possible, address the reader directly, instead of calling them **users**. +Use the [second person](#you-your-yours), **you**, instead. + +Use: + +- You can configure a pipeline. + +Instead of: + +- Users can configure a pipeline. + ## utilize Do not use **utilize**. Use **use** instead. It's more succinct and easier for non-native English speakers to understand. @@ -700,8 +863,13 @@ Do not use Latin abbreviations. Use **with**, **through**, or **by using** inste Try to avoid **we** and focus instead on how the user can accomplish something in GitLab. -- Do: Use widgets when you have work you want to organize. -- Do not: We created a feature for you to add widgets. +Use: + +- Use widgets when you have work you want to organize. + +Instead of: + +- We created a feature for you to add widgets. One exception: You can use **we recommend** instead of **it is recommended** or **GitLab recommends**. ([Vale](../testing.md#vale) rule: [`SubstitutionSuggestions.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/SubstitutionSuggestions.yml)) @@ -709,5 +877,18 @@ One exception: You can use **we recommend** instead of **it is recommended** or Do not use **whitelist**. Another option is **allowlist**. ([Vale](../testing.md#vale) rule: [`InclusionCultural.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/InclusionCultural.yml)) +## you, your, yours + +Use **you**, **your**, and **yours** instead of [**the user** and **the user's**](#user-users). +Documentation should be from the [point of view](https://design.gitlab.com/content/voice-tone#point-of-view) of the reader. + +Use: + +- You can configure a pipeline. + +Instead of: + +- Users can configure a pipeline. + <!-- vale on --> <!-- markdownlint-enable --> diff --git a/doc/development/documentation/workflow.md b/doc/development/documentation/workflow.md index 782cd3411b1..49ad51874e3 100644 --- a/doc/development/documentation/workflow.md +++ b/doc/development/documentation/workflow.md @@ -4,66 +4,44 @@ group: unassigned info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Documentation process +# How to update GitLab documentation -The process for creating and maintaining GitLab product documentation allows -anyone to contribute a merge request or create an issue for GitLab -documentation. - -Documentation updates relating to new features or feature enhancements must -use the [feature workflow process](https://about.gitlab.com/handbook/engineering/ux/technical-writing/workflow/#for-a-product-change) described in the GitLab Handbook. - -## Who updates the docs? - -*Anyone* can contribute! You can create a merge request for documentation when: +Anyone can contribute to the GitLab documentation! You can create a merge request for documentation when: - You find errors or other room for improvement in existing documentation. - You have an idea for all-new documentation that would help a GitLab user or administrator to accomplish their work with GitLab. -## Documentation labels - -Regardless of the type of issue or merge request, certain labels are required when documentation -is added or updated. The following are added by the issue or merge request author: - -- An appropriate [type label](../contributing/issue_workflow.md#type-labels). -- The [stage label](../contributing/issue_workflow.md#stage-labels) and - [group label](../contributing/issue_workflow.md#group-labels). For example, `~devops::create` and - `~group::source code`. -- The `~documentation` [specialization label](../contributing/issue_workflow.md#specialization-labels). - -The following are also added by members of the Technical Writing team: - -- A documentation [scoped label](../../user/project/labels.md#scoped-labels) with the - `docs::` prefix. For example, `~docs::improvement`. -- The `~Technical Writing` [team label](../contributing/issue_workflow.md#team-labels). - -Documentation changes that are not associated with the release of a new or updated feature -do not take the `~"type::feature"` label, but still need the `~documentation` label. - -They may include: - -- Documentation created or updated to improve accuracy, completeness, ease of use, or any reason - other than a [feature change](https://about.gitlab.com/handbook/engineering/ux/technical-writing/workflow/#for-a-product-change). -- Addressing gaps in existing documentation, or making improvements to existing documentation. -- Work on special projects related to the documentation. +If you are working on a feature or enhancement, use the +[feature workflow process described in the GitLab Handbook](https://about.gitlab.com/handbook/engineering/ux/technical-writing/workflow/#for-a-product-change). ## How to update the docs -To update GitLab documentation: - -1. Either: - - Click the **Edit this Page** link at the bottom of any page on <https://docs.gitlab.com>. - - Navigate to one of the repositories and documentation paths listed on the - [GitLab Documentation guidelines](index.md) page. -1. Follow the described standards and processes listed on the page, including: - - The [Structure and template](structure.md) page. - - The [Style Guide](styleguide/index.md). - - The [Markdown Guide](https://about.gitlab.com/handbook/markdown-guide/). -1. Follow the [Merge Request Guidelines](../contributing/merge_request_workflow.md#merge-request-guidelines). - -NOTE: -Work in a fork if you do not have the Developer role in the GitLab project. +If you are not a GitLab team member, or do not have the Developer role for the GitLab repository, to update GitLab documentation: + +1. Select an issue you'd like to work on. + - You don't need an issue to open a merge request. + - For a Hackathon, in the issue, in a comment, mention the person who opened the issue and ask for the issue to be assigned to you. + To be fair to other contributors, if you see someone has already asked to work on the issue, choose another issue. + If you are looking for issues to work on and don't see any that suit you, you can always fix [Vale](testing.md#vale) issues. +1. Go to the [GitLab repository](https://gitlab.com/gitlab-org/gitlab). +1. In the top-right, select **Fork**. Forking makes a copy of the repository on GitLab.com. +1. In your fork, find the documentation page by going to the `\doc` directory. +1. If you know Git, make your changes and open a merge request. + If not, follow these steps: + 1. In the top right, select **Edit**, make the changes, and **Save**. + 1. From the left menu, select **Merge requests**. + 1. For the source branch, select your fork and branch. If you did not create a branch, select `master`. + For the target branch, select the [GitLab repository](https://gitlab.com/gitlab-org/gitlab) `master` branch. + 1. For the commit message, use 3-5 words, start with a capital letter, and do not end with a period. + 1. Select **Commit changes**. A merge request opens. + 1. Select the **Documentation** template. In the description, write a brief summary of the changes and link to the related issue, if there is one. + +If you need help while working on the page, view: + +- The [Style Guide](styleguide/index.md). +- The [Word list](styleguide/word_list.md) +- The [Markdown Guide](https://about.gitlab.com/handbook/markdown-guide/). ### Ask for help @@ -83,6 +61,22 @@ To identify someone who can help you: If you are a member of the GitLab Slack workspace, you can request help in `#docs`. +## Documentation labels + +When you author an issue or merge request, you must add these labels: + +- A [type label](../contributing/issue_workflow.md#type-labels). +- A [stage label](../contributing/issue_workflow.md#stage-labels) and [group label](../contributing/issue_workflow.md#group-labels). + For example, `~devops::create` and `~group::source code`. +- A `~documentation` [specialization label](../contributing/issue_workflow.md#specialization-labels). + +A member of the Technical Writing team adds these labels: + +- A [documentation scoped label](../../user/project/labels.md#scoped-labels) with the + `docs::` prefix. For example, `~docs::improvement`. +- The [`~Technical Writing` team label](../contributing/issue_workflow.md#team-labels). +- A type label: either `~"type::feature"` or `~"type::maintenance"`. + ### Reviewing and merging Anyone with the [Maintainer role](../../user/permissions.md) to the relevant GitLab project can @@ -130,9 +124,9 @@ immediately after merge by the developer or maintainer. For this, create an issue using the [Doc Review description template](https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Doc%20Review) and link to it from the merged merge request that introduced the documentation change. -Circumstances, where a regular pre-merge Technical Writer review might be skipped, include: +Circumstances in which a regular pre-merge Technical Writer review might be skipped include: -- There is a short amount of time left before the milestone release. If less than three +- There is a short amount of time left before the milestone release. If fewer than three days are remaining, seek a post-merge review and ping the writer via Slack to ensure the review is completed as soon as possible. - The size of the change is small and you have a high degree of confidence diff --git a/doc/development/ee_features.md b/doc/development/ee_features.md index 7f74d9660e9..70d7ea7c1a5 100644 --- a/doc/development/ee_features.md +++ b/doc/development/ee_features.md @@ -1079,6 +1079,48 @@ export default { - **EE extra HTML** - For the templates that have extra HTML in EE we should move it into a new component and use the `ee_else_ce` dynamic import +#### Testing modules using EE/CE aliases + +When writing Frontend tests, if the module under test imports other modules with `ee_else_ce/...` and these modules are also needed by the relevant test, then the relevant test **must** import these modules with `ee_else_ce/...`. This avoids unexpected EE or FOSS failures, and helps ensure the EE behaves like CE when it is unlicensed. + +For example: + +```vue +<script> +// ~/foo/component_under_test.vue + +import FriendComponent from 'ee_else_ce/components/friend.vue;' + +export default { + name: 'ComponentUnderTest', + components: { FriendComponent }. +} +</script> + +<template> + <friend-component /> +</template> +``` + +```javascript +// spec/frontend/foo/component_under_test_spec.js + +// ... +// because we referenced the component using ee_else_ce we have to do the same in the spec. +import Friend from 'ee_else_ce/components/friend.vue;' + +describe('ComponentUnderTest', () => { + const findFriend = () => wrapper.find(Friend); + + it('renders friend', () => { + // This would fail in CE if we did `ee/component...` + // and would fail in EE if we did `~/component...` + expect(findFriend().exists()).toBe(true); + }); +}); + +``` + ### Non Vue Files For regular JS files, the approach is similar. diff --git a/doc/development/event_tracking/backend.md b/doc/development/event_tracking/backend.md deleted file mode 100644 index 3931f0b35ab..00000000000 --- a/doc/development/event_tracking/backend.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'https://about.gitlab.com/handbook/product/product-intelligence-guide/' -remove_date: '2021-12-01' ---- - -This document was moved to [another location](https://about.gitlab.com/handbook/product/product-intelligence-guide/). - -<!-- This redirect file can be deleted after 2021-12-01. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/event_tracking/frontend.md b/doc/development/event_tracking/frontend.md deleted file mode 100644 index 3931f0b35ab..00000000000 --- a/doc/development/event_tracking/frontend.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'https://about.gitlab.com/handbook/product/product-intelligence-guide/' -remove_date: '2021-12-01' ---- - -This document was moved to [another location](https://about.gitlab.com/handbook/product/product-intelligence-guide/). - -<!-- This redirect file can be deleted after 2021-12-01. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/event_tracking/index.md b/doc/development/event_tracking/index.md deleted file mode 100644 index 3931f0b35ab..00000000000 --- a/doc/development/event_tracking/index.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'https://about.gitlab.com/handbook/product/product-intelligence-guide/' -remove_date: '2021-12-01' ---- - -This document was moved to [another location](https://about.gitlab.com/handbook/product/product-intelligence-guide/). - -<!-- This redirect file can be deleted after 2021-12-01. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/experiment_guide/gitlab_experiment.md b/doc/development/experiment_guide/gitlab_experiment.md index af4512dcde0..288823bb41f 100644 --- a/doc/development/experiment_guide/gitlab_experiment.md +++ b/doc/development/experiment_guide/gitlab_experiment.md @@ -20,7 +20,7 @@ concepts which may seem confusing or advanced without understanding the underpin of how GitLab uses feature flags in development. One concept: GLEX supports experiments with multiple variants, which are sometimes referred to as A/B/n tests. -The [`gitlab-experiment` project](https://gitlab.com/gitlab-org/gitlab-experiment) +The [`gitlab-experiment` project](https://gitlab.com/gitlab-org/ruby/gems/gitlab-experiment) exists in a separate repository, so it can be shared across any GitLab property that uses Ruby. You should feel comfortable reading the documentation on that project as well if you want to dig into more advanced topics. @@ -394,26 +394,6 @@ You may be asked from time to time to track a specific record ID in experiments. The approach is largely up to the PM and engineer creating the implementation. No recommendations are provided here at this time. -### Record experiment subjects - -Snowplow tracking of identifiable users or groups is prohibited, but you can still -determine if an experiment is successful or not. We're allowed to record the ID of -a namespace, project or user in our database. Therefore, we can tell the experiment -to record their ID together with the assigned experiment variant in the -`experiment_subjects` database table for later analysis. - -For the recording to work, the experiment's context must include a `namespace`, -`group`, `project`, `user`, or `actor`. - -To record the experiment subject when you first assign a variant, call `record!` in -the experiment's block: - -```ruby -experiment(:pill_color, actor: current_user) do |e| - e.record! -end -``` - ## Test with RSpec This gem provides some RSpec helpers and custom matchers. These are in flux as of GitLab 13.10. diff --git a/doc/development/experiment_guide/index.md b/doc/development/experiment_guide/index.md index 1b1f756d4c0..9937cb2ebd1 100644 --- a/doc/development/experiment_guide/index.md +++ b/doc/development/experiment_guide/index.md @@ -38,13 +38,13 @@ being removed before the [experiment cleanup process](https://about.gitlab.com/h If, as a reviewer or maintainer, you find code that would usually fail review but is acceptable for now, mention your concerns with a note that there's no need to change the code. The author can then add a comment to this piece of code -and link to the issue that resolves the experiment. If the experiment is -successful and becomes part of the product, any follow up issues should be -addressed. +and link to the issue that resolves the experiment. The author or reviewer can add a link to this concern in the +experiment rollout issue under the `Experiment Successful Cleanup Concerns` section of the description. +If the experiment is successful and becomes part of the product, any items that appear under this section will be addressed. ## Implementing an experiment -[`GLEX`](https://gitlab.com/gitlab-org/gitlab-experiment) - or `Gitlab::Experiment`, the `gitlab-experiment` gem - is the preferred option for implementing an experiment in GitLab. +[`GLEX`](https://gitlab.com/gitlab-org/ruby/gems/gitlab-experiment) - or `Gitlab::Experiment`, the `gitlab-experiment` gem - is the preferred option for implementing an experiment in GitLab. For more information, see [Implementing an A/B/n experiment using GLEX](gitlab_experiment.md). diff --git a/doc/development/fe_guide/event_tracking.md b/doc/development/fe_guide/event_tracking.md deleted file mode 100644 index 3931f0b35ab..00000000000 --- a/doc/development/fe_guide/event_tracking.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'https://about.gitlab.com/handbook/product/product-intelligence-guide/' -remove_date: '2021-12-01' ---- - -This document was moved to [another location](https://about.gitlab.com/handbook/product/product-intelligence-guide/). - -<!-- This redirect file can be deleted after 2021-12-01. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/fe_guide/graphql.md b/doc/development/fe_guide/graphql.md index 44d43a32803..ed71f612061 100644 --- a/doc/development/fe_guide/graphql.md +++ b/doc/development/fe_guide/graphql.md @@ -171,6 +171,40 @@ import { getIdFromGraphQLId } from '~/graphql_shared/utils'; const primaryKeyId = getIdFromGraphQLId(data.id); ``` +**It is required** to query global `id` for every GraphQL type that has an `id` in the schema: + +```javascript +query allReleases(...) { + project(...) { + id // Project has an ID in GraphQL schema so should fetch it + releases(...) { + nodes { + // Release has no ID property in GraphQL schema + name + tagName + tagPath + assets { + count + links { + nodes { + id // Link has an ID in GraphQL schema so should fetch it + name + } + } + } + } + pageInfo { + // PageInfo no ID property in GraphQL schema + startCursor + hasPreviousPage + hasNextPage + endCursor + } + } + } +} +``` + ## Immutability and cache updates From Apollo version 3.0.0 all the cache updates need to be immutable. It needs to be replaced entirely @@ -808,11 +842,11 @@ export default { #### Polling and Performance -While the Apollo client has support for simple polling, for performance reasons, our [Etag-based caching](../polling.md) is preferred to hitting the database each time. +While the Apollo client has support for simple polling, for performance reasons, our [ETag-based caching](../polling.md) is preferred to hitting the database each time. -Once the backend is set up, there are a few changes to make on the frontend. +After the ETag resource is set up to be cached from backend, there are a few changes to make on the frontend. -First, get your resource ETag path from the backend. In the example of the pipelines graph, this is called the `graphql_resource_etag`. This will be used to create new headers to add to the Apollo context: +First, get your ETag resource from the backend, which should be in the form of a URL path. In the example of the pipelines graph, this is called the `graphql_resource_etag`, which is used to create new headers to add to the Apollo context: ```javascript /* pipelines/components/graph/utils.js */ @@ -847,7 +881,51 @@ apollo: { }, ``` -Then, because ETags depend on the request being a `GET` instead of GraphQL's usual `POST`, but our default link library does not support `GET` we need to let our default Apollo client know to use a different library. +Here, the apollo query is watching for changes in `graphqlResourceEtag`. If your ETag resource dynamically changes, you should make sure the resource you are sending in the query headers is also updated. To do this, you can store and update the ETag resource dynamically in the local cache. + +You can see an example of this in the pipeline status of the pipeline editor. The pipeline editor watches for changes in the latest pipeline. When the user creates a new commit, we update the pipeline query to poll for changes in the new pipeline. + +```graphql +# pipeline_etag.query.graphql + +query getPipelineEtag { + pipelineEtag @client +} +``` + +```javascript +/* pipeline_editor/components/header/pipeline_status.vue */ + +import getPipelineEtag from '~/pipeline_editor/graphql/queries/client/pipeline_etag.query.graphql'; + +apollo: { + pipelineEtag: { + query: getPipelineEtag, + }, + pipeline: { + context() { + return getQueryHeaders(this.pipelineEtag); + }, + query: getPipelineQuery, + pollInterval: POLL_INTERVAL, + }, +} + +/* pipeline_editor/components/commit/commit_section.vue */ + +await this.$apollo.mutate({ + mutation: commitCIFile, + update(store, { data }) { + const pipelineEtag = data?.commitCreate?.commit?.commitPipelinePath; + + if (pipelineEtag) { + store.writeQuery({ query: getPipelineEtag, data: { pipelineEtag } }); + } + }, +}); +``` + +ETags depend on the request being a `GET` instead of GraphQL's usual `POST`. Our default link library does not support `GET` requests, so we must let our default Apollo client know to use a different library. Keep in mind, this means your app cannot batch queries. ```javascript /* componentMountIndex.js */ @@ -862,10 +940,35 @@ const apolloProvider = new VueApollo({ }); ``` -Keep in mind, this means your app will not batch queries. +Finally, we can add a visibility check so that the component pauses polling when the browser tab is not active. This should lessen the request load on the page. + +```javascript +/* component.vue */ + +import { toggleQueryPollingByVisibility } from '~/pipelines/components/graph/utils'; + +export default { + mounted() { + toggleQueryPollingByVisibility(this.$apollo.queries.pipeline, POLL_INTERVAL); + }, +}; +``` + +You can use [this MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/59672/) as a reference on how to fully implement ETag caching on the frontend. Once subscriptions are mature, this process can be replaced by using them and we can remove the separate link library and return to batching queries. +##### How to test ETag caching + +You can test that your implementation works by checking requests on the network tab. If there are no changes in your ETag resource, all polled requests should: + +- Be `GET` requests instead of `POST` requests. +- Have an HTTP status of `304` instead of `200`. + +Make sure that caching is not disabled in your developer tools when testing. + +If you are using Chrome and keep seeing `200` HTTP status codes, it might be this bug: [Developer tools show 200 instead of 304](https://bugs.chromium.org/p/chromium/issues/detail?id=1269602). In this case, inspect the response headers' source to confirm that the request was actually cached and did return with a `304` status code. + #### Subscriptions We use [subscriptions](https://www.apollographql.com/docs/react/data/subscriptions/) to receive real-time updates from GraphQL API via websockets. Currently, the number of existing subscriptions is limited, you can check a list of available ones in [GraphqiQL explorer](https://gitlab.com/-/graphql-explorer) diff --git a/doc/development/fe_guide/haml.md b/doc/development/fe_guide/haml.md index f905fdad77e..803bb89118c 100644 --- a/doc/development/fe_guide/haml.md +++ b/doc/development/fe_guide/haml.md @@ -59,8 +59,12 @@ When using the GitLab UI form builder, the following components are available fo NOTE: Currently only the listed components are available but more components are planned. +<!-- vale gitlab.Spelling = NO --> + #### gitlab_ui_checkbox_component +<!-- vale gitlab.Spelling = YES --> + [GitLab UI Docs](https://gitlab-org.gitlab.io/gitlab-ui/?path=/story/base-form-form-checkbox--default) | Argument | Description | Type | Required (default value) | @@ -73,8 +77,12 @@ Currently only the listed components are available but more components are plann | `unchecked_value` | Value when checkbox is unchecked. | `String` | `false` (`'0'`) | | `label_options` | Options that are passed to [Rails `label` method](https://api.rubyonrails.org/classes/ActionView/Helpers/FormBuilder.html#method-i-label). | `Hash` | `false` (`{}`) | +<!-- vale gitlab.Spelling = NO --> + #### gitlab_ui_radio_component +<!-- vale gitlab.Spelling = YES --> + [GitLab UI Docs](https://gitlab-org.gitlab.io/gitlab-ui/?path=/story/base-form-form-radio--default) | Argument | Description | Type | Required (default value) | diff --git a/doc/development/fe_guide/vue.md b/doc/development/fe_guide/vue.md index 5d5d37e0398..b947d90cc11 100644 --- a/doc/development/fe_guide/vue.md +++ b/doc/development/fe_guide/vue.md @@ -73,7 +73,7 @@ component, is that you avoid the need to create a fixture or an HTML element in ##### provide/inject Vue supports dependency injection through [provide/inject](https://vuejs.org/v2/api/#provide-inject). -In the component the `inject` configuration accesses the values `provide` passes down. +In the component the `inject` configuration accesses the values `provide` passes down. This example of a Vue app initialization shows how the `provide` configuration passes a value from HAML to the component: ```javascript @@ -308,7 +308,7 @@ This folder holds all components that are specific to this new feature. If you need to use or create a component that is likely to be used somewhere else, please refer to `vue_shared/components`. -A good rule of thumb to know when you should create a component is to think if +A good guideline to know when you should create a component is to think if it could be reusable elsewhere. For example, tables are used in a quite amount of places across GitLab, a table @@ -336,7 +336,7 @@ In the [Vue documentation](https://vuejs.org/v2/api/#Options-Data) the Data func > The data object for the Vue instance. Vue recursively converts its properties into getter/setters to make it "reactive". The object must be plain: native objects such as browser API objects and -prototype properties are ignored. A rule of thumb is that data should just be data - it is not +prototype properties are ignored. A guideline is that data should just be data - it is not recommended to observe objects with their own stateful behavior. Based on the Vue guidance: diff --git a/doc/development/fe_guide/vue3_migration.md b/doc/development/fe_guide/vue3_migration.md index c6f480deb22..2b783eb21b7 100644 --- a/doc/development/fe_guide/vue3_migration.md +++ b/doc/development/fe_guide/vue3_migration.md @@ -31,7 +31,7 @@ Component's computed properties / methods or external helpers. **When to use** -If you are in a Vue app that doesn't use any event hub, try to avoid adding a new one unless absolutely necessary. For example, if you need a child component to react to its parent's event, it's preferred to pass a prop down. Then, use the watch property on that prop in the child component to create the desired side effect. +If you are in a Vue app that doesn't use any event hub, try to avoid adding a new one unless absolutely necessary. For example, if you need a child component to react to its parent's event, it's preferred to pass a prop down. Then, use the watch property on that prop in the child component to create the desired side effect. If you need cross-component communication (between different Vue apps), then perhaps introducing a hub is the right decision. diff --git a/doc/development/feature_flags/controls.md b/doc/development/feature_flags/controls.md index abb100c659e..4843b58c3fd 100644 --- a/doc/development/feature_flags/controls.md +++ b/doc/development/feature_flags/controls.md @@ -12,7 +12,8 @@ info: "See the Technical Writers assigned to Development Guidelines: https://abo To be able to turn on/off features behind feature flags in any of the GitLab Inc. provided environments such as staging and production, you need to have access to the [ChatOps](../chatops_on_gitlabcom.md) bot. The ChatOps bot -is currently running on the ops instance, which is different from <https://gitlab.com> or <https://dev.gitlab.org>. +is currently running on the ops instance, which is different from +[GitLab.com](https://gitlab.com) or [`dev.gitlab.org`](https://dev.gitlab.org). Follow the ChatOps document to [request access](../chatops_on_gitlabcom.md#requesting-access). @@ -35,12 +36,12 @@ This allows you to separate rolling out a feature from a deploy, making it easier to measure the impact of both separately. The GitLab feature library (using -[Flipper](https://github.com/jnunemaker/flipper), and covered in the [Feature -Flags process](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/) guide) supports rolling out changes to a percentage of +[Flipper](https://github.com/jnunemaker/flipper), and covered in the +[Feature Flags process](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/) guide) supports rolling out changes to a percentage of time to users. This in turn can be controlled using [GitLab ChatOps](../../ci/chatops/index.md). -For an up to date list of feature flag commands please see [the source -code](https://gitlab.com/gitlab-com/chatops/blob/master/lib/chatops/commands/feature.rb). +For an up to date list of feature flag commands please see +[the source code](https://gitlab.com/gitlab-com/chatops/blob/master/lib/chatops/commands/feature.rb). Note that all the examples in that file must be preceded by `/chatops run`. @@ -50,15 +51,16 @@ change feature flags or you do not [have access](#access). ### Enabling a feature for pre-production testing -As a first step in a feature rollout, you should enable the feature on <https://about.staging.gitlab.com> -and <https://dev.gitlab.org>. +As a first step in a feature rollout, you should enable the feature on +[`about.staging.gitlab.com`](https://about.staging.gitlab.com) +and [`dev.gitlab.org`](https://dev.gitlab.org). These two environments have different scopes. `dev.gitlab.org` is a production CE environment that has internal GitLab Inc. traffic and is used for some development and other related work. `staging.gitlab.com` has a smaller subset of GitLab.com database and repositories and does not have regular traffic. Staging is an EE instance and can give you -a (very) rough estimate of how your feature will look/behave on GitLab.com. +a (very) rough estimate of how your feature will look and behave on GitLab.com. Both of these instances are connected to Sentry so make sure you check the projects there for any exceptions while testing your feature after enabling the feature flag. @@ -97,7 +99,7 @@ Guidelines: #### Process When enabling a feature flag rollout, the system will automatically block the -chatops command from succeeding if there are active `"severity::1"` or `~"severity::2"` +ChatOps command from succeeding if there are active `"severity::1"` or `~"severity::2"` incidents or in-progress change issues, for example: ```shell @@ -227,7 +229,7 @@ Note, that if an actor based feature gate is present, switching the `default_enabled` attribute of the YAML definition from `false` to `true` will not have any effect. The feature gate must be deleted first. -For example, a feature flag is set via chatops: +For example, a feature flag is set via ChatOps: ```shell /chatops run feature set --project=gitlab-org/gitlab some_feature true @@ -265,7 +267,7 @@ To disable a feature flag that has been enabled for a specific project you can r You cannot selectively disable feature flags for a specific project/group/user without applying a [specific method of implementing](index.md#selectively-disable-by-actor) the feature flags. -If a feature flag is disabled via chatops, that will take precedence over the `default_enabled` value in the YML. In other words, you could have a feature enabled for on-premise installations but not for GitLab.com. +If a feature flag is disabled via ChatOps, that will take precedence over the `default_enabled` value in the YML. In other words, you could have a feature enabled for on-premise installations but not for GitLab.com. ### Feature flag change logging diff --git a/doc/development/features_inside_dot_gitlab.md b/doc/development/features_inside_dot_gitlab.md index 4631ab3a471..3a9decaad69 100644 --- a/doc/development/features_inside_dot_gitlab.md +++ b/doc/development/features_inside_dot_gitlab.md @@ -12,7 +12,7 @@ When implementing new features, please refer to these existing features to avoid - [Custom Dashboards](../operations/metrics/dashboards/index.md#add-a-new-dashboard-to-your-project): `.gitlab/dashboards/`. - [Issue Templates](../user/project/description_templates.md#create-an-issue-template): `.gitlab/issue_templates/`. - [Merge Request Templates](../user/project/description_templates.md#create-a-merge-request-template): `.gitlab/merge_request_templates/`. -- [GitLab Kubernetes Agents](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/configuration_repository.md#layout): `.gitlab/agents/`. +- [GitLab Agent](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/configuration_repository.md#layout): `.gitlab/agents/`. - [CODEOWNERS](../user/project/code_owners.md#set-up-code-owners): `.gitlab/CODEOWNERS`. - [Route Maps](../ci/review_apps/#route-maps): `.gitlab/route-map.yml`. - [Customize Auto DevOps Helm Values](../topics/autodevops/customize.md#customize-values-for-helm-chart): `.gitlab/auto-deploy-values.yaml`. diff --git a/doc/development/filtering_by_label.md b/doc/development/filtering_by_label.md index 6f9811f7e05..9e759744a1a 100644 --- a/doc/development/filtering_by_label.md +++ b/doc/development/filtering_by_label.md @@ -47,9 +47,9 @@ This is more complicated than is ideal. It makes the query construction more prone to errors (such as [issue #15557](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/15557)). -## Attempt A: WHERE EXISTS +## Attempt A: `WHERE EXISTS` -### Attempt A1: use multiple subqueries with WHERE EXISTS +### Attempt A1: use multiple subqueries with `WHERE EXISTS` In [issue #37137](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/37137) and its associated [merge request](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/14022), @@ -82,7 +82,7 @@ AND (EXISTS ( While this worked without schema changes, and did improve readability somewhat, it did not improve query performance. -### Attempt A2: use label IDs in the WHERE EXISTS clause +### Attempt A2: use label IDs in the `WHERE EXISTS` clause In [merge request #34503](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/34503), we followed a similar approach to A1. But this time, we did a separate query to fetch the IDs of the labels used in the filter so that we avoid the `JOIN` in the `EXISTS` clause and filter directly by diff --git a/doc/development/go_guide/go_upgrade.md b/doc/development/go_guide/go_upgrade.md index 53f2d7d176a..889849799bc 100644 --- a/doc/development/go_guide/go_upgrade.md +++ b/doc/development/go_guide/go_upgrade.md @@ -32,7 +32,7 @@ Individual Golang projects need to support multiple Go versions because: - We must support the [official Omnibus GitLab Go version](#updating-go-version), which may be behind the latest minor release. - When Omnibus switches Go version, we still may need to support the old one for security backports. -These 3 requirements may easily be satisfied by keeping support for the [3 latest minor versions of Go](https://golang.org/dl/). +These 3 requirements may easily be satisfied by keeping support for the [3 latest minor versions of Go](https://go.dev/dl/). It is ok to drop support for the oldest Go version and support only the 2 latest releases, if this is enough to support backports to the last 3 minor GitLab releases. @@ -52,12 +52,12 @@ in case of a critical security release. We should always: - Use the same Go version for Omnibus GitLab and Cloud Native GitLab. -- Use a [supported version](https://golang.org/doc/devel/release#policy). +- Use a [supported version](https://go.dev/doc/devel/release#policy). - Use the most recent patch-level for that version to keep up with security fixes. Changing the version affects every project being compiled, so it's important to ensure that all projects have been updated to test against the new Go version -before changing the package builders to use it. Despite [Go's compatibility promise](https://golang.org/doc/go1compat), +before changing the package builders to use it. Despite [Go's compatibility promise](https://go.dev/doc/go1compat), changes between minor versions can expose bugs or cause problems in our projects. ### Upgrade process @@ -134,7 +134,7 @@ if you need help finding the correct person or labels: | GitLab Compose Kit | [Issuer Tracker](https://gitlab.com/gitlab-org/gitlab-compose-kit/-/issues) | | GitLab Container Registry | [Issue Tracker](https://gitlab.com/gitlab-org/container-registry) | | GitLab Elasticsearch Indexer | [Issue Tracker](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer/-/issues) | -| GitLab Kubernetes Agent (KAS) | [Issue Tracker](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/issues) | +| GitLab Agent Server (KAS) | [Issue Tracker](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/issues) | | GitLab Pages | [Issue Tracker](https://gitlab.com/gitlab-org/gitlab-pages/-/issues) | | GitLab Quality Images | [Issue Tracker](https://gitlab.com/gitlab-org/gitlab-build-images/-/issues) | | GitLab Shell | [Issue Tracker](https://gitlab.com/gitlab-org/gitlab-shell/-/issues) | diff --git a/doc/development/go_guide/index.md b/doc/development/go_guide/index.md index 9bf8b7ef89a..a5661a77da3 100644 --- a/doc/development/go_guide/index.md +++ b/doc/development/go_guide/index.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Go standards and style guidelines This document describes various guidelines and best practices for GitLab -projects using the [Go language](https://golang.org). +projects using the [Go language](https://go.dev/). ## Overview @@ -103,7 +103,7 @@ projects: - Use `goimports` before committing. [`goimports`](https://pkg.go.dev/golang.org/x/tools/cmd/goimports) is a tool that automatically formats Go source code using - [`Gofmt`](https://golang.org/cmd/gofmt/), in addition to formatting import lines, + [`Gofmt`](https://pkg.go.dev/cmd/gofmt), in addition to formatting import lines, adding missing ones and removing unreferenced ones. Most editors/IDEs allow you to run commands before/after saving a file, you can set it @@ -196,7 +196,7 @@ deploy a new pod, migrating the data automatically. ### Testing frameworks We should not use any specific library or framework for testing, as the -[standard library](https://golang.org/pkg/) provides already everything to get +[standard library](https://pkg.go.dev/std) provides already everything to get started. If there is a need for more sophisticated testing tools, the following external dependencies might be worth considering in case we decide to use a specific library or framework: @@ -279,7 +279,7 @@ to make the test output easily readable. ### Benchmarks Programs handling a lot of IO or complex operations should always include -[benchmarks](https://golang.org/pkg/testing/#hdr-Benchmarks), to ensure +[benchmarks](https://pkg.go.dev/testing#hdr-Benchmarks), to ensure performance consistency over time. ## Error handling @@ -435,7 +435,7 @@ The following are some style guidelines that are specific to the Secure Team. Use `goimports -local gitlab.com/gitlab-org` before committing. [`goimports`](https://pkg.go.dev/golang.org/x/tools/cmd/goimports) is a tool that automatically formats Go source code using -[`Gofmt`](https://golang.org/cmd/gofmt/), in addition to formatting import lines, +[`Gofmt`](https://pkg.go.dev/cmd/gofmt), in addition to formatting import lines, adding missing ones and removing unreferenced ones. By using the `-local gitlab.com/gitlab-org` option, `goimports` groups locally referenced packages separately from external ones. See @@ -452,7 +452,7 @@ If the scanner report is small, less than 35 lines, then feel free to [inline th #### Test Diffs -The [go-cmp]<https://github.com/google/go-cmp> package should be used when comparing large structs in tests. It makes it possible to output a specific diff where the two structs differ, rather than seeing the whole of both structs printed out in the test logs. Here is a small example: +The [go-cmp](https://github.com/google/go-cmp) package should be used when comparing large structs in tests. It makes it possible to output a specific diff where the two structs differ, rather than seeing the whole of both structs printed out in the test logs. Here is a small example: ```golang package main diff --git a/doc/development/graphql_guide/authorization.md b/doc/development/graphql_guide/authorization.md index d7edd01cda2..717a6d29fbc 100644 --- a/doc/development/graphql_guide/authorization.md +++ b/doc/development/graphql_guide/authorization.md @@ -43,6 +43,11 @@ such as short pages, which can expose the presence of confidential resources. See [`authorization_spec.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/graphql/features/authorization_spec.rb) for examples of all the authorization schemes discussed here. +<!-- + NOTE: if you change this heading (or the location to this file), make sure to update + the referenced link in rubocop/cop/graphql/authorize_types.rb +--> + ## Type authorization Authorize a type by passing an ability to the `authorize` method. All diff --git a/doc/development/graphql_guide/index.md b/doc/development/graphql_guide/index.md index cc97e41df05..412825e06d3 100644 --- a/doc/development/graphql_guide/index.md +++ b/doc/development/graphql_guide/index.md @@ -21,3 +21,4 @@ feedback, and suggestions. - [GraphQL BatchLoader](batchloader.md): development documentation on the BatchLoader. - [GraphQL pagination](pagination.md): development documentation on pagination. - [GraphQL Pro](graphql_pro.md): information on our GraphQL Pro subscription. +- [GraphQL monitoring](monitoring.md): tips on how to use our monitoring tools to inspect GraphQL queries. diff --git a/doc/development/graphql_guide/monitoring.md b/doc/development/graphql_guide/monitoring.md new file mode 100644 index 00000000000..28d1a4a9046 --- /dev/null +++ b/doc/development/graphql_guide/monitoring.md @@ -0,0 +1,89 @@ +--- +stage: Ecosystem +group: Integrations +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Monitoring GraphQL + +This page gives tips on how to analyze GraphQL data in our monitoring tools. +Please contribute your own tips to improve this document. + +## Kibana + +We use Kibana to filter GraphQL query logs. Sign in to [Kibana](https://log.gprd.gitlab.net/) +with a `@gitlab.com` email address. + +In Kibana we can inspect two kinds of GraphQL logs: + +- Logs of each GraphQL query executed within the request. +- Logs of the full request, which due to [query multiplexing](https://graphql-ruby.org/queries/multiplex.html) + may have executed multiple queries. + +### Logs of each GraphQL query + +In a [multiplex query](https://graphql-ruby.org/queries/multiplex.html), each individual query +is logged separately. We can use subcomponent filtering to inspect these logs. +[Visit Kibana with this filter enabled](https://log.gprd.gitlab.net/goto/a0da8c9a1e9c1f533a058b7d29d13956) +or set up the subcomponent filter using these steps: + +1. Add a filter: + 1. Filter: `json.subcomponent` + 1. Operator: `is` + 1. Value: `graphql_json` +1. Select **Refresh**. + +You can select Kibana fields from the **Available fields** section of the sidebar to +add columns to the log table, or [visit this view](https://log.gprd.gitlab.net/goto/5826d3d3affb41cac52e637ffc205905), +which already has a set of Kibana fields selected. Some relevant Kibana fields include: + +| Kibana field | Description | +| --- | --- | +| `json.operation_name` | The [operation name](https://graphql.org/learn/queries/#operation-name) used by the client. | +| `json.operation_fingerprint`| The [fingerprint](https://graphql-ruby.org/api-doc/1.12.20/GraphQL/Query#fingerprint-instance_method) of the query, used to recognize repeated queries over time. | +| `json.meta.caller_id` | Appears as `graphql:<operation_name>` for queries that came from the GitLab frontend, otherwise as `graphql:unknown`. Can be used to identify internal versus external queries. | +| `json.query_string` | The query string itself. | +| `json.is_mutation` | `true` when a mutation, `false` when not. | +| `json.query_analysis.used_fields` | List of GraphQL fields selected by the query. | +| `json.query_analysis.used_deprecated_fields` | List of deprecated GraphQL fields selected by the query. | +| `json.query_analysis.duration_s` | Duration of query execution in seconds. | +| `json.query_analysis.complexity` | The [complexity](../api_graphql_styleguide.md#max-complexity) score of the query. | + +#### Useful filters + +Combine the [subcomponent filter](#logs-of-each-graphql-query) with the following Kibana filters to further interrogate the query logs. + +##### Queries that used a particular field + +Filter logs by queries that used a particular field: + +1. Add a filter: + 1. Filter: `json.query_analysis.used_fields` + 1. Operator: `is` + 1. Value: `Type.myField`, where `Type.myField` is the type name and field name as it + appears in [our GraphQL reference documentation](../../api/graphql/reference/index.md). +1. Select **Refresh**. + +##### Queries that used a deprecated field + +Filter logs of queries that used a particular deprecated field by following the +[steps above](#queries-that-used-a-particular-field) but use the `json.graphql.used_deprecated_fields` +filter instead. + +### Logs of the full request + +The full request logs encompass log data for all [multiplexed queries](https://graphql-ruby.org/queries/multiplex.html) +in the request, as well as data from time spent outside of `GraphQLController#execute`. + +To see the full request logs, do **not** apply the `json.subcomponent` [filter](#logs-of-each-graphql-query), and instead: + +1. Add a filter: + 1. Filter: `json.meta.caller_id` + 1. Operator: `is` + 1. Value: `GraphqlController#execute` +1. Select **Refresh**. + +Some differences from the [query logs](#logs-of-each-graphql-query) described above: + +- Some of the [Kibana fields mentioned above](#logs-of-each-graphql-query) are not available to the full request logs. +- The names of filters differ. For example, instead of `json.query_analysis.used_fields` you select `json.graphql.used_fields`. diff --git a/doc/development/i18n/externalization.md b/doc/development/i18n/externalization.md index 52a7f839286..65cf8911e12 100644 --- a/doc/development/i18n/externalization.md +++ b/doc/development/i18n/externalization.md @@ -133,11 +133,9 @@ You can mark that content for translation with: The `~/locale` module exports the following key functions for externalization: -- `__()` (double underscore parenthesis) -- `s__()` (namespaced double underscore parenthesis) -- `__()` Mark content for translation (note the double underscore). -- `s__()` Mark namespaced content for translation -- `n__()` Mark pluralized content for translation +- `__()` Mark content for translation (double underscore parenthesis). +- `s__()` Mark namespaced content for translation (s double underscore parenthesis). +- `n__()` Mark pluralized content for translation (n double underscore parenthesis). ```javascript import { __, s__, n__ } from '~/locale'; @@ -822,11 +820,11 @@ bin/rake gettext:regenerate ``` This command updates the `locale/gitlab.pot` file with the newly externalized strings and removes -any unused strings. Once the changes are on the default branch, [CrowdIn](https://translate.gitlab.com) +any unused strings. Once the changes are on the default branch, [Crowdin](https://translate.gitlab.com) picks them up and presents them for translation. You don't need to check in any changes to the `locale/[language]/gitlab.po` files. They are updated -automatically when [translations from CrowdIn are merged](merging_translations.md). +automatically when [translations from Crowdin are merged](merging_translations.md). If there are merge conflicts in the `gitlab.pot` file, you can delete the file and regenerate it using the same command. diff --git a/doc/development/i18n/index.md b/doc/development/i18n/index.md index c22bb6ff020..0870a0e6750 100644 --- a/doc/development/i18n/index.md +++ b/doc/development/i18n/index.md @@ -32,8 +32,8 @@ See [Externalization for GitLab](externalization.md). ## Translate strings The translation process is managed at [https://translate.gitlab.com](https://translate.gitlab.com) -using [CrowdIn](https://crowdin.com/). -You must create a CrowdIn account before you can submit translations. Once you are signed in, select +using [Crowdin](https://crowdin.com/). +You must create a Crowdin account before you can submit translations. Once you are signed in, select the language you wish to contribute translations to. Voting for translations is also valuable, helping to confirm good translations and flag inaccurate @@ -54,4 +54,4 @@ instructions on becoming a proofreader yourself. Translations are typically included in the next major or minor release. -See [Merging translations from CrowdIn](merging_translations.md). +See [Merging translations from Crowdin](merging_translations.md). diff --git a/doc/development/i18n/merging_translations.md b/doc/development/i18n/merging_translations.md index f89190fc316..43f19f8a9d6 100644 --- a/doc/development/i18n/merging_translations.md +++ b/doc/development/i18n/merging_translations.md @@ -4,27 +4,27 @@ group: Import info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Merging translations from CrowdIn +# Merging translations from Crowdin -CrowdIn automatically syncs the `gitlab.pot` file with the CrowdIn service, presenting +Crowdin automatically syncs the `gitlab.pot` file with the Crowdin service, presenting newly added externalized strings to the community of translators. -The [GitLab CrowdIn Bot](https://gitlab.com/gitlab-crowdin-bot) also creates merge requests +The [GitLab Crowdin Bot](https://gitlab.com/gitlab-crowdin-bot) also creates merge requests to take newly approved translation submissions and merge them into the `locale/<language>/gitlab.po` files. Check the [merge requests created by `gitlab-crowdin-bot`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests?scope=all&state=opened&author_username=gitlab-crowdin-bot) to see new and merged merge requests. ## Validation -By default CrowdIn commits translations with `[skip ci]` in the commit +By default Crowdin commits translations with `[skip ci]` in the commit message. This avoids an excessive number of pipelines from running. Before merging translations, make sure to trigger a pipeline to validate -translations. Static analysis validates things CrowdIn doesn't do. Create +translations. Static analysis validates things Crowdin doesn't do. Create a new pipeline at [`https://gitlab.com/gitlab-org/gitlab/pipelines/new`](https://gitlab.com/gitlab-org/gitlab/pipelines/new) (requires the Developer role) for the `master-i18n` branch. If there are validation errors, the easiest solution is to disapprove -the offending string in CrowdIn, leaving a comment with what is +the offending string in Crowdin, leaving a comment with what is required to fix the errors. There's an [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/23256) that suggests automating this process. Disapproving excludes the @@ -32,18 +32,18 @@ invalid translation. The merge request is then updated within a few minutes. If the translation fails validation due to angle brackets (`<` or `>`), -it should be disapproved in CrowdIn. Our strings must use [variables](externalization.md#html) +it should be disapproved in Crowdin. Our strings must use [variables](externalization.md#html) for HTML instead. -It might be useful to pause the integration on the CrowdIn side for a +It might be useful to pause the integration on the Crowdin side for a moment so translations don't keep coming. You can do this by clicking -**Pause sync** on the [CrowdIn integration settings page](https://translate.gitlab.com/project/gitlab-ee/settings#integration). +**Pause sync** on the [Crowdin integration settings page](https://translate.gitlab.com/project/gitlab-ee/settings#integration). ## Merging translations After all translations are determined to be appropriate and the pipelines pass, you can merge the translations into the default branch. When merging translations, -be sure to select the **Remove source branch** checkbox. This causes CrowdIn +be sure to select the **Remove source branch** checkbox. This causes Crowdin to recreate the `master-i18n` branch from the default branch after merging the new translation. @@ -51,26 +51,26 @@ We are discussing [automating this entire process](https://gitlab.com/gitlab-org ## Recreate the merge request -CrowdIn creates a new merge request as soon as the old one is closed +Crowdin creates a new merge request as soon as the old one is closed or merged. But it does not recreate the `master-i18n` branch every -time. To force CrowdIn to recreate the branch, close any [open merge requests](https://gitlab.com/gitlab-org/gitlab/-/merge_requests?scope=all&state=opened&author_username=gitlab-crowdin-bot) +time. To force Crowdin to recreate the branch, close any [open merge requests](https://gitlab.com/gitlab-org/gitlab/-/merge_requests?scope=all&state=opened&author_username=gitlab-crowdin-bot) and delete the [`master-18n`](https://gitlab.com/gitlab-org/gitlab/-/branches/all?utf8=✓&search=master-i18n) branch. This might be needed when the merge request contains failures that have been fixed on the default branch. -## Recreate the GitLab integration in CrowdIn +## Recreate the GitLab integration in Crowdin NOTE: These instructions work only for GitLab Team Members. -If for some reason the GitLab integration in CrowdIn doesn't exist, you can +If for some reason the GitLab integration in Crowdin doesn't exist, you can recreate it with the following steps: 1. Sign in to GitLab as `gitlab-crowdin-bot`. (If you're a GitLab Team Member, find credentials in the GitLab shared [1Password account](https://about.gitlab.com/handbook/security/#1password-for-teams).) -1. Sign in to CrowdIn with the GitLab integration. +1. Sign in to Crowdin with the GitLab integration. 1. Go to **Settings > Integrations > GitLab > Set Up Integration**. 1. Select the `gitlab-org/gitlab` repository. 1. In **Select Branches for Translation**, select `master`. @@ -78,7 +78,7 @@ recreate it with the following steps: ## Manually update the translation levels -There's no automated way to pull the translation levels from CrowdIn, to display +There's no automated way to pull the translation levels from Crowdin, to display this information in the language selection dropdown. Therefore, the translation levels are hard-coded in the `TRANSLATION_LEVELS` constant in [`i18n.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/i18n.rb), and must be regularly updated. diff --git a/doc/development/i18n/translation.md b/doc/development/i18n/translation.md index 04a9f68abec..61f9d7a25c3 100644 --- a/doc/development/i18n/translation.md +++ b/doc/development/i18n/translation.md @@ -6,9 +6,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Translating GitLab -For managing the translation process, we use [CrowdIn](https://crowdin.com). +For managing the translation process, we use [Crowdin](https://crowdin.com). To contribute translations at [`translate.gitlab.com`](https://translate.gitlab.com), -you must create a CrowdIn account. You may create a new account or use any of their supported +you must create a Crowdin account. You may create a new account or use any of their supported sign-in services. ## Language selections @@ -16,12 +16,12 @@ sign-in services. GitLab is being translated into many languages. To select a language to contribute to: 1. Find the language that you want to contribute to, in the - [GitLab CrowdIn project](https://crowdin.com/project/gitlab-ee). + [GitLab Crowdin project](https://crowdin.com/project/gitlab-ee). - If the language you want is available, proceed to the next step. - If the language you want is not available, [open an issue](https://gitlab.com/gitlab-org/gitlab/-/issues?scope=all&utf8=✓&state=all&label_name[]=Category%3AInternationalization). - Notify our CrowdIn administrators by including `@gitlab-org/manage/import` in your issue. + Notify our Crowdin administrators by including `@gitlab-org/manage/import` in your issue. - After the issue and any merge requests are complete, restart this procedure. 1. View the list of files and folders. Select `gitlab.pot` to open the translation editor. @@ -30,7 +30,7 @@ GitLab is being translated into many languages. To select a language to contribu The online translation editor is the easiest way to contribute translations. - + - Strings for translation are listed in the left panel. - Translations are entered into the central panel. Multiple translations are required for strings diff --git a/doc/development/image_scaling.md b/doc/development/image_scaling.md index 82ca8cf8e83..e1ffbdb766a 100644 --- a/doc/development/image_scaling.md +++ b/doc/development/image_scaling.md @@ -37,8 +37,8 @@ Furthermore, configuration in Workhorse can lead to the image scaler rejecting a For instance, here are two different URLs that serve the GitLab project avatar both in its original size and scaled down to 64 pixels. Only the second request will trigger the image scaler: -- [`/uploads/-/system/project/avatar/278964/logo-extra-whitespace.png`](https://assets.gitlab-static.net/uploads/-/system/project/avatar/278964/logo-extra-whitespace.png) -- [`/uploads/-/system/project/avatar/278964/logo-extra-whitespace.png?width=64`](https://assets.gitlab-static.net/uploads/-/system/project/avatar/278964/logo-extra-whitespace.png?width=64) +- [`/uploads/-/system/project/avatar/278964/logo-extra-whitespace.png`](https://gitlab.com/uploads/-/system/project/avatar/278964/logo-extra-whitespace.png) +- [`/uploads/-/system/project/avatar/278964/logo-extra-whitespace.png?width=64`](https://gitlab.com/uploads/-/system/project/avatar/278964/logo-extra-whitespace.png?width=64) ## Where do we scale images? diff --git a/doc/development/index.md b/doc/development/index.md index fa49d43d46c..1398104abda 100644 --- a/doc/development/index.md +++ b/doc/development/index.md @@ -139,8 +139,9 @@ In these cases, use the following workflow: and approval from the VP of Development, the DRI for Development Guidelines, @clefelhocz1. -1. After all approvals are complete, assign the merge request to the - Technical Writer for [Development Guidelines](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines) +1. After all approvals are complete, review the page's metadata to + [find a Technical Writer](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments) + who can help you merge the changes. for final content review and merge. The Technical Writer may ask for additional approvals as previously suggested before merging the MR. diff --git a/doc/development/integrations/codesandbox.md b/doc/development/integrations/codesandbox.md index caef1cd045b..8fd1f0e331d 100644 --- a/doc/development/integrations/codesandbox.md +++ b/doc/development/integrations/codesandbox.md @@ -18,7 +18,7 @@ Before using CodeSandbox with your local GitLab instance, you must: Follow the GDK [NGINX configuration instructions](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/nginx.md) to enable HTTPS for GDK. 1. Clone the [`codesandbox-client` project](https://github.com/codesandbox/codesandbox-client) locally. If you plan on contributing upstream, you might want to fork and clone first. -1. (Optional) Use correct `python` and `nodejs` versions. Otherwise, `yarn` may fail to +1. Optional. Use correct `python` and `nodejs` versions. Otherwise, `yarn` may fail to install or build some packages. If you're using `asdf` you can run the following commands: ```shell diff --git a/doc/development/integrations/secure.md b/doc/development/integrations/secure.md index 34293845d17..356e731aa87 100644 --- a/doc/development/integrations/secure.md +++ b/doc/development/integrations/secure.md @@ -298,7 +298,7 @@ this makes it possible to debug the problem without having to change the log lev #### common `logutil` package -If you are using [go](https://golang.org/) and +If you are using [go](https://go.dev/) and [common](https://gitlab.com/gitlab-org/security-products/analyzers/common), then it is suggested that you use [Logrus](https://github.com/Sirupsen/logrus) and [common's `logutil` package](https://gitlab.com/gitlab-org/security-products/analyzers/common/-/tree/master/logutil) diff --git a/doc/development/interacting_components.md b/doc/development/interacting_components.md index f6fec0cde8c..c137faf464c 100644 --- a/doc/development/interacting_components.md +++ b/doc/development/interacting_components.md @@ -29,5 +29,5 @@ See also [File Storage in GitLab](file_storage.md). ### Forks GitLab supports a great amount of features for [merge requests](../user/project/merge_requests/index.md). One -of them is the ability to create merge requests from and to [forks](../user/project/working_with_projects.md#fork-a-project), +of them is the ability to create merge requests from and to [forks](../user/project/repository/forking_workflow.md#creating-a-fork), which should also be highly considered and tested upon development phase. diff --git a/doc/development/internal_api/index.md b/doc/development/internal_api/index.md index 0fe9a5362cf..543c5f40f88 100644 --- a/doc/development/internal_api/index.md +++ b/doc/development/internal_api/index.md @@ -42,7 +42,7 @@ file, and include the token Base64 encoded in a `secret_token` parameter or in the `Gitlab-Shared-Secret` header. NOTE: -The internal API used by GitLab Pages, and GitLab Kubernetes Agent Server (`kas`) uses JSON Web Token (JWT) +The internal API used by GitLab Pages, and GitLab Agent Server (`kas`) uses JSON Web Token (JWT) authentication, which is different from GitLab Shell. ## Git Authentication @@ -400,13 +400,13 @@ Example response: } ``` -## Kubernetes agent endpoints +## GitLab Agent endpoints > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/41045) in GitLab 13.4. > - This feature is not deployed on GitLab.com > - It's not recommended for production use. -The following endpoints are used by the GitLab Kubernetes Agent Server (`kas`) +The following endpoints are used by the GitLab Agent Server (`kas`) for various purposes. These endpoints are all authenticated using JWT. The JWT secret is stored in a file @@ -414,11 +414,11 @@ specified in `config/gitlab.yml`. By default, the location is in the root of the GitLab Rails app in a file called `.gitlab_kas_secret`. WARNING: -The Kubernetes agent is under development and is not recommended for production use. +The GitLab Agent is under development and is not recommended for production use. -### Kubernetes agent information +### GitLab Agent information -Called from GitLab Kubernetes Agent Server (`kas`) to retrieve agent +Called from GitLab Agent Server (`kas`) to retrieve agent information for the given agent token. This returns the Gitaly connection information for the agent's project in order for `kas` to fetch and update the agent's configuration. @@ -434,9 +434,9 @@ curl --request GET --header "Gitlab-Kas-Api-Request: <JWT token>" \ --header "Authorization: Bearer <agent token>" "http://localhost:3000/api/v4/internal/kubernetes/agent_info" ``` -### Kubernetes agent project information +### GitLab Agent project information -Called from GitLab Kubernetes Agent Server (`kas`) to retrieve project +Called from GitLab Agent Server (`kas`) to retrieve project information for the given agent token. This returns the Gitaly connection for the requested project. GitLab `kas` uses this to configure the agent to fetch Kubernetes resources from the project repository to @@ -460,9 +460,9 @@ curl --request GET --header "Gitlab-Kas-Api-Request: <JWT token>" \ --header "Authorization: Bearer <agent token>" "http://localhost:3000/api/v4/internal/kubernetes/project_info?id=7" ``` -### Kubernetes agent usage metrics +### GitLab Agent usage metrics -Called from GitLab Kubernetes Agent Server (`kas`) to increase the usage +Called from GitLab Agent Server (`kas`) to increase the usage metric counters. | Attribute | Type | Required | Description | @@ -481,9 +481,9 @@ curl --request POST --header "Gitlab-Kas-Api-Request: <JWT token>" --header "Con --data '{"gitops_sync_count":1}' "http://localhost:3000/api/v4/internal/kubernetes/usage_metrics" ``` -### Kubernetes agent alert metrics +### GitLab Agent alert metrics -Called from GitLab Kubernetes Agent Server (KAS) to save alerts derived from Cilium on Kubernetes +Called from GitLab Agent Server (KAS) to save alerts derived from Cilium on Kubernetes Cluster. | Attribute | Type | Required | Description | @@ -505,7 +505,7 @@ curl --request POST --header "Gitlab-Kas-Api-Request: <JWT token>" \ ### Create Starboard vulnerability -Called from the GitLab Kubernetes Agent Server (`kas`) to create a security vulnerability +Called from the GitLab Agent Server (`kas`) to create a security vulnerability from a Starboard vulnerability report. This request is idempotent. Multiple requests with the same data create a single vulnerability. diff --git a/doc/development/licensed_feature_availability.md b/doc/development/licensed_feature_availability.md index 10e6d717a18..629d0027ffe 100644 --- a/doc/development/licensed_feature_availability.md +++ b/doc/development/licensed_feature_availability.md @@ -29,7 +29,7 @@ project.feature_available?(:feature_symbol) ## Restricting global features (instance) However, for features such as [Geo](../administration/geo/index.md) and -[Load balancing](../administration/database_load_balancing.md), which cannot be restricted +[Database Load Balancing](../administration/postgresql/database_load_balancing.md), which cannot be restricted to only a subset of projects or namespaces, the check is made directly in the instance license. diff --git a/doc/development/merge_request_performance_guidelines.md b/doc/development/merge_request_performance_guidelines.md index cbf3c09b28b..69e9f7d16e3 100644 --- a/doc/development/merge_request_performance_guidelines.md +++ b/doc/development/merge_request_performance_guidelines.md @@ -160,10 +160,10 @@ query. This in turn makes it much harder for this code to overload a database. ## Use read replicas when possible -In a DB cluster we have many read replicas and one primary. A classic use of scaling the DB is to have read-only actions be performed by the replicas. We use [load balancing](../administration/database_load_balancing.md) to distribute this load. This allows for the replicas to grow as the pressure on the DB grows. +In a DB cluster we have many read replicas and one primary. A classic use of scaling the DB is to have read-only actions be performed by the replicas. We use [load balancing](../administration/postgresql/database_load_balancing.md) to distribute this load. This allows for the replicas to grow as the pressure on the DB grows. By default, queries use read-only replicas, but due to -[primary sticking](../administration/database_load_balancing.md#primary-sticking), GitLab uses the +[primary sticking](../administration/postgresql/database_load_balancing.md#primary-sticking), GitLab uses the primary for some time and reverts to secondaries after they have either caught up or after 30 seconds. Doing this can lead to a considerable amount of unnecessary load on the primary. To prevent switching to the primary [merge request 56849](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/56849) introduced the @@ -187,7 +187,7 @@ Internally, our database load balancer classifies the queries based on their mai - Sidekiq background jobs After the above queries are executed, GitLab -[sticks to the primary](../administration/database_load_balancing.md#primary-sticking). +[sticks to the primary](../administration/postgresql/database_load_balancing.md#primary-sticking). To make the inside queries prefer using the replicas, [merge request 59086](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/59086) introduced `fallback_to_replicas_for_ambiguous_queries`. This MR is also an example of how we redirected a @@ -205,7 +205,7 @@ Keeping the old behavior requires marking CTEs with the keyword `MATERIALIZED`. When building CTE statements, use the `Gitlab::SQL::CTE` class [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/56976) in GitLab 13.11. By default, this `Gitlab::SQL::CTE` class forces materialization through adding the `MATERIALIZED` keyword for PostgreSQL 12 and higher. `Gitlab::SQL::CTE` automatically omits materialization when PostgreSQL 11 is running -(this behavior is implemented using a custom arel node `Gitlab::Database::AsWithMaterialized` under the surface). +(this behavior is implemented using a custom Arel node `Gitlab::Database::AsWithMaterialized` under the surface). WARNING: We plan to drop the support for PostgreSQL 11. Upgrading to GitLab 14.0 requires PostgreSQL 12 or higher. diff --git a/doc/development/multi_version_compatibility.md b/doc/development/multi_version_compatibility.md index 27a7cd6e85e..27c4edf15f4 100644 --- a/doc/development/multi_version_compatibility.md +++ b/doc/development/multi_version_compatibility.md @@ -64,7 +64,7 @@ To illustrate how these problems arise, take a look at this example: In this example, you can imagine that we are updating by one monthly release. But refer to [How long must code be backwards-compatible?](#how-long-must-code-be-backwards-compatible). -| Update step | Postgres DB | Web nodes | API nodes | Sidekiq nodes | Compatibility concerns | +| Update step | PostgreSQL DB | Web nodes | API nodes | Sidekiq nodes | Compatibility concerns | | --- | --- | --- | --- | --- | --- | | Initial state | 🙂 | 🙂 | 🙂 | 🙂 | | | Ran pre-deployment migrations | 🚢 except post-deploy migrations | 🙂 | 🙂 | 🙂 | Rails code in 🙂 is making DB calls to 🚢 | @@ -102,7 +102,7 @@ But the problem isn't just that there are many nodes. The bigger problem is that - "Web app nodes": Handle web requests - "API app nodes": Handle API requests - "Sidekiq app nodes": Handle Sidekiq jobs -- "Postgres database": Handle internal Postgres calls +- "PostgreSQL database": Handle internal PostgreSQL calls - "Redis database": Handle internal Redis calls - "Gitaly nodes": Handle internal Gitaly calls @@ -110,7 +110,7 @@ During an update, there will be [two different versions of GitLab running in dif ## Doesn't the order of update steps matter? -Yes! We have specific instructions for [zero-downtime updates](../update/index.md#upgrading-without-downtime) because it allows us to ignore some permutations of compatibility. This is why we don't worry about Rails code making DB calls to an old Postgres database schema. +Yes! We have specific instructions for [zero-downtime updates](../update/index.md#upgrading-without-downtime) because it allows us to ignore some permutations of compatibility. This is why we don't worry about Rails code making DB calls to an old PostgreSQL database schema. ## I've identified a potential backwards compatibility problem, what can I do about it? diff --git a/doc/development/new_fe_guide/modules/widget_extensions.md b/doc/development/new_fe_guide/modules/widget_extensions.md index d1f6099e908..b833ba7c630 100644 --- a/doc/development/new_fe_guide/modules/widget_extensions.md +++ b/doc/development/new_fe_guide/modules/widget_extensions.md @@ -54,3 +54,26 @@ import issueExtension from '~/vue_merge_request_widget/extensions/issues'; // Register the imported extension registerExtension(issueExtension); ``` + +## Fetching errors + +If `fetchCollapsedData()` or `fetchFullData()` methods throw an error: + +- The loading state of the extension is updated to `LOADING_STATES.collapsedError` and `LOADING_STATES.expandedError` + respectively. +- The extensions header displays an error icon and updates the text to be either: + - The text defined in `$options.i18n.error`. + - "Failed to load" if `$options.i18n.error` is not defined. +- The error is sent to Sentry to log that it occurred. + +To customise the error text, you need to add it to the `i18n` object in your extension: + +```javascript +export default { + //... + i18n: { + //... + error: __('Your error text'), + }, +}; +``` diff --git a/doc/development/performance.md b/doc/development/performance.md index e59f7fb154b..b5294c8359d 100644 --- a/doc/development/performance.md +++ b/doc/development/performance.md @@ -365,7 +365,7 @@ This patch is available by default for [GCK](https://gitlab.com/gitlab-org/gitlab-compose-kit/-/merge_requests/149) and can additionally be enabled for [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/advanced.md#apply-custom-patches-for-ruby). -This patch provides the following metrics that make it easier to understand efficiency of memory use for a given codepath: +This patch provides the following metrics that make it easier to understand efficiency of memory use for a given code path: - `mem_total_bytes`: the number of bytes consumed both due to new objects being allocated into existing object slots plus additional memory allocated for large objects (that is, `mem_bytes + slot_size * mem_objects`). @@ -384,7 +384,7 @@ and `100M mem_bytes`. You can view the current usage on [GitLab.com](https://log There are two ways of measuring your own code: 1. Review `api_json.log`, `development_json.log`, `sidekiq.log` that includes memory allocation counters. -1. Use `Gitlab::Memory::Instrumentation.with_memory_allocations` for a given codeblock and log it. +1. Use `Gitlab::Memory::Instrumentation.with_memory_allocations` for a given code block and log it. 1. Use [Measuring module](service_measurement.md) ```json diff --git a/doc/development/pipelines.md b/doc/development/pipelines.md index 71a11d2024c..c7443032d78 100644 --- a/doc/development/pipelines.md +++ b/doc/development/pipelines.md @@ -68,6 +68,7 @@ In addition, there are a few circumstances where we would always run the full RS - when the `pipeline:run-all-rspec` label is set on the merge request - when the merge request is created by an automation (e.g. Gitaly update or MR targeting a stable branch) +- when the merge request is created in a security mirror - when any CI config file is changed (i.e. `.gitlab-ci.yml` or `.gitlab/ci/**/*`) ### Jest minimal jobs @@ -83,6 +84,7 @@ In addition, there are a few circumstances where we would always run the full Je - when the `pipeline:run-all-jest` label is set on the merge request - when the merge request is created by an automation (e.g. Gitaly update or MR targeting a stable branch) +- when the merge request is created in a security mirror - when any CI config file is changed (i.e. `.gitlab-ci.yml` or `.gitlab/ci/**/*`) - when any frontend "core" file is changed (i.e. `package.json`, `yarn.lock`, `babel.config.js`, `jest.config.*.js`, `config/helpers/**/*.js`) - when any vendored JavaScript file is changed (i.e. `vendor/assets/javascripts/**/*`) @@ -220,6 +222,20 @@ The `* as-if-jh` jobs are run in addition to the regular EE-context jobs. The `j The intent is to ensure that a change doesn't introduce a failure after the `gitlab-org/gitlab` project is synced to the `gitlab-jh/gitlab` project. +## `undercover` RSpec test + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/74859) in GitLab 14.6. + +The `rspec:undercoverage` job runs [`undercover`](https://rubygems.org/gems/undercover) +to detect, and fail if any changes introduced in the merge request has zero coverage. + +The `rsepc:undercoverage` job obtains coverage data from the `rspec:coverage` +job. + +In the event of an emergency, or false positive from this job, add the +`pipeline:skip-undercoverage` label to the merge request to allow this job to +fail. + ## PostgreSQL versions testing Our test suite runs against PG12 as GitLab.com runs on PG12 and @@ -820,7 +836,7 @@ We no longer use this optimization for `gitlab-org/gitlab` because the [pack-obj allows Gitaly to serve the full CI/CD fetch traffic now. See [Git fetch caching](#git-fetch-caching). The pre-clone step works by using the `CI_PRE_CLONE_SCRIPT` variable -[defined by GitLab.com shared runners](../ci/runners/runner_cloud/linux_runner_cloud.md#pre-clone-script). +[defined by GitLab.com shared runners](../ci/runners/saas/linux_saas_runner.md#pre-clone-script). --- diff --git a/doc/development/policies.md b/doc/development/policies.md index e5191f00d9a..9a977a49329 100644 --- a/doc/development/policies.md +++ b/doc/development/policies.md @@ -79,7 +79,7 @@ cop](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/49771). ## Scores, Order, Performance -To see how the rules get evaluated into a judgment, it is useful in a console to use `policy.debug(:some_ability)`. This prints the rules in the order they are evaluated. +To see how the rules get evaluated into a judgment, open a Rails console and run: `policy.debug(:some_ability)`. This prints the rules in the order they are evaluated. For example, let's say you wanted to debug `IssuePolicy`. You might run the debugger in this way: @@ -222,7 +222,7 @@ delegation would end up with only children whose parents enjoy green vegetables eating it. But a parent may well give their child broccoli, even if they dislike it themselves, because it is good for their child. -The solution it to override the `:eat_broccoli` ability in the child policy: +The solution is to override the `:eat_broccoli` ability in the child policy: ```ruby class ChildPolicy < BasePolicy diff --git a/doc/development/product_analytics/event_dictionary.md b/doc/development/product_analytics/event_dictionary.md deleted file mode 100644 index 3931f0b35ab..00000000000 --- a/doc/development/product_analytics/event_dictionary.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'https://about.gitlab.com/handbook/product/product-intelligence-guide/' -remove_date: '2021-12-01' ---- - -This document was moved to [another location](https://about.gitlab.com/handbook/product/product-intelligence-guide/). - -<!-- This redirect file can be deleted after 2021-12-01. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/product_analytics/index.md b/doc/development/product_analytics/index.md deleted file mode 100644 index 3931f0b35ab..00000000000 --- a/doc/development/product_analytics/index.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'https://about.gitlab.com/handbook/product/product-intelligence-guide/' -remove_date: '2021-12-01' ---- - -This document was moved to [another location](https://about.gitlab.com/handbook/product/product-intelligence-guide/). - -<!-- This redirect file can be deleted after 2021-12-01. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/rails_update.md b/doc/development/rails_update.md index f25d68a8900..1a30e606c17 100644 --- a/doc/development/rails_update.md +++ b/doc/development/rails_update.md @@ -71,7 +71,7 @@ To efficiently and quickly find which Rails change caused the spec failure you c gem 'rails', ENV['RAILS_VERSION'], path: ENV['RAILS_FOLDER'] ``` -1. Set the `RAILS_FOLDER` env variable with the folder you cloned Rails into: +1. Set the `RAILS_FOLDER` environment variable with the folder you cloned Rails into: ```shell export RAILS_FOLDER="<GDK_FOLDER>/rails" diff --git a/doc/development/reactive_caching.md b/doc/development/reactive_caching.md index 5aaf4c72e64..b4deffe162a 100644 --- a/doc/development/reactive_caching.md +++ b/doc/development/reactive_caching.md @@ -228,7 +228,7 @@ self.reactive_cache_lease_timeout = 2.minutes - It defaults to 1 minute. ```ruby -self.reactive_cache_lease_timeout = 1.minute +self.reactive_cache_refresh_interval = 1.minute ``` #### `self.reactive_cache_lifetime` diff --git a/doc/development/ruby_upgrade.md b/doc/development/ruby_upgrade.md index f9816986b2d..2102a256645 100644 --- a/doc/development/ruby_upgrade.md +++ b/doc/development/ruby_upgrade.md @@ -146,7 +146,7 @@ When upgrading Ruby, consider updating the following repositories: - [Gitaly](https://gitlab.com/gitlab-org/gitaly) ([example](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3771)) - [GitLab Labkit](https://gitlab.com/gitlab-org/labkit-ruby) ([example](https://gitlab.com/gitlab-org/labkit-ruby/-/merge_requests/79)) - [GitLab Exporter](https://gitlab.com/gitlab-org/gitlab-exporter) ([example](https://gitlab.com/gitlab-org/gitlab-exporter/-/merge_requests/150)) -- [GitLab Experiment](https://gitlab.com/gitlab-org/gitlab-experiment) ([example](https://gitlab.com/gitlab-org/gitlab-experiment/-/merge_requests/128)) +- [GitLab Experiment](https://gitlab.com/gitlab-org/ruby/gems/gitlab-experiment) ([example](https://gitlab.com/gitlab-org/ruby/gems/gitlab-experiment/-/merge_requests/128)) - [Gollum Lib](https://gitlab.com/gitlab-org/gollum-lib) ([example](https://gitlab.com/gitlab-org/gollum-lib/-/merge_requests/21)) - [GitLab Helm Chart](https://gitlab.com/gitlab-org/charts/gitlab) ([example](https://gitlab.com/gitlab-org/charts/gitlab/-/merge_requests/2162)) - [GitLab Sidekiq fetcher](https://gitlab.com/gitlab-org/sidekiq-reliable-fetch) ([example](https://gitlab.com/gitlab-org/sidekiq-reliable-fetch/-/merge_requests/33)) @@ -190,7 +190,7 @@ via `gdk update`. This pause is a good time to assess the risk of this upgrade for GitLab SaaS. For Ruby upgrades that are high risk, such as major version upgrades, it is recommended to coordinate the changes with the infrastructure team through a [change management request](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/). -Create this issue early to give everyone enough time to schedule and prepare changes. +Create this issue early to give everyone enough time to schedule and prepare changes. ### Make it the default Ruby @@ -205,7 +205,7 @@ in that repository. This change is only necessary when the minor or major versio ([example](https://gitlab.com/gitlab-org/gitlab-compose-kit/-/merge_requests/176).) As mentioned above, if the impact of the Ruby upgrade on SaaS availability is uncertain, it is -prudent to skip this step until you have verified that it runs smootly in production via a staged +prudent to skip this step until you have verified that it runs smoothly in production via a staged rollout. In this case, go to the next step first, and then, after the verification period has passed, promote the new Ruby to be the new default. diff --git a/doc/development/scalability.md b/doc/development/scalability.md index fdae66b7abc..7a3f3c7097d 100644 --- a/doc/development/scalability.md +++ b/doc/development/scalability.md @@ -123,8 +123,7 @@ the read replicas. [Omnibus ships with Patroni](../administration/postgresql/rep #### Load-balancing -GitLab EE has [application support for load balancing using read -replicas](../administration/database_load_balancing.md). This load balancer does +GitLab EE has [application support for load balancing using read replicas](../administration/postgresql/database_load_balancing.md). This load balancer does some actions that aren't traditionally available in standard load balancers. For example, the application considers a replica only if its replication lag is low (for example, WAL data behind by less than 100 MB). diff --git a/doc/development/secure_coding_guidelines.md b/doc/development/secure_coding_guidelines.md index 65fdb815f87..21655d6a8fb 100644 --- a/doc/development/secure_coding_guidelines.md +++ b/doc/development/secure_coding_guidelines.md @@ -188,7 +188,7 @@ and [possessive quantifiers](https://www.regular-expressions.info/possessive.htm #### Go -Go's [`regexp`](https://golang.org/pkg/regexp/) package uses `re2` and isn't vulnerable to backtracking issues. +Go's [`regexp`](https://pkg.go.dev/regexp) package uses `re2` and isn't vulnerable to backtracking issues. ## Further Links @@ -544,7 +544,7 @@ This outputs `1` followed by the content of `/etc/passwd`. ### TLS minimum recommended version -As we have [moved away from supporting TLS 1.0 and 1.1](https://about.gitlab.com/blog/2018/10/15/gitlab-to-deprecate-older-tls/), we should only use TLS 1.2 and above. +As we have [moved away from supporting TLS 1.0 and 1.1](https://about.gitlab.com/blog/2018/10/15/gitlab-to-deprecate-older-tls/), you must use TLS 1.2 and above. #### Ciphers diff --git a/doc/development/service_ping/dictionary.md b/doc/development/service_ping/dictionary.md deleted file mode 100644 index 810c789bc03..00000000000 --- a/doc/development/service_ping/dictionary.md +++ /dev/null @@ -1,4 +0,0 @@ ---- -redirect_to: 'https://metrics.gitlab.com/index.html' -remove_date: '2021-11-10' ---- diff --git a/doc/development/service_ping/implement.md b/doc/development/service_ping/implement.md index 65a8b4c1cad..c32789740c3 100644 --- a/doc/development/service_ping/implement.md +++ b/doc/development/service_ping/implement.md @@ -26,6 +26,10 @@ To implement a new metric in Service Ping, follow these steps: 1. [Verify your metric](#verify-your-metric) 1. [Set up and test Service Ping locally](#set-up-and-test-service-ping-locally) +NOTE: +When you add or change a Service Metric, you must migrate metrics to [instrumentation classes](metrics_instrumentation.md). +For information about the progress on migrating Service ping metrics, see this [epic](https://gitlab.com/groups/gitlab-org/-/epics/5547). + ## Instrumentation classes We recommend you use [instrumentation classes](metrics_instrumentation.md) in `usage_data.rb` where possible. @@ -795,7 +799,7 @@ To set up Service Ping locally, you must: 1. [Set up local repositories](#set-up-local-repositories). 1. [Test local setup](#test-local-setup). -1. (Optional) [Test Prometheus-based Service Ping](#test-prometheus-based-service-ping). +1. Optional. [Test Prometheus-based Service Ping](#test-prometheus-based-service-ping). ### Set up local repositories diff --git a/doc/development/service_ping/index.md b/doc/development/service_ping/index.md index 6ddbe2f9646..1f751eea4d8 100644 --- a/doc/development/service_ping/index.md +++ b/doc/development/service_ping/index.md @@ -8,40 +8,24 @@ info: To determine the technical writer assigned to the Stage/Group associated w > Introduced in GitLab Ultimate 11.2, more statistics. -This guide describes Service Ping's purpose and how it's implemented. - -For more information about Product Intelligence, see: - -- [Product Intelligence Guide](https://about.gitlab.com/handbook/product/product-intelligence-guide/) -- [Snowplow Guide](../snowplow/index.md) - -More links: - -- [Product Intelligence Direction](https://about.gitlab.com/direction/product-intelligence/) -- [Data Analysis Process](https://about.gitlab.com/handbook/business-technology/data-team/#data-analysis-process/) -- [Data for Product Managers](https://about.gitlab.com/handbook/business-technology/data-team/programs/data-for-product-managers/) -- [Data Infrastructure](https://about.gitlab.com/handbook/business-technology/data-team/platform/infrastructure/) - -## What is Service Ping? - -Service Ping is a process in GitLab that collects and sends a weekly payload to GitLab Inc. +Service Ping is a GitLab process that collects and sends a weekly payload to GitLab. The payload provides important high-level data that helps our product, support, -and sales teams understand how GitLab is used. For example, the data helps to: +and sales teams understand how GitLab is used. The data helps to: - Compare counts month over month (or week over week) to get a rough sense for how an instance uses different product features. - Collect other facts that help us classify and understand GitLab installations. -- Calculate our Stage Monthly Active Users (SMAU), which helps to measure the success of our stages +- Calculate our stage monthly active users (SMAU), which helps to measure the success of our stages and features. -Service Ping information is not anonymous. It's linked to the instance's hostname. However, it does +Service Ping information is not anonymous. It's linked to the instance's hostname, but does not contain project names, usernames, or any other specific data. -Sending a Service Ping payload is optional and can be [disabled](#disable-service-ping) on any self-managed instance. -When Service Ping is enabled, GitLab gathers data from the other instances +Sending a Service Ping payload is optional and you can [disable](#disable-service-ping) it on any +self-managed instance. When Service Ping is enabled, GitLab gathers data from the other instances and can show your instance's usage statistics to your users. -### Terminology +## Service Ping terminology We use the following terminology to describe the Service Ping components: @@ -53,12 +37,18 @@ We use the following terminology to describe the Service Ping components: - **MAU**: monthly active users. - **WAU**: weekly active users. -### Why should we enable Service Ping? +### Why enable Service Ping? + +The main purpose of Service Ping is to build a better GitLab. We collect data about how GitLab is used +to understand feature or stage adoption and usage. This data gives an insight into how GitLab adds +value and helps our team understand the reasons why people use GitLab, and with this knowledge we're able to +make better product decisions. + +There are several other benefits to enabling Service Ping: -- The main purpose of Service Ping is to build a better GitLab. Data about how GitLab is used is collected to better understand feature/stage adoption and usage, which helps us understand how GitLab is adding value and helps our team better understand the reasons why people use GitLab and with this knowledge we're able to make better product decisions. - As a benefit of having Service Ping active, GitLab lets you analyze the users' activities over time of your GitLab installation. - As a benefit of having Service Ping active, GitLab provides you with [DevOps Score](../../user/admin_area/analytics/dev_ops_report.md#devops-score), which gives you an overview of your entire instance's adoption of Concurrent DevOps from planning to monitoring. -- You get better, more proactive support. (assuming that our TAMs and support organization used the data to deliver more value) +- You get better, more proactive support (assuming that our TAMs and support organization used the data to deliver more value). - You get insight and advice into how to get the most value out of your investment in GitLab. Wouldn't you want to know that a number of features or values are not being adopted in your organization? - You get a report that illustrates how you compare against other similar organizations (anonymized), with specific advice and recommendations on how to improve your DevOps processes. - Service Ping is enabled by default. To disable it, see [Disable Service Ping](#disable-service-ping). @@ -78,7 +68,7 @@ Because of these limitations we recommend you: > Introduced in GitLab 14.1. -In GitLab versions 14.1 and later, free self-managed users running [GitLab EE](../ee_features.md) can receive paid features by registering with GitLab and sending us activity data through [Service Ping](#what-is-service-ping). Features introduced here do not remove the feature from its paid tier. Users can continue to access the features in a paid tier without sharing usage data. +In GitLab versions 14.1 and later, free self-managed users running [GitLab EE](../ee_features.md) can receive paid features by registering with GitLab and sending us activity data through Service Ping. Features introduced here do not remove the feature from its paid tier. Users can continue to access the features in a paid tier without sharing usage data. #### Features available in 14.1 and later @@ -484,20 +474,34 @@ To generate Service Ping, use [Teleport](https://goteleport.com/docs/) or a deta 1. Connect to bastion with agent forwarding: - `$ ssh -A lb-bastion.gprd.gitlab.com`. + ```shell + ssh -A lb-bastion.gprd.gitlab.com + ``` + 1. Create named screen: - `$ screen -S <username>_usage_ping_<date>`. + ```shell + screen -S <username>_usage_ping_<date> + ``` + 1. Connect to console host: + + ```shell + ssh $USER-rails@console-01-sv-gprd.c.gitlab-production.internal + ``` - `$ ssh $USER-rails@console-01-sv-gprd.c.gitlab-production.internal`. -1. Run `ServicePing::SubmitService.new.execute` -1. Detach from screen: +1. Run: - `ctrl + a, ctrl + d` -1. Exit from bastion: + ```shell + ServicePing::SubmitService.new.execute + ``` - `$ exit` +1. To detach from screen, press `ctrl + A`, `ctrl + D`. +1. Exit from bastion: + + ```shell + exit + ``` ### Verification (After approx 30 hours) @@ -511,22 +515,49 @@ To generate Service Ping, use [Teleport](https://goteleport.com/docs/) or a deta 1. Reconnect to bastion: - `$ ssh -A lb-bastion.gprd.gitlab.com` + ```shell + ssh -A lb-bastion.gprd.gitlab.com + ``` + 1. Find your screen session: - `$ screen -ls` + ```shell + screen -ls + ``` + 1. Attach to your screen session: - `$ screen -x 14226.mwawrzyniak_usage_ping_2021_01_22` -1. Check the last payload in `raw_usage_data` table: `RawUsageData.last.payload` -1. Check the when the payload was sent: `RawUsageData.last.sent_at` + ```shell + screen -x 14226.mwawrzyniak_usage_ping_2021_01_22 + ``` + +1. Check the last payload in `raw_usage_data` table: + + ```shell + RawUsageData.last.payload + ``` + +1. Check the when the payload was sent: + + ```shell + RawUsageData.last.sent_at + ``` + +### Skip database write operations + +To skip database write operations, DevOps report creation, and storage of usage data payload, pass an optional argument: + +```shell +skip_db_write: +ServicePing::SubmitService.new(skip_db_write: true).execute +``` ## Troubleshooting ### Cannot disable Service Ping using the configuration file The method to disable Service Ping using the GitLab configuration file does not work in -GitLab versions 9.3.0 to 13.12.3. To disable it, you need to use the Admin Area in +GitLab versions 9.3.0 to 13.12.3. To disable it, you must use the Admin Area in the GitLab UI instead. For more information, see [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/333269). @@ -603,3 +634,12 @@ To work around this bug, you have two options: 1. Expand **Usage Statistics**. 1. Clear the **Enable Service Ping** checkbox. 1. Select **Save Changes**. + +## Related topics + +- [Product Intelligence Guide](https://about.gitlab.com/handbook/product/product-intelligence-guide/) +- [Snowplow Guide](../snowplow/index.md) +- [Product Intelligence Direction](https://about.gitlab.com/direction/product-intelligence/) +- [Data Analysis Process](https://about.gitlab.com/handbook/business-technology/data-team/#data-analysis-process/) +- [Data for Product Managers](https://about.gitlab.com/handbook/business-technology/data-team/programs/data-for-product-managers/) +- [Data Infrastructure](https://about.gitlab.com/handbook/business-technology/data-team/platform/infrastructure/) diff --git a/doc/development/service_ping/metrics_dictionary.md b/doc/development/service_ping/metrics_dictionary.md index b3c404de150..808c5064cf3 100644 --- a/doc/development/service_ping/metrics_dictionary.md +++ b/doc/development/service_ping/metrics_dictionary.md @@ -181,6 +181,7 @@ product_group: group::product intelligence value_type: string status: active milestone: 9.1 +instrumentation_class: UuidMetric introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/1521 time_frame: none data_source: database @@ -199,22 +200,23 @@ The GitLab codebase provides a dedicated [generator](https://gitlab.com/gitlab-o For uniqueness, the generated files include a timestamp prefix in ISO 8601 format. -The generator takes a list of key paths and 2 options as arguments. It creates metric YAML definitions in the corresponding location: +The generator takes a list of key paths and 3 options as arguments. It creates metric YAML definitions in the corresponding location: - `--ee`, `--no-ee` Indicates if metric is for EE. - `--dir=DIR` Indicates the metric directory. It must be one of: `counts_7d`, `7d`, `counts_28d`, `28d`, `counts_all`, `all`, `settings`, `license`. +- `--class_name=CLASS_NAME` Indicates the instrumentation class. For example `UsersCreatingIssuesMetric`, `UuidMetric` **Single metric example** ```shell -bundle exec rails generate gitlab:usage_metric_definition counts.issues --dir=7d +bundle exec rails generate gitlab:usage_metric_definition counts.issues --dir=7d --class_name=CountIssues create config/metrics/counts_7d/issues.yml ``` **Multiple metrics example** ```shell -bundle exec rails generate gitlab:usage_metric_definition counts.issues counts.users --dir=7d +bundle exec rails generate gitlab:usage_metric_definition counts.issues counts.users --dir=7d --class_name=CountUsersCreatingIssues create config/metrics/counts_7d/issues.yml create config/metrics/counts_7d/users.yml ``` @@ -223,7 +225,7 @@ NOTE: To create a metric definition used in EE, add the `--ee` flag. ```shell -bundle exec rails generate gitlab:usage_metric_definition counts.issues --ee --dir=7d +bundle exec rails generate gitlab:usage_metric_definition counts.issues --ee --dir=7d --class_name=CountUsersCreatingIssues create ee/config/metrics/counts_7d/issues.yml ``` @@ -236,9 +238,9 @@ A YAML metric definition is required for each metric. A dedicated generator is p The generator takes `category` and `event` arguments, as the root key is `redis_hll_counters`, and creates two metric definitions for weekly and monthly time frames: ```shell -bundle exec rails generate gitlab:usage_metric_definition:redis_hll issues i_closed -create config/metrics/counts_7d/i_closed_weekly.yml -create config/metrics/counts_28d/i_closed_monthly.yml +bundle exec rails generate gitlab:usage_metric_definition:redis_hll issues count_users_closing_issues +create config/metrics/counts_7d/count_users_closing_issues_weekly.yml +create config/metrics/counts_28d/count_users_closing_issues_monthly.yml ``` To create a metric definition used in EE, add the `--ee` flag. diff --git a/doc/development/service_ping/metrics_lifecycle.md b/doc/development/service_ping/metrics_lifecycle.md index 46040146de2..ebfab6341e9 100644 --- a/doc/development/service_ping/metrics_lifecycle.md +++ b/doc/development/service_ping/metrics_lifecycle.md @@ -55,23 +55,24 @@ The correct approach is to add a new metric for GitLab 12.6 release with updated and update existing business analysis artefacts to use `example_metric_without_archived` instead of `example_metric` -## Deprecate a metric +## Remove a metric + +WARNING: +If a metric is not used in Sisense or any other system after 6 months, the +Product Intelligence team marks it as inactive and assigns it to the group owner for review. + +We are working on automating this process. See [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/338466) for details. -If a metric is obsolete and you no longer use it, you can mark it as deprecated. +Product Intelligence removes metrics from Service Ping if they are not used in any Sisense dashboard. -For an example of the metric deprecation process take a look at this [example merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/59899) +For an example of the metric removal process, see this [example issue](https://gitlab.com/gitlab-org/gitlab/-/issues/297029). -To deprecate a metric: +To remove a metric: 1. Check the following YAML files and verify the metric is not used in an aggregate: - [`config/metrics/aggregates/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/aggregates/) - [`ee/config/metrics/aggregates/*.yaml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/aggregates/) -1. Create an issue in the [GitLab Data Team - project](https://gitlab.com/gitlab-data/analytics/-/issues). Ask for - confirmation that the metric is not used by other teams, or in any of the SiSense - dashboards. - 1. Verify the metric is not used to calculate the conversational index. The conversational index is a measure that reports back to self-managed instances to inform administrators of the progress of DevOps adoption for the instance. @@ -81,70 +82,6 @@ To deprecate a metric: to view the metrics that are used. The metrics are represented as the keys that are passed as a field argument into the `get_value` method. -1. Document the deprecation in the metric's YAML definition. Set - the `status:` attribute to `deprecated`, for example: - - ```yaml - --- - key_path: analytics_unique_visits.analytics_unique_visits_for_any_target_monthly - description: Visits to any of the pages listed above per month - product_section: dev - product_stage: manage - product_group: group::analytics - product_category: - value_type: number - status: deprecated - time_frame: 28d - data_source: - distribution: - - ce - tier: - - free - ``` - -1. Replace the metric's instrumentation with a fixed value. This avoids wasting - resources to calculate the deprecated metric. In - [`lib/gitlab/usage_data.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data.rb) - or - [`ee/lib/ee/gitlab/usage_data.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/ee/gitlab/usage_data.rb), - replace the code that calculates the metric's value with a fixed value that - indicates it's deprecated: - - ```ruby - module Gitlab - class UsageData - DEPRECATED_VALUE = -1000 - - def analytics_unique_visits_data - results['analytics_unique_visits_for_any_target'] = redis_usage_data { unique_visit_service.unique_visits_for(targets: :analytics) } - results['analytics_unique_visits_for_any_target_monthly'] = DEPRECATED_VALUE - - { analytics_unique_visits: results } - end - # ... - end - end - ``` - -## Remove a metric - -### Removal policy - -WARNING: -A metric that is not used in Sisense or any other system after 6 months is marked by the -Product Intelligence team as inactive and is assigned to the group owner for review. - -We are working on automating this process. See [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/338466) for details. - -Metrics can be removed from Service Ping if they: - -- Were previously [deprecated](#deprecate-a-metric). -- Are not used in any Sisense dashboard. - -For an example of the metric removal process take a look at this [example issue](https://gitlab.com/gitlab-org/gitlab/-/issues/297029) - -### To remove a deprecated metric - 1. Verify that removing the metric from the Service Ping payload does not cause errors in [Version App](https://gitlab.com/gitlab-services/version-gitlab-com) when the updated payload is collected and processed. Version App collects @@ -159,9 +96,6 @@ For an example of the metric removal process take a look at this [example issue] Ask for confirmation that the metric is not referred to in any SiSense dashboards and can be safely removed from Service Ping. Use this [example issue](https://gitlab.com/gitlab-data/analytics/-/issues/7539) for guidance. - This step can be skipped if verification done during [deprecation process](#deprecate-a-metric) - reported that metric is not required by any data transformation in Snowflake data warehouse nor it is - used by any of SiSense dashboards. 1. After you verify the metric can be safely removed, update the attributes of the metric's YAML definition: diff --git a/doc/development/session.md b/doc/development/session.md index bee857da323..61a130e3a53 100644 --- a/doc/development/session.md +++ b/doc/development/session.md @@ -51,12 +51,12 @@ Session data can be accessed directly through Redis. This can let you check up o ```ruby # Get a list of sessions -session_ids = Gitlab::Redis::SharedState.with do |redis| - redis.smembers("#{Gitlab::Redis::SharedState::USER_SESSIONS_LOOKUP_NAMESPACE}:#{user.id}") +session_ids = Gitlab::Redis::Sessions.with do |redis| + redis.smembers("#{Gitlab::Redis::Sessions::USER_SESSIONS_LOOKUP_NAMESPACE}:#{user.id}") end # Retrieve a specific session -session_data = Gitlab::Redis::SharedState.with { |redis| redis.get("#{Gitlab::Redis::SharedState::SESSION_NAMESPACE}:#{session_id}") } +session_data = Gitlab::Redis::Sessions.with { |redis| redis.get("#{Gitlab::Redis::Sessions::SESSION_NAMESPACE}:#{session_id}") } Marshal.load(session_data) ``` diff --git a/doc/development/shell_scripting_guide/index.md b/doc/development/shell_scripting_guide/index.md index d3b446d45da..3d58fabad72 100644 --- a/doc/development/shell_scripting_guide/index.md +++ b/doc/development/shell_scripting_guide/index.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w GitLab consists of many various services and sub-projects. The majority of their backend code is written in [Ruby](https://www.ruby-lang.org) and -[Go](https://golang.org). However, some of them use shell scripts for +[Go](https://go.dev/). However, some of them use shell scripts for automation of routine system administration tasks like deployment, installation, etc. It's being done either for historical reasons or as an effort to minimize the dependencies, for instance, for Docker images. diff --git a/doc/development/sidekiq_style_guide.md b/doc/development/sidekiq_style_guide.md index e28a328888d..2137a7d83e6 100644 --- a/doc/development/sidekiq_style_guide.md +++ b/doc/development/sidekiq_style_guide.md @@ -342,6 +342,7 @@ end > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/69372) in GitLab 14.3. > - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/338350) in GitLab 14.4. +> - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/338350) in GitLab 14.6. The deduplication always take into account the latest binary replication pointer, not the first one. This happens because we drop the same job scheduled for the second time and the Write-Ahead Log (WAL) is lost. @@ -353,15 +354,11 @@ This way we are always comparing the latest binary replication pointer, making sure that we read from the replica that is fully caught up. FLAG: -On self-managed GitLab, by default this feature is not available. -To make it available, -ask an administrator to [enable the preserve_latest_wal_locations_for_idempotent_jobs flag](../administration/feature_flags.md). -FLAG: -On self-managed GitLab, by default this feature is not available. -To make it available, -ask an administrator to [enable the `preserve_latest_wal_locations_for_idempotent_jobs` flag](../administration/feature_flags.md). +On self-managed GitLab, by default this feature is available. To hide the feature, ask an administrator to +[disable the feature flag](../administration/feature_flags.md) named `preserve_latest_wal_locations_for_idempotent_jobs`. + This feature flag is related to GitLab development and is not intended to be used by GitLab administrators, though. -On GitLab.com, this feature is available but can be configured by GitLab.com administrators only. +On GitLab.com, this feature is available. ## Limited capacity worker @@ -574,7 +571,7 @@ of reading a stale record is non-zero due to replicas potentially lagging behind When the number of jobs that rely on the database increases, ensuring immediate data consistency can put unsustainable load on the primary database server. We therefore added the ability to use -[database load balancing for Sidekiq workers](../administration/database_load_balancing.md#load-balancing-for-sidekiq). +[Database Load Balancing for Sidekiq workers](../administration/postgresql/database_load_balancing.md). By configuring a worker's `data_consistency` field, we can then allow the scheduler to target read replicas under several strategies outlined below. diff --git a/doc/development/snowplow/implementation.md b/doc/development/snowplow/implementation.md index fe1de789eae..6da4896c7e7 100644 --- a/doc/development/snowplow/implementation.md +++ b/doc/development/snowplow/implementation.md @@ -13,12 +13,12 @@ This page describes how to: ## Snowplow JavaScript frontend tracking -GitLab provides a `Tracking` interface that wraps the [Snowplow JavaScript tracker](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/javascript-trackers/) -to track custom events. +GitLab provides a `Tracking` interface that wraps the [Snowplow JavaScript tracker](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/javascript-trackers/) +to track custom events. For the recommended frontend tracking implementation, see [Usage recommendations](#usage-recommendations). -Tracking implementations must have an `action` and a `category`. You can provide additional +Tracking implementations must have an `action` and a `category`. You can provide additional categories from the [structured event taxonomy](index.md#structured-event-taxonomy) with an `extra` object that accepts key-value pairs. @@ -60,15 +60,15 @@ The following example shows `data-track-*` attributes assigned to a button: | `data-track-action` | true | Action the user is taking. Clicks must be prepended with `click` and activations must be prepended with `activate`. For example, focusing a form field is `activate_form_input` and clicking a button is `click_button`. Replaces `data-track-event`, which was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/290962) in GitLab 13.11. | | `data-track-label` | false | The specific element or object to act on. This can be: the label of the element, for example, a tab labeled 'Create from template' for `create_from_template`; a unique identifier if no text is available, for example, `groups_dropdown_close` for closing the Groups dropdown in the top bar; or the name or title attribute of a record being created. | | `data-track-property` | false | Any additional property of the element, or object being acted on. | -| `data-track-value` | false | Describes a numeric value or something directly related to the event. This could be the value of an input. For example, `10` when clicking `internal` visibility. If omitted, this is the element's `value` property or `undefined`. For checkboxes, the default value is the element's checked attribute or `0` when unchecked. | +| `data-track-value` | false | Describes a numeric value (decimal) directly related to the event. This could be the value of an input. For example, `10` when clicking `internal` visibility. If omitted, this is the element's `value` property or `undefined`. For checkboxes, the default value is the element's checked attribute or `0` when unchecked. | | `data-track-extra` | false | A key-value pair object passed as a valid JSON string. This attribute is added to the `extra` property in our [`gitlab_standard`](schemas.md#gitlab_standard) schema. | | `data-track-context` | false | The `context` as described in our [Structured event taxonomy](index.md#structured-event-taxonomy). | #### Event listeners Event listeners bind at the document level to handle click events in elements with data attributes. -This allows them to be handled when the DOM re-renders or changes. Document-level binding reduces -the likelihood that click events stop propagating up the DOM tree. +This allows them to be handled when the DOM re-renders or changes. Document-level binding reduces +the likelihood that click events stop propagating up the DOM tree. If click events stop propagating, you must implement listeners and [Vue component tracking](#implement-vue-component-tracking) or [raw JavaScript tracking](#implement-raw-javascript-tracking). @@ -102,12 +102,12 @@ track_action: "click_button" }) ### Implement Vue component tracking For custom event tracking, use a Vue `mixin` in components. Vue `mixin` exposes the `Tracking.event` -static method and the `track` method. You can specify tracking options in `data` or `computed`. -These options override any defaults and allow the values to be dynamic from props or based on state. +static method and the `track` method. You can specify tracking options in `data` or `computed`. +These options override any defaults and allow the values to be dynamic from props or based on state. -Several default options are passed when an event is tracked from the component: +Several default options are passed when an event is tracked from the component: -- `category`: If you don't specify, by default `document.body.dataset.page` is used. +- `category`: If you don't specify, by default `document.body.dataset.page` is used. - `label` - `property` - `value` @@ -121,7 +121,7 @@ To implement Vue component tracking: const trackingMixin = Tracking.mixin; ``` -1. Provide categories to track the event from the component. For example, to track all events in a +1. Provide categories to track the event from the component. For example, to track all events in a component with a label, use the `label` category: ```javascript @@ -293,14 +293,14 @@ describe('MyTracking', () => { ### Form tracking -To enable Snowplow automatic [form tracking](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/javascript-trackers/javascript-tracker/javascript-tracker-v2/tracking-specific-events/#form-tracking): +To enable Snowplow automatic [form tracking](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/javascript-trackers/javascript-tracker/javascript-tracker-v2/tracking-specific-events/#form-tracking): -1. Call `Tracking.enableFormTracking` when the DOM is ready. +1. Call `Tracking.enableFormTracking` when the DOM is ready. 1. Provide a `config` object that includes at least one of the following elements: - `forms` determines the forms to track. Identified by the CSS class name. - `fields` determines the fields inside the tracked forms to track. Identified by the field `name`. 1. Optional. Provide a list of contexts as the second argument. The [`gitlab_standard`](schemas.md#gitlab_standard) schema is excluded from these events. - + ```javascript Tracking.enableFormTracking({ forms: { allow: ['sign-in-form', 'password-recovery-form'] }, @@ -339,7 +339,7 @@ Backend tracking provides: - User behavior tracking - Instrumentation to monitor and visualize performance over time in a section or aspect of code. -To add custom event tracking and instrumentation, call the `GitLab::Tracking.event` class method. +To add custom event tracking and instrumentation, call the `GitLab::Tracking.event` class method. For example: ```ruby @@ -361,7 +361,7 @@ Use the following arguments: | `action` | String | | The action being taken. For example, a controller action such as `create`, or an Active Record callback. | | `label` | String | nil | The specific element or object to act on. This can be one of the following: the label of the element, for example, a tab labeled 'Create from template' for `create_from_template`; a unique identifier if no text is available, for example, `groups_dropdown_close` for closing the Groups dropdown in the top bar; or the name or title attribute of a record being created. | | `property` | String | nil | Any additional property of the element, or object being acted on. | -| `value` | Numeric | nil | Describes a numeric value or something directly related to the event. This could be the value of an input. For example, `10` when clicking `internal` visibility. | +| `value` | Numeric | nil | Describes a numeric value (decimal) directly related to the event. This could be the value of an input. For example, `10` when clicking `internal` visibility. | | `context` | Array\[SelfDescribingJSON\] | nil | An array of custom contexts to send with this event. Most events should not have any custom contexts. | | `project` | Project | nil | The project associated with the event. | | `user` | User | nil | The user associated with the event. | @@ -370,7 +370,7 @@ Use the following arguments: ### Unit testing -To test backend Snowplow events, use the `expect_snowplow_event` helper. For more information, see +To test backend Snowplow events, use the `expect_snowplow_event` helper. For more information, see [testing best practices](../testing_guide/best_practices.md#test-snowplow-events). ### Performance @@ -419,17 +419,24 @@ Snowplow Inspector Chrome Extension is a browser extension for testing frontend [Snowplow Micro](https://snowplowanalytics.com/blog/2019/07/17/introducing-snowplow-micro/) is a Docker-based solution for testing backend and frontend in a local development environment. Snowplow Micro -records the same events as the full Snowplow pipeline. To query events, use the Snowplow Micro API. +records the same events as the full Snowplow pipeline. To query events, use the Snowplow Micro API. + +It can be set up automatically using [GitLab Development Kit (GDK)](https://gitlab.com/gitlab-org/gitlab-development-kit). +See the [how-to docs](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/snowplow_micro.md) for more details. + +Optionally, you can set it up manually, using the following instructions. + +#### Set up Snowplow Micro manually -To install and run Snowplow Micro, complete these steps to modify the +To install and run Snowplow Micro, complete these steps to modify the [GitLab Development Kit (GDK)](https://gitlab.com/gitlab-org/gitlab-development-kit): 1. Ensure [Docker is installed](https://docs.docker.com/get-docker/) and running. -1. To install Snowplow Micro, clone the settings in +1. To install Snowplow Micro, clone the settings in [this project](https://gitlab.com/gitlab-org/snowplow-micro-configuration). -1. Navigate to the directory with the cloned project, +1. Navigate to the directory with the cloned project, and start the appropriate Docker container: ```shell diff --git a/doc/development/snowplow/index.md b/doc/development/snowplow/index.md index f4f41221699..dbc7a25075f 100644 --- a/doc/development/snowplow/index.md +++ b/doc/development/snowplow/index.md @@ -90,7 +90,7 @@ Each click event provides attributes that describe the event. | action | text | true | The action the user takes, or aspect that's being instrumented. The first word must describe the action or aspect. For example, clicks must be `click`, activations must be `activate`, creations must be `create`. Use underscores to describe what was acted on. For example, activating a form field is `activate_form_input`, an interface action like clicking on a dropdown is `click_dropdown`, a behavior like creating a project record from the backend is `create_project`. | | label | text | false | The specific element or object to act on. This can be one of the following: the label of the element, for example, a tab labeled 'Create from template' for `create_from_template`; a unique identifier if no text is available, for example, `groups_dropdown_close` for closing the Groups dropdown in the top bar; or the name or title attribute of a record being created. | | property | text | false | Any additional property of the element, or object being acted on. | -| value | decimal | false | Describes a numeric value or something directly related to the event. This could be the value of an input. For example, `10` when clicking `internal` visibility. | +| value | decimal | false | Describes a numeric value (decimal) directly related to the event. This could be the value of an input. For example, `10` when clicking `internal` visibility. | ### Examples @@ -149,6 +149,19 @@ ORDER BY page_view_start DESC LIMIT 100 ``` +#### Query JSON formatted data + +```sql +SELECT + derived_tstamp, + contexts:data[0]:data:extra:old_format as CURRENT_FORMAT, + contexts:data[0]:data:extra:value as UPDATED_FORMAT +FROM legacy.snowplow_structured_events_all +WHERE event_action in ('wiki_format_updated') +ORDER BY derived_tstamp DESC +LIMIT 100 +``` + ### Web-specific parameters Snowplow JavaScript adds [web-specific parameters](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/snowplow-tracker-protocol/#Web-specific_parameters) to all web events by default. diff --git a/doc/development/snowplow/schemas.md b/doc/development/snowplow/schemas.md index 5b9e4f5256e..f66e0566a9c 100644 --- a/doc/development/snowplow/schemas.md +++ b/doc/development/snowplow/schemas.md @@ -18,6 +18,7 @@ The [`StandardContext`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/g |----------------|---------------------|-----------------------|---------------------------------------------------------------------------------------------| | `project_id` | **{dotted-circle}** | integer | | | `namespace_id` | **{dotted-circle}** | integer | | +| `user_id` | **{dotted-circle}** | integer | User database record ID attribute. This file undergoes a pseudonymization process at the collector level. | | `environment` | **{check-circle}** | string (max 32 chars) | Name of the source environment, such as `production` or `staging` | | `source` | **{check-circle}** | string (max 32 chars) | Name of the source application, such as `gitlab-rails` or `gitlab-javascript` | | `plan` | **{dotted-circle}** | string (max 32 chars) | Name of the plan for the namespace, such as `free`, `premium`, or `ultimate`. Automatically picked from the `namespace`. | diff --git a/doc/development/sql.md b/doc/development/sql.md index 129280598fe..e2208caf35a 100644 --- a/doc/development/sql.md +++ b/doc/development/sql.md @@ -249,6 +249,9 @@ In line with our `CodeReuse/ActiveRecord` cop, you should only use forms like use the `ApplicationRecord`-provided `.pluck_primary_key` helper method instead. In the latter, you should add a small helper method to the relevant model. +If you have strong reasons to use `pluck`, it could make sense to limit the number +of records plucked. `MAX_PLUCK` defaults to `1_000` in `ApplicationRecord`. + ## Inherit from ApplicationRecord Most models in the GitLab codebase should inherit from `ApplicationRecord`, diff --git a/doc/development/stage_group_dashboards.md b/doc/development/stage_group_dashboards.md index a887558e473..88e9141574e 100644 --- a/doc/development/stage_group_dashboards.md +++ b/doc/development/stage_group_dashboards.md @@ -60,8 +60,8 @@ component can have 2 indicators: We're working on making this target configurable per endpoint in [this project](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/525). Learn - how to [customize the request - apdex](application_slis/rails_request_apdex.md), this new apdex + how to + [customize the request Apdex](application_slis/rails_request_apdex.md), this new Apdex measurement is not yet part of the error budget. For Sidekiq job execution, the threshold depends on the [job diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index 6a739d9e1a5..0f768a51b66 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -483,6 +483,43 @@ expect(page).to have_css '[data-testid="weight"]', text: 2 expect(page).to have_css '.atwho-view ul', visible: true ``` +##### Interacting with modals + +Use the `within_modal` helper to interact with [GitLab UI modals](https://gitlab-org.gitlab.io/gitlab-ui/?path=/story/base-modal--default). + +```ruby +include Spec::Support::Helpers::ModalHelpers + +within_modal do + expect(page).to have_link _('UI testing docs') + + fill_in _('Search projects'), with: 'gitlab' + + click_button 'Continue' +end +``` + +Furthermore, you can use `accept_gl_confirm` for confirmation modals that only need to be accepted. +This is helpful when migrating [`window.confirm()`](https://developer.mozilla.org/en-US/docs/Web/API/Window/confirm) to [`confirmAction`](https://gitlab.com/gitlab-org/gitlab/-/blob/ee280ed2b763d1278ad38c6e7e8a0aff092f617a/app/assets/javascripts/lib/utils/confirm_via_gl_modal/confirm_via_gl_modal.js#L3). + +```ruby +include Spec::Support::Helpers::ModalHelpers + +accept_gl_confirm do + click_button 'Delete user' +end +``` + +You can also pass the expected confirmation message and button text to `accept_gl_confirm`. + +```ruby +include Spec::Support::Helpers::ModalHelpers + +accept_gl_confirm('Are you sure you want to delete this user?', button_text: 'Delete') do + click_button 'Delete user' +end +``` + ##### Other useful methods After you retrieve an element using a [finder method](#finders), you can invoke a number of @@ -702,6 +739,42 @@ it 'is overdue' do end ``` +#### RSpec helpers + +You can use the `:freeze_time` and `:time_travel_to` RSpec metadata tag helpers to help reduce the amount of +boilerplate code needed to wrap entire specs with the [`ActiveSupport::Testing::TimeHelpers`](https://api.rubyonrails.org/v6.0.3.1/classes/ActiveSupport/Testing/TimeHelpers.html) +methods. + +```ruby +describe 'specs which require time to be frozen', :freeze_time do + it 'freezes time' do + right_now = Time.now + + expect(Time.now).to eq(right_now) + end +end + +describe 'specs which require time to be frozen to a specific date and/or time', time_travel_to: '2020-02-02 10:30:45 -0700' do + it 'freezes time to the specified date and time' do + expect(Time.now).to eq(Time.new(2020, 2, 2, 17, 30, 45, '+00:00')) + end +end +``` + +[Under the hood](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/support/time_travel.rb), these helpers use the `around(:each)` hook and the block syntax of the +[`ActiveSupport::Testing::TimeHelpers`](https://api.rubyonrails.org/v6.0.3.1/classes/ActiveSupport/Testing/TimeHelpers.html) +methods: + +```ruby +around(:each) do |example| + freeze_time { example.run } +end + +around(:each) do |example| + travel_to(date_or_time) { example.run } +end +``` + ### Feature flags in tests This section was moved to [developing with feature flags](../feature_flags/index.md). @@ -891,7 +964,7 @@ creates and deletes indices between examples to ensure a clean index, so that th for polluting the tests with nonessential data. Most tests for Elasticsearch logic relate to: -- Creating data in Postgres and waiting for it to be indexed in Elasticsearch. +- Creating data in PostgreSQL and waiting for it to be indexed in Elasticsearch. - Searching for that data. - Ensuring that the test gives the expected result. @@ -907,7 +980,7 @@ You do NOT need to add `:clean_gitlab_redis_shared_state` manually. Specs using Elasticsearch require that you: -- Create data in Postgres and then index it into Elasticsearch. +- Create data in PostgreSQL and then index it into Elasticsearch. - Enable Application Settings for Elasticsearch (which is disabled by default). To do so, use: @@ -921,7 +994,7 @@ end Additionally, you can use the `ensure_elasticsearch_index!` method to overcome the asynchronous nature of Elasticsearch. It uses the [Elasticsearch Refresh API](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-refresh.html#refresh-api-desc) to make sure all operations performed on an index since the last refresh are available for search. This method is typically -called after loading data into Postgres to ensure the data is indexed and searchable. +called after loading data into PostgreSQL to ensure the data is indexed and searchable. #### Test Snowplow events diff --git a/doc/development/testing_guide/end_to_end/best_practices.md b/doc/development/testing_guide/end_to_end/best_practices.md index 9491c89c2a0..543feaa967c 100644 --- a/doc/development/testing_guide/end_to_end/best_practices.md +++ b/doc/development/testing_guide/end_to_end/best_practices.md @@ -16,7 +16,7 @@ In case custom inflection logic is needed, custom inflectors are added in the [q ## Link a test to its test case -Every test should have a corresponding test case as well as a results issue in the [Quality Test Cases project](https://gitlab.com/gitlab-org/quality/testcases/). +Every test should have a corresponding test case in the [GitLab project Test Cases](https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases) as well as a results issue in the [Quality Test Cases project](https://gitlab.com/gitlab-org/quality/testcases/-/issues). It's recommended that you reuse the issue created to plan the test as the results issue. If a test case or results issue does not already exist you can create them yourself. Alternatively, you can run the test in a pipeline that has reporting enabled and the test-case reporter will automatically create a new test case and/or results issue and link the results issue to it's corresponding test case. @@ -30,11 +30,11 @@ For example: ```ruby RSpec.describe 'Stage' do describe 'General description of the feature under test' do - it 'test name', testcase: 'https://gitlab.com/gitlab-org/quality/testcases/-/quality/test_cases/:test_case_id' do + it 'test name', testcase: 'https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/:test_case_id' do ... end - it 'another test', testcase: 'https://gitlab.com/gitlab-org/quality/testcases/-/quality/test_cases/:another_test_case_id' do + it 'another test', testcase: 'https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/:another_test_case_id' do ... end end @@ -92,7 +92,7 @@ end There would be two associated test cases, one for each shared example, with the following content: -[Test 1 Test Case](https://gitlab.com/gitlab-org/quality/testcases/-/quality/test_cases/1491): +[Test 1 Test Case](https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/347774): ````markdown ```markdown @@ -115,12 +115,12 @@ pushes and merges Active and historical test results: -https://gitlab.com/gitlab-org/quality/testcases/-/issues/600 +https://gitlab.com/gitlab-org/quality/testcases/-/issues/2177 ``` ```` -[Test 1 Results Issue](https://gitlab.com/gitlab-org/quality/testcases/-/issues/600): +[Test 1 Results Issue](https://gitlab.com/gitlab-org/quality/testcases/-/issues/2177): ````markdown ```markdown @@ -142,7 +142,7 @@ pushes and merges ``` ```` -[Test 2 Test Case](https://gitlab.com/gitlab-org/quality/testcases/-/quality/test_cases/602): +[Test 2 Test Case](https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/347775): ````markdown ```markdown @@ -165,12 +165,12 @@ user fails to push Active and historical test results: -https://gitlab.com/gitlab-org/quality/testcases/-/issues/602 +https://gitlab.com/gitlab-org/quality/testcases/-/issues/2176 ``` ```` -[Test 2 Results Issue](https://gitlab.com/gitlab-org/quality/testcases/-/issues/602): +[Test 2 Results Issue](https://gitlab.com/gitlab-org/quality/testcases/-/issues/2176): ````markdown ```markdown diff --git a/doc/development/testing_guide/end_to_end/feature_flags.md b/doc/development/testing_guide/end_to_end/feature_flags.md index 52212410cc6..de34e6a1872 100644 --- a/doc/development/testing_guide/end_to_end/feature_flags.md +++ b/doc/development/testing_guide/end_to_end/feature_flags.md @@ -67,6 +67,57 @@ If no scope is provided, the feature flag is set instance-wide: Runtime::Feature.enable(:feature_flag_name) ``` +## Working with selectors + +A new feature often replaces a `vue` component or a `haml` file with a new one. +In most cases, the new file or component is accessible only with a feature flag. +This approach becomes problematic when tests must pass both with, and without, +the feature flag enabled. To ensure tests pass in both scenarios: + +1. Create another selector inside the new component or file. +1. Give it the same name as the old one. + +Selectors are connected to a specific frontend file in the [page object](page_objects.md), +and checked for availability inside our `qa:selectors` test. If the mentioned selector +is missing inside that frontend file, the test fails. To ensure selectors are +available when a feature flag is enabled or disabled, add the new selector to the +[page object](page_objects.md), leaving the old selector in place. +The test uses the correct selector and still detects missing selectors. + +If a new feature changes an existing frontend file that already has a selector, +you can add a new selector with the same name. However, only one of the selectors +displays on the page. You should: + +1. Disable the other with the feature flag. +1. Add a comment in the frontend file to delete the old selector from the frontend + file and from the page object file when the feature flag is removed. + +### Example before + +```ruby +# This is the link to the old file +view 'app/views/devise/passwords/edit.html.haml' do + # The new selector should have the same name + element :password_field + ... +end +``` + +### Example after + +```ruby +view 'app/views/devise/passwords/edit.html.haml' do + element :password_field + ... +end + +# Now it can verify the selector is available +view 'app/views/devise/passwords/new_edit_behind_ff.html.haml' do + # The selector has the same name + element :password_field +end +``` + ## Running a scenario with a feature flag enabled It's also possible to run an entire scenario with a feature flag enabled, without having to edit diff --git a/doc/development/testing_guide/end_to_end/resources.md b/doc/development/testing_guide/end_to_end/resources.md index 0819a2f7b54..abba1d51f07 100644 --- a/doc/development/testing_guide/end_to_end/resources.md +++ b/doc/development/testing_guide/end_to_end/resources.md @@ -4,14 +4,21 @@ group: unassigned info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Resource class in GitLab QA +# Resource classes in GitLab QA -Resources are primarily created using Browser UI steps, but can also -be created via the API or the CLI. +Resources are primarily created using Browser UI steps, but can also be created via the API or the CLI. + +A typical resource class is used to create a new resource that can be used in a single test. However, several tests can +end up creating the same kind of resource and use it in ways that mean it could have been +used by more than one test. Creating a new resource each time is not efficient. Therefore, we can also create reusable +resources that are created once and can then be used by many tests. + +In the following section the content focuses on single-use resources, however it also applies to reusable resources. +Information specific to [reusable resources is detailed below](#reusable-resources). ## How to properly implement a resource class? -All resource classes should inherit from `Resource::Base`. +All non-reusable resource classes should inherit from `Resource::Base`. There is only one mandatory method to implement to define a resource class. This is the `#fabricate!` method, which is used to build the resource via the @@ -391,6 +398,96 @@ end In this case, the result is similar to calling `Resource::Shirt.fabricate!`. +## Reusable resources + +Reusable resources are created by the first test that needs a particular kind of resource, and then any test that needs +the same kind of resource can reuse it instead of creating a new one. + +The `ReusableProject` resource is an example of this class: + +```ruby +module QA + module Resource + class ReusableProject < Project # A reusable resource inherits from the resource class that we want to be able to reuse. + prepend Reusable # The Reusable module mixes in some methods that help implement reuse. + + def initialize + super # A ReusableProject is a Project so it should be initialized as one. + + # Some Project attributes aren't valid and need to be overridden. For example, a ReusableProject keeps its name once it's created, + # so we don't add a random string to the name specified. + @add_name_uuid = false + + # It has a default name, and a different name can be specified when a resource is first created. However, the same name must be + # provided any time that instance of the resource is used. + @name = "reusable_project" + + # Several instances of a ReusableProject can exists as long as each is identified via a unique value for `reuse_as`. + @reuse_as = :default_project + end + + # All reusable resource classes must validate that an instance meets the conditions that allow reuse. For example, + # by confirming that the name specified for the instance is valid and doesn't conflict with other instances. + def validate_reuse_preconditions + raise ResourceReuseError unless reused_name_valid? + end + end + end +end +``` + +Consider some examples of how a reusable resource is used: + +```ruby +# This will create a project. +default_project = Resource::ReusableProject.fabricate_via_api! +default_project.name # => "reusable_project" +default_project.reuse_as # => :default_project +``` + +Then in another test we could reuse the project: + +```ruby +# This will fetch the project created above rather than creating a new one. +default_project_again = Resource::ReusableProject.fabricate_via_api! +default_project_again.name # => "reusable_project" +default_project_again.reuse_as # => :default_project +``` + +We can also create another project that we want to change in a way that might not be suitable for tests using the +default project: + +```ruby +project_with_member = Resource::ReusableProject.fabricate_via_api! do |project| + project.name = "project-with-member" + project.reuse_as = :project_with_member +end + +project_with_member.add_member(user) +``` + +Another test can reuse that project: + +```ruby +project_still_has_member = Resource::ReusableProject.fabricate_via_api! do |project| + project.name = "project-with-member" + project.reuse_as = :project_with_member +end + +expect(project_still_has_member).to have_member(user) +``` + +However, if we don't provide the name again an error will be raised: + +```ruby +Resource::ReusableProject.fabricate_via_api! do |project| + project.reuse_as = :project_with_member +end + +# => ResourceReuseError will be raised because it will try to use the default name, "reusable_project", which doesn't +# match the name specified when the project was first fabricated. +``` + ## Where to ask for help? If you need more information, ask for help on `#quality` channel on Slack diff --git a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md index 3749511fef5..cb15dbe023f 100644 --- a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md +++ b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md @@ -20,6 +20,7 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec | `:github` | The test requires a GitHub personal access token. | | `:group_saml` | The test requires a GitLab instance that has SAML SSO enabled at the group level. Interacts with an external SAML identity provider. Paired with the `:orchestrated` tag. | | `:instance_saml` | The test requires a GitLab instance that has SAML SSO enabled at the instance level. Interacts with an external SAML identity provider. Paired with the `:orchestrated` tag. | +| `:service_ping_disabled` | The test interacts with the GitLab configuration service ping at the instance level to turn admin setting service ping checkbox on or off. This tag will have the test run only in the `service_ping_disabled` job and must be paired with the `:orchestrated` and `:requires_admin` tags. | | `:jira` | The test requires a Jira Server. [GitLab-QA](https://gitlab.com/gitlab-org/gitlab-qa) provisions the Jira Server in a Docker container when the `Test::Integration::Jira` test scenario is run. | `:kubernetes` | The test includes a GitLab instance that is configured to be run behind an SSH tunnel, allowing a TLS-accessible GitLab. This test also includes provisioning of at least one Kubernetes cluster to test against. _This tag is often be paired with `:orchestrated`._ | | `:ldap_no_server` | The test requires a GitLab instance to be configured to use LDAP. To be used with the `:orchestrated` tag. It does not spin up an LDAP server at orchestration time. Instead, it creates the LDAP server at runtime. | @@ -43,6 +44,6 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec | `:skip_signup_disabled` | The test uses UI to sign up a new user and is skipped in any environment that does not allow new user registration via the UI. | | `:smoke` | The test belongs to the test suite which verifies basic functionality of a GitLab instance.| | `:smtp` | The test requires a GitLab instance to be configured to use an SMTP server. Tests SMTP notification email delivery from GitLab by using MailHog. | -| `:testcase` | The link to the test case issue in the [Quality Test Cases project](https://gitlab.com/gitlab-org/quality/testcases/). | +| `:testcase` | The link to the test case issue in the [GitLab Project Test Cases](https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases). | | `:transient` | The test tests transient bugs. It is excluded by default. | | `:issue`, `:issue_${num}` | Optional links to issues which might be related to the spec. Helps keeping track of related issues and can also be used by tools that create test reports. Currently added automatically to `Allure` test report. Multiple tags can be used by adding optional number postfix like `issue_1`, `issue_2` etc. | diff --git a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md index 95984a701e7..ef3e0624395 100644 --- a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md +++ b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md @@ -294,7 +294,7 @@ Geo requires an EE license. To visit the Geo sites in your browser, you need a r export EE_LICENSE=$(cat <path/to/your/gitlab_license>) ``` -1. (Optional) Pull the GitLab image +1. Optional. Pull the GitLab image This step is optional because pulling the Docker image is part of the [`Test::Integration::Geo` orchestrated scenario](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/d8c5c40607c2be0eda58bbca1b9f534b00889a0b/lib/gitlab/qa/scenario/test/integration/geo.rb). However, it's easier to monitor the download progress if you pull the image first, and the scenario skips this step after checking that the image is up to date. @@ -496,12 +496,12 @@ Tests that are tagged with `:mobile` can be run against specified mobile devices Running directly against an environment like staging is not recommended because Sauce Labs test logs expose credentials. Therefore, it is best practice and the default to use a tunnel. -Tunnel installation instructions are here [https://docs.saucelabs.com/secure-connections/sauce-connect/installation]. To start the tunnel, after following the installation above, copy the run command in Sauce Labs > Tunnels (must be logged in to Sauce Labs with the credentials found in 1Password) and run in terminal. +For tunnel installation instructions, read [Sauce Connect Proxy Installation](https://docs.saucelabs.com/secure-connections/sauce-connect/installation). To start the tunnel, after following the installation above, copy the run command in Sauce Labs > Tunnels (must be logged in to Sauce Labs with the credentials found in 1Password) and run in terminal. NOTE: It is highly recommended to use `GITLAB_QA_ACCESS_TOKEN` to speed up tests and reduce flakiness. -`QA_REMOTE_MOBILE_DEVICE_NAME` can be any device name listed in [https://saucelabs.com/platform/supported-browsers-devices] under Emulators/simulators and the latest versions of Android or iOS. `QA_BROWSER` must be set to `safari` for iOS devices and `chrome` for Android devices. +`QA_REMOTE_MOBILE_DEVICE_NAME` can be any device name listed in [Supported browsers and devices](https://saucelabs.com/platform/supported-browsers-devices) under Emulators/simulators and the latest versions of Android or iOS. `QA_BROWSER` must be set to `safari` for iOS devices and `chrome` for Android devices. 1. To test against a local instance with a tunnel running, in `gitlab/qa` run: diff --git a/doc/development/testing_guide/flaky_tests.md b/doc/development/testing_guide/flaky_tests.md index 9489020de5d..d2e68ea7715 100644 --- a/doc/development/testing_guide/flaky_tests.md +++ b/doc/development/testing_guide/flaky_tests.md @@ -82,26 +82,19 @@ These flaky tests can fail depending on the order they run with other tests. For - <https://gitlab.com/gitlab-org/gitlab/-/issues/327668> -To identify the tests that lead to such failure, we can use `rspec --bisect`, +To identify the tests that lead to such failure, we can use `scripts/rspec_bisect_flaky`, which would give us the minimal test combination to reproduce the failure: -```shell -rspec --bisect ee/spec/services/ee/merge_requests/update_service_spec.rb ee/spec/services/ee/notes/quick_actions_service_spec.rb ee/spec/services/epic_links/create_service_spec.rb ee/spec/services/ee/issuable/bulk_update_service_spec.rb -Bisect started using options: "ee/spec/services/ee/merge_requests/update_service_spec.rb ee/spec/services/ee/notes/quick_actions_service_spec.rb ee/spec/services/epic_links/create_service_spec.rb ee/spec/services/ee/issuable/bulk_update_service_spec.rb" -Running suite to find failures... (2 minutes 18.4 seconds) -Starting bisect with 3 failing examples and 144 non-failing examples. -Checking that failure(s) are order-dependent... failure appears to be order-dependent - -Round 1: bisecting over non-failing examples 1-144 . ignoring examples 1-72 (1 minute 11.33 seconds) -... -Round 7: bisecting over non-failing examples 132-133 . ignoring example 132 (43.78 seconds) -Bisect complete! Reduced necessary non-failing examples from 144 to 1 in 8 minutes 31 seconds. - -The minimal reproduction command is: - rspec ./ee/spec/services/ee/issuable/bulk_update_service_spec.rb[1:2:1:1:1:1,1:2:1:2:1:1,1:2:1:3:1] ./ee/spec/services/epic_links/create_service_spec.rb[1:1:2:2:6:4] -``` +1. First obtain the list of specs that ran before the flaky test. You can search + for the list under `Knapsack node specs:` in the CI job output log. +1. Save the list of specs as a file, and run: + + ```shell + cat knapsack_specs.txt | xargs scripts/rspec_bisect_flaky + ``` -We can reproduce the test failure with the reproduction command above. If we change the order of the tests, the test would pass. +If there is an order-dependency issue, the script above will print the minimal +reproduction. ### Time-sensitive flaky tests diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md index 3096386d7c3..eb35227a50c 100644 --- a/doc/development/testing_guide/frontend_testing.md +++ b/doc/development/testing_guide/frontend_testing.md @@ -395,7 +395,7 @@ it('passes', () => { ``` To modify only the hash, use either the `setWindowLocation` helper, or assign -directly to `window.location.hash`, e.g.: +directly to `window.location.hash`, for example: ```javascript it('passes', () => { @@ -423,9 +423,7 @@ it('passes', () => { ### Waiting in tests Sometimes a test needs to wait for something to happen in the application before it continues. -Avoid using [`setTimeout`](https://developer.mozilla.org/en-US/docs/Web/API/setTimeout) - -because it makes the reason for waiting unclear. Instead use one of the following approaches. +Avoid using [`setTimeout`](https://developer.mozilla.org/en-US/docs/Web/API/setTimeout) because it makes the reason for waiting unclear. Instead use one of the following approaches. #### Promises and Ajax calls @@ -633,8 +631,8 @@ The latter is useful when you have `setInterval` in the code. **Remember:** our Non-determinism is the breeding ground for flaky and brittle specs. Such specs end up breaking the CI pipeline, interrupting the work flow of other contributors. -1. Make sure your test subject's collaborators (e.g., Axios, apollo, Lodash helpers) and test environment (e.g., Date) behave consistently across systems and over time. -1. Make sure tests are focused and not doing "extra work" (e.g., needlessly creating the test subject more than once in an individual test) +1. Make sure your test subject's collaborators (for example, Axios, Apollo, Lodash helpers) and test environment (for example, Date) behave consistently across systems and over time. +1. Make sure tests are focused and not doing "extra work" (for example, needlessly creating the test subject more than once in an individual test). ### Faking `Date` for determinism @@ -650,7 +648,7 @@ describe('cool/component', () => { // Default fake `Date` const TODAY = new Date(); - // NOTE: `useFakeDate` cannot be called during test execution (i.e. inside `it`, `beforeEach`, `beforeAll`, etc.). + // NOTE: `useFakeDate` cannot be called during test execution (that is, inside `it`, `beforeEach`, `beforeAll`, etc.). describe("on Ada Lovelace's Birthday", () => { useFakeDate(1815, 11, 10) @@ -670,7 +668,7 @@ Similarly, if you really need to use the real `Date` class, then you can import ```javascript import { useRealDate } from 'helpers/fake_date'; -// NOTE: `useRealDate` cannot be called during test execution (i.e. inside `it`, `beforeEach`, `beforeAll`, etc.). +// NOTE: `useRealDate` cannot be called during test execution (that is, inside `it`, `beforeEach`, `beforeAll`, etc.). describe('with real date', () => { useRealDate(); }); @@ -702,30 +700,33 @@ The more challenging part are mocks, which can be used for functions or even dep ### Manual module mocks Manual mocks are used to mock modules across the entire Jest environment. This is a very powerful testing tool that helps simplify -unit testing by mocking out modules which cannot be easily consumed in our test environment. +unit testing by mocking out modules that cannot be easily consumed in our test environment. -> **WARNING:** Do not use manual mocks if a mock should not be consistently applied in every spec (i.e. it's only needed by a few specs). +> **WARNING:** Do not use manual mocks if a mock should not be consistently applied in every spec (that is, it's only needed by a few specs). > Instead, consider using [`jest.mock(..)`](https://jestjs.io/docs/jest-object#jestmockmodulename-factory-options) > (or a similar mocking function) in the relevant spec file. #### Where should I put manual mocks? Jest supports [manual module mocks](https://jestjs.io/docs/manual-mocks) by placing a mock in a `__mocks__/` directory next to the source module -(e.g. `app/assets/javascripts/ide/__mocks__`). **Don't do this.** We want to keep all of our test-related code in one place (the `spec/` folder). +(for example, `app/assets/javascripts/ide/__mocks__`). **Don't do this.** We want to keep all of our test-related code in one place (the `spec/` folder). If a manual mock is needed for a `node_modules` package, use the `spec/frontend/__mocks__` folder. Here's an example of a [Jest mock for the package `monaco-editor`](https://gitlab.com/gitlab-org/gitlab/-/blob/b7f914cddec9fc5971238cdf12766e79fa1629d7/spec/frontend/__mocks__/monaco-editor/index.js#L1). -If a manual mock is needed for a CE module, place it in `spec/frontend/mocks/ce`. +If a manual mock is needed for a CE module, place the implementation in +`spec/frontend/__helpers__/mocks` and add a line to the `frontend/test_setup` +(or the `frontend/shared_test_setup`) that looks something like: -- Files in `spec/frontend/mocks/ce` mocks the corresponding CE module from `app/assets/javascripts`, mirroring the source module's path. - - Example: `spec/frontend/mocks/ce/lib/utils/axios_utils` mocks the module `~/lib/utils/axios_utils`. -- We don't support mocking EE modules yet. -- If a mock is found for which a source module doesn't exist, the test suite fails. 'Virtual' mocks, or mocks that don't have a 1-to-1 association with a source module, are not supported yet. +```javascript +// "~/lib/utils/axios_utils" is the path to the real module +// "helpers/mocks/axios_utils" is the path to the mocked implementation +jest.mock('~/lib/utils/axios_utils', () => jest.requireActual('helpers/mocks/axios_utils')); +``` #### Manual mock examples -- [`mocks/axios_utils`](https://gitlab.com/gitlab-org/gitlab/-/blob/bd20aeb64c4eed117831556c54b40ff4aee9bfd1/spec/frontend/mocks/ce/lib/utils/axios_utils.js#L1) - +- [`__helpers__/mocks/axios_utils`](https://gitlab.com/gitlab-org/gitlab/-/blob/a50edd12b3b1531389624086b6381a042c8143ef/spec/frontend/__helpers__/mocks/axios_utils.js#L1) - This mock is helpful because we don't want any unmocked requests to pass any tests. Also, we are able to inject some test helpers such as `axios.waitForAll`. - [`__mocks__/mousetrap/index.js`](https://gitlab.com/gitlab-org/gitlab/-/blob/cd4c086d894226445be9d18294a060ba46572435/spec/frontend/__mocks__/mousetrap/index.js#L1) - This mock is helpful because the module itself uses AMD format which webpack understands, but is incompatible with the jest environment. This mock doesn't remove @@ -897,7 +898,7 @@ it.each([ NOTE: Only use template literal block if pretty print is not needed for spec output. For example, empty strings, nested objects etc. -For example, when testing the difference between an empty search string and a non-empty search string, the use of the array block syntax with the pretty print option would be preferred. That way the differences between an empty string e.g. `''` and a non-empty string e.g. `'search string'` would be visible in the spec output. Whereas with a template literal block, the empty string would be shown as a space, which could lead to a confusing developer experience +For example, when testing the difference between an empty search string and a non-empty search string, the use of the array block syntax with the pretty print option would be preferred. That way the differences between an empty string (`''`) and a non-empty string (`'search string'`) would be visible in the spec output. Whereas with a template literal block, the empty string would be shown as a space, which could lead to a confusing developer experience. ```javascript // bad @@ -1190,7 +1191,7 @@ it('renders the component correctly', () => { }) ``` -The above test will create two snapshots, what's important is to decide which of the snapshots provide more value for the codebase safety i.e. if one of these snapshots changes, does that highlight a possible un-wanted break in the codebase? This can help catch unexpected changes if something in an underlying dependency changes without our knowledge. +The above test will create two snapshots. It's important to decide which of the snapshots provide more value for codebase safety. That is, if one of these snapshots changes, does that highlight a possible break in the codebase? This can help catch unexpected changes if something in an underlying dependency changes without our knowledge. ### Pros and Cons diff --git a/doc/development/testing_guide/img/review-app-parent-pipeline.png b/doc/development/testing_guide/img/review-app-parent-pipeline.png Binary files differindex 5686d5f6ebe..dd64d7d1be5 100644 --- a/doc/development/testing_guide/img/review-app-parent-pipeline.png +++ b/doc/development/testing_guide/img/review-app-parent-pipeline.png diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md index 3190ab6c899..31a807697c5 100644 --- a/doc/development/testing_guide/review_apps.md +++ b/doc/development/testing_guide/review_apps.md @@ -36,6 +36,12 @@ On every [pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/125315730) in browser performance testing using a [Sitespeed.io Container](../../user/project/merge_requests/browser_performance_testing.md). +## Sample Data for Review Apps + +Upon deployment of a review app, project data is created from the [`sample-gitlab-project`](https://gitlab.com/gitlab-org/sample-data-templates/sample-gitlab-project) template project. This aims to provide projects with prepopulated resources to facilitate manual and exploratory testing. + +The sample projects will be created in the `root` user namespace and can be accessed from the personal projects list for that user. + ## How to ### Redeploy Review App from a clean slate @@ -81,26 +87,22 @@ the GitLab handbook information for the [shared 1Password account](https://about ### Run a Rails console 1. Make sure you [have access to the cluster](#get-access-to-the-gcp-review-apps-cluster) and the `container.pods.exec` permission first. -1. [Filter Workloads by your Review App slug](https://console.cloud.google.com/kubernetes/workload?project=gitlab-review-apps), - e.g. `review-qa-raise-e-12chm0`. -1. Find and open the `task-runner` Deployment, e.g. `review-qa-raise-e-12chm0-task-runner`. -1. Click on the Pod in the "Managed pods" section, e.g. `review-qa-raise-e-12chm0-task-runner-d5455cc8-2lsvz`. -1. Click on the `KUBECTL` dropdown, then `Exec` -> `task-runner`. -1. Replace `-c task-runner -- ls` with `-it -- gitlab-rails console` from the +1. [Filter Workloads by your Review App slug](https://console.cloud.google.com/kubernetes/workload?project=gitlab-review-apps). For example, `review-qa-raise-e-12chm0`. +1. Find and open the `toolbox` Deployment. For example, `review-qa-raise-e-12chm0-toolbox`. +1. Click on the Pod in the "Managed pods" section. For example, `review-qa-raise-e-12chm0-toolbox-d5455cc8-2lsvz`. +1. Click on the `KUBECTL` dropdown, then `Exec` -> `toolbox`. +1. Replace `-c toolbox -- ls` with `-it -- gitlab-rails console` from the default command or - - Run `kubectl exec --namespace review-qa-raise-e-12chm0 review-qa-raise-e-12chm0-task-runner-d5455cc8-2lsvz -it -- gitlab-rails console` and - - Replace `review-qa-raise-e-12chm0-task-runner-d5455cc8-2lsvz` + - Run `kubectl exec --namespace review-qa-raise-e-12chm0 review-qa-raise-e-12chm0-toolbox-d5455cc8-2lsvz -it -- gitlab-rails console` and + - Replace `review-qa-raise-e-12chm0-toolbox-d5455cc8-2lsvz` with your Pod's name. ### Dig into a Pod's logs 1. Make sure you [have access to the cluster](#get-access-to-the-gcp-review-apps-cluster) and the `container.pods.getLogs` permission first. -1. [Filter Workloads by your Review App slug](https://console.cloud.google.com/kubernetes/workload?project=gitlab-review-apps), - e.g. `review-qa-raise-e-12chm0`. -1. Find and open the `migrations` Deployment, e.g. - `review-qa-raise-e-12chm0-migrations.1`. -1. Click on the Pod in the "Managed pods" section, e.g. - `review-qa-raise-e-12chm0-migrations.1-nqwtx`. +1. [Filter Workloads by your Review App slug](https://console.cloud.google.com/kubernetes/workload?project=gitlab-review-apps). For example, `review-qa-raise-e-12chm0`. +1. Find and open the `migrations` Deployment. For example, `review-qa-raise-e-12chm0-migrations.1`. +1. Click on the Pod in the "Managed pods" section. For example, `review-qa-raise-e-12chm0-migrations.1-nqwtx`. 1. Click on the `Container logs` link. Alternatively, you could use the [Logs Explorer](https://console.cloud.google.com/logs/query;query=?project=gitlab-review-apps) which provides more utility to search logs. An example query for a pod name is as follows: @@ -161,7 +163,7 @@ subgraph "CNG-mirror pipeline" - The `review-build-cng` job automatically starts only if your MR includes [CI or frontend changes](../pipelines.md#changes-patterns). In other cases, the job is manual. - The [`CNG-mirror`](https://gitlab.com/gitlab-org/build/CNG-mirror/pipelines/44364657) pipeline creates the Docker images of - each component (e.g. `gitlab-rails-ee`, `gitlab-shell`, `gitaly` etc.) + each component (for example, `gitlab-rails-ee`, `gitlab-shell`, `gitaly` etc.) based on the commit from the [GitLab pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/125315730) and stores them in its [registry](https://gitlab.com/gitlab-org/build/CNG-mirror/container_registry). - We use the [`CNG-mirror`](https://gitlab.com/gitlab-org/build/CNG-mirror) project so that the `CNG`, (Cloud @@ -192,7 +194,7 @@ subgraph "CNG-mirror pipeline" - If the `review-deploy` job keeps failing (and a manual retry didn't help), please post a message in the `#g_qe_engineering_productivity` channel and/or create a `~"Engineering Productivity"` `~"ep::review apps"` `~"type::bug"` issue with a link to your merge request. Note that the deployment failure can - reveal an actual problem introduced in your merge request (i.e. this isn't + reveal an actual problem introduced in your merge request (that is, this isn't necessarily a transient failure)! - If the `review-qa-smoke` job keeps failing (note that we already retry it twice), please check the job's logs: you could discover an actual problem introduced in @@ -231,7 +233,7 @@ that were not removed along with the Kubernetes resources. The cluster is configured via Terraform in the [`engineering-productivity-infrastructure`](https://gitlab.com/gitlab-org/quality/engineering-productivity-infrastructure) project. Node pool image type must be `Container-Optimized OS (cos)`, not `Container-Optimized OS with Containerd (cos_containerd)`, -due to this [known issue on GitLab Runner Kubernetes executor](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4755) +due to this [known issue on the Kubernetes executor for GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4755) ### Helm @@ -251,7 +253,7 @@ aids in identifying load spikes on the cluster, and if nodes are problematic or ### Database related errors in `review-deploy` or `review-qa-smoke` Occasionally the state of a Review App's database could diverge from the database schema. This could be caused by -changes to migration files or schema, such as a migration being renamed or deleted. This typically manifest in migration errors such as: +changes to migration files or schema, such as a migration being renamed or deleted. This typically manifests in migration errors such as: - migration job failing with a column that already exists - migration job failing with a column that does not exist @@ -283,7 +285,7 @@ If the Docker image does not exist: - Verify the `image.repository` and `image.tag` options in the `helm upgrade --install` command match the repository names used by CNG-mirror pipeline. - Look further in the corresponding downstream CNG-mirror pipeline in `review-build-cng` job. -### Node count is always increasing (i.e. never stabilizing or decreasing) +### Node count is always increasing (never stabilizing or decreasing) **Potential cause:** @@ -370,10 +372,10 @@ effectively preventing all the Review Apps from getting a DNS record assigned, making them unreachable via domain name. This in turn prevented other components of the Review App to properly start -(e.g. `gitlab-runner`). +(for example, `gitlab-runner`). -After some digging, we found that new mounts were failing, when being performed -with transient scopes (e.g. pods) of `systemd-mount`: +After some digging, we found that new mounts fail when performed +with transient scopes (for example, pods) of `systemd-mount`: ```plaintext MountVolume.SetUp failed for volume "dns-gitlab-review-app-external-dns-token-sj5jm" : mount failed: exit status 1 @@ -399,8 +401,8 @@ For the record, the debugging steps to find out this issue were: instances** then click the "SSH" button for the node where the `dns-gitlab-review-app-external-dns` pod runs) 1. In the node: `systemctl --version` => `systemd 232` 1. Gather some more information: - - `mount | grep kube | wc -l` => e.g. 290 - - `systemctl list-units --all | grep -i var-lib-kube | wc -l` => e.g. 142 + - `mount | grep kube | wc -l` (returns a count, for example, 290) + - `systemctl list-units --all | grep -i var-lib-kube | wc -l` (returns a count, for example, 142) 1. Check how many pods are in a bad state: - Get all pods running a given node: `kubectl get pods --field-selector=spec.nodeName=NODE_NAME` - Get all the `Running` pods on a given node: `kubectl get pods --field-selector=spec.nodeName=NODE_NAME | grep Running` diff --git a/doc/development/testing_guide/testing_levels.md b/doc/development/testing_guide/testing_levels.md index 29cdfab713e..9ca2d0db93c 100644 --- a/doc/development/testing_guide/testing_levels.md +++ b/doc/development/testing_guide/testing_levels.md @@ -223,14 +223,14 @@ graph RL Formal definition: <https://en.wikipedia.org/wiki/Integration_testing> These kind of tests ensure that individual parts of the application work well -together, without the overhead of the actual app environment (i.e. the browser). +together, without the overhead of the actual app environment (such as the browser). These tests should assert at the request/response level: status code, headers, body. -They're useful to test permissions, redirections, what view is rendered etc. +They're useful, for example, to test permissions, redirections, API endpoints, what view is rendered, and so forth. | Code path | Tests path | Testing engine | Notes | | --------- | ---------- | -------------- | ----- | -| `app/controllers/` | `spec/requests/`, `spec/controllers` | RSpec | Request specs are preferred over legacy controller specs. | +| `app/controllers/` | `spec/requests/`, `spec/controllers` | RSpec | Request specs are preferred over legacy controller specs. Request specs are encouraged for API endpoints. | | `app/mailers/` | `spec/mailers/` | RSpec | | | `lib/api/` | `spec/requests/api/` | RSpec | | | `app/assets/javascripts/` | `spec/frontend/` | Jest | [More details below](#frontend-integration-tests) | @@ -502,9 +502,7 @@ These tests run against the UI and ensure that basic functionality is working. ### GitLab QA orchestrator -[GitLab QA orchestrator](https://gitlab.com/gitlab-org/gitlab-qa) is a tool that allows to test that all these pieces -integrate well together by building a Docker image for a given version of GitLab -Rails and running end-to-end tests (i.e. using Capybara) against it. +[GitLab QA orchestrator](https://gitlab.com/gitlab-org/gitlab-qa) is a tool that allows you to test that all these pieces integrate well together by building a Docker image for a given version of GitLab Rails and running end-to-end tests (using Capybara) against it. Learn more in the [GitLab QA orchestrator README](https://gitlab.com/gitlab-org/gitlab-qa/tree/master/README.md). diff --git a/doc/development/usage_ping/dictionary.md b/doc/development/usage_ping/dictionary.md deleted file mode 100644 index 810c789bc03..00000000000 --- a/doc/development/usage_ping/dictionary.md +++ /dev/null @@ -1,4 +0,0 @@ ---- -redirect_to: 'https://metrics.gitlab.com/index.html' -remove_date: '2021-11-10' ---- diff --git a/doc/development/work_items_widgets.md b/doc/development/work_items_widgets.md new file mode 100644 index 00000000000..92919c10a9f --- /dev/null +++ b/doc/development/work_items_widgets.md @@ -0,0 +1,114 @@ +--- +stage: Plan +group: Project Management +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- +# Work items widgets + +## Frontend architecture + +Widgets for work items are heavily inspired by [Frontend widgets](fe_guide/widgets.md). +You can expect some differences, because work items are architecturally different from issuables. + +GraphQL (Vue Apollo) constitutes the core of work items widgets' stack. + +### Retrieve widget information for work items + +To display a work item page, the frontend must know which widgets are available +on the work item it is attempting to display. To do so, it needs to fetch the +list of widgets, using a query like this: + +```plaintext +query WorkItem($workItemId: ID!) { + workItem(workItemId: $id) @client { + id + type + widgets { + nodes { + type + } + } + } +} +``` + +### GraphQL queries and mutations + +GraphQL queries and mutations are work item agnostic. Work item queries and mutations +should happen at the widget level, so widgets are standalone reusable components. +The work item query and mutation should support any work item type and be dynamic. +They should allow you to query and mutate any work item attribute by specifying a widget identifier. + +In this query example, the description widget uses the query and mutation to +display and update the description of any work item: + +```plaintext +query { + workItem(input: { + workItemId: "gid://gitlab/AnyWorkItem/2207", + widgetIdentifier: "description", + }) { + id + type + widgets { + nodes { + ... on DescriptionWidget { + contentText + } + } + } + } +} + +``` + +Mutation example: + +```plaintext +mutation { + updateWorkItem(input: { + workItemId: "gid://gitlab/AnyWorkItem/2207", + widgetIdentifier: "description", + value: "the updated description" + }) { + workItem { + id + description + } + } +} + +``` + +### Widget's responsibility and structure + +A widget is responsible for displaying and updating a single attribute, such as +title, description, or labels. Widgets must support any type of work item. +To maximize component reusability, widgets should be field wrappers owning the +work item query and mutation of the attribute it's responsible for. + +A field component is a generic and simple component. It has no knowledge of the +attribute or work item details, such as input field, date selector, or dropdown. + +Widgets must be configurable to support various use cases, depending on work items. +When building widgets, use slots to provide extra context while minimizing +the use of props and injected attributes. + +### Examples + +We have a [dropdown field component](https://gitlab.com/gitlab-org/gitlab/-/blob/eea9ad536fa2d28ee6c09ed7d9207f803142eed7/app/assets/javascripts/vue_shared/components/dropdown/dropdown_widget/dropdown_widget.vue) +for use as reference. + +Any work item widget can wrap the dropdown component. The widget has knowledge of +the attribute it mutates, and owns the mutation for it. Multiple widgets can use +the same field component. For example: + +- Title and description widgets use the input field component. +- Start and end date use the date selector component. +- Labels, milestones, and assignees selectors use the dropdown component. + +Some frontend widgets already use the dropdown component. Use them as a reference +for work items widgets development: + +- `ee/app/assets/javascripts/boards/components/assignee_select.vue` +- `ee/app/assets/javascripts/boards/components/milestone_select.vue` |
