diff options
Diffstat (limited to 'doc/development')
126 files changed, 4397 insertions, 2363 deletions
diff --git a/doc/development/README.md b/doc/development/README.md index b041e48f0c2..22cc21e12b9 100644 --- a/doc/development/README.md +++ b/doc/development/README.md @@ -9,7 +9,7 @@ description: 'Learn how to contribute to GitLab.' - Set up GitLab's development environment with [GitLab Development Kit (GDK)](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/README.md) - [GitLab contributing guide](contributing/index.md) - - [Issues workflow](contributing/issue_workflow.md). For information on: + - [Issues workflow](contributing/issue_workflow.md) for more information on: - Issue tracker guidelines. - Triaging. - Labels. @@ -17,7 +17,7 @@ description: 'Learn how to contribute to GitLab.' - Issue weight. - Regression issues. - Technical or UX debt. - - [Merge requests workflow](contributing/merge_request_workflow.md). For + - [Merge requests workflow](contributing/merge_request_workflow.md) for more information on: - Merge request guidelines. - Contribution acceptance criteria. @@ -57,10 +57,8 @@ Complementary reads: - [GitLab utilities](utilities.md) - [Issuable-like Rails models](issuable-like-models.md) - [Logging](logging.md) -- [API styleguide](api_styleguide.md) Use this styleguide if you are - contributing to the API -- [GraphQL API styleguide](api_graphql_styleguide.md) Use this - styleguide if you are contributing to the [GraphQL API](../api/graphql/index.md) +- [API style guide](api_styleguide.md) for contributing to the API +- [GraphQL API style guide](api_graphql_styleguide.md) for contributing to the [GraphQL API](../api/graphql/index.md) - [Sidekiq guidelines](sidekiq_style_guide.md) for working with Sidekiq workers - [Working with Gitaly](gitaly.md) - [Manage feature flags](feature_flags/index.md) @@ -100,6 +98,7 @@ Complementary reads: - [Rails initializers](rails_initializers.md) - [Code comments](code_comments.md) - [Renaming features](renaming_features.md) +- [Windows Development on GCP](windows.md) ## Performance guides @@ -117,7 +116,7 @@ Complementary reads: ### Tooling - [Understanding EXPLAIN plans](understanding_explain_plans.md) -- [explain.depesz.com](https://explain.depesz.com/) for visualising the output +- [explain.depesz.com](https://explain.depesz.com/) for visualizing the output of `EXPLAIN` - [pgFormatter](http://sqlformat.darold.net/) a PostgreSQL SQL syntax beautifier @@ -178,7 +177,7 @@ Complementary reads: ## Documentation guides - [Writing documentation](documentation/index.md) -- [Documentation styleguide](documentation/styleguide.md) +- [Documentation style guide](documentation/styleguide.md) - [Markdown](../user/markdown.md) ## Internationalization (i18n) guides @@ -189,11 +188,11 @@ Complementary reads: ## Telemetry guides -- [Introduction](../telemetry/index.md) -- [Frontend tracking guide](../telemetry/frontend.md) -- [Backend tracking guide](../telemetry/backend.md) +- [Telemetry guide](telemetry/index.md) +- [Usage Ping guide](telemetry/usage_ping.md) +- [Snowplow guide](telemetry/snowplow.md) -## Experiment Guide +## Experiment guide - [Introduction](experiment_guide/index.md) @@ -221,9 +220,10 @@ Complementary reads: - [Defining relations between files using projections](projections.md) - [Reference processing](./reference_processing.md) +- [Compatibility with multiple versions of the application running at the same time](multi_version_compatibility.md) ## Other GitLab Development Kit (GDK) guides - [Run full Auto DevOps cycle in a GDK instance](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/auto_devops.md) -- [Using GitLab Runner with GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/runner.md) -- [Using the Web IDE terminal with GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/master/doc/howto/web_ide_terminal_gdk_setup.md) +- [Using GitLab Runner with the GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/runner.md) +- [Using the Web IDE terminal with the GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/master/doc/howto/web_ide_terminal_gdk_setup.md) diff --git a/doc/development/api_graphql_styleguide.md b/doc/development/api_graphql_styleguide.md index 036eddd7c37..6d3c0cf0eac 100644 --- a/doc/development/api_graphql_styleguide.md +++ b/doc/development/api_graphql_styleguide.md @@ -1,6 +1,6 @@ -# GraphQL API +# GraphQL API style guide -This document outlines the styleguide for GitLab's [GraphQL API](../api/graphql/index.md). +This document outlines the style guide for GitLab's [GraphQL API](../api/graphql/index.md). ## How GitLab implements GraphQL @@ -12,7 +12,7 @@ which is exposed as an API endpoint at `/api/graphql`. ## Deep Dive -In March 2019, Nick Thomas hosted a [Deep Dive](https://gitlab.com/gitlab-org/create-stage/issues/1) +In March 2019, Nick Thomas hosted a Deep Dive (GitLab team members only: `https://gitlab.com/gitlab-org/create-stage/issues/1`) on GitLab's [GraphQL API](../api/graphql/index.md) to share his domain specific knowledge with anyone who may work in this part of the code base in the future. You can find the [recording on YouTube](https://www.youtube.com/watch?v=-9L_1MWrjkg), and the slides on @@ -102,7 +102,7 @@ be `id` fields. Further reading: - [GraphQL Best Practices Guide](https://graphql.org/learn/best-practices/#nullability) -- [Using nullability in GraphQL](https://blog.apollographql.com/using-nullability-in-graphql-2254f84c4ed7) +- [Using nullability in GraphQL](https://www.apollographql.com/blog/using-nullability-in-graphql-2254f84c4ed7) ### Exposing Global IDs @@ -110,7 +110,7 @@ When exposing an `ID` field on a type, we will by default try to expose a global ID by calling `to_global_id` on the resource being rendered. -To override this behaviour, you can implement an `id` method on the +To override this behavior, you can implement an `id` method on the type for which you are exposing an ID. Please make sure that when exposing a `GraphQL::ID_TYPE` using a custom method that it is globally unique. @@ -365,7 +365,7 @@ field :token, GraphQL::STRING_TYPE, null: true, The original `description:` of the field should be maintained, and should _not_ be updated to mention the deprecation. -### Deprecation reason styleguide +### Deprecation reason style guide Where the reason for deprecation is due to the field being replaced with another field, the `reason` must be: @@ -446,7 +446,7 @@ Descriptions of fields and arguments are viewable to users through: - The [GraphiQL explorer](#graphiql). - The [static GraphQL API reference](../api/graphql/#reference). -### Description styleguide +### Description style guide To ensure consistency, the following should be followed whenever adding or updating descriptions: @@ -598,7 +598,7 @@ ID. Otherwise use a [globally unique ID](#exposing-global-ids). We already have a `FullPathLoader` that can be included in other resolvers to quickly find Projects and Namespaces which will have a -lot of dependant objects. +lot of dependent objects. To limit the amount of queries performed, we can use `BatchLoader`. diff --git a/doc/development/api_styleguide.md b/doc/development/api_styleguide.md index 78e35023766..e9a41e7ec58 100644 --- a/doc/development/api_styleguide.md +++ b/doc/development/api_styleguide.md @@ -1,6 +1,6 @@ -# API styleguide +# API style guide -This styleguide recommends best practices for API development. +This style guide recommends best practices for API development. ## Instance variables @@ -9,7 +9,7 @@ to access them as we do in Rails views), local variables are fine. ## Entities -Always use an [Entity] to present the endpoint's payload. +Always use an [Entity](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/api/entities) to present the endpoint's payload. ## Documentation @@ -29,7 +29,7 @@ for a good example): - If the endpoint is deprecated, and if so, when will it be removed - `params` for the method parameters. This acts as description, - [validation, and coercion of the parameters] + [validation, and coercion of the parameters](https://github.com/ruby-grape/grape#parameter-validation-and-coercion) A good example is as follows: @@ -100,13 +100,13 @@ Model.create(foo: params[:foo]) ## Using HTTP status helpers -For non-200 HTTP responses, use the provided helpers in `lib/api/helpers.rb` to ensure correct behaviour (`not_found!`, `no_content!` etc.). These will `throw` inside Grape and abort the execution of your endpoint. +For non-200 HTTP responses, use the provided helpers in `lib/api/helpers.rb` to ensure correct behavior (`not_found!`, `no_content!` etc.). These will `throw` inside Grape and abort the execution of your endpoint. For `DELETE` requests, you should also generally use the `destroy_conditionally!` helper which by default returns a `204 No Content` response on success, or a `412 Precondition Failed` response if the given `If-Unmodified-Since` header is out of range. This helper calls `#destroy` on the passed resource, but you can also implement a custom deletion method by passing a block. ## Using API path helpers in GitLab Rails codebase -Because we support [installing GitLab under a relative URL], one must take this +Because we support [installing GitLab under a relative URL](../install/relative_url.md), one must take this into account when using API path helpers generated by Grape. Any such API path helper usage must be in wrapped into the `expose_path` helper call. @@ -121,10 +121,6 @@ For instance: The [internal API](./internal_api.md) is documented for internal use. Please keep it up to date so we know what endpoints different components are making use of. -[Entity]: https://gitlab.com/gitlab-org/gitlab/blob/master/lib/api/entities -[validation, and coercion of the parameters]: https://github.com/ruby-grape/grape#parameter-validation-and-coercion -[installing GitLab under a relative URL]: https://docs.gitlab.com/ee/install/relative_url.html - ## Avoiding N+1 problems In order to avoid N+1 problems that are common when returning collections @@ -146,7 +142,7 @@ data](https://gitlab.com/gitlab-org/gitlab/blob/19f74903240e209736c7668132e6a5a7 for `Todo` _targets_ when returned in the Todos API. For more context and discussion about preloading see -[this merge request](https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/25711) +[this merge request](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/25711) which introduced the scope. ### Verifying with tests diff --git a/doc/development/application_limits.md b/doc/development/application_limits.md index b6e777dee15..b13e2994c52 100644 --- a/doc/development/application_limits.md +++ b/doc/development/application_limits.md @@ -11,7 +11,7 @@ coordinate with others to [document](../administration/instance_limits.md) and communicate those limits. There is a guide about [introducing application -limits](https://about.gitlab.com/handbook/product/#introducing-application-limits). +limits](https://about.gitlab.com/handbook/product/product-management/process/index.html#introducing-application-limits). ## Development @@ -90,7 +90,7 @@ project.actual_limits.exceeded?(:project_hooks, 10) #### `Limitable` concern -The [`Limitable` concern](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/app/models/concerns/limitable.rb) +The [`Limitable` concern](https://gitlab.com/gitlab-org/gitlab/blob/master/app/models/concerns/limitable.rb) can be used to validate that a model does not exceed the limits. It ensures that the count of the records for the current model does not exceed the defined limit. diff --git a/doc/development/architecture.md b/doc/development/architecture.md index bbd5ca3c494..f52a5d30c1f 100644 --- a/doc/development/architecture.md +++ b/doc/development/architecture.md @@ -8,7 +8,7 @@ New versions of GitLab are released in stable branches and the master branch is For information, see the [GitLab Release Process](https://gitlab.com/gitlab-org/release/docs/-/tree/master#gitlab-release-process). -Both EE and CE require some add-on components called GitLab Shell and Gitaly. These components are available from the [GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell/-/tree/master) and [Gitaly](https://gitlab.com/gitlab-org/gitaly/-/tree/master) repositories respectively. New versions are usually tags but staying on the master branch will give you the latest stable version. New releases are generally around the same time as GitLab CE releases with exception for informal security updates deemed critical. +Both EE and CE require some add-on components called GitLab Shell and Gitaly. These components are available from the [GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell/-/tree/master) and [Gitaly](https://gitlab.com/gitlab-org/gitaly/-/tree/master) repositories respectively. New versions are usually tags but staying on the master branch will give you the latest stable version. New releases are generally around the same time as GitLab CE releases with the exception of informal security updates deemed critical. ## Components diff --git a/doc/development/auto_devops.md b/doc/development/auto_devops.md index dba68974dd5..7d0c020ef96 100644 --- a/doc/development/auto_devops.md +++ b/doc/development/auto_devops.md @@ -1,7 +1,11 @@ # Auto DevOps development guide This document provides a development guide for contributors to -[Auto DevOps](../topics/autodevops/index.md) +[Auto DevOps](../topics/autodevops/index.md). + +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +An [Auto DevOps technical walk-through](https://youtu.be/G7RTLeToz9E) +is also available on YouTube. ## Development diff --git a/doc/development/background_migrations.md b/doc/development/background_migrations.md index c28008c94b8..0747224db30 100644 --- a/doc/development/background_migrations.md +++ b/doc/development/background_migrations.md @@ -157,7 +157,7 @@ roughly be as follows: enough times to be marked as dead. 1. Remove the old column. -This may also require a bump to the [import/export version][import-export], if +This may also require a bump to the [import/export version](../user/project/settings/import_export.md), if importing a project from a prior version of GitLab requires the data to be in the new format. @@ -283,7 +283,7 @@ end The final step runs for any un-migrated rows after all of the jobs have been processed. This is in case a Sidekiq process running the background migrations received SIGKILL, leading to the jobs being lost. (See -[more reliable Sidekiq queue][reliable-sidekiq] for more information.) +[more reliable Sidekiq queue](https://gitlab.com/gitlab-org/gitlab-foss/issues/36791) for more information.) If the application does not depend on the data being 100% migrated (for instance, the data is advisory, and not mission-critical), then this final step @@ -312,7 +312,7 @@ to migrate you database down and up, which can result in other background migrations being called. That means that using `spy` test doubles with `have_received` is encouraged, instead of using regular test doubles, because your expectations defined in a `it` block can conflict with what is being -called in RSpec hooks. See [issue #35351][issue-rspec-hooks] +called in RSpec hooks. See [issue #35351](https://gitlab.com/gitlab-org/gitlab/issues/18839) for more details. ## Best practices @@ -329,8 +329,3 @@ for more details. 1. Make sure to discuss the numbers with a database specialist, the migration may add more pressure on DB than you expect (measure on staging, or ask someone to measure on production). - -[migrations-readme]: https://gitlab.com/gitlab-org/gitlab/blob/master/spec/migrations/README.md -[issue-rspec-hooks]: https://gitlab.com/gitlab-org/gitlab/issues/18839 -[reliable-sidekiq]: https://gitlab.com/gitlab-org/gitlab-foss/issues/36791 -[import-export]: ../user/project/settings/import_export.md diff --git a/doc/development/changelog.md b/doc/development/changelog.md index 2007c26403c..a0e9f9d3be3 100644 --- a/doc/development/changelog.md +++ b/doc/development/changelog.md @@ -101,20 +101,20 @@ automatically. Its simplest usage is to provide the value for `title`: -```text +```plaintext bin/changelog 'Hey DZ, I added a feature to GitLab!' ``` If you want to generate a changelog entry for GitLab EE, you will need to pass the `--ee` option: -```text +```plaintext bin/changelog --ee 'Hey DZ, I added a feature to GitLab!' ``` At this point the script would ask you to select the category of the change (mapped to the `type` field in the entry): -```text +```plaintext >> Please specify the category of your change: 1. New feature 2. Bug fix @@ -132,7 +132,7 @@ the command above on a branch called `feature/hey-dz`, it will generate a The command will output the path of the generated file and its contents: -```text +```plaintext create changelogs/unreleased/my-feature.yml --- title: Hey DZ, I added a feature to GitLab! @@ -162,7 +162,7 @@ If you use **`--amend`** and don't provide a title, it will automatically use the "subject" of the previous commit, which is the first line of the commit message: -```text +```plaintext $ git show --oneline ab88683 Added an awesome new feature to GitLab @@ -180,7 +180,7 @@ type: Use **`--force`** or **`-f`** to overwrite an existing changelog entry if it already exists. -```text +```plaintext $ bin/changelog 'Hey DZ, I added a feature to GitLab!' error changelogs/unreleased/feature-hey-dz.yml already exists! Use `--force` to overwrite. @@ -198,7 +198,7 @@ type: Use the **`--merge-request`** or **`-m`** argument to provide the `merge_request` value: -```text +```plaintext $ bin/changelog 'Hey DZ, I added a feature to GitLab!' -m 1983 create changelogs/unreleased/feature-hey-dz.yml --- @@ -213,7 +213,7 @@ type: Use the **`--dry-run`** or **`-n`** argument to prevent actually writing or committing anything: -```text +```plaintext $ bin/changelog --amend --dry-run create changelogs/unreleased/feature-hey-dz.yml --- @@ -230,7 +230,7 @@ $ ls changelogs/unreleased/ Use the **`--git-username`** or **`-u`** argument to automatically fill in the `author` value with your configured Git `user.name` value: -```text +```plaintext $ git config user.name Jane Doe @@ -247,7 +247,7 @@ type: Use the **`--type`** or **`-t`** argument to provide the `type` value: -```text +```plaintext $ bin/changelog 'Hey DZ, I added a feature to GitLab!' -t added create changelogs/unreleased/feature-hey-dz.yml --- diff --git a/doc/development/cicd/img/ci_architecture.png b/doc/development/cicd/img/ci_architecture.png Binary files differnew file mode 100644 index 00000000000..b888f2f07aa --- /dev/null +++ b/doc/development/cicd/img/ci_architecture.png diff --git a/doc/development/cicd/index.md b/doc/development/cicd/index.md index ed33eb01308..db3f58d1d22 100644 --- a/doc/development/cicd/index.md +++ b/doc/development/cicd/index.md @@ -2,6 +2,81 @@ Development guides that are specific to CI/CD are listed here. +## CI Architecture overview + +The following is a simplified diagram of the CI architecture. Some details are left out in order to focus on +the main components. + + +<!-- Editable diagram available at https://app.diagrams.net/#G1LFl-KW4fgpBPzz8VIH9rsOlAH4t0xwKj --> + +On the left side we have the events that can trigger a pipeline based on various events (trigged by a user or automation): + +- A `git push` is the most common event that triggers a pipeline. +- The [Web API](../../api/pipelines.md#create-a-new-pipeline). +- A user clicking the "Run Pipeline" button in the UI. +- When a [merge request is created or updated](../../ci/merge_request_pipelines/index.md#pipelines-for-merge-requests). +- When an MR is added to a [Merge Train](../../ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md#merge-trains-premium). +- A [scheduled pipeline](../../ci/pipelines/schedules.md#pipeline-schedules). +- When project is [subscribed to an upstream project](../../ci/multi_project_pipelines.md#trigger-a-pipeline-when-an-upstream-project-is-rebuilt). +- When [Auto DevOps](../../topics/autodevops/index.md) is enabled. +- When GitHub integration is used with [external pull requests](../../ci/ci_cd_for_external_repos/index.md#pipelines-for-external-pull-requests). +- When an upstream pipeline contains a [bridge job](../../ci/yaml/README.md#trigger) which triggers a downstream pipeline. + +Triggering any of these events will invoke the [`CreatePipelineService`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/ci/create_pipeline_service.rb) +which takes as input event data and the user triggering it, then will attempt to create a pipeline. + +The `CreatePipelineService` relies heavily on the [`YAML Processor`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/yaml_processor.rb) +component, which is responsible for taking in a YAML blob as input and returns the abstract data structure of a +pipeline (including stages and all jobs). This component also validates the structure of the YAML while +processing it, and returns any syntax or semantic errors. The `YAML Processor` component is where we define +[all the keywords](../../ci/yaml/README.md) available to structure a pipeline. + +The `CreatePipelineService` receives the abstract data structure returned by the `YAML Processor`, +which then converts it to persisted models (pipeline, stages, jobs, etc.). After that, the pipeline is ready +to be processed. Processing a pipeline means running the jobs in order of execution (stage or DAG) +until either one of the following: + +- All expected jobs have been executed. +- Failures interrupt the pipeline execution. + +The component that processes a pipeline is [`ProcessPipelineService`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/ci/process_pipeline_service.rb), +which is responsible for moving all the pipeline's jobs to a completed state. When a pipeline is created, all its +jobs are initially in `created` state. This services looks at what jobs in `created` stage are eligible +to be processed based on the pipeline structure. Then it moves them into the `pending` state, which means +they can now [be picked up by a Runner](#job-scheduling). After a job has been executed it can complete +successfully or fail. Each status transition for job within a pipeline triggers this service again, which +looks for the next jobs to be transitioned towards completion. While doing that, `ProcessPipelineService` +updates the status of jobs, stages and the overall pipeline. + +On the right side of the diagram we have a list of [Runners](../../ci/runners/README.md#configuring-gitlab-runners) +connected to the GitLab instance. These can be Shared Runners, Group Runners or Project-specific Runners. +The communication between Runners and the Rails server occurs through a set of API endpoints, grouped as +the `Runner API Gateway`. + +We can register, delete and verify Runners, which also causes read/write queries to the database. After a Runner is connected, +it keeps asking for the next job to execute. This invokes the [`RegisterJobService`](https://gitlab.com/gitlab-org/gitlab/blob/master/app/services/ci/register_job_service.rb) +which will pick the next job and assign it to the Runner. At this point the job will transition to a +`running` state, which again triggers `ProcessPipelineService` due to the status change. +For more details read [Job scheduling](#job-scheduling)). + +While a job is being executed, the Runner sends logs back to the server as well any possible artifacts +that need to be stored. Also, a job may depend on artifacts from previous jobs in order to run. In this +case the Runner will download them using a dedicated API endpoint. + +Artifacts are stored in object storage, while metadata is kept in the database. An important example of artifacts +is reports (JUnit, SAST, DAST, etc.) which are parsed and rendered in the merge request. + +Job status transitions are not all automated. A user may run [manual jobs](../../ci/yaml/README.md#whenmanual), cancel a pipeline, retry +specific failed jobs or the entire pipeline. Anything that +causes a job to change status will trigger `ProcessPipelineService`, as it's responsible for +tracking the status of the entire pipeline. + +A special type of job is the [bridge job](../../ci/yaml/README.md#trigger) which is executed server-side +when transitioning to the `pending` state. This job is responsible for creating a downstream pipeline, such as +a multi-project or child pipeline. The workflow loop starts again +from the `CreatePipelineService` every time a downstream pipeline is triggered. + ## Job scheduling When a Pipeline is created all its jobs are created at once for all stages, with an initial state of `created`. This makes it possible to visualize the full content of a pipeline. diff --git a/doc/development/code_review.md b/doc/development/code_review.md index 5220f29bd60..a5ad7dc0f46 100644 --- a/doc/development/code_review.md +++ b/doc/development/code_review.md @@ -64,7 +64,7 @@ It picks reviewers and maintainers from the list at the page, with these behaviors: 1. It will not pick people whose [GitLab status](../user/profile/index.md#current-status) - contains the string 'OOO'. + contains the string 'OOO', or the emoji is `:palm_tree:` or `:beach:`. 1. [Trainee maintainers](https://about.gitlab.com/handbook/engineering/workflow/code-review/#trainee-maintainer) are three times as likely to be picked as other reviewers. 1. It always picks the same reviewers and maintainers for the same @@ -76,26 +76,36 @@ page, with these behaviors: As described in the section on the responsibility of the maintainer below, you are recommended to get your merge request approved and merged by maintainer(s) - with [domain expertise](#domain-experts). +with [domain expertise](#domain-experts). -1. If your merge request includes backend changes [^1], it must be +1. If your merge request includes backend changes (*1*), it must be **approved by a [backend maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_maintainers_backend)**. -1. If your merge request includes database migrations or changes to expensive queries [^2], it must be +1. If your merge request includes database migrations or changes to expensive queries (*2*), it must be **approved by a [database maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_maintainers_database)**. Read the [database review guidelines](database_review.md) for more details. -1. If your merge request includes frontend changes [^1], it must be +1. If your merge request includes frontend changes (*1*), it must be **approved by a [frontend maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_maintainers_frontend)**. -1. If your merge request includes UX changes [^1], it must be +1. If your merge request includes UX changes (*1*), it must be **approved by a [UX team member](https://about.gitlab.com/company/team/)**. -1. If your merge request includes adding a new JavaScript library [^1], it must be +1. If your merge request includes adding a new JavaScript library (*1*), it must be **approved by a [frontend lead](https://about.gitlab.com/company/team/)**. -1. If your merge request includes adding a new UI/UX paradigm [^1], it must be +1. If your merge request includes adding a new UI/UX paradigm (*1*), it must be **approved by a [UX lead](https://about.gitlab.com/company/team/)**. 1. If your merge request includes a new dependency or a filesystem change, it must be **approved by a [Distribution team member](https://about.gitlab.com/company/team/)**. See how to work with the [Distribution team](https://about.gitlab.com/handbook/engineering/development/enablement/distribution/#how-to-work-with-distribution) for more details. 1. If your merge request includes documentation changes, it must be **approved by a [Technical writer](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers)**, based on the appropriate [product category](https://about.gitlab.com/handbook/product/categories/). +1. If your merge request includes Quality and non-Quality-related changes (*3*), it must be **approved + by a [Software Engineer in Test](https://about.gitlab.com/handbook/engineering/quality/#individual-contributors)**. +1. If your merge request includes _only_ Quality-related changes (*3*), it must be **approved + by a [Quality maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_maintainers_qa)**. + +- (*1*): Please note that specs other than JavaScript specs are considered backend code. +- (*2*): We encourage you to seek guidance from a database maintainer if your merge + request is potentially introducing expensive queries. It is most efficient to comment + on the line of code in question with the SQL queries so they can give their advice. +- (*3*): Quality-related changes include all files within the `qa` directory. #### Security requirements @@ -314,7 +324,25 @@ Before taking the decision to merge: - Set the milestone. - Consider warnings and errors from danger bot, code quality, and other reports. Unless a strong case can be made for the violation, these should be resolved - before merge. A comment must to be posted if the MR is merged with any failed job. + before merging. A comment must to be posted if the MR is merged with any failed job. +- If the MR contains both Quality and non-Quality-related changes, the MR should be merged by the relevant maintainer for user-facing changes (backend, frontend, or database) after the Quality related changes are approved by a Software Engineer in Test. + +If a merge request is fundamentally ready, but needs only trivial fixes (such as +typos), consider demonstrating a [bias for +action](https://about.gitlab.com/handbook/values/#bias-for-action) by making +those changes directly without going back to the author. You can do this by +using the [suggest changes](../user/discussions/index.md#suggest-changes) feature to apply +your own suggestions to the merge request. Note that: + +- If the changes are not straightforward, please prefer assigning the merge request back + to the author. +- **Before applying suggestions**, edit the merge request to make sure + [squash and + merge](../user/project/merge_requests/squash_and_merge.md#squash-and-merge) + is enabled, otherwise, the pipeline's Danger job will fail. + - If a merge request does not have squash and merge enabled, and it + has more than one commit, then see the note below about rewriting + commit history. When ready to merge: @@ -463,6 +491,21 @@ When a merge request author has been blocked for longer than the `Review-response` SLO, they are free to remind the reviewer through Slack or assign another reviewer. +### Customer critical merge requests + +A merge request may benefit from being considered a customer critical priority because there is a significant benefit to the business in doing so. + +Properties of customer critical merge requests: + +- The [Senior Director of Development](https://about.gitlab.com/job-families/engineering/engineering-management/#senior-director-engineering) ([@clefelhocz1](https://gitlab.com/clefelhocz1)) is the DRI for deciding if a merge request will be customer critical. +- The DRI will assign the `customer-critical-merge-request` label to the merge request. +- It is required that the reviewer(s) and maintainer(s) involved with a customer critical merge request are engaged as soon as this decision is made. +- It is required to prioritize work for those involved on a customer critical merge request so that they have the time available necessary to focus on it. +- It is required to adhere to GitLab [values](https://about.gitlab.com/handbook/values.md) and processes when working on customer critical merge requests, taking particular note of family and friends first/work second, definition of done, iteration, and release when it's ready. +- Customer critical merge requests are required to not reduce security, introduce data-loss risk, reduce availability, nor break existing functionality per the process for [prioritizing technical decisions](https://about.gitlab.com/handbook/engineering/#prioritizing-technical-decisions.md). +- On customer critical requests, it is _recommended_ that those involved _consider_ coordinating synchronously (Zoom, Slack) in addition to asynchronously (merge requests comments) if they believe this will reduce elapsed time to merge even though this _may_ sacrifice [efficiency](https://about.gitlab.com/company/culture/all-remote/asynchronous/#evaluating-efficiency.md). +- After a customer critical merge request is merged, a retrospective must be completed with the intention of reducing the frequency of future customer critical merge requests. + ## Examples How code reviews are conducted can surprise new contributors. Here are some examples of code reviews that should help to orient you as to what to expect. @@ -495,6 +538,3 @@ Largely based on the [thoughtbot code review guide](https://github.com/thoughtbo --- [Return to Development documentation](README.md) - -[^1]: Please note that specs other than JavaScript specs are considered backend code. -[^2]: We encourage you to seek guidance from a database maintainer if your merge request is potentially introducing expensive queries. It is most efficient to comment on the line of code in question with the SQL queries so they can give their advice. diff --git a/doc/development/contributing/index.md b/doc/development/contributing/index.md index 8270e52e0ab..aebff633f58 100644 --- a/doc/development/contributing/index.md +++ b/doc/development/contributing/index.md @@ -3,35 +3,40 @@ Thank you for your interest in contributing to GitLab. This guide details how to contribute to GitLab in a way that is easy for everyone. -For a first-time step-by-step guide to the contribution process, please see -["Contributing to GitLab"](https://about.gitlab.com/community/contribute/). +For a first-time step-by-step guide to the contribution process, see our +[Contributing to GitLab](https://about.gitlab.com/community/contribute/) page. -Looking for something to work on? Look for issues with the label [`Accepting merge requests`](#i-want-to-contribute). +Looking for something to work on? Look for issues with the label +[`~Accepting merge requests`](#how-to-contribute). -GitLab comes in two flavors, GitLab Community Edition (CE) our free and open -source edition, and GitLab Enterprise Edition (EE) which is our commercial -edition. Throughout this guide you will see references to CE and EE for -abbreviation. +GitLab comes in two flavors: -To get an overview of GitLab community membership including those that would be reviewing or merging your contributions, please visit [the community roles page](community_roles.md). +- GitLab Community Edition (CE), our free and open source edition. +- GitLab Enterprise Edition (EE), which is our commercial edition. + +Throughout this guide you will see references to CE and EE for abbreviation. + +To get an overview of GitLab community membership, including those that would review or merge +your contributions, visit [the community roles page](community_roles.md). If you want to know how the GitLab [core team](https://about.gitlab.com/community/core-team/) -operates please see [the GitLab contributing process](https://gitlab.com/gitlab-org/gitlab/blob/master/PROCESS.md). +operates, see [the GitLab contributing process](https://gitlab.com/gitlab-org/gitlab/blob/master/PROCESS.md). -[GitLab Inc engineers should refer to the engineering workflow document](https://about.gitlab.com/handbook/engineering/workflow/) +GitLab Inc engineers should refer to the [engineering workflow document](https://about.gitlab.com/handbook/engineering/workflow/). ## Security vulnerability disclosure -Please report suspected security vulnerabilities in private to +Report suspected security vulnerabilities in private to `support@gitlab.com`, also see the [disclosure section on the GitLab.com website](https://about.gitlab.com/security/disclosure/). -Please do **NOT** create publicly viewable issues for suspected security -vulnerabilities. + +DANGER: **Danger:** +Do **NOT** create publicly viewable issues for suspected security vulnerabilities. ## Code of conduct We want to create a welcoming environment for everyone who is interested in contributing. -Please visit our [Code of Conduct page](https://about.gitlab.com/community/contribute/code-of-conduct/) to learn more about our commitment to an open and welcoming environment. +Visit our [Code of Conduct page](https://about.gitlab.com/community/contribute/code-of-conduct/) to learn more about our commitment to an open and welcoming environment. ## Closing policy for issues and merge requests @@ -40,39 +45,56 @@ and merge requests is limited. Out of respect for our volunteers, issues and merge requests not in line with the guidelines listed in this document may be closed without notice. -Please treat our volunteers with courtesy and respect, it will go a long way +Treat our volunteers with courtesy and respect, it will go a long way towards getting your issue resolved. Issues and merge requests should be in English and contain appropriate language for audiences of all ages. -If a contributor is no longer actively working on a submitted merge request -we can decide that the merge request will be finished by one of our -[Merge request coaches](https://about.gitlab.com/company/team/) or close the merge request. We make this decision -based on how important the change is for our product vision. If a merge request -coach is going to finish the merge request we assign the -~"coach will finish" label. When a team member picks up a community contribution, +If a contributor is no longer actively working on a submitted merge request, +we can: + +- Decide that the merge request will be finished by one of our + [Merge request coaches](https://about.gitlab.com/company/team/). +- Close the merge request. + +We make this decision based on how important the change is for our product vision. If a merge +request coach is going to finish the merge request, we assign the +`~coach will finish` label. + +When a team member picks up a community contribution, we credit the original author by adding a changelog entry crediting the author and optionally include the original author on at least one of the commits within the MR. ## Helping others -Please help other GitLab users when you can. -The methods people will use to seek help can be found on the [getting help page](https://about.gitlab.com/get-help/). +Help other GitLab users when you can. +The methods people use to seek help can be found on the [getting help page](https://about.gitlab.com/get-help/). -Sign up for the mailing list, answer GitLab questions on StackOverflow or -respond in the IRC channel. +Sign up for the mailing list, answer GitLab questions on StackOverflow or respond in the IRC channel. -## I want to contribute +## How to contribute If you want to contribute to GitLab, -[issues with the `Accepting merge requests` label](issue_workflow.md#label-for-community-contributors) +[issues with the `~Accepting merge requests` label](issue_workflow.md#label-for-community-contributors) are a great place to start. + If you have any questions or need help visit [Getting Help](https://about.gitlab.com/get-help/) to -learn how to communicate with GitLab. If you're looking for a Gitter or Slack channel -please consider we favor -[asynchronous communication](https://about.gitlab.com/handbook/communication/#internal-communication) over real time communication. Thanks for your contribution! +learn how to communicate with GitLab. We have a [Gitter channel for contributors](https://gitter.im/gitlab/contributors), +however we favor +[asynchronous communication](https://about.gitlab.com/handbook/communication/#internal-communication) over real time communication. + +Thanks for your contribution! + +### GitLab Development Kit + +The GitLab Development Kit (GDK) helps contributors run a local GitLab instance with all the +required dependencies. It can be used to test changes to GitLab and related projects before raising +a Merge Request. + +For more information, see the [`gitlab-development-kit`](https://gitlab.com/gitlab-org/gitlab-development-kit) +project. ## Contribution Flow @@ -92,7 +114,7 @@ When submitting code to GitLab, you may feel that your contribution requires the When your code contains more than 500 changes, any major breaking changes, or an external library, `@mention` a maintainer in the merge request. If you are not sure who to mention, the reviewer will add one early in the merge request process. -## Issues workflow +### Issues workflow This [documentation](issue_workflow.md) outlines the current issue workflow: @@ -105,7 +127,7 @@ This [documentation](issue_workflow.md) outlines the current issue workflow: - [Technical and UX debt](issue_workflow.md#technical-and-ux-debt) - [Technical debt in follow-up issues](issue_workflow.md#technical-debt-in-follow-up-issues) -## Merge requests workflow +### Merge requests workflow This [documentation](merge_request_workflow.md) outlines the current merge request process. @@ -120,13 +142,15 @@ This [documentation](style_guides.md) outlines the current style guidelines. ## Implement design & UI elements -This [design documentation](design.md) outlines the current process for implementing -design & UI elements. +This [design documentation](design.md) outlines the current process for implementing design and UI +elements. -## Getting an Enterprise Edition License +## Contribute documentation -If you need a license for contributing to an EE-feature, please [follow these instructions](https://about.gitlab.com/handbook/marketing/community-relations/code-contributor-program/#for-contributors-to-the-gitlab-enterprise-edition-ee). +For information on how to contribute documentation, see GitLab +[documentation guidelines](../documentation/index.md). ---- +## Getting an Enterprise Edition License -[Return to Development documentation](../README.md) +If you need a license for contributing to an EE-feature, see +[relevant information](https://about.gitlab.com/handbook/marketing/community-relations/code-contributor-program/#for-contributors-to-the-gitlab-enterprise-edition-ee). diff --git a/doc/development/contributing/issue_workflow.md b/doc/development/contributing/issue_workflow.md index 13ff35ed65c..be416bf636e 100644 --- a/doc/development/contributing/issue_workflow.md +++ b/doc/development/contributing/issue_workflow.md @@ -36,7 +36,7 @@ project. To allow for asynchronous issue handling, we use [milestones](https://gitlab.com/groups/gitlab-org/-/milestones) and [labels](https://gitlab.com/gitlab-org/gitlab/-/labels). Leads and product managers handle most of the -scheduling into milestones. Labelling is a task for everyone. +scheduling into milestones. Labeling is a task for everyone. Most issues will have labels for at least one of the following: @@ -156,9 +156,9 @@ As a team needs some way to collect the work their members are planning to be as Normally there is a 1:1 relationship between Stage labels and Group labels. In the spirit of "Everyone can contribute", any issue can be picked up by any group, depending on current priorities. When picking up an issue belonging to a different -group, it should be relabelled. For example, if an issue labelled `~"devops::create"` +group, it should be relabeled. For example, if an issue labeled `~"devops::create"` and `~"group::knowledge"` is picked up by someone in the Access group of the Plan stage, -the issue should be relabelled as `~"group::access"` while keeping the original +the issue should be relabeled as `~"group::access"` while keeping the original `~"devops::create"` unchanged. We also use stage and group labels to help quantify our [throughput](https://about.gitlab.com/handbook/engineering/management/throughput/). @@ -264,7 +264,7 @@ release. There are three levels of Release Scoping labels: milestone. If these issues are not done in the current release, they will strongly be considered for the next release. - ~"Next Patch Release": Issues to put in the next patch release. Work on these - first, and add the "Pick Into X" label to the merge request, along with the + first, and add the `~"Pick into X.Y"` label to the merge request, along with the appropriate milestone. Each issue scheduled for the current milestone should be labeled ~Deliverable @@ -347,7 +347,7 @@ there is the ~"stewardship" label. This label is to be used for issues in which the stewardship of GitLab is a topic of discussion. For instance if GitLab Inc. is planning to add -features from GitLab EE to GitLab CE, related issues would be labelled with +features from GitLab EE to GitLab CE, related issues would be labeled with ~"stewardship". A recent example of this was the issue for @@ -461,7 +461,7 @@ fixing in the same MR, but not worth creating a follow-up issue for. Renaming a method that is used in many places to make its intent slightly clearer may be worth fixing, but it should not happen in the same MR, and is generally not worth the overhead of having an issue of its own. These issues would invariably -be labelled `~P4 ~S4` if we were to create them. +be labeled `~P4 ~S4` if we were to create them. More severe technical debt can have implications for development velocity. If it isn't addressed in a timely manner, the codebase becomes needlessly difficult diff --git a/doc/development/contributing/merge_request_workflow.md b/doc/development/contributing/merge_request_workflow.md index 0ac08ec2eae..a70fadfe8a9 100644 --- a/doc/development/contributing/merge_request_workflow.md +++ b/doc/development/contributing/merge_request_workflow.md @@ -17,10 +17,10 @@ wireframes of the proposed feature if it will also change the UI. Merge requests should be submitted to the appropriate project at GitLab.com, for example [GitLab](https://gitlab.com/gitlab-org/gitlab/-/merge_requests), [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests), -[GitLab Omnibus](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests), etc. +[Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests), etc. If you are new to GitLab development (or web development in general), see the -[I want to contribute!](index.md#i-want-to-contribute) section to get started with +[how to contribute](index.md#how-to-contribute) section to get started with some potentially easy issues. To start developing GitLab, download the [GitLab Development Kit](https://gitlab.com/gitlab-org/gitlab-development-kit) @@ -138,7 +138,7 @@ For more information see [How to Write a Git Commit Message](https://chris.beams Example commit message template that can be used on your machine that embodies the above (guide for [how to apply template](https://codeinthehole.com/tips/a-useful-template-for-commit-messages/)): -```text +```plaintext # (If applied, this commit will...) <subject> (Max 50 char) # |<---- Using a Maximum Of 50 Characters ---->| @@ -244,7 +244,7 @@ request: 1. The [CI environment preparation](https://gitlab.com/gitlab-org/gitlab/blob/master/scripts/prepare_build.sh). 1. The [Omnibus package creator](https://gitlab.com/gitlab-org/omnibus-gitlab). -### Incremental improvements +## Incremental improvements We allow engineering time to fix small problems (with or without an issue) that are incremental improvements, such as: diff --git a/doc/development/contributing/style_guides.md b/doc/development/contributing/style_guides.md index 9a0b654f47f..9e4870eadc4 100644 --- a/doc/development/contributing/style_guides.md +++ b/doc/development/contributing/style_guides.md @@ -53,7 +53,7 @@ should enable all RuboCop rules to avoid style-related discussions/nitpicking/back-and-forth in reviews. Additionally, we have a dedicated -[newlines styleguide](../newlines_styleguide.md), as well as dedicated +[newlines style guide](../newlines_styleguide.md), as well as dedicated [test-specific style guides and best practices](../testing_guide/index.md). ## Database migrations diff --git a/doc/development/creating_enums.md b/doc/development/creating_enums.md index 79ed465b121..e2ebad538d9 100644 --- a/doc/development/creating_enums.md +++ b/doc/development/creating_enums.md @@ -13,3 +13,92 @@ def change add_column :ci_job_artifacts, :file_format, :integer, limit: 2 end ``` + +## All of the key/value pairs should be defined in FOSS + +**Summary:** All enums needs to be defined in FOSS, if a model is also part of the FOSS. + +```ruby +class Model < ApplicationRecord + enum platform: { + aws: 0, + gcp: 1 # EE-only + } +end +``` + +When you add a new key/value pair to a `enum` and if it's EE-specific, you might be +tempted to organize the `enum` as the following: + +```ruby +# Define `failure_reason` enum in `Pipeline` model: +class Pipeline < ApplicationRecord + enum failure_reason: ::PipelineEnums.failure_reasons +end +``` + +```ruby +# Define key/value pairs that used in FOSS and EE: +module PipelineEnums + def self.failure_reasons + { unknown_failure: 0, config_error: 1 } + end +end + +PipelineEnums.prepend_if_ee('EE::PipelineEnums') +``` + +```ruby +# Define key/value pairs that used in EE only: +module EE + module PipelineEnums + override :failure_reasons + def failure_reasons + super.merge(activity_limit_exceeded: 2) + end + end +end +``` + +This works as-is, however, it has a couple of downside that: + +- Someone could define a key/value pair in EE that is **conflicted** with a value defined in FOSS. + e.g. Define `activity_limit_exceeded: 1` in `EE::PipelineEnums`. +- When it happens, the feature works totally different. + e.g. We cannot figure out `failure_reason` is either `config_error` or `activity_limit_exceeded`. +- When it happens, we have to ship a database migration to fix the data integrity, + which might be impossible if you cannot recover the original value. + +Also, you might observe a workaround for this concern by setting an offset in EE's values. +For example, this example sets `1000` as the offset: + +```ruby +module EE + module PipelineEnums + override :failure_reasons + def failure_reasons + super.merge(activity_limit_exceeded: 1_000, size_limit_exceeded: 1_001) + end + end +end +``` + +This looks working as a workaround, however, this approach has some donwside that: + +- Features could move from EE to FOSS or vice versa. Therefore, the offset might be mixed between FOSS and EE in the future. + e.g. When you move `activity_limit_exceeded` to FOSS, you'll see `{ unknown_failure: 0, config_error: 1, activity_limit_exceeded: 1_000 }`. +- The integer column for the `enum` is likely created [as `SMALLINT`](#creating-enums). + Therefore, you need to be careful of that the offset doesn't exceed the maximum value of 2 bytes integer. + +As a conclusion, you should define all of the key/value pairs in FOSS. +For example, you can simply write the following code in the above case: + +```ruby +class Pipeline < ApplicationRecord + enum failure_reason: { + unknown_failure: 0, + config_error: 1, + activity_limit_exceeded: 2 + } +end +``` diff --git a/doc/development/dangerbot.md b/doc/development/dangerbot.md index e090281f2bf..eda9e1e21cc 100644 --- a/doc/development/dangerbot.md +++ b/doc/development/dangerbot.md @@ -65,7 +65,7 @@ First, be aware of GitLab's [commitment to dogfooding](https://about.gitlab.com/ The code we write for Danger is GitLab-specific, and it **may not** be most appropriate place to implement functionality that addresses a need we encounter. Our users, customers, and even our own satellite projects, such as [Gitaly](https://gitlab.com/gitlab-org/gitaly), -often face similar challenges, after all. Think about how you could fulfil the +often face similar challenges, after all. Think about how you could fulfill the same need while ensuring everyone can benefit from the work, and do that instead if you can. @@ -145,7 +145,7 @@ at GitLab so far: fork. That way the danger comments will be made from CI using that API token instead. Making the variable - [masked](../ci/variables/README.md#masked-variables) will make sure + [masked](../ci/variables/README.md#mask-a-custom-variable) will make sure it doesn't show up in the job logs. The variable cannot be - [protected](../ci/variables/README.md#protected-environment-variables), + [protected](../ci/variables/README.md#protect-a-custom-variable), as it needs to be present for all feature branches. diff --git a/doc/development/database/add_foreign_key_to_existing_column.md b/doc/development/database/add_foreign_key_to_existing_column.md index e08f0a3bd1e..b8817eddeec 100644 --- a/doc/development/database/add_foreign_key_to_existing_column.md +++ b/doc/development/database/add_foreign_key_to_existing_column.md @@ -64,7 +64,7 @@ class AddNotValidForeignKeyToEmailsUser < ActiveRecord::Migration[5.2] def up # safe to use: it requires short lock on the table since we don't validate the foreign key - add_foreign_key :emails, :users, on_delete: :cascade, validate: false # rubocop:disable Migration/AddConcurrentForeignKey + add_foreign_key :emails, :users, on_delete: :cascade, validate: false end def down diff --git a/doc/development/database_debugging.md b/doc/development/database_debugging.md index 46cb869fea3..3753baa3c63 100644 --- a/doc/development/database_debugging.md +++ b/doc/development/database_debugging.md @@ -5,27 +5,33 @@ run into some head-banging database problems. An easy first step is to search for your error in Slack, or search for `GitLab <my error>` with Google. ---- +Available `RAILS_ENV`: -Available `RAILS_ENV` - -- `production` (generally not for your main GDK db, but you may need this for e.g. Omnibus) -- `development` (this is your main GDK db) -- `test` (used for tests like rspec) +- `production` (generally not for your main GDK database, but you may need this for other installations such as Omnibus). +- `development` (this is your main GDK db). +- `test` (used for tests like RSpec). ## Nuke everything and start over -If you just want to delete everything and start over with an empty DB (~1 minute): +If you just want to delete everything and start over with an empty DB (approximately 1 minute): -- `bundle exec rake db:reset RAILS_ENV=development` +```shell +bundle exec rake db:reset RAILS_ENV=development +``` -If you just want to delete everything and start over with dummy data (~4 minutes). This also does `db:reset` and runs DB-specific migrations: +If you just want to delete everything and start over with dummy data (approximately 4 minutes). This +also does `db:reset` and runs DB-specific migrations: -- `bundle exec rake dev:setup RAILS_ENV=development` +```shell +bundle exec rake dev:setup RAILS_ENV=development +``` -If your test DB is giving you problems, it is safe to nuke it because it doesn't contain important data: +If your test DB is giving you problems, it is safe to nuke it because it doesn't contain important +data: -- `bundle exec rake db:reset RAILS_ENV=test` +```shell +bundle exec rake db:reset RAILS_ENV=test +``` ## Migration wrangling diff --git a/doc/development/database_review.md b/doc/development/database_review.md index f2db0ab4fd5..aa7ebb3756f 100644 --- a/doc/development/database_review.md +++ b/doc/development/database_review.md @@ -74,12 +74,12 @@ the following preparations into account. #### Preparation when adding migrations -- Ensure `db/structure.sql` is updated. +- Ensure `db/structure.sql` is updated as [documented](migration_style_guide.md#schema-changes). - Make migrations reversible by using the `change` method or include a `down` method when using `up`. - Include either a rollback procedure or describe how to rollback changes. -- Add the output of both migrating and rolling back for all migrations into the MR description - - Ensure the down method reverts the changes in `db/structure.sql` - - Update the migration output whenever you modify the migrations during the review process +- Add the output of both migrating and rolling back for all migrations into the MR description. + - Ensure the down method reverts the changes in `db/structure.sql`. + - Update the migration output whenever you modify the migrations during the review process. - Add tests for the migration in `spec/migrations` if necessary. See [Testing Rails migrations at GitLab](testing_guide/testing_migrations_guide.md) for more details. - When [high-traffic](https://gitlab.com/gitlab-org/gitlab/-/blob/master/rubocop/migration_helpers.rb#L12) tables are involved in the migration, use the [`with_lock_retries`](migration_style_guide.md#retry-mechanism-when-acquiring-database-locks) helper method. Review the relevant [examples in our documentation](migration_style_guide.md#examples) for use cases and solutions. - Ensure RuboCop checks are not disabled unless there's a valid reason to. diff --git a/doc/development/diffs.md b/doc/development/diffs.md index 3f855d90756..e065e0acc6f 100644 --- a/doc/development/diffs.md +++ b/doc/development/diffs.md @@ -8,12 +8,7 @@ Currently we rely on different sources to present diffs, these include: ## Deep Dive -In January 2019, Oswaldo Ferreira hosted a [Deep Dive] on GitLab's Diffs and Commenting on Diffs functionality to share his domain specific knowledge with anyone who may work in this part of the code base in the future. You can find the [recording on YouTube], and the slides on [Google Slides] and in [PDF]. Everything covered in this deep dive was accurate as of GitLab 11.7, and while specific details may have changed since then, it should still serve as a good introduction. - -[Deep Dive]: https://gitlab.com/gitlab-org/create-stage/issues/1 -[recording on YouTube]: https://www.youtube.com/watch?v=K6G3gMcFyek -[Google Slides]: https://docs.google.com/presentation/d/1bGutFH2AT3bxOPZuLMGl1ANWHqFnrxwQwjiwAZkF-TU/edit -[PDF]: https://gitlab.com/gitlab-org/create-stage/uploads/b5ad2f336e0afcfe0f99db0af0ccc71a/Create_Deep_Dive__Diffs_and_commenting_on_diffs.pdf +In January 2019, Oswaldo Ferreira hosted a Deep Dive (GitLab team members only: `https://gitlab.com/gitlab-org/create-stage/issues/1`) on GitLab's Diffs and Commenting on Diffs functionality to share his domain specific knowledge with anyone who may work in this part of the code base in the future. You can find the [recording on YouTube](https://www.youtube.com/watch?v=K6G3gMcFyek), and the slides on [Google Slides](https://docs.google.com/presentation/d/1bGutFH2AT3bxOPZuLMGl1ANWHqFnrxwQwjiwAZkF-TU/edit) and in [PDF](https://gitlab.com/gitlab-org/create-stage/uploads/b5ad2f336e0afcfe0f99db0af0ccc71a/). Everything covered in this deep dive was accurate as of GitLab 11.7, and while specific details may have changed since then, it should still serve as a good introduction. ## Architecture overview diff --git a/doc/development/documentation/feature_flags.md b/doc/development/documentation/feature_flags.md new file mode 100644 index 00000000000..373c5a4ee7d --- /dev/null +++ b/doc/development/documentation/feature_flags.md @@ -0,0 +1,188 @@ +--- +type: reference +description: "GitLab development - how to document features deployed behind feature flags" +--- + +# Document features deployed behind feature flags + +GitLab uses [Feature Flags](../feature_flags/index.md) to strategically roll +out the deployment of its own features. The way we document a feature behind a +feature flag depends on its state (enabled or disabled). When the state +changes, the developer who made the change **must update the documentation** +accordingly. + +## Criteria + +According to the process of [deploying GitLab features behind feature flags](../feature_flags/process.md): + +> - _By default, feature flags should be off._ +> - _Feature flags should remain in the codebase for a short period as possible to reduce the need for feature flag accounting._ +> - _In order to build a final release and present the feature for self-managed users, the feature flag should be at least defaulted to on._ + +See how to document them below, according to the state of the flag: + +- [Features disabled by default](#features-disabled-by-default). +- [Features that became enabled by default](#features-that-became-enabled-by-default). +- [Features directly enabled by default](#features-directly-enabled-by-default). +- [Features with the feature flag removed](#features-with-flag-removed). + +NOTE: **Note:** +The [`**(CORE ONLY)**`](styleguide.md#product-badges) badge or equivalent for +the feature's tier should be added to the line and heading that refers to +enabling/disabling feature flags as Admin access is required to do so, +therefore, it indicates that it cannot be done by regular users of GitLab.com. + +### Features disabled by default + +For features disabled by default, if they cannot be used yet due to lack of +completeness, or if they're still under internal evaluation (for example, for +performance implications) do **not document them**: add (or merge) the docs +only when the feature is safe and ready to use and test by end users. + +For feature flags disabled by default, if they can be used by end users: + +- Say that it's disabled by default. +- Say whether it's enabled on GitLab.com. +- Say whether it's recommended for production use. +- Document how to enable and disable it. + +For example, for a feature disabled by default, disabled on GitLab.com, and +not ready for production use: + +````markdown +# Feature Name + +> - [Introduced](link-to-issue) in GitLab 12.0. +> - It's deployed behind a feature flag, disabled by default. +> - It's disabled on GitLab.com. +> - It's not recommended for production use. +> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#anchor-to-section). **(CORE ONLY)** + +(...) + +### Enable or disable <Feature Name> **(CORE ONLY)** + +<Feature Name> is under development and not ready for production use. It is +deployed behind a feature flag that is **disabled by default**. +[GitLab administrators with access to the GitLab Rails console](../path/to/administration/feature_flags.md) +can enable it for your instance. + +To enable it: + +```ruby +Feature.enable(:<feature flag>) +``` + +To disable it: + +```ruby +Feature.disable(:<feature flag>) +``` +```` + +Adjust the blurb according to the state of the feature you're documenting. + +### Features that became enabled by default + +For features that became enabled by default: + +- Say that it became enabled by default. +- Say whether it's enabled on GitLab.com. +- Say whether it's recommended for production use. +- Document how to disable and enable it. + +For example, for a feature initially deployed disabled by default, that became enabled by default, that is enabled on GitLab.com, and ready for production use: + +````markdown +# Feature Name + +> - [Introduced](link-to-issue) in GitLab 12.0. +> - It was deployed behind a feature flag, disabled by default. +> - [Became enabled by default](link-to-issue) on GitLab 12.1. +> - It's enabled on GitLab.com. +> - It's recommended for production use. +> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#anchor-to-section). **(CORE ONLY)** + +(...) + +### Enable or disable <Feature Name> **(CORE ONLY)** + +<Feature Name> is under development but ready for production use. +It is deployed behind a feature flag that is **enabled by default**. +[GitLab administrators with access to the GitLab Rails console](..path/to/administration/feature_flags.md) +can opt to disable it for your instance. + +To disable it: + +```ruby +Feature.disable(:<feature flag>) +``` + +To enable it: + +```ruby +Feature.enable(:<feature flag>) +``` +```` + +Adjust the blurb according to the state of the feature you're documenting. + +### Features directly enabled by default + +For features enabled by default: + +- Say it's enabled by default. +- Say whether it's enabled on GitLab.com. +- Say whether it's recommended for production use. +- Document how to disable and enable it. + +For example, for a feature enabled by default, enabled on GitLab.com, and ready for production use: + +````markdown +# Feature Name + +> - [Introduced](link-to-issue) in GitLab 12.0. +> - It's deployed behind a feature flag, enabled by default. +> - It's enabled on GitLab.com. +> - It's recommended for production use. +> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#anchor-to-section). **(CORE ONLY)** + +(...) + +### Enable or disable <Feature Name> **(CORE ONLY)** + +<Feature Name> is under development but ready for production use. +It is deployed behind a feature flag that is **enabled by default**. +[GitLab administrators with access to the GitLab Rails console](..path/to/administration/feature_flags.md) +can opt to disable it for your instance. + +To disable it: + +```ruby +Feature.disable(:<feature flag>) +``` + +To enable it: + +```ruby +Feature.enable(:<feature flag>) +``` +```` + +Adjust the blurb according to the state of the feature you're documenting. + +### Features with flag removed + +Once the feature is ready and the flag has been removed, clean up the +documentation. Remove the feature flag mention keeping only a note that +mentions the flag in the version history notes: + +````markdown +# Feature Name + +> - [Introduced](link-to-issue) in GitLab 12.0. +> - [Feature flag removed](link-to-issue) in GitLab 12.2. + +(...) + +```` diff --git a/doc/development/documentation/index.md b/doc/development/documentation/index.md index 7a0e187b70a..256d3f5d86c 100644 --- a/doc/development/documentation/index.md +++ b/doc/development/documentation/index.md @@ -6,13 +6,14 @@ description: Learn how to contribute to GitLab Documentation. GitLab's documentation is [intended as the single source of truth (SSOT)](https://about.gitlab.com/handbook/documentation/) for information about how to configure, use, and troubleshoot GitLab. The documentation contains use cases and usage instructions for every GitLab feature, organized by product area and subject. This includes topics and workflows that span multiple GitLab features, and the use of GitLab with other applications. -In addition to this page, the following resources can help you craft and contribute documentation: +In addition to this page, the following resources can help you craft and contribute to documentation: - [Style Guide](styleguide.md) - What belongs in the docs, language guidelines, Markdown standards to follow, links, and more. - [Structure and template](structure.md) - Learn the typical parts of a doc page and how to write each one. - [Documentation process](workflow.md). - [Markdown Guide](../../user/markdown.md) - A reference for all Markdown syntax supported by GitLab. - [Site architecture](site_architecture/index.md) - How <https://docs.gitlab.com> is built. +- [Documentation for feature flags](feature_flags.md) - How to write and update documentation for GitLab features deployed behind feature flags. ## Source files and rendered web locations @@ -51,7 +52,7 @@ docs-only merge requests using the following guide: [Contributions to GitLab docs](workflow.md) are welcome from the entire GitLab community. -To ensure that GitLab docs are current, there are special processes and responsibilities for all [feature changes](feature-change-workflow.md)—i.e. development work that impacts the appearance, usage, or administration of a feature. +To ensure that GitLab docs are current, there are special processes and responsibilities for all [feature changes](feature-change-workflow.md), that is development work that impacts the appearance, usage, or administration of a feature. However, anyone can contribute [documentation improvements](improvement-workflow.md) that are not associated with a feature change. For example, adding a new doc on how to accomplish a use case that's already possible with GitLab or with third-party tools and GitLab. @@ -76,13 +77,24 @@ whether the move is necessary), and ensure that a technical writer reviews this change prior to merging. If you indeed need to change a document's location, do not remove the old -document, but instead replace all of its content with a new line: +document, but instead replace all of its content with the following: -```md -This document was moved to [another location](path/to/new_doc.md). +```markdown +--- +redirect_to: '../path/to/file/index.md' +--- + +This document was moved to [another location](../path/to/file/index.md). ``` -where `path/to/new_doc.md` is the relative path to the root directory `doc/`. +Where `../path/to/file/index.md` is usually the relative path to the old document. + +The `redirect_to` variable supports both full and relative URLs, for example +`https://docs.gitlab.com/ee/path/to/file.html`, `../path/to/file.html`, `path/to/file.md`. +It ensures that the redirect will work for <https://docs.gitlab.com> and any `*.md` paths +will be compiled to `*.html`. +The new line underneath the front matter informs the user that the document +changed location and is useful for someone that browses that file from the repository. For example, if you move `doc/workflow/lfs/index.md` to `doc/administration/lfs.md`, then the steps would be: @@ -90,13 +102,17 @@ For example, if you move `doc/workflow/lfs/index.md` to 1. Copy `doc/workflow/lfs/index.md` to `doc/administration/lfs.md` 1. Replace the contents of `doc/workflow/lfs/index.md` with: - ```md + ```markdown + --- + redirect_to: '../../administration/lfs.md' + --- + This document was moved to [another location](../../administration/lfs.md). ``` 1. Find and replace any occurrences of the old location with the new one. - A quick way to find them is to use `git grep`. First go to the root directory - where you cloned the `gitlab` repository and then do: + A quick way to find them is to use `git grep` on the repository you changed + the file from: ```shell git grep -n "workflow/lfs/lfs_administration" @@ -123,24 +139,6 @@ Things to note: built-in help page, that's why we omit it in `git grep`. - Use the checklist on the "Change documentation location" MR description template. -### Alternative redirection method - -You can also replace the content -of the old file with a frontmatter containing a redirect link: - -```yaml ---- -redirect_to: '../path/to/file/README.md' ---- -``` - -It supports both full and relative URLs, e.g. `https://docs.gitlab.com/ee/path/to/file.html`, `../path/to/file.html`, `path/to/file.md`. Note that any `*.md` paths will be compiled to `*.html`. - -NOTE: **Note:** -This redirection method will not provide a redirect fallback on GitLab `/help`. When using -it, make sure to add a link to the new page on the doc, otherwise it's a dead end for users that -land on the doc via `/help`. - ### Redirections for pages with Disqus comments If the documentation page being relocated already has Disqus comments, @@ -150,12 +148,12 @@ Disqus uses an identifier per page, and for <https://docs.gitlab.com>, the page is configured to be the page URL. Therefore, when we change the document location, we need to preserve the old URL as the same Disqus identifier. -To do that, add to the frontmatter the variable `disqus_identifier`, -using the old URL as value. For example, let's say I moved the document +To do that, add to the front matter the variable `disqus_identifier`, +using the old URL as value. For example, let's say we moved the document available under `https://docs.gitlab.com/my-old-location/README.html` to a new location, `https://docs.gitlab.com/my-new-location/index.html`. -Into the **new document** frontmatter add the following: +Into the **new document** front matter, we add the following: ```yaml --- @@ -173,8 +171,8 @@ Before getting started, make sure you read the introductory section [documentation workflow](workflow.md). - Use the current [merge request description template](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/merge_request_templates/Documentation.md) -- Label the MR `Documentation` -- Assign the correct milestone (see note below) +- Label the MR `Documentation` (can only be done by people with `developer` access, for example, GitLab team members) +- Assign the correct milestone per note below (can only be done by people with `developer` access, for example, GitLab team members) Documentation will be merged if it is an improvement on existing content, represents a good-faith effort to follow the template and style standards, @@ -185,7 +183,7 @@ in a follow-up MR or issue. NOTE: **Note:** If the release version you want to add the documentation to has already been -frozen or released, use the label `Pick into X.Y` to get it merged into +frozen or released, use the label `~"Pick into X.Y"` to get it merged into the correct release. Avoid picking into a past release as much as you can, as it increases the work of the release managers. @@ -205,7 +203,7 @@ to the MR. For example, let's say your merge request has a milestone set to 11.3, which will be released on 2018-09-22. If it gets merged on 2018-09-15, it will be available online on 2018-09-15, but, as the feature freeze date has passed, if -the MR does not have a "pick into 11.3" label, the milestone has to be changed +the MR does not have a `~"Pick into 11.3"` label, the milestone has to be changed to 11.4 and it will be shipped with all GitLab packages only on 2018-10-22, with GitLab 11.4. Meaning, it will only be available under `/help` from GitLab 11.4 onward, but available on <https://docs.gitlab.com/> on the same day it was merged. @@ -277,7 +275,7 @@ You can combine one or more of the following: ### GitLab `/help` tests -Several [rspec tests](https://gitlab.com/gitlab-org/gitlab/blob/master/spec/features/help_pages_spec.rb) +Several [RSpec tests](https://gitlab.com/gitlab-org/gitlab/blob/master/spec/features/help_pages_spec.rb) are run to ensure GitLab documentation renders and works correctly. In particular, that [main docs landing page](../../README.md) will work correctly from `/help`. For example, [GitLab.com's `/help`](https://gitlab.com/help). @@ -300,8 +298,8 @@ To preview your changes to documentation locally, follow this The live preview is currently enabled for the following projects: -- <https://gitlab.com/gitlab-org/gitlab> -- <https://gitlab.com/gitlab-org/gitlab-runner> +- [`gitlab`](https://gitlab.com/gitlab-org/gitlab) +- [`gitlab-runner`](https://gitlab.com/gitlab-org/gitlab-runner) If your merge request has docs changes, you can use the manual `review-docs-deploy` job to deploy the docs review app for your merge request. @@ -407,8 +405,6 @@ merge request with new or changed docs is submitted, are: - [`internal_anchors`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/docs.gitlab-ci.yml#L69) checks that all internal anchors (ex: `[link](../index.md#internal_anchor)`) are valid. -- If any code or the `doc/README.md` file is changed, a full pipeline will run, which - runs tests for [`/help`](#gitlab-help-tests). ### Running tests @@ -489,14 +485,14 @@ markdownlint can be used [on the command line](https://github.com/igorshubovych/ either on a single Markdown file or on all Markdown files in a project. For example, to run markdownlint on all documentation in the [`gitlab` project](https://gitlab.com/gitlab-org/gitlab), run the following commands from within your `gitlab` project root directory, which will -automatically detect the [`.markdownlint.json`](#markdownlint-configuration) config +automatically detect the [`.markdownlint.json`](#markdownlint-configuration) configuration file in the root of the project, and test all files in `/doc` and its subdirectories: ```shell markdownlint 'doc/**/*.md' ``` -If you wish to use a different config file, use the `-c` flag: +If you wish to use a different configuration file, use the `-c` flag: ```shell markdownlint -c <config-file-name> 'doc/**/*.md' @@ -510,7 +506,7 @@ such as: - [Atom](https://atom.io/packages/linter-node-markdownlint) It is best to use the [same configuration file](#markdownlint-configuration) as what -is in use in the four repos that are the sources for <https://docs.gitlab.com>. Each +is in use in the four repositories that are the sources for <https://docs.gitlab.com>. Each plugin/extension has different requirements regarding the configuration file, which is explained in each editor's docs. @@ -519,12 +515,12 @@ is explained in each editor's docs. Each formatting issue that markdownlint checks has an associated [rule](https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md#rules). These rules are configured in the `.markdownlint.json` files located in the root of -four repos that are the sources for <https://docs.gitlab.com>: +four repositories that are the sources for <https://docs.gitlab.com>: -- <https://gitlab.com/gitlab-org/gitlab/blob/master/.markdownlint.json> -- <https://gitlab.com/gitlab-org/gitlab-runner/blob/master/.markdownlint.json> -- <https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/.markdownlint.json> -- <https://gitlab.com/charts/gitlab/blob/master/.markdownlint.json> +- [`gitlab`](https://gitlab.com/gitlab-org/gitlab/blob/master/.markdownlint.json) +- [`gitlab-runner`](https://gitlab.com/gitlab-org/gitlab-runner/blob/master/.markdownlint.json) +- [`omnibus-gitlab`](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/.markdownlint.json) +- [`charts`](https://gitlab.com/charts/gitlab/blob/master/.markdownlint.json) By default all rules are enabled, so the configuration file is used to disable unwanted rules, and also to configure optional parameters for enabled rules as needed. You can @@ -534,7 +530,7 @@ on or off when markdownlint was enabled on the docs. #### Vale -[Vale](https://errata-ai.github.io/vale/) is a grammar, style, and word usage linter +[Vale](https://errata-ai.gitbook.io/vale/) is a grammar, style, and word usage linter for the English language. Vale's configuration is stored in the [`.vale.ini`](https://gitlab.com/gitlab-org/gitlab/blob/master/.vale.ini) file located in the root directory of the [GitLab repository](https://gitlab.com/gitlab-org/gitlab). @@ -554,10 +550,28 @@ You can also [configure the text editor of your choice](https://errata-ai.github.io/vale/#local-use-by-a-single-writer) to display the results. +Vale's test results [are displayed](#testing) in CI pipelines. + +##### Disable a Vale test + +You can disable a specific Vale linting rule or all Vale linting rules for any portion of a document: + +- To disable a specific rule, add a `<!-- vale gitlab.rulename = NO -->` tag + before the text, and a `<!-- vale gitlab.rulename = YES -->` tag after the text, + replacing `rulename` with the filename of a test in the [GitLab styles](https://gitlab.com/gitlab-org/gitlab/-/tree/master/doc/.linting/vale/styles/gitlab) directory. +- To disable all Vale linting rules, add a `<!-- vale off -->` tag before the text, + and a `<!-- vale on -->` tag after the text. + +Whenever possible, exclude only the problematic rule and line(s). +In some cases, such as list items, you may need to disable linting for the entire +list until ["Ignore comments are not honored in a Markdown file"](https://github.com/errata-ai/vale/issues/175) is resolved. + +For more information, see [Vale's documentation](https://errata-ai.gitbook.io/vale/getting-started/markup#markup-based-configuration). + ## Danger Bot GitLab uses [Danger](https://github.com/danger/danger) for some elements in code review. For docs changes in merge requests, whenever a change to files under `/doc` is made, Danger Bot leaves a comment with further instructions about the documentation -process. This is configured in the Dangerfile in the GitLab repo under +process. This is configured in the `Dangerfile` in the GitLab repository under [/danger/documentation/](https://gitlab.com/gitlab-org/gitlab/tree/master/danger/documentation). diff --git a/doc/development/documentation/site_architecture/release_process.md b/doc/development/documentation/site_architecture/release_process.md index a8da565124b..13d6540fa35 100644 --- a/doc/development/documentation/site_architecture/release_process.md +++ b/doc/development/documentation/site_architecture/release_process.md @@ -20,7 +20,7 @@ Although build images are built automatically via GitLab CI/CD, you can build and tag all tooling images locally: 1. Make sure you have [Docker installed](https://docs.docker.com/install/). -1. Make sure you're on the `dockerfiles/` directory of the `gitlab-docs` repo. +1. Make sure you're in the `dockerfiles/` directory of the `gitlab-docs` repository. 1. Build the images: ```shell @@ -46,7 +46,7 @@ products, we need to add a 1. Check that there is a [stable branch created](https://gitlab.com/gitlab-org/charts/gitlab/-/branches) for the new chart version. If you're unsure or can't find it, drop a line in the `#g_delivery` channel. -1. Make sure you're on the root path of the `gitlab-docs` repo. +1. Make sure you're in the root path of the `gitlab-docs` repository. 1. Open `content/_data/chart_versions.yaml` and add the new stable branch version using the version mapping. Note that only the `major.minor` version is needed. 1. Create a new merge request and merge it. @@ -61,14 +61,14 @@ this first step. The single docs version must be created before the release merge request, but this needs to happen when the stable branches for all products have been created. -1. Make sure you're on the root path of the `gitlab-docs` repo. +1. Make sure you're in the root path of the `gitlab-docs` repository. 1. Make sure your `master` is updated: ```shell git pull origin master ``` -1. Run the raketask to create the single version: +1. Run the Rake task to create the single version: ```shell ./bin/rake "release:single[12.0]" @@ -109,7 +109,7 @@ Visit `http://localhost:4000/12.0/` to see if everything works correctly. Now it's time to create the monthly release merge request that adds the new version and rotates the old one: -1. Make sure you're on the root path of the `gitlab-docs` repo. +1. Make sure you're in the root path of the `gitlab-docs` repository. 1. Create a branch `release-X-Y`: ```shell @@ -158,10 +158,10 @@ the dropdown are included in the unmerged `release-X-Y` branch. The content of `content/_data/versions.yaml` needs to change for all online versions: -1. Run the raketask that will create all the respective merge requests needed to +1. Run the Rake task that will create all the respective merge requests needed to update the dropdowns and will be set to automatically be merged when their pipelines succeed. The `release-X-Y` branch needs to be present locally, - otherwise the raketask will fail: + otherwise the Rake task will fail: ```shell ./bin/rake release:dropdowns diff --git a/doc/development/documentation/structure.md b/doc/development/documentation/structure.md index 84ba47eba78..d19383bee27 100644 --- a/doc/development/documentation/structure.md +++ b/doc/development/documentation/structure.md @@ -20,7 +20,7 @@ Every feature or use case document should include the following content in the f with exceptions and details noted below and in the template included on this page. - **Title**: Top-level heading with the feature name, or a use case name, which would start with - a verb, like Configuring, Enabling, etc. + a verb, like Configuring, Enabling, and so on. - **Introduction**: A couple sentences about the subject matter and what's to be found on this page. Describe what the feature or topic is, what it does, and in what context it should be used. There is no need to add a title called "Introduction" or "Overview," because people rarely @@ -41,7 +41,7 @@ and other logical divisions such as pre- and post-deployment steps. To start a new document, respect the file tree and file name guidelines, as well as the style guidelines. Use the following template: -```md +```markdown <!--Follow the Style Guide when working on this document. https://docs.gitlab.com/ee/development/documentation/styleguide.html When done, remove all of this commented-out text, except a commented-out Troubleshooting section, which, if empty, can be left in place to encourage future use.--> @@ -130,7 +130,7 @@ Notes: ## Help and feedback section The "help and feedback" section (introduced by [!319](https://gitlab.com/gitlab-org/gitlab-docs/-/merge_requests/319)) displayed at the end of each document -can be omitted from the doc by adding a key into the its frontmatter: +can be omitted from the doc by adding a key into the its front matter: ```yaml --- @@ -148,7 +148,7 @@ We also have integrated the docs site with Disqus (introduced by allowing our users to post comments. To omit only the comments from the feedback section, use the following -key on the frontmatter: +key on the front matter: ```yaml --- @@ -159,7 +159,7 @@ comments: false We are only hiding comments in main index pages, such as [the main documentation index](../../README.md), since its content is too broad to comment on. Before omitting Disqus, you must check with a technical writer. -Note that once `feedback: false` is added to the frontmatter, it will automatically omit +Note that once `feedback: false` is added to the front matter, it will automatically omit Disqus, therefore, don't add both keys to the same document. The click events in the feedback section are tracked with Google Tag Manager. The diff --git a/doc/development/documentation/styleguide.md b/doc/development/documentation/styleguide.md index 0007f6d6e2f..6d146051e13 100644 --- a/doc/development/documentation/styleguide.md +++ b/doc/development/documentation/styleguide.md @@ -17,9 +17,9 @@ that apply to all GitLab content, not just documentation. ### Why a single source of truth -The documentation of GitLab products and features is the SSOT for all information related to implementation, usage, and troubleshooting. It evolves continually, in keeping with new products and features, and with improvements for clarity, accuracy, and completeness. +The documentation of GitLab products and features is the SSOT for all information related to implementation, usage, and troubleshooting. It evolves continuously, in keeping with new products and features, and with improvements for clarity, accuracy, and completeness. -This policy prevents information silos, ensuring that it remains easy to find information about GitLab products. +This policy prevents information silos, making it easier to find information about GitLab products. It also informs decisions about the kinds of content we include in our documentation. @@ -46,12 +46,12 @@ In the software industry, it is a best practice to organize documentation in dif 1. Explanation 1. Reference (for example, a glossary) -At GitLab, we have so many product changes in our monthly releases that we can't afford to continually update multiple types of information. +At GitLab, we have so many product changes in our monthly releases that we can't afford to continuously update multiple types of information. If we have multiple types, the information will become outdated. Therefore, we have a [single template](structure.md) for documentation. We currently do not distinguish specific document types, although we are open to reconsidering this policy once the documentation has reached a future stage of maturity and quality. If you are reading this, then despite our -continual improvement efforts, that point hasn't been reached. +continuous improvement efforts, that point hasn't been reached. ### Link instead of summarize @@ -61,7 +61,7 @@ Instead, link to the SSOT and explain why it is important to consume the informa ### Organize by topic, not by type -Beyond top-level audience-type folders (for example, `administration`), we organize content by topic, not by type, so that it can be located as easily as possible within the single-source-of-truth (SSOT) section for the subject matter. +Beyond top-level audience-type folders (for example, `administration`), we organize content by topic, not by type, so it can be located as easily as possible within the single-source-of-truth (SSOT) section for the subject matter. For example, do not create groupings of similar media types. For example: @@ -76,7 +76,7 @@ and cross-link between any related content. ### Docs-first methodology -We employ a **docs-first methodology** to help ensure that the docs remain a complete and trusted resource, and to make communicating about the use of GitLab more efficient. +We employ a **docs-first methodology** to help ensure the docs remain a complete and trusted resource, and to make communicating about the use of GitLab more efficient. - If the answer to a question exists in documentation, share the link to the docs instead of rephrasing the information. - When you encounter new information not available in GitLab’s documentation (for example, when working on a support case or testing a feature), your first step should be to create a merge request (MR) to add this information to the docs. You can then share the MR in order to communicate this information. @@ -129,13 +129,13 @@ correctly, but is not the current standard for GitLab documentation). A rule that could cause confusion is `MD044/proper-names`, as it might not be immediately clear what caused markdownlint to fail, or how to correct the failure. This rule checks a list of known words, listed in the `.markdownlint.json` file in each project, -to verify that proper capitalization and backticks are used. Words in backticks will +to verify proper use of capitalization and backticks. Words in backticks will be ignored by markdownlint. In general, product names should follow the exact capitalization of the official names of the products, protocols, and so on. -Some examples that will fail if incorrect capitalization is used: +Some examples fail if incorrect capitalization is used: - MinIO (needs capital `IO`) - NGINX (needs all capitals) @@ -248,10 +248,6 @@ GitLab documentation should be clear and easy to understand. - Be clear, concise, and stick to the goal of the documentation. - Write in US English with US grammar. - Use inclusive language. -- Avoid jargon. -- Avoid uncommon words. -- Don't write in the first person singular. - - Instead of "I" or "me," use "we," "you," "us," or "one." ### Point of view @@ -285,21 +281,53 @@ because it’s friendly and easy to understand. Some features are also objects. For example, "GitLab's Merge Requests support X" and "Create a new merge request for Z." +### Language to avoid + +When creating documentation, limit or avoid the use of the following verb +tenses, words, and phrases: + +- Avoid jargon. +- Avoid uncommon words. +- Don't write in the first person singular. + - Instead of "I" or "me," use "we," "you," "us," or "one." + - When possible, stay user focused by writing in the second person ("you" or + the imperative). +- Don't overuse "that". In many cases, you can remove "that" from a sentence + and improve readability. - Avoid use of the future tense: - - Instead of "after you execute this command, GitLab will display the result", use "after you execute this command, GitLab displays the result". - - Only use the future tense to convey when the action or result will actually occur at a future time. -- Do not use slashes to clump different words together or as a replacement for the word "or": - - Instead of "and/or," consider using "or," or use another sensible construction. - - Other examples include "clone/fetch," author/assignee," and "namespace/repository name." Break apart any such instances in an appropriate way. - - Exceptions to this rule include commonly accepted technical terms such as CI/CD, TCP/IP, and so on. -- Do not use "may" and "might" interchangeably: - - Use "might" to indicate the probability of something occurring. "If you skip this step, the import process might fail." - - Use "may" to indicate giving permission for someone to do something, or consider using "can" instead. "You may select either option on this screen." Or, "you can select either option on this screen." -- We discourage use of Latin abbreviations, such as "e.g.," "i.e.," or "etc.," -as even native users of English might misunderstand them. + - Instead of "after you execute this command, GitLab will display the + result", use "after you execute this command, GitLab displays the result". + - Only use the future tense to convey when the action or result will actually + occur at a future time. +- Don't use slashes to clump different words together or as a replacement for + the word "or": + - Instead of "and/or," consider using "or," or use another sensible + construction. + - Other examples include "clone/fetch," author/assignee," and + "namespace/repository name." Break apart any such instances in an + appropriate way. + - Exceptions to this rule include commonly accepted technical terms, such as + CI/CD and TCP/IP. +- <!-- vale gitlab.LatinTerms = NO --> + We discourage use of Latin abbreviations, such as "e.g.," "i.e.," or "etc.," + as even native users of English might misunderstand them. - Instead of "i.e.," use "that is." - Instead of "e.g.," use "for example," "such as," "for instance," or "like." - - Instead of "etc.," either use "and so on" or consider editing it out, since it can be vague. + - Instead of "etc.," either use "and so on" or consider editing it out, since + it can be vague. + <!-- vale gitlab.rulename = NO --> +- Avoid using the word *currently* when talking about the product or its + features. The documentation describes the product as it is, and not as it + will be at some indeterminate point in the future. + +### Word usage clarifications + +- Don't use "may" and "might" interchangeably: + - Use "might" to indicate the probability of something occurring. "If you + skip this step, the import process might fail." + - Use "may" to indicate giving permission for someone to do something, or + consider using "can" instead. "You may select either option on this + screen." Or, "You can select either option on this screen." ### Contractions @@ -353,6 +381,8 @@ as even native users of English might misunderstand them. | Requests to localhost are not allowed | Requests to localhost aren't allowed | | Specified URL cannot be used | Specified URL can't be used | +<!-- vale on --> + ## Text - [Write in Markdown](#markdown). @@ -360,7 +390,7 @@ as even native users of English might misunderstand them. - Insert an empty line for new paragraphs. - Insert an empty line between different markups (for example, after every paragraph, header, list, and so on). Example: - ```md + ```markdown ## Header Paragraph. @@ -417,7 +447,7 @@ Only use ordered lists when their items describe a sequence of steps to follow. Do: -```md +```markdown These are the steps to do something: 1. First, do the first step. @@ -427,7 +457,7 @@ These are the steps to do something: Don't: -```md +```markdown This is a list of available features: 1. Feature 1 @@ -453,7 +483,7 @@ This is a list of available features: all with a period. - Separate list items from explanatory text with a colon (`:`). For example: - ```md + ```markdown The list is as follows: - First item: this explains the first item. @@ -511,7 +541,7 @@ In unordered lists (using `-`), this means two spaces for each level of indentat - Unordered list item 3 - ```text + ```plaintext a codeblock that will next inside list item 3 ``` @@ -534,7 +564,7 @@ For ordered lists, use three spaces for each level of indentation: 1. Ordered list item 3 - ```text + ```plaintext a codeblock that will next inside list item 3 ``` @@ -560,9 +590,47 @@ to mix types, that is also possible, as long as you don't mix items at the same - Unordered list item three. ``` +## Tables + +Tables should be used to describe complex information in a straightforward +manner. Note that in many cases, an unordered list is sufficient to describe a +list of items with a single, simple description per item. But, if you have data +that is best described by a matrix, tables are the best choice for use. + +### Creation guidelines + +Due to accessibility and scanability requirements, tables should not have any +empty cells. If there is no otherwise meaningful value for a cell, consider entering +*N/A* (for 'not applicable') or *none*. + +To help tables be easier to maintain, consider adding additional spaces to the +column widths to make them consistent. For example: + +```markdown +| App name | Description | Requirements | +|:---------|:---------------------|:---------------| +| App 1 | Description text 1. | Requirements 1 | +| App 2 | Description text 2. | None | +``` + +Consider installing a plugin or extension in your editor for formatting tables: + +- [Markdown Table Prettifier](https://marketplace.visualstudio.com/items?itemName=darkriszty.markdown-table-prettify) for Visual Studio Code +- [Markdown Table Formatter](https://packagecontrol.io/packages/Markdown%20Table%20Formatter) for Sublime Text +- [Markdown Table Formatter](https://atom.io/packages/markdown-table-formatter) for Atom + +### Feature tables + +When creating tables of lists of features (such as whether or not features are +available to certain roles on the [Permissions](../../user/permissions.md#project-members-permissions) +page), use the following phrases (based on the SVG icons): + +- *No*: **{dotted-circle}** No +- *Yes*: **{check-circle}** Yes + ## Quotes -Valid for Markdown content only, not for frontmatter entries: +Valid for Markdown content only, not for front matter entries: - Standard quotes: double quotes (`"`). Example: "This is wrapped in double quotes". - Quote within a quote: double quotes (`"`) wrap single quotes (`'`). Example: "I am 'quoting' something within a quote". @@ -724,15 +792,17 @@ Instead: Example: -```md +```markdown For more information, see the [confidential issue](../../user/project/issues/confidential_issues.md) `https://gitlab.com/gitlab-org/gitlab-foss/issues/<issue_number>`. ``` ### Link to specific lines of code -When linking to specifics lines within a file, link to a commit instead of to the branch. +When linking to specific lines within a file, link to a commit instead of to the branch. Lines of code change through time, therefore, linking to a line by using the commit link -ensures the user lands on the line you're referring to. +ensures the user lands on the line you're referring to. The **Permalink** button, which is +available when viewing a file within a project, makes it easy to generate a link to the +most recent commit of the given file. - **Do:** `[link to line 3](https://gitlab.com/gitlab-org/gitlab/-/blob/11f17c56d8b7f0b752562d78a4298a3a95b5ce66/.gitlab/issue_templates/Feature%20proposal.md#L3)` - **Don't:** `[link to line 3](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/Feature%20proposal.md#L3).` @@ -801,7 +871,7 @@ known tool is [`pngquant`](https://pngquant.org/), which is cross-platform and open source. Install it by visiting the official website and following the instructions for your OS. -GitLab has a [raketask](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/tasks/pngquant.rake) +GitLab has a [Rake task](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/tasks/pngquant.rake) that you can use to automate the process. In the root directory of your local copy of `https://gitlab.com/gitlab-org/gitlab`, run in a terminal: @@ -838,7 +908,7 @@ Do not upload videos to the product repositories. [Link](#link-to-video) or [emb To link out to a video, include a YouTube icon so that readers can quickly and easily scan the page for videos before reading: -```md +```markdown <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> For an overview, see [Video Title](link-to-video). ``` @@ -965,7 +1035,7 @@ of language classes available. | `ruby` | Alias: `rb`. | | `shell` | Aliases: `bash` or `sh`. | | `sql` | | -| `toml` | Runner configuration examples, and other toml formatted configuration files. | +| `toml` | Runner configuration examples, and other TOML-formatted configuration files. | | `typescript` | Alias: `ts`. | | `xml` | | | `yaml` | Alias: `yml`. | @@ -1012,7 +1082,7 @@ This will ensure that the source Markdown remains readable and should help with The following are examples of source Markdown for menu items with their published output: -```md +```markdown 1. Go to **{home}** **Project overview > Details** 1. Go to **{doc-text}** **Repository > Branches** 1. Go to **{issues}** **Issues > List** @@ -1073,7 +1143,7 @@ of users. Weigh the costs of distracting users to whom the content is not relevant against the cost of users missing the content if it were not expressed as a note. -```md +```markdown NOTE: **Note:** This is something to note. ``` @@ -1085,7 +1155,7 @@ This is something to note. ### Tip -```md +```markdown TIP: **Tip:** This is a tip. ``` @@ -1097,7 +1167,7 @@ This is a tip. ### Caution -```md +```markdown CAUTION: **Caution:** This is something to be cautious about. ``` @@ -1109,7 +1179,7 @@ This is something to be cautious about. ### Danger -```md +```markdown DANGER: **Danger:** This is a breaking change, a bug, or something very important to note. ``` @@ -1123,7 +1193,7 @@ This is a breaking change, a bug, or something very important to note. For highlighting a text within a blue blockquote, use this format: -```md +```markdown > This is a blockquote. ``` @@ -1135,7 +1205,7 @@ If the text spans across multiple lines it's OK to split the line. For multiple paragraphs, use the symbol `>` before every line: -```md +```markdown > This is the first paragraph. > > This is the second paragraph. @@ -1216,7 +1286,7 @@ a helpful link back to how the feature was developed. to the following should be added immediately below the heading as a blockquote: - `> Introduced in GitLab 11.3.`. -- Whenever possible, version text should have a link to the issue, merge request, or epic that introduced the feature. +- Whenever possible, version text should have a link to the _completed_ issue, merge request, or epic that introduced the feature. An issue is preferred over a merge request, and a merge request is preferred over an epic. For example: - `> [Introduced](<link-to-issue>) in GitLab 11.3.`. @@ -1228,21 +1298,35 @@ a helpful link back to how the feature was developed. - If listing information for multiple version as a feature evolves, add the information to a block-quoted bullet list. For example: - ```md + ```markdown > - [Introduced](<link-to-issue>) in GitLab 11.3. > - Enabled by default in GitLab 11.4. ``` - If a feature is moved to another tier: - ```md + ```markdown > - [Introduced](<link-to-issue>) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.5. > - [Moved](<link-to-issue>) to [GitLab Starter](https://about.gitlab.com/pricing/) in 11.8. > - [Moved](<link-to-issue>) to GitLab Core in 12.0. ``` NOTE: **Note:** -Version text must be on its own line and surounded by blank lines to render correctly. +Version text must be on its own line and surrounded by blank lines to render correctly. + +### Versions in the past or future + +When describing functionality available in past or future versions, use: + +- **Earlier**, and not **older** or **before**. +- **Later**, and not **newer** or **after**. + +For example: + +- Available in GitLab 12.3 and earlier. +- Available in GitLab 12.4 and later. +- If using GitLab 11.4 or earlier, ... +- If using GitLab 10.6 or later, ... ### Importance of referencing GitLab versions and tiers @@ -1333,7 +1417,7 @@ avoid duplication, link to the special document that can be found in [`doc/administration/restart_gitlab.md`](../../administration/restart_gitlab.md). Usually the text will read like: -```md +```markdown Save the file and [reconfigure GitLab](../../administration/restart_gitlab.md) for the changes to take effect. ``` @@ -1369,7 +1453,7 @@ When there is a list of steps to perform, usually that entails editing the configuration file and reconfiguring/restarting GitLab. In such case, follow the style below as a guide: -````md +````markdown **For Omnibus installations** 1. Edit `/etc/gitlab/gitlab.rb`: @@ -1416,38 +1500,8 @@ can facilitate this by making sure the troubleshooting content addresses: ## Feature flags -Sometimes features are shipped with feature flags, either: - -- On by default, but providing the option to turn the feature off. -- Off by default, but providing the option to turn the feature on. - -When documenting feature flags for a feature, include: - -- Why a feature flag is necessary. Some of the reasons are - [outlined in the handbook](https://about.gitlab.com/handbook/product/#alpha-beta-ga). -- That administrative access is required to make a feature flag change. -- What to ask for when requesting a change to a feature flag's state. - -NOTE: **Note:** -The [Product Manager for the relevant group](https://about.gitlab.com/handbook/product/categories/#devops-stages) -must review and approve the addition or removal of any mentions of using feature flags before the doc change is merged. - -The following is sample text for adding feature flag documentation for a feature that is -off by default: - -````md -### Enabling the feature - -This feature comes with the `:feature_flag` feature flag disabled by default. In some cases, -this feature is incompatible with an old configuration. To turn on the feature, -ask a GitLab administrator with Rails console access to run the following command: - -```ruby -Feature.enable(:feature_flag) -``` -```` - -For guidance on developing with feature flags, see +Learn how to [document features deployed behind flags](feature_flags.md). +For guidance on developing GitLab with feature flags, see [Feature flags in development of GitLab](../feature_flags/index.md). ## API @@ -1475,7 +1529,7 @@ The following can be used as a template to get started: One or two sentence description of what endpoint does. -```text +```plaintext METHOD /endpoint ``` @@ -1537,7 +1591,7 @@ You can use the following fake tokens as examples. Use the following table headers to describe the methods. Attributes should always be in code blocks using backticks (`` ` ``). -```md +```markdown | Attribute | Type | Required | Description | |:----------|:-----|:---------|:------------| ``` @@ -1614,7 +1668,7 @@ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --form "title= ``` The above example is run by and administrator and will add an SSH public key -titled `ssh-key` to user's account which has an id of 25. +titled `ssh-key` to user's account which has an ID of 25. #### Escape special characters diff --git a/doc/development/documentation/workflow.md b/doc/development/documentation/workflow.md index 01f528dcfa4..ab6200155bf 100644 --- a/doc/development/documentation/workflow.md +++ b/doc/development/documentation/workflow.md @@ -1,19 +1,20 @@ # Documentation process -The process for creating and maintaining GitLab product documentation depends on whether the -documentation is associated with: +The process for creating and maintaining GitLab product documentation allows +anyone to contribute a merge request or create an issue for GitLab's +documentation. -- [A new feature or feature enhancement](#for-a-product-change). - - Delivered for a specific milestone and associated with specific code changes. This documentation - has the highest priority. +NOTE: **Note:** +Documentation updates relating to new features or feature enhancements must +use the [feature workflow process](https://about.gitlab.com/handbook/engineering/ux/technical-writing/workflow/#for-a-product-change) described in the GitLab Handbook. -- [Changes outside a specific milestone](#for-all-other-documentation). +## Who updates the docs? - It is usually not associated with a specific code change and has a lower priority. +*Anyone* can contribute! You can create a merge request for documentation when: -Documentation is not usually required when a "backstage feature" is added or changed, and does not -directly affect the way that any user or administrator interacts with GitLab. +- You find errors or other room for improvement in existing documentation. +- You have an idea for all-new documentation that would help a GitLab user or administrator to + accomplish their work with GitLab. ## Documentation labels @@ -32,372 +33,17 @@ The following are also added by members of the Technical Writing team: `docs::` prefix. For example, `~docs::improvement`. - The `~Technical Writing` [team label](../contributing/issue_workflow.md#team-labels). -## For a product change - -This documentation is required for any new or changed feature and is: - -- Created or updated as part of feature development, almost always in the same merge request as the - feature code. Including documentation in the same merge request as the code eliminates the - possibility that code and documentation get out of sync. -- Required with the delivery of a feature for a specific milestone as part of GitLab's - [definition of done](../contributing/merge_request_workflow.md#definition-of-done). -- Often linked from the release post. - -### Roles and responsibilities - -Documentation for specific milestones involves the: - -- Developer of a feature or enhancement. -- Product Manager for the group delivering the new feature or feature enhancement. -- Technical Writer assigned to the group. - -Each role is described below. - -#### Developers - -Developers are the primary author of documentation for a feature or feature enhancement. They are -responsible for: - -- Developing initial content required for a feature. -- Liaising with their Product Manager to understand what documentation must be delivered, and when. -- Requesting technical reviews from other developers within their group. -- Requesting documentation reviews from the Technical Writer - [assigned to the DevOps stage group](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments) - that is delivering the new feature or feature enhancements. - -TIP: **Tip:** -Community Contributors can ask for additional help from GitLab team members. - -##### Authoring - -Because the documentation is an essential part of the product, if a ~feature issue also contains the -~documentation label, you must ship the new or updated documentation with the code of the feature. - -Technical Writers are happy to help, as requested and planned on an issue-by-issue basis. - -For feature issues requiring documentation, follow the process below unless otherwise agreed with -the Product Manager and Technical Writer for a given issue: - -- Include any new and edited documentation, either in: - - The merge request introducing the code. - - A separate merge request raised around the same time. -- Use the [documentation requirements](#documentation-requirements) developed by the Product Manager - in the issue and discuss any further documentation plans or ideas as needed. - - If the new or changed documentation requires extensive collaboration or conversation, a - separate, linked issue can be used for the planning process. - -- Use the [Documentation guidelines](index.md), as well as other resources linked from there, - including: - - Documentation [Structure and template](structure.md) page. - - [Style Guide](styleguide.md). - - [Markdown Guide](https://about.gitlab.com/handbook/engineering/ux/technical-writing/markdown-guide/). -- Contact the Technical Writer for the relevant [DevOps stage](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments) - in your issue or merge request, or within `#docs` on GitLab Slack, if you: - - Need any help to choose the correct place for documentation. - - Want to discuss a documentation idea or outline. - - Want to request any other help. -- If you are working on documentation in a separate merge request, ensure the documentation is - merged as close as possible to the code merge. -- A policy for documenting [feature-flagged](../feature_flags/index.md) issues is forthcoming and you - are welcome to join the [discussion](https://gitlab.com/gitlab-org/gitlab/issues/26347). - -##### Reviews and merging - -Reviewers help ensure: - -- Accuracy. -- Clarity. -- Completeness. -- Adherence to: - - [Documentation requirements](#documentation-requirements) in the issue. - - [Documentation guidelines](index.md). - - [Style guide](styleguide.md). - -Prior to merging, documentation changes committed by the developer must be reviewed by: - -- The code reviewer for the merge request. This is known as a technical review. -- Optionally, others involved in the work such as other developers or the Product Manager. -- The Technical Writer for the DevOps stage group, except in exceptional circumstances where a - [post-merge review](#post-merge-reviews) can be requested. -- A maintainer of the project. - -#### Product Managers - -Product Managers are responsible for the [documentation requirements](#documentation-requirements) -for a feature or feature enhancement. They can also: - -- Liaise with the Technical Writer for discussion and collaboration. -- Review documentation themselves. - -For issues requiring any new or updated documentation, the Product Manager must: - -- Add the ~documentation label. -- Confirm or add the [documentation requirements](#documentation-requirements). -- Ensure the issue contains: - - Any new or updated feature name. - - Overview, description, and use cases when applicable (as required by the - [documentation structure and template](structure.md)). - -Everyone is encouraged to draft the documentation requirements in the issue. However, a Product -Manager will: - -- When the issue is assigned a release milestone, review and update the Documentation details. -- By the kickoff, finalize the documentation details. - -#### Technical Writers - -Technical Writers are responsible for: - -- Participating in issues discussions and reviewing MRs for the upcoming milestone. -- Reviewing documentation requirements in issues when called upon. -- Answering questions, and helping and providing advice throughout the authoring and editing - process. -- Reviewing all significant new and updated documentation content, whether before merge or after it - is merged. -- Assisting the developer and Product Manager with feature documentation delivery. - -##### Planning - -The Technical Writer: - -- Reviews their group's `~feature` issues that are part of the next milestone to get a sense of the - scope of content likely to be authored. -- Recommends the `~documentation` label on issues from that list which don't have it but should, or - inquires with the PM to determine if documentation is truly required. -- For `~direction` issues from that list, reads the full issue and reviews its Documentation - requirements section. Addresses any recommendations or questions with the PMs and others - collaborating on the issue in order to refine or expand the Documentation requirements. - -##### Collaboration - -By default, the developer will work on documentation changes independently, but -the developer, Product Manager, or Technical Writer can propose a broader collaboration for -any given issue. - -Additionally, Technical Writers are available for questions at any time. - -##### Review - -Technical Writers: - -- Provide non-blocking reviews of all documentation changes, before or after the change is merged. -- Confirm that the documentation is: - - Clear. - - Grammatically correct. - - Discoverable. - - Navigable. -- Ensures that the documentation avoids: - - Redundancy. - - Bad file locations. - - Typos. - - Broken links. - -The Technical Writer will review the documentation to check that the developer and -code reviewer have ensured: - -- Clarity. -- Appropriate location, making sure the documentation is in the correct directories (often - reflecting how the product is structured) and has the correct name. -- Syntax, typos, and broken links. -- Improvements to the content. -- Accordance with the: - - [Documentation Style Guide](styleguide.md). - - [Structure and Template](structure.md) doc. - -### When documentation is required - -Documentation [is required](../contributing/merge_request_workflow.md#definition-of-done) for a -milestone when: - -- A new or enhanced feature is shipped that impacts the user or administrator experience. -- There are changes to the UI or API. -- A process, workflow, or previously documented feature is changed. -- A feature is deprecated or removed. - -NOTE: **Note:** -Documentation refactoring unrelated to a feature change is covered in the -[other process](#for-all-other-documentation), so that time-sensitive documentation updates are -prioritized. - -### Documentation requirements - -Requirements for the documentation of a feature should be included as part of the -issue for planning that feature in a **Documentation** section within the issue description. Issues -created using the [**Feature Proposal** template](https://gitlab.com/gitlab-org/gitlab/raw/master/.gitlab/issue_templates/Feature%20proposal.md) -have this section by default. - -Anyone can add these details, but the Product Manager who assigns the issue to a specific release -milestone will ensure these details are present and finalized by the time of that milestone's kickoff. - -Developers, Technical Writers, and others may help further refine this plan at any time on request. - -The following details should be included: - -- What concepts and procedures should the documentation guide and enable the user to understand or - accomplish? -- To this end, what new page(s) are needed, if any? What pages or subsections need updates? - Consider changes and additions to user, admin, and API documentation. -- For any guide or instruction set, should it help address a single use case, or be flexible to - address a certain range of use cases? -- Do we need to update a previously recommended workflow? Should we link the new feature from - various relevant locations? Consider all ways documentation should be affected. -- Are there any key terms or task descriptions that should be included so that the documentation is - found in relevant searches? -- Include suggested titles of any pages or subsection headings, if applicable. -- List any documentation that should be cross-linked, if applicable. - -### Including docs with code - -Currently, the Technical Writing team strongly encourages including documentation in -the same merge request as the code that it relates to, but this is not strictly mandatory. -It's still common for documentation to be added in an MR separate from the feature MR. - -Engineering teams may elect to adopt a workflow where it is **mandatory** that docs -are included in the code MR, as part of their [definition of done](../contributing/merge_request_workflow.md#definition-of-done). -When a team adopts this workflow, that team's engineers must include their docs in the **same** -MR as their feature code, at all times. - -#### Downsides of separate docs MRs - -A workflow that has documentation separated into its own MR has many downsides. - -If the documentation merges **before** the feature: - -- GitLab.com users might try to use the feature before it's released, driving support tickets. -- If the feature is delayed, the documentation might not be pulled/reverted in time and could be - accidentally included in the self-managed package for that release. - -If the documentation merges **after** the feature: - -- The feature might be included in the self-managed package, but without any documentation - if the docs MR misses the cutoff. -- A feature might show up in the GitLab.com UI before any documentation exists for it. - Users surprised by this feature will search for documentation and won't find it, possibly driving - support tickets. - -Having two separate MRs means: - -- Two different people might be responsible for merging one feature, which is not workable - with an asynchronous work style. The feature might merge while the technical writer is asleep, - creating a potentially lengthy delay between the two merges. -- If the docs MR is assigned to the same maintainer as responsible for the feature - code MR, they will have to review and juggle two MRs instead of dealing with just one. - -Documentation quality might be lower, because: - -- Having docs in a separate MR will mean far fewer people will see and verify them, - increasing the likelihood that issues will be missed. -- In a "split" workflow, engineers might only create the documentation MR once the - feature MR is ready, or almost ready. This gives the technical writer little time - to learn about the feature in order to do a good review. It also increases pressure - on them to review and merge faster than desired, letting problems slip in due to haste. - -#### Benefits of always including docs with code - -Including docs with code (and doing it early in the development process) has many benefits: - -- There are no timing issues connected to releases: - - If a feature slips to the next release, the documentation slips too. - - If the feature *just* makes it into a release, the docs *just* make it in too. - - If a feature makes it to GitLab.com early, the documentation will be ready for - our early adopters. -- Only a single person will be responsible for merging the feature (the code maintainer). -- The technical writer will have more time to gain an understanding of the feature - and will be better able to verify the content of the docs in the Review App or GDK. - They will also be able to offer advice for improving the UI text or offer additional use cases. -- The documentation will have increased visibility: - - Everyone involved in the merge request will see the docs. This could include product - managers, multiple engineers with deep domain knowledge, as well as the code reviewers - and maintainer. They will be more likely to catch issues with examples, as well - as background or concepts that the technical writer may not be aware of. - - Increasing visibility of the documentation also has the side effect of improving - *other* engineers' documentation. By reviewing each other's MRs, each engineer's - own documentation skills will improve. -- Thinking about the documentation early can help engineers generate better examples, - as they will need to think about what examples a user will want, and will need to - make sure the code they write implements that example properly. - -#### Docs with code as a workflow - -In order to have docs included with code as a mandatory workflow, some changes might -need to happen to a team's current workflow: - -- The engineers must strive to include the docs early in the development process, - to give ample time for review, not just from the technical writer, but also the - code reviewer and maintainer. -- Reviewers and maintainers must also review the docs during code reviews, to make - sure the described processes match the expected use of the feature, and that examples - are correct. They do *not* need to worry about style, grammar, and so on. -- The technical writer must be assigned the MR directly and not only pinged. Thanks - to the ability to have [multiple assignees for any MR](../../user/project/merge_requests/getting_started.md#multiple-assignees-starter), - this can be done at any time, but must be before the code maintainer review. It's - common to have both the docs and code reviews happening at the same time, with the - author, reviewer and technical writer discussing the docs together. -- When the docs are ready, the technical writer will click **Approve** and usually - will no longer be involved in the MR. If the feature changes during code review and - the docs are updated, the technical writer must be reassigned the MR to verify the - update. -- Maintainers are allowed to merge features with the docs "as-is", even if the technical - writer has not given final approval yet. The **docs reviews must not be blockers**. Therefore - it's important to get the docs included and assigned to the technical writers early. - If the feature is merged before final docs approval, the maintainer must create - a [post-merge follow-up issue](#post-merge-reviews), and assign it to both the engineer - and technical writer. - -Maintainers are allowed to merge features with the docs "as-is" even if the -technical writer has not given final approval yet but the merge request has -all other required approvals. - -You can visualize the parallel workflow for code and docs reviews as: - -```mermaid -graph TD - A("Feature MR Created (Engineer)") --> |Assign| B("Code Review (reviewer)") - B --> |"Approve / Reassign"| C("Code Review (maintainer)") - C --> |Approve| F("Merge (maintainer)") - A --> D("Docs Added (Engineer)") - D --> |Assign| E("Docs Review (Tech Writer)") - E --> |Approve| F -``` - -For complex features split over multiple merge requests: - -- If a merge request is implementing components for a future feature, but the components - are not accessible to users yet, then no documentation should be included. -- If a merge request will expose a feature to users in any way, such as an enabled - UI element, an API endpoint, or anything similar, then that MR **must** have docs. - Note that this may mean multiple docs additions could happen in the buildup to the - implementation of a single large feature, for example API docs and feature usage docs. -- If it's unclear which engineer should add the feature documentation into their - MR, the engineering manager should decide during planning, and tie the documentation - to the last MR that must be merged before a feature is considered released. - This is often, but not always, a frontend MR. - -## For all other documentation - Documentation changes that are not associated with the release of a new or updated feature do not take the `~feature` label, but still need the `~documentation` label. They may include: - Documentation created or updated to improve accuracy, completeness, ease of use, or any reason - other than a [feature change](#for-a-product-change). + other than a [feature change](https://about.gitlab.com/handbook/engineering/ux/technical-writing/workflow/#for-a-product-change). - Addressing gaps in existing documentation, or making improvements to existing documentation. - Work on special projects related to the documentation. -TIP: **Tip:** -Anyone can contribute a merge request or create an issue for GitLab's documentation. - -### Who updates the docs - -Anyone can contribute! You can create a merge request for documentation when: - -- You find errors or other room for improvement in existing documentation. -- You have an idea for all-new documentation that would help a GitLab user or administrator to - accomplish their work with GitLab. - -### How to update the docs +## How to update the docs To update GitLab documentation: @@ -464,7 +110,7 @@ The process involves the following: The process is reflected in the **Documentation** [merge request template](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/merge_request_templates/Documentation.md). -### Other ways to help +## Other ways to help If you have ideas for further documentation resources please [create an issue](https://gitlab.com/gitlab-org/gitlab/issues/new?issuable_template=Documentation) diff --git a/doc/development/ee_features.md b/doc/development/ee_features.md index bd70d5b87ba..3951b0516e8 100644 --- a/doc/development/ee_features.md +++ b/doc/development/ee_features.md @@ -3,7 +3,8 @@ - **Write the code and the tests.**: As with any code, EE features should have good test coverage to prevent regressions. - **Write documentation.**: Add documentation to the `doc/` directory. Describe - the feature and include screenshots, if applicable. + the feature and include screenshots, if applicable. Indicate [what editions](documentation/styleguide.md#product-badges) + the feature applies to. - **Submit a MR to the `www-gitlab-com` project.**: Add the new feature to the [EE features list](https://about.gitlab.com/features/). @@ -111,7 +112,7 @@ There are a few gotchas with it: smaller methods. Or create a "hook" method that is empty in CE, and with the EE-specific implementation in EE. - when the original implementation contains a guard clause (e.g. - `return unless condition`), we cannot easily extend the behaviour by + `return unless condition`), we cannot easily extend the behavior by overriding the method, because we can't know when the overridden method (i.e. calling `super` in the overriding method) would want to stop early. In this case, we shouldn't just override it, but update the original method @@ -131,7 +132,7 @@ There are a few gotchas with it: ``` Instead of just overriding `Base#execute`, we should update it and extract - the behaviour into another method: + the behavior into another method: ```ruby class Base @@ -513,7 +514,7 @@ do that, so we'll follow regular object-oriented practices that we define the interface first here. For example, suppose we have a few more optional parameters for EE. We can move the -paramters out of the `Grape::API` class to a helper module, so we can inject it +parameters out of the `Grape::API` class to a helper module, so we can inject it before it would be used in the class. ```ruby @@ -613,9 +614,9 @@ module EE end ``` -#### EE-specific behaviour +#### EE-specific behavior -Sometimes we need EE-specific behaviour in some of the APIs. Normally we could +Sometimes we need EE-specific behavior in some of the APIs. Normally we could use EE methods to override CE methods, however API routes are not methods and therefore can't be simply overridden. We need to extract them into a standalone method, or introduce some "hooks" where we could inject behavior in the CE diff --git a/doc/development/elasticsearch.md b/doc/development/elasticsearch.md index 758cecce315..185f536fc01 100644 --- a/doc/development/elasticsearch.md +++ b/doc/development/elasticsearch.md @@ -7,7 +7,7 @@ the [Elasticsearch integration documentation](../integration/elasticsearch.md#en ## Deep Dive -In June 2019, Mario de la Ossa hosted a [Deep Dive](https://gitlab.com/gitlab-org/create-stage/issues/1) on GitLab's [Elasticsearch integration](../integration/elasticsearch.md) to share his domain specific knowledge with anyone who may work in this part of the code base in the future. You can find the [recording on YouTube](https://www.youtube.com/watch?v=vrvl-tN2EaA), and the slides on [Google Slides](https://docs.google.com/presentation/d/1H-pCzI_LNrgrL5pJAIQgvLX8Ji0-jIKOg1QeJQzChug/edit) and in [PDF](https://gitlab.com/gitlab-org/create-stage/uploads/c5aa32b6b07476fa8b597004899ec538/Elasticsearch_Deep_Dive.pdf). Everything covered in this deep dive was accurate as of GitLab 12.0, and while specific details may have changed since then, it should still serve as a good introduction. +In June 2019, Mario de la Ossa hosted a Deep Dive (GitLab team members only: `https://gitlab.com/gitlab-org/create-stage/issues/1`) on GitLab's [Elasticsearch integration](../integration/elasticsearch.md) to share his domain specific knowledge with anyone who may work in this part of the code base in the future. You can find the [recording on YouTube](https://www.youtube.com/watch?v=vrvl-tN2EaA), and the slides on [Google Slides](https://docs.google.com/presentation/d/1H-pCzI_LNrgrL5pJAIQgvLX8Ji0-jIKOg1QeJQzChug/edit) and in [PDF](https://gitlab.com/gitlab-org/create-stage/uploads/c5aa32b6b07476fa8b597004899ec538/Elasticsearch_Deep_Dive.pdf). Everything covered in this deep dive was accurate as of GitLab 12.0, and while specific details may have changed since then, it should still serve as a good introduction. ## Supported Versions diff --git a/doc/development/emails.md b/doc/development/emails.md index a84895eef5b..e6e2ea8aae7 100644 --- a/doc/development/emails.md +++ b/doc/development/emails.md @@ -14,13 +14,11 @@ Please note that [S/MIME signed](../administration/smime_signing_email.md) email Rails provides a way to preview our mailer templates in HTML and plaintext using dummy data. -The previews live in [`app/mailers/previews`][previews] and can be viewed at +The previews live in [`app/mailers/previews`](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/app/mailers/previews) and can be viewed at [`/rails/mailers`](http://localhost:3000/rails/mailers). See the [Rails guides](https://guides.rubyonrails.org/action_mailer_basics.html#previewing-emails) for more information. -[previews]: https://gitlab.com/gitlab-org/gitlab-foss/tree/master/app/mailers/previews - ## Incoming email 1. Go to the GitLab installation directory. @@ -59,6 +57,9 @@ See the [Rails guides](https://guides.rubyonrails.org/action_mailer_basics.html# mailbox: "inbox" # The IDLE command timeout. idle_timeout: 60 + + # Whether to expunge (permanently remove) messages from the mailbox when they are deleted after delivery + expunge_deleted: false ``` As mentioned, the part after `+` is ignored, and this will end up in the mailbox for `gitlab-incoming@gmail.com`. @@ -87,7 +88,7 @@ for the format of the email key: - Actions are always at the end, separated by `-`. For example `-issue` or `-merge-request` - If your feature is related to a project, the key begins with the project identifiers (project path slug - and project id), separated by `-`. For example, `gitlab-org-gitlab-foss-20` + and project ID), separated by `-`. For example, `gitlab-org-gitlab-foss-20` - Additional information, such as an author's token, can be added between the project identifiers and the action, separated by `-`. For example, `gitlab-org-gitlab-foss-20-Author_Token12345678-issue` - You register your handlers in `lib/gitlab/email/handler.rb` diff --git a/doc/development/event_tracking/backend.md b/doc/development/event_tracking/backend.md index dc4d7279671..93e135772ef 100644 --- a/doc/development/event_tracking/backend.md +++ b/doc/development/event_tracking/backend.md @@ -1,5 +1,5 @@ --- -redirect_to: '../../telemetry/backend.md' +redirect_to: '../telemetry/index.md' --- -This document was moved to [another location](../../telemetry/backend.md). +This document was moved to [another location](../telemetry/index.md). diff --git a/doc/development/event_tracking/frontend.md b/doc/development/event_tracking/frontend.md index 0e98daf15bb..93e135772ef 100644 --- a/doc/development/event_tracking/frontend.md +++ b/doc/development/event_tracking/frontend.md @@ -1,5 +1,5 @@ --- -redirect_to: '../../telemetry/frontend.md' +redirect_to: '../telemetry/index.md' --- -This document was moved to [another location](../../telemetry/frontend.md). +This document was moved to [another location](../telemetry/index.md). diff --git a/doc/development/event_tracking/index.md b/doc/development/event_tracking/index.md index ae555e99c6b..93e135772ef 100644 --- a/doc/development/event_tracking/index.md +++ b/doc/development/event_tracking/index.md @@ -1,5 +1,5 @@ --- -redirect_to: '../../telemetry/index.md' +redirect_to: '../telemetry/index.md' --- -This document was moved to [another location](../../telemetry/index.md). +This document was moved to [another location](../telemetry/index.md). diff --git a/doc/development/experiment_guide/index.md b/doc/development/experiment_guide/index.md index ffa95d86876..f0e05139cba 100644 --- a/doc/development/experiment_guide/index.md +++ b/doc/development/experiment_guide/index.md @@ -32,10 +32,9 @@ The author then adds a comment to this piece of code and adds a link to the issu #... }, # Add your experiment here: - sign_up_flow: { - feature_toggle: :experimental_sign_up_flow, # Feature flag that will be used - environment: ::Gitlab.dev_env_or_com?, # Target environment - enabled_ratio: 0.1 # Percentage of users that will be part of the experiment. 10% of the users would be part of this experiments. + signup_flow: { + environment: ::Gitlab.dev_env_or_com?, # Target environment, defaults to enabled for development and GitLab.com + tracking_category: 'Growth::Acquisition::Experiment::SignUpFlow' # Used for providing the category when setting up tracking data } }.freeze ``` @@ -46,7 +45,7 @@ The author then adds a comment to this piece of code and adds a link to the issu class RegistrationController < Applicationcontroller def show # experiment_enabled?(:feature_name) is also available in views and helpers - if experiment_enabled?(:sign_up_flow) + if experiment_enabled?(:signup_flow) # render the experiment else # render the original version @@ -55,13 +54,18 @@ The author then adds a comment to this piece of code and adds a link to the issu end ``` -- Track necessary events. See the [telemetry guide](../../telemetry/index.md) for details. +- Track necessary events. See the [telemetry guide](../telemetry/index.md) for details. - After the merge request is merged, use [`chatops`](../../ci/chatops/README.md) in the -[appropriate channel](../feature_flags/controls.md#where-to-run-commands) to enable the feature flag and start the experiment. +[appropriate channel](../feature_flags/controls.md#communicate-the-change) to start the experiment for 10% of the users. +The feature flag should have the name of the experiment with the `_experiment_percentage` suffix appended. For visibility, please also share any commands run against production in the `#s_growth` channel: ```shell - /chatops run feature set --project=gitlab-org/gitlab experimental_sign_up_flow true + /chatops run feature set signup_flow_experiment_percentage 10 ``` - If you notice issues with the experiment, you can disable the experiment by setting the feature flag to `false` again. + If you notice issues with the experiment, you can disable the experiment by removing the feature flag: + + ```shell + /chatops run feature delete signup_flow_experiment_percentage + ``` diff --git a/doc/development/fe_guide/accessibility.md b/doc/development/fe_guide/accessibility.md index 4fd9a4fed60..998c71135fb 100644 --- a/doc/development/fe_guide/accessibility.md +++ b/doc/development/fe_guide/accessibility.md @@ -2,19 +2,12 @@ ## Resources -[Chrome Accessibility Developer Tools][chrome-accessibility-developer-tools] +[Chrome Accessibility Developer Tools](https://github.com/GoogleChrome/accessibility-developer-tools) are useful for testing for potential accessibility problems in GitLab. -The [axe][axe-website] browser extension (available for [Firefox][axe-firefox-extension] and [Chrome][axe-chrome-extension]) is +The [axe](https://www.deque.com/axe/) browser extension (available for [Firefox](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/) and [Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd)) is also a handy tool for running audits and getting feedback on markup, CSS and even potentially problematic color usages. Accessibility best-practices and more in-depth information are available on -[the Audit Rules page][audit-rules] for the Chrome Accessibility Developer Tools. The "[awesome a11y][awesome-a11y]" list is also a +[the Audit Rules page](https://github.com/GoogleChrome/accessibility-developer-tools/wiki/Audit-Rules) for the Chrome Accessibility Developer Tools. The [Awesome Accessibility](https://github.com/brunopulis/awesome-a11y) list is also a useful compilation of accessibility-related material. - -[chrome-accessibility-developer-tools]: https://github.com/GoogleChrome/accessibility-developer-tools -[audit-rules]: https://github.com/GoogleChrome/accessibility-developer-tools/wiki/Audit-Rules -[axe-website]: https://www.deque.com/axe/ -[axe-firefox-extension]: https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/ -[axe-chrome-extension]: https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd -[awesome-a11y]: https://github.com/brunopulis/awesome-a11y diff --git a/doc/development/fe_guide/axios.md b/doc/development/fe_guide/axios.md index 6e7cf523f36..f8d301dac5e 100644 --- a/doc/development/fe_guide/axios.md +++ b/doc/development/fe_guide/axios.md @@ -1,13 +1,13 @@ # Axios -We use [axios][axios] to communicate with the server in Vue applications and most new code. +We use [axios](https://github.com/axios/axios) to communicate with the server in Vue applications and most new code. In order to guarantee all defaults are set you *should not use `axios` directly*, you should import `axios` from `axios_utils`. ## CSRF token All our request require a CSRF token. -To guarantee this token is set, we are importing [axios][axios], setting the token, and exporting `axios` . +To guarantee this token is set, we are importing [axios](https://github.com/axios/axios), setting the token, and exporting `axios` . This exported module should be used instead of directly using `axios` to ensure the token is set. @@ -32,19 +32,16 @@ This exported module should be used instead of directly using `axios` to ensure ## Mock axios response in tests -To help us mock the responses we are using [axios-mock-adapter][axios-mock-adapter]. +To help us mock the responses we are using [axios-mock-adapter](https://github.com/ctimmerm/axios-mock-adapter). -Advantages over [`spyOn()`]: +Advantages over [`spyOn()`](https://jasmine.github.io/api/edge/global.html#spyOn): - no need to create response objects - does not allow call through (which we want to avoid) - simple API to test error cases - provides `replyOnce()` to allow for different responses -We have also decided against using [axios interceptors] because they are not suitable for mocking. - -[axios interceptors]: https://github.com/axios/axios#interceptors -[`spyOn()`]: https://jasmine.github.io/api/edge/global.html#spyOn +We have also decided against using [axios interceptors](https://github.com/axios/axios#interceptors) because they are not suitable for mocking. ### Example @@ -77,8 +74,3 @@ Because polling function requires a header object, we need to always include an ```javascript mock.onGet('/users').reply(200, { foo: 'bar' }, {}); ``` - -[axios]: https://github.com/axios/axios -[axios-instance]: #creating-an-instance -[axios-interceptors]: https://github.com/axios/axios#interceptors -[axios-mock-adapter]: https://github.com/ctimmerm/axios-mock-adapter diff --git a/doc/development/fe_guide/design_patterns.md b/doc/development/fe_guide/design_patterns.md index 72a7861ffcb..5b7b16a9a8a 100644 --- a/doc/development/fe_guide/design_patterns.md +++ b/doc/development/fe_guide/design_patterns.md @@ -75,6 +75,4 @@ class Foo { new Foo({ container: '.my-element' }); ``` -You can find an example of the above in this [class][container-class-example]; - -[container-class-example]: https://gitlab.com/gitlab-org/gitlab/blob/master/app/assets/javascripts/mini_pipeline_graph_dropdown.js +You can find an example of the above in this [class](https://gitlab.com/gitlab-org/gitlab/blob/master/app/assets/javascripts/mini_pipeline_graph_dropdown.js); diff --git a/doc/development/fe_guide/droplab/droplab.md b/doc/development/fe_guide/droplab/droplab.md index 4d7c882dc09..83bc4216403 100644 --- a/doc/development/fe_guide/droplab/droplab.md +++ b/doc/development/fe_guide/droplab/droplab.md @@ -26,7 +26,7 @@ If you do not provide any arguments, it will globally query and instantiate all <ul> ``` -```js +```javascript const droplab = new DropLab(); droplab.init(); ``` @@ -47,7 +47,7 @@ You can add static list items. <ul> ``` -```js +```javascript const droplab = new DropLab(); droplab.init(); ``` @@ -65,7 +65,7 @@ a non-global instance of DropLab using the `DropLab.prototype.init` method. <ul> ``` -```js +```javascript const trigger = document.getElementById('trigger'); const list = document.getElementById('list'); @@ -83,7 +83,7 @@ You can also add hooks to an existing DropLab instance using `DropLab.prototype. <ul id="list" data-dropdown><!-- ... --><ul> ``` -```js +```javascript const droplab = new DropLab(); droplab.init(); @@ -114,7 +114,7 @@ for all `data-dynamic` dropdown lists tracked by that DropLab instance. </ul> ``` -```js +```javascript const droplab = new DropLab(); droplab.init().addData([{ @@ -137,7 +137,7 @@ the data as the second argument and the `id` of the trigger element as the first </ul> ``` -```js +```javascript const droplab = new DropLab(); droplab.init().addData('trigger', [{ @@ -167,7 +167,7 @@ dropdown lists, one of which is dynamic. </div> ``` -```js +```javascript const droplab = new DropLab(); droplab.init().addData('trigger', [{ @@ -224,7 +224,7 @@ Some plugins require configuration values, the config object can be passed as th <ul id="list" data-dropdown><!-- ... --><ul> ``` -```js +```javascript const droplab = new DropLab(); const trigger = document.getElementById('trigger'); @@ -249,7 +249,7 @@ droplab.init(trigger, list, [droplabAjax], { When plugins are initialised for a droplab trigger+dropdown, DropLab will call the plugins `init` function, so this must be implemented in the plugin. -```js +```javascript class MyPlugin { static init() { this.someProp = 'someProp'; diff --git a/doc/development/fe_guide/droplab/plugins/ajax.md b/doc/development/fe_guide/droplab/plugins/ajax.md index 77ba6f739e6..abc208e7568 100644 --- a/doc/development/fe_guide/droplab/plugins/ajax.md +++ b/doc/development/fe_guide/droplab/plugins/ajax.md @@ -18,7 +18,7 @@ Add the `Ajax` object to the plugins array of a `DropLab.prototype.init` or `Dro <ul id="list" data-dropdown><!-- ... --><ul> ``` -```js +```javascript const droplab = new DropLab(); const trigger = document.getElementById('trigger'); diff --git a/doc/development/fe_guide/droplab/plugins/filter.md b/doc/development/fe_guide/droplab/plugins/filter.md index b867394a241..876149e4872 100644 --- a/doc/development/fe_guide/droplab/plugins/filter.md +++ b/doc/development/fe_guide/droplab/plugins/filter.md @@ -18,7 +18,7 @@ Add the `Filter` object to the plugins array of a `DropLab.prototype.init` or `D <ul> ``` -```js +```javascript const droplab = new DropLab(); const trigger = document.getElementById('trigger'); diff --git a/doc/development/fe_guide/droplab/plugins/input_setter.md b/doc/development/fe_guide/droplab/plugins/input_setter.md index db492da478a..9b2e1e8faab 100644 --- a/doc/development/fe_guide/droplab/plugins/input_setter.md +++ b/doc/development/fe_guide/droplab/plugins/input_setter.md @@ -23,7 +23,7 @@ You can also set the `InputSetter` config to an array of objects, which will all <ul> ``` -```js +```javascript const droplab = new DropLab(); const trigger = document.getElementById('trigger'); diff --git a/doc/development/fe_guide/event_tracking.md b/doc/development/fe_guide/event_tracking.md index ae555e99c6b..93e135772ef 100644 --- a/doc/development/fe_guide/event_tracking.md +++ b/doc/development/fe_guide/event_tracking.md @@ -1,5 +1,5 @@ --- -redirect_to: '../../telemetry/index.md' +redirect_to: '../telemetry/index.md' --- -This document was moved to [another location](../../telemetry/index.md). +This document was moved to [another location](../telemetry/index.md). diff --git a/doc/development/fe_guide/frontend_faq.md b/doc/development/fe_guide/frontend_faq.md index 3b0b1d8f0da..8f8f162609a 100644 --- a/doc/development/fe_guide/frontend_faq.md +++ b/doc/development/fe_guide/frontend_faq.md @@ -39,7 +39,7 @@ bundle exec rake routes | grep "issues" ### 2. `modal_copy_button` vs `clipboard_button` -The `clipboard_button` uses the `copy_to_clipboard.js` behaviour, which is +The `clipboard_button` uses the `copy_to_clipboard.js` behavior, which is initialized on page load, so if there are vue-based clipboard buttons that don't exist at page load (such as ones in a `GlModal`), they do not have the click handlers associated with the clipboard package. diff --git a/doc/development/fe_guide/graphql.md b/doc/development/fe_guide/graphql.md index 8ccc734ee35..caf84d04490 100644 --- a/doc/development/fe_guide/graphql.md +++ b/doc/development/fe_guide/graphql.md @@ -10,13 +10,13 @@ their execution by clicking **Execute query** button on the top left:  -We use [Apollo] and [Vue Apollo][vue-apollo] for working with GraphQL +We use [Apollo](https://www.apollographql.com/) and [Vue Apollo](https://github.com/vuejs/vue-apollo) for working with GraphQL on the frontend. ## Apollo Client To save duplicated clients getting created in different apps, we have a -[default client][default-client] that should be used. This setups the +[default client](https://gitlab.com/gitlab-org/gitlab/blob/master/app/assets/javascripts/lib/graphql.js) that should be used. This setups the Apollo client with the correct URL and also sets the CSRF headers. Default client accepts two parameters: `resolvers` and `config`. @@ -73,7 +73,7 @@ More about fragments: ## Usage in Vue -To use Vue Apollo, import the [Vue Apollo][vue-apollo] plugin as well +To use Vue Apollo, import the [Vue Apollo](https://github.com/vuejs/vue-apollo) plugin as well as the default client. This should be created at the same point the Vue application is mounted. @@ -94,7 +94,7 @@ new Vue({ }); ``` -Read more about [Vue Apollo][vue-apollo] in the [Vue Apollo documentation](https://vue-apollo.netlify.com/guide/). +Read more about [Vue Apollo](https://github.com/vuejs/vue-apollo) in the [Vue Apollo documentation](https://vue-apollo.netlify.com/guide/). ### Local state with Apollo @@ -212,7 +212,7 @@ Read more about local state management with Apollo in the [Vue Apollo documentat When Apollo Client is used within Vuex and fetched data is stored in the Vuex store, there is no need in keeping Apollo Client cache enabled. Otherwise we would have data from the API stored in two places - Vuex store and Apollo Client cache. More to say, with Apollo default settings, a subsequent fetch from the GraphQL API could result in fetching data from Apollo cache (in the case where we have the same query and variables). To prevent this behavior, we need to disable Apollo Client cache passing a valid `fetchPolicy` option to its constructor: -```js +```javascript import fetchPolicies from '~/graphql_shared/fetch_policy_constants'; export const gqClient = createGqClient( @@ -298,7 +298,8 @@ handleClick() { GitLab's GraphQL API uses [Relay-style cursor pagination](https://www.apollographql.com/docs/react/data/pagination/#cursor-based) for connection types. This means a "cursor" is used to keep track of where in the data -set the next items should be fetched from. +set the next items should be fetched from. [GraphQL Ruby Connection Concepts](https://graphql-ruby.org/pagination/connection_concepts.html) +is a good overview and introduction to connections. Every connection type (for example, `DesignConnection` and `DiscussionConnection`) has a field `pageInfo` that contains an information required for pagination: @@ -426,7 +427,7 @@ This should be resolved in the scope of the issue #### Mocking response as component data -With [Vue test utils][vue-test-utils] it is easy to quickly test components that +With [Vue test utils](https://vue-test-utils.vuejs.org/) it is easy to quickly test components that fetch GraphQL queries. The simplest way is to use `shallowMount` and then set the data on the component @@ -598,11 +599,4 @@ defaultClient.query({ query }) .then(result => console.log(result)); ``` -Read more about the [Apollo] client in the [Apollo documentation](https://www.apollographql.com/docs/tutorial/client/). - -[Apollo]: https://www.apollographql.com/ -[vue-apollo]: https://github.com/vuejs/vue-apollo -[feature-flags]: ../feature_flags.md -[default-client]: https://gitlab.com/gitlab-org/gitlab/blob/master/app/assets/javascripts/lib/graphql.js -[vue-test-utils]: https://vue-test-utils.vuejs.org/ -[apollo-link-state]: https://www.apollographql.com/docs/link/links/state.html +Read more about the [Apollo](https://www.apollographql.com/) client in the [Apollo documentation](https://www.apollographql.com/docs/tutorial/client/). diff --git a/doc/development/fe_guide/icons.md b/doc/development/fe_guide/icons.md index ea321330c41..4fb738f5466 100644 --- a/doc/development/fe_guide/icons.md +++ b/doc/development/fe_guide/icons.md @@ -1,8 +1,8 @@ # Icons and SVG Illustrations -We manage our own Icon and Illustration library in the [`gitlab-svgs`][gitlab-svgs] repository. -This repository is published on [npm][npm] and managed as a dependency via yarn. -You can browse all available Icons and Illustrations [here][svg-preview]. +We manage our own Icon and Illustration library in the [`gitlab-svgs`](https://gitlab.com/gitlab-org/gitlab-svgs) repository. +This repository is published on [npm](https://www.npmjs.com/package/@gitlab/svgs) and managed as a dependency via yarn. +You can browse all available Icons and Illustrations [here](https://gitlab-org.gitlab.io/gitlab-svgs). To upgrade to a new version run `yarn upgrade @gitlab/svgs`. ## Icons @@ -22,7 +22,7 @@ sprite_icon(icon_name, size: nil, css_class: '') ``` - **icon_name** Use the icon_name that you can find in the SVG Sprite - ([Overview is available here][svg-preview]). + ([Overview is available here](https://gitlab-org.gitlab.io/gitlab-svgs)). - **size (optional)** Use one of the following sizes : 16, 24, 32, 48, 72 (this will be translated into a `s16` class) - **css_class (optional)** If you want to add additional css classes @@ -42,7 +42,7 @@ sprite_icon(icon_name, size: nil, css_class: '') ### Usage in Vue -[GitLab UI][gitlab-ui], our components library, provides a component to display sprite icons. +[GitLab UI](https://gitlab-org.gitlab.io/gitlab-ui/), our components library, provides a component to display sprite icons. Sample usage : @@ -65,7 +65,7 @@ export default { </template> ``` -- **name** Name of the Icon in the SVG Sprite ([Overview is available here][svg-preview]). +- **name** Name of the Icon in the SVG Sprite ([Overview is available here](https://gitlab-org.gitlab.io/gitlab-svgs)). - **size (optional)** Number value for the size which is then mapped to a specific CSS class (Available Sizes: 8, 12, 16, 18, 24, 32, 48, 72 are mapped to `sXX` css classes) - **css-classes (optional)** Additional CSS Classes to add to the svg tag. @@ -111,8 +111,3 @@ export default { <img :src="svgIllustrationPath" /> </template> ``` - -[npm]: https://www.npmjs.com/package/@gitlab/svgs -[gitlab-svgs]: https://gitlab.com/gitlab-org/gitlab-svgs -[svg-preview]: https://gitlab-org.gitlab.io/gitlab-svgs -[gitlab-ui]: https://gitlab-org.gitlab.io/gitlab-ui/ diff --git a/doc/development/fe_guide/index.md b/doc/development/fe_guide/index.md index 790cd94cec4..8fd6ba459b9 100644 --- a/doc/development/fe_guide/index.md +++ b/doc/development/fe_guide/index.md @@ -5,17 +5,17 @@ across GitLab's frontend team. ## Overview -GitLab is built on top of [Ruby on Rails](https://rubyonrails.org) using [Haml][haml] and also a JavaScript based Frontend with [Vue.js](https://vuejs.org). -Be wary of [the limitations that come with using Hamlit][hamlit-limits]. We also use [SCSS](https://sass-lang.com) and plain JavaScript with -modern ECMAScript standards supported through [Babel][babel] and ES module support through [webpack][webpack]. +GitLab is built on top of [Ruby on Rails](https://rubyonrails.org) using [Haml](http://haml.info/) and also a JavaScript based Frontend with [Vue.js](https://vuejs.org). +Be wary of [the limitations that come with using Hamlit](https://github.com/k0kubun/hamlit/blob/master/REFERENCE.md#limitations). We also use [SCSS](https://sass-lang.com) and plain JavaScript with +modern ECMAScript standards supported through [Babel](https://babeljs.io/) and ES module support through [webpack](https://webpack.js.org/). Working with our frontend assets requires Node (v10.13.0 or greater) and Yarn (v1.10.0 or greater). You can find information on how to install these on our -[installation guide][install]. +[installation guide](../../install/installation.md#4-node). ### Browser Support -For our currently-supported browsers, see our [requirements][requirements]. +For our currently-supported browsers, see our [requirements](../../install/requirements.md#supported-web-browsers). Use [BrowserStack](https://www.browserstack.com/) to test with our supported browsers. Login to BrowserStack with the credentials saved in GitLab's [shared 1Password account](https://about.gitlab.com/handbook/security/#1password-for-teams). @@ -83,7 +83,7 @@ Read the [frontend's FAQ](frontend_faq.md) for common small pieces of helpful in See the relevant style guides for our guidelines and for information on linting: - [JavaScript](style/javascript.md). Our guide is based on -the excellent [Airbnb][airbnb-js-style-guide] style guide with a few small +the excellent [Airbnb](https://github.com/airbnb/javascript) style guide with a few small changes. - [SCSS](style/scss.md): our SCSS conventions which are enforced through [`scss-lint`](https://github.com/sds/scss-lint). - [HTML](style/html.md). Guidelines for writing HTML code consistent with the rest of the codebase. @@ -109,14 +109,3 @@ Our accessibility standards and resources. Frontend internationalization support is described in [this document](../i18n/). The [externalization part of the guide](../i18n/externalization.md) explains the helpers/methods available. - -[haml]: http://haml.info/ -[hamlit]: https://github.com/k0kubun/hamlit -[hamlit-limits]: https://github.com/k0kubun/hamlit/blob/master/REFERENCE.md#limitations -[babel]: https://babeljs.io/ -[webpack]: https://webpack.js.org/ -[jquery]: https://jquery.com/ -[axios]: https://github.com/axios/axios -[airbnb-js-style-guide]: https://github.com/airbnb/javascript -[install]: ../../install/installation.md#4-node -[requirements]: ../../install/requirements.md#supported-web-browsers diff --git a/doc/development/fe_guide/performance.md b/doc/development/fe_guide/performance.md index fcba8758c2f..aaa6bb16fab 100644 --- a/doc/development/fe_guide/performance.md +++ b/doc/development/fe_guide/performance.md @@ -41,9 +41,9 @@ But in general it should be handled automatically through a `MutationObserver` i Only animate `opacity` & `transform` properties. Other properties (such as `top`, `left`, `margin`, and `padding`) all cause Layout to be recalculated, which is much more expensive. For details on this, see "Styles that Affect Layout" in -[High Performance Animations][high-perf-animations]. +[High Performance Animations](https://www.html5rocks.com/en/tutorials/speed/high-performance-animations/). -If you _do_ need to change layout (e.g. a sidebar that pushes main content over), prefer [FLIP][flip] to change expensive +If you _do_ need to change layout (e.g. a sidebar that pushes main content over), prefer [FLIP](https://aerotwist.com/blog/flip-your-animations/) to change expensive properties once, and handle the actual animation with transforms. ## Reducing Asset Footprint @@ -160,18 +160,13 @@ General tips: - If some functionality can reasonably be achieved without adding extra libraries, avoid them. - Use page-specific JavaScript as described above to load libraries that are only needed on certain pages. - Use code-splitting dynamic imports wherever possible to lazy-load code that is not needed initially. -- [High Performance Animations][high-perf-animations] +- [High Performance Animations](https://www.html5rocks.com/en/tutorials/speed/high-performance-animations/) --- ## Additional Resources - [WebPage Test](https://www.webpagetest.org) for testing site loading time and size. -- [Google PageSpeed Insights][pagespeed-insights] grades web pages and provides feedback to improve the page. +- [Google PageSpeed Insights](https://developers.google.com/speed/pagespeed/insights/) grades web pages and provides feedback to improve the page. - [Profiling with Chrome DevTools](https://developers.google.com/web/tools/chrome-devtools/) -- [Browser Diet][browser-diet] is a community-built guide that catalogues practical tips for improving web page performance. - -[pagespeed-insights]: https://developers.google.com/speed/pagespeed/insights/ -[browser-diet]: https://browserdiet.com/ -[high-perf-animations]: https://www.html5rocks.com/en/tutorials/speed/high-performance-animations/ -[flip]: https://aerotwist.com/blog/flip-your-animations/ +- [Browser Diet](https://browserdiet.com/) is a community-built guide that catalogues practical tips for improving web page performance. diff --git a/doc/development/fe_guide/security.md b/doc/development/fe_guide/security.md index 7dba61d6b45..a001dd83ab7 100644 --- a/doc/development/fe_guide/security.md +++ b/doc/development/fe_guide/security.md @@ -2,8 +2,8 @@ ## Resources -[Mozilla’s HTTP Observatory CLI][observatory-cli] and the -[Qualys SSL Labs Server Test][qualys-ssl] are good resources for finding +[Mozilla’s HTTP Observatory CLI](https://github.com/mozilla/http-observatory-cli) and the +[Qualys SSL Labs Server Test](https://www.ssllabs.com/ssltest/analyze.html) are good resources for finding potential problems and ensuring compliance with security best practices. <!-- Uncomment these sections when CSP/SRI are implemented. @@ -29,14 +29,14 @@ Some exceptions include: - Connecting with GitHub, Bitbucket, GitLab.com, etc. to allow project importing. - Connecting with Google, Twitter, GitHub, etc. to allow OAuth authentication. -We use [the Secure Headers gem][secure_headers] to enable Content +We use [the Secure Headers gem](https://github.com/twitter/secureheaders) to enable Content Security Policy headers in the GitLab Rails app. Some resources on implementing Content Security Policy: -- [MDN Article on CSP][mdn-csp] -- [GitHub’s CSP Journey on the GitHub Engineering Blog][github-eng-csp] -- The Dropbox Engineering Blog's series on CSP: [1][dropbox-csp-1], [2][dropbox-csp-2], [3][dropbox-csp-3], [4][dropbox-csp-4] +- [MDN Article on CSP](https://developer.mozilla.org/en-US/docs/Web/Security/CSP) +- [GitHub’s CSP Journey on the GitHub Engineering Blog](http://githubengineering.com/githubs-csp-journey/) +- The Dropbox Engineering Blog's series on CSP: [1](https://blogs.dropbox.com/tech/2015/09/on-csp-reporting-and-filtering/), [2](https://blogs.dropbox.com/tech/2015/09/unsafe-inline-and-nonce-deployment/), [3](https://blogs.dropbox.com/tech/2015/09/csp-the-unexpected-eval/), [4](https://blogs.dropbox.com/tech/2015/09/csp-third-party-integrations-and-privilege-separation/) ### Subresource Integrity (SRI) @@ -52,8 +52,8 @@ All CSS and JavaScript assets should use Subresource Integrity. Some resources on implementing Subresource Integrity: -- [MDN Article on SRI][mdn-sri] -- [Subresource Integrity on the GitHub Engineering Blog][github-eng-sri] +- [MDN Article on SRI](https://developer.mozilla.org/en-us/docs/web/security/subresource_integrity) +- [Subresource Integrity on the GitHub Engineering Blog](http://githubengineering.com/subresource-integrity/) --> @@ -67,7 +67,7 @@ such as with reCAPTCHA, which cannot be used without an `iframe`. ## Avoiding inline scripts and styles -In order to protect users from [XSS vulnerabilities][xss], we will disable +In order to protect users from [XSS vulnerabilities](https://en.wikipedia.org/wiki/Cross-site_scripting), we will disable inline scripts in the future using Content Security Policy. While inline scripts can be useful, they're also a security concern. If @@ -77,16 +77,3 @@ inject scripts into the web app. Inline styles should be avoided in almost all cases, they should only be used when no alternatives can be found. This allows reusability of styles as well as readability. - -[observatory-cli]: https://github.com/mozilla/http-observatory-cli -[qualys-ssl]: https://www.ssllabs.com/ssltest/analyze.html -[secure_headers]: https://github.com/twitter/secureheaders -[mdn-csp]: https://developer.mozilla.org/en-US/docs/Web/Security/CSP -[github-eng-csp]: http://githubengineering.com/githubs-csp-journey/ -[dropbox-csp-1]: https://blogs.dropbox.com/tech/2015/09/on-csp-reporting-and-filtering/ -[dropbox-csp-2]: https://blogs.dropbox.com/tech/2015/09/unsafe-inline-and-nonce-deployment/ -[dropbox-csp-3]: https://blogs.dropbox.com/tech/2015/09/csp-the-unexpected-eval/ -[dropbox-csp-4]: https://blogs.dropbox.com/tech/2015/09/csp-third-party-integrations-and-privilege-separation/ -[mdn-sri]: https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity -[github-eng-sri]: http://githubengineering.com/subresource-integrity/ -[xss]: https://en.wikipedia.org/wiki/Cross-site_scripting diff --git a/doc/development/fe_guide/style/javascript.md b/doc/development/fe_guide/style/javascript.md index 7951c702601..b69a6f1941c 100644 --- a/doc/development/fe_guide/style/javascript.md +++ b/doc/development/fe_guide/style/javascript.md @@ -184,6 +184,9 @@ This can help to quickly understand the control flow. // bad if (isThingNull) return ''; +if (isThingNull) + return ''; + // good if (isThingNull) { return ''; @@ -192,7 +195,7 @@ if (isThingNull) { ## ESLint -ESLint behaviour can be found in our [tooling guide](../tooling.md). +ESLint behavior can be found in our [tooling guide](../tooling.md). ## IIFEs diff --git a/doc/development/fe_guide/style/scss.md b/doc/development/fe_guide/style/scss.md index 6c0247ad00c..336c9b8ca35 100644 --- a/doc/development/fe_guide/style/scss.md +++ b/doc/development/fe_guide/style/scss.md @@ -9,30 +9,25 @@ easy to maintain, and performant for the end-user. ## Rules +Our CSS is a mixture of current and legacy approaches. That means sometimes it may be difficult to follow this guide to the letter; it means you will definitely run into exceptions, where following the guide is difficult to impossible without outsized effort. In those cases, you may work with your reviewers and maintainers to identify an approach that does not fit these rules. Please endeavor to limit these cases. + ### Utility Classes -As part of the effort for [cleaning up our CSS and moving our components into `gitlab-ui`](https://gitlab.com/groups/gitlab-org/-/epics/950) -led by the [GitLab UI WG](https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_requests/20623) we prefer the use of utility classes over adding new CSS. However, complex CSS can be addressed by adding component classes. +In order to reduce the generation of more CSS as our site grows, prefer the use of utility classes over adding new CSS. In complex cases, CSS can be addressed by adding component classes. #### Where are utility classes defined? -- [Bootstrap's Utility Classes](https://getbootstrap.com/docs/4.3/utilities/) -- [`common.scss`](https://gitlab.com/gitlab-org/gitlab/blob/master/app/assets/stylesheets/framework/common.scss) (old) -- [`utilities.scss`](https://gitlab.com/gitlab-org/gitlab/blob/master/app/assets/stylesheets/utilities.scss) (new) +Prefer the use of [utility classes defined in GitLab UI](https://gitlab.com/gitlab-org/gitlab-ui/-/blob/master/doc/css.md#utilities). An easy list of classes can also be [seen on Unpkg](https://unpkg.com/browse/@gitlab/ui/src/scss/utilities.scss). -#### Where should I put new utility classes? +Classes in [`utilities.scss`](https://gitlab.com/gitlab-org/gitlab/blob/master/app/assets/stylesheets/utilities.scss) and [`common.scss`](https://gitlab.com/gitlab-org/gitlab/blob/master/app/assets/stylesheets/framework/common.scss) are being deprecated. Classes in [`common.scss`](https://gitlab.com/gitlab-org/gitlab/blob/master/app/assets/stylesheets/framework/common.scss) that use non-design system values should be avoided in favor of conformant values. -New utility classes should be added to [`utilities.scss`](https://gitlab.com/gitlab-org/gitlab/blob/master/app/assets/stylesheets/utilities.scss). Existing classes include: +Avoid [Bootstrap's Utility Classes](https://getbootstrap.com/docs/4.3/utilities/). -| Name | Pattern | Example | -|------|---------|---------| -| Background color | `.bg-{variant}-{shade}` | `.bg-warning-400` | -| Text color | `.text-{variant}-{shade}` | `.text-success-500` | -| Font size | `.text-{size}` | `.text-2` | +#### Where should I put new utility classes? -- `{variant}` is one of 'primary', 'secondary', 'success', 'warning', 'error' -- `{shade}` is one of the shades listed on [colors](https://design.gitlab.com/product-foundations/colors/) -- `{size}` is a number from 1-6 from our [Type scale](https://design.gitlab.com/product-foundations/typography/) +If a class you need has not been added to GitLab UI, you get to add it! Follow the naming patterns documented in the [utility files](https://gitlab.com/gitlab-org/gitlab-ui/-/tree/master/src/scss/utility-mixins) and refer to [GitLab UI's CSS documentation](https://gitlab.com/gitlab-org/gitlab-ui/-/blob/master/doc/contributing/adding_css.md#adding-utility-mixins) for more details, especially about adding responsive and stateful rules. + +If it is not possible to wait for a GitLab UI update (generally one day), add the class to [`utilities.scss`](https://gitlab.com/gitlab-org/gitlab/blob/master/app/assets/stylesheets/utilities.scss) following the same naming conventions documented in GitLab UI. A follow—up issue to backport the class to GitLab UI and delete it from GitLab should be opened. #### When should I create component classes? @@ -77,6 +72,24 @@ CSS classes should use the `lowercase-hyphenated` format rather than } ``` +Class names should be used instead of tag name selectors. +Using tag name selectors are discouraged in CSS because +they can affect unintended elements in the hierarchy. +Also, since they are not meaningful names, they do not +add meaning to the code. + +```scss +// Bad +ul { + color: #fff; +} + +// Good +.class-name { + color: #fff; +} +``` + ### Formatting You should always use a space before a brace, braces should be on the same @@ -254,8 +267,8 @@ documentation includes [a full list of their linters](https://github.com/sds/scs ### Fixing issues If you want to automate changing a large portion of the codebase to conform to -the SCSS style guide, you can use [CSSComb][csscomb]. First install -[Node][node] and [NPM][npm], then run `npm install csscomb -g` to install +the SCSS style guide, you can use [CSSComb](https://github.com/csscomb/csscomb.js). First install +[Node](https://github.com/nodejs/node) and [NPM](https://www.npmjs.com/), then run `npm install csscomb -g` to install CSSComb globally (system-wide). Run it in the GitLab directory with `csscomb app/assets/stylesheets` to automatically fix issues with CSS/SCSS. @@ -279,7 +292,3 @@ Make sure a comment is added on the line above the `disable` rule, otherwise the linter will throw a warning. `DisableLinterReason` is enabled to make sure the style guide isn't being ignored, and to communicate to others why the style guide is ignored in this instance. - -[csscomb]: https://github.com/csscomb/csscomb.js -[node]: https://github.com/nodejs/node -[npm]: https://www.npmjs.com/ diff --git a/doc/development/fe_guide/style/vue.md b/doc/development/fe_guide/style/vue.md index a7d8fc61752..46305cc7217 100644 --- a/doc/development/fe_guide/style/vue.md +++ b/doc/development/fe_guide/style/vue.md @@ -53,7 +53,7 @@ Please check this [rules](https://github.com/vuejs/eslint-plugin-vue#bulb-rules) ## Naming -1. **Extensions**: Use `.vue` extension for Vue components. Do not use `.js` as file extension ([#34371]). +1. **Extensions**: Use `.vue` extension for Vue components. Do not use `.js` as file extension ([#34371](https://gitlab.com/gitlab-org/gitlab-foss/issues/34371)). 1. **Reference Naming**: Use PascalCase for their instances: ```javascript @@ -89,8 +89,6 @@ Please check this [rules](https://github.com/vuejs/eslint-plugin-vue#bulb-rules) <component my-prop="prop" /> ``` -[#34371]: https://gitlab.com/gitlab-org/gitlab-foss/issues/34371 - ## Alignment 1. Follow these alignment styles for the template method: @@ -327,7 +325,7 @@ When using `v-for` you need to provide a *unique* `:key` attribute for each item </div> ``` -1. When the elements being iterated don't have a unique id, you can use the array index as the `:key` attribute +1. When the elements being iterated don't have a unique ID, you can use the array index as the `:key` attribute ```html <div diff --git a/doc/development/fe_guide/tooling.md b/doc/development/fe_guide/tooling.md index 0e8f5ed05ed..585cd969c96 100644 --- a/doc/development/fe_guide/tooling.md +++ b/doc/development/fe_guide/tooling.md @@ -56,13 +56,13 @@ When declaring multiple globals, always use one `/* global [name] */` line per v ## Formatting with Prettier -Our code is automatically formatted with [Prettier](https://prettier.io) to follow our style guides. Prettier is taking care of formatting .js, .vue, and .scss files based on the standard prettier rules. You can find all settings for Prettier in `.prettierrc`. +Our code is automatically formatted with [Prettier](https://prettier.io) to follow our style guides. Prettier is taking care of formatting `.js`, `.vue`, and `.scss` files based on the standard prettier rules. You can find all settings for Prettier in `.prettierrc`. ### Editor The easiest way to include prettier in your workflow is by setting up your preferred editor (all major editors are supported) accordingly. We suggest setting up prettier to run automatically when each file is saved. Find [here](https://prettier.io/docs/en/editors.html) the best way to set it up in your preferred editor. -Please take care that you only let Prettier format the same file types as the global Yarn script does (.js, .vue, and .scss). In VSCode by example you can easily exclude file formats in your settings file: +Please take care that you only let Prettier format the same file types as the global Yarn script does (`.js`, `.vue`, and `.scss`). In VSCode by example you can easily exclude file formats in your settings file: ```json "prettier.disableLanguages": [ diff --git a/doc/development/fe_guide/vue.md b/doc/development/fe_guide/vue.md index aeedd57fd83..972c2ded9c9 100644 --- a/doc/development/fe_guide/vue.md +++ b/doc/development/fe_guide/vue.md @@ -12,17 +12,17 @@ What is described in the following sections can be found in these examples: ## Vue architecture -All new features built with Vue.js must follow a [Flux architecture][flux]. +All new features built with Vue.js must follow a [Flux architecture](https://facebook.github.io/flux/). The main goal we are trying to achieve is to have only one data flow and only one data entry. In order to achieve this goal we use [vuex](#vuex). -You can also read about this architecture in vue docs about [state management][state-management] -and about [one way data flow][one-way-data-flow]. +You can also read about this architecture in vue docs about [state management](https://vuejs.org/v2/guide/state-management.html#Simple-State-Management-from-Scratch) +and about [one way data flow](https://vuejs.org/v2/guide/components.html#One-Way-Data-Flow). ### Components and Store -In some features implemented with Vue.js, like the [issue board][issue-boards] -or [environments table][environments-table] +In some features implemented with Vue.js, like the [issue board](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/app/assets/javascripts/boards) +or [environments table](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/app/assets/javascripts/environments) you can find a clear separation of concerns: ```plaintext @@ -47,7 +47,7 @@ of the new feature should be. The Store and the Service should be imported and initialized in this file and provided as a prop to the main component. -Be sure to read about [page-specific JavaScript][page_specific_javascript]. +Be sure to read about [page-specific JavaScript](./performance.md#page-specific-javascript). ### Bootstrapping Gotchas @@ -162,7 +162,7 @@ For example, tables are used in a quite amount of places across GitLab, a table would be a good fit for a component. On the other hand, a table cell used only in one table would not be a good use of this pattern. -You can read more about components in Vue.js site, [Component System][component-system] +You can read more about components in Vue.js site, [Component System](https://vuejs.org/v2/guide/#Composing-with-Components). ### A folder for the Store @@ -189,96 +189,135 @@ Each Vue component has a unique output. This output is always present in the ren Although we can test each method of a Vue component individually, our goal must be to test the output of the render/template function, which represents the state at all times. -Make use of the [axios mock adapter](axios.md#mock-axios-response-in-tests) to mock data returned. - -Here's how we would test the Todo App above: +Here's an example of a well structured unit test for [this Vue component](#appendix---vue-component-subject-under-test): ```javascript -import Vue from 'vue'; -import axios from '~/lib/utils/axios_utils'; +import { shallowMount } from '@vue/test-utils'; +import { GlLoadingIcon } from '@gitlab/ui'; import MockAdapter from 'axios-mock-adapter'; +import axios from '~/lib/utils/axios_utils'; +import App from '~/todos/app.vue'; + +const TEST_TODOS = [ + { text: 'Lorem ipsum test text' }, + { text: 'Lorem ipsum 2' }, +]; +const TEST_NEW_TODO = 'New todo title'; +const TEST_TODO_PATH = '/todos'; -describe('Todos App', () => { - let vm; +describe('~/todos/app.vue', () => { + let wrapper; let mock; beforeEach(() => { - // Create a mock adapter for stubbing axios API requests + // IMPORTANT: Use axios-mock-adapter for stubbing axios API requests mock = new MockAdapter(axios); - - const Component = Vue.extend(component); - - // Mount the Component - vm = new Component().$mount(); + mock.onGet(TEST_TODO_PATH).reply(200, TEST_TODOS); + mock.onPost(TEST_TODO_PATH).reply(200); }); afterEach(() => { - // Reset the mock adapter - mock.restore(); - // Destroy the mounted component - vm.$destroy(); - }); + // IMPORTANT: Clean up the component instance and axios mock adapter + wrapper.destroy(); + wrapper = null; - it('should render the loading state while the request is being made', () => { - expect(vm.$el.querySelector('i.fa-spin')).toBeDefined(); + mock.restore(); }); - it('should render todos returned by the endpoint', done => { - // Mock the get request on the API endpoint to return data - mock.onGet('/todos').replyOnce(200, [ - { - title: 'This is a todo', - text: 'This is the text', + // NOTE: It is very helpful to separate setting up the component from + // its collaborators (i.e. Vuex, axios, etc.) + const createWrapper = (props = {}) => { + wrapper = shallowMount(App, { + propsData: { + path: TEST_TODO_PATH, + ...props, }, - ]); + }); + }; + // NOTE: Helper methods greatly help test maintainability and readability. + const findLoader = () => wrapper.find(GlLoadingIcon); + const findAddButton = () => wrapper.find('[data-testid="add-button"]'); + const findTextInput = () => wrapper.find('[data-testid="text-input"]'); + const findTodoData = () => wrapper.findAll('[data-testid="todo-item"]').wrappers.map(wrapper => ({ text: wrapper.text() })); + + describe('when mounted and loading', () => { + beforeEach(() => { + // Create request which will never resolve + mock.onGet(TEST_TODO_PATH).reply(() => new Promise(() => {})); + createWrapper(); + }); - Vue.nextTick(() => { - const items = vm.$el.querySelectorAll('.js-todo-list div') - expect(items.length).toBe(1); - expect(items[0].textContent).toContain('This is the text'); - done(); + it('should render the loading state', () => { + expect(findLoader().exists()).toBe(true); }); }); - it('should add a todos on button click', (done) => { + describe('when todos are loaded', () => { + beforeEach(() => { + createWrapper(); + // IMPORTANT: This component fetches data asynchronously on mount, so let's wait for the Vue template to update + return wrapper.vm.$nextTick(); + }); - // Mock the put request and check that the sent data object is correct - mock.onPut('/todos').replyOnce((req) => { - expect(req.data).toContain('text'); - expect(req.data).toContain('title'); + it('should not show loading', () => { + expect(findLoader().exists()).toBe(false); + }); - return [201, {}]; + it('should render todos', () => { + expect(findTodoData()).toEqual(TEST_TODOS); }); - vm.$el.querySelector('.js-add-todo').click(); + it('when todo is added, should post new todo', () => { + findTextInput().vm.$emit('update', TEST_NEW_TODO) + findAddButton().vm.$emit('click'); - // Add a new interceptor to mock the add Todo request - Vue.nextTick(() => { - expect(vm.$el.querySelectorAll('.js-todo-list div').length).toBe(2); - done(); + return wrapper.vm.$nextTick() + .then(() => { + expect(mock.history.post.map(x => JSON.parse(x.data))).toEqual([{ text: TEST_NEW_TODO }]); + }); }); }); }); ``` -### `mountComponent` helper +### Test the component's output -There is a helper in `spec/javascripts/helpers/vue_mount_component_helper.js` that allows you to mount a component with the given props: +The main return value of a Vue component is the rendered output. In order to test the component we +need to test the rendered output. [Vue](https://vuejs.org/v2/guide/unit-testing.html) guide's to unit test show us exactly that: + +### Events + +We should test for events emitted in response to an action within our component, this is useful to verify the correct events are being fired with the correct arguments. + +For any DOM events we should use [`trigger`](https://vue-test-utils.vuejs.org/api/wrapper/#trigger) to fire out event. ```javascript -import Vue from 'vue'; -import mountComponent from 'spec/helpers/vue_mount_component_helper' -import component from 'component.vue' +// Assuming SomeButton renders: <button>Some button</button> +wrapper = mount(SomeButton); -const Component = Vue.extend(component); -const data = {prop: 'foo'}; -const vm = mountComponent(Component, data); +... +it('should fire the click event', () => { + const btn = wrapper.find('button') + + btn.trigger('click'); + ... +}) ``` -### Test the component's output +When we need to fire a Vue event, we should use [`emit`](https://vuejs.org/v2/guide/components-custom-events.html) to fire our event. -The main return value of a Vue component is the rendered output. In order to test the component we -need to test the rendered output. [Vue][vue-test] guide's to unit test show us exactly that: +```javascript +wrapper = shallowMount(DropdownItem); + +... + +it('should fire the itemClicked event', () => { + DropdownItem.vm.$emit('itemClicked'); + ... +}) +``` + +We should verify an event has been fired by asserting against the result of the [`emitted()`](https://vue-test-utils.vuejs.org/api/wrapper/#emitted) method ## Vue.js Expert Role @@ -287,15 +326,50 @@ One should apply to be a Vue.js expert by opening an MR when the Merge Request's - Deep understanding of Vue and Vuex reactivity - Vue and Vuex code are structured according to both official and our guidelines - Full understanding of testing a Vue and Vuex application -- Vuex code follows the [documented pattern](vuex.md#actions-pattern-request-and-receive-namespaces) +- Vuex code follows the [documented pattern](vuex.md#naming-pattern-request-and-receive-namespaces) - Knowledge about the existing Vue and Vuex applications and existing reusable components -[issue-boards]: https://gitlab.com/gitlab-org/gitlab-foss/tree/master/app/assets/javascripts/boards -[environments-table]: https://gitlab.com/gitlab-org/gitlab-foss/tree/master/app/assets/javascripts/environments -[page_specific_javascript]: ./performance.md#page-specific-javascript -[component-system]: https://vuejs.org/v2/guide/#Composing-with-Components -[state-management]: https://vuejs.org/v2/guide/state-management.html#Simple-State-Management-from-Scratch -[one-way-data-flow]: https://vuejs.org/v2/guide/components.html#One-Way-Data-Flow -[vue-test]: https://vuejs.org/v2/guide/unit-testing.html -[flux]: https://facebook.github.io/flux/ -[axios]: https://github.com/axios/axios +## Vue 2 -> Vue 3 Migration + +> This section is added temporarily to support the efforts to migrate the codebase from Vue 2.x to Vue 3.x + +Currently, we recommend to minimize adding certain features to the codebase to prevent increasing the tech debt for the eventual migration: + +- filters; +- event buses; +- functional templated +- `slot` attributes + +You can find more details on [Migration to Vue 3](vue3_migration.md) + +## Appendix - Vue component subject under test + +This is the template for the example component which is tested in the [Testing Vue components](#testing-vue-components) section: + +```html +<template> + <div class="content"> + <gl-loading-icon v-if="isLoading" /> + <template v-else> + <div + v-for="todo in todos" + :key="todo.id" + :class="{ 'gl-strike': todo.isDone }" + data-testid="todo-item" + >{{ toddo.text }}</div> + <footer class="gl-border-t-1 gl-mt-3 gl-pt-3"> + <gl-form-input + type="text" + v-model="todoText" + data-testid="text-input" + > + <gl-button + variant="success" + data-testid="add-button" + @click="addTodo" + >Add</gl-button> + </footer> + </template> + </div> +</template> +``` diff --git a/doc/development/fe_guide/vue3_migration.md b/doc/development/fe_guide/vue3_migration.md new file mode 100644 index 00000000000..7ab48db7f76 --- /dev/null +++ b/doc/development/fe_guide/vue3_migration.md @@ -0,0 +1,124 @@ +# Migration to Vue 3 + +In order to prepare for the eventual migration to Vue 3.x, we should be wary about adding the following features to the codebase: + +## Vue filters + +**Why?** + +Filters [are removed](https://github.com/vuejs/rfcs/blob/master/active-rfcs/0015-remove-filters.md) from the Vue 3 API completely. + +**What to use instead** + +Component's computed properties / methods or external helpers. + +## Event hub + +**Why?** + +`$on`, `$once`, and `$off` methods [are removed](https://github.com/vuejs/rfcs/blob/master/active-rfcs/0020-events-api-change.md) from the Vue instance, so in Vue 3 it can't be used to create an event hub. + +**What to use instead** + +Vue docs recommend using [mitt](https://github.com/developit/mitt) library. It's relatively small (200 bytes gzipped) and has a simple API: + +```javascript +import mitt from 'mitt' + +const emitter = mitt() + +// listen to an event +emitter.on('foo', e => console.log('foo', e) ) + +// listen to all events +emitter.on('*', (type, e) => console.log(type, e) ) + +// fire an event +emitter.emit('foo', { a: 'b' }) + +// working with handler references: +function onFoo() {} + +emitter.on('foo', onFoo) // listen +emitter.off('foo', onFoo) // unlisten +``` + +**Event hub factory** + +To make it easier for you to migrate existing event hubs to the new recommended approach, or simply +to create new ones, we have created a factory that you can use to instantiate a new mitt-based +event hub. + +```javascript +import createEventHub from '~/helpers/event_hub_factory'; + +export default createEventHub(); +``` + +Event hubs created with the factory expose the same methods as Vue 2 event hubs (`$on`, `$once`, `$off` and +`$emit`), making them backward compatible with our previous approach. + +## \<template functional> + +**Why?** + +In Vue 3, `{ functional: true }` option [is removed](https://github.com/vuejs/rfcs/blob/functional-async-api-change/active-rfcs/0007-functional-async-api-change.md) and `<template functional>` is no longer supported. + +**What to use instead** + +Functional components must be written as plain functions: + +```javascript +import { h } from 'vue' + +const FunctionalComp = (props, slots) => { + return h('div', `Hello! ${props.name}`) +} +``` + +## Old slots syntax with `slot` attribute + +**Why?** + +In Vue 2.6 `slot` attribute was already deprecated in favor of `v-slot` directive but its usage is still allowed and sometimes we prefer using them because it simplifies unit tests (with old syntax, slots are rendered on `shallowMount`). However, in Vue 3 we can't use old syntax anymore. + +**What to use instead** + +The syntax with `v-slot` directive. To fix rendering slots in `shallowMount`, we need to stub a child component with slots explicitly. + +```html +<!-- MyAwesomeComponent.vue --> +<script> +import SomeChildComponent from './some_child_component.vue' + +export default { + components: { + SomeChildComponent + } +} + +</script> + +<template> + <div> + <h1>Hello GitLab!</h1> + <some-child-component> + <template #header> + Header content + </template> + </some-child-component> + </div> +</template> +``` + +```javascript +// MyAwesomeComponent.spec.js + +import SomeChildComponent from '~/some_child_component.vue' + +shallowMount(MyAwesomeComponent, { + stubs: { + SomeChildComponent + } +}) +``` diff --git a/doc/development/fe_guide/vuex.md b/doc/development/fe_guide/vuex.md index 65661b0cc3b..e7be67b8da5 100644 --- a/doc/development/fe_guide/vuex.md +++ b/doc/development/fe_guide/vuex.md @@ -1,6 +1,6 @@ # Vuex -When there's a clear benefit to separating state management from components (e.g. due to state complexity) we recommend using [Vuex][vuex-docs] over any other Flux pattern. Otherwise, feel free to manage state within the components. +When there's a clear benefit to separating state management from components (e.g. due to state complexity) we recommend using [Vuex](https://vuex.vuejs.org) over any other Flux pattern. Otherwise, feel free to manage state within the components. Vuex should be strongly considered when: @@ -9,7 +9,7 @@ Vuex should be strongly considered when: - There are complex interactions with Backend, e.g. multiple API calls - The app involves interacting with backend via both traditional REST API and GraphQL (especially when moving the REST API over to GraphQL is a pending backend task) -_Note:_ All of the below is explained in more detail in the official [Vuex documentation][vuex-docs]. +_Note:_ All of the below is explained in more detail in the official [Vuex documentation](https://vuex.vuejs.org). ## Separation of concerns @@ -86,71 +86,35 @@ You can use `mapState` to access state properties in the components. An action is a payload of information to send data from our application to our store. -An action is usually composed by a `type` and a `payload` and they describe what happened. -Enforcing that every change is described as an action lets us have a clear understanding of what is going on in the app. +An action is usually composed by a `type` and a `payload` and they describe what happened. Unlike [mutations](#mutationsjs), actions can contain asynchronous operations - that's why we always need to handle asynchronous logic in actions. -In this file, we will write the actions that will call the respective mutations: +In this file, we will write the actions that will call mutations for handling a list of users: ```javascript import * as types from './mutation_types'; import axios from '~/lib/utils/axios_utils'; import createFlash from '~/flash'; - export const requestUsers = ({ commit }) => commit(types.REQUEST_USERS); - export const receiveUsersSuccess = ({ commit }, data) => commit(types.RECEIVE_USERS_SUCCESS, data); - export const receiveUsersError = ({ commit }, error) => commit(types.RECEIVE_USERS_ERROR, error); - export const fetchUsers = ({ state, dispatch }) => { - dispatch('requestUsers'); + commit(types.REQUEST_USERS); axios.get(state.endpoint) - .then(({ data }) => dispatch('receiveUsersSuccess', data)) + .then(({ data }) => commit(types.RECEIVE_USERS_SUCCESS, data)) .catch((error) => { - dispatch('receiveUsersError', error) + commit(types.RECEIVE_USERS_ERROR, error) createFlash('There was an error') }); } - export const requestAddUser = ({ commit }) => commit(types.REQUEST_ADD_USER); - export const receiveAddUserSuccess = ({ commit }, data) => commit(types.RECEIVE_ADD_USER_SUCCESS, data); - export const receiveAddUserError = ({ commit }, error) => commit(types.REQUEST_ADD_USER_ERROR, error); - export const addUser = ({ state, dispatch }, user) => { - dispatch('requestAddUser'); + commit(types.REQUEST_ADD_USER); axios.post(state.endpoint, user) - .then(({ data }) => dispatch('receiveAddUserSuccess', data)) - .catch((error) => dispatch('receiveAddUserError', error)); + .then(({ data }) => commit(types.RECEIVE_ADD_USER_SUCCESS, data)) + .catch((error) => commit(types.REQUEST_ADD_USER_ERROR, error)); } ``` -#### Actions Pattern: `request` and `receive` namespaces - -When a request is made we often want to show a loading state to the user. - -Instead of creating an action to toggle the loading state and dispatch it in the component, -create: - -1. An action `requestSomething`, to toggle the loading state -1. An action `receiveSomethingSuccess`, to handle the success callback -1. An action `receiveSomethingError`, to handle the error callback -1. An action `fetchSomething` to make the request. - 1. In case your application does more than a `GET` request you can use these as examples: - - `POST`: `createSomething` - - `PUT`: `updateSomething` - - `DELETE`: `deleteSomething` - -The component MUST only dispatch the `fetchNamespace` action. Actions namespaced with `request` or `receive` should not be called from the component -The `fetch` action will be responsible to dispatch `requestNamespace`, `receiveNamespaceSuccess` and `receiveNamespaceError` - -By following this pattern we guarantee: - -1. All applications follow the same pattern, making it easier for anyone to maintain the code -1. All data in the application follows the same lifecycle pattern -1. Actions are contained and human friendly -1. Unit tests are easier -1. Actions are simple and straightforward - #### Dispatching actions To dispatch an action from a component, use the `mapActions` helper: @@ -181,6 +145,8 @@ Remember that actions only describe that something happened, they don't describe **Never commit a mutation directly from a component** +Instead, you should create an action that will commit a mutation. + ```javascript import * as types from './mutation_types'; @@ -210,6 +176,31 @@ Remember that actions only describe that something happened, they don't describe }; ``` +#### Naming Pattern: `REQUEST` and `RECEIVE` namespaces + +When a request is made we often want to show a loading state to the user. + +Instead of creating an mutation to toggle the loading state, we should: + +1. A mutation with type `REQUEST_SOMETHING`, to toggle the loading state +1. A mutation with type `RECEIVE_SOMETHING_SUCCESS`, to handle the success callback +1. A mutation with type `RECEIVE_SOMETHING_ERROR`, to handle the error callback +1. An action `fetchSomething` to make the request and commit mutations on mentioned cases + 1. In case your application does more than a `GET` request you can use these as examples: + - `POST`: `createSomething` + - `PUT`: `updateSomething` + - `DELETE`: `deleteSomething` + +As a result, we can dispatch the `fetchNamespace` action from the component and it will be responsible to commit `REQUEST_NAMESPACE`, `RECEIVE_NAMESPACE_SUCCESS` and `RECEIVE_NAMESPACE_ERROR` mutations. + +> Previously, we were dispatching actions from the `fetchNamespace` action instead of committing mutation, so please don't be confused if you find a different pattern in the older parts of the codebase. However, we encourage leveraging a new pattern whenever you write new Vuex stores + +By following this pattern we guarantee: + +1. All applications follow the same pattern, making it easier for anyone to maintain the code +1. All data in the application follows the same lifecycle pattern +1. Unit tests are easier + ### `getters.js` Sometimes we may need to get derived state based on store state, like filtering for a specific prop. @@ -477,8 +468,6 @@ To prevent this error from happening, you need to export an empty function as `d export default () => {}; ``` -[vuex-docs]: https://vuex.vuejs.org - ### Two way data binding When storing form data in Vuex, it is sometimes necessary to update the value stored. The store should never be mutated directly, and an action should be used instead. diff --git a/doc/development/feature_flags/controls.md b/doc/development/feature_flags/controls.md index aa32285b27b..309aecd7978 100644 --- a/doc/development/feature_flags/controls.md +++ b/doc/development/feature_flags/controls.md @@ -16,32 +16,6 @@ run: /chatops run feature --help ``` -## Where to run commands - -To increase visibility, we recommend that GitLab team members run feature flag -related Chatops commands within certain Slack channels based on the environment -and related feature. For the [staging](https://staging.gitlab.com) -and [development](https://dev.gitlab.org) environments of GitLab.com, -the commands should run in a channel for the stage the feature is relevant too. - -For example, use the `#s_monitor` channel for features developed by the -Monitor stage, Health group. - -For all production environment Chatops commands, use the `#production` channel. - -As per the template, where a feature would have a (potentially) significant user -impact and the feature is being enabled instance wide prior to release, please copy -the Slack message and repost in the `#support_gitlab-com` channel for added visibility -and awareness, preferably with a link to the issue, MR, or docs. - -Regardless of the channel in which the Chatops command is ran, any feature flag change that affects GitLab.com will automatically be logged in an issue. - -The issue is created in the [gl-infra/feature-flag-log](https://gitlab.com/gitlab-com/gl-infra/feature-flag-log/issues?scope=all&utf8=%E2%9C%93&state=closed) project, and it will at minimum log the Slack handle of person enabling a feature flag, the time, and the name of the flag being changed. - -The issue is then also posted to GitLab Inc. internal [Grafana dashboard](https://dashboards.gitlab.net/) as an annotation marker to make the change even more visible. - -Changes to the issue format can be submitted in the [Chatops project](https://gitlab.com/gitlab-com/chatops). - ## Rolling out changes When the changes are deployed to the environments it is time to start @@ -67,19 +41,11 @@ If you get an error "Whoops! This action is not allowed. This incident will be reported." that means your Slack account is not allowed to change feature flags or you do not [have access](#access). -### Enabling feature for preproduction testing +### Enabling a feature for preproduction testing As a first step in a feature rollout, you should enable the feature on <https://staging.gitlab.com> and <https://dev.gitlab.org>. -For example, to enable a feature for 25% of all users, run the following in -Slack: - -```shell -/chatops run feature set new_navigation_bar 25 --dev -/chatops run feature set new_navigation_bar 25 --staging -``` - These two environments have different scopes. `dev.gitlab.org` is a production CE environment that has internal GitLab Inc. traffic and is used for some development and other related work. @@ -89,13 +55,65 @@ a (very) rough estimate of how your feature will look/behave on GitLab.com. Both of these instances are connected to Sentry so make sure you check the projects there for any exceptions while testing your feature after enabling the feature flag. -Once you are confident enough that these environments are in a good state with your -feature enabled, you can roll out the change to GitLab.com. +For these preproduction environments, the commands should be run in a +Slack channel for the stage the feature is relevant to. For example, use the +`#s_monitor` channel for features developed by the Monitor stage, Health +group. + +To enable a feature for 25% of all users, run the following in Slack: + +```shell +/chatops run feature set new_navigation_bar 25 --dev +/chatops run feature set new_navigation_bar 25 --staging +``` ### Enabling a feature for GitLab.com -Similar to above, to enable a feature for 25% of all users, run the following in -Slack: +When a feature has successfully been +[enabled on a preproduction](#enabling-a-feature-for-preproduction-testing) +environment and verified as safe and working, you can roll out the +change to GitLab.com (production). + +#### Communicate the change + +Some feature flag changes on GitLab.com should be communicated with +parts of the company. The developer responsible needs to determine +whether this is necessary and the appropriate level of communication. +This depends on the feature and what sort of impact it might have. + +As a guideline: + +- For simple features that are low-risk, and easily rolled back, then + just proceed to [enabling the feature in `#production`](#process). +- For features that will impact user experience consider notifying + `#support_gitlab-com` beforehand. +- For features with significant downstream effects (e.g.: turning on/off + Elasticsearch indexing) consider coordinating with `#production` + beforehand. + +#### Process + +Before toggling any feature flag, check that there are no ongoing +significant incidents on GitLab.com. You can do this by checking the +`#production` and `#incident-management` Slack channels, or looking for +[open incident issues](https://gitlab.com/gitlab-com/gl-infra/production/issues/?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=incident) +(although check the dates and times). + +We do not want to introduce changes during an incident, as it can make +diagnosis and resolution of the incident much harder to achieve, and +also will largely invalidate your rollout process as you will be unable +to assess whether the rollout was without problems or not. + +If there is any doubt, ask in `#production`. + +The following `/chatops` commands should be performed in the Slack +`#production` channel. + +When you begin to enable the feature, please link to the relevant +Feature Flag Rollout Issue within a Slack thread of the first `/chatops` +command you make so people can understand the change if they need to. + +To enable a feature for 25% of all users, run the following in Slack: ```shell /chatops run feature set new_navigation_bar 25 @@ -150,6 +168,23 @@ NOTE: **Note:** **Percentage of time** rollout is not a good idea if what you want is to make sure a feature is always on or off to the users. +### Feature flag change logging + +Any feature flag change that affects GitLab.com (production) will +automatically be logged in an issue. + +The issue is created in the +[gl-infra/feature-flag-log](https://gitlab.com/gitlab-com/gl-infra/feature-flag-log/issues?scope=all&utf8=%E2%9C%93&state=closed) +project, and it will at minimum log the Slack handle of person enabling +a feature flag, the time, and the name of the flag being changed. + +The issue is then also posted to GitLab's internal +[Grafana dashboard](https://dashboards.gitlab.net/) as an annotation +marker to make the change even more visible. + +Changes to the issue format can be submitted in the +[Chatops project](https://gitlab.com/gitlab-com/chatops). + ## Cleaning up Once the change is deemed stable, submit a new merge request to remove the @@ -157,7 +192,7 @@ feature flag. This ensures the change is available to all users and self-managed instances. Make sure to add the ~"feature flag" label to this merge request so release managers are aware the changes are hidden behind a feature flag. If the merge request has to be picked into a stable branch, make sure to also add the -appropriate "Pick into X" label (e.g. "Pick into XX.X"). +appropriate `~"Pick into X.Y"` label (e.g. `~"Pick into 13.0"`). See [the process document](process.md#including-a-feature-behind-feature-flag-in-the-final-release) for further details. When a feature gate has been removed from the code base, the feature diff --git a/doc/development/feature_flags/index.md b/doc/development/feature_flags/index.md index f5915f2c0a8..bd0bd8f2018 100644 --- a/doc/development/feature_flags/index.md +++ b/doc/development/feature_flags/index.md @@ -1,6 +1,7 @@ # Feature flags in development of GitLab -Feature flags can be used to gradually roll out changes, be +[Feature Flags](../../user/project/operations/feature_flags.md) +can be used to gradually roll out changes, be it a new feature, or a performance improvement. By using feature flags, we can comfortably measure the impact of our changes, while still being able to easily disable those changes, without having to revert an entire release. @@ -10,6 +11,5 @@ Before using feature flags for GitLab's development, read through the following: - [Process for using features flags](process.md). - [Developing with feature flags](development.md). - [Controlling feature flags](controls.md). - -When documenting feature flags, see [Feature flags](../documentation/styleguide.md#feature-flags) -in the Documentation Style Guide. +- [Documenting features deployed behind feature flags](../documentation/feature_flags.md). +- [How GitLab administrators can enable and disable features behind flags](../../administration/feature_flags.md). diff --git a/doc/development/feature_flags/process.md b/doc/development/feature_flags/process.md index 0cca4117f1f..57360f5b771 100644 --- a/doc/development/feature_flags/process.md +++ b/doc/development/feature_flags/process.md @@ -63,6 +63,9 @@ from when the merge request is first reviewed to when the change is deployed to GitLab.com. However, it is recommended to allow 10-14 days for this activity to account for unforeseen problems. +Feature flags must be [documented according to their state (enabled/disabled)](../documentation/feature_flags.md), +and when the state changes, docs **must** be updated accordingly. + NOTE: **Note:** Take into consideration that such action can make the feature available on GitLab.com shortly after the change to the feature flag is merged. diff --git a/doc/development/file_storage.md b/doc/development/file_storage.md index 230288844d7..e8ae5a11d48 100644 --- a/doc/development/file_storage.md +++ b/doc/development/file_storage.md @@ -21,7 +21,7 @@ There are many places where file uploading is used, according to contexts: - CI Artifacts (archive, metadata, trace) - LFS Objects - Merge request diffs - - Design Management design thumbnails (EE) + - Design Management design thumbnails ## Disk storage @@ -30,18 +30,18 @@ they are still not 100% standardized. You can see them below: | Description | In DB? | Relative path (from CarrierWave.root) | Uploader class | model_type | | ------------------------------------- | ------ | ----------------------------------------------------------- | ---------------------- | ---------- | -| Instance logo | yes | uploads/-/system/appearance/logo/:id/:filename | `AttachmentUploader` | Appearance | -| Header logo | yes | uploads/-/system/appearance/header_logo/:id/:filename | `AttachmentUploader` | Appearance | -| Group avatars | yes | uploads/-/system/group/avatar/:id/:filename | `AvatarUploader` | Group | -| User avatars | yes | uploads/-/system/user/avatar/:id/:filename | `AvatarUploader` | User | -| User snippet attachments | yes | uploads/-/system/personal_snippet/:id/:random_hex/:filename | `PersonalFileUploader` | Snippet | -| Project avatars | yes | uploads/-/system/project/avatar/:id/:filename | `AvatarUploader` | Project | -| Issues/MR/Notes Markdown attachments | yes | uploads/:project_path_with_namespace/:random_hex/:filename | `FileUploader` | Project | -| Issues/MR/Notes Legacy Markdown attachments | no | uploads/-/system/note/attachment/:id/:filename | `AttachmentUploader` | Note | -| Design Management design thumbnails (EE) | yes | uploads/-/system/design_management/action/image_v432x230/:id/:filename | `DesignManagement::DesignV432x230Uploader` | DesignManagement::Action | -| CI Artifacts (CE) | yes | `shared/artifacts/:disk_hash[0..1]/:disk_hash[2..3]/:disk_hash/:year_:month_:date/:job_id/:job_artifact_id` (:disk_hash is SHA256 digest of project_id) | `JobArtifactUploader` | Ci::JobArtifact | -| LFS Objects (CE) | yes | shared/lfs-objects/:hex/:hex/:object_hash | `LfsObjectUploader` | LfsObject | -| External merge request diffs | yes | shared/external-diffs/merge_request_diffs/mr-:parent_id/diff-:id | `ExternalDiffUploader` | MergeRequestDiff | +| Instance logo | yes | `uploads/-/system/appearance/logo/:id/:filename` | `AttachmentUploader` | Appearance | +| Header logo | yes | `uploads/-/system/appearance/header_logo/:id/:filename` | `AttachmentUploader` | Appearance | +| Group avatars | yes | `uploads/-/system/group/avatar/:id/:filename` | `AvatarUploader` | Group | +| User avatars | yes | `uploads/-/system/user/avatar/:id/:filename` | `AvatarUploader` | User | +| User snippet attachments | yes | `uploads/-/system/personal_snippet/:id/:random_hex/:filename` | `PersonalFileUploader` | Snippet | +| Project avatars | yes | `uploads/-/system/project/avatar/:id/:filename` | `AvatarUploader` | Project | +| Issues/MR/Notes Markdown attachments | yes | `uploads/:project_path_with_namespace/:random_hex/:filename` | `FileUploader` | Project | +| Issues/MR/Notes Legacy Markdown attachments | no | `uploads/-/system/note/attachment/:id/:filename` | `AttachmentUploader` | Note | +| Design Management design thumbnails | yes | `uploads/-/system/design_management/action/image_v432x230/:id/:filename` | `DesignManagement::DesignV432x230Uploader` | DesignManagement::Action | +| CI Artifacts (CE) | yes | `shared/artifacts/:disk_hash[0..1]/:disk_hash[2..3]/:disk_hash/:year_:month_:date/:job_id/:job_artifact_id` (`:disk_hash` is SHA256 digest of `project_id`) | `JobArtifactUploader` | Ci::JobArtifact | +| LFS Objects (CE) | yes | `shared/lfs-objects/:hex/:hex/:object_hash` | `LfsObjectUploader` | LfsObject | +| External merge request diffs | yes | `shared/external-diffs/merge_request_diffs/mr-:parent_id/diff-:id` | `ExternalDiffUploader` | MergeRequestDiff | CI Artifacts and LFS Objects behave differently in CE and EE. In CE they inherit the `GitlabUploader` while in EE they inherit the `ObjectStorage` and store files in and S3 API compatible object store. diff --git a/doc/development/filtering_by_label.md b/doc/development/filtering_by_label.md index 6f6d7afc040..19dece0d5c9 100644 --- a/doc/development/filtering_by_label.md +++ b/doc/development/filtering_by_label.md @@ -80,7 +80,7 @@ it did not improve query performance. ## Attempt B: Denormalize using an array column Having [removed MySQL support in GitLab 12.1](https://about.gitlab.com/blog/2019/06/27/removing-mysql-support/), -using [PostgreSQL's arrays](https://www.postgresql.org/docs/9.6/arrays.html) became more +using [PostgreSQL's arrays](https://www.postgresql.org/docs/11/arrays.html) became more tractable as we didn't have to support two databases. We discussed denormalizing the `label_links` table for querying in [issue #49651](https://gitlab.com/gitlab-org/gitlab-foss/issues/49651), @@ -91,7 +91,7 @@ and `epics`: `issues.label_ids` would be an array column of label IDs, and `issues.label_titles` would be an array of label titles. These array columns can be complemented with [GIN -indexes](https://www.postgresql.org/docs/9.6/gin-intro.html) to improve +indexes](https://www.postgresql.org/docs/11/gin-intro.html) to improve matching. ### Attempt B1: store label IDs for each object diff --git a/doc/development/geo.md b/doc/development/geo.md index b922fdfa119..bf56340f8ec 100644 --- a/doc/development/geo.md +++ b/doc/development/geo.md @@ -216,15 +216,11 @@ bundle exec rake geo:db:migrate Foreign Data Wrapper ([FDW](#fdw)) is used by the [Geo Log Cursor](#geo-log-cursor) and improves the performance of many synchronization operations. -FDW is a PostgreSQL extension ([`postgres_fdw`](https://www.postgresql.org/docs/current/postgres-fdw.html)) that is enabled within +FDW is a PostgreSQL extension ([`postgres_fdw`](https://www.postgresql.org/docs/11/postgres-fdw.html)) that is enabled within the Geo Tracking Database (on a **secondary** node), which allows it to connect to the readonly database replica and perform queries and filter data from both instances. -While FDW is available in older versions of PostgreSQL, we needed to -raise the minimum required version to 9.6 as this includes many -performance improvements to the FDW implementation. - This persistent connection is configured as an FDW server named `gitlab_secondary`. This configuration exists within the database's user context only. To access the `gitlab_secondary`, GitLab needs to use the diff --git a/doc/development/geo/framework.md b/doc/development/geo/framework.md index 026d3543955..d72f7cc4cc1 100644 --- a/doc/development/geo/framework.md +++ b/doc/development/geo/framework.md @@ -84,10 +84,8 @@ module Geo model_record.file end - private - # Specify the model this replicator belongs to - def model + def self.model ::Packages::PackageFile end end @@ -163,47 +161,7 @@ state. For example, to add support for files referenced by a `Widget` model with a `widgets` table, you would perform the following steps: -1. Add verification state fields to the `widgets` table so the Geo primary can - track verification state: - - ```ruby - # frozen_string_literal: true - - class AddVerificationStateToWidgets < ActiveRecord::Migration[6.0] - DOWNTIME = false - - def change - add_column :widgets, :verification_retry_at, :datetime_with_timezone - add_column :widgets, :verified_at, :datetime_with_timezone - add_column :widgets, :verification_checksum, :string - add_column :widgets, :verification_failure, :string - add_column :widgets, :verification_retry_count, :integer - end - end - ``` - -1. Add a partial index on `verification_failure` to ensure re-verification can - be performed efficiently: - - ```ruby - # frozen_string_literal: true - - class AddVerificationFailureIndexToWidgets < ActiveRecord::Migration[6.0] - include Gitlab::Database::MigrationHelpers - - DOWNTIME = false - - disable_ddl_transaction! - - def up - add_concurrent_index :widgets, :verification_failure, where: "(verification_failure IS NOT NULL)", name: "widgets_verification_failure_partial" - end - - def down - remove_concurrent_index :widgets, :verification_failure - end - end - ``` +#### Replication 1. Include `Gitlab::Geo::ReplicableModel` in the `Widget` class, and specify the Replicator class `with_replicator Geo::WidgetReplicator`. @@ -270,7 +228,7 @@ For example, to add support for files referenced by a `Widget` model with a ```ruby # frozen_string_literal: true - class CreateWidgetRegistry < ActiveRecord::Migration[5.2] + class CreateWidgetRegistry < ActiveRecord::Migration[6.0] DOWNTIME = false def change @@ -334,7 +292,7 @@ For example, to add support for files referenced by a `Widget` model with a end ``` -1. Create `ee/spec/models/geo/widget_registry.rb`: +1. Create `ee/spec/models/geo/widget_registry_spec.rb`: ```ruby # frozen_string_literal: true @@ -350,4 +308,206 @@ For example, to add support for files referenced by a `Widget` model with a end ``` -Widget files should now be replicated and verified by Geo! +Widgets should now be replicated by Geo! + +#### Verification + +1. Add verification state fields to the `widgets` table so the Geo primary can + track verification state: + + ```ruby + # frozen_string_literal: true + + class AddVerificationStateToWidgets < ActiveRecord::Migration[6.0] + DOWNTIME = false + + def change + add_column :widgets, :verification_retry_at, :datetime_with_timezone + add_column :widgets, :verified_at, :datetime_with_timezone + add_column :widgets, :verification_checksum, :binary, using: 'verification_checksum::bytea' + add_column :widgets, :verification_failure, :string + add_column :widgets, :verification_retry_count, :integer + end + end + ``` + +1. Add a partial index on `verification_failure` and `verification_checksum` to ensure + re-verification can be performed efficiently: + + ```ruby + # frozen_string_literal: true + + class AddVerificationFailureIndexToWidgets < ActiveRecord::Migration[6.0] + include Gitlab::Database::MigrationHelpers + + DOWNTIME = false + + disable_ddl_transaction! + + def up + add_concurrent_index :widgets, :verification_failure, where: "(verification_failure IS NOT NULL)", name: "widgets_verification_failure_partial" + add_concurrent_index :widgets, :verification_checksum, where: "(verification_checksum IS NOT NULL)", name: "widgets_verification_checksum_partial" + end + + def down + remove_concurrent_index :widgets, :verification_failure + remove_concurrent_index :widgets, :verification_checksum + end + end + ``` + +1. Add fields `widget_count`, `widget_checksummed_count`, and `widget_checksum_failed_count` + to `GeoNodeStatus#RESOURCE_STATUS_FIELDS` array in `ee/app/models/geo_node_status.rb`. +1. Add the same fields to `GeoNodeStatus#PROMETHEUS_METRICS` hash in + `ee/app/models/geo_node_status.rb`. +1. Add the same fields to `Sidekiq metrics` table in + `doc/administration/monitoring/prometheus/gitlab_metrics.md`. +1. Add the same fields to `GET /geo_nodes/status` example response in `doc/api/geo_nodes.md`. +1. Modify `GeoNodeStatus#load_verification_data` to make sure the fields mantioned above + are set: + + ```ruby + self.widget_count = Geo::WidgetReplicator.model.count + self.widget_checksummed_count = Geo::WidgetReplicator.checksummed.count + self.widget_checksum_failed_count = Geo::WidgetReplicator.checksum_failed.count + ``` + +1. Make sure `Widget` model has `checksummed` and `checksum_failed` scopes. +1. Update `ee/spec/fixtures/api/schemas/public_api/v4/geo_node_status.json` with new fields. +1. Update `GeoNodeStatus#PROMETHEUS_METRICS` hash in `ee/app/models/geo_node_status.rb` with new fields. +1. Update `Sidekiq metrics` table in `doc/administration/monitoring/prometheus/gitlab_metrics.md` with new fields. +1. Update `GET /geo_nodes/status` example response in `doc/api/geo_nodes.md` with new fields. +1. Update `ee/spec/models/geo_node_status_spec.rb` and `ee/spec/factories/geo_node_statuses.rb` with new fields. + +To do: Add verification on secondaries. This should be done as part of +[Geo: Self Service Framework - First Implementation for Package File verification](https://gitlab.com/groups/gitlab-org/-/epics/1817) + +Widgets should now be verified by Geo! + +#### GraphQL API + +1. Add a new field to `GeoNodeType` in + `ee/app/graphql/types/geo/geo_node_type.rb`: + + ```ruby + field :widget_registries, ::Types::Geo::WidgetRegistryType.connection_type, + null: true, + resolver: ::Resolvers::Geo::WidgetRegistriesResolver, + description: 'Find widget registries on this Geo node', + feature_flag: :geo_self_service_framework + ``` + +1. Add the new `widget_registries` field name to the `expected_fields` array in + `ee/spec/graphql/types/geo/geo_node_type_spec.rb`. + +1. Create `ee/app/graphql/resolvers/geo/widget_registries_resolver.rb`: + + ```ruby + # frozen_string_literal: true + + module Resolvers + module Geo + class WidgetRegistriesResolver < BaseResolver + include RegistriesResolver + end + end + end + ``` + +1. Create `ee/spec/graphql/resolvers/geo/widget_registries_resolver_spec.rb`: + + ```ruby + # frozen_string_literal: true + + require 'spec_helper' + + describe Resolvers::Geo::WidgetRegistriesResolver do + it_behaves_like 'a Geo registries resolver', :widget_registry + end + ``` + +1. Create `ee/app/finders/geo/widget_registry_finder.rb`: + + ```ruby + # frozen_string_literal: true + + module Geo + class WidgetRegistryFinder + include FrameworkRegistryFinder + end + end + ``` + +1. Create `ee/spec/finders/geo/widget_registry_finder_spec.rb`: + + ```ruby + # frozen_string_literal: true + + require 'spec_helper' + + describe Geo::WidgetRegistryFinder do + it_behaves_like 'a framework registry finder', :widget_registry + end + ``` + +1. Create `ee/app/graphql/types/geo/package_file_registry_type.rb`: + + ```ruby + # frozen_string_literal: true + + module Types + module Geo + # rubocop:disable Graphql/AuthorizeTypes because it is included + class WidgetRegistryType < BaseObject + include ::Types::Geo::RegistryType + + graphql_name 'WidgetRegistry' + description 'Represents the sync and verification state of a widget' + + field :widget_id, GraphQL::ID_TYPE, null: false, description: 'ID of the Widget' + end + end + end + ``` + +1. Create `ee/spec/graphql/types/geo/widget_registry_type_spec.rb`: + + ```ruby + # frozen_string_literal: true + + require 'spec_helper' + + describe GitlabSchema.types['WidgetRegistry'] do + it_behaves_like 'a Geo registry type' + + it 'has the expected fields (other than those included in RegistryType)' do + expected_fields = %i[widget_id] + + expect(described_class).to have_graphql_fields(*expected_fields).at_least + end + end + ``` + +1. Add integration tests for providing Widget registry data to the frontend via + the GraphQL API, by duplicating and modifying the following shared examples + in `ee/spec/requests/api/graphql/geo/registries_spec.rb`: + + ```ruby + it_behaves_like 'gets registries for', { + field_name: 'widgetRegistries', + registry_class_name: 'WidgetRegistry', + registry_factory: :widget_registry, + registry_foreign_key_field_name: 'widgetId' + } + ``` + +Individual widget synchronization and verification data should now be available +via the GraphQL API! + +#### Admin UI + +To do: This should be done as part of +[Geo: Implement frontend for Self-Service Framework replicables](https://gitlab.com/groups/gitlab-org/-/epics/2525) + +Widget sync and verification data (aggregate and individual) should now be +available in the Admin UI! diff --git a/doc/development/gitaly.md b/doc/development/gitaly.md index 43e4d4d0e72..5e5cae7228b 100644 --- a/doc/development/gitaly.md +++ b/doc/development/gitaly.md @@ -5,7 +5,7 @@ Workhorse and GitLab-Shell. ## Deep Dive -In May 2019, Bob Van Landuyt hosted a [Deep Dive](https://gitlab.com/gitlab-org/create-stage/issues/1) +In May 2019, Bob Van Landuyt hosted a Deep Dive (GitLab team members only: `https://gitlab.com/gitlab-org/create-stage/issues/1`) on GitLab's [Gitaly project](https://gitlab.com/gitlab-org/gitaly) and how to contribute to it as a Ruby developer, to share his domain specific knowledge with anyone who may work in this part of the code base in the future. @@ -77,7 +77,7 @@ If your test-suite is failing with Gitaly issues, as a first step, try running: rm -rf tmp/tests/gitaly ``` -During rspec tests, the Gitaly instance will write logs to `gitlab/log/gitaly-test.log`. +During RSpec tests, the Gitaly instance will write logs to `gitlab/log/gitaly-test.log`. ## Legacy Rugged code diff --git a/doc/development/go_guide/index.md b/doc/development/go_guide/index.md index c1dfb220df8..fe69a4205f8 100644 --- a/doc/development/go_guide/index.md +++ b/doc/development/go_guide/index.md @@ -63,15 +63,20 @@ file and ask your manager to review and merge. ```yaml projects: gitlab: reviewer go - gitlab-foss: reviewer go ``` ## Code style and format - Avoid global variables, even in packages. By doing so you will introduce side effects if the package is included multiple times. -- Use `go fmt` before committing ([Gofmt](https://golang.org/cmd/gofmt/) is a - tool that automatically formats Go source code). +- Use `goimports` before committing. + [goimports](https://godoc.org/golang.org/x/tools/cmd/goimports) + is a tool that automatically formats Go source code using + [Gofmt](https://golang.org/cmd/gofmt/), in addition to formatting import lines, + adding missing ones and removing unreferenced ones. + + Most editors/IDEs will allow you to run commands before/after saving a file, you can set it + up to run `goimports` so that it's applied to every file when saving. - Place private methods below the first caller method in the source file. ### Automatic linting @@ -244,6 +249,59 @@ Programs handling a lot of IO or complex operations should always include [benchmarks](https://golang.org/pkg/testing/#hdr-Benchmarks), to ensure performance consistency over time. +## Error handling + +### Adding context + +Adding context before you return the error can be helpful, instead of +just returning the error. This allows developers to understand what the +program was trying to do when it entered the error state making it much +easier to debug. + +For example: + +```go +// Wrap the error +return nil, fmt.Errorf("get cache %s: %w", f.Name, err) + +// Just add context +return nil, fmt.Errorf("saving cache %s: %v", f.Name, err) +``` + +A few things to keep in mind when adding context: + +- Decide if you want to expose the underlying error + to the caller. If so, use `%w`, if not, you can use `%v`. +- Don't use words like `failed`, `error`, `didn't`. As it's an error, + the user already knows that something failed and this might lead to + having strings like `failed xx failed xx failed xx`. Explain _what_ + failed instead. +- Error strings should not be capitalized or end with punctuation or a + newline. You can use `golint` to check for this. + +### Naming + +- When using sentinel errors they should always be named like `ErrXxx`. +- When creating a new error type they should always be named like + `XxxError`. + +### Checking Error types + +- To check error equality don't use `==`. Use + [`errors.Is`](https://pkg.go.dev/errors?tab=doc#Is) instead (for Go + versions >= 1.13). +- To check if the error is of a certain type don't use type assertion, + use [`errors.As`](https://pkg.go.dev/errors?tab=doc#As) instead (for + Go versions >= 1.13). + +### References for working with errors + +- [Go 1.13 errors](https://blog.golang.org/go1.13-errors). +- [Programing with + errors](https://peter.bourgon.org/blog/2019/09/11/programming-with-errors.html). +- [Don’t just check errors, handle them + gracefully](https://dave.cheney.net/2016/04/27/dont-just-check-errors-handle-them-gracefully). + ## CLIs Every Go program is launched from the command line. @@ -368,13 +426,13 @@ Once you've picked a new Go version to use, the steps to update Omnibus and CNG are: - [Create a merge request in the CNG project](https://gitlab.com/gitlab-org/build/CNG/edit/master/ci_files/variables.yml?branch_name=update-go-version), - updating the `GO_VERSION` in `ci_files/variables.yml`. + updating the `GO_VERSION` in `ci_files/variables.yml`. - Create a merge request in the [`gitlab-omnibus-builder` project](https://gitlab.com/gitlab-org/gitlab-omnibus-builder), - updating every file in the `docker/` directory so the `GO_VERSION` is set - appropriately. [Here's an example](https://gitlab.com/gitlab-org/gitlab-omnibus-builder/-/merge_requests/125/diffs). + updating every file in the `docker/` directory so the `GO_VERSION` is set + appropriately. [Here's an example](https://gitlab.com/gitlab-org/gitlab-omnibus-builder/-/merge_requests/125/diffs). - Tag a new release of `gitlab-omnibus-builder` containing the change. -- [Create a merge request in the `gitlab-omnibus` project](https://gitlab.com/gitlab-org/omnibus-gitlab/edit/master/.gitlab-ci.yml?branch_name=update-gitlab-omnibus-builder-version), - updating the `BUILDER_IMAGE_REVISION` to match the newly-created tag. +- [Create a merge request in the `omnibus-gitlab` project](https://gitlab.com/gitlab-org/omnibus-gitlab/edit/master/.gitlab-ci.yml?branch_name=update-gitlab-omnibus-builder-version), + updating the `BUILDER_IMAGE_REVISION` to match the newly-created tag. To reduce unnecessary differences between two distribution methods, Omnibus and CNG **should always use the same Go version**. diff --git a/doc/development/hash_indexes.md b/doc/development/hash_indexes.md index 417ea18e22f..bc962ac0cd6 100644 --- a/doc/development/hash_indexes.md +++ b/doc/development/hash_indexes.md @@ -14,7 +14,7 @@ documentation: > answers to queries that subsequently use them. For these reasons, hash index > use is presently discouraged. -RuboCop is configured to register an offence when it detects the use of a hash +RuboCop is configured to register an offense when it detects the use of a hash index. Instead of using hash indexes you should use regular btree indexes. diff --git a/doc/development/i18n/externalization.md b/doc/development/i18n/externalization.md index 7ddcd426fd7..a81e656fc27 100644 --- a/doc/development/i18n/externalization.md +++ b/doc/development/i18n/externalization.md @@ -20,7 +20,7 @@ The following tools are used: 1. [`gettext_i18n_rails`](https://github.com/grosser/gettext_i18n_rails): this gem allow us to translate content from models, views and controllers. Also - it gives us access to the following raketasks: + it gives us access to the following Rake tasks: - `rake gettext:find`: Parses almost all the files from the Rails application looking for content that has been marked for translation. Finally, it updates the PO files with the new content that @@ -30,7 +30,7 @@ The following tools are used: 1. [`gettext_i18n_rails_js`](https://github.com/webhippie/gettext_i18n_rails_js): this gem is useful to make the translations available in JavaScript. It - provides the following raketask: + provides the following Rake task: - `rake gettext:po_to_json`: Reads the contents from the PO files and generates JSON files containing all the available translations. @@ -131,7 +131,7 @@ You can mark that content for translation with: In JavaScript we added the `__()` (double underscore parenthesis) function that you can import from the `~/locale` file. For instance: -```js +```javascript import { __ } from '~/locale'; const label = __('Subscribe'); ``` @@ -167,7 +167,7 @@ For example use `%{created_at}` in Ruby but `%{createdAt}` in JavaScript. Make s - In JavaScript (when Vue cannot be used): - ```js + ```javascript import { __, sprintf } from '~/locale'; sprintf(__('Hello %{username}'), { username: 'Joe' }); // => 'Hello Joe' @@ -180,7 +180,7 @@ For example use `%{created_at}` in Ruby but `%{createdAt}` in JavaScript. Make s escape any interpolated dynamic values yourself, for instance using `escape` from `lodash`. - ```js + ```javascript import { escape } from 'lodash'; import { __, sprintf } from '~/locale'; @@ -220,14 +220,14 @@ For example use `%{created_at}` in Ruby but `%{createdAt}` in JavaScript. Make s - In JavaScript: - ```js + ```javascript n__('Apple', 'Apples', 3) // => 'Apples' ``` Using interpolation: - ```js + ```javascript n__('Last day', 'Last %d days', x) // => When x == 1: 'Last day' // => When x == 2: 'Last 2 days' @@ -274,7 +274,7 @@ Namespaces should be PascalCase. - In JavaScript: - ```js + ```javascript s__('OpenedNDaysAgo|Opened') ``` @@ -285,7 +285,7 @@ guidelines for more details](translation.md#namespaced-strings). - In JavaScript: -```js +```javascript import { createDateTimeFormat } from '~/locale'; const dateFormat = createDateTimeFormat({ year: 'numeric', month: 'long', day: 'numeric' }); @@ -372,7 +372,7 @@ structure is the same in all languages. For instance, the following: -```js +```javascript {{ s__("mrWidget|Set by") }} {{ author.name }} {{ s__("mrWidget|to be merged automatically when the pipeline succeeds") }} @@ -380,7 +380,7 @@ For instance, the following: should be externalized as follows: -```js +```javascript {{ sprintf(s__("mrWidget|Set by %{author} to be merged automatically when the pipeline succeeds"), { author: author.name }) }} ``` @@ -439,7 +439,7 @@ This also applies when using links in between translated sentences, otherwise th - In JavaScript (when Vue cannot be used), instead of: - ```js + ```javascript {{ sprintf(s__("ClusterIntegration|Learn more about %{link}"), { link: '<a href="https://cloud.google.com/compute/docs/regions-zones/regions-zones" target="_blank" rel="noopener noreferrer">zones</a>' @@ -449,7 +449,7 @@ This also applies when using links in between translated sentences, otherwise th Set the link starting and ending HTML fragments as placeholders like so: - ```js + ```javascript {{ sprintf(s__("ClusterIntegration|Learn more about %{linkStart}zones%{linkEnd}"), { linkStart: '<a href="https://cloud.google.com/compute/docs/regions-zones/regions-zones" target="_blank" rel="noopener noreferrer">', @@ -536,9 +536,9 @@ The linter will take the following into account: - Variable usage - Only one unnamed (`%d`) variable, since the order of variables might change in different languages - - All variables used in the message-id are used in the translation + - All variables used in the message ID are used in the translation - There should be no variables used in a translation that aren't in the - message-id + message ID - Errors during translation. The errors are grouped per file, and per message ID: @@ -562,7 +562,7 @@ Errors in `locale/zh_TW/gitlab.po`: In this output the `locale/zh_HK/gitlab.po` has syntax errors. The `locale/zh_TW/gitlab.po` has variables that are used in the translation that -aren't in the message with id `1 pipeline`. +aren't in the message with ID `1 pipeline`. ## Adding a new language diff --git a/doc/development/i18n/proofreader.md b/doc/development/i18n/proofreader.md index 1170103490b..837da349f7e 100644 --- a/doc/development/i18n/proofreader.md +++ b/doc/development/i18n/proofreader.md @@ -87,7 +87,7 @@ are very appreciative of the work done by translators and proofreaders! - Mark Minakou - [GitLab](https://gitlab.com/sandzhaj), [Crowdin](https://crowdin.com/profile/sandzhaj) - NickVolynkin - [Crowdin](https://crowdin.com/profile/NickVolynkin) - Andrey Komarov - [GitLab](https://gitlab.com/elkamarado), [Crowdin](https://crowdin.com/profile/kamarado) - - Iaroslav Postovalov - [GitLab](https://gitlab/CMDR_Tvis), [Crowdin](https://crowdin.com/profile/CMDR_Tvis) + - Iaroslav Postovalov - [GitLab](https://gitlab.com/CMDR_Tvis), [Crowdin](https://crowdin.com/profile/CMDR_Tvis) - Serbian (Cyrillic) - Proofreaders needed. - Serbian (Latin) diff --git a/doc/development/img/snowplow_flow.png b/doc/development/img/snowplow_flow.png Binary files differnew file mode 100644 index 00000000000..5996cf01537 --- /dev/null +++ b/doc/development/img/snowplow_flow.png diff --git a/doc/development/img/telemetry_system_overview.png b/doc/development/img/telemetry_system_overview.png Binary files differnew file mode 100644 index 00000000000..1667039e8cd --- /dev/null +++ b/doc/development/img/telemetry_system_overview.png diff --git a/doc/development/import_export.md b/doc/development/import_export.md index 5a7d176d1d3..6d1b6929667 100644 --- a/doc/development/import_export.md +++ b/doc/development/import_export.md @@ -331,3 +331,78 @@ module Projects wiki_repo_saver, lfs_saver].all?(&:save) end ``` + +## Test fixtures + +Fixtures used in Import/Export specs live in `spec/fixtures/lib/gitlab/import_export`. There are both Project and Group fixtures. + +There are two versions of each of these fixtures: + +- A human readable single JSON file with all objects, called either `project.json` or `group.json`. +- A folder named `tree`, containing a tree of files in `ndjson` format. **Please do not edit files under this folder manually unless strictly necessary.** + +The tools to generate the NDJSON tree from the human-readable JSON files live in the [`gitlab-org/memory-team/team-tools`](https://gitlab.com/gitlab-org/memory-team/team-tools/-/blob/master/import-export/) project. + +### Project + +**Please use `legacy-project-json-to-ndjson.sh` to generate the NDJSON tree.** + +The NDJSON tree will look like this: + +```shell +tree +├── project +│ ├── auto_devops.ndjson +│ ├── boards.ndjson +│ ├── ci_cd_settings.ndjson +│ ├── ci_pipelines.ndjson +│ ├── container_expiration_policy.ndjson +│ ├── custom_attributes.ndjson +│ ├── error_tracking_setting.ndjson +│ ├── external_pull_requests.ndjson +│ ├── issues.ndjson +│ ├── labels.ndjson +│ ├── merge_requests.ndjson +│ ├── milestones.ndjson +│ ├── pipeline_schedules.ndjson +│ ├── project_badges.ndjson +│ ├── project_feature.ndjson +│ ├── project_members.ndjson +│ ├── protected_branches.ndjson +│ ├── protected_tags.ndjson +│ ├── releases.ndjson +│ ├── services.ndjson +│ ├── snippets.ndjson +│ └── triggers.ndjson +└── project.json +``` + +### Group + +**Please use `legacy-group-json-to-ndjson.rb` to generate the NDJSON tree.** + +The NDJSON tree will look like this: + +```shell +tree +└── groups + ├── 4351 + │ ├── badges.ndjson + │ ├── boards.ndjson + │ ├── epics.ndjson + │ ├── labels.ndjson + │ ├── members.ndjson + │ └── milestones.ndjson + ├── 4352 + │ ├── badges.ndjson + │ ├── boards.ndjson + │ ├── epics.ndjson + │ ├── labels.ndjson + │ ├── members.ndjson + │ └── milestones.ndjson + ├── _all.ndjson + ├── 4351.json + └── 4352.json +``` + +CAUTION: **Caution:** When updating these fixtures, please ensure you update both `json` files and `tree` folder, as the tests apply to both. diff --git a/doc/development/import_project.md b/doc/development/import_project.md index 78efc6ce2ab..f222a6533e8 100644 --- a/doc/development/import_project.md +++ b/doc/development/import_project.md @@ -61,10 +61,9 @@ Parameters: | `namespace_path` | string | yes | Namespace path | | `project_path` | string | yes | Project name | | `archive_path` | string | yes | Path to the exported project tarball you want to import | -| `measurement_enabled` | boolean | no | Measure execution time, number of SQL calls and GC count | ```shell -bundle exec rake "gitlab:import_export:import[root, root, testingprojectimport, /path/to/file.tar.gz, true]" +bundle exec rake "gitlab:import_export:import[root, root, testingprojectimport, /path/to/file.tar.gz]" ``` ### Importing via the Rails console diff --git a/doc/development/instrumentation.md b/doc/development/instrumentation.md index fd01d0ea405..d72e1c6635e 100644 --- a/doc/development/instrumentation.md +++ b/doc/development/instrumentation.md @@ -5,7 +5,7 @@ blocks of Ruby code. Method instrumentation is the primary form of instrumentation with block-based instrumentation only being used when we want to drill down to specific regions of code within a method. -Please refer to [Telemetry](../telemetry/index.md) if you are tracking product usage patterns. +Please refer to [Telemetry](telemetry/index.md) if you are tracking product usage patterns. ## Instrumenting Methods diff --git a/doc/development/integrations/example_vuln.png b/doc/development/integrations/example_vuln.png Binary files differnew file mode 100644 index 00000000000..f7a3c8b38f2 --- /dev/null +++ b/doc/development/integrations/example_vuln.png diff --git a/doc/development/integrations/secure.md b/doc/development/integrations/secure.md index b38e45778fb..b0e1e28ba8b 100644 --- a/doc/development/integrations/secure.md +++ b/doc/development/integrations/secure.md @@ -2,14 +2,23 @@ Integrating a security scanner into GitLab consists of providing end users with a [CI job definition](../../ci/yaml/README.md#introduction) -they can add to their CI configuration files, to scan their GitLab projects. +they can add to their CI configuration files to scan their GitLab projects. +This CI job should then output its results in a GitLab-specified format. These results are then +automatically presented in various places in GitLab, such as the Pipeline view, Merge Request +widget, and Security Dashboard. + The scanning job is usually based on a [Docker image](https://docs.docker.com/) that contains the scanner and all its dependencies in a self-contained environment. -This page documents requirements and guidelines for writing CI jobs implementing a security scanner, -as well as requirements and guidelines for the Docker image itself. + +This page documents requirements and guidelines for writing CI jobs that implement a security +scanner, as well as requirements and guidelines for the Docker image. ## Job definition +This section desribes several important fields to add to the security scanner's job +definition file. Full documentation on these and other available fields can be viewed +in the [CI documentation](../../ci/yaml/README.md#image). + ### Name For consistency, scanning jobs should be named after the scanner, in lower case. @@ -26,8 +35,8 @@ containing the security scanner. ### Script The [`script`](../../ci/yaml/README.md#script) keyword -is used to specify the command that the job runs. -Because the `script` cannot be left empty, it must be set to the command that performs the scan. +is used to specify the commands to run the scanner. +Because the `script` entry can't be left empty, it must be set to the command that performs the scan. It is not possible to rely on the predefined `ENTRYPOINT` and `CMD` of the Docker image to perform the scan automatically, without passing any command. @@ -53,44 +62,41 @@ so the [`allow_failure`](../../ci/yaml/README.md#allow_failure) parameter should ### Artifacts Scanning jobs must declare a report that corresponds to the type of scanning they perform, -using the [`artifacts:reports`](../../ci/yaml/README.md#artifactsreports) keyword. +using the [`artifacts:reports`](../../ci/pipelines/job_artifacts.md#artifactsreports) keyword. Valid reports are: `dependency_scanning`, `container_scanning`, `dast`, and `sast`. For example, here is the definition of a SAST job that generates a file named `gl-sast-report.json`, and uploads it as a SAST report: ```yaml -mysec_dependency_scanning: +mysec_sast_scanning: image: registry.gitlab.com/secure/mysec artifacts: reports: sast: gl-sast-report.json ``` -`gl-sast-report.json` is an example file path. See [the Output file section](#output-file) for more details. -It is processed as a SAST report because it is declared as such in the job definition. +Note that `gl-sast-report.json` is an example file path but any other file name can be used. See +[the Output file section](#output-file) for more details. It's processed as a SAST report because +it's declared under the `reports:sast` key in the job definition, not because of the file name. ### Policies -Scanning jobs should be skipped unless the corresponding feature is listed -in the `GITLAB_FEATURES` variable (comma-separated list of values). -So Dependency Scanning, Container Scanning, SAST, and DAST should be skipped -unless `GITLAB_FEATURES` contains `dependency_scanning`, `container_scanning`, `sast`, and `dast`, respectively. -See [GitLab CI/CD predefined variables](../../ci/variables/predefined_variables.md). - -Also, scanning jobs should be skipped when the corresponding variable prefixed with `_DISABLED` is present. -See `DEPENDENCY_SCANNING_DISABLED`, `CONTAINER_SCANNING_DISABLED`, `SAST_DISABLED`, and `DAST_DISABLED` -in [Auto DevOps documentation](../../topics/autodevops/customize.md#disable-jobs). +Certain GitLab workflows, such as [AutoDevOps](../../topics/autodevops/customize.md#disable-jobs), +define variables to indicate that given scans should be disabled. You can check for this by looking +for variables such as `DEPENDENCY_SCANNING_DISABLED`, `CONTAINER_SCANNING_DISABLED`, +`SAST_DISABLED`, and `DAST_DISABLED`. If appropriate based on the scanner type, you should then +disable running the custom scanner. -Finally, SAST and Dependency Scanning job definitions should use -`CI_PROJECT_REPOSITORY_LANGUAGES` (comma-separated list of values) -in order to skip the job when the language or technology is not supported. +GitLab also defines a `CI_PROJECT_REPOSITORY_LANGUAGES` variable, which provides the list of +languages in the repo. Depending on this value, your scanner may or may not do something different. Language detection currently relies on the [`linguist`](https://github.com/github/linguist) Ruby gem. See [GitLab CI/CD prefined variables](../../ci/variables/predefined_variables.md#variables-reference). -For instance, here is how to skip the Dependency Scanning job `mysec_dependency_scanning` -unless the project repository contains Java source code, -and the `dependency_scanning` feature is enabled: +#### Policy checking example + +This example shows how to skip a custom Dependency Scanning job, `mysec_dependency_scanning`, unless +the project repository contains Java source code and the `dependency_scanning` feature is enabled: ```yaml mysec_dependency_scanning: @@ -111,6 +117,8 @@ for a particular branch or when a particular set of files changes. The Docker image is a self-contained environment that combines the scanner with all the libraries and tools it depends on. +Packaging your scanner into a Docker image makes its dependencies and configuration always present, +regardless of the individual machine the scanner runs on. ### Image size @@ -144,7 +152,7 @@ It also generates text output on the standard output and standard error streams, All CI variables are passed to the scanner as environment variables. The scanned project is described by the [predefined CI variables](../../ci/variables/README.md). -#### SAST, Dependency Scanning +#### SAST and Dependency Scanning SAST and Dependency Scanning scanners must scan the files in the project directory, given by the `CI_PROJECT_DIR` variable. @@ -178,7 +186,7 @@ It is recommended to name the output file after the type of scanning, and to use Since all Secure reports are JSON files, it is recommended to use `.json` as a file extension. For instance, a suggested file name for a Dependency Scanning report is `gl-dependency-scanning.json`. -The [`artifacts:reports`](../../ci/yaml/README.md#artifactsreports) keyword +The [`artifacts:reports`](../../ci/pipelines/job_artifacts.md#artifactsreports) keyword of the job definition must be consistent with the file path where the Security report is written. For instance, if a Dependency Scanning analyzer writes its report to the CI project directory, and if this report file name is `depscan.json`, @@ -223,11 +231,8 @@ The DAST variant of the report JSON format is not documented at the moment. ### Version -The documentation of -[SAST](../../user/application_security/sast/index.md#reports-json-format), -[Dependency Scanning](../../user/application_security/dependency_scanning/index.md#reports-json-format), -and [Container Scanning](../../user/application_security/container_scanning/index.md#reports-json-format) -describes the Secure report format version. +This field specifies the version of the report schema you are using. Please reference individual scanner +pages for the specific versions to use. ### Vulnerabilities @@ -251,12 +256,17 @@ The `id` should not collide with any other scanner another integrator would prov #### Name, message, and description -The `name` and `message` fields contain a short description of the vulnerability, -whereas the `description` field provides more details. +The `name` and `message` fields contain a short description of the vulnerability. +The `description` field provides more details. -The `name` is context-free and contains no information on where the vulnerability has been found, +The `name` field is context-free and contains no information on where the vulnerability has been found, whereas the `message` may repeat the location. +As a visual example, this screenshot highlights where these fields are used when viewing a +vulnerability as part of a pipeline view. + + + For instance, a `message` for a vulnerability reported by Dependency Scanning gives information on the vulnerable dependency, which is redundant with the `location` field of the vulnerability. @@ -288,21 +298,17 @@ It should not repeat the other fields of the vulnerability object. In particular, the `description` should not repeat the `location` (what is affected) or the `solution` (how to mitigate the risk). -There is a proposal to remove either the `name` or the `message`, to remove ambiguities. -See [issue #36779](https://gitlab.com/gitlab-org/gitlab/issues/36779). - #### Solution -The `solution` field may contain instructions users should follow to fix the vulnerability or to mitigate the risk. -It is intended for users whereas the `remediations` objects are processed automatically by GitLab. +You can use the `solution` field to instruct users how to fix the identified vulnerability or to mitigate +the risk. End-users interact with this field, whereas GitLab automatically processes the +`remediations` objects. #### Identifiers -The `identifiers` array describes the vulnerability flaw that has been detected. -An identifier object has a `type` and a `value`; -these technical fields are used to tell if two identifiers are the same. -It also has a `name` and a `url`; -these fields are used to display the identifier in the user interface. +The `identifiers` array describes the detected vulnerability. An identifier object's `type` and +`value` fields are used to tell if two identifiers are the same. The user interface uses the +object's `name` and `url` fields to display the identifier. It is recommended to reuse the identifiers the GitLab scanners already define: @@ -316,18 +322,15 @@ It is recommended to reuse the identifiers the GitLab scanners already define: | [RHSA](https://access.redhat.com/errata/#/) | `rhsa` | RHSA-2020:0111 | | [ELSA](https://linux.oracle.com/security/) | `elsa` | ELSA-2020-0085 | -The generic identifiers listed above are defined in the [common library](https://gitlab.com/gitlab-org/security-products/analyzers/common); -this library is shared by the analyzers maintained by GitLab, -and this is where you can [contribute](https://gitlab.com/gitlab-org/security-products/analyzers/common/blob/master/issue/identifier.go) new generic identifiers. -Analyzers may also produce vendor-specific or product-specific identifiers; -these do not belong to the [common library](https://gitlab.com/gitlab-org/security-products/analyzers/common). +The generic identifiers listed above are defined in the [common library](https://gitlab.com/gitlab-org/security-products/analyzers/common), +which is shared by the analyzers that GitLab maintains. You can [contribute](https://gitlab.com/gitlab-org/security-products/analyzers/common/blob/master/issue/identifier.go) +new generic identifiers to if needed. Analyzers may also produce vendor-specific or product-specific +identifiers, which don't belong in the [common library](https://gitlab.com/gitlab-org/security-products/analyzers/common). The first item of the `identifiers` array is called the primary identifier. The primary identifier is particularly important, because it is used to -[track vulnerabilities](#tracking-merging-vulnerabilities) -as new commits are pushed to the repository. - -Identifiers are used to [merge duplicate vulnerabilities](#tracking-merging-vulnerabilities) +[track vulnerabilities](#tracking-and-merging-vulnerabilities) as new commits are pushed to the repository. +Identifiers are also used to [merge duplicate vulnerabilities](#tracking-and-merging-vulnerabilities) reported for the same commit, except for `CWE` and `WASC`. ### Location @@ -336,7 +339,7 @@ The `location` indicates where the vulnerability has been detected. The format of the location depends on the type of scanning. Internally GitLab extracts some attributes of the `location` to generate the **location fingerprint**, -which is used to [track vulnerabilities](#tracking-merging-vulnerabilities) +which is used to track vulnerabilities as new commits are pushed to the repository. The attributes used to generate the location fingerprint also depend on the type of scanning. @@ -426,12 +429,12 @@ combines `file`, `start_line`, and `end_line`, so these attributes are mandatory. All other attributes are optional. -### Tracking, merging vulnerabilities +### Tracking and merging vulnerabilities Users may give feedback on a vulnerability: -- they may dismiss a vulnerability if it does not apply to their projects -- or they may create an issue for a vulnerability, if there is a possible threat +- They may dismiss a vulnerability if it doesn't apply to their projects +- They may create an issue for a vulnerability if there's a possible threat GitLab tracks vulnerabilities so that user feedback is not lost when new Git commits are pushed to the repository. @@ -470,18 +473,57 @@ Valid values are: `Ignore`, `Unknown`, `Experimental`, `Low`, `Medium`, `High`, ### Remediations The `remediations` field of the report is an array of remediation objects. -Each remediation describes a patch that can be applied to automatically fix +Each remediation describes a patch that can be applied to +[automatically fix](../../user/application_security/#solutions-for-vulnerabilities-auto-remediation) a set of vulnerabilities. +Here is an example of a report that contains remediations. + +```json +{ + "vulnerabilities": [ + { + "category": "dependency_scanning", + "name": "Regular Expression Denial of Service", + "id": "123e4567-e89b-12d3-a456-426655440000", + "solution": "Upgrade to new versions.", + "scanner": { + "id": "gemnasium", + "name": "Gemnasium" + }, + "identifiers": [ + { + "type": "gemnasium", + "name": "Gemnasium-642735a5-1425-428d-8d4e-3c854885a3c9", + "value": "642735a5-1425-428d-8d4e-3c854885a3c9" + } + ] + } + ], + "remediations": [ + { + "fixes": [ + { + "id": "123e4567-e89b-12d3-a456-426655440000" + } + ], + "summary": "Upgrade to new version", + "diff": "ZGlmZiAtLWdpdCBhL3lhcm4ubG9jayBiL3lhcm4ubG9jawppbmRleCAwZWNjOTJmLi43ZmE0NTU0IDEwMDY0NAotLS0gYS95Y==" + } + ] +} +``` + #### Summary -The `summary` field is an overview of how the vulnerabilities can be fixed. +The `summary` field is an overview of how the vulnerabilities can be fixed. This field is required. #### Fixed vulnerabilities The `fixes` field is an array of objects that reference the vulnerabilities fixed by the -remediation. `fixes[].id` contains a fixed vulnerability's unique identifier. +remediation. `fixes[].id` contains a fixed vulnerability's [unique identifier](#id). This field is required. #### Diff -The `diff` field is a base64-encoded remediation code diff, compatible with [`git apply`](https://git-scm.com/docs/git-format-patch#_discussion). +The `diff` field is a base64-encoded remediation code diff, compatible with +[`git apply`](https://git-scm.com/docs/git-format-patch#_discussion). This field is required. diff --git a/doc/development/integrations/secure_partner_integration.md b/doc/development/integrations/secure_partner_integration.md index d8badda4125..59336b0e6a1 100644 --- a/doc/development/integrations/secure_partner_integration.md +++ b/doc/development/integrations/secure_partner_integration.md @@ -68,7 +68,7 @@ and complete an intgration with the Secure stage. 1. Ensure your pipeline jobs create a report artifact that GitLab can process to successfully display your own product's results with the rest of GitLab. - See detailed [technical directions](secure.md) for this step. - - Read more about [job report artifacts](../../ci/yaml/README.md#artifactsreports). + - Read more about [job report artifacts](../../ci/pipelines/job_artifacts.md#artifactsreports). - Read about [job artifacts](../../user/project/pipelines/job_artifacts.md). - Your report artifact must be in one of our currently supported formats. For more information, see the [documentation on reports](secure.md#report). diff --git a/doc/development/interacting_components.md b/doc/development/interacting_components.md index 4639bd7cc20..697c64986b1 100644 --- a/doc/development/interacting_components.md +++ b/doc/development/interacting_components.md @@ -10,7 +10,7 @@ when making _backend_ changes that might involve multiple features or [component ## Uploads GitLab supports uploads to [object storage](https://docs.gitlab.com/charts/advanced/external-object-storage/). That means every feature and -change that affects uploads should also be tested against [object storage], +change that affects uploads should also be tested against [object storage](https://docs.gitlab.com/charts/advanced/external-object-storage/), which is _not_ enabled by default in [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit). When working on a related feature, make sure to enable and test it diff --git a/doc/development/internal_api.md b/doc/development/internal_api.md index e8fca7535b0..5b53f223eb0 100644 --- a/doc/development/internal_api.md +++ b/doc/development/internal_api.md @@ -40,7 +40,7 @@ POST /internal/allowed | Attribute | Type | Required | Description | |:----------|:-------|:---------|:------------| -| `key_id` | string | no | Id of the SSH-key used to connect to GitLab-shell | +| `key_id` | string | no | ID of the SSH-key used to connect to GitLab-shell | | `username` | string | no | Username from the certificate used to connect to GitLab-Shell | | `project` | string | no (if `gl_repository` is passed) | Path to the project | | `gl_repository` | string | no (if `project` is passed) | Path to the project | @@ -93,7 +93,7 @@ information for LFS clients when the repository is accessed over SSH. | Attribute | Type | Required | Description | |:----------|:-------|:---------|:------------| -| `key_id` | string | no | Id of the SSH-key used to connect to GitLab-shell | +| `key_id` | string | no | ID of the SSH-key used to connect to GitLab-shell | | `username`| string | no | Username from the certificate used to connect to GitLab-Shell | | `project` | string | no | Path to the project | @@ -151,14 +151,14 @@ Example response: - GitLab-shell -## Get user for user id or key +## Get user for user ID or key This endpoint is used when a user performs `ssh git@gitlab.com`. It discovers the user associated with an SSH key. | Attribute | Type | Required | Description | |:----------|:-------|:---------|:------------| -| `key_id` | integer | no | The id of the SSH key used as found in the authorized-keys file or through the `/authorized_keys` check | +| `key_id` | integer | no | The ID of the SSH key used as found in the authorized-keys file or through the `/authorized_keys` check | | `username` | string | no | Username of the user being looked up, used by GitLab-shell when authenticating using a certificate | ```plaintext @@ -223,7 +223,7 @@ recovery codes based on their SSH key | Attribute | Type | Required | Description | |:----------|:-------|:---------|:------------| -| `key_id` | integer | no | The id of the SSH key used as found in the authorized-keys file or through the `/authorized_keys` check | +| `key_id` | integer | no | The ID of the SSH key used as found in the authorized-keys file or through the `/authorized_keys` check | | `user_id` | integer | no | **Deprecated** User_id for which to generate new recovery codes | ```plaintext diff --git a/doc/development/lfs.md b/doc/development/lfs.md index e64bc0f7d3a..32e2e3d1bde 100644 --- a/doc/development/lfs.md +++ b/doc/development/lfs.md @@ -2,7 +2,7 @@ ## Deep Dive -In April 2019, Francisco Javier López hosted a [Deep Dive](https://gitlab.com/gitlab-org/create-stage/issues/1) +In April 2019, Francisco Javier López hosted a Deep Dive (GitLab team members only: `https://gitlab.com/gitlab-org/create-stage/issues/1`) on GitLab's [Git LFS](../topics/git/lfs/index.md) implementation to share his domain specific knowledge with anyone who may work in this part of the code base in the future. You can find the [recording on YouTube](https://www.youtube.com/watch?v=Yyxwcksr0Qc), diff --git a/doc/development/licensed_feature_availability.md b/doc/development/licensed_feature_availability.md index 29e4ace157b..4f962b6f5e2 100644 --- a/doc/development/licensed_feature_availability.md +++ b/doc/development/licensed_feature_availability.md @@ -35,3 +35,7 @@ the instance license. ```ruby License.feature_available?(:feature_symbol) ``` + +## Enabling promo features on GitLab.com + +A paid feature can be made available to everyone on GitLab.com by enabling the feature flag `"promo_#{feature}"`. diff --git a/doc/development/licensing.md b/doc/development/licensing.md index 06a71967d06..e1be1faa61b 100644 --- a/doc/development/licensing.md +++ b/doc/development/licensing.md @@ -42,66 +42,6 @@ For all of the above, please include `--why "Reason"` and `--who "My Name"` so t More detailed information on how the gem and its commands work is available in the [License Finder README](https://github.com/pivotal/LicenseFinder). -## Acceptable Licenses +## Additional information -Libraries with the following licenses are acceptable for use: - -- [MIT License](https://choosealicense.com/licenses/mit/) (the MIT Expat License specifically): The MIT License requires that the license itself is included with all copies of the source. It is a permissive (non-copyleft) license as defined by the Open Source Initiative. -- [GNU Lesser General Public License (GNU LGPL)](https://choosealicense.com/licenses/lgpl-3.0/) (version 2, version 3): GPL constraints regarding modification and redistribution under the same license are not required of projects using an LGPL library, only upon modification of the LGPL-licensed library itself. -- [Apache 2.0 License](https://choosealicense.com/licenses/apache-2.0/): A permissive license that also provides an express grant of patent rights from contributors to users. -- [Ruby 1.8 License](https://github.com/ruby/ruby/blob/ruby_1_8_6/COPYING): Dual-licensed under either itself or the GPLv2, defer to the Ruby License itself. Acceptable because of point 3b: "You may distribute the software in object code or binary form, provided that you do at least ONE of the following: b) accompany the distribution with the machine-readable source of the software." -- [Ruby 1.9 License](https://www.ruby-lang.org/en/about/license.txt): Dual-licensed under either itself or the BSD 2-Clause License, defer to BSD 2-Clause. -- [BSD 2-Clause License](https://opensource.org/licenses/BSD-2-Clause): A permissive (non-copyleft) license as defined by the Open Source Initiative. -- [BSD 3-Clause License](https://opensource.org/licenses/BSD-3-Clause) (also known as New BSD or Modified BSD): A permissive (non-copyleft) license as defined by the Open Source Initiative -- [ISC License](https://opensource.org/licenses/ISC) (also known as the OpenBSD License): A permissive (non-copyleft) license as defined by the Open Source Initiative. -- [Creative Commons Zero (CC0)](https://creativecommons.org/publicdomain/zero/1.0/): A public domain dedication, recommended as a way to disclaim copyright on your work to the maximum extent possible. -- [Unlicense](https://unlicense.org): Another public domain dedication. -- [OWFa 1.0](http://www.openwebfoundation.org/legal/the-owf-1-0-agreements/owfa-1-0): An open-source license and patent grant designed for specifications. -- [JSON License](https://www.json.org/license.html): Equivalent to the MIT license plus the statement, "The Software shall be used for Good, not Evil." - -## Unacceptable Licenses - -Libraries with the following licenses require legal approval for use: - -- [GNU GPL](https://choosealicense.com/licenses/gpl-3.0/) (version 1, [version 2](http://www.gnu.org/licenses/gpl-2.0.txt), [version 3](http://www.gnu.org/licenses/gpl-3.0.txt), or any future versions): GPL-licensed libraries cannot be linked to from non-GPL projects. -- [GNU AGPLv3](https://choosealicense.com/licenses/agpl-3.0/): AGPL-licensed libraries cannot be linked to from non-GPL projects. -- [Open Software License (OSL)](https://opensource.org/licenses/OSL-3.0): is a copyleft license. In addition, the FSF [recommend against its use](https://www.gnu.org/licenses/license-list.en.html#OSL). -- [WTFPL](https://wtfpl.net): is a public domain dedication [rejected by the OSI (3.2)](https://opensource.org/minutes20090304). Also has a strong language which is not in accordance with our diversity policy. - -## GPL Cooperation Commitment - -Before filing or continuing to prosecute any legal proceeding or claim (other than a Defensive Action) arising from termination of a Covered License, GitLab commits to extend to the person or entity (“you”) accused of violating the Covered License the following provisions regarding cure and reinstatement, taken from GPL version 3. As used here, the term ‘this License’ refers to the specific Covered License being enforced. - -However, if you cease all violation of this License, then your license from a particular copyright holder is reinstated (a) provisionally, unless and until the copyright holder explicitly and finally terminates your license, and (b) permanently, if the copyright holder fails to notify you of the violation by some reasonable means prior to 60 days after the cessation. - -Moreover, your license from a particular copyright holder is reinstated permanently if the copyright holder notifies you of the violation by some reasonable means, this is the first time you have received notice of violation of this License (for any work) from that copyright holder, and you cure the violation prior to 30 days after your receipt of the notice. - -GitLab intends this Commitment to be irrevocable, and binding and enforceable against GitLab and assignees of or successors to GitLab’s copyrights. - -GitLab may modify this Commitment by publishing a new edition on this page or a successor location. - -Definitions - -‘Covered License’ means the GNU General Public License, version 2 (GPLv2), the GNU Lesser General Public License, version 2.1 (LGPLv2.1), or the GNU Library General Public License, version 2 (LGPLv2), all as published by the Free Software Foundation. - -‘Defensive Action’ means a legal proceeding or claim that GitLab brings against you in response to a prior proceeding or claim initiated by you or your affiliate. - -GitLab means GitLab Inc. and its affiliates and subsidiaries. - -## Requesting Approval for Licenses or any other Intellectual Property - -Libraries that are not already approved and listed on the [Acceptable Licenses](#acceptable-licenses) list or that may be listed on the [Unacceptable Licenses](#unacceptable-licenses) list may be submitted to the legal team for review and use on a case-by-case basis. Please email `legal@gitlab.com` with the details of how the software will be used, whether or not it will be modified, and how it will be distributed (if at all). After a decision has been made, the original requestor is responsible for updating this document, if applicable. Not all approvals will be approved for universal use and may continue to remain on the Unacceptable License list. - -All inquiries relating to patents should be directed to the Legal team. - -## Notes - -Decisions regarding the GNU GPL licenses are based on information provided by [The GNU Project](http://www.gnu.org/licenses/gpl-faq.html#IfLibraryIsGPL), as well as [the Open Source Initiative](https://opensource.org/faq#linking-proprietary-code), which both state that linking GPL libraries makes the program itself GPL. - -If a gem uses a license which is not listed above, open an issue and ask. If a license is not included in the "acceptable" list, operate under the assumption that it is not acceptable. - -Keep in mind that each license has its own restrictions (typically defined in their body text). Please make sure to comply with those restrictions at all times whenever an external library is used. - -Dependencies which are only used in development or test environment are exempt from license requirements, as they're not distributed for use in production. - -**NOTE:** This document is **not** legal advice, nor is it comprehensive. It should not be taken as such. +Please see the [Open Source](https://about.gitlab.com/handbook/engineering/open-source/#using-open-source-libraries) page for more information on licensing. diff --git a/doc/development/logging.md b/doc/development/logging.md index ef2d2d7022d..e7d48e4d278 100644 --- a/doc/development/logging.md +++ b/doc/development/logging.md @@ -167,7 +167,8 @@ Resources: #### Logging durations Similar to timezones, choosing the right time unit to log can impose avoidable overhead. So, whenever -challenged to choose between seconds, milliseconds or any other unit, lean towards _seconds_ as float. +challenged to choose between seconds, milliseconds or any other unit, lean towards _seconds_ as float +(with microseconds precision, i.e. `Gitlab::InstrumentationHelper::DURATION_PRECISION`). In order to make it easier to track timings in the logs, make sure the log key has `_s` as suffix and `duration` within its name (e.g., `view_duration_s`). @@ -260,6 +261,8 @@ I, [2020-01-13T19:01:17.091Z #11056] INFO -- : {"message"=>"Message", "project_ lifecycle, which can then be added to the web request or Sidekiq logs. +The API, Rails and Sidekiq logs contain fields starting with `meta.` with this context information. + Entry points can be seen at: - [`ApplicationController`](https://gitlab.com/gitlab-org/gitlab/blob/master/app/controllers/application_controller.rb) @@ -360,4 +363,4 @@ end project. See [this example](https://gitlab.com/gitlab-cookbooks/gitlab_fluentd/-/merge_requests/51/diffs). 1. Be sure to update the [GitLab CE/EE documentation](../administration/logs.md) and the [GitLab.com - runbooks](https://gitlab.com/gitlab-com/runbooks/blob/master/howto/logging.md). + runbooks](https://gitlab.com/gitlab-com/runbooks/blob/master/docs/logging/README.md). diff --git a/doc/development/mass_insert.md b/doc/development/mass_insert.md index 47f993a921e..c19850ca67e 100644 --- a/doc/development/mass_insert.md +++ b/doc/development/mass_insert.md @@ -1,7 +1,7 @@ -# Mass Inserting Rails Models +# Mass inserting Rails models -Setting the environment variable [`MASS_INSERT=1`](rake_tasks.md#env-variables) -when running `rake setup` will create millions of records, but these records +Setting the environment variable [`MASS_INSERT=1`](rake_tasks.md#environment-variables) +when running [`rake setup`](rake_tasks.md) will create millions of records, but these records aren't visible to the `root` user by default. To make any number of the mass-inserted projects visible to the `root` user, run diff --git a/doc/development/merge_request_performance_guidelines.md b/doc/development/merge_request_performance_guidelines.md index 4a9b5d26aef..b51fc681e27 100644 --- a/doc/development/merge_request_performance_guidelines.md +++ b/doc/development/merge_request_performance_guidelines.md @@ -24,7 +24,7 @@ The term `SHOULD` per the [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt) means > carefully weighed before choosing a different course. Ideally, each of these tradeoffs should be documented -in the separate issues, labelled accordingly and linked +in the separate issues, labeled accordingly and linked to original issue and epic. ## Impact Analysis diff --git a/doc/development/migration_style_guide.md b/doc/development/migration_style_guide.md index 3e993243855..4cf546173de 100644 --- a/doc/development/migration_style_guide.md +++ b/doc/development/migration_style_guide.md @@ -35,16 +35,36 @@ and post-deployment migrations (`db/post_migrate`) are run after the deployment ## Schema Changes -Migrations that make changes to the database schema (e.g. adding a column) can -only be added in the monthly release, patch releases may only contain data -migrations _unless_ schema changes are absolutely required to solve a problem. +Changes to the schema should be commited to `db/structure.sql`. This +file is automatically generated by Rails, so you normally should not +edit this file by hand. If your migration is adding a column to a +table, that column will be added at the bottom. Please do not reorder +columns manually for existing tables as this will cause confusing to +other people using `db/structure.sql` generated by Rails. + +When your local database in your GDK is diverging from the schema from +`master` it might be hard to cleanly commit the schema changes to +Git. In that case you can use the `scripts/regenerate-schema` script to +regenerate a clean `db/structure.sql` for the migrations you're +adding. This script will apply all migrations found in `db/migrate` +or `db/post_migrate`, so if there are any migrations you don't want to +commit to the schema, rename or remove them. If your branch is not +targetting `master` you can set the `TARGET` environment variable. + +```shell +# Regenerate schema against `master` +scripts/regenerate-schema + +# Regenerate schema against `12-9-stable-ee` +TARGET=12-9-stable-ee scripts/regenerate-schema +``` ## What Requires Downtime? The document ["What Requires Downtime?"](what_requires_downtime.md) specifies various database operations, such as -- [adding, dropping, and renaming columns](what_requires_downtime.md#adding-columns) +- [dropping and renaming columns](what_requires_downtime.md#dropping-columns) - [changing column constraints and types](what_requires_downtime.md#changing-column-constraints) - [adding and dropping indexes, tables, and foreign keys](what_requires_downtime.md#adding-indexes) @@ -307,6 +327,34 @@ def down end ``` +**Usage with `disable_ddl_transaction!`** + +Generally the `with_lock_retries` helper should work with `disabled_ddl_transaction!`. A custom RuboCop rule ensures that only allowed methods can be placed within the lock retries block. + +```ruby +disable_ddl_transaction! + +def up + with_lock_retries do + add_column :users, :name, :text + end + + add_text_limit :users, :name, 255 # Includes constraint validation (full table scan) +end +``` + +The RuboCop rule generally allows standard Rails migration methods, listed below. This example will cause a rubocop offense: + +```ruby +disabled_ddl_transaction! + +def up + with_lock_retries do + add_concurrent_index :users, :name + end +end +``` + ### When to use the helper method The `with_lock_retries` helper method can be used when you normally use @@ -330,8 +378,6 @@ Example changes: - `change_column_default` - `create_table` / `drop_table` -**Note:** `with_lock_retries` method **cannot** be used with `disable_ddl_transaction!`. - **Note:** `with_lock_retries` method **cannot** be used within the `change` method, you must manually define the `up` and `down` methods to make the migration reversible. ### How the helper method works @@ -506,34 +552,12 @@ You can read more about adding [foreign key constraints to an existing column](d ## Adding Columns With Default Values -When adding columns with default values to non-empty tables, you must use -`add_column_with_default`. This method ensures the table is updated without -requiring downtime. This method is not reversible so you must manually define -the `up` and `down` methods in your migration class. - -For example, to add the column `foo` to the `projects` table with a default -value of `10` you'd write the following: - -```ruby -class MyMigration < ActiveRecord::Migration[4.2] - include Gitlab::Database::MigrationHelpers - disable_ddl_transaction! - - def up - add_column_with_default(:projects, :foo, :integer, default: 10) - end - - def down - remove_column(:projects, :foo) - end -end -``` +With PostgreSQL 11 being the minimum version since GitLab 13.0, adding columns with default values has become much easier and +the standard `add_column` helper should be used in all cases. -Keep in mind that this operation can easily take 10-15 minutes to complete on -larger installations (e.g. GitLab.com). As a result, you should only add -default values if absolutely necessary. There is a RuboCop cop that will fail if -this method is used on some tables that are very large on GitLab.com, which -would cause other issues. +Before PostgreSQL 11, adding a column with a default was problematic as it would +have caused a full table rewrite. The corresponding helper `add_column_with_default` +has been deprecated and will be removed in a later release. ## Changing the column default @@ -574,8 +598,7 @@ without requiring `disable_ddl_transaction!`. ## Updating an existing column To update an existing column to a particular value, you can use -`update_column_in_batches` (`add_column_with_default` uses this internally to -fill in the default value). This will split the updates into batches, so we +`update_column_in_batches`. This will split the updates into batches, so we don't update too many rows at in a single statement. This updates the column `foo` in the `projects` table to 10, where `some_column` @@ -701,7 +724,7 @@ set the limit to 8-bytes. This will allow the column to hold a value up to Rails migration example: ```ruby -add_column_with_default(:projects, :foo, :integer, default: 10, limit: 8) +add_column(:projects, :foo, :integer, default: 10, limit: 8) ``` ## Timestamp column type diff --git a/doc/development/multi_version_compatibility.md b/doc/development/multi_version_compatibility.md new file mode 100644 index 00000000000..aedd5c1ffb7 --- /dev/null +++ b/doc/development/multi_version_compatibility.md @@ -0,0 +1,62 @@ +# Compatibility with multiple versions of the application running at the same time + +When adding or changing features, we must be aware that there may be multiple versions of the application running +at the same time and connected to the same PostgreSQL and Redis databases. This could happen during a rolling deploy +when the servers are updated one by one. + +During a rolling deploy, post-deployment DB migrations are run after all the servers have been updated. This means the +servers could be in these intermediate states: + +1. Old application code running with new DB migrations already executed +1. New application code running with new DB migrations but without new post-deployment DB migrations + +We must make sure that the application works properly in these states. + +For GitLab.com, we also run a set of canary servers which run a more recent version of the application. Users with +the canary cookie set would be handled by these servers. Some URL patterns may also be forced to the canary servers, +even without the cookie being set. This also means that some pages may match the pattern and get handled by canary servers, +but AJAX requests to URLs (like the GraphQL endpoint) won't match the pattern. + +With this canary setup, we'd be in this mixed-versions state for an extended period of time until canary is promoted to +production and post-deployment migrations run. + +## Examples of previous incidents + +### Some links to issues and MRs were broken + +When we moved MR routes, users on the new servers were redirected to the new URLs. When these users shared these new URLs in +Markdown (or anywhere else), they were broken links for users on the old servers. + +For more information, see [the relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/118840). + +### Stale cache in issue or merge request descriptions and comments + +We bumped the Markdown cache version and found a bug when a user edited a description or comment which was generated from a different Markdown +cache version. The cached HTML wasn't generated properly after saving. In most cases, this wouldn't have happened because users would have +viewed the Markdown before clicking **Edit** and that would mean the Markdown cache is refreshed. But because we run mixed versions, this is +more likely to happen. Another user on a different version could view the same page and refresh the cache to the other version behind the scenes. + +For more information, see [the relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/208255). + +### Project service templates incorrectly copied + +We changed the column which indicates whether a service is a template. When we create services, we copy attributes from the template +and set this column to `false`. The old servers were still updating the old column, but that was fine because we had a DB trigger +that updated the new column from the old one. For the new servers though, they were only updating the new column and that same trigger +was now working against us and setting it back to the wrong value. + +For more information, see [the relevant issue](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/9176). + +### Sidebar wasn't loading for some users + +We changed the data type of one GraphQL field. When a user opened an issue page from the new servers and the GraphQL AJAX request went +to the old servers, a type mismatch happened, which resulted in a JavaScript error that prevented the sidebar from loading. + +For more information, see [the relevant issue](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/1772). + +### CI artifact uploads were failing + +We added a `NOT NULL` constraint to a column and marked it as a `NOT VALID` constraint so that it is not enforced on existing rows. +But even with that, this was still a problem because the old servers were still inserting new rows with null values. + +For more information, see [the relevant issue](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/1944). diff --git a/doc/development/namespaces_storage_statistics.md b/doc/development/namespaces_storage_statistics.md index f175739e55e..3065d4f84a2 100644 --- a/doc/development/namespaces_storage_statistics.md +++ b/doc/development/namespaces_storage_statistics.md @@ -38,7 +38,7 @@ alternative method. ### Attempt A: PostgreSQL materialized view -Model can be updated through a refresh strategy based on a project routes SQL and a [materialized view](https://www.postgresql.org/docs/9.6/rules-materializedviews.html): +Model can be updated through a refresh strategy based on a project routes SQL and a [materialized view](https://www.postgresql.org/docs/11/rules-materializedviews.html): ```sql SELECT split_part("rs".path, '/', 1) as root_path, @@ -111,11 +111,11 @@ Directly relate the root namespace to its child namespaces, so whenever a namespace is created without a parent, this one is tagged with the root namespace ID: -| id | root_id | parent_id -|:---|:--------|:---------- -| 1 | 1 | NULL -| 2 | 1 | 1 -| 3 | 1 | 2 +| ID | root ID | parent ID | +|:---|:--------|:----------| +| 1 | 1 | NULL | +| 2 | 1 | 1 | +| 3 | 1 | 2 | To aggregate the statistics inside a namespace we'd execute something like: diff --git a/doc/development/new_fe_guide/development/accessibility.md b/doc/development/new_fe_guide/development/accessibility.md index 8d9f7ca5069..aa76a9fec07 100644 --- a/doc/development/new_fe_guide/development/accessibility.md +++ b/doc/development/new_fe_guide/development/accessibility.md @@ -6,13 +6,13 @@ Using semantic HTML plays a key role when it comes to accessibility. WAI-ARIA (the Accessible Rich Internet Applications specification) defines a way to make Web content and Web applications more accessible to people with disabilities. -> Note: It is [recommended][using-aria] to use semantic elements as the primary method to achieve accessibility rather than adding aria attributes. Adding aria attributes should be seen as a secondary method for creating accessible elements. +> Note: It is [recommended](https://www.w3.org/TR/using-aria/#notes2) to use semantic elements as the primary method to achieve accessibility rather than adding aria attributes. Adding aria attributes should be seen as a secondary method for creating accessible elements. ### Role The `role` attribute describes the role the element plays in the context of the document. -Check the list of WAI-ARIA roles [here][roles] +Check the list of WAI-ARIA roles [here](https://www.w3.org/TR/wai-aria-1.1/#landmark_roles) ## Icons @@ -36,20 +36,11 @@ In forms we should use the `for` attribute in the label statement: ## Testing -1. On MacOS you can use [VoiceOver][voice-over] by pressing `cmd+F5`. -1. On Windows you can use [Narrator][narrator] by pressing Windows logo key + Ctrl + Enter. +1. On MacOS you can use [VoiceOver](https://www.apple.com/accessibility/mac/vision/) by pressing `cmd+F5`. +1. On Windows you can use [Narrator](https://www.microsoft.com/en-us/accessibility/windows) by pressing Windows logo key + Ctrl + Enter. ## Online resources -- [Chrome Accessibility Developer Tools][dev-tools] for testing accessibility -- [Audit Rules Page][audit-rules] for best practices -- [Lighthouse Accessibility Score][lighthouse] for accessibility audits - -[using-aria]: https://www.w3.org/TR/using-aria/#notes2 -[dev-tools]: https://github.com/GoogleChrome/accessibility-developer-tools -[audit-rules]: https://github.com/GoogleChrome/accessibility-developer-tools/wiki/Audit-Rules -[aria-w3c]: https://www.w3.org/TR/wai-aria-1.1/ -[roles]: https://www.w3.org/TR/wai-aria-1.1/#landmark_roles -[voice-over]: https://www.apple.com/accessibility/mac/vision/ -[narrator]: https://www.microsoft.com/en-us/accessibility/windows -[lighthouse]: https://developers.google.com/web/tools/lighthouse/scoring#a11y +- [Chrome Accessibility Developer Tools](https://github.com/GoogleChrome/accessibility-developer-tools) for testing accessibility +- [Audit Rules Page](https://github.com/GoogleChrome/accessibility-developer-tools/wiki/Audit-Rules) for best practices +- [Lighthouse Accessibility Score](https://developers.google.com/web/tools/lighthouse/scoring#a11y) for accessibility audits diff --git a/doc/development/new_fe_guide/modules/dirty_submit.md b/doc/development/new_fe_guide/modules/dirty_submit.md index 30cf8017820..dd336ad3a90 100644 --- a/doc/development/new_fe_guide/modules/dirty_submit.md +++ b/doc/development/new_fe_guide/modules/dirty_submit.md @@ -13,7 +13,7 @@ within the GitLab project. ## Usage -```js +```javascript import dirtySubmitFactory from './dirty_submit/dirty_submit_form'; new DirtySubmitForm(document.querySelector('form')); diff --git a/doc/development/newlines_styleguide.md b/doc/development/newlines_styleguide.md index a13adc2f13e..e298ba1d935 100644 --- a/doc/development/newlines_styleguide.md +++ b/doc/development/newlines_styleguide.md @@ -1,4 +1,4 @@ -# Newlines styleguide +# Newlines style guide This style guide recommends best practices for newlines in Ruby code. diff --git a/doc/development/ordering_table_columns.md b/doc/development/ordering_table_columns.md index cbfd05e731d..b68602ea30d 100644 --- a/doc/development/ordering_table_columns.md +++ b/doc/development/ordering_table_columns.md @@ -77,64 +77,64 @@ always be at the end of a table. Let's use the `events` table as an example, which currently has the following layout: -| Column | Type | Size | -|:------------|:----------------------------|:---------| -| id | integer | 4 bytes | -| target_type | character varying | variable | -| target_id | integer | 4 bytes | -| title | character varying | variable | -| data | text | variable | -| project_id | integer | 4 bytes | -| created_at | timestamp without time zone | 8 bytes | -| updated_at | timestamp without time zone | 8 bytes | -| action | integer | 4 bytes | -| author_id | integer | 4 bytes | +| Column | Type | Size | +|:--------------|:----------------------------|:---------| +| `id` | integer | 4 bytes | +| `target_type` | character varying | variable | +| `target_id` | integer | 4 bytes | +| `title` | character varying | variable | +| `data` | text | variable | +| `project_id` | integer | 4 bytes | +| `created_at` | timestamp without time zone | 8 bytes | +| `updated_at` | timestamp without time zone | 8 bytes | +| `action` | integer | 4 bytes | +| `author_id` | integer | 4 bytes | After adding padding to align the columns this would translate to columns being divided into fixed size chunks as follows: -| Chunk Size | Columns | -|:-----------|:------------------| -| 8 bytes | id | -| variable | target_type | -| 8 bytes | target_id | -| variable | title | -| variable | data | -| 8 bytes | project_id | -| 8 bytes | created_at | -| 8 bytes | updated_at | -| 8 bytes | action, author_id | +| Chunk Size | Columns | +|:-----------|:----------------------| +| 8 bytes | `id` | +| variable | `target_type` | +| 8 bytes | `target_id` | +| variable | `title` | +| variable | `data` | +| 8 bytes | `project_id` | +| 8 bytes | `created_at` | +| 8 bytes | `updated_at` | +| 8 bytes | `action`, `author_id` | This means that excluding the variable sized data and tuple header, we need at least 8 * 6 = 48 bytes per row. We can optimise this by using the following column order instead: -| Column | Type | Size | -|:------------|:----------------------------|:---------| -| created_at | timestamp without time zone | 8 bytes | -| updated_at | timestamp without time zone | 8 bytes | -| id | integer | 4 bytes | -| target_id | integer | 4 bytes | -| project_id | integer | 4 bytes | -| action | integer | 4 bytes | -| author_id | integer | 4 bytes | -| target_type | character varying | variable | -| title | character varying | variable | -| data | text | variable | +| Column | Type | Size | +|:--------------|:----------------------------|:---------| +| `created_at` | timestamp without time zone | 8 bytes | +| `updated_at` | timestamp without time zone | 8 bytes | +| `id` | integer | 4 bytes | +| `target_id` | integer | 4 bytes | +| `project_id` | integer | 4 bytes | +| `action` | integer | 4 bytes | +| `author_id` | integer | 4 bytes | +| `target_type` | character varying | variable | +| `title` | character varying | variable | +| `data` | text | variable | This would produce the following chunks: -| Chunk Size | Columns | -|:-----------|:-------------------| -| 8 bytes | created_at | -| 8 bytes | updated_at | -| 8 bytes | id, target_id | -| 8 bytes | project_id, action | -| 8 bytes | author_id | -| variable | target_type | -| variable | title | -| variable | data | +| Chunk Size | Columns | +|:-----------|:-----------------------| +| 8 bytes | `created_at` | +| 8 bytes | `updated_at` | +| 8 bytes | `id`, `target_id` | +| 8 bytes | `project_id`, `action` | +| 8 bytes | `author_id` | +| variable | `target_type` | +| variable | `title` | +| variable | `data` | Here we only need 40 bytes per row excluding the variable sized data and 24-byte tuple header. 8 bytes being saved may not sound like much, but for tables as diff --git a/doc/development/packages.md b/doc/development/packages.md index 66b50ce12c8..11fc3faf4ab 100644 --- a/doc/development/packages.md +++ b/doc/development/packages.md @@ -74,7 +74,7 @@ It is using the top-level group or namespace as the defining portion of the name To avoid name conflict for instance-level endpoints you will need to define a package naming convention that gives a way to identify the project that the package belongs to. This generally involves using the project -id or full project path in the package name. See +ID or full project path in the package name. See [Conan's naming convention](../user/packages/conan_repository/index.md#package-recipe-naming-convention) as an example. For group and project-level endpoints, naming can be less constrained and it will be up to the group and project @@ -110,7 +110,7 @@ The way new package systems are integrated in GitLab is using an [MVC](https://a Required actions are all the additional requests that GitLab will need to handle so the corresponding package manager CLI can work properly. It could be a search feature or an endpoint providing meta information about a package. For example: - For NuGet, the search request was implemented during the first MVC iteration, to support Visual Studio. -- For NPM, there is a metadata endpoint used by `npm` to get the tarball url. +- For NPM, there is a metadata endpoint used by `npm` to get the tarball URL. For the first MVC iteration, it's recommended to stay at the project level of the [remote hierarchy](#remote-hierarchy). Other levels can be tackled with [future Merge Requests](#future-work). @@ -139,7 +139,7 @@ During this phase, the idea is to collect as much information as possible about - **Requests**: Which requests are needed to have a working MVC. Ideally, produce a list of all the requests needed for the MVC (including required actions). Further investigation could provide an example for each request with the request and the response bodies. -- **Upload**: Carefully analyse how the upload process works. This will probably be the most +- **Upload**: Carefully analyze how the upload process works. This will probably be the most complex request to implement. A detailed analysis is desired here as uploads can be encoded in different ways (body or multipart) and can even be in a totally different format (for example, a JSON structure where the package file is a Base64 value of @@ -159,7 +159,7 @@ During this phase, the idea is to collect as much information as possible about The analysis usually takes a full milestone to complete, though it's not impossible to start the implementation in the same milestone. -In particular, the upload request can have some [requirements in the GitLab Workhorse project](#file-uploads). This project has a different release cycle than the rails backend. It's **strongly** recommended that you open an issue there as soon as the upload request analysis is done. This way GitLab Worhorse is already ready when the upload request is implemented on the rails backend. +In particular, the upload request can have some [requirements in the GitLab Workhorse project](#file-uploads). This project has a different release cycle than the rails backend. It's **strongly** recommended that you open an issue there as soon as the upload request analysis is done. This way GitLab Workhorse is already ready when the upload request is implemented on the rails backend. ### Implementation @@ -196,8 +196,8 @@ information like the file `name`, `side`, `sha1`, etc. If there is specific data necessary to be stored for only one package system support, consider creating a separate metadata model. See `packages_maven_metadata` table -and `Packages::MavenMetadatum` model as an example for package specific data, and `packages_conan_file_metadata` table -and `Packages::ConanFileMetadatum` model as an example for package file specific data. +and `Packages::Maven::Metadatum` model as an example for package specific data, and `packages_conan_file_metadata` table +and `Packages::Conan::FileMetadatum` model as an example for package file specific data. If there is package specific behavior for a given package manager, add those methods to the metadata models and delegate from the package model. diff --git a/doc/development/performance.md b/doc/development/performance.md index 5068103ff16..2e73161a11f 100644 --- a/doc/development/performance.md +++ b/doc/development/performance.md @@ -46,7 +46,7 @@ GitLab provides built-in tools to help improve performance and availability: GitLab team members can use [GitLab.com's performance monitoring systems](https://about.gitlab.com/handbook/engineering/monitoring/) located at <https://dashboards.gitlab.net>, this requires you to log in using your `@gitlab.com` email address. Non-GitLab team-members are advised to set up their -own InfluxDB and Grafana stack. +own Prometheus and Grafana stack. ## Benchmarks @@ -109,7 +109,7 @@ In short: By collecting snapshots of process state at regular intervals, profiling allows you to see where time is spent in a process. The [StackProf](https://github.com/tmm1/stackprof) gem is included in GitLab's development environment, allowing you to investigate -the behaviour of suspect code in detail. +the behavior of suspect code in detail. It's important to note that profiling an application *alters its performance*, and will generally be done *in an unrepresentative environment*. In particular, diff --git a/doc/development/permissions.md b/doc/development/permissions.md index bca137337fc..0772389bf9e 100644 --- a/doc/development/permissions.md +++ b/doc/development/permissions.md @@ -9,9 +9,9 @@ anything that deals with permissions, all of them should be considered. Groups and projects can have the following visibility levels: -- public (20) - an entity is visible to everyone -- internal (10) - an entity is visible to logged in users -- private (0) - an entity is visible only to the approved members of the entity +- public (`20`) - an entity is visible to everyone +- internal (`10`) - an entity is visible to logged in users +- private (`0`) - an entity is visible only to the approved members of the entity The visibility level of a group can be changed only if all subgroups and subprojects have the same or lower visibility level. (e.g., a group can be set diff --git a/doc/development/pipelines.md b/doc/development/pipelines.md index 9ba6dfc110a..39ca846c1cc 100644 --- a/doc/development/pipelines.md +++ b/doc/development/pipelines.md @@ -19,6 +19,9 @@ The current stages are: <https://gitlab.com/gitlab-org/gitlab-foss>. - `prepare`: This stage includes jobs that prepare artifacts that are needed by jobs in subsequent stages. +- `build-images`: This stage includes jobs that prepare docker images + that are needed by jobs in subsequent stages or downstream pipelines. +- `fixtures`: This stage includes jobs that prepare fixtures needed by frontend tests. - `test`: This stage includes most of the tests, DB/migration jobs, and static analysis jobs. - `post-test`: This stage includes jobs that build reports or gather data from the `test` stage's jobs (e.g. coverage, Knapsack metadata etc.). @@ -30,7 +33,6 @@ The current stages are: that is deployed in the previous stage. - `post-qa`: This stage includes jobs that build reports or gather data from the `qa` stage's jobs (e.g. Review App performance report). -- `notification`: This stage includes jobs that sends notifications about pipeline status. - `pages`: This stage includes a job that deploys the various reports as GitLab Pages (e.g. <https://gitlab-org.gitlab.io/gitlab/coverage-ruby/>, <https://gitlab-org.gitlab.io/gitlab/coverage-javascript/>, @@ -68,12 +70,9 @@ that are scoped to a single [configuration parameter](../ci/yaml/README.md#confi | `.default-retry` | Allows a job to [retry](../ci/yaml/README.md#retry) upon `unknown_failure`, `api_failure`, `runner_system_failure`, `job_execution_timeout`, or `stuck_or_timeout_failure`. | | `.default-before_script` | Allows a job to use a default `before_script` definition suitable for Ruby/Rails tasks that may need a database running (e.g. tests). | | `.default-cache` | Allows a job to use a default `cache` definition suitable for Ruby/Rails and frontend tasks. | -| `.use-pg9` | Allows a job to use the `postgres:9.6.17` and `redis:alpine` services. | -| `.use-pg10` | Allows a job to use the `postgres:10.12` and `redis:alpine` services. | | `.use-pg11` | Allows a job to use the `postgres:11.6` and `redis:alpine` services. | -| `.use-pg9-ee` | Same as `.use-pg9` but also use the `docker.elastic.co/elasticsearch/elasticsearch:6.4.2` services. | -| `.use-pg10-ee` | Same as `.use-pg10` but also use the `docker.elastic.co/elasticsearch/elasticsearch:6.4.2` services. | | `.use-pg11-ee` | Same as `.use-pg11` but also use the `docker.elastic.co/elasticsearch/elasticsearch:6.4.2` services. | +| `.use-kaniko` | Allows a job to use the `kaniko` tool to build Docker images. | | `.as-if-foss` | Simulate the FOSS project by setting the `FOSS_ONLY='1'` environment variable. | ## `workflow:rules` @@ -129,11 +128,12 @@ and included in `rules` definitions via [YAML anchors](../ci/yaml/README.md#anch | `changes:` patterns | Description | |------------------------------|--------------------------------------------------------------------------| +| `ci-patterns` | Only create job for CI config-related changes. | | `yaml-patterns` | Only create job for YAML-related changes. | | `docs-patterns` | Only create job for docs-related changes. | -| `frontend-dependency-patterns` | Only create job when frontend dependencies are updated (i.e. `package.json`, and `yarn.lock`). changes. | +| `frontend-dependency-patterns` | Only create job when frontend dependencies are updated (i.e. `package.json`, and `yarn.lock`). changes. | | `frontend-patterns` | Only create job for frontend-related changes. | -| `backstage-patterns` | Only create job for backstage-related changes (i.e. Danger, fixtures, RuboCop, specs). | +| `backstage-patterns` | Only create job for backstage-related changes (i.e. Danger, fixtures, RuboCop, specs). | | `code-patterns` | Only create job for code-related changes. | | `qa-patterns` | Only create job for QA-related changes. | | `code-backstage-patterns` | Combination of `code-patterns` and `backstage-patterns`. | @@ -151,112 +151,360 @@ request, be sure to start the `dont-interrupt-me` job before pushing. ## PostgreSQL versions testing +### Current versions testing + +| Where? | PostgreSQL version | +| ------ | ------ | +| MRs | 11 | +| `master` (non-scheduled pipelines) | 11 | +| 2-hourly scheduled pipelines | 11 | + +### Long-term plan + We follow the [PostgreSQL versions shipped with Omnibus GitLab](https://docs.gitlab.com/omnibus/package-information/postgresql_versions.html): -| | 12.10 (April 2020) | 13.0 (May 2020) | 13.1 (June 2020) | 13.2 (July 2020) | 13.3 (August 2020) | 13.4, 13.5 | 13.6 (November 2020) | 14.0 (May 2021?) | +| PostgreSQL version | 12.10 (April 2020) | 13.0 (May 2020) | 13.1 (June 2020) | 13.2 (July 2020) | 13.3 (August 2020) | 13.4, 13.5 | 13.6 (November 2020) | 14.0 (May 2021?) | | ------ | ------------------ | --------------- | ---------------- | ---------------- | ------------------ | ------------ | -------------------- | ---------------- | -| PG9.6 | nightly | - | - | - | - | - | - | - | -| PG10 | `master` | - | - | - | - | - | - | - | -| PG11 | MRs/`master` | MRs/`master` | MRs/`master` | MRs/`master` | MRs/`master` | MRs/`master` | nightly | - | -| PG12 | - | - | - | - | `master` | `master` | MRs/`master` | `master` | -| PG13 | - | - | - | - | - | - | - | MRs/`master` | +| PG9.6 | MRs/`master`/`2-hour`/`nightly` | - | - | - | - | - | - | - | +| PG10 | `nightly` | - | - | - | - | - | - | - | +| PG11 | `master`/`2-hour` | MRs/`master`/`2-hour`/`nightly` | MRs/`master`/`2-hour`/`nightly` | MRs/`master`/`2-hour`/`nightly` | MRs/`master`/`2-hour`/`nightly` | MRs/`master`/`2-hour`/`nightly` | `nightly` | - | +| PG12 | - | - | - | - | `master`/`2-hour` | `master`/`2-hour` | MRs/`master`/`2-hour`/`nightly` | `master`/`2-hour` | +| PG13 | - | - | - | - | - | - | - | MRs/`master`/`2-hour`/`nightly` | + +## Pipeline types + +Since we use the [`rules:`](../ci/yaml/README.md#rules) and [`needs:`](../ci/yaml/README.md#needs) keywords extensively, +we have four main pipeline types which are described below. Note that an MR that includes multiple types of changes would +have a pipelines that include jobs from multiple types (e.g. a combination of docs-only and code-only pipelines). + +### Docs-only MR pipeline + +Reference pipeline: <https://gitlab.com/gitlab-org/gitlab/pipelines/135236627> + +```mermaid +graph LR + subgraph "No needed jobs"; + 1-1["danger-review (3.5 minutes)"]; + click 1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8100542&udv=0" + 1-50["docs lint (6.75 minutes)"]; + click 1-50 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356757&udv=0" + end +``` + +### Code-only MR pipeline + +Reference pipeline: <https://gitlab.com/gitlab-org/gitlab/pipelines/136295694> + +```mermaid +graph RL; + classDef criticalPath fill:#f66; + + subgraph "No needed jobs"; + 1-1["danger-review (3.5 minutes)"]; + click 1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8100542&udv=0" + 1-2["build-qa-image (3.4 minutes)"]; + click 1-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914325&udv=0" + 1-3["compile-assets pull-cache (9.06 minutes)"]; + click 1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914317&udv=0" + 1-4["compile-assets pull-cache as-if-foss (8.35 minutes)"]; + click 1-4 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356616&udv=0" + 1-5["gitlab:assets:compile pull-cache (22 minutes)"]; + click 1-5 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914312&udv=0" + 1-6["setup-test-env (8.22 minutes)"]; + click 1-6 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914315&udv=0" + 1-7["review-stop-failed-deployment"]; + 1-8["dependency_scanning"]; + 1-9["qa:internal, qa:internal-as-if-foss"]; + 1-11["qa:selectors, qa:selectors-as-if-foss"]; + 1-14["retrieve-tests-metadata (1.5 minutes)"]; + click 1-14 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356697&udv=0" + 1-15["code_quality"]; + 1-16["brakeman-sast"]; + 1-17["eslint-sast"]; + 1-18["kubesec-sast"]; + 1-19["nodejs-scan-sast"]; + 1-20["secrets-sast"]; + + class 1-3 criticalPath; + class 1-6 criticalPath; + end + + 2_1-1["graphql-reference-verify (5 minutes)"]; + click 2_1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356715&udv=0" + 2_1-2["memory-static (4.75 minutes)"]; + click 2_1-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356721&udv=0" + 2_1-3["run-dev-fixtures (5 minutes)"]; + click 2_1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356729&udv=0" + 2_1-4["run-dev-fixtures-ee (5 minutes)"]; + click 2_1-4 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356731&udv=0" + subgraph "Needs `setup-test-env`"; + 2_1-1 & 2_1-2 & 2_1-3 & 2_1-4 --> 1-6; + end + + 2_2-1["static-analysis (17 minutes)"]; + click 2_2-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914471&udv=0" + 2_2-2["frontend-fixtures (17.2 minutes)"]; + class 2_2-2 criticalPath; + click 2_2-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7910143&udv=0" + 2_2-3["frontend-fixtures-as-if-foss (8.75 minutes)"]; + click 2_2-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7910154&udv=0" + 2_2-4["memory-on-boot (7.19 minutes)"]; + click 2_2-4 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356727&udv=0" + 2_2-5["webpack-dev-server (6.1 minutes)"]; + click 2_2-5 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8404303&udv=0" + subgraph "Needs `setup-test-env` & `compile-assets`"; + 2_2-1 & 2_2-2 & 2_2-4 & 2_2-5 --> 1-6 & 1-3; + 2_2-3 --> 1-6 & 1-4; + end + + 2_3-1["build-assets-image (2.5 minutes)"]; + subgraph "Needs `gitlab:assets:compile`"; + 2_3-1 --> 1-5 + end + + 2_4-1["package-and-qa (manual)"]; + subgraph "Needs `build-qa-image`"; + 2_4-1 --> 1-2; + click 2_4-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914305&udv=0" + end + + 2_5-1["rspec & db jobs (12-22 minutes)"]; + subgraph "Needs `compile-assets`, `setup-test-env`, & `retrieve-tests-metadata`"; + 2_5-1 --> 1-3 & 1-6 & 1-14; + class 2_5-1 criticalPath; + click 2_5-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations" + end + + 3_1-1["jest (15 minutes)"]; + class 3_1-1 criticalPath; + click 3_1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914204&udv=0" + 3_1-2["karma (8 minutes)"]; + click 3_1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914200&udv=0" + 3_1-3["jest-as-if-foss (19.7 minutes)"]; + click 3_1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914205&udv=0" + 3_1-4["karma-as-if-foss (7.5 minutes)"]; + click 3_1-4 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914203&udv=0" + subgraph "Needs `frontend-fixtures`"; + 3_1-1 & 3_1-2 --> 2_2-2; + 3_1-3 & 3_1-4 --> 2_2-3; + end + + 3_2-1["rspec:coverage (6.5 minutes)"]; + subgraph "Depends on `rspec` jobs"; + 3_2-1 -.->|"(don't use needs because of limitations)"| 2_5-1; + class 3_2-1 criticalPath; + click 3_2-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7248745&udv=0" + end + + 4_1-1["coverage-frontend (3.6 minutes)"]; + subgraph "Needs `jest`"; + 4_1-1 --> 3_1-1; + class 4_1-1 criticalPath; + click 4_1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7910777&udv=0" + end +``` + +### Frontend-only MR pipeline + +Reference pipeline: <https://gitlab.com/gitlab-org/gitlab/pipelines/134661039> + +```mermaid +graph RL; + classDef criticalPath fill:#f66; + + subgraph "No needed jobs"; + 1-1["danger-review (3.5 minutes)"]; + click 1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8100542&udv=0" + 1-2["build-qa-image (3.4 minutes)"]; + click 1-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914325&udv=0" + 1-3["compile-assets pull-cache (9.06 minutes)"]; + click 1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914317&udv=0" + 1-4["compile-assets pull-cache as-if-foss (8.35 minutes)"]; + click 1-4 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356616&udv=0" + 1-5["gitlab:assets:compile pull-cache (22 minutes)"]; + click 1-5 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914312&udv=0" + 1-6["setup-test-env (8.22 minutes)"]; + click 1-6 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914315&udv=0" + 1-7["review-stop-failed-deployment"]; + 1-8["dependency_scanning"]; + 1-9["qa:internal, qa:internal-as-if-foss"]; + 1-11["qa:selectors, qa:selectors-as-if-foss"]; + 1-14["retrieve-tests-metadata (1.5 minutes)"]; + click 1-14 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356697&udv=0" + 1-15["code_quality"]; + 1-16["brakeman-sast"]; + 1-17["eslint-sast"]; + 1-18["kubesec-sast"]; + 1-19["nodejs-scan-sast"]; + 1-20["secrets-sast"]; + + class 1-3 criticalPath; + class 1-5 criticalPath; + class 1-6 criticalPath; + end + + 2_1-1["graphql-reference-verify (5 minutes)"]; + click 2_1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356715&udv=0" + 2_1-2["memory-static (4.75 minutes)"]; + click 2_1-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356721&udv=0" + 2_1-3["run-dev-fixtures (5 minutes)"]; + click 2_1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356729&udv=0" + 2_1-4["run-dev-fixtures-ee (5 minutes)"]; + click 2_1-4 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356731&udv=0" + subgraph "Needs `setup-test-env`"; + 2_1-1 & 2_1-2 & 2_1-3 & 2_1-4 --> 1-6; + end + + 2_2-1["static-analysis (17 minutes)"]; + click 2_2-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914471&udv=0" + 2_2-2["frontend-fixtures (17.2 minutes)"]; + class 2_2-2 criticalPath; + click 2_2-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7910143&udv=0" + 2_2-3["frontend-fixtures-as-if-foss (8.75 minutes)"]; + click 2_2-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7910154&udv=0" + 2_2-4["memory-on-boot (7.19 minutes)"]; + click 2_2-4 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356727&udv=0" + 2_2-5["webpack-dev-server (6.1 minutes)"]; + click 2_2-5 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8404303&udv=0" + subgraph "Needs `setup-test-env` & `compile-assets`"; + 2_2-1 & 2_2-2 & 2_2-4 & 2_2-5 --> 1-6 & 1-3; + 2_2-3 --> 1-6 & 1-4; + end + + 2_3-1["build-assets-image (2.5 minutes)"]; + class 2_3-1 criticalPath; + subgraph "Needs `gitlab:assets:compile`"; + 2_3-1 --> 1-5 + end -## Directed acyclic graph + 2_4-1["package-and-qa (manual)"]; + subgraph "Needs `build-qa-image` & `build-assets-image`"; + 2_4-1 --> 1-2 & 2_3-1; + click 2_4-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914305&udv=0" + end + + 2_5-1["rspec & db jobs (12-22 minutes)"]; + subgraph "Needs `compile-assets`, `setup-test-env, & `retrieve-tests-metadata`"; + 2_5-1 --> 1-3 & 1-6 & 1-14; + class 2_5-1 criticalPath; + click 2_5-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations" + end + + 2_6-1["review-build-cng (27.3 minutes)"]; + subgraph "Needs `build-assets-image`"; + 2_6-1 --> 2_3-1; + class 2_6-1 criticalPath; + click 2_6-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914314&udv=0" + end + + 3_1-1["jest (15 minutes)"]; + class 3_1-1 criticalPath; + click 3_1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914204&udv=0" + 3_1-2["karma (8 minutes)"]; + click 3_1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914200&udv=0" + 3_1-3["jest-as-if-foss (19.7 minutes)"]; + click 3_1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914205&udv=0" + 3_1-4["karma-as-if-foss (7.5 minutes)"]; + click 3_1-4 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914203&udv=0" + subgraph "Needs `frontend-fixtures`"; + 3_1-1 & 3_1-3 --> 2_2-2; + 3_1-2 & 3_1-4 --> 2_2-3; + end + + 3_2-1["rspec:coverage (6.5 minutes)"]; + subgraph "Depends on `rspec` jobs"; + 3_2-1 -.->|"(don't use needs because of limitations)"| 2_5-1; + class 3_2-1 criticalPath; + click 3_2-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7248745&udv=0" + end -We're using the [`needs:`](../ci/yaml/README.md#needs) keyword to -execute jobs out of order for the following jobs: + 4_1-1["coverage-frontend (3.6 minutes)"]; + subgraph "Needs `jest`"; + 4_1-1 --> 3_1-1; + class 4_1-1 criticalPath; + click 4_1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7910777&udv=0" + end + + 3_3-1["review-deploy (6 minutes)"]; + subgraph "Played by `review-build-cng`"; + 3_3-1 --> 2_6-1; + class 3_3-1 criticalPath; + click 3_3-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6721130&udv=0" + end + + 4_2-1["review-qa-smoke (8 minutes)"]; + click 4_2-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6729805&udv=0" + 4_2-2["review-performance (4 minutes)"]; + click 4_2-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356817&udv=0" + 4_2-3["dast (18 minutes)"]; + click 4_2-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356819&udv=0" + class 4_2-3 criticalPath; + subgraph "Played by `review-deploy`"; + 4_2-1 & 4_2-2 & 4_2-3 -.->|"(don't use needs because of limitations)"| 3_3-1; + end +``` + +### QA-only MR pipeline + +Reference pipeline: <https://gitlab.com/gitlab-org/gitlab/pipelines/134645109> ```mermaid graph RL; - A[setup-test-env]; - B["gitlab:assets:compile pull-push-cache<br/>(canonical master only)"]; - C["gitlab:assets:compile pull-cache<br/>(canonical default refs only)"]; - D["cache gems<br/>(master and tags only)"]; - E[review-build-cng]; - F[build-qa-image]; - G[review-deploy]; - I["karma, jest"]; - I2["karma-as-if-foss, jest-as-if-foss<br/>(EE default refs only)"]; - J["compile-assets pull-push-cache<br/>(master only)"]; - J2["compile-assets pull-push-cache as-if-foss<br/>(EE master only)"]; - K[compile-assets pull-cache]; - K2["compile-assets pull-cache as-if-foss<br/>(EE default refs only)"]; - U[frontend-fixtures]; - U2["frontend-fixtures-as-if-foss<br/>(EE default refs only)"]; - V["webpack-dev-server, static-analysis"]; - M[coverage]; - O[coverage-frontend]; - N["pages (master only)"]; - Q[package-and-qa]; - S["RSpec<br/>(e.g. rspec unit pg10)"] - T[retrieve-tests-metadata]; - QA["qa:internal, qa:selectors"]; - QA2["qa:internal-as-if-foss, qa:selectors-as-if-foss<br/>(EE default refs only)"]; - X["docs lint, code_quality, sast, dependency_scanning, danger-review"]; - -subgraph "`prepare` stage" - A - B - C - F - K - K2 - J - J2 - T - end - -subgraph "`fixture` stage" - U -.-> |needs and depends on| A; - U -.-> |needs and depends on| K; - U2 -.-> |needs and depends on| A; - U2 -.-> |needs and depends on| K2; - end - -subgraph "`test` stage" - D -.-> |needs| A; - I -.-> |needs and depends on| U; - I2 -.-> |needs and depends on| U2; - L -.-> |needs and depends on| A; - S -.-> |needs and depends on| A; - S -.-> |needs and depends on| K; - S -.-> |needs and depends on| T; - L["db:*, gitlab:setup, graphql-docs-verify, downtime_check"] -.-> |needs| A; - V -.-> |needs and depends on| K; - X -.-> |needs| T; - QA -.-> |needs| T; - QA2 -.-> |needs| T; - end - -subgraph "`post-test` stage" - M --> |happens after| S - O --> |needs `jest`| I - end - -subgraph "`review-prepare` stage" - E -.-> |needs| C; - end - -subgraph "`review` stage" - G -.-> |needs| E - end - -subgraph "`qa` stage" - Q -.-> |needs| C; - Q -.-> |needs| F; - QA1["review-qa-smoke, review-qa-all, review-performance, dast"] -.-> |needs| G; - end - -subgraph "`post-qa` stage" - PQA1["parallel-spec-reports"] -.-> |depends on `review-qa-all`| QA1; + classDef criticalPath fill:#f66; + + subgraph "No needed jobs"; + 1-1["danger-review (3.5 minutes)"]; + click 1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8100542&udv=0" + 1-2["build-qa-image (3.4 minutes)"]; + click 1-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914325&udv=0" + 1-3["compile-assets pull-cache (9.06 minutes)"]; + click 1-3 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914317&udv=0" + 1-4["compile-assets pull-cache as-if-foss (8.35 minutes)"]; + click 1-4 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356616&udv=0" + 1-5["gitlab:assets:compile pull-cache (22 minutes)"]; + click 1-5 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914312&udv=0" + 1-6["setup-test-env (8.22 minutes)"]; + click 1-6 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914315&udv=0" + 1-7["review-stop-failed-deployment"]; + 1-8["dependency_scanning"]; + 1-9["qa:internal, qa:internal-as-if-foss"]; + 1-11["qa:selectors, qa:selectors-as-if-foss"]; + 1-14["retrieve-tests-metadata (1.5 minutes)"]; + click 1-14 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356697&udv=0" + 1-15["code_quality"]; + 1-16["brakeman-sast"]; + 1-17["eslint-sast"]; + 1-18["kubesec-sast"]; + 1-19["nodejs-scan-sast"]; + 1-20["secrets-sast"]; + + class 1-5 criticalPath; + end + + 2_1-1["graphql-reference-verify (5 minutes)"]; + click 2_1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8356715&udv=0" + subgraph "Needs `setup-test-env`"; + 2_1-1 --> 1-6; + end + + 2_2-1["static-analysis (17 minutes)"]; + click 2_2-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914471&udv=0" + subgraph "Needs `setup-test-env` & `compile-assets`"; + 2_2-1 --> 1-6 & 1-3; + end + + 2_3-1["build-assets-image (2.5 minutes)"]; + subgraph "Needs `gitlab:assets:compile`"; + 2_3-1 --> 1-5 + class 2_3-1 criticalPath; end -subgraph "`pages` stage" - N -.-> |depends on| C; - N -.-> |depends on karma| I; - N -.-> |depends on| M; - N --> |happens after| PQA1 - end + 2_4-1["package-and-qa (108 minutes)"]; + subgraph "Needs `build-qa-image` & `build-assets-image`"; + 2_4-1 --> 1-2 & 2_3-1; + class 2_4-1 criticalPath; + click 2_4-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914305&udv=0" + end ``` ## Test jobs @@ -268,6 +516,87 @@ for more information. Consult the [Review Apps](testing_guide/review_apps.md) dedicated page for more information. +## As-if-FOSS jobs + +The `* as-if-foss` jobs allows to run GitLab's test suite "as-if-FOSS", meaning as if the jobs would run in the context +of the `gitlab-org/gitlab-foss` project. These jobs are only created in the following cases: + +- `master` commits (pushes and scheduled pipelines). +- `gitlab-org/security/gitlab` merge requests. +- Merge requests which include `RUN AS-IF-FOSS` in their title. +- Merge requests that changes the CI config. + +The `* as-if-foss` jobs have the `FOSS_ONLY='1'` variable set and gets their EE-specific +folders removed before the tests start running. + +The intent is to ensure that a change won't introduce a failure once the `gitlab-org/gitlab` project will be synced to +the `gitlab-org/gitlab-foss` project. + +## Pre-clone step + +The `gitlab-org/gitlab` project on GitLab.com uses a [pre-clone step](https://gitlab.com/gitlab-org/gitlab/issues/39134) +to seed the project with a recent archive of the repository. This is done for +several reasons: + +- It speeds up builds because a 800 MB download only takes seconds, as opposed to a full Git clone. +- It significantly reduces load on the file server, as smaller deltas mean less time spent in `git pack-objects`. + +The pre-clone step works by using the `CI_PRE_CLONE_SCRIPT` variable +[defined by GitLab.com shared runners](../user/gitlab_com/index.md#pre-clone-script). + +The `CI_PRE_CLONE_SCRIPT` is currently defined as a project CI/CD +variable: + +```shell +echo "Downloading archived master..." +wget -O /tmp/gitlab.tar.gz https://storage.googleapis.com/gitlab-ci-git-repo-cache/project-278964/gitlab-master.tar.gz + +if [ ! -f /tmp/gitlab.tar.gz ]; then + echo "Repository cache not available, cloning a new directory..." + exit +fi + +rm -rf $CI_PROJECT_DIR +echo "Extracting tarball into $CI_PROJECT_DIR..." +mkdir -p $CI_PROJECT_DIR +cd $CI_PROJECT_DIR +tar xzf /tmp/gitlab.tar.gz +rm -f /tmp/gitlab.tar.gz +chmod a+w $CI_PROJECT_DIR +``` + +The first step of the script downloads `gitlab-master.tar.gz` from +Google Cloud Storage. There is a [GitLab CI job named `cache-repo`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/cache-repo.gitlab-ci.yml#L5) +that is responsible for keeping that archive up-to-date. Every two hours +on a scheduled pipeline, it does the following: + +1. Creates a fresh clone of the `gitlab-org/gitlab` repository on GitLab.com. +1. Saves the data as a `.tar.gz`. +1. Uploads it into the Google Cloud Storage bucket. + +When a CI job runs with this configuration, you'll see something like +this: + +```shell +$ eval "$CI_PRE_CLONE_SCRIPT" +Downloading archived master... +Extracting tarball into /builds/group/project... +Fetching changes... +Reinitialized existing Git repository in /builds/group/project/.git/ +``` + +Note that the `Reinitialized existing Git repository` message shows that +the pre-clone step worked. The runner runs `git init`, which +overwrites the Git configuration with the appropriate settings to fetch +from the GitLab repository. + +`CI_REPO_CACHE_CREDENTIALS` contains the Google Cloud service account +JSON for uploading to the `gitlab-ci-git-repo-cache` bucket. These +credentials are stored in the 1Password GitLab.com Production vault. + +Note that this bucket should be located in the same continent as the +runner, or [network egress charges will apply](https://cloud.google.com/storage/pricing). + --- [Return to Development documentation](README.md) diff --git a/doc/development/policies.md b/doc/development/policies.md index 4d045411156..62442de825a 100644 --- a/doc/development/policies.md +++ b/doc/development/policies.md @@ -95,7 +95,7 @@ Each line represents a rule that was evaluated. There are a few things to note: Here you can see that the first four rules were evaluated `false` for which user and subject. For example, you can see in the last line that -the rule was activated because the user `root` had Reporter access to +the rule was activated because the user `john` had Reporter access to `Project/4`. When a policy is asked whether a particular ability is allowed @@ -123,7 +123,7 @@ class FooPolicy < BasePolicy end ``` -Naively, if we call `Ability.can?(user1, :some_ability, foo)` and `Ability.can?(user2, :some_ability, foo)`, we would have to calculate the condition twice - since they are for different users. But if we use the `scope: :subject` option: +Naively, if we call `Ability.allowed?(user1, :some_ability, foo)` and `Ability.allowed?(user2, :some_ability, foo)`, we would have to calculate the condition twice - since they are for different users. But if we use the `scope: :subject` option: ```ruby condition(:expensive_condition, scope: :subject) { @subject.expensive_query? } diff --git a/doc/development/polymorphic_associations.md b/doc/development/polymorphic_associations.md index 5d69c8add38..b6567704d8e 100644 --- a/doc/development/polymorphic_associations.md +++ b/doc/development/polymorphic_associations.md @@ -4,7 +4,7 @@ Rails makes it possible to define so called "polymorphic associations". This usually works by adding two columns to a table: a target type column, and a -target id. For example, at the time of writing we have such a setup for +target ID. For example, at the time of writing we have such a setup for `members` with the following columns: - `source_type`: a string defining the model to use, can be either `Project` or diff --git a/doc/development/profiling.md b/doc/development/profiling.md index 04713055117..2589329fc83 100644 --- a/doc/development/profiling.md +++ b/doc/development/profiling.md @@ -87,7 +87,7 @@ printer.print(File.open('/tmp/profile.html', 'w')) [GitLab-Profiler](https://gitlab.com/gitlab-com/gitlab-profiler) is a project that builds on this to add some additional niceties, such as allowing -configuration with a single Yaml file for multiple URLs, and uploading of the +configuration with a single YAML file for multiple URLs, and uploading of the profile and log output to S3. For GitLab.com, you can find the latest results here (restricted to GitLab Team members only): diff --git a/doc/development/query_recorder.md b/doc/development/query_recorder.md index f2c5b8b9848..b28fdb1252b 100644 --- a/doc/development/query_recorder.md +++ b/doc/development/query_recorder.md @@ -47,71 +47,71 @@ This could lead to false successes where subsequent "requests" could have querie There are multiple ways to find the source of queries. -1. The `QueryRecorder` `data` attribute stores queries by `file_name:line_number:method_name`. - Each entry is a `hash` with the following fields: - - - `count`: the number of times a query from this `file_name:line_number:method_name` was called - - `occurrences`: the actual `SQL` of each call - - `backtrace`: the stack trace of each call (if either of the two following options were enabled) - - `QueryRecorder#find_query` allows filtering queries by their `file_name:line_number:method_name` and - `count` attributes. For example: - - ```ruby - control = ActiveRecord::QueryRecorder.new(skip_cached: false) { visit_some_page } - control.find_query(/.*note.rb.*/, 0, first_only: true) - ``` - - `QueryRecorder#occurrences_by_line_method` returns a sorted array based on `data`, sorted by `count`. - -1. You can output the call backtrace for the specific `QueryRecorder` instance you want - by using `ActiveRecord::QueryRecorder.new(query_recorder_debug: true)`. The output - will be in `test.log` - -1. Using the environment variable `QUERY_RECORDER_DEBUG`, the call backtrace will be output for all tests. - - To enable this, run the specs with the `QUERY_RECORDER_DEBUG` environment variable set. For example: - - ```shell - QUERY_RECORDER_DEBUG=1 bundle exec rspec spec/requests/api/projects_spec.rb - ``` - - This will log calls to QueryRecorder into the `test.log` file. For example: - - ```plaintext - QueryRecorder SQL: SELECT COUNT(*) FROM "issues" WHERE "issues"."deleted_at" IS NULL AND "issues"."project_id" = $1 AND ("issues"."state" IN ('opened')) AND "issues"."confidential" = $2 - --> /home/user/gitlab/gdk/gitlab/spec/support/query_recorder.rb:19:in `callback' - --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/notifications/fanout.rb:127:in `finish' - --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/notifications/fanout.rb:46:in `block in finish' - --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/notifications/fanout.rb:46:in `each' - --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/notifications/fanout.rb:46:in `finish' - --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/notifications/instrumenter.rb:36:in `finish' - --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/notifications/instrumenter.rb:25:in `instrument' - --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract_adapter.rb:478:in `log' - --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/postgresql_adapter.rb:601:in `exec_cache' - --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/postgresql_adapter.rb:585:in `execute_and_clear' - --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/postgresql/database_statements.rb:160:in `exec_query' - --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/database_statements.rb:356:in `select' - --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/database_statements.rb:32:in `select_all' - --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/query_cache.rb:68:in `block in select_all' - --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/query_cache.rb:83:in `cache_sql' - --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/query_cache.rb:68:in `select_all' - --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/relation/calculations.rb:270:in `execute_simple_calculation' - --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/relation/calculations.rb:227:in `perform_calculation' - --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/relation/calculations.rb:133:in `calculate' - --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/relation/calculations.rb:48:in `count' - --> /home/user/gitlab/gdk/gitlab/app/services/base_count_service.rb:20:in `uncached_count' - --> /home/user/gitlab/gdk/gitlab/app/services/base_count_service.rb:12:in `block in count' - --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/cache.rb:299:in `block in fetch' - --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/cache.rb:585:in `block in save_block_result_to_cache' - --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/cache.rb:547:in `block in instrument' - --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/notifications.rb:166:in `instrument' - --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/cache.rb:547:in `instrument' - --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/cache.rb:584:in `save_block_result_to_cache' - --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/cache.rb:299:in `fetch' - --> /home/user/gitlab/gdk/gitlab/app/services/base_count_service.rb:12:in `count' - --> /home/user/gitlab/gdk/gitlab/app/models/project.rb:1296:in `open_issues_count' - ``` +- Inspect the `QueryRecorder` `data` attribute. It stores queries by `file_name:line_number:method_name`. + Each entry is a `hash` with the following fields: + + - `count`: the number of times a query from this `file_name:line_number:method_name` was called + - `occurrences`: the actual `SQL` of each call + - `backtrace`: the stack trace of each call (if either of the two following options were enabled) + + `QueryRecorder#find_query` allows filtering queries by their `file_name:line_number:method_name` and + `count` attributes. For example: + + ```ruby + control = ActiveRecord::QueryRecorder.new(skip_cached: false) { visit_some_page } + control.find_query(/.*note.rb.*/, 0, first_only: true) + ``` + + `QueryRecorder#occurrences_by_line_method` returns a sorted array based on `data`, sorted by `count`. + +- View the call backtrace for the specific `QueryRecorder` instance you want + by using `ActiveRecord::QueryRecorder.new(query_recorder_debug: true)`. The output + will be in `test.log` + +- Enable the call backtrace for all tests using the `QUERY_RECORDER_DEBUG` environment variable. + + To enable this, run the specs with the `QUERY_RECORDER_DEBUG` environment variable set. For example: + + ```shell + QUERY_RECORDER_DEBUG=1 bundle exec rspec spec/requests/api/projects_spec.rb + ``` + + This will log calls to QueryRecorder into the `test.log` file. For example: + + ```sql + QueryRecorder SQL: SELECT COUNT(*) FROM "issues" WHERE "issues"."deleted_at" IS NULL AND "issues"."project_id" = $1 AND ("issues"."state" IN ('opened')) AND "issues"."confidential" = $2 + --> /home/user/gitlab/gdk/gitlab/spec/support/query_recorder.rb:19:in `callback' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/notifications/fanout.rb:127:in `finish' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/notifications/fanout.rb:46:in `block in finish' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/notifications/fanout.rb:46:in `each' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/notifications/fanout.rb:46:in `finish' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/notifications/instrumenter.rb:36:in `finish' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/notifications/instrumenter.rb:25:in `instrument' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract_adapter.rb:478:in `log' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/postgresql_adapter.rb:601:in `exec_cache' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/postgresql_adapter.rb:585:in `execute_and_clear' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/postgresql/database_statements.rb:160:in `exec_query' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/database_statements.rb:356:in `select' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/database_statements.rb:32:in `select_all' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/query_cache.rb:68:in `block in select_all' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/query_cache.rb:83:in `cache_sql' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/query_cache.rb:68:in `select_all' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/relation/calculations.rb:270:in `execute_simple_calculation' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/relation/calculations.rb:227:in `perform_calculation' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/relation/calculations.rb:133:in `calculate' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/relation/calculations.rb:48:in `count' + --> /home/user/gitlab/gdk/gitlab/app/services/base_count_service.rb:20:in `uncached_count' + --> /home/user/gitlab/gdk/gitlab/app/services/base_count_service.rb:12:in `block in count' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/cache.rb:299:in `block in fetch' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/cache.rb:585:in `block in save_block_result_to_cache' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/cache.rb:547:in `block in instrument' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/notifications.rb:166:in `instrument' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/cache.rb:547:in `instrument' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/cache.rb:584:in `save_block_result_to_cache' + --> /home/user/.rbenv/versions/2.3.5/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/cache.rb:299:in `fetch' + --> /home/user/gitlab/gdk/gitlab/app/services/base_count_service.rb:12:in `count' + --> /home/user/gitlab/gdk/gitlab/app/models/project.rb:1296:in `open_issues_count' + ``` ## See also diff --git a/doc/development/rake_tasks.md b/doc/development/rake_tasks.md index 61d5277b1e6..96acce5e4df 100644 --- a/doc/development/rake_tasks.md +++ b/doc/development/rake_tasks.md @@ -1,8 +1,10 @@ # Rake tasks for developers -## Set up db with developer seeds +Rake tasks are available for developers and others contributing to GitLab. -Note that if your db user does not have advanced privileges you must create the db manually before running this command. +## Set up database with developer seeds + +Note that if your database user does not have advanced privileges, you must create the database manually before running this command. ```shell bundle exec rake setup @@ -12,13 +14,15 @@ The `setup` task is an alias for `gitlab:setup`. This tasks calls `db:reset` to create the database, and calls `db:seed_fu` to seed the database. Note: `db:setup` calls `db:seed` but this does nothing. -### Env variables +### Environment variables **MASS_INSERT**: Create millions of users (2m), projects (5m) and its relations. It's highly recommended to run the seed with it to catch slow queries while developing. Expect the process to take up to 20 extra minutes. -**LARGE_PROJECTS**: Create large projects (through import) from a predefined set of urls. +See also [Mass inserting Rails models](mass_insert.md). + +**LARGE_PROJECTS**: Create large projects (through import) from a predefined set of URLs. ### Seeding issues for all or a given project @@ -86,7 +90,7 @@ echo 'yes' | bundle exec rake setup To save you from answering `yes` manually. -### Discard stdout +### Discard `stdout` Since the script would print a lot of information, it could be slowing down your terminal, and it would generate more than 20G logs if you just redirect @@ -97,8 +101,8 @@ it to a file. If we don't care about the output, we could just redirect it to echo 'yes' | bundle exec rake setup > /dev/null ``` -Note that since you can't see the questions from stdout, you might just want -to `echo 'yes'` to keep it running. It would still print the errors on stderr +Note that since you can't see the questions from `stdout`, you might just want +to `echo 'yes'` to keep it running. It would still print the errors on `stderr` so no worries about missing errors. ### Extra Project seed options @@ -113,33 +117,35 @@ There are a few environment flags you can pass to change how projects are seeded In order to run the test you can use the following commands: -- `bin/rake spec` to run the rspec suite +- `bin/rake spec` to run the RSpec suite - `bin/rake spec:unit` to run only the unit tests - `bin/rake spec:integration` to run only the integration tests - `bin/rake spec:system` to run only the system tests - `bin/rake karma` to run the Karma test suite -Note: `bin/rake spec` takes significant time to pass. -Instead of running full test suite locally you can save a lot of time by running -a single test or directory related to your changes. After you submit merge request +`bin/rake spec` takes significant time to pass. +Instead of running the full test suite locally, you can save a lot of time by running +a single test or directory related to your changes. After you submit a merge request, CI will run full test suite for you. Green CI status in the merge request means full test suite is passed. -Note: You can't run `rspec .` since this will try to run all the `_spec.rb` +You can't run `rspec .` since this will try to run all the `_spec.rb` files it can find, also the ones in `/tmp` -Note: You can pass RSpec command line options to the `spec:unit`, -`spec:integration`, and `spec:system` tasks, e.g. `bin/rake "spec:unit[--tag ~geo --dry-run]"`. +You can pass RSpec command line options to the `spec:unit`, +`spec:integration`, and `spec:system` tasks. For example, `bin/rake "spec:unit[--tag ~geo --dry-run]"`. -To run a single test file you can use: +For an RSpec test, to run a single test file you can run: -- `bin/rspec spec/controllers/commit_controller_spec.rb` for a rspec test +```shell +bin/rspec spec/controllers/commit_controller_spec.rb +``` To run several tests inside one directory: -- `bin/rspec spec/requests/api/` for the rspec tests if you want to test API only +- `bin/rspec spec/requests/api/` for the RSpec tests if you want to test API only -### Speed-up tests, Rake tasks, and migrations +### Speed up tests, Rake tasks, and migrations [Spring](https://github.com/rails/spring) is a Rails application preloader. It speeds up development by keeping your application running in the background so @@ -172,18 +178,16 @@ This will compile and minify all JavaScript and CSS assets and copy them along with all other frontend assets (images, fonts, etc) into `/public/assets` where they can be easily inspected. -## Updating Emoji Aliases +## Emoji tasks -To update the Emoji aliases file (used for Emoji autocomplete) you must run the +To update the Emoji aliases file (used for Emoji autocomplete), run the following: ```shell bundle exec rake gemojione:aliases ``` -## Updating Emoji Digests - -To update the Emoji digests file (used for Emoji autocomplete) you must run the +To update the Emoji digests file (used for Emoji autocomplete), run the following: ```shell @@ -193,9 +197,7 @@ bundle exec rake gemojione:digests This will update the file `fixtures/emojis/digests.json` based on the currently available Emoji. -## Emoji Sprites - -Generating a sprite file containing all the Emoji can be done by running: +To generate a sprite file containing all the Emoji, run: ```shell bundle exec rake gemojione:sprite @@ -206,7 +208,7 @@ such changes, first generate the `emoji.png` spritesheet with the above Rake task, then check the dimensions of the new spritesheet and update the `SPRITESHEET_WIDTH` and `SPRITESHEET_HEIGHT` constants accordingly. -## Updating project templates +## Update project templates Starting a project from a template needs this project to be exported. On a up to date master branch run: @@ -249,7 +251,7 @@ bundle exec rake db:obsolete_ignored_columns Feel free to remove their definitions from their `ignored_columns` definitions. -## Update GraphQL Documentation and Schema definitions +## Update GraphQL documentation and schema definitions To generate GraphQL documentation based on the GitLab schema, run: diff --git a/doc/development/reactive_caching.md b/doc/development/reactive_caching.md index bc5fbb58af9..f3386305e93 100644 --- a/doc/development/reactive_caching.md +++ b/doc/development/reactive_caching.md @@ -85,6 +85,9 @@ The ReactiveCaching concern can be used in models as well as `project_services` 1. Implement the `calculate_reactive_cache` method in your model/service. 1. Call `with_reactive_cache` in your model/service where the cached value is needed. +1. If the `calculate_reactive_cache` method above submits requests to external services +(e.g. Prometheus, K8s), make sure to change the +[`reactive_cache_work_type` accordingly](#selfreactive_cache_work_type). ### In controllers @@ -244,6 +247,13 @@ and will silently raise `ReactiveCaching::ExceededReactiveCacheLimit` on Sentry. self.reactive_cache_hard_limit = 5.megabytes ``` +#### `self.reactive_cache_work_type` + +- This is the type of work performed by the `calculate_reactive_cache` method. Based on this attribute, +it's able to pick the right worker to process the caching job. Make sure to +set it as `:external_dependency` if the work performs any external request +(e.g. Kubernetes, Sentry). + #### `self.reactive_cache_worker_finder` - This is the method used by the background worker to find or generate the object on @@ -256,7 +266,7 @@ which `calculate_reactive_cache` can be called. end ``` -- The default behaviour can be overridden by defining a custom `reactive_cache_worker_finder`. +- The default behavior can be overridden by defining a custom `reactive_cache_worker_finder`. ```ruby class Foo < ApplicationRecord diff --git a/doc/development/repository_mirroring.md b/doc/development/repository_mirroring.md index 0a0c91821cf..1d4dbe88399 100644 --- a/doc/development/repository_mirroring.md +++ b/doc/development/repository_mirroring.md @@ -2,9 +2,10 @@ ## Deep Dive -In December 2018, Tiago Botelho hosted a [Deep Dive] on GitLab's [Pull Repository Mirroring functionality] to share his domain specific knowledge with anyone who may work in this part of the code base in the future. You can find the [recording on YouTube], and the slides in [PDF]. Everything covered in this deep dive was accurate as of GitLab 11.6, and while specific details may have changed since then, it should still serve as a good introduction. - -[Deep Dive]: https://gitlab.com/gitlab-org/create-stage/issues/1 -[Pull Repository Mirroring functionality]: ../user/project/repository/repository_mirroring.md#pulling-from-a-remote-repository-starter -[recording on YouTube]: https://www.youtube.com/watch?v=sSZq0fpdY-Y -[PDF]: https://gitlab.com/gitlab-org/create-stage/uploads/8693404888a941fd851f8a8ecdec9675/Gitlab_Create_-_Pull_Mirroring_Deep_Dive.pdf +In December 2018, Tiago Botelho hosted a Deep Dive (GitLab team members only: `https://gitlab.com/gitlab-org/create-stage/issues/1`) +on GitLab's [Pull Repository Mirroring functionality](../user/project/repository/repository_mirroring.md#pulling-from-a-remote-repository-starter) +to share his domain specific knowledge with anyone who may work in this part of the +code base in the future. You can find the [recording on YouTube](https://www.youtube.com/watch?v=sSZq0fpdY-Y), +and the slides in [PDF](https://gitlab.com/gitlab-org/create-stage/uploads/8693404888a941fd851f8a8ecdec9675/Gitlab_Create_-_Pull_Mirroring_Deep_Dive.pdf). +Everything covered in this deep dive was accurate as of GitLab 11.6, and while specific +details may have changed since then, it should still serve as a good introduction. diff --git a/doc/development/scalability.md b/doc/development/scalability.md index 9607bce3bc4..ba25e169d66 100644 --- a/doc/development/scalability.md +++ b/doc/development/scalability.md @@ -214,7 +214,7 @@ Redis process. Single-core: Like PgBouncer, a single Redis process can only use one core. It does not support multi-threading. -Dumb secondaries: Redis secondaries (aka slaves) don't actually +Dumb secondaries: Redis secondaries (aka replicas) don't actually handle any load. Unlike PostgreSQL secondaries, they don't even serve read queries. They simply replicate data from the primary and take over only when the primary fails. @@ -231,8 +231,8 @@ election to determine a new leader. No leader: A Redis cluster can get into a mode where there are no primaries. For example, this can happen if Redis nodes are misconfigured to follow the wrong node. Sometimes this requires forcing one node to -become a primary via the [`SLAVEOF NO ONE` -command](https://redis.io/commands/slaveof). +become a primary via the [`REPLICAOF NO ONE` +command](https://redis.io/commands/replicaof). ### Sidekiq diff --git a/doc/development/secure_coding_guidelines.md b/doc/development/secure_coding_guidelines.md index 0367db8939a..b473c310647 100644 --- a/doc/development/secure_coding_guidelines.md +++ b/doc/development/secure_coding_guidelines.md @@ -1,4 +1,4 @@ -# Security Guidelines +# Secure Coding Guidelines This document contains descriptions and guidelines for addressing security vulnerabilities commonly identified in the GitLab codebase. They are intended @@ -26,7 +26,7 @@ Improper permission handling can have significant impacts on the security of an Some situations may reveal [sensitive data](https://gitlab.com/gitlab-com/gl-infra/production/issues/477) or allow a malicious actor to perform [harmful actions](https://gitlab.com/gitlab-org/gitlab/issues/8180). The overall impact depends heavily on what resources can be accessed or modified improperly. -A common vulnerability when permission checks are missing is called [IDOR](https://www.owasp.org/index.php/Testing_for_Insecure_Direct_Object_References_(OTG-AUTHZ-004)) for Insecure Direct Object References. +A common vulnerability when permission checks are missing is called [IDOR](https://owasp.org/www-project-web-security-testing-guide/latest/4-Web_Application_Security_Testing/05-Authorization_Testing/04-Testing_for_Insecure_Direct_Object_References) for Insecure Direct Object References. ### When to Consider @@ -49,8 +49,8 @@ Be careful to **also test [visibility levels](https://gitlab.com/gitlab-org/gitl Some example of well implemented access controls and tests: 1. [example1](https://dev.gitlab.org/gitlab/gitlab-ee/merge_requests/710/diffs?diff_id=13750#af40ef0eaae3c1e018809e1d88086e32bccaca40_43_43) -1. [example2](https://dev.gitlab.org/gitlab/gitlabhq/merge_requests/2511/diffs#ed3aaab1510f43b032ce345909a887e5b167e196_142_155) -1. [example3](https://dev.gitlab.org/gitlab/gitlabhq/merge_requests/3170/diffs?diff_id=17494) +1. [example2](https://dev.gitlab.org/gitlab/gitlabhq/-/merge_requests/2511/diffs#ed3aaab1510f43b032ce345909a887e5b167e196_142_155) +1. [example3](https://dev.gitlab.org/gitlab/gitlabhq/-/merge_requests/3170/diffs?diff_id=17494) **NB:** any input from development team is welcome, e.g. about rubocop rules. @@ -74,11 +74,11 @@ text = "foo\nbar" p text.match /^bar$/ ``` -The output of this example is `#<MatchData "bar">`, as Ruby treats the input `text` line by line. In order to match the whole __string__ the Regex anchors `\A` and `\z` should be used according to [Rubular](https://rubular.com/). +The output of this example is `#<MatchData "bar">`, as Ruby treats the input `text` line by line. In order to match the whole __string__ the Regex anchors `\A` and `\z` should be used. #### Impact -This Ruby Regex speciality can have security impact, as often regular expressions are used for validations or to impose restrictions on user-input. +This Ruby Regex specialty can have security impact, as often regular expressions are used for validations or to impose restrictions on user-input. #### Examples @@ -104,15 +104,64 @@ Here `params[:ip]` should not contain anything else but numbers and dots. Howeve In most cases the anchors `\A` for beginning of text and `\z` for end of text should be used instead of `^` and `$`. -### Further Links +## Denial of Service (ReDoS) + +[ReDoS](https://owasp.org/www-community/attacks/Regular_expression_Denial_of_Service_-_ReDoS) is a possible attack if the attacker knows +or controls the regular expression (regex) used, and is able to enter user input to match against the bad regular expression. + +### Impact + +The resource, for example Unicorn, Puma, or Sidekiq, can be made to hang as it takes a long time to evaulate the bad regex match. + +### Examples + +GitLab-specific examples can be found in the following merge requests: + +- [MR25314](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25314) +- [MR25122](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25122#note_289087459) + +Consider the following example application, which defines a check using a regular expression. A user entering `user@aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!.com` as the email on a form will hang the web server. + +```ruby +class Email < ApplicationRecord + DOMAIN_MATCH = Regexp.new('([a-zA-Z0-9]+)+\.com') + + validates :domain_matches + + private + + def domain_matches + errors.add(:email, 'does not match') if email =~ DOMAIN_MATCH + end +``` + +### Mitigation + +GitLab has `Gitlab::UntrustedRegexp` which internally uses the [`re2`](https://github.com/google/re2/wiki/Syntax) library. +By utilizing `re2`, we get a strict limit on total execution time, and a smaller subset of available regex features. + +All user-provided regexes should use `Gitlab::UntrustedRegexp`. + +For other regexes, here are a few guidelines: + +- Remove unnecessary backtracking. +- Avoid nested quantifiers if possible. +- Try to be as precise as possible in your regex and avoid the `.` if something else can be used (e.g.: Use `_[^_]+_` instead of `_.*_` to match `_text here_`). + +An example can be found [in this commit](https://gitlab.com/gitlab-org/gitlab/commit/717824144f8181bef524592eab882dd7525a60ef). + +## Further Links - [Rubular](https://rubular.com/) is a nice online tool to fiddle with Ruby Regexps. +- [Runaway Regular Expressions](https://www.regular-expressions.info/catastrophic.html) +- [The impact of regular expression denial of service (ReDoS) in practice: an empirical study at the ecosystem scale](http://people.cs.vt.edu/~davisjam/downloads/publications/DavisCoghlanServantLee-EcosystemREDOS-ESECFSE18.pdf). This research paper discusses approaches to automatically detect ReDoS vulnerabilities. +- [Freezing the web: A study of redos vulnerabilities in JavaScript-based web servers](https://www.usenix.org/system/files/conference/usenixsecurity18/sec18-staicu.pdf). Another research paper about detecting ReDoS vulnerabilities. ## Server Side Request Forgery (SSRF) ### Description -A [Server-side Request Forgery (SSRF)][1] is an attack in which an attacker +A [Server-side Request Forgery (SSRF)](https://www.hackerone.com/blog-How-To-Server-Side-Request-Forgery-SSRF) is an attack in which an attacker is able coerce a application into making an outbound request to an unintended resource. This resource is usually internal. In GitLab, the connection most commonly uses HTTP, but an SSRF can be performed with any protocol, such as @@ -122,8 +171,6 @@ With an SSRF attack, the UI may or may not show the response. The latter is called a Blind SSRF. While the impact is reduced, it can still be useful for attackers, especially for mapping internal network services as part of recon. -[1]: https://www.hackerone.com/blog-How-To-Server-Side-Request-Forgery-SSRF - ### Impact The impact of an SSRF can vary, depending on what the application server @@ -155,16 +202,14 @@ The preferred SSRF mitigations within GitLab are: #### GitLab HTTP Library -The [GitLab::HTTP][2] wrapper library has grown to include mitigations for all of the GitLab-known SSRF vectors. It is also configured to respect the +The [GitLab::HTTP](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/http.rb) wrapper library has grown to include mitigations for all of the GitLab-known SSRF vectors. It is also configured to respect the `Outbound requests` options that allow instance administrators to block all internal connections, or limit the networks to which connections can be made. In some cases, it has been possible to configure GitLab::HTTP as the HTTP connection library for 3rd-party gems. This is preferrable over re-implementing the mitigations for a new feature. -- [More details](https://dev.gitlab.org/gitlab/gitlabhq/merge_requests/2530/diffs) - -[2]: https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/http.rb +- [More details](https://dev.gitlab.org/gitlab/gitlabhq/-/merge_requests/2530/diffs) #### Feature-specific Mitigations @@ -185,9 +230,7 @@ For situtions in which a whitelist or GitLab:HTTP cannot be used, it will be nec - For HTTP connections: Disable redirects or validate the redirect destination - To mitigate DNS rebinding attacks, validate and use the first IP address received -See [url_blocker_spec.rb][3] for examples of SSRF payloads - -[3]: https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/lib/gitlab/url_blocker_spec.rb +See [url_blocker_spec.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/lib/gitlab/url_blocker_spec.rb) for examples of SSRF payloads ## XSS guidelines @@ -236,7 +279,7 @@ For any and all input fields, ensure to define expectations on the type/format o - Validate the input using a [whitelist approach](https://youtu.be/2VFavqfDS6w?t=7816) to only allow characters through which you are expecting to receive for the field. - Input which fails validation should be **rejected**, and not sanitized. -Note that blacklists should be avoided, as it is near impossible to block all [variations of XSS](https://www.owasp.org/index.php/XSS_Filter_Evasion_Cheat_Sheet). +Note that blacklists should be avoided, as it is near impossible to block all [variations of XSS](https://owasp.org/www-community/xss-filter-evasion-cheatsheet). #### Output encoding diff --git a/doc/development/shell_scripting_guide/index.md b/doc/development/shell_scripting_guide/index.md index 387ef01bdcf..99cd1b9d67f 100644 --- a/doc/development/shell_scripting_guide/index.md +++ b/doc/development/shell_scripting_guide/index.md @@ -80,7 +80,20 @@ We format shell scripts according to the [Google Shell Style Guide](https://goog so the following `shfmt` invocation should be applied to the project's script files: ```shell -shfmt -i 2 -ci scripts/**/*.sh +shfmt -i 2 -ci -w scripts/**/*.sh +``` + +In addition to the [Linting](#linting) GitLab CI/CD job, all projects with shell scripts should also +use this job: + +```yaml +shfmt: + image: mvdan/shfmt:v3.1.0-alpine + stage: test + before_script: + - shfmt -version + script: + - shfmt -i 2 -ci -d scripts # path to your shell scripts ``` TIP: **Tip:** @@ -88,11 +101,6 @@ By default, shfmt will use the [shell detection](https://github.com/mvdan/sh#shf and ignore files starting with a period. To override this, use `-ln` flag to specify the shell dialect: `-ln posix` or `-ln bash`. -NOTE: **Note:** -Currently, the `shfmt` tool [is not shipped](https://github.com/mvdan/sh/issues/68) as a Docker image containing -a Linux shell. This makes it impossible to use the [official Docker image](https://hub.docker.com/r/mvdan/shfmt) -in GitLab Runner. This [may change](https://github.com/mvdan/sh/issues/68#issuecomment-507721371) in future. - ## Testing NOTE: **Note:** diff --git a/doc/development/sidekiq_style_guide.md b/doc/development/sidekiq_style_guide.md index 0ca25f43345..a5d0eecdc7b 100644 --- a/doc/development/sidekiq_style_guide.md +++ b/doc/development/sidekiq_style_guide.md @@ -82,6 +82,11 @@ As a general rule, a worker can be considered idempotent if: A good example of that would be a cache expiration worker. +NOTE: **Note:** +A job scheduled for an idempotent worker will automatically be +[deduplicated](#deduplication) when an unstarted job with the same +arguments is already in the queue. + ### Ensuring a worker is idempotent Make sure the worker tests pass using the following shared example: @@ -118,8 +123,50 @@ It's encouraged to only have the `idempotent!` call in the top-most worker class the `perform` method is defined in another class or module. NOTE: **Note:** -Note that a cop will fail if the worker class is not marked as idempotent. -Consider skipping the cop if you're not confident your job can safely run multiple times. +If the worker class is not marked as idempotent, a cop will fail. +Consider skipping the cop if you're not confident your job can safely +run multiple times. + +### Deduplication + +When a job for an idempotent worker is enqueued while another +unstarted job is already in the queue, GitLab drops the second +job. The work is skipped because the same work would be +done by the job that was scheduled first; by the time the second +job executed, the first job would do nothing. + +For example, `AuthorizedProjectsWorker` takes a user ID. When the +worker runs, it recalculates a user's authorizations. GitLab schedules +this job each time an action potentially changes a user's +authorizations. If the same user is added to two projects at the +same time, the second job can be skipped if the first job hasn't +begun, because when the first job runs, it creates the +authorizations for both projects. + +GitLab doesn't skip jobs scheduled in the future, as we assume that +the state will have changed by the time the job is scheduled to +execute. + +More [deduplication strategies have been suggested](https://gitlab.com/gitlab-com/gl-infra/scalability/issues/195). If you are implementing a worker that +could benefit from a different strategy, please comment in the issue. + +If the automatic deduplication were to cause issues in certain +queues. This can be temporarily disabled by enabling a feature flag +named `disable_<queue name>_deduplication`. For example to disable +deduplication for the `AuthorizedProjectsWorker`, we would enable the +feature flag `disable_authorized_projects_deduplication`. + +From chatops: + +```shell +/chatops run feature set disable_authorized_projects_deduplication true +``` + +From the rails console: + +```ruby +Feature.enable!(:disable_authorized_projects_deduplication) +``` ## Job urgency diff --git a/doc/development/telemetry/index.md b/doc/development/telemetry/index.md new file mode 100644 index 00000000000..32f63d5221e --- /dev/null +++ b/doc/development/telemetry/index.md @@ -0,0 +1,165 @@ +# Telemetry Guide + +At GitLab, we collect telemetry for the purpose of helping us build a better GitLab. Data about how GitLab is used is collected to better understand what parts of GitLab needs improvement and what features to build next. Telemetry also helps our team better understand the reasons why people use GitLab and with this knowledge we are able to make better product decisions. + +We also encourage users to enable tracking, and we embrace full transparency with our tracking approach so it can be easily understood and trusted. By enabling tracking, users can: + +- Contribute back to the wider community. +- Help GitLab improve on the product. + +This documentation consists of three guides providing an overview of Telemetry at GitLab. + +Telemetry Guide: + + 1. [Our tracking tools](#our-tracking-tools) + 1. [What data can be tracked](#what-data-can-be-tracked) + 1. [Telemetry systems overview](#telemetry-systems-overview) + +[Usage Ping Guide](usage_ping.md) + + 1. [What is Usage Ping](usage_ping.md#what-is-usage-ping) + 1. [Usage Ping payload](usage_ping.md#usage-ping-payload) + 1. [Disabling Usage Ping](usage_ping.md#disabling-usage-ping) + 1. [Usage Ping request flow](usage_ping.md#usage-ping-request-flow) + 1. [How Usage Ping works](usage_ping.md#how-usage-ping-works) + 1. [Implementing Usage Ping](usage_ping.md#implementing-usage-ping) + 1. [Developing and testing usage ping](usage_ping.md#developing-and-testing-usage-ping) + +[Snowplow Guide](snowplow.md) + +1. [What is Snowplow](snowplow.md#what-is-snowplow) +1. [Snowplow schema](snowplow.md#snowplow-schema) +1. [Enabling Snowplow](snowplow.md#enabling-snowplow) +1. [Snowplow request flow](snowplow.md#snowplow-request-flow) +1. [Implementing Snowplow JS (Frontend) tracking](snowplow.md#implementing-snowplow-js-frontend-tracking) +1. [Implementing Snowplow Ruby (Backend) tracking](snowplow.md#implementing-snowplow-ruby-backend-tracking) +1. [Developing and testing Snowplow](snowplow.md#developing-and-testing-snowplow) + +More useful links: + +- [Telemetry Direction](https://about.gitlab.com/direction/telemetry/) +- [Data Analysis Process](https://about.gitlab.com/handbook/business-ops/data-team/#-data-analysis-process) +- [Data for Product Managers](https://about.gitlab.com/handbook/business-ops/data-team/data-for-product-managers/) +- [Data Infrastructure](https://about.gitlab.com/handbook/business-ops/data-team/data-infrastructure/) + +## Our tracking tools + +In this section we will explain the six different technologies we use to gather product usage data. + +**Snowplow JS (Frontend)** + +Snowplow is an enterprise-grade marketing and product analytics platform which helps track the way users engage with our website and application. [Snowplow JS](https://github.com/snowplow/snowplow/wiki/javascript-tracker) is a frontend tracker for client-side events. + +**Snowplow Ruby (Backend)** + +Snowplow is an enterprise-grade marketing and product analytics platform which helps track the way users engage with our website and application. [Snowplow Ruby](https://github.com/snowplow/snowplow/wiki/ruby-tracker) is a backend tracker for server-side events. + +**Usage Ping** + +Usage Ping is a method for GitLab Inc to collect usage data on a GitLab instance. Usage Ping is primarily composed of row counts for different tables in the instance’s database. By comparing these counts month over month (or week over week), we can get a rough sense for how an instance is using the different features within the product. This high-level data is used to help our product, support, and sales teams. + +Read more about how this works in the [Usage Ping guide](usage_ping.md) + +**Database import** + +Database imports are full imports of data into GitLab's data warehouse. For GitLab.com, the PostgreSQL database is loaded into Snowflake data warehouse every 6 hours. For more details, see the [data team handbook](https://about.gitlab.com/handbook/business-ops/data-team/#extract-and-load). + +**Log system** + +System logs are the application logs generated from running the GitLab Rails application. For more details, see the [log system](../../administration/logs.md) and [logging infrastructure](https://gitlab.com/gitlab-com/runbooks/tree/master/logging/doc#logging-infrastructure-overview). + +## What data can be tracked + +Our different tracking tools allows us to track different types of events. The event types and examples of what data can be tracked are outlined below. + +| Event Type | Snowplow JS (Frontend) | Snowplow Ruby (Backend) | Usage Ping | Database import | Log system | +| ------ | ------ | ------ | ------ | ------ | ------ | +| Database counts | ❌ | ❌ | ✅ | ✅ | ❌ | +| Pageview events | ✅ | ✅ | ❌ | ❌ | ❌ | +| UI events | ✅ | ❌ | ❌ | ❌ | ❌ | +| CRUD and API events | ❌ | ✅ | ❌ | ❌ | ❌ | +| Event funnels | ✅ | ✅ | ❌ | ❌ | ❌ | +| PostgreSQL Data | ❌ | ❌ | ❌ | ✅ | ❌ | +| Logs | ❌ | ❌ | ❌ | ❌ | ✅ | +| External services | ❌ | ❌ | ❌ | ❌ | ❌ | + +**Database counts** + +- How many Projects have been created by unique users +- How many users logged in the past 28 day + +Database counts are row counts for different tables in an instance’s database. These are SQL count queries which have been filtered, grouped, or aggregated which provide high level usage data. The full list of available tables can be found in [structure.sql](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/structure.sql) + +**Pageview events** + +- How many sessions visited the /dashboard/groups page + +**UI Events** + +- How many sessions clicked on a button or link +- How many sessions closed a modal + +UI events are any interface-driven actions from the browser including click data. + +**CRUD or API events** + +- How many Git pushes were made +- How many GraphQL queries were made +- How many requests were made to a Rails action or controller. + +These are backend events that include the creation, read, update, deletion of records and other events that might be triggered from layers that aren't necessarily only available in the interface. + +**Event funnels** + +- How many sessions performed action A, B, then C +- What is our conversion rate from step A to B? + +**PostgreSQL data** + +These are raw database records which can be explored using business intelligence tools like Sisense. The full list of available tables can be found in [structure.sql](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/structure.sql) + +**Logs** + +These are raw logs such as the [Production logs](../../administration/logs.md#production_jsonlog), [API logs](../../administration/logs.md#api_jsonlog), or [Sidekiq logs](../../administration/logs.md#sidekiqlog). See the [overview of Logging Infrastructure](https://gitlab.com/gitlab-com/runbooks/tree/master/logging/doc#logging-infrastructure-overview) for more details. + +**External services** + +These are external services a GitLab instance interacts with such as an [external storage provider](../../administration/static_objects_external_storage.md) or an [external container registry](../../administration/packages/container_registry.md#use-an-external-container-registry-with-gitlab-as-an-auth-endpoint). These services must be able to send data back into a GitLab instance for data to be tracked. + +## Telemetry systems overview + +The systems overview is a simplified diagram showing the interactions between GitLab Inc and self-managed nstances. + + + +[Source file](https://app.diagrams.net/#G13DVpN-XnhWGz9tqReIj8pp1UE4ehk_EC) + +### GitLab Inc + +For Telemetry purposes, GitLab Inc has three major components: + +1. [Data Infrastructure](https://about.gitlab.com/handbook/business-ops/data-team/data-infrastructure/): This contains everything managed by our data team including Sisense Dashboards for visualization, Snowflake for Data Warehousing, incoming data sources such as PostgreSQL Pipeline and S3 Bucket, and lastly our data collectors [GitLab.com's Snowplow Collector](https://about.gitlab.com/handbook/engineering/infrastructure/library/snowplow/) and GitLab's Versions Application. +1. GitLab.com: This is the production GitLab application which is made up of a Client and Server. On the Client or browser side, a Snowplow JS Tracker (Frontend) is used to track client-side events. On the Server or application side, a Snowplow Ruby Tracker (Backend) is used to track server-side events. The server also contains Usage Ping which leverages a PostgreSQL database and a Redis in-memory data store to report on usage data. Lastly, the server also contains System Logs which are generated from running the GitLab application. +1. [Monitoring infrastructure](https://about.gitlab.com/handbook/engineering/monitoring/): This is the infrastructure used to ensure GitLab.com is operating smoothly. System Logs are sent from GitLab.com to our monitoring infrastructure and collected by a FluentD collector. From FluentD, logs are either sent to long term Google Cloud Services cold storage via Stackdriver, or, they are sent to our Elastic Cluster via Cloud Pub/Sub which can be explored in real-time using Kibana + +### Self-managed + +For Telemetry purposes, self-managed instances have two major components: + +1. Data infrastructure: Having a data infrastructure setup is optional on self-managed instances. If you'd like to collect Snowplow tracking events for your self-managed instance, you can setup your own self-managed Snowplow collector and configure your Snowplow events to point to your own collector. +1. GitLab: A self-managed GitLab instance contains all of the same components as GitLab.com mentioned above. + +### Differences between GitLab Inc and Self-managed + +As shown by the orange lines, on GitLab.com Snowplow JS, Snowplow Ruby, Usage Ping, and PostgreSQL database imports all flow into GitLab Inc's data fnfrastructure. However, on self-managed, only Usage Ping flows into GitLab Inc's data infrastructure. + +As shown by the green lines, on GitLab.com system logs flow into GitLab Inc's monitoring infrastructure. On self-managed, there are no logs sent to GitLab Inc's monitoring infrastructure. + +The differences between GitLab.com and self-managed are summarized below: + +| Environment | Snowplow JS (Frontend) | Snowplow Ruby (Backend) | Usage Ping | Database import | Logs system | +| ------ | ------ | ------ | ------ | ------ | ------ | +| GitLab.com | ✅ | ✅ | ✅ | ✅ | ✅ | +| Self-Managed | ❌(1) | ❌(1) | ✅ | ❌ | ❌ | + +Note (1): Snowplow JS and Snowplow Ruby are available on self-managed, however, the Snowplow Collector endpoint is set to a self-managed Snowplow Collector which GitLab Inc does not have access to. diff --git a/doc/development/telemetry/snowplow.md b/doc/development/telemetry/snowplow.md new file mode 100644 index 00000000000..aeaad6e5624 --- /dev/null +++ b/doc/development/telemetry/snowplow.md @@ -0,0 +1,393 @@ +# Snowplow Guide + +This guide provides a details about how Snowplow works. It includes the following sections: + +1. [What is Snowplow](#what-is-snowplow) +1. [Snowplow schema](#snowplow-schema) +1. [Enabling Snowplow](#enabling-snowplow) +1. [Snowplow request flow](#snowplow-request-flow) +1. [Implementing Snowplow JS (Frontend) tracking](#implementing-snowplow-js-frontend-tracking) +1. [Implementing Snowplow Ruby (Backend) tracking](#implementing-snowplow-ruby-backend-tracking) +1. [Developing and testing Snowplow](#developing-and-testing-snowplow) + +For more information about Telemetry, see: + +- [Telemetry Guide](index.md) +- [Usage Ping Guide](usage_ping.md) + +More useful links: + +- [Telemetry Direction](https://about.gitlab.com/direction/telemetry/) +- [Data Analysis Process](https://about.gitlab.com/handbook/business-ops/data-team/#-data-analysis-process) +- [Data for Product Managers](https://about.gitlab.com/handbook/business-ops/data-team/data-for-product-managers/) +- [Data Infrastructure](https://about.gitlab.com/handbook/business-ops/data-team/data-infrastructure/) + +## What is Snowplow + +Snowplow is an enterprise-grade marketing and product analytics platform which helps track the way users engage with our website and application. + +From [Snowplow's documentation](https://github.com/snowplow/snowplow), Snowplow consists of six loosely-coupled sub-systems: + +- **Trackers** fire Snowplow events. Currently Snowplow has 12 trackers, covering web, mobile, desktop, server and IoT +- **Collectors** receive Snowplow events from trackers. Currently we have three different event collectors, sinking events either to Amazon S3, Apache Kafka or Amazon Kinesis +- **Enrich** cleans up the raw Snowplow events, enriches them and puts them into storage. Currently we have a Hadoop-based enrichment process, and a Kinesis- or Kafka-based process +- **Storage** is where the Snowplow events live. Currently we store the Snowplow events in a flat file structure on S3, and in the Redshift and PostgreSQL databases +- **Data modeling** is where event-level data is joined with other data sets and aggregated into smaller data sets, and business logic is applied. This produces a clean set of tables which make it easier to perform analysis on the data. We have data models for Redshift and Looker +- **Analytics** are performed on the Snowplow events or on the aggregate tables. + + +>  + +## Snowplow schema + +We currently have many definitions of Snowplow's schema. We have an active issue to [standardize this schema](https://gitlab.com/gitlab-org/gitlab/issues/207930) including the following definitions: + +- Frontend and backend taxonomy as listed below +- [Feature instrumentation taxonomy](https://about.gitlab.com/handbook/product/feature-instrumentation/#taxonomy) +- [Self describing events](https://github.com/snowplow/snowplow/wiki/Custom-events#self-describing-events) +- [Iglu schema](https://gitlab.com/gitlab-org/iglu/) +- [Snowplow authored events](https://github.com/snowplow/snowplow/wiki/Snowplow-authored-events) + +## Enabling Snowplow + +Tracking can be enabled at: + +- The instance level, which will enable tracking on both the frontend and backend layers. +- User level, though user tracking can be disabled on a per-user basis. GitLab tracking respects the [Do Not Track](https://www.eff.org/issues/do-not-track) standard, so any user who has enabled the Do Not Track option in their browser will also not be tracked from a user level. + +We utilize Snowplow for the majority of our tracking strategy and it is enabled on GitLab.com. On a self-managed instance, Snowplow can be enabled by navigating to: + +- **Admin Area > Settings > Integrations** in the UI. +- `admin/application_settings/integrations` in your browser. + +The following configuration is required: + +| Name | Value | +| ------------- | ------------------------- | +| Collector | `snowplow.trx.gitlab.net` | +| Site ID | `gitlab` | +| Cookie domain | `.gitlab.com` | + +## Snowplow request flow + +The following example shows a basic request/response flow between a Snowplow JS / Ruby Trackers on GitLab.com, [the GitLab.com Snowplow Collector](https://about.gitlab.com/handbook/engineering/infrastructure/library/snowplow/), GitLab's S3 Bucket, GitLab's Snowflake Data Warehouse, and Sisense.: + +```mermaid +sequenceDiagram + participant Snowplow JS (Frontend) + participant Snowplow Ruby (Backend) + participant GitLab.com Snowplow Collector + participant S3 Bucket + participant Snowflake DW + participant Sisense Dashboards + Snowplow JS (Frontend) ->> GitLab.com Snowplow Collector: FE Tracking event + Snowplow Ruby (Backend) ->> GitLab.com Snowplow Collector: BE Tracking event + loop Process using Kinesis Stream + GitLab.com Snowplow Collector ->> GitLab.com Snowplow Collector: Log raw events + GitLab.com Snowplow Collector ->> GitLab.com Snowplow Collector: Enrich events + GitLab.com Snowplow Collector ->> GitLab.com Snowplow Collector: Write to disk + end + GitLab.com Snowplow Collector ->> S3 Bucket: Kinesis Firehose + S3 Bucket->>Snowflake DW: Import data + Snowflake DW->>Snowflake DW: Transform data using dbt + Snowflake DW->>Sisense Dashboards: Data available for querying +``` + +## Implementing Snowplow JS (Frontend) tracking + +GitLab provides `Tracking`, an interface that wraps the [Snowplow JavaScript Tracker](https://github.com/snowplow/snowplow/wiki/javascript-tracker) for tracking custom events. There are a few ways to utilize tracking, but each generally requires at minimum, a `category` and an `action`. Additional data can be provided that adheres to our [Feature instrumentation taxonomy](https://about.gitlab.com/handbook/product/feature-instrumentation/#taxonomy). + +| field | type | default value | description | +|:-----------|:-------|:---------------------------|:------------| +| `category` | string | document.body.dataset.page | Page or subsection of a page that events are being captured within. | +| `action` | string | 'generic' | Action the user is taking. Clicks should be `click` and activations should be `activate`, so for example, focusing a form field would be `activate_form_input`, and clicking a button would be `click_button`. | +| `data` | object | {} | Additional data such as `label`, `property`, `value`, and `context` as described [in our Feature Instrumentation taxonomy](https://about.gitlab.com/handbook/product/feature-instrumentation/#taxonomy). | + +### Tracking in HAML (or Vue Templates) + +When working within HAML (or Vue templates) we can add `data-track-*` attributes to elements of interest. All elements that have a `data-track-event` attribute will automatically have event tracking bound on clicks. + +Below is an example of `data-track-*` attributes assigned to a button: + +```haml +%button.btn{ data: { track: { event: "click_button", label: "template_preview", property: "my-template" } } } +``` + +```html +<button class="btn" + data-track-event="click_button" + data-track-label="template_preview" + data-track-property="my-template" +/> +``` + +Event listeners are bound at the document level to handle click events on or within elements with these data attributes. This allows for them to be properly handled on rerendering and changes to the DOM, but it's important to know that because of the way these events are bound, click events shouldn't be stopped from propagating up the DOM tree. If for any reason click events are being stopped from propagating, you'll need to implement your own listeners and follow the instructions in [Tracking in raw JavaScript](#tracking-in-raw-javascript). + +Below is a list of supported `data-track-*` attributes: + +| attribute | required | description | +|:----------------------|:---------|:------------| +| `data-track-event` | true | Action the user is taking. Clicks must be prepended with `click` and activations must be prepended with `activate`. For example, focusing a form field would be `activate_form_input` and clicking a button would be `click_button`. | +| `data-track-label` | false | The `label` as described [in our Feature Instrumentation taxonomy](https://about.gitlab.com/handbook/product/feature-instrumentation/#taxonomy). | +| `data-track-property` | false | The `property` as described [in our Feature Instrumentation taxonomy](https://about.gitlab.com/handbook/product/feature-instrumentation/#taxonomy). | +| `data-track-value` | false | The `value` as described [in our Feature Instrumentation taxonomy](https://about.gitlab.com/handbook/product/feature-instrumentation/#taxonomy). If omitted, this will be the element's `value` property or an empty string. For checkboxes, the default value will be the element's checked attribute or `false` when unchecked. | +| `data-track-context` | false | The `context` as described [in our Feature Instrumentation taxonomy](https://about.gitlab.com/handbook/product/feature-instrumentation/#taxonomy). | + +### Tracking within Vue components + +There's a tracking Vue mixin that can be used in components if more complex tracking is required. To use it, first import the `Tracking` library and request a mixin. + +```javascript +import Tracking from '~/tracking'; +const trackingMixin = Tracking.mixin({ label: 'right_sidebar' }); +``` + +You can provide default options that will be passed along whenever an event is tracked from within your component. For instance, if all events within a component should be tracked with a given `label`, you can provide one at this time. Available defaults are `category`, `label`, `property`, and `value`. If no category is specified, `document.body.dataset.page` is used as the default. + +You can then use the mixin normally in your component with the `mixin`, Vue declaration. The mixin also provides the ability to specify tracking options in `data` or `computed`. These will override any defaults and allows the values to be dynamic from props, or based on state. + +```javascript +export default { + mixins: [trackingMixin], + // ...[component implementation]... + data() { + return { + expanded: false, + tracking: { + label: 'left_sidebar' + } + }; + }, +} +``` + +The mixin provides a `track` method that can be called within the template, or from component methods. An example of the whole implementation might look like the following. + +```javascript +export default { + mixins: [Tracking.mixin({ label: 'right_sidebar' })], + data() { + return { + expanded: false, + }; + }, + methods: { + toggle() { + this.expanded = !this.expanded; + this.track('click_toggle', { value: this.expanded }) + } + } +}; +``` + +And if needed within the template, you can use the `track` method directly as well. + +```html +<template> + <div> + <a class="toggle" @click.prevent="toggle">Toggle</a> + <div v-if="expanded"> + <p>Hello world!</p> + <a @click.prevent="track('click_action')">Track an event</a> + </div> + </div> +</template> +``` + +### Tracking in raw JavaScript + +Custom event tracking and instrumentation can be added by directly calling the `Tracking.event` static function. The following example demonstrates tracking a click on a button by calling `Tracking.event` manually. + +```javascript +import Tracking from '~/tracking'; + +const button = document.getElementById('create_from_template_button'); +button.addEventListener('click', () => { + Tracking.event('dashboard:projects:index', 'click_button', { + label: 'create_from_template', + property: 'template_preview', + value: 'rails', + }); +}) +``` + +### Tests and test helpers + +In Jest particularly in vue tests, you can use the following: + +```javascript +import { mockTracking } from 'helpers/tracking_helper'; + +describe('MyTracking', () => { + let spy; + + beforeEach(() => { + spy = mockTracking('_category_', wrapper.element, jest.spyOn); + }); + + it('tracks an event when clicked on feedback', () => { + wrapper.find('.discover-feedback-icon').trigger('click'); + + expect(spy).toHaveBeenCalledWith('_category_', 'click_button', { + label: 'security-discover-feedback-cta', + property: '0', + }); + }); +}); + +``` + +In obsolete Karma tests it's used as below: + +```javascript +import { mockTracking, triggerEvent } from 'spec/helpers/tracking_helper'; + +describe('my component', () => { + let trackingSpy; + + beforeEach(() => { + trackingSpy = mockTracking('_category_', vm.$el, spyOn); + }); + + const triggerEvent = () => { + // action which should trigger a event + }; + + it('tracks an event when toggled', () => { + expect(trackingSpy).not.toHaveBeenCalled(); + + triggerEvent('a.toggle'); + + expect(trackingSpy).toHaveBeenCalledWith('_category_', 'click_edit_button', { + label: 'right_sidebar', + property: 'confidentiality', + }); + }); +}); +``` + +## Implementing Snowplow Ruby (Backend) tracking + +GitLab provides `Gitlab::Tracking`, an interface that wraps the [Snowplow Ruby Tracker](https://github.com/snowplow/snowplow/wiki/ruby-tracker) for tracking custom events. + +Custom event tracking and instrumentation can be added by directly calling the `GitLab::Tracking.event` class method, which accepts the following arguments: + +| argument | type | default value | description | +|:-----------|:-------|:---------------------------|:------------| +| `category` | string | 'application' | Area or aspect of the application. This could be `HealthCheckController` or `Lfs::FileTransformer` for instance. | +| `action` | string | 'generic' | The action being taken, which can be anything from a controller action like `create` to something like an Active Record callback. | +| `data` | object | {} | Additional data such as `label`, `property`, `value`, and `context` as described [in our Feature Instrumentation taxonomy](https://about.gitlab.com/handbook/product/feature-instrumentation/#taxonomy). These will be set as empty strings if you don't provide them. | + +Tracking can be viewed as either tracking user behavior, or can be utilized for instrumentation to monitor and visual performance over time in an area or aspect of code. + +For example: + +```ruby +class Projects::CreateService < BaseService + def execute + project = Project.create(params) + + Gitlab::Tracking.event('Projects::CreateService', 'create_project', + label: project.errors.full_messages.to_sentence, + value: project.valid? + ) + end +end +``` + +### Performance + +We use the [AsyncEmitter](https://github.com/snowplow/snowplow/wiki/Ruby-Tracker#52-the-asyncemitter-class) when tracking events, which allows for instrumentation calls to be run in a background thread. This is still an active area of development. + +## Developing and testing Snowplow + +There are several tools for developing and testing Snowplow Event + +| Testing Tool | Frontend Tracking | Backend Tracking | Local Development Environment | Production Environment | +| ------ | ------ | ------ | ------ | ------ | +| Snowplow Analytics Debugger Chrome Extension | ✅ | ❌ | ✅ | ✅ | +| Snowplow Inspector Chrome Extension | ✅ | ❌ | ✅ | ✅ | +| Snowplow Micro | ✅ | ✅ | ✅ | ❌ | +| Snowplow Mini | ✅ | ✅ | ❌ | ✅ | + +### Snowplow Analytics Debugger Chrome Extension + +Snowplow Analytics Debugger is a browser extension for testing frontend events. This works on production, staging and local development environments. + +1. Install [Snowplow Analytics Debugger](https://chrome.google.com/webstore/detail/snowplow-analytics-debugg/jbnlcgeengmijcghameodeaenefieedm) chrome browser extension +1. Open Chrome DevTools to the Snowplow Analytics Debugger tab +1. Learn more at [Igloo Analytics](https://www.iglooanalytics.com/blog/snowplow-analytics-debugger-chrome-extension.html) + +### Snowplow Inspector Chrome Extension + +Snowplow Inspector Chrome Extension is a browser extension for testing frontend events. This works on production, staging and local development environments. + +1. Install [Snowplow Inspector](https://chrome.google.com/webstore/detail/snowplow-inspector/maplkdomeamdlngconidoefjpogkmljm?hl=en) +1. Open the chrome extension by pressing the Snowplow Inspector icon beside the address bar +1. Click around on a webpage with Snowplow and you should see JavaScript events firing in the inspector window. + +### Snowplow Micro + +Snowplow Micro is a very small version of a full Snowplow data collection pipeline: small enough that it can be launched by a test suite. Events can be recorded into Snowplow Micro just as they can a full Snowplow pipeline. Micro then exposes an API that can be queried. + +Snowplow Micro is a docker-based solution for testing frontend and backend events in a local development environment. You need to modify GDK using the instructions below to set this up. + +- Read [Introducing Snowplow Micro](https://snowplowanalytics.com/blog/2019/07/17/introducing-snowplow-micro/) +- Look at the [Snowplow Micro repo](https://github.com/snowplow-incubator/snowplow-micro) +- Watch our [installation guide recording](https://www.youtube.com/watch?v=OX46fo_A0Ag) + +1. Install [Snowplow Micro](https://github.com/snowplow-incubator/snowplow-micro) + +``` bash +docker run --mount type=bind,source=$(pwd)/example,destination=/config -p 9090:9090 snowplow/snowplow-micro:latest --collector-config /config/micro.conf --iglu /config/iglu.json +``` + +1. Install snowplow micro by cloning the settings in [this project](https://gitlab.com/a_akgun/snowplow-micro). + + ``` bash + git clone git@gitlab.com:a_akgun/snowplow-micro.git + ./snowplow-micro.sh + ``` + +1. Update port in SQL (needed to set 9090) + + ``` bash + gdk psql -d gitlabhq_development + update application_settings set snowplow_collector_hostname='localhost:9090', snowplow_enabled=true, snowplow_cookie_domain='.gitlab.com'; + ``` + +1. Update `app/assets/javascripts/tracking.js` to [remove this line](https://gitlab.com/snippets/1918635): + + ``` javascript + forceSecureTracker: true + ``` + +1. Update `lib/gitlab/tracking.rb` to [add these lines](https://gitlab.com/snippets/1918635): + + ``` ruby + protocol: 'http', + port: 9090, + ``` + +1. Update `lib/gitlab/tracking.rb` to [change async emitter from https to http](https://gitlab.com/snippets/1918635): + + ``` ruby + SnowplowTracker::AsyncEmitter.new(Gitlab::CurrentSettings.snowplow_collector_hostname, protocol: 'http'), + ``` + +1. Enable Snowplow in the admin area, Settings::Integrations::Snowplow to point to: + `http://localhost:3000/admin/application_settings/integrations#js-snowplow-settings` +1. `gdk restart` +1. Send a test Snowplow event from the Rails console + + ``` ruby + Gitlab::Tracking.self_describing_event('iglu:com.gitlab/pageview_context/jsonschema/1-0-0', { page_type: ‘MY_TYPE' }, context: nil ) + ``` + +### Snowplow Mini + +[Snowplow Mini](https://github.com/snowplow/snowplow-mini) is an easily-deployable, single-instance version of Snowplow. + +Snowplow Mini can be used for testing frontend and backend events on a production, staging and local development environment. + +For GitLab.com, we are currently setting up a [QA and Testing environment](https://gitlab.com/gitlab-org/telemetry/-/issues/266) using Snowplow Mini. diff --git a/doc/development/telemetry/usage_ping.md b/doc/development/telemetry/usage_ping.md new file mode 100644 index 00000000000..e9b959eaa96 --- /dev/null +++ b/doc/development/telemetry/usage_ping.md @@ -0,0 +1,489 @@ +# Usage Ping Guide + +> - [Introduced][ee-557] in GitLab Enterprise Edition 8.10. +> - More statistics [were added][ee-735] in GitLab Enterprise Edition 8.12. +> - [Moved to GitLab Core][ce-23361] in 9.1. +> - More statistics [were added][ee-6602] in GitLab Ultimate 11.2. + +This guide provides a details about how usage ping works. It includes the following sections: + +1. [What is Usage Ping](#what-is-usage-ping) +1. [Usage Ping payload](#usage-ping-payload) +1. [Disabling Usage Ping](#disabling-usage-ping) +1. [Usage Ping request flow](#usage-ping-request-flow) +1. [How Usage Ping works](#how-usage-ping-works) +1. [Implementing Usage Ping](#implementing-usage-ping) +1. [Developing and testing usage ping](#developing-and-testing-usage-ping) + +For more information about Telemetry, see: + +- [Telemetry Guide](index.md) +- [Snowplow Guide](snowplow.md) + +More useful links: + +- [Telemetry Direction](https://about.gitlab.com/direction/telemetry/) +- [Data Analysis Process](https://about.gitlab.com/handbook/business-ops/data-team/#-data-analysis-process) +- [Data for Product Managers](https://about.gitlab.com/handbook/business-ops/data-team/data-for-product-managers/) +- [Data Infrastructure](https://about.gitlab.com/handbook/business-ops/data-team/data-infrastructure/) + +## What is Usage Ping + +- GitLab sends a weekly payload containing usage data to GitLab Inc. The usage ping uses high-level data to help our product, support, and sales teams. It does not send any project names, usernames, or any other specific data. The information from the usage ping is not anonymous, it is linked to the hostname of the instance. Sending usage ping is optional, and any instance can disable analytics. +- The usage data is primarily composed of row counts for different tables in the instance’s database. By comparing these counts month over month (or week over week), we can get a rough sense for how an instance is using the different features within the product. +- Usage ping is important to GitLab as we use it to calculate our and Stage Monthly Active Users (SMAU) which helps us measure the success of our stages and features. +- Once usage ping is enabled, GitLab will gather data from the other instances and will be able to show usage statistics of your instance to your users. + +### Why Should We Enable Usage Ping? + +- The main purpose of Usage Ping is to build a better GitLab. Data about how GitLab is used is collected to better understand feature/stage adoption and usage, which helps us understand how GitLab is adding value and helps our team better understand the reasons why people use GitLab and with this knowledge we are able to make better product decisions. +- As a benefit of having the usage ping active, GitLab lets you analyze the users’ activities over time of your GitLab installation. +- As a benefit of having the usage ping active, GitLab provides you with The DevOps Score,which gives you an overview of your entire instance’s adoption of Concurrent DevOps from planning to monitoring. +- You will get better, more proactive support. (assuming that our TAMs and support organization used the data to deliver more value) +- You will get insight and advice into how to get the most value out of your investment in GitLab. Wouldn't you want to know that a number of features or values are not being adopted in your organization? +- You get a report that illustrates how you compare against other similar organizations (anonymized), with specific advice and recommendations on how to improve your DevOps processes. + +### Limitations + +- Usage Ping does not track frontend events things like page views, link clicks, or user sessions and only focuses on aggregated backend events. +- Because of these limitations we recommend instrumenting your products with Snowplow for more detailed analytics on GitLab.com and use Usage Ping to track aggregated backend events on self-managed. + +## Usage Ping payload + +You can view the exact JSON payload sent to GitLab Inc. in the administration panel. To view the payload: + +1. Navigate to the **Admin Area > Settings > Metrics and profiling**. +1. Expand the **Usage statistics** section. +1. Click the **Preview payload** button. + +Here is an example of the payload structure + +``` json +{ + "uuid": "0000000-0000-0000-0000-000000000000", + "hostname": "example.com", + "version": "12.10.0-pre", + "installation_type": "omnibus-gitlab", + "active_user_count": 999, + "recorded_at": "2020-04-17T07:43:54.162+00:00", + "edition": "EEU", + "license_md5": "00000000000000000000000000000000", + "license_id": null, + "historical_max_users": 999, + "licensee": { + "Name": "ABC, Inc.", + "Email": "email@example.com", + "Company": "ABC, Inc." + }, + "license_user_count": 999, + "license_starts_at": "2020-01-01", + "license_expires_at": "2021-01-01", + "license_plan": "ultimate", + "license_add_ons": { + }, + "license_trial": false, + "counts": { + "assignee_lists": 999, + "boards": 999, + "ci_builds": 999, + ... + }, + "container_registry_enabled": true, + "dependency_proxy_enabled": false, + "gitlab_shared_runners_enabled": true, + "gravatar_enabled": true, + "influxdb_metrics_enabled": true, + "ldap_enabled": false, + "mattermost_enabled": false, + "omniauth_enabled": true, + "prometheus_metrics_enabled": false, + "reply_by_email_enabled": "incoming+%{key}@incoming.gitlab.com", + "signup_enabled": true, + "web_ide_clientside_preview_enabled": true, + "ingress_modsecurity_enabled": true, + "projects_with_expiration_policy_disabled": 999, + "projects_with_expiration_policy_enabled": 999, + ... + "elasticsearch_enabled": true, + "license_trial_ends_on": null, + "geo_enabled": false, + "git": { + "version": { + "major": 2, + "minor": 26, + "patch": 1 + } + }, + "gitaly": { + "version": "12.10.0-rc1-93-g40980d40", + "servers": 56, + "filesystems": [ + "EXT_2_3_4" + ] + }, + "gitlab_pages": { + "enabled": true, + "version": "1.17.0" + }, + "database": { + "adapter": "postgresql", + "version": "9.6.15" + }, + "app_server": { + "type": "console" + }, + "avg_cycle_analytics": { + "issue": { + "average": 999, + "sd": 999, + "missing": 999 + }, + "plan": { + "average": null, + "sd": 999, + "missing": 999 + }, + "code": { + "average": null, + "sd": 999, + "missing": 999 + }, + "test": { + "average": null, + "sd": 999, + "missing": 999 + }, + "review": { + "average": null, + "sd": 999, + "missing": 999 + }, + "staging": { + "average": null, + "sd": 999, + "missing": 999 + }, + "production": { + "average": null, + "sd": 999, + "missing": 999 + }, + "total": 999 + }, + "usage_activity_by_stage": { + "configure": { + "project_clusters_enabled": 999, + ... + }, + "create": { + "merge_requests": 999, + ... + }, + "manage": { + "events": 999, + ... + }, + "monitor": { + "clusters": 999, + ... + }, + "package": { + "projects_with_packages": 999 + }, + "plan": { + "issues": 999, + ... + }, + "release": { + "deployments": 999, + ... + }, + "secure": { + "user_container_scanning_jobs": 999, + ... + }, + "verify": { + "ci_builds": 999, + ... + } + }, + "usage_activity_by_stage_monthly": { + "configure": { + "project_clusters_enabled": 999, + ... + }, + "create": { + "merge_requests": 999, + ... + }, + "manage": { + "events": 999, + ... + }, + "monitor": { + "clusters": 999, + ... + }, + "package": { + "projects_with_packages": 999 + }, + "plan": { + "issues": 999, + ... + }, + "release": { + "deployments": 999, + ... + }, + "secure": { + "user_container_scanning_jobs": 999, + ... + }, + "verify": { + "ci_builds": 999, + ... + } + } +} +``` + +## Disabling usage ping + +The usage ping is opt-out. If you want to deactivate this feature, go to the Settings page of your administration panel and uncheck the Usage Ping checkbox. + +To disable the usage ping and prevent it from being configured in future through the administration panel, Omnibus installs can set the following in [`gitlab.rb`](https://docs.gitlab.com/omnibus/settings/configuration.html#configuration-options): + +```ruby +gitlab_rails['usage_ping_enabled'] = false +``` + +And source installs can set the following in `gitlab.yml`: + +```yaml +production: &base + # ... + gitlab: + # ... + usage_ping_enabled: false +``` + +## Usage Ping Request Flow + +The following example shows a basic request/response flow between a GitLab Instance, the Versions Application, the License Application, Salesforce, GitLab's S3 Bucket, GitLab's Snowflake Data Warehouse, and Sisense.: + +```mermaid +sequenceDiagram + participant GitLab Instance + participant Versions Application + participant Licenses Application + participant Salesforce + participant S3 Bucket + participant Snowflake DW + participant Sisense Dashboards + GitLab Instance->>Versions Application: Send Usage Ping + loop Process usage data + Versions Application->>Versions Application: Parse usage data + Versions Application->>Versions Application: Write to database + Versions Application->>Versions Application: Update license ping time + end + loop Process data for Salesforce + Versions Application-xLicenses Application: Request Zuora subscription id + Licenses Application-xVersions Application: Zuora subscription id + Versions Application-xSalesforce: Request Zuora account id by Zuora subscription id + Salesforce-xVersions Application: Zuora account id + Versions Application-xSalesforce: Usage data for the Zuora account + end + Versions Application->>S3 Bucket: Export Versions database + S3 Bucket->>Snowflake DW: Import data + Snowflake DW->>Snowflake DW: Transform data using dbt + Snowflake DW->>Sisense Dashboards: Data available for querying + Versions Application->>GitLab Instance: DevOps Score (Conversational Development Index) +``` + +## How Usage Ping works + +1. The Usage Ping [cron job](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/gitlab_usage_ping_worker.rb#L30) is set in Sidekiq to run weekly. +1. When the cron job runs, it calls [GitLab::UsageData.to_json](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/submit_usage_ping_service.rb#L22). +1. GitLab::UsageData.to_json [cascades down](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data.rb#L22) to ~400+ other counter method calls. +1. The response of all methods calls are [merged together](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data.rb#L14) into a single JSON payload in GitLab::UsageData.to_json. +1. The JSON payload is then [posted to the Versions application]( https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/submit_usage_ping_service.rb#L20). + +## Implementing Usage Ping + +Usage Ping consists of four types of counters which are all found in `usage_data.rb`: + +- **Ordinary Batch Counters:** Simple count of a given ActiveRecord_Relation +- **Distinct Batch Counters:** Distinct count of a given ActiveRecord_Relation on given column +- **Alternative Counters:** Used for settings and configurations +- **Redis Counters:** Used for in-memory counts. This method is being deprecated due to data inaccuracies and will be replaced with a persistent method. + +Note: Only use the provided counter methods. Each counter method contains a built in fail safe to isolate each counter to avoid breaking the entire Usage Ping. + +### Why batch counting + +For large tables, PostgreSQL can take a long time to count rows due to MVCC [(Multi-version Concurrency Control)](https://en.wikipedia.org/wiki/Multiversion_concurrency_control). Batch counting is a counting method where a single large query is broken into multiple smaller queries. For example, instead of a single query querying 1,000,000 records, with batch counting, you can execute 100 queries of 10,000 records each. Batch counting is useful for avoiding database timeouts as each batch query is significantly shorter than one single long running query. + +For GitLab.com, there are extremely large tables with 15 second query timeouts, so, we use batch counting to avoid encountering timeouts. Here are the sizes of some GitLab.com tables: + +| Table | Row counts in millions | +| ------ | ------ | +| merge_request_diff_commits | 2280 | +| ci_build_trace_sections | 1764 | +| merge_request_diff_files | 1082 | +| events | 514 | + +There are two batch counting methods provided, `Ordinary Batch Counters` and `Distinct Batch Counters`. Batch counting requires indexes on columns to calculate max, min, and range queries. In some cases, a specialized index may need to be added on the columns involved in a counter. + +### Ordinary Batch Counters + +Handles `ActiveRecord::StatementInvalid` error + +Simple count of a given ActiveRecord_Relation + +Method: `count(relation, column = nil, batch: true, start: nil, finish: nil)` + +Arguments: + +- `relation` the ActiveRecord_Relation to perform the count +- `column` the column to perform the count on, by default is the primary key +- `batch`: default `true` in order to use batch counting +- `start`: custom start of the batch counting in order to avoid complex min calculations +- `end`: custom end of the batch counting in order to avoid complex min calculations + +Examples: + +```ruby +count(User.active) +count(::Clusters::Cluster.aws_installed.enabled, :cluster_id) +count(::Clusters::Cluster.aws_installed.enabled, :cluster_id, start: ::Clusters::Cluster.minimum(:id), finish: ::Clusters::Cluster.maximum(:id)) +``` + +### Distinct Batch Counters + +Handles `ActiveRecord::StatementInvalid` error + +Distinct count of a given ActiveRecord_Relation on given column + +Method: `distinct_count(relation, column = nil, batch: true, start: nil, finish: nil)` + +Arguments: + +- `relation` the ActiveRecord_Relation to perform the count +- `column` the column to perform the distinct count, by default is the primary key +- `batch`: default `true` in order to use batch counting +- `start`: custom start of the batch counting in order to avoid complex min calculations +- `end`: custom end of the batch counting in order to avoid complex min calculations + +Examples: + +```ruby +distinct_count(::Project, :creator_id) +distinct_count(::Note.with_suggestions.where(time_period), :author_id, start: ::User.minimum(:id), finish: ::User.maximum(:id)) +distinct_count(::Clusters::Applications::CertManager.where(time_period).available.joins(:cluster), 'clusters.user_id') +``` + +### Redis Counters + +Handles `::Redis::CommandError` and `Gitlab::UsageDataCounters::BaseCounter::UnknownEvent` +returns -1 when a block is sent or hash with all values -1 when a `counter(Gitlab::UsageDataCounters)` is sent +different behavior due to 2 different implementations of Redis counter + +Method: `redis_usage_data(counter, &block)` + +Arguments: + +- `counter`: a counter from `Gitlab::UsageDataCounters`, that has `fallback_totals` method implemented +- or a `block`: wich is evaluated + +Example of usage: + +```ruby +redis_usage_data(Gitlab::UsageDataCounters::WikiPageCounter) +redis_usage_data { ::Gitlab::UsageCounters::PodLogs.usage_totals[:total] } +``` + +Note that Redis counters are in the [process of being deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/216330) and you should instead try to use Snowplow events instead. We're in the process of building [self-managed event tracking](https://gitlab.com/gitlab-org/telemetry/-/issues/373) and once this is available, we will convert all Redis counters into Snowplow events. + +### Alternative Counters + +Handles `StandardError` and fallbacks into -1 this way not all measures fail if we encounter one exception. +Mainly used for settings and configurations. + +Method: `alt_usage_data(value = nil, fallback: -1, &block)` + +Arguments: + +- `value`: a simple static value in wich case the value is simply returned. +- or a `block`: wich is evaluated +- `fallback: -1`: the common value used for any metrics that are failing. + +Example of usage: + +```ruby +alt_usage_data { Gitlab::VERSION } +alt_usage_data { Gitlab::CurrentSettings.uuid } +alt_usage_data(999) +``` + +## Developing and testing Usage Ping + +### 1. Use your Rails console to manually test counters + +```ruby +# count +Gitlab::UsageData.count(User.active) +Gitlab::UsageData.count(::Clusters::Cluster.aws_installed.enabled, :cluster_id) + +# count distinct +Gitlab::UsageData.distinct_count(::Project, :creator_id) +Gitlab::UsageData.distinct_count(::Note.with_suggestions.where(time_period), :author_id, start: ::User.minimum(:id), finish: ::User.maximum(:id)) +``` + +### 2. Generate the SQL query + +Your Rails console will give back the generated SQL queries. + +Example: + +```ruby + pry(main)> Gitlab::UsageData.count(User.active) + (0.4ms) SELECT "features"."key" FROM "features" + (0.7ms) SELECT MIN("users"."id") FROM "users" WHERE ("users"."state" IN ('active')) AND (ghost IS NOT TRUE) AND ("users"."user_type" IS NULL OR "users"."user_type" NOT IN (2, 1, 3)) + (0.6ms) SELECT MAX("users"."id") FROM "users" WHERE ("users"."state" IN ('active')) AND (ghost IS NOT TRUE) AND ("users"."user_type" IS NULL OR "users"."user_type" NOT IN (2, 1, 3)) + (0.5ms) SELECT COUNT("users"."id") FROM "users" WHERE ("users"."state" IN ('active')) AND (ghost IS NOT TRUE) AND ("users"."user_type" IS NULL OR "users"."user_type" NOT IN (2, 1, 3)) AND "users"."id" BETWEEN 0 AND 99999 +``` + +### 3. Optimize queries with #database-lab + +Paste the SQL query into `#database-lab` to see how the query performs at scale. + +- #database-lab is a Slack channel which uses a production-sized environment to test your queries +- GitLab.com’s production database has a 15 second timeout. +- For each query we require an execution time of under 1 second due do cold caches which can 10x this time. +- Add a specialized index on columns involved to reduce your the execution time. + +In order to have an understanding of the queries execution we add in the MR description the following information + +For counters that have a `time_period` test and add information for both cases. + +- with `time_period = {}` for all time period +- and `time_period = { created_at: 28.days.ago..Time.current }` for last 28 days period + +Execution plan and query time before and after optimization + +Using database-lab and [explain.depesz.com](https://explain.depesz.com/) see more details in [database review guide](../database_review.md#preparation-when-adding-or-modifying-queries) + +Query generated for the index and time + +Using database-lab + +Migration output for up and down execution + +Examples of query optimization work: + +- [Example 1](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26445) +- [Example 2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26871) + +### 4. Ask for a Telemetry Review + +On GitLab.com, we have DangerBot setup to monitor Telemetry related files and DangerBot will recommend a Telemetry review. Simply `@gitlab-org/growth/telemetry/engineers` in your MR for a review. diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index 62767180077..f0137e542cc 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -17,24 +17,22 @@ our test design. We can find some helpful heuristics documented in the Handbook ## Test speed -GitLab has a massive test suite that, without [parallelization], can take hours +GitLab has a massive test suite that, without [parallelization](ci.md#test-suite-parallelization-on-the-ci), can take hours to run. It's important that we make an effort to write tests that are accurate and effective _as well as_ fast. Here are some things to keep in mind regarding test performance: -- `double` and `spy` are faster than `FactoryBot.build(...)` +- `instance_double` and `spy` are faster than `FactoryBot.build(...)` - `FactoryBot.build(...)` and `.build_stubbed` are faster than `.create`. - Don't `create` an object when `build`, `build_stubbed`, `attributes_for`, - `spy`, or `double` will do. Database persistence is slow! + `spy`, or `instance_double` will do. Database persistence is slow! - Don't mark a feature as requiring JavaScript (through `:js` in RSpec) unless it's _actually_ required for the test to be valid. Headless browser testing is slow! -[parallelization]: ci.md#test-suite-parallelization-on-the-ci - ## RSpec -To run rspec tests: +To run RSpec tests: ```shell # run all tests @@ -71,6 +69,8 @@ FDOC=1 bin/rspec spec/[path]/[to]/[spec].rb - Use `Gitlab.config.gitlab.host` rather than hard coding `'localhost'` - Don't assert against the absolute value of a sequence-generated attribute (see [Gotchas](../gotchas.md#do-not-assert-against-the-absolute-value-of-a-sequence-generated-attribute)). +- Avoid using `expect_any_instance_of` or `allow_any_instance_of` (see + [Gotchas](../gotchas.md#do-not-assert-against-the-absolute-value-of-a-sequence-generated-attribute)). - Don't supply the `:each` argument to hooks since it's the default. - On `before` and `after` hooks, prefer it scoped to `:context` over `:all` - When using `evaluate_script("$('.js-foo').testSomething()")` (or `execute_script`) which acts on a given element, @@ -320,26 +320,48 @@ stub_feature_flags(ci_live_trace: false) Feature.enabled?(:ci_live_trace) # => false ``` -If you wish to set up a test where a feature flag is disabled for some -actors and not others, you can specify this in options passed to the -helper. For example, to disable the `ci_live_trace` feature flag for a -specifc project: +If you wish to set up a test where a feature flag is enabled only +for some actors and not others, you can specify this in options +passed to the helper. For example, to enable the `ci_live_trace` +feature flag for a specifc project: ```ruby project1, project2 = build_list(:project, 2) -# Feature will only be disabled for project1 -stub_feature_flags(ci_live_trace: { enabled: false, thing: project1 }) +# Feature will only be enabled for project1 +stub_feature_flags(ci_live_trace: project1) + +Feature.enabled?(:ci_live_trace) # => false +Feature.enabled?(:ci_live_trace, project1) # => true +Feature.enabled?(:ci_live_trace, project2) # => false +``` + +This represents an actual behavior of FlipperGate: -Feature.enabled?(:ci_live_trace, project1) # => false -Feature.enabled?(:ci_live_trace, project2) # => true +1. You can enable an override for a specified actor to be enabled +1. You can disable (remove) an override for a specified actor, + fallbacking to default state +1. There's no way to model that you explicitly disable a specified actor + +```ruby +Feature.enable(:my_feature) +Feature.disable(:my_feature, project1) +Feature.enabled?(:my_feature) # => true +Feature.enabled?(:my_feature, project1) # => true +``` + +```ruby +Feature.disable(:my_feature2) +Feature.enable(:my_feature2, project1) +Feature.enabled?(:my_feature2) # => false +Feature.enabled?(:my_feature2, project1) # => true ``` ### Pristine test environments The code exercised by a single GitLab test may access and modify many items of data. Without careful preparation before a test runs, and cleanup afterward, -data can be changed by a test in such a way that it affects the behaviour of +data can be changed by a test in such a way that it affects the behavior of following tests. This should be avoided at all costs! Fortunately, the existing test framework handles most cases already. @@ -493,7 +515,7 @@ range of inputs. By specifying the test case once, alongside a table of inputs and the expected output for each, your tests can be made easier to read and more compact. -We use the [rspec-parameterized](https://github.com/tomykaira/rspec-parameterized) +We use the [RSpec::Parameterized](https://github.com/tomykaira/rspec-parameterized) gem. A short example, using the table syntax and checking Ruby equality for a range of inputs, might look like this: @@ -526,7 +548,7 @@ objects, FactoryBot-created objects etc. can lead to ### Prometheus tests Prometheus metrics may be preserved from one test run to another. To ensure that metrics are -reset before each example, add the `:prometheus` tag to the Rspec test. +reset before each example, add the `:prometheus` tag to the RSpec test. ### Matchers @@ -651,7 +673,7 @@ end ### Factories -GitLab uses [factory_bot] as a test fixture replacement. +GitLab uses [factory_bot](https://github.com/thoughtbot/factory_bot) as a test fixture replacement. - Factory definitions live in `spec/factories/`, named using the pluralization of their corresponding model (`User` factories are defined in `users.rb`). @@ -666,8 +688,6 @@ GitLab uses [factory_bot] as a test fixture replacement. - Factories don't have to be limited to `ActiveRecord` objects. [See example](https://gitlab.com/gitlab-org/gitlab-foss/commit/0b8cefd3b2385a21cfed779bd659978c0402766d). -[factory_bot]: https://github.com/thoughtbot/factory_bot - ### Fixtures All fixtures should be placed under `spec/fixtures/`. diff --git a/doc/development/testing_guide/end_to_end/beginners_guide.md b/doc/development/testing_guide/end_to_end/beginners_guide.md new file mode 100644 index 00000000000..73960a2f74d --- /dev/null +++ b/doc/development/testing_guide/end_to_end/beginners_guide.md @@ -0,0 +1,340 @@ +# Beginner's guide to writing end-to-end tests + +In this tutorial, you will learn about the creation of end-to-end (_e2e_) tests +for [GitLab Community Edition](https://about.gitlab.com/install/?version=ce) and +[GitLab Enterprise Edition](https://about.gitlab.com/install/). + +By the end of this tutorial, you will be able to: + +- Determine whether an end-to-end test is needed. +- Understand the directory structure within `qa/`. +- Write a basic end-to-end test that will validate login features. +- Develop any missing [page object](page_objects.md) libraries. + +## Before you write a test + +Before you write tests, your +[GitLab Development Kit (GDK)](https://gitlab.com/gitlab-org/gitlab-development-kit) +must be configured to run the specs. The end-to-end tests: + +- Are contained within the `qa/` directory. +- Should be independent and + [idempotent](https://en.wikipedia.org/wiki/Idempotence#Computer_science_meaning). +- Create [resources](resources.md) (such as project, issue, user) on an ad-hoc basis. +- Test the UI and API interfaces, and use the API to efficiently set up the UI tests. + +TIP: **Tip:** +For more information, see [End-to-end testing Best Practices](best_practices.md). + +## Determine if end-to-end tests are needed + +Check the code coverage of a specific feature before writing end-to-end tests, +for both [GitLab Community Edition](https://gitlab-org.gitlab.io/gitlab-foss/coverage-ruby/#_AllFiles) +and [GitLab Enterprise Edition](https://gitlab-org.gitlab.io/gitlab/coverage-ruby/#_AllFiles) projects. +Does sufficient test coverage exist at the unit, feature, or integration levels? +If you answered *yes*, then you *don't* need an end-to-end test. + +For information about the distribution of tests per level in GitLab, see +[Testing Levels](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/development/testing_guide/testing_levels.md). + +- See the + [How to test at the correct level?](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/development/testing_guide/testing_levels.md#how-to-test-at-the-correct-level) + section of the [Testing levels](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/development/testing_guide/testing_levels.md) document. +- Review how often the feature changes. Stable features that don't change very often + might not be worth covering with end-to-end tests if they are already covered + in lower level tests. +- Finally, discuss the proposed test with the developer(s) involved in implementing + the feature and the lower-level tests. + +CAUTION: **Caution:** +Check both [GitLab Community Edition](https://gitlab-org.gitlab.io/gitlab-foss/coverage-ruby/#_AllFiles) and +[GitLab Enterprise Edition](https://gitlab-org.gitlab.io/gitlab/coverage-ruby/#_AllFiles) coverage projects +for previously-written tests for this feature. For analyzing the code coverage, +you must understand which application files implement specific features. + +NOTE: **Note:** +In this tutorial we're writing a login end-to-end test, even though it has been +sufficiently covered by lower-level testing, because it's the first step for most +end-to-end flows, and is easiest to understand. + +## Identify the DevOps stage + +The GitLab QA end-to-end tests are organized by the different +[stages in the DevOps lifecycle](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/qa/qa/specs/features/browser_ui). +Determine where the test should be placed by +[stage](https://about.gitlab.com/handbook/product/categories/#devops-stages), +determine which feature the test will belong to, and then place it in a subdirectory +under the stage. + + + +NOTE: **Note:** +If the test is Enterprise Edition only, the test will be created in the `features/ee` +directory, but follow the same DevOps lifecycle format. + +## Create a skeleton test + +In the first part of this tutorial we will be testing login, which is owned by the +Manage stage. Inside `qa/specs/features/browser_ui/1_manage/login`, create a +file `basic_login_spec.rb`. + +### The outer `context` block + +Specs have an outer `context` indicating the DevOps stage. + +```ruby +# frozen_string_literal: true + +module QA + context 'Manage' do + + end +end +``` + +### The `describe` block + +Inside of our outer `context`, describe the feature to test. In this case, `Login`. + +```ruby +# frozen_string_literal: true + +module QA + context 'Manage' do + describe 'Login' do + + end + end +end +``` + +### The `it` blocks (examples) + +Every test suite contains at least one `it` block (example). A good way to start +writing end-to-end tests is to write test case descriptions as `it` blocks: + +```ruby +module QA + context 'Manage' do + describe 'Login' do + it 'can login' do + + end + + it 'can logout' do + + end + end + end +end +``` + +## Write the test + +An important question is "What do we test?" and even more importantly, "How do we test?" + +Begin by logging in. + +```ruby +# frozen_string_literal: true + +module QA + context 'Manage' do + describe 'Login' do + it 'can login' do + Flow::Login.sign_in + + end + + it 'can logout' do + Flow::Login.sign_in + + end + end + end +end +``` + +After [running the spec](#run-the-spec), our test should login and end; then we +should answer the question "What do we test?" + +```ruby +# frozen_string_literal: true + +module QA + context 'Manage' do + describe 'Login' do + it 'can login' do + Flow::Login.sign_in + + Page::Main::Menu.perform do |menu| + expect(menu).to be_signed_in + end + end + + it 'can logout' do + Flow::Login.sign_in + + Page::Main::Menu.perform do |menu| + menu.sign_out + + expect(menu).not_to be_signed_in + end + end + end + end +end +``` + +**What do we test?** + +1. Can we log in? +1. Can we log out? + +**How do we test?** + +1. Check if the user avatar appears in the top navigation. +1. Check if the user avatar *does not* appear in the top navigation. + +NOTE: **Note:** +Behind the scenes, `be_signed_in` is a +[predicate matcher](https://relishapp.com/rspec/rspec-expectations/v/3-8/docs/built-in-matchers/predicate-matchers) +that [implements checking the user avatar](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/page/main/menu.rb#L74). + +## De-duplicate your code + +Refactor your test to use a `before` block for test setup, since it's duplicating +a call to `sign_in`. + +```ruby +# frozen_string_literal: true + +module QA + context 'Manage' do + describe 'Login' do + before do + Flow::Login.sign_in + end + + it 'can login' do + Page::Main::Menu.perform do |menu| + expect(menu).to be_signed_in + end + end + + it 'can logout' do + Page::Main::Menu.perform do |menu| + menu.sign_out + + expect(menu).not_to be_signed_in + end + end + end + end +end +``` + +The `before` block is essentially a `before(:each)` and is run before each example, +ensuring we now log in at the beginning of each test. + +## Test setup using resources and page objects + +Next, let's test something other than Login. Let's test Issues, which are owned by the Plan +stage, so [create a file](#identify-the-devops-stage) in +`qa/specs/features/browser_ui/3_create/issues` called `issues_spec.rb`. + +```ruby +# frozen_string_literal: true + +module QA + context 'Plan' do + describe 'Issues' do + let(:issue) do + Resource::Issue.fabricate_via_api! do |issue| + issue.title = 'My issue' + issue.description = 'This is an issue specific to this test' + end + end + + before do + Flow::Login.sign_in + issue.visit! + end + + it 'can close an issue' do + Page::Project::Issue::Show.perform do |show| + show.click_close_issue_button + + expect(show).to be_closed + end + end + end + end +end +``` + +Note the following important points: + +- At the start of our example, we will be at the `page/issue/show.rb` [page](page_objects.md). +- Our test fabricates only what it needs, when it needs it. +- The issue is fabricated through the API to save time. +- GitLab prefers `let()` over instance variables. See + [best practices](../best_practices.md#let-variables). +- `be_closed` is not implemented in `page/project/issue/show.rb` yet, but will be + implemented in the next step. + +The issue is fabricated as a [Resource](resources.md), which is a GitLab entity +you can create through the UI or API. Other examples include: + +- A [Merge Request](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/resource/merge_request.rb). +- A [User](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/resource/user.rb). +- A [Project](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/resource/project.rb). +- A [Group](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/resource/group.rb). + +## Write the page object + +A [Page Object](page_objects.md) is a class in our suite that represents a page +within GitLab. The **Login** page would be one example. Since our page object for +the **Issue Show** page already exists, add the `closed?` method. + +```ruby +module Page::Project::Issue + class Show + view 'app/views/projects/issues/show.html.haml' do + element :closed_status_box + end + + def closed? + has_element?(:closed_status_box) + end + end +end +``` + +Next, define the element `closed_status_box` within your view, so your Page Object +can see it. + +```haml +-#=> app/views/projects/issues/show.html.haml +.issuable-status-box.status-box.status-box-issue-closed{ ..., data: { qa_selector: 'closed_status_box' } } +``` + +## Run the spec + +Before running the spec, confirm: + +- The GDK is installed. +- The GDK is running on port 3000 locally. +- No additional [RSpec metadata tags](rspec_metadata_tests.md) have been applied. +- Your working directory is `qa/` within your GDK GitLab installation. + +To run the spec, run the following command: + +```ruby +bundle exec bin/qa Test::Instance::All http://localhost:3000 -- <test_file> +``` + +Where `<test_file>` is: + +- `qa/specs/features/browser_ui/1_manage/login/login_spec.rb` when running the Login example. +- `qa/specs/features/browser_ui/2_plan/issues/issue_spec.rb` when running the Issue example. diff --git a/doc/development/testing_guide/end_to_end/img/gl-devops-lifecycle-by-stage-numbers_V12_10.png b/doc/development/testing_guide/end_to_end/img/gl-devops-lifecycle-by-stage-numbers_V12_10.png Binary files differnew file mode 100644 index 00000000000..d11305c3686 --- /dev/null +++ b/doc/development/testing_guide/end_to_end/img/gl-devops-lifecycle-by-stage-numbers_V12_10.png diff --git a/doc/development/testing_guide/end_to_end/index.md b/doc/development/testing_guide/end_to_end/index.md index 443b7b06a24..e6a683e9148 100644 --- a/doc/development/testing_guide/end_to_end/index.md +++ b/doc/development/testing_guide/end_to_end/index.md @@ -180,7 +180,7 @@ instance-level scenarios](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/ Continued reading: -- [Quick Start Guide](quick_start_guide.md) +- [Beginner's Guide](beginners_guide.md) - [Style Guide](style_guide.md) - [Best Practices](best_practices.md) - [Testing with feature flags](feature_flags.md) diff --git a/doc/development/testing_guide/end_to_end/page_objects.md b/doc/development/testing_guide/end_to_end/page_objects.md index 22e7375be1f..9d4fa5316c4 100644 --- a/doc/development/testing_guide/end_to_end/page_objects.md +++ b/doc/development/testing_guide/end_to_end/page_objects.md @@ -238,6 +238,53 @@ the view for code in a library. In such rare cases it's reasonable to use CSS selectors in page object methods, with a comment explaining why an `element` can't be added. +### Define Page concerns + +Some pages share common behaviors, and/or are prepended with EE-specific modules that adds EE-specific methods. + +These modules must: + +1. Extend from the `QA::Page::PageConcern` module, with `extend QA::Page::PageConcern`. +1. Override the `self.prepended` method if they need to `include`/`prepend` other modules themselves, and/or define + `view` or `elements`. +1. Call `super` as the first thing in `self.prepended`. +1. Include/prepend other modules and define their `view`/`elements` in a `base.class_eval` block to ensure they're + defined in the class that prepends the module. + +These steps ensure the sanity selectors check will detect problems properly. + +For example, `qa/qa/ee/page/merge_request/show.rb` adds EE-specific methods to `qa/qa/page/merge_request/show.rb` (with +`QA::Page::MergeRequest::Show.prepend_if_ee('QA::EE::Page::MergeRequest::Show')`) and following is how it's implemented +(only showing the relevant part and refering to the 4 steps described above with inline comments): + +```ruby +module QA + module EE + module Page + module MergeRequest + module Show + extend QA::Page::PageConcern # 1. + + def self.prepended(base) # 2. + super # 3. + + base.class_eval do # 4. + prepend Page::Component::LicenseManagement + + view 'app/assets/javascripts/vue_merge_request_widget/components/states/sha_mismatch.vue' do + element :head_mismatch, "The source branch HEAD has recently changed." + end + + [...] + end + end + end + end + end + end +end +``` + ## Running the test locally During development, you can run the `qa:selectors` test by running diff --git a/doc/development/testing_guide/end_to_end/quick_start_guide.md b/doc/development/testing_guide/end_to_end/quick_start_guide.md deleted file mode 100644 index 0ae3f375284..00000000000 --- a/doc/development/testing_guide/end_to_end/quick_start_guide.md +++ /dev/null @@ -1,621 +0,0 @@ -# Writing end-to-end tests step-by-step - -In this tutorial, you will find different examples, and the steps involved, in the creation of end-to-end (_e2e_) tests for GitLab CE and GitLab EE, using GitLab QA. - -When referring to end-to-end tests in this document, this means testing a specific feature end-to-end such as: - -- A user logging in. -- The creation of a project. -- The management of labels. -- Breaking down epics into sub-epics and issues. - -## Important information before we start writing tests - -It's important to understand that end-to-end tests of isolated features, such as the ones described in the above note, doesn't mean that everything needs to happen through the GUI. - -If you don't exactly understand what we mean by **not everything needs to happen through the GUI,** please make sure you've read the [best practices](best_practices.md) before moving on. - -## This document covers the following items - -- [0.](#0-are-end-to-end-tests-needed) Identifying if end-to-end tests are really needed -- [1.](#1-identifying-the-devops-stage) Identifying the [DevOps stage](https://about.gitlab.com/stages-devops-lifecycle/) of the feature that you are going to cover with end-to-end tests -- [2.](#2-test-skeleton) Creating the skeleton of the test file (`*_spec.rb`) -- [3.](#3-test-cases-mvc) The [MVC](https://about.gitlab.com/handbook/values/#minimum-viable-change-mvc) of the test cases' logic -- [4.](#4-extracting-duplicated-code) Extracting duplicated code into methods -- [5.](#5-tests-pre-conditions-using-resources-and-page-objects) Tests' pre-conditions (`before :context` and `before`) using resources and [Page Objects](page_objects.md) -- [6.](#6-optimization) Optimizing the test suite -- [7.](#7-resources) Using and implementing resources -- [8.](#8-page-objects) Moving element definitions and methods to [Page Objects](page_objects.md) - -### 0. Are end-to-end tests needed? - -At GitLab we respect the [test pyramid](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/development/testing_guide/testing_levels.md), and so, we recommend you check the code coverage of a specific feature before writing end-to-end tests, for both [CE](https://gitlab-org.gitlab.io/gitlab-foss/coverage-ruby/#_AllFiles) and [EE](https://gitlab-org.gitlab.io/gitlab/coverage-ruby/#_AllFiles) projects. - -Sometimes you may notice that there is already good coverage in lower test levels, and we can stay confident that if we break a feature, we will still have quick feedback about it, even without having end-to-end tests. - -> For analyzing the code coverage, you will also need to understand which application files implement specific functionalities. - -#### Some other guidelines are as follows - -- Take a look at the [How to test at the correct level?](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/development/testing_guide/testing_levels.md#how-to-test-at-the-correct-level) section of the [Testing levels](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/development/testing_guide/testing_levels.md) document - -- Look into the frequency in which such a feature is changed (_Stable features that don't change very often might not be worth covering with end-to-end tests if they're already covered in lower levels_) - -- Finally, discuss with the developer(s) involved in developing the feature and the tests themselves, to get their feeling - -If after this analysis you still think that end-to-end tests are needed, keep reading. - -### 1. Identifying the DevOps stage - -The GitLab QA end-to-end tests are organized by the different [stages in the DevOps lifecycle](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/qa/qa/specs/features/browser_ui), and so, if you are creating tests for issue creation, for instance, you would locate the spec files under the `qa/qa/specs/features/browser_ui/2_plan/` directory since issue creation is part of the Plan stage. - - In another case of a test for listing merged merge requests (MRs), the test should go under the `qa/qa/specs/features/browser_ui/3_create/` directory since merge requests are a feature from the Create stage. - -> There may be sub-directories inside the stages directories, for different features. For example: `.../browser_ui/2_plan/ee_epics/` and `.../browser_ui/2_plan/issues/`. - -Now, let's say we want to create tests for the [scoped labels](https://about.gitlab.com/releases/2019/04/22/gitlab-11-10-released/#scoped-labels) feature, available on GitLab EE Premium (this feature is part of the Plan stage.) - -> Because these tests are for a feature available only on GitLab EE, we need to create them in the [EE repository](https://gitlab.com/gitlab-org/gitlab). - -Since [there is no specific directory for this feature](https://gitlab.com/gitlab-org/gitlab/tree/master/qa/qa/specs/features/browser_ui/2_plan), we should create a sub-directory for it. - -Under `.../browser_ui/2_plan/`, let's create a sub-directory called `ee_scoped_labels/`. - -> Notice that since this feature is only available for GitLab EE we prefix the sub-directory with `ee_`. - -### 2. Test skeleton - -Inside the newly created sub-directory, let's create a file describing the test suite (e.g. `editing_scoped_labels_spec.rb`.) - -#### The `context` and `describe` blocks - -Specs have an outer `context` that indicates the DevOps stage. The next level is the `describe` block, that briefly states the subject of the test suite. See the following example: - -```ruby -module QA - context 'Plan' do - describe 'Editing scoped labels on issues' do - end - end -end -``` - -#### The `it` blocks - -Every test suite is composed of at least one `it` block, and a good way to start writing end-to-end tests is by writing test cases descriptions as `it` blocks. These might help you to think of different test scenarios. Take a look at the following example: - -```ruby -module QA - context 'Plan' do - describe 'Editing scoped labels on issues' do - it 'replaces an existing label if it has the same key' do - end - - it 'keeps both scoped labels when adding a label with a different key' do - end - end - end -end -``` - -### 3. Test cases MVC - -For the [MVC](https://about.gitlab.com/handbook/values/#minimum-viable-change-mvc) of our test cases, let's say that we already have the application in the state needed for the tests, and then let's focus on the logic of the test cases only. - -To evolve the test cases drafted on step 2, let's imagine that the user is already logged into a GitLab EE instance, they already have at least a Premium license in use, there is already a project created, there is already an issue opened in the project, the issue already has a scoped label (e.g. `animal::fox`), there are other scoped labels (for the same scope and for a different scope (e.g. `animal::dolphin` and `plant::orchid`), and finally, the user is already on the issue's page. Let's also suppose that for every test case the application is in a clean state, meaning that one test case won't affect another. - -> Note: there are different approaches to creating an application state for end-to-end tests. Some of them are very time consuming and subject to failures, such as when using the GUI for all the pre-conditions of the tests. On the other hand, other approaches are more efficient, such as using the public APIs. The latter is more efficient since it doesn't depend on the GUI. We won't focus on this part yet, but it's good to keep it in mind. - -Let's now focus on the first test case. - -```ruby -it 'replaces an existing label if it has the same key' do - # This implementation is only for tutorial purposes. We normally encapsulate elements in Page Objects (which we cover on section 8). - page.find('.block.labels .edit-link').click - page.find('.dropdown-menu-labels .dropdown-input-field').send_keys ['animal::dolphin', :enter] - page.find('#content-body').click - page.refresh - - labels_block = page.find(%q([data-qa-selector="labels_block"])) - - expect(labels_block).to have_content('animal::dolphin') - expect(labels_block).not_to have_content('animal::fox') - expect(page).to have_content('added animal::dolphin label and removed animal::fox') -end -``` - -> Notice that the test itself is simple. The most challenging part is the creation of the application state, which will be covered later. -> -> The exemplified test case's MVC is not enough for the change to be merged, but it helps to build up the test logic. The reason is that we do not want to use locators directly in the tests, and tests **must** use [Page Objects](page_objects.md) before they can be merged. This way we better separate the responsibilities, where the Page Objects encapsulate elements and methods that allow us to interact with pages, while the spec files describe the test cases in more business-related language. - -Below are the steps that the test covers: - -1. The test finds the 'Edit' link for the labels and clicks on it. -1. Then it fills in the 'Assign labels' input field with the value 'animal::dolphin' and press enters. -1. Then it clicks in the content body to apply the label and refreshes the page. -1. Finally, the expectations check that the previous scoped label was removed and that the new one was added. - -Let's now see how the second test case would look. - -```ruby -it 'keeps both scoped labels when adding a label with a different key' do - # This implementation is only for tutorial purposes. We normally encapsulate elements in Page Objects (which we cover on section 8). - page.find('.block.labels .edit-link').click - page.find('.dropdown-menu-labels .dropdown-input-field').send_keys ['plant::orchid', :enter] - page.find('#content-body').click - page.refresh - - labels_block = page.find(%q([data-qa-selector="labels_block"])) - - expect(labels_block).to have_content('animal::fox') - expect(labels_block).to have_content('plant::orchid') - expect(page).to have_content('added animal::fox') - expect(page).to have_content('added plant::orchid') -end -``` - -> Note that elements are always located using CSS selectors, and a good practice is to add test-specific selectors (this is called "testability"). For example, the `labels_block` element uses the CSS selector [`data-qa-selector="labels_block"`](page_objects.md#data-qa-selector-vs-qa-selector), which was added specifically for testing purposes. - -Below are the steps that the test covers: - -1. The test finds the 'Edit' link for the labels and clicks on it. -1. Then it fills in the 'Assign labels' input field with the value 'plant::orchid' and press enters. -1. Then it clicks in the content body to apply the label and refreshes the page. -1. Finally, the expectations check that both scoped labels are present. - -> Similar to the previous test, this one is also very straightforward, but there is some code duplication. Let's address it. - -### 4. Extracting duplicated code - -If we refactor the tests created on step 3 we could come up with something like this: - -```ruby -before do - ... - - @initial_label = 'animal::fox' - @new_label_same_scope = 'animal::dolphin' - @new_label_different_scope = 'plant::orchid' - - ... -end - -it 'replaces an existing label if it has the same key' do - select_label_and_refresh @new_label_same_scope - - labels_block = page.find(%q([data-qa-selector="labels_block"])) - - expect(labels_block).to have_content(@new_label_same_scope) - expect(labels_block).not_to have_content(@initial_label) - expect(page).to have_content("added #{@new_label_same_scope}") - expect(page).to have_content("and removed #{@initial_label}") -end - -it 'keeps both scoped label when adding a label with a different key' do - select_label_and_refresh @new_label_different_scope - - labels_block = page.find(%q([data-qa-selector="labels_block"])) - - expect(labels_blocks).to have_content(@new_label_different_scope) - expect(labels_blocks).to have_content(@initial_label) - expect(page).to have_content("added #{@new_label_different_scope}") - expect(page).to have_content("added #{@initial_label}") -end - -def select_label_and_refresh(label) - page.find('.block.labels .edit-link').click - page.find('.dropdown-menu-labels .dropdown-input-field').send_keys [label, :enter] - page.find('#content-body').click - page.refresh -end -``` - -First, we remove the duplication of strings by defining the global variables `@initial_label`, `@new_label_same_scope` and `@new_label_different_scope` in the `before` block, and by using them in the expectations. - -Then, by creating a reusable `select_label_and_refresh` method, we remove the code duplication of this action, and later we can move this method to a Page Object class that will be created for easier maintenance purposes. - -Notice that the reusable method is created at the bottom of the file. This helps readability, -where reading the code should be similar to reading a newspaper: - -- High-level information is at the top, like the title and summary of the news. -- Low level, or more specific information, is at the bottom. - -### 5. Tests' pre-conditions using resources and Page Objects - -In this section, we will address the previously mentioned subject of creating the application state for the tests, using the `before :context` and `before` blocks, together with resources and Page Objects. - -#### `before :context` - -A pre-condition for the entire test suite is defined in the `before :context` block. - -> For our test suite, due to the need of the tests being completely independent of each other, we won't use the `before :context` block. The `before :context` block would make the tests dependent on each other because the first test changes the label of the issue, and the second one depends on the `'animal::fox'` label being set. - -TIP: **Tip:** In case of a test suite with only one `it` block it's ok to use only the `before` block (see below) with all the test's pre-conditions. - -#### `before` - -As the pre-conditions for our test suite, the things that needs to happen before each test starts are: - -- The user logging in; -- A premium license already being set; -- A project being created with an issue and labels already set; -- The issue page being opened with only one scoped label applied to it. - -> When running end-to-end tests as part of the GitLab's continuous integration process [a license is already set as an environment variable](https://gitlab.com/gitlab-org/gitlab/blob/1a60d926740db10e3b5724713285780a4f470531/qa/qa/ee/strategy.rb#L20). For running tests locally you can set up such license by following the document [what tests can be run?](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md), based on the [supported GitLab environment variables](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#supported-gitlab-environment-variables). - -#### Implementation - -In the following code we will focus only on the test suite's pre-conditions: - -```ruby -module QA - context 'Plan' do - describe 'Editing scoped labels on issues' do - before do - Runtime::Browser.visit(:gitlab, Page::Main::Login) - Page::Main::Login.perform(&:sign_in_using_credentials) - - @initial_label = 'animal::fox' - @new_label_same_scope = 'animal::dolphin' - @new_label_different_scope = 'plant::orchid' - - issue = Resource::Issue.fabricate_via_api! do |issue| - issue.title = 'Issue to test the scoped labels' - issue.labels = [@initial_label] - end - - [@new_label_same_scope, @new_label_different_scope].each do |label| - Resource::Label.fabricate_via_api! do |l| - l.project = issue.project - l.title = label - end - end - - issue.visit! - end - - it 'replaces an existing label if it has the same key' do - ... - end - - it 'keeps both scoped labels when adding a label with a different key' do - ... - end - - def select_label_and_refresh(label) - ... - end - end - end -end -``` - -In the `before` block we create all the application state needed for the tests to run. We do that by using the `Runtime::Browser.visit` method to go to the login page, by performing a `sign_in_using_credentials` from the `Login` Page Object, by fabricating resources via APIs (`issue`, and `Resource::Label`), and by using the `issue.visit!` to visit the issue page. - -> A project is created in the background by creating the `issue` resource. -> -> When creating the [Resources](resources.md), notice that when calling the `fabricate_via_api` method, we pass some attribute:values, like `title`, and `labels` for the `issue` resource; and `project` and `title` for the `label` resource. -> -> What's important to understand here is that by creating the application state mostly using the public APIs we save a lot of time in the test suite setup stage. -> -> Soon we will cover the use of the already existing resources' methods and the creation of your own `fabricate_via_api` methods for resources where this is still not available, but first, let's optimize our implementation. - -### 6. Optimization - -As already mentioned in the [best practices](best_practices.md) document, end-to-end tests are very costly in terms of execution time, and it's our responsibility as software engineers to ensure that we optimize them as much as possible. - -> Note that end-to-end tests are slow to run and so they can have several actions and assertions in a single test, which helps us get feedback from the tests sooner. In comparison, unit tests are much faster to run and can exercise every little piece of the application in isolation, and so they usually have only one assertion per test. - -Some improvements that we could make in our test suite to optimize its time to run are: - -1. Having a single test case (an `it` block) that exercises both scenarios to avoid "wasting" time in the tests' pre-conditions, instead of having two different test cases. -1. Making the selection of labels more performant by allowing for the selection of more than one label in the same reusable method. - -Let's look at a suggestion that addresses the above points, one by one: - -```ruby -module QA - context 'Plan' do - describe 'Editing scoped labels on issues' do - before do - ... - end - - it 'correctly applies scoped labels depending on if they are from the same or a different scope' do - select_labels_and_refresh [@new_label_same_scope, @new_label_different_scope] - - labels_block = page.all(%q([data-qa-selector="labels_block"])) - - expect(labels_block).to have_content(@new_label_same_scope) - expect(labels_block).to have_content(@new_label_different_scope) - expect(labels_block).not_to have_content(@initial_label) - expect(page).to have_content("added #{@initial_label}") - expect(page).to have_content("added #{@new_label_same_scope} #{@new_label_different_scope} labels and removed #{@initial_label}") - end - - def select_labels_and_refresh(labels) - find('.block.labels .edit-link').click - labels.each do |label| - find('.dropdown-menu-labels .dropdown-input-field').send_keys [label, :enter] - end - find('#content-body').click - refresh - end - end - end - end -``` - -To address point 1, we changed the test implementation from two `it` blocks into a single one that exercises both scenarios. Now the new test description is: `'correctly applies the scoped labels depending if they are from the same or a different scope'`. It's a long description, but it describes well what the test does. - -> Notice that the implementation of the new and unique `it` block had to change a little bit. Below we describe in details what it does. - -1. It selects two scoped labels simultaneously, one from the same scope of the one already applied in the issue during the setup phase (in the `before` block), and another one from a different scope. -1. It asserts that the correct labels are visible in the `labels_block`, and that the labels were correctly added and removed; -1. Finally, the `select_label_and_refresh` method is changed to `select_labels_and_refresh`, which accepts an array of labels instead of a single label, and it iterates on them for faster label selection (this is what is used in step 1 explained above.) - -### 7. Resources - -**Note:** When writing this document, some code that is now merged to master was not implemented yet, but we left them here for the readers to understand the whole process of end-to-end test creation. - -You can think of [Resources](resources.md) as anything that can be created on GitLab CE or EE, either through the GUI, the API, or the CLI. - -With that in mind, resources can be a project, an epic, an issue, a label, a commit, etc. - -As you saw in the tests' pre-conditions and the optimization sections, we're already creating some of these resources. We are doing that by calling the `fabricate_via_api!` method. - -> We could be using the `fabricate!` method instead, which would use the `fabricate_via_api!` method if it exists, and fallback to GUI fabrication otherwise, but we recommend being explicit to make it clear what the test does. Also, we always recommend fabricating resources via API since this makes tests faster and more reliable. - -For our test suite example, the resources that we need to create don't have the necessary code for the `fabricate_via_api!` method to correctly work (e.g., the issue and label resources), so we will have to create them. - -#### Implementation - -In the following we describe the changes needed in each of the resource files mentioned above. - -**Issue resource** - -Now, let's make it possible to create an issue resource through the API. - -First, in the [issue resource](https://gitlab.com/gitlab-org/gitlab/blob/d3584e80b4236acdf393d815d604801573af72cc/qa/qa/resource/issue.rb), let's expose its id and labels attributes. - -Add the following `attribute :id` and `attribute :labels` right above the [`attribute :title`](https://gitlab.com/gitlab-org/gitlab/blob/d3584e80b4236acdf393d815d604801573af72cc/qa/qa/resource/issue.rb#L15). - -> This line is needed to allow for the issue fabrication, and for labels to be automatically added to the issue when fabricating it via API. -> -> We add the attributes above the existing attribute to keep them alphabetically organized. - -Then, let's initialize an instance variable for labels to allow an empty array as default value when such information is not passed during the resource fabrication, since this optional. [Between the attributes and the `fabricate!` method](https://gitlab.com/gitlab-org/gitlab/blob/1a1f1408728f19b2aa15887cd20bddab7e70c8bd/qa/qa/resource/issue.rb#L18), add the following: - -```ruby -def initialize - @labels = [] -end -``` - -Next, add the following code right below the [`fabricate!`](https://gitlab.com/gitlab-org/gitlab/blob/d3584e80b4236acdf393d815d604801573af72cc/qa/qa/resource/issue.rb#L27) method. - -```ruby -def api_get_path - "/projects/#{project.id}/issues/#{id}" -end - -def api_post_path - "/projects/#{project.id}/issues" -end - -def api_post_body - { - labels: labels, - title: title - } -end -``` - -By defining the `api_get_path` method, we allow the [`ApiFabricator`](https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/resource/api_fabricator.rb) module to know which path to use to get a single issue. - -> This `GET` path can be found in the [public API documentation](../../../api/issues.md#single-issue). - -By defining the `api_post_path` method, we allow the [`ApiFabricator`](https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/resource/api_fabricator.rb) module to know which path to use to create a new issue in a specific project. - -> This `POST` path can be found in the [public API documentation](../../../api/issues.md#new-issue). - -By defining the `api_post_body` method, we allow the [`ApiFabricator.api_post`](https://gitlab.com/gitlab-org/gitlab/blob/a9177ca1812bac57e2b2fa4560e1d5dd8ffac38b/qa/qa/resource/api_fabricator.rb#L68) method to know which data to send when making the `POST` request. - -> Notice that we pass both `labels` and `title` attributes in the `api_post_body`, where `labels` receives an array of labels, and [`title` is required](../../../api/issues.md#new-issue). Also, notice that we keep them alphabetically organized. - -**Label resource** - -Finally, let's make it possible to create label resources through the API. - -Add the following code right below the [`fabricate!`](https://gitlab.com/gitlab-org/gitlab/blob/a9177ca1812bac57e2b2fa4560e1d5dd8ffac38b/qa/qa/resource/label.rb#L36) method. - -```ruby -def resource_web_url(resource) - super -rescue ResourceURLMissingError - # this particular resource does not expose a web_url property -end - -def api_get_path - raise NotImplementedError, "The Labels API doesn't expose a single-resource endpoint so this method cannot be properly implemented." -end - -def api_post_path - "/projects/#{project.id}/labels" -end - -def api_post_body - { - color: @color, - name: @title - } -end -``` - -By defining the `resource_web_url(resource)` method, we override the one from the [`ApiFabricator`](https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/resource/api_fabricator.rb#L44) module. We do that to avoid failing the test due to this particular resource not exposing a `web_url` property. - -By defining the `api_get_path` method, we **would** allow for the [`ApiFabricator`](https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/resource/api_fabricator.rb) module to know which path to use to get a single label, but since there's no path available for that in the public API, we raise a `NotImplementedError` instead. - -By defining the `api_post_path` method, we allow for the [`ApiFabricator`](https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/resource/api_fabricator.rb) module to know which path to use to create a new label in a specific project. - -By defining the `api_post_body` method, we allow for the [`ApiFabricator.api_post`](https://gitlab.com/gitlab-org/gitlab/blob/a9177ca1812bac57e2b2fa4560e1d5dd8ffac38b/qa/qa/resource/api_fabricator.rb#L68) method to know which data to send when making the `POST` request. - -> Notice that we pass both `color` and `name` attributes in the `api_post_body` since [those are required](../../../api/labels.md#create-a-new-label). Also, notice that we keep them alphabetically organized. - -### 8. Page Objects - -Page Objects are used in end-to-end tests for maintenance reasons, where a page's elements and methods are defined to be reused in any test. - -> Page Objects are auto-loaded in the [`qa/qa.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa.rb) file and available in all the test files (`*_spec.rb`). - -Take a look at the [Page Objects](page_objects.md) documentation. - -Now, let's go back to our example. - -As you may have noticed, we are defining elements with CSS selectors and the `select_labels_and_refresh` method directly in the test file, and this is an anti-pattern since we need to better separate the responsibilities. - -To address this issue, we will move the implementation to Page Objects, and the test suite will only focus on the business rules that we are testing. - -#### Updates in the test file - -As in a test-driven development approach, let's start changing the test file even before the Page Object implementation is in place. - -Replace the code of the `it` block in the test file by the following: - -```ruby -module QA - context 'Plan' do - describe 'Editing scoped labels on issues' do - before do - ... - end - - it 'correctly applies scoped labels depending on if they are from the same or a different scope' do - Page::Project::Issue::Show.perform do |issue_page| - issue_page.select_labels_and_refresh [@new_label_same_scope, @new_label_different_scope] - - expect(page).to have_content("added #{@initial_label}") - expect(page).to have_content("added #{@new_label_same_scope} #{@new_label_different_scope} labels and removed #{@initial_label}") - expect(issue_page.text_of_labels_block).to have_content(@new_label_same_scope) - expect(issue_page.text_of_labels_block).to have_content(@new_label_different_scope) - expect(issue_page.text_of_labels_block).not_to have_content(@initial_label) - end - end - end - end -end -``` - -Notice that `select_labels_and_refresh` is now a method from the issue Page Object (which is not yet implemented), and that we verify the labels' text by using `text_of_labels_block`, instead of via the `labels_block` element. The `text_of_labels_block` method will also be implemented in the issue Page Object. - -Let's now update the Issue Page Object. - -#### Updates in the Issue Page Object - -> Page Objects are located in the `qa/qa/page/` directory, and its sub-directories. - -The file we will have to change is the [Issue Page Object](https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/page/project/issue/show.rb). - -First, add the following code right below the definition of an already implemented view (keep in mind that view's definitions and their elements should be alphabetically ordered): - -```ruby -view 'app/helpers/dropdowns_helper.rb' do - element :dropdown_input_field -end - -view 'app/views/shared/issuable/_sidebar.html.haml' do - element :dropdown_menu_labels - element :edit_link_labels - element :labels_block -end -``` - -Similarly to what we did before, let's first change the Page Object even without the elements being defined in the view (`_sidebar.html.haml`) and the `dropdowns_helper.rb` files, and later we will update them by adding the appropriate CSS selectors. - -Now, let's implement the methods `select_labels_and_refresh` and `text_of_labels_block`. - -Somewhere between the definition of the views and the private methods, add the following snippet of code (these should also be alphabetically ordered for organization reasons): - -```ruby -def select_labels_and_refresh(labels) - click_element(:edit_link_labels) - labels.each do |label| - within_element(:dropdown_menu_labels, text: label) do - send_keys_to_element(:dropdown_input_field, [label, :enter]) - end - end - click_body - labels.each do |label| - has_element?(:labels_block, text: label) - end - refresh -end - -def text_of_labels_block - find_element(:labels_block) -end -``` - -##### Details of `select_labels_and_refresh` - -Notice that we have not only moved the `select_labels_and_refresh` method, but we have also changed its implementation to: - -1. Click the `:edit_link_labels` element previously defined, instead of using `find('.block.labels .edit-link').click` -1. Use `within_element(:dropdown_menu_labels, text: label)`, and inside of it, we call `send_keys_to_element(:dropdown_input_field, [label, :enter])`, which is a method that we will implement in the `QA::Page::Base` class to replace `find('.dropdown-menu-labels .dropdown-input-field').send_keys [label, :enter]` -1. Use `click_body` after iterating on each label, instead of using `find('#content-body').click` -1. Iterate on every label again, and then we use `has_element?(:labels_block, text: label)` after clicking the page body (which applies the labels), and before refreshing the page, to avoid test flakiness due to refreshing too fast. - -##### Details of `text_of_labels_block` - -The `text_of_labels_block` method is a simple method that returns the `:labels_block` element (`find_element(:labels_block)`). - -#### Updates in the view (*.html.haml) and `dropdowns_helper.rb` files - -Now let's change the view and the `dropdowns_helper` files to add the selectors that relate to the [Page Objects](page_objects.md). - -In [`app/views/shared/issuable/_sidebar.html.haml:105`](https://gitlab.com/gitlab-org/gitlab/blob/7ca12defc7a965987b162a6ebef302f95dc8867f/app/views/shared/issuable/_sidebar.html.haml#L105), add a `data: { qa_selector: 'edit_link_labels' }` data attribute. - -The code should look like this: - -```haml -= link_to _('Edit'), '#', class: 'js-sidebar-dropdown-toggle edit-link float-right', data: { qa_selector: 'edit_link_labels' } -``` - -In the same file, on [line 121](https://gitlab.com/gitlab-org/gitlab/blob/7ca12defc7a965987b162a6ebef302f95dc8867f/app/views/shared/issuable/_sidebar.html.haml#L121), add a `data: { qa_selector: 'dropdown_menu_labels' }` data attribute. - -The code should look like this: - -```haml -.dropdown-menu.dropdown-select.dropdown-menu-paging.dropdown-menu-labels.dropdown-menu-selectable.dropdown-extended-height{ data: { qa_selector: 'dropdown_menu_labels' } } -``` - -In [`app/helpers/dropdowns_helper.rb:94`](https://gitlab.com/gitlab-org/gitlab/blob/7ca12defc7a965987b162a6ebef302f95dc8867f/app/helpers/dropdowns_helper.rb#L94), add a `data: { qa_selector: 'dropdown_input_field' }` data attribute. - -The code should look like this: - -```ruby -filter_output = search_field_tag search_id, nil, class: "dropdown-input-field", placeholder: placeholder, autocomplete: 'off', data: { qa_selector: 'dropdown_input_field' } -``` - -> `data-qa-*` data attributes and CSS classes starting with `qa-` are used solely for the purpose of QA and testing. -> By defining these, we add **testability** to the application. -> -> When defining a data attribute like: `qa_selector: 'labels_block'`, it should match the element definition: `element :labels_block`. We use a [sanity test](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/doc/development/testing_guide/end_to_end/page_objects.md#how-did-we-solve-fragile-tests-problem) to check that defined elements have their respective selectors in the specified views. - -#### Updates in the `QA::Page::Base` class - -The last thing that we have to do is to update `QA::Page::Base` class to add the `send_keys_to_element` method on it. - -Add the following snippet of code somewhere where class methods are defined (remember to organize methods alphabetically, and if you see a place where this standard is not being followed, it would be helpful if you could rearrange it): - -```ruby -def send_keys_to_element(name, keys) - find_element(name).send_keys(keys) -end -``` - -This method receives an element (`name`) and the `keys` that it will send to that element, and the keys are an array that can receive strings, or "special" keys, like `:enter`. - -As you might remember, in the Issue Page Object we call this method like this: `send_keys_to_element(:dropdown_input_field, [label, :enter])`. - -With that, you should be able to start writing end-to-end tests yourself. *Congratulations!* diff --git a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md index f8dc3366904..b7c93d205a3 100644 --- a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md +++ b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md @@ -14,3 +14,5 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec | `:reliable` | The test has been [promoted to a reliable test](https://about.gitlab.com/handbook/engineering/quality/guidelines/reliable-tests/#promoting-an-existing-test-to-reliable) meaning it passes consistently in all pipelines, including merge requests. | | `:requires_admin` | The test requires an admin account. Tests with the tag are excluded when run against Canary and Production environments. | | `:runner` | The test depends on and will set up a GitLab Runner instance, typically to run a pipeline. | +| `:gitaly_ha` | The test will run against a GitLab instance where repositories are stored on redundant Gitaly nodes behind a Praefect node. All nodes are [separate containers](../../../administration/gitaly/praefect.md#requirements-for-configuring-a-gitaly-cluster). Tests that use this tag have a longer setup time since there are three additional containers that need to be started. | +| `:skip_live_env` | The test will be excluded when run against live deployed environments such as Staging, Canary, and Production. | diff --git a/doc/development/testing_guide/end_to_end/style_guide.md b/doc/development/testing_guide/end_to_end/style_guide.md index 7f4616f394b..9c02af12d5d 100644 --- a/doc/development/testing_guide/end_to_end/style_guide.md +++ b/doc/development/testing_guide/end_to_end/style_guide.md @@ -66,6 +66,7 @@ We follow a simple formula roughly based on hungarian notation. - `_placeholder`: a temporary element that appears while content is loading. For example, the elements that are displayed instead of discussions while the discussions are being fetched. - `_radio` - `_tab` + - `_menu_item` *Note: If none of the listed types are suitable, please open a merge request to add an appropriate type to the list.* diff --git a/doc/development/testing_guide/flaky_tests.md b/doc/development/testing_guide/flaky_tests.md index 1e53e92fad5..6c1b06ce59a 100644 --- a/doc/development/testing_guide/flaky_tests.md +++ b/doc/development/testing_guide/flaky_tests.md @@ -40,7 +40,7 @@ Quarantined tests are run on the CI in dedicated jobs that are allowed to fail: ## Automatic retries and flaky tests detection -On our CI, we use [rspec-retry](https://github.com/NoRedInk/rspec-retry) to automatically retry a failing example a few +On our CI, we use [RSpec::Retry](https://github.com/NoRedInk/rspec-retry) to automatically retry a failing example a few times (see [`spec/spec_helper.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/spec/spec_helper.rb) for the precise retries count). We also use a home-made `RspecFlaky::Listener` listener which records flaky diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md index 2a3fcf122a6..52d538c7159 100644 --- a/doc/development/testing_guide/frontend_testing.md +++ b/doc/development/testing_guide/frontend_testing.md @@ -5,7 +5,7 @@ at GitLab. We use Karma with Jasmine and Jest for JavaScript unit and integratio and RSpec feature tests with Capybara for e2e (end-to-end) integration testing. Unit and feature tests need to be written for all new features. -Most of the time, you should use [RSpec] for your feature tests. +Most of the time, you should use [RSpec](https://github.com/rspec/rspec-rails#feature-specs) for your feature tests. Regression tests should be written for bug fixes to prevent them from recurring in the future. @@ -15,7 +15,7 @@ information on general testing practices at GitLab. ## Vue.js testing -If you are looking for a guide on Vue component testing, you can jump right away to this [section][vue-test]. +If you are looking for a guide on Vue component testing, you can jump right away to this [section](../fe_guide/vue.md#testing-vue-components). ## Jest @@ -30,7 +30,7 @@ Jest tests can be found in `/spec/frontend` and `/ee/spec/frontend` in EE. ## Karma test suite -While GitLab is switching over to [Jest][jest] you'll still find Karma tests in our application. [Karma][karma] is a test runner which uses [Jasmine] as its test +While GitLab is switching over to [Jest](https://jestjs.io) you'll still find Karma tests in our application. [Karma](http://karma-runner.github.io/) is a test runner which uses [Jasmine](https://jasmine.github.io/) as its test framework. Jest also uses Jasmine as foundation, that's why it's looking quite similar. Karma tests live in `spec/javascripts/` and `/ee/spec/javascripts` in EE. @@ -549,7 +549,8 @@ TBU Jasmine provides stubbing and mocking capabilities. There are some subtle differences in how to use it within Karma and Jest. -Stubs or spies are often used synonymously. In Jest it's quite easy thanks to the `.spyOn` method. [Official docs][jestspy] +Stubs or spies are often used synonymously. In Jest it's quite easy thanks to the `.spyOn` method. +[Official docs](https://jestjs.io/docs/en/jest-object#jestspyonobject-methodname) The more challenging part are mocks, which can be used for functions or even dependencies. ### Manual module mocks @@ -637,12 +638,12 @@ Karma allows something similar, but it's way more costly. Running Karma with `yarn run karma-start` will compile the JavaScript assets and run a server at `http://localhost:9876/` where it will automatically -run the tests on any browser which connects to it. You can enter that url on +run the tests on any browser which connects to it. You can enter that URL on multiple browsers at once to have it run the tests on each in parallel. While Karma is running, any changes you make will instantly trigger a recompile and retest of the **entire test suite**, so you can see instantly if you've broken -a test with your changes. You can use [Jasmine focused][jasmine-focus] or +a test with your changes. You can use [Jasmine focused](https://jasmine.github.io/2.5/focused_specs.html) or excluded tests (with `fdescribe` or `xdescribe`) to get Karma to run only the tests you want while you're working on a specific feature, but make sure to remove these directives when you commit your code. @@ -831,43 +832,6 @@ testAction( Check an example in [spec/javascripts/ide/stores/actions_spec.jsspec/javascripts/ide/stores/actions_spec.js](https://gitlab.com/gitlab-org/gitlab/blob/master/spec/javascripts/ide/stores/actions_spec.js). -### Vue Helper: `mountComponent` - -To make mounting a Vue component easier and more readable, we have a few helpers available in `spec/helpers/vue_mount_component_helper`: - -- `createComponentWithStore` -- `mountComponentWithStore` - -Examples of usage: - -```javascript -beforeEach(() => { - vm = createComponentWithStore(Component, store); - - vm.$store.state.currentBranchId = 'master'; - - vm.$mount(); -}); -``` - -```javascript -beforeEach(() => { - vm = mountComponentWithStore(Component, { - el: '#dummy-element', - store, - props: { badge }, - }); -}); -``` - -Don't forget to clean up: - -```javascript -afterEach(() => { - vm.$destroy(); -}); -``` - ### Wait until axios requests finish The axios utils mock module located in `spec/frontend/mocks/ce/lib/utils/axios_utils.js` contains two helper methods for Jest tests that spawn HTTP requests. @@ -906,13 +870,3 @@ You can download any older version of Firefox from the releases FTP server, <htt --- [Return to Testing documentation](index.md) - -<!-- URL References --> - -[jasmine-focus]: https://jasmine.github.io/2.5/focused_specs.html -[karma]: http://karma-runner.github.io/ -[vue-test]: ../fe_guide/vue.md#testing-vue-components -[rspec]: https://github.com/rspec/rspec-rails#feature-specs -[jasmine]: https://jasmine.github.io/ -[jest]: https://jestjs.io -[jestspy]: https://jestjs.io/docs/en/jest-object#jestspyonobject-methodname diff --git a/doc/development/testing_guide/index.md b/doc/development/testing_guide/index.md index 25da72b6b6a..f0fb06910f8 100644 --- a/doc/development/testing_guide/index.md +++ b/doc/development/testing_guide/index.md @@ -4,15 +4,15 @@ This document describes various guidelines and best practices for automated testing of the GitLab project. It is meant to be an _extension_ of the [thoughtbot testing -styleguide](https://github.com/thoughtbot/guides/tree/master/style/testing). If +style guide](https://github.com/thoughtbot/guides/tree/master/style/testing). If this guide defines a rule that contradicts the thoughtbot guide, this guide takes precedence. Some guidelines may be repeated verbatim to stress their importance. ## Overview -GitLab is built on top of [Ruby on Rails](https://rubyonrails.org/), and we're using [RSpec] for all -the backend tests, with [Capybara] for end-to-end integration testing. +GitLab is built on top of [Ruby on Rails](https://rubyonrails.org/), and we're using [RSpec](https://github.com/rspec/rspec-rails#feature-specs) for all +the backend tests, with [Capybara](https://github.com/teamcapybara/capybara) for end-to-end integration testing. On the frontend side, we're using [Jest](https://jestjs.io/) and [Karma](http://karma-runner.github.io/)/[Jasmine](https://jasmine.github.io/) for JavaScript unit and integration testing. @@ -58,14 +58,10 @@ Everything you should know about how to test Rake tasks. ## [End-to-end tests](end_to_end/index.md) Everything you should know about how to run end-to-end tests using -[GitLab QA][gitlab-qa] testing framework. +[GitLab QA](ttps://gitlab.com/gitlab-org/gitlab-qa) testing framework. ## [Migrations tests](testing_migrations_guide.md) Everything you should know about how to test migrations. [Return to Development documentation](../README.md) - -[RSpec]: https://github.com/rspec/rspec-rails#feature-specs -[Capybara]: https://github.com/teamcapybara/capybara -[gitlab-qa]: https://gitlab.com/gitlab-org/gitlab-qa diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md index 9eb5d5add8a..58acf937d77 100644 --- a/doc/development/testing_guide/review_apps.md +++ b/doc/development/testing_guide/review_apps.md @@ -102,10 +102,10 @@ subgraph "CNG-mirror pipeline" ### Auto-stopping of Review Apps Review Apps are automatically stopped 2 days after the last deployment thanks to -the [Environment auto-stop](../../ci/environments.md#environments-auto-stop) feature. +the [Environment auto-stop](../../ci/environments/index.md#environments-auto-stop) feature. If you need your Review App to stay up for a longer time, you can -[pin its environment](../../ci/environments.md#auto-stop-example) or retry the +[pin its environment](../../ci/environments/index.md#auto-stop-example) or retry the `review-deploy` job to update the "latest deployed at" time. The `review-cleanup` job that automatically runs in scheduled @@ -153,7 +153,13 @@ used by the `review-deploy` and `review-stop` jobs. ### Get access to the GCP Review Apps cluster You need to [open an access request (internal link)](https://gitlab.com/gitlab-com/access-requests/issues/new) -for the `gcp-review-apps-sg` GCP group. +for the `gcp-review-apps-sg` GCP group. In order to join a group, you must specify the desired GCP role in your access request. +The role is what will grant you specific permissions in order to engage with Review App containers. + +Here are some permissions you may want to have, and the roles that grant them: + +- `container.pods.getLogs` - Required to [retrieve pod logs](#dig-into-a-pods-logs). Granted by [Viewer (`roles/viewer`)](https://cloud.google.com/iam/docs/understanding-roles#kubernetes-engine-roles). +- `container.pods.exec` - Required to [run a Rails console](#run-a-rails-console). Granted by [Kubernetes Engine Developer (`roles/container.developer`)](https://cloud.google.com/iam/docs/understanding-roles#kubernetes-engine-roles). ### Log into my Review App @@ -175,7 +181,7 @@ secure note named `gitlab-{ce,ee} Review App's root password`. ### Run a Rails console -1. Make sure you [have access to the cluster](#get-access-to-the-gcp-review-apps-cluster) first. +1. Make sure you [have access to the cluster](#get-access-to-the-gcp-review-apps-cluster) and the `container.pods.exec` permission first. 1. [Filter Workloads by your Review App slug](https://console.cloud.google.com/kubernetes/workload?project=gitlab-review-apps), e.g. `review-qa-raise-e-12chm0`. 1. Find and open the `task-runner` Deployment, e.g. `review-qa-raise-e-12chm0-task-runner`. @@ -191,7 +197,7 @@ secure note named `gitlab-{ce,ee} Review App's root password`. ### Dig into a Pod's logs -1. Make sure you [have access to the cluster](#get-access-to-the-gcp-review-apps-cluster) first. +1. Make sure you [have access to the cluster](#get-access-to-the-gcp-review-apps-cluster) and the `container.pods.getLogs` permission first. 1. [Filter Workloads by your Review App slug](https://console.cloud.google.com/kubernetes/workload?project=gitlab-review-apps), e.g. `review-qa-raise-e-12chm0`. 1. Find and open the `migrations` Deployment, e.g. @@ -209,6 +215,31 @@ Leading indicators may be health check failures leading to restarts or majority The [Review Apps Overview dashboard](https://app.google.stackdriver.com/dashboards/6798952013815386466?project=gitlab-review-apps&timeDomain=1d) aids in identifying load spikes on the cluster, and if nodes are problematic or the entire cluster is trending towards unhealthy. +### Release failed with `ImagePullBackOff` + +**Potential cause:** + +If you see an `ImagePullBackoff` status, check for a missing Docker image. + +**Where to look for further debugging:** + +To check that the Docker images were created, run the following Docker command: + +```shell +`DOCKER_CLI_EXPERIMENTAL=enabled docker manifest repository:tag` +``` + +The output of this command indicates if the Docker image exists. For example: + +```shell +DOCKER_CLI_EXPERIMENTAL=enabled docker manifest inspect registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-rails-ee:39467-allow-a-release-s-associated-milestones-to-be-edited-thro +``` + +If the Docker image does not exist: + +- Verify the `image.repository` and `image.tag` options in the `helm upgrade --install` command match the repository names used by CNG-mirror pipeline. +- Look further in the corresponding downstream CNG-mirror pipeline in `review-build-cng` job. + ### Node count is always increasing (i.e. never stabilizing or decreasing) **Potential cause:** diff --git a/doc/development/testing_guide/testing_levels.md b/doc/development/testing_guide/testing_levels.md index 295aa6609a8..9285a910ecf 100644 --- a/doc/development/testing_guide/testing_levels.md +++ b/doc/development/testing_guide/testing_levels.md @@ -312,9 +312,7 @@ of a controller test. Testing a fat controller usually involves a lot of stubbin controller.instance_variable_set(:@user, user) ``` -and use methods which are deprecated in Rails 5 ([#23768]). - -[#23768]: https://gitlab.com/gitlab-org/gitlab/-/issues/16260 +and use methods which are deprecated in Rails 5 ([#23768](https://gitlab.com/gitlab-org/gitlab/-/issues/16260)). ### About Karma @@ -356,7 +354,7 @@ possible). | Tests path | Testing engine | Notes | | ---------- | -------------- | ----- | -| `spec/features/` | [Capybara] + [RSpec] | If your test has the `:js` metadata, the browser driver will be [Poltergeist], otherwise it's using [RackTest]. | +| `spec/features/` | [Capybara](https://github.com/teamcapybara/capybara) + [RSpec](https://github.com/rspec/rspec-rails#feature-specs) | If your test has the `:js` metadata, the browser driver will be [Poltergeist](https://github.com/teamcapybara/capybara#poltergeist), otherwise it's using [RackTest](https://github.com/teamcapybara/capybara#racktest). | ### Frontend feature tests @@ -460,9 +458,6 @@ The reasons why we should follow these best practices are as follows: of tests). This is slower than transactions, however, so we want to use truncation only when necessary. -[Poltergeist]: https://github.com/teamcapybara/capybara#poltergeist -[RackTest]: https://github.com/teamcapybara/capybara#racktest - ## Black-box tests at the system level, aka end-to-end tests Formal definitions: @@ -470,11 +465,11 @@ Formal definitions: - <https://en.wikipedia.org/wiki/System_testing> - <https://en.wikipedia.org/wiki/Black-box_testing> -GitLab consists of [multiple pieces] such as [GitLab Shell], [GitLab Workhorse], -[Gitaly], [GitLab Pages], [GitLab Runner], and GitLab Rails. All theses pieces -are configured and packaged by [GitLab Omnibus]. +GitLab consists of [multiple pieces](../architecture.md#components) such as [GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell), [GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse), +[Gitaly](https://gitlab.com/gitlab-org/gitaly), [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages), [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner), and GitLab Rails. All theses pieces +are configured and packaged by [Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab). -The QA framework and instance-level scenarios are [part of GitLab Rails] so that +The QA framework and instance-level scenarios are [part of GitLab Rails](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/qa) so that they're always in-sync with the codebase (especially the views). Note that: @@ -483,11 +478,11 @@ Note that: - data needed for the tests can only be created using the GUI or the API - expectations can only be made against the browser page and API responses -Every new feature should come with a [test plan]. +Every new feature should come with a [test plan](https://gitlab.com/gitlab-org/gitlab/tree/master/.gitlab/issue_templates/Test%20plan.md). | Tests path | Testing engine | Notes | | ---------- | -------------- | ----- | -| `qa/qa/specs/features/` | [Capybara] + [RSpec] + Custom QA framework | Tests should be placed under their corresponding [Product category] | +| `qa/qa/specs/features/` | [Capybara](https://github.com/teamcapybara/capybara) + [RSpec](https://github.com/rspec/rspec-rails#feature-specs) + Custom QA framework | Tests should be placed under their corresponding [Product category](https://about.gitlab.com/handbook/product/categories/) | > See [end-to-end tests](end_to_end/index.md) for more information. @@ -495,17 +490,6 @@ Note that `qa/spec` contains unit tests of the QA framework itself, not to be confused with the application's [unit tests](#unit-tests) or [end-to-end tests](#black-box-tests-at-the-system-level-aka-end-to-end-tests). -[multiple pieces]: ../architecture.md#components -[GitLab Shell]: https://gitlab.com/gitlab-org/gitlab-shell -[GitLab Workhorse]: https://gitlab.com/gitlab-org/gitlab-workhorse -[Gitaly]: https://gitlab.com/gitlab-org/gitaly -[GitLab Pages]: https://gitlab.com/gitlab-org/gitlab-pages -[GitLab Runner]: https://gitlab.com/gitlab-org/gitlab-runner -[GitLab Omnibus]: https://gitlab.com/gitlab-org/omnibus-gitlab -[part of GitLab Rails]: https://gitlab.com/gitlab-org/gitlab-foss/tree/master/qa -[test plan]: https://gitlab.com/gitlab-org/gitlab/tree/master/.gitlab/issue_templates/Test%20plan.md -[Product category]: https://about.gitlab.com/handbook/product/categories/ - ### Smoke tests Smoke tests are quick tests that may be run at any time (especially after the @@ -517,14 +501,11 @@ These tests run against the UI and ensure that basic functionality is working. ### GitLab QA orchestrator -[GitLab QA orchestrator] is a tool that allows to test that all these pieces +[GitLab QA orchestrator](https://gitlab.com/gitlab-org/gitlab-qa) is a tool that allows to test that all these pieces integrate well together by building a Docker image for a given version of GitLab Rails and running end-to-end tests (i.e. using Capybara) against it. -Learn more in the [GitLab QA orchestrator README][gitlab-qa-readme]. - -[GitLab QA orchestrator]: https://gitlab.com/gitlab-org/gitlab-qa -[gitlab-qa-readme]: https://gitlab.com/gitlab-org/gitlab-qa/tree/master/README.md +Learn more in the [GitLab QA orchestrator README](https://gitlab.com/gitlab-org/gitlab-qa/tree/master/README.md). ## EE-specific tests @@ -538,7 +519,9 @@ trade-off: - Unit tests are usually cheap, and you should consider them like the basement of your house: you need them to be confident that your code is behaving correctly. However if you run only unit tests without integration / system - tests, you might [miss] the [big] / [picture] ! + tests, you might [miss](https://twitter.com/ThePracticalDev/status/850748070698651649) the + [big](https://twitter.com/timbray/status/822470746773409794) / + [picture](https://twitter.com/withzombies/status/829716565834752000) ! - Integration tests are a bit more expensive, but don't abuse them. A system test is often better than an integration test that is stubbing a lot of internals. - System tests are expensive (compared to unit tests), even more if they require @@ -546,8 +529,8 @@ trade-off: section. Another way to see it is to think about the "cost of tests", this is well -explained [in this article][tests-cost] and the basic idea is that the cost of a -test includes: +explained [in this article](https://medium.com/table-xi/high-cost-tests-and-high-value-tests-a86e27a54df#.2ulyh3a4e) +and the basic idea is that the cost of a test includes: - The time it takes to write the test - The time it takes to run the test every time the suite runs @@ -557,18 +540,11 @@ test includes: ### Frontend-related tests -There are cases where the behaviour you are testing is not worth the time spent +There are cases where the behavior you are testing is not worth the time spent running the full application, for example, if you are testing styling, animation, edge cases or small actions that don't involve the backend, you should write an integration test using Jasmine. -[miss]: https://twitter.com/ThePracticalDev/status/850748070698651649 -[big]: https://twitter.com/timbray/status/822470746773409794 -[picture]: https://twitter.com/withzombies/status/829716565834752000 -[tests-cost]: https://medium.com/table-xi/high-cost-tests-and-high-value-tests-a86e27a54df#.2ulyh3a4e -[RSpec]: https://github.com/rspec/rspec-rails#feature-specs -[Capybara]: https://github.com/teamcapybara/capybara - --- [Return to Testing documentation](index.md) diff --git a/doc/development/uploads.md b/doc/development/uploads.md index fbdb15d82e1..1dd6ab5496a 100644 --- a/doc/development/uploads.md +++ b/doc/development/uploads.md @@ -11,7 +11,7 @@ The following graph explains machine boundaries in a scalable GitLab installatio ```mermaid graph TB subgraph "load balancers" - LB(HA Proxy) + LB(Proxy) end subgraph "Shared storage" diff --git a/doc/development/value_stream_analytics.md b/doc/development/value_stream_analytics.md index 3ceb523ab73..69b07eb7c86 100644 --- a/doc/development/value_stream_analytics.md +++ b/doc/development/value_stream_analytics.md @@ -185,7 +185,7 @@ Currently supported parents: The [original implementation](https://gitlab.com/gitlab-org/gitlab/issues/847) of value stream analytics defined 7 stages. These stages are always available for each parent, however altering these stages is not possible. -To make things efficient and reduce the number of records created, the default stages are expressed as in-memory objects (not persisted). When the user creates a custom stage for the first time, all the stages will be persisted. This behaviour is implemented in the value stream analytics service objects. +To make things efficient and reduce the number of records created, the default stages are expressed as in-memory objects (not persisted). When the user creates a custom stage for the first time, all the stages will be persisted. This behavior is implemented in the value stream analytics service objects. The reason for this was that we'd like to add the abilities to hide and order stages later on. diff --git a/doc/development/verifying_database_capabilities.md b/doc/development/verifying_database_capabilities.md index 1413c782c5d..f6c78e51299 100644 --- a/doc/development/verifying_database_capabilities.md +++ b/doc/development/verifying_database_capabilities.md @@ -2,7 +2,7 @@ Sometimes certain bits of code may only work on a certain database version. While we try to avoid such code as much as possible sometimes it is -necessary to add database (version) specific behaviour. +necessary to add database (version) specific behavior. To facilitate this we have the following methods that you can use: @@ -12,7 +12,7 @@ To facilitate this we have the following methods that you can use: This allows you to write code such as: ```ruby -if Gitlab::Database.version.to_f >= 9.6 +if Gitlab::Database.version.to_f >= 11.7 run_really_fast_query else run_fast_query diff --git a/doc/development/what_requires_downtime.md b/doc/development/what_requires_downtime.md index 9ece6eff41e..8ea9f70fc7a 100644 --- a/doc/development/what_requires_downtime.md +++ b/doc/development/what_requires_downtime.md @@ -5,38 +5,6 @@ GitLab offline, others do require a downtime period. This guide describes various operations, their impact, and how to perform them without requiring downtime. -## Adding Columns - -You can safely add a new column to an existing table as long as it does **not** -have a default value. For example, this query would not require downtime: - -```sql -ALTER TABLE projects ADD COLUMN random_value int; -``` - -Add a column _with_ a default however does require downtime. For example, -consider this query: - -```sql -ALTER TABLE projects ADD COLUMN random_value int DEFAULT 42; -``` - -This requires updating every single row in the `projects` table so that -`random_value` is set to `42` by default. This requires updating all rows and -indexes in a table. This in turn acquires enough locks on the table for it to -effectively block any other queries. - -Adding a column with a default value _can_ be done without requiring downtime -when using the migration helper method -`Gitlab::Database::MigrationHelpers#add_column_with_default`. This method works -similar to `add_column` except it updates existing rows in batches without -blocking access to the table being modified. See ["Adding Columns With Default -Values"](migration_style_guide.md#adding-columns-with-default-values) for more -information on how to use this method. - -Note that usage of `add_column_with_default` with `allow_null: false` to also add -a `NOT NULL` constraint is [discouraged](https://gitlab.com/gitlab-org/gitlab/issues/38060). - ## Dropping Columns Removing columns is tricky because running GitLab processes may still be using @@ -171,8 +139,39 @@ Adding or removing a NOT NULL clause (or another constraint) can typically be done without requiring downtime. However, this does require that any application changes are deployed _first_. Thus, changing the constraints of a column should happen in a post-deployment migration. -NOTE: Avoid using `change_column` as it produces inefficient query because it re-defines -the whole column type. For example, to add a NOT NULL constraint, prefer `change_column_null` + +NOTE: Avoid using `change_column` as it produces an inefficient query because it re-defines +the whole column type. + +To add a NOT NULL constraint, use the `add_not_null_constraint` migration helper: + +```ruby +# A post-deployment migration in db/post_migrate +class AddNotNull < ActiveRecord::Migration[4.2] + include Gitlab::Database::MigrationHelpers + + disable_ddl_transaction! + + def up + add_not_null_constraint :users, :username + end + + def down + remove_not_null_constraint :users, :username + end +end +``` + +If the column to be updated requires cleaning first (e.g. there are `NULL` values), you should: + +1. Add the `NOT NULL` constraint with `validate: false` + + `add_not_null_constraint :users, :username, validate: false` + +1. Clean up the data with a data migration +1. Validate the `NOT NULL` constraint with a followup migration + + `validate_not_null_constraint :users, :username` ## Changing Column Types diff --git a/doc/development/windows.md b/doc/development/windows.md new file mode 100644 index 00000000000..b5309002222 --- /dev/null +++ b/doc/development/windows.md @@ -0,0 +1,139 @@ +--- +type: reference, howto +--- + +# Windows Development + +There are times in development where a Windows development machine is needed. +This is a guide for how to get a Windows development virtual machine on Google Cloud Platform +(GCP) with the same preinstalled tools as the GitLab shared Windows runners. + +## Why Windows in Google Cloud? + +Use of Microsoft Windows operating systems on company laptops is banned under GitLab's [Approved Operating Systems policy](https://about.gitlab.com/handbook/security/approved_os.html#windows). + +This can make it difficult to develop features for the Windows platforms. Using GCP will allow us to have a temporary Windows machine that can be removed once we're done with it. + +## Shared Windows runners + +You can use the shared Windows runners in the case that you don't need a full Windows development machine. +The [GitLab 12.7 Release Post](https://about.gitlab.com/releases/2020/01/22/gitlab-12-7-released/#windows-shared-runners-on-gitlabcom-beta) +and [Windows shared runner beta blog post](https://about.gitlab.com/blog/2020/01/21/windows-shared-runner-beta/#getting-started) both +outline quite a bit of useful information. + +To use the shared Windows runners add the following `tags` to relevant jobs in your `.gitlab-ci.yml` file: + +```yaml +tags: + - shared-windows + - windows + - windows-1809 +``` + +A list of software preinstalled on the Windows images is available at: [Preinstalled software](https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/gcp/windows-containers/blob/master/cookbooks/preinstalled-software/README.md). + +## GCP Windows image for development + +The [shared Windows GitLab +runners](https://about.gitlab.com/releases/2020/01/22/gitlab-12-7-released/#windows-shared-runners-on-gitlabcom-beta) +are built with [Packer](https://www.packer.io/). + +The Infrastructure as Code repository for building the Google Cloud images is available at: +[GitLab Google Cloud Platform Shared Runner Images](https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/gcp/windows-containers). + +### Build image + +There is a chance that your Google Cloud group may already have an image +built. Search the available images before you do the work to build your +own. + +Build a Google Cloud image with the above shared runners repo by doing the following: + +1. Install [Packer](https://www.packer.io/) (tested to work with version 1.5.1). +1. Install Packer Windows Update Provisioner. + 1. Clone the repository <https://github.com/rgl/packer-provisioner-windows-update> and `cd` into the cloned directory. + 1. Run the command `go build -o packer-provisioner-windows-update` (requires `go` to be installed). + 1. Verify `packer-provisioner-windows-update` is in the `PATH` environment variable. +1. Add all [required environment variables](https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/gcp/windows-containers/-/blob/master/packer.json#L2-10) + in the `packer.json` file to your environment (perhaps use [direnv](https://direnv.net/)). +1. Build the image by running the command: `packer build packer.json`. + +## How to use a Windows image in GCP + +1. In a web browser, go to <https://console.cloud.google.com/compute/images>. +1. Filter images by the name you used when creating image, `windows` is likely all you need to filter by. +1. Click the image's name. +1. Click the **CREATE INSTANCE** link. +1. Important: Change name to what you'd like as you can't change it later. +1. Optional: Change Region to be closest to you as well as any other option you'd like. +1. Click **Create** at the bottom of the page. +1. Click the name of your newly created VM Instance (optionally you can filter to find it). +1. Click **Set Windows password**. +1. Optional: Set a username or use default. +1. Click **Next**. +1. Copy and save the password as it won't be shown again. +1. Click **RDP** down arrow. +1. Click **Download the RDP file**. +1. Open the downloaded RDP file with the Windows remote desktop app (<https://docs.microsoft.com/en-us/windows-server/remote/remote-desktop-services/clients/remote-desktop-clients>). +1. Click **Continue** to accept the certificate. +1. Enter the password and click **Next**. + +You should now be remoted into a Windows machine with a command prompt. + +### Optional: Use GCP VM Instance as a runner + +- Register the runner with a project: `gitlab-runner.exe register`. +- Install the runner:`gitlab-runner.exe install`. +- Start the runner: `gitlab-runner.exe start`. + +For more information, see [Install GitLab Runner on Windows](https://docs.gitlab.com/runner/install/windows.html) +and [Registering Runners](https://docs.gitlab.com/runner/register/index.html). + +## Developer tips + +Here are a few tips on GCP and Windows. + +### GCP cost savings + +To minimise the cost of your GCP VM instance, stop it when you're not using it. +If you do, you'll need to redownload the RDP file from the console as the IP +address changes every time you stop and start it. + +### chocolatey + +Chocolatey is a package manager for Windows. You can search for packages on <https://chocolatey.org/>. + +- `choco install vim` + +### Visual Studio (install / usage for full GUI) + +You can install Visual Studio and run it within the Windows Remote Desktop app. + +Install it by running: `choco install visualstudio2019community` + +Start it by running: `"C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\devenv.exe" .` + +### .NET 3 support + +You can install .NET version 3 support with the following `DISM` command: + +`DISM /Online /Enable-Feature /FeatureName:NetFx3 /All` + +### nix -> Windows cmd tips + +The first tip for using the Windows command shell is to open Powershell and use that instead. + +Start Powershell: `start powershell`. + +Powershell has aliases for all of the following commands so you don't have to learn the native commands: + +- `ls` ---> `dir` +- `rm` ---> `del` +- `rm -rf nonemptydir` ---> `rmdir /S nonemptydir` +- `/` ---> `\` (path separator) +- `cat` ---> `type` +- `mv` ---> `move` +- Redirection works the same (i.e. `>` and `2>&1`) +- `.\some.exe` to call a local executable +- curl is available +- `..` and `.` are available |
