diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2022-03-18 20:02:30 +0000 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2022-03-18 20:02:30 +0000 |
| commit | 41fe97390ceddf945f3d967b8fdb3de4c66b7dea (patch) | |
| tree | 9c8d89a8624828992f06d892cd2f43818ff5dcc8 /doc/development | |
| parent | 0804d2dc31052fb45a1efecedc8e06ce9bc32862 (diff) | |
| download | gitlab-ce-41fe97390ceddf945f3d967b8fdb3de4c66b7dea.tar.gz | |
Add latest changes from gitlab-org/gitlab@14-9-stable-eev14.9.0-rc42
Diffstat (limited to 'doc/development')
115 files changed, 3906 insertions, 1429 deletions
diff --git a/doc/development/adding_database_indexes.md b/doc/development/adding_database_indexes.md index 0e8e8289464..d263d9b5eb5 100644 --- a/doc/development/adding_database_indexes.md +++ b/doc/development/adding_database_indexes.md @@ -275,11 +275,11 @@ You can verify if the MR was deployed to GitLab.com by executing `/chatops run auto_deploy status <merge_sha>`. To verify existence of the index, you can: -- Use a meta-command in #database-lab, such as: `\d <index_name>` - - Ensure that the index is not [`invalid`](https://www.postgresql.org/docs/12/sql-createindex.html#:~:text=The%20psql%20%5Cd%20command%20will%20report%20such%20an%20index%20as%20INVALID) -- Ask someone in #database to check if the index exists +- Use a meta-command in #database-lab, such as: `\d <index_name>`. + - Ensure that the index is not [`invalid`](https://www.postgresql.org/docs/12/sql-createindex.html#:~:text=The%20psql%20%5Cd%20command%20will%20report%20such%20an%20index%20as%20INVALID). +- Ask someone in #database to check if the index exists. - With proper access, you can also verify directly on production or in a -production clone +production clone. ### Add a migration to create the index synchronously diff --git a/doc/development/agent/gitops.md b/doc/development/agent/gitops.md deleted file mode 100644 index 7c741408ae6..00000000000 --- a/doc/development/agent/gitops.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/gitops.md' -remove_date: '2022-02-01' ---- - -This file was moved to [another location](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/gitops.md). - -<!-- This redirect file can be deleted after <2022-02-01>. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/agent/identity.md b/doc/development/agent/identity.md deleted file mode 100644 index 6caf108a32a..00000000000 --- a/doc/development/agent/identity.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/identity_and_auth.md' -remove_date: '2022-02-01' ---- - -This file was moved to [another location](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/identity_and_auth.md). - -<!-- This redirect file can be deleted after <2022-02-01>. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/agent/index.md b/doc/development/agent/index.md deleted file mode 100644 index 474f8a02933..00000000000 --- a/doc/development/agent/index.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/architecture.md' -remove_date: '2022-02-01' ---- - -This file was moved to [another location](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/architecture.md). - -<!-- This redirect file can be deleted after <2022-02-01>. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/agent/local.md b/doc/development/agent/local.md deleted file mode 100644 index a4b29bea838..00000000000 --- a/doc/development/agent/local.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/local.md' -remove_date: '2022-02-01' ---- - -This file was moved to [another location](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/local.md). - -<!-- This redirect file can be deleted after <2022-02-01>. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/agent/repository_overview.md b/doc/development/agent/repository_overview.md deleted file mode 100644 index 8ea9dceb32a..00000000000 --- a/doc/development/agent/repository_overview.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/repository_overview.md' -remove_date: '2022-02-01' ---- - -This file was moved to [another location](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/repository_overview.md). - -<!-- This redirect file can be deleted after <2022-02-01>. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/agent/routing.md b/doc/development/agent/routing.md deleted file mode 100644 index 364267a45fe..00000000000 --- a/doc/development/agent/routing.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/kas_request_routing.md' -remove_date: '2022-02-01' ---- - -This file was moved to [another location](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/kas_request_routing.md). - -<!-- This redirect file can be deleted after <2022-02-01>. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/agent/user_stories.md b/doc/development/agent/user_stories.md deleted file mode 100644 index 2ed4bbdc9f6..00000000000 --- a/doc/development/agent/user_stories.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/user_stories.md' -remove_date: '2022-02-01' ---- - -This file was moved to [another location](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/user_stories.md). - -<!-- This redirect file can be deleted after <2022-02-01>. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/api_graphql_styleguide.md b/doc/development/api_graphql_styleguide.md index afd745533c9..417ccba26a0 100644 --- a/doc/development/api_graphql_styleguide.md +++ b/doc/development/api_graphql_styleguide.md @@ -442,10 +442,10 @@ booleans: ```ruby class MergeRequestPermissionsType < BasePermissionType - present_using MergeRequestPresenter - graphql_name 'MergeRequestPermissions' + present_using MergeRequestPresenter + abilities :admin_merge_request, :update_merge_request, :create_note ability_field :resolve_note, @@ -1329,6 +1329,10 @@ class UserUpdateMutation < BaseMutation end ``` +Due to changes in the `1.13` version of the `graphql-ruby` gem, `graphql_name` should be the first +line of the class to ensure that type names are generated correctly. The `Graphql::GraphqlNamePosition` cop enforces this. +See [issue #27536](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/27536#note_840245581) for further context. + Our GraphQL mutation names are historically inconsistent, but new mutation names should follow the convention `'{Resource}{Action}'` or `'{Resource}{Action}{Attribute}'`. @@ -1511,9 +1515,9 @@ GraphQL-name of the mutation: ```ruby module Types class MutationType < BaseObject - include Gitlab::Graphql::MountMutation + graphql_name 'Mutation' - graphql_name "Mutation" + include Gitlab::Graphql::MountMutation mount_mutation Mutations::MergeRequests::SetDraft end diff --git a/doc/development/architecture.md b/doc/development/architecture.md index e8d04d68565..dd432dd5e37 100644 --- a/doc/development/architecture.md +++ b/doc/development/architecture.md @@ -101,11 +101,11 @@ understand the GitLab architecture. A complete architecture diagram is available in our [component diagram](#component-diagram) below. - + <!-- -To update this diagram, GitLab team members can edit this source file: -https://docs.google.com/drawings/d/1fBzAyklyveF-i-2q-OHUIqDkYfjjxC4mq5shwKSZHLs/edit. +To update this diagram, use and update this source file: +https://miro.com/app/board/uXjVOH3lzXo=/ --> ### Component diagram @@ -151,7 +151,7 @@ graph LR NGINX -- TCP 8150 --> GitLabKas NGINX --> Registry %% inbound from GitLabShell - GitLabShell --TCP 8080 -->Puma + GitLabShell --> GitLabWorkhorse %% services Puma["Puma (GitLab Rails)"] @@ -349,7 +349,7 @@ Component statuses are linked to configuration documentation for each component. | [GitLab Exporter](#gitlab-exporter) | Generates a variety of GitLab metrics | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | CE & EE | | [GitLab Geo Node](#gitlab-geo) | Geographically distributed GitLab nodes | ⚙ | ⚙ | ❌ | ❌ | ✅ | ❌ | ⚙ | EE Only | | [GitLab Pages](#gitlab-pages) | Hosts static websites | ⚙ | ⚙ | ❌ | ❌ | ✅ | ⚙ | ⚙ | CE & EE | -| [GitLab Agent](#gitlab-agent) | Integrate Kubernetes clusters in a cloud-native way | ⚙ | ⚙ | ⚙ | ❌ | ❌ | ⤓ | ⚙ | EE Only | +| [GitLab agent](#gitlab-agent) | Integrate Kubernetes clusters in a cloud-native way | ⚙ | ⚙ | ⚙ | ❌ | ❌ | ⤓ | ⚙ | EE Only | | [GitLab self-monitoring: Alertmanager](#alertmanager) | Deduplicates, groups, and routes alerts from Prometheus | ⚙ | ⚙ | ✅ | ⚙ | ✅ | ❌ | ❌ | CE & EE | | [GitLab self-monitoring: Grafana](#grafana) | Metrics dashboard | ✅ | ✅ | ⚙ | ⤓ | ✅ | ❌ | ⚙ | CE & EE | | [GitLab self-monitoring: Jaeger](#jaeger) | View traces generated by the GitLab instance | ❌ | ⚙ | ⚙ | ❌ | ❌ | ⤓ | ⚙ | CE & EE | @@ -499,14 +499,14 @@ Geo is a premium feature built to help speed up the development of distributed t GitLab Exporter is a process designed in house that allows us to export metrics about GitLab application internals to Prometheus. You can read more [in the project's README](https://gitlab.com/gitlab-org/gitlab-exporter). -#### GitLab Agent +#### GitLab agent - [Project page](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent) - Configuration: - [Omnibus](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template) - [Charts](https://docs.gitlab.com/charts/charts/gitlab/kas/index.html) -The [GitLab Agent](../user/clusters/agent/index.md) is an active in-cluster +The [GitLab agent](../user/clusters/agent/index.md) is an active in-cluster component for solving GitLab and Kubernetes integration tasks in a secure and cloud-native way. diff --git a/doc/development/backend/create_source_code_be/index.md b/doc/development/backend/create_source_code_be/index.md new file mode 100644 index 00000000000..6421ca3754a --- /dev/null +++ b/doc/development/backend/create_source_code_be/index.md @@ -0,0 +1,143 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Create: Source Code Backend + +The Create:Source Code BE team focuses on the GitLab suite of Source Code Management +(SCM) tools. It is responsible for all backend aspects of the product categories +that fall under the [Source Code group](https://about.gitlab.com/handbook/product/categories/#source-code-group) +of the [Create stage](https://about.gitlab.com/handbook/product/categories/#create-stage) +of the [DevOps lifecycle](https://about.gitlab.com/handbook/product/categories/#devops-stages). + +We interface with the Gitaly and Code Review teams, and work closely with the +[Create:Source Code Frontend team](https://about.gitlab.com/handbook/engineering/development/dev/create-source-code-fe). The features +we work with are listed on the +[Features by Group Page](https://about.gitlab.com/handbook/product/categories/features/#createsource-code-group). + +The team works across three codebases: Workhorse, GitLab Shell and GitLab Rails. + +## Workhorse + +GitLab Workhorse is a smart reverse proxy for GitLab. It handles "large" HTTP +requests such as file downloads, file uploads, `git push`, `git pull` and `git` archive downloads. + +Workhorse itself is not a feature, but there are several features in GitLab +that would not work efficiently without Workhorse. + +Workhorse documentation is available in the [Workhorse repository](https://gitlab.com/gitlab-org/gitlab/tree/master/workhorse). + +## GitLab Shell + +GitLab Shell handles Git SSH sessions for GitLab and modifies the list of authorized keys. +For more information, [refer to the README](https://gitlab.com/gitlab-org/gitlab-shell/-/blob/main/README.md). +for GitLab Shell. + +## GitLab Rails + +### Source code API endpoints + +| Endpoint | Threshold | Source | +| -----------------------------------------------------------------------------------|---------------------------------------|--------------------------------------------------------------------------------------| +| `DELETE /api/:version/projects/:id/protected_branches/:name` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/protected_branches.rb) | +| `GET /api/:version/internal/authorized_keys` | `:high` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/internal/base.rb) | | | +| `GET /api/:version/internal/lfs` | `:high` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/internal/lfs.rb)| +| `GET /api/:version/projects/:id/approval_rules` | `:low` | | +| `GET /api/:version/projects/:id/approval_settings` | default | | +| `GET /api/:version/projects/:id/approvals` | default | | +| `GET /api/:version/projects/:id/forks` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/projects.rb) | +| `GET /api/:version/projects/:id/groups` | default | [source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/projects.rb) | +| `GET /api/:version/projects/:id/languages` | `:medium` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/projects.rb) | +| `GET /api/:version/projects/:id/merge_request_approval_setting` | `:medium` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/api/merge_request_approval_settings.rb) | +| `GET /api/:version/projects/:id/merge_requests/:merge_request_iid/approval_rules` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/api/merge_request_approval_rules.rb) | +| `GET /api/:version/projects/:id/merge_requests/:merge_request_iid/approval_settings` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/api/project_approval_settings.rb) | +| `GET /api/:version/projects/:id/merge_requests/:merge_request_iid/approval_state` | `:low` | [source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/merge_request_approvals.rb) | +| `GET /api/:version/projects/:id/merge_requests/:merge_request_iid/approvals` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/merge_request_approvals.rb) | +| `GET /api/:version/projects/:id/protected_branches` | default |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/protected_branches.rb) | +| `GET /api/:version/projects/:id/protected_branches/:name` | default |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/protected_branches.rb) | +| `GET /api/:version/projects/:id/protected_tags` | default | | +| `GET /api/:version/projects/:id/protected_tags/:name` | default | | +| `GET /api/:version/projects/:id/push_rule` | default | | +| `GET /api/:version/projects/:id/remote_mirrors` | default | | +| `GET /api/:version/projects/:id/repository/archive` | default | | +| `GET /api/:version/projects/:id/repository/blobs/:sha` | default | | +| `GET /api/:version/projects/:id/repository/blobs/:sha/raw` | default | | +| `GET /api/:version/projects/:id/repository/branches` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/branches.rb) | +| `GET /api/:version/projects/:id/repository/branches/:branch` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/branches.rb) | +| `GET /api/:version/projects/:id/repository/commits` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/commits.rb)| +| `GET /api/:version/projects/:id/repository/commits/:sha` | default | [source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/commits.rb) | +| `GET /api/:version/projects/:id/repository/commits/:sha/comments` | default | [source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/commits.rb) | +| `GET /api/:version/projects/:id/repository/commits/:sha/diff` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/commits.rb) | +| `GET /api/:version/projects/:id/repository/commits/:sha/merge_requests` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/commits.rb)| +| `GET /api/:version/projects/:id/repository/commits/:sha/refs` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/commits.rb) | +| `GET /api/:version/projects/:id/repository/compare` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/repositories.rb) | +| `GET /api/:version/projects/:id/repository/contributors` | default | | +| `GET /api/:version/projects/:id/repository/files/:file_path` | default | | +| `GET /api/:version/projects/:id/repository/files/:file_path/raw` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/files.rb) | +| `GET /api/:version/projects/:id/repository/tags` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/tags.rb) | +| `GET /api/:version/projects/:id/repository/tree` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/repositories.rb) | +| `GET /api/:version/projects/:id/statistics` | default | | +| `GraphqlController#execute` | default | | +| `HEAD /api/:version/projects/:id/repository/files/:file_path` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/files.rb) | +| `HEAD /api/:version/projects/:id/repository/files/:file_path/raw` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/files.rb) | +| `POST /api/:version/internal/allowed` | default | [source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/internal/base.rb) | +| `POST /api/:version/internal/lfs_authenticate` | `:high` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/internal/base.rb) | +| `POST /api/:version/internal/post_receive` | default | [source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/internal/base.rb) | +| `POST /api/:version/internal/pre_receive` | `:high` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/internal/base.rb) | +| `POST /api/:version/projects/:id/approvals` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/api/project_approvals.rb) | +| `POST /api/:version/projects/:id/merge_requests/:merge_request_iid/approvals` | `:low` | [source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/merge_request_approvals.rb) | +| `POST /api/:version/projects/:id/merge_requests/:merge_request_iid/approve` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/merge_request_approvals.rb) | +| `POST /api/:version/projects/:id/merge_requests/:merge_request_iid/unapprove` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/merge_request_approvals.rb)| +| `POST /api/:version/projects/:id/protected_branches` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/protected_branches.rb)| +| `POST /api/:version/projects/:id/repository/commits` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/commits.rb)| +| `POST /api/:version/projects/:id/repository/files/:file_path` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/files.rb) | +| `PUT /api/:version/projects/:id/push_rule` | default | | +| `PUT /api/:version/projects/:id/repository/files/:file_path` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/files.rb) | +| `Projects::BlameController#show` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/projects/blame_controller.rb) | +| `Projects::BlobController#create` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/projects/blob_controller.rb) | +| `Projects::BlobController#diff` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/projects/blob_controller.rb) | +| `Projects::BlobController#edit` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/projects/blob_controller.rb) | +| `Projects::BlobController#show` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/projects/blob_controller.rb) | +| `Projects::BlobController#update` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/projects/blob_controller.rb) | +| `Projects::BranchesController#create` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/projects/branches_controller.rb) | +| `Projects::BranchesController#destroy` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/projects/branches_controller.rb) | +| `Projects::BranchesController#diverging_commit_counts` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/projects/branches_controller.rb) | +| `Projects::BranchesController#index` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/projects/branches_controller.rb) | +| `Projects::BranchesController#new` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/projects/branches_controller.rb) | +| `Projects::CommitController#branches` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/projects/commit_controller.rb) | +| `Projects::CommitController#merge_requests` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/projects/commit_controller.rb) | +| `Projects::CommitController#pipelines` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/projects/commit_controller.rb) | +| `Projects::CommitController#show` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/projects/commit_controller.rb) | +| `Projects::CommitsController#show` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/projects/commits_controller.rb)| +| `Projects::CommitsController#signatures` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/projects/commits_controller.rb) | +| `Projects::CompareController#create` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/projects/commits_controller.rb) | +| `Projects::CompareController#index` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/projects/compare_controller.rb) | +| `Projects::CompareController#show` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/projects/compare_controller.rb) | +| `Projects::CompareController#signatures` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/projects/compare_controller.rb) | +| `Projects::FindFileController#list` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/projects/find_file_controller.rb) | +| `Projects::FindFileController#show` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/projects/find_file_controller.rb) | +| `Projects::ForksController#index` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/projects/forks_controller.rb) | +| `Projects::GraphsController#show` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/projects/graphs_controller.rb) | +| `Projects::NetworkController#show` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/projects/network_controller.rb) | +| `Projects::PathLocksController#index` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/controllers/projects/path_locks_controller.rb) | +| `Projects::RawController#show` | default | | +| `Projects::RefsController#logs_tree` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/projects/refs_controller.rb) | +| `Projects::RefsController#switch` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/projects/refs_controller.rb) | +| `Projects::RepositoriesController#archive` | default | | +| `Projects::Settings::RepositoryController#show` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/projects/settings/repository_controller.rb) | +| `Projects::TagsController#index` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/projects/tags_controller.rb) | +| `Projects::TagsController#new` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/projects/tags_controller.rb) | +| `Projects::TagsController#show` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/projects/tags_controller.rb) | +| `Projects::TemplatesController#names` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/projects/templates_controller.rb) | +| `Projects::TreeController#show` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/projects/tree_controller.rb) | +| `ProjectsController#refs` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/projects_controller.rb) | +| `Repositories::GitHttpController#git_receive_pack` | default | | +| `Repositories::GitHttpController#git_upload_pack` | default | | +| `Repositories::GitHttpController#info_refs` | default | | +| `Repositories::LfsApiController#batch` | `:medium` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/repositories/lfs_api_controller.rb) | +| `Repositories::LfsLocksApiController#verify` | default | | +| `Repositories::LfsStorageController#download` | `:medium` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/repositories/lfs_storage_controller.rb) | +| `Repositories::LfsStorageController#upload_authorize` | `:medium` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/repositories/lfs_storage_controller.rb) | +| `Repositories::LfsStorageController#upload_finalize` | `:low` |[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/repositories/lfs_storage_controller.rb) | diff --git a/doc/development/background_migrations.md b/doc/development/background_migrations.md index 49835085f96..9fffbd25518 100644 --- a/doc/development/background_migrations.md +++ b/doc/development/background_migrations.md @@ -489,7 +489,8 @@ View the production Sidekiq log and filter for: - `json.class: BackgroundMigrationWorker` - `json.job_status: fail` -- `json.meta.caller_id: <MyBackgroundMigrationClassName>` +- `json.meta.caller_id: <MyBackgroundMigrationSchedulingMigrationClassName>` +- `json.args: <MyBackgroundMigrationClassName>` Looking at the `json.error_class`, `json.error_message` and `json.error_backtrace` values may be helpful in understanding why the jobs failed. diff --git a/doc/development/bulk_import.md b/doc/development/bulk_import.md index ff0c8a19ca1..a2620faed35 100644 --- a/doc/development/bulk_import.md +++ b/doc/development/bulk_import.md @@ -1,7 +1,7 @@ --- stage: Manage group: Import -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # GitLab Group Migration diff --git a/doc/development/caching.md b/doc/development/caching.md index 20847832e37..7c51bd595f7 100644 --- a/doc/development/caching.md +++ b/doc/development/caching.md @@ -265,6 +265,13 @@ All the time! - As the lookup is similar to a cache lookup (in the GitLab implementation), we can use the same key for both. This is how `Gitlab::Cache.fetch_once` works. +#### Possible downsides + +- Adding new attributes to a cached object using `Gitlab::JsonCache` + and `Gitlab::SafeRequestStore`, for example, can lead to stale data issues + where the cache data doesn't have the appropriate value for the new attribute + (see this past [incident](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6372)). + ### When to use SQL caching Rails uses this automatically for identical queries in a request, so no action is diff --git a/doc/development/changelog.md b/doc/development/changelog.md index b51db69c2f7..b98ed6cb109 100644 --- a/doc/development/changelog.md +++ b/doc/development/changelog.md @@ -101,6 +101,7 @@ EE: true - _Any_ contribution from a community member, no matter how small, **may** have a changelog entry regardless of these guidelines if the contributor wants one. - Any [GLEX experiment](experiment_guide/gitlab_experiment.md) changes **should not** have a changelog entry. +- An MR that includes only documentation changes **should not** have a changelog entry. For more information, see [how to handle changelog entries with feature flags](feature_flags/index.md#changelog). diff --git a/doc/development/cicd/index.md b/doc/development/cicd/index.md index 2779b457fb9..8677d5b08e3 100644 --- a/doc/development/cicd/index.md +++ b/doc/development/cicd/index.md @@ -7,9 +7,10 @@ type: index, concepts, howto # CI/CD development documentation **(FREE)** -Development guides that are specific to CI/CD are listed here. +Development guides that are specific to CI/CD are listed here: -If you are creating new CI/CD templates, please read [the development guide for GitLab CI/CD templates](templates.md). +- If you are creating new CI/CD templates, please read [the development guide for GitLab CI/CD templates](templates.md). +- If you are adding a new keyword or changing the CI schema, check the [CI schema guide](schema.md) See the [CI/CD YAML reference documentation guide](cicd_reference_documentation_guide.md) to learn how to update the [reference page](../../ci/yaml/index.md). @@ -29,7 +30,7 @@ On the left side we have the events that can trigger a pipeline based on various - A user clicking the "Run pipeline" button in the UI. - When a [merge request is created or updated](../../ci/pipelines/merge_request_pipelines.md). - When an MR is added to a [Merge Train](../../ci/pipelines/merge_trains.md#merge-trains). -- A [scheduled pipeline](../../ci/pipelines/schedules.md#pipeline-schedules). +- A [scheduled pipeline](../../ci/pipelines/schedules.md). - When project is [subscribed to an upstream project](../../ci/pipelines/multi_project_pipelines.md#trigger-a-pipeline-when-an-upstream-project-is-rebuilt). - When [Auto DevOps](../../topics/autodevops/index.md) is enabled. - When GitHub integration is used with [external pull requests](../../ci/ci_cd_for_external_repos/index.md#pipelines-for-external-pull-requests). diff --git a/doc/development/cicd/schema.md b/doc/development/cicd/schema.md new file mode 100644 index 00000000000..b63d951b881 --- /dev/null +++ b/doc/development/cicd/schema.md @@ -0,0 +1,146 @@ +--- +stage: Verify +group: Pipeline Authoring +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: index, howto +--- + +# Contribute to the CI Schema **(FREE)** + +The [pipeline editor](../../ci/pipeline_editor/index.md) uses a CI schema to enhance +the authoring experience of our CI configuration files. With the CI schema, the editor can: + +- Validate the content of the CI configuration file as it is being written in the editor. +- Provide autocomplete functionality and suggest available keywords. +- Provide definitions of keywords through annotations. + +As the rules and keywords for configuring our CI configuration files change, so too +should our CI schema. + +This feature is behind the [`schema_linting`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/feature_flags/development/schema_linting.yml) +feature flag for self-managed instances, and is enabled for GitLab.com. + +## JSON Schemas + +The CI schema follows the [JSON Schema Draft-07](https://json-schema.org/draft-07/json-schema-release-notes.html) +specification. Although the CI configuration file is written in YAML, it is converted +into JSON by using `monaco-yaml` before it is validated by the CI schema. + +If you're new to JSON schemas, consider checking out +[this guide](https://json-schema.org/learn/getting-started-step-by-step) for +a step-by-step introduction on how to work with JSON schemas. + +## Update Keywords + +The CI schema is at [`app/assets/javascripts/editor/schema/ci.json`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/javascripts/editor/schema/ci.json). +It contains all the keywords available for authoring CI configuration files. +Check the [keyword reference](../../ci/yaml/index.md) for a comprehensive list of +all available keywords. + +All keywords are defined under `definitions`. We use these definitions as +[references](https://json-schema.org/learn/getting-started-step-by-step#references) +to share common data structures across the schema. + +For example, this defines the `retry` keyword: + +```json +{ + "definitions": { + "retry": { + "description": "Retry a job if it fails. Can be a simple integer or object definition.", + "oneOf": [ + { + "$ref": "#/definitions/retry_max" + }, + { + "type": "object", + "additionalProperties": false, + "properties": { + "max": { + "$ref": "#/definitions/retry_max" + }, + "when": { + "description": "Either a single or array of error types to trigger job retry.", + "oneOf": [ + { + "$ref": "#/definitions/retry_errors" + }, + { + "type": "array", + "items": { + "$ref": "#/definitions/retry_errors" + } + } + ] + } + } + } + ] + } + } +} +``` + +With this definition, the `retry` keyword is both a property of +the `job_template` definition and the `default` global keyword. Global keywords +that configure pipeline behavior (such as `workflow` and `stages`) are defined +under the topmost **properties** key. + +```json +{ + "properties": { + "default": { + "type": "object", + "properties": { + "retry": { + "$ref": "#/definitions/retry" + }, + } + } + }, + "definitions": { + "job_template": { + "properties": { + "retry": { + "$ref": "#/definitions/retry" + } + }, + } + } +} +``` + +## Guidelines for updating the schema + +- Keep definitions atomic when possible, to be flexible with + referencing keywords. For example, `workflow:rules` uses only a subset of + properties in the `rules` definition. The `rules` properties have their + own definitions, so we can reference them individually. +- When adding new keywords, consider adding a `description` with a link to the + keyword definition in the documentation. This information shows up in the annotations + when the user hovers over the keyword. +- For each property, consider if a `minimum`, `maximum`, or + `default` values are required. Some values might be required, and in others we can set + blank. In the blank case, we can add the following to the definition: + +```json +{ + "keyword": { + "oneOf": [ + { + "type": "null" + }, + ... + ] + } +} +``` + +## Test the schema + +For now, the CI schema can only be tested manually. To verify the behavior is correct: + +1. Enable the `schema_linting` feature flag. +1. Go to **CI/CD** > **Editor**. +1. Write your CI/CD configuration in the editor and verify that the schema validates + it correctly. diff --git a/doc/development/cicd/templates.md b/doc/development/cicd/templates.md index d7edad842b8..c6f59a7e452 100644 --- a/doc/development/cicd/templates.md +++ b/doc/development/cicd/templates.md @@ -96,7 +96,7 @@ Additional points to keep in mind when authoring templates: |------------------------------------------------------|--------------------|---------------| | Can use global keywords, including `stages`. | Yes | No | | Can define jobs. | Yes | Yes | -| Can be selected in the new file UI | Yes | Yes | +| Can be selected in the new file UI | Yes | No | | Can include other job templates with `include` | Yes | No | | Can include other pipeline templates with `include`. | No | No | @@ -105,6 +105,16 @@ Additional points to keep in mind when authoring templates: To make templates easier to follow, templates should all use clear syntax styles, with a consistent format. +The `before_script`, `script`, and `after_script` keywords of every job are linted +using [ShellCheck](https://www.shellcheck.net/) and should follow the +[Shell scripting standards and style guidelines](../shell_scripting_guide/index.md) +as much as possible. + +ShellCheck assumes that the script is designed to run using [Bash](https://www.gnu.org/software/bash/). +Templates which use scripts for shells that aren't compatible with the Bash ShellCheck +rules can be excluded from ShellCheck linting. To exclude a script, add it to the +`EXCLUDED_TEMPLATES` list in [`scripts/lint_templates_bash.rb`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/scripts/lint_templates_bash.rb). + #### Do not hardcode the default branch Use [`$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH`](../../ci/variables/predefined_variables.md) @@ -219,6 +229,30 @@ job1: - echo ${ERROR_MESSAGE} ``` +#### Use all-caps naming for non-local variables + +If you are expecting a variable to be provided via the CI/CD settings, or via the +`variables` keyword, that variable must use all-caps naming with underscores (`_`) +separating words. + +```yaml +.with_login: + before_script: + # SECRET_TOKEN should be provided via the project settings + - docker login -u my-user -p "$SECRET_TOKEN my-registry +``` + +Lower-case naming can optionally be used for variables which are defined locally in +one of the `script` keywords: + +```yaml +job1: + script: + - response="$(curl "https://example.com/json")" + - message="$(echo "$response" | jq -r .message)" + - 'echo "Server responded with: $message"' +``` + ### Backward compatibility A template might be dynamically included with the `include:template:` keyword. If diff --git a/doc/development/code_review.md b/doc/development/code_review.md index 3664ca7642a..ec913df8e4a 100644 --- a/doc/development/code_review.md +++ b/doc/development/code_review.md @@ -74,19 +74,27 @@ It picks reviewers and maintainers from the list at the page, with these behaviors: 1. It doesn't pick people whose Slack or [GitLab status](../user/profile/index.md#set-your-current-status): - - contains the string 'OOO', 'PTO', 'Parental Leave', or 'Friends and Family' - - emoji is `:palm_tree:`, `:beach:`, `:beach_umbrella:`, `:beach_with_umbrella:`, `:ferris_wheel:`, `:thermometer:`, `:face_with_thermometer:`, `:red_circle:`, `:bulb:`, `:sun_with_face:`. - - GitLab user busy indicator is set to true + - Contains the string 'OOO', 'PTO', 'Parental Leave', or 'Friends and Family'. + - GitLab user **Busy** indicator is set to `True`. + - Emoji is any of: + - 🌴 `:palm_tree:` + - 🏖️ `:beach:`, `:beach_umbrella:`, or `:beach_with_umbrella:` + - 🎡 `:ferris_wheel:` + - 🌡️ `:thermometer:` + - 🤒 `:face_with_thermometer:` + - 🔴 `:red_circle:` + - 💡 `:bulb:` + - 🌞 `:sun_with_face:` 1. [Trainee maintainers](https://about.gitlab.com/handbook/engineering/workflow/code-review/#trainee-maintainer) are three times as likely to be picked as other reviewers. 1. Team members whose Slack or [GitLab status](../user/profile/index.md#set-your-current-status) emoji is 🔵 `:large_blue_circle:` are more likely to be picked. This applies to both reviewers and trainee maintainers. - - Reviewers with `:large_blue_circle:` are two times as likely to be picked as other reviewers. - - Trainee maintainers with `:large_blue_circle:` are four times as likely to be picked as other reviewers. + - Reviewers with 🔵 `:large_blue_circle:` are two times as likely to be picked as other reviewers. + - Trainee maintainers with 🔵 `:large_blue_circle:` are four times as likely to be picked as other reviewers. 1. People whose [GitLab status](../user/profile/index.md#set-your-current-status) emoji - is 🔶 `:large_orange_diamond:` are half as likely to be picked. This applies to both reviewers and trainee maintainers. + is 🔶 `:large_orange_diamond:` or 🔸 `:small_orange_diamond:` are half as likely to be picked. This applies to both reviewers and trainee maintainers. 1. It always picks the same reviewers and maintainers for the same - branch name (unless their OOO status changes, as in point 1). It + branch name (unless their out-of-office (OOO) status changes, as in point 1). It removes leading `ce-` and `ee-`, and trailing `-ce` and `-ee`, so that it can be stable for backport branches. @@ -624,7 +632,7 @@ Enterprise Edition instance. This has some implications: [added to Omnibus](https://docs.gitlab.com/omnibus/settings/gitlab.yml#adding-a-new-setting-to-gitlabyml). 1. **File system access** is not possible in a [cloud-native architecture](architecture.md#adapting-existing-and-introducing-new-components). Ensure that we support object storage for any file storage we need to perform. For more - information, see the [uploads documentation](uploads.md). + information, see the [uploads documentation](uploads/index.md). ### Review turnaround time diff --git a/doc/development/contributing/design.md b/doc/development/contributing/design.md index efa8d4b0c41..463a7ee0e0b 100644 --- a/doc/development/contributing/design.md +++ b/doc/development/contributing/design.md @@ -35,7 +35,7 @@ Check these aspects both when _designing_ and _reviewing_ UI changes. - Use clear and consistent [terminology](https://design.gitlab.com/content/terminology/). - Check grammar and spelling. - Consider help content and follow its [guidelines](https://design.gitlab.com/usability/helping-users/). -- Request review from the [appropriate Technical Writer](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers), +- Request review from the [appropriate Technical Writer](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments), indicating any specific files or lines they should review, and how to preview or understand the location/context of the text from the user's perspective. diff --git a/doc/development/contributing/issue_workflow.md b/doc/development/contributing/issue_workflow.md index ad8403d242c..4db686b9b1e 100644 --- a/doc/development/contributing/issue_workflow.md +++ b/doc/development/contributing/issue_workflow.md @@ -45,7 +45,7 @@ scheduling into milestones. Labeling is a task for everyone. (For some projects, Most issues will have labels for at least one of the following: -- Type. For example: `~"type::feature"`, `~"type::bug"`, or `~"type::tooling"`. +- Type. For example: `~"type::feature"`, `~"type::bug"`, or `~"type::maintenance"`. - Stage. For example: `~"devops::plan"` or `~"devops::create"`. - Group. For example: `~"group::source code"`, `~"group::knowledge"`, or `~"group::editor"`. - Category. For example: `~"Category:Code Analytics"`, `~"Category:DevOps Reports"`, or `~"Category:Templates"`. @@ -72,19 +72,7 @@ labels, you can _always_ add the type, stage, group, and often the category/feat Type labels are very important. They define what kind of issue this is. Every issue should have one and only one. -The current type labels are: - -- `~"type::feature"` - - `~"feature::addition"` - - `~"feature::enhancement"` -- `~"type::maintenance"` -- `~"type::bug"` -- `~"type::tooling"` - - `~"tooling::pipelines"` - - `~"tooling::workflow"` -- `~"support request"` -- `~meta` -- `~documentation` +The current type labels are [available in the handbook](https://about.gitlab.com/handbook/engineering/metrics/#work-type-classification) A number of type labels have a priority assigned to them, which automatically makes them float to the top, depending on their importance. diff --git a/doc/development/contributing/merge_request_workflow.md b/doc/development/contributing/merge_request_workflow.md index 02148f2a717..a9b4d13ab06 100644 --- a/doc/development/contributing/merge_request_workflow.md +++ b/doc/development/contributing/merge_request_workflow.md @@ -81,7 +81,7 @@ request is as follows: 1. If your MR touches code that executes shell commands, reads or opens files, or handles paths to files on disk, make sure it adheres to the [shell command guidelines](../shell_commands.md) -1. If your code needs to handle file storage, see the [uploads documentation](../uploads.md). +1. If your code needs to handle file storage, see the [uploads documentation](../uploads/index.md). 1. If your merge request adds one or more migrations, make sure to execute all migrations on a fresh database before the MR is reviewed. If the review leads to large changes in the MR, execute the migrations again once the review is complete. @@ -264,8 +264,11 @@ requirements. 1. Peer member testing is optional but recommended when the risk of a change is high. This includes when the changes are [far-reaching](https://about.gitlab.com/handbook/engineering/development/#reducing-the-impact-of-far-reaching-work) or are for [components critical for security](../code_review.md#security). 1. Regressions and bugs are covered with tests that reduce the risk of the issue happening again. +1. Code affected by a feature flag is covered by [automated tests with the feature flag enabled and disabled](../feature_flags/index.md#feature-flags-in-tests), or both + states are tested as part of peer member testing or as part of the rollout plan. 1. [Performance guidelines](../merge_request_performance_guidelines.md) have been followed. 1. [Secure coding guidelines](https://gitlab.com/gitlab-com/gl-security/security-guidelines) have been followed. +1. [Application and rate limit guidelines](../merge_request_application_and_rate_limit_guidelines.md) have been followed. 1. [Documented](../documentation/index.md) in the `/doc` directory. 1. [Changelog entry added](../changelog.md), if necessary. 1. Reviewed by relevant reviewers, and all concerns are addressed for Availability, Regressions, and Security. Documentation reviews should take place as soon as possible, but they should not block a merge request. diff --git a/doc/development/contributing/style_guides.md b/doc/development/contributing/style_guides.md index da926005466..7a4ebbdbadf 100644 --- a/doc/development/contributing/style_guides.md +++ b/doc/development/contributing/style_guides.md @@ -159,25 +159,22 @@ When the number of RuboCop exceptions exceed the default [`exclude-limit` of 15] we may want to resolve exceptions over multiple commits. To minimize confusion, we should track our progress through the exception list. -When auto-generating the `.rubocop_todo.yml` exception list for a particular Cop, -and more than 15 files are affected, we should add the exception list to -a different file in the directory `.rubocop_todo/`. For example, the configuration for the cop -`Gitlab/NamespacedClass` is in `.rubocop_todo/gitlab/namespaced_class.yml`. - -This ensures that our list isn't mistakenly removed by another auto generation of -the `.rubocop_todo.yml`. This also allows us greater visibility into the exceptions -which are currently being resolved. - -One way to generate the initial list is to run the Rake task `rubocop:todo:generate`: +The preferred way to [generate the initial list or a list for specific RuboCop rules](../rake_tasks.md#generate-initial-rubocop-todo-list) +is to run the Rake task `rubocop:todo:generate`: ```shell +# Initial list bundle exec rake rubocop:todo:generate + +# List for specific RuboCop rules +bundle exec rake 'rubocop:todo:generate[Gitlab/NamespacedClass,Lint/Syntax]' ``` -You can then move the list from the freshly generated `.rubocop_todo.yml` for the Cop being actively -resolved and place it in the directory `.rubocop_todo/`. In this scenario, do not commit -auto-generated changes to the `.rubocop_todo.yml`, as an `exclude limit` that is higher than 15 -makes the `.rubocop_todo.yml` hard to parse. +This Rake task creates or updates the exception list in `.rubocop_todo/`. For +example, the configuration for the RuboCop rule `Gitlab/NamespacedClass` is +located in `.rubocop_todo/gitlab/namespaced_class.yml`. + +Make sure to commit any changes in `.rubocop_todo/` after running the Rake task. ### Reveal existing RuboCop exceptions diff --git a/doc/development/contributing/verify/index.md b/doc/development/contributing/verify/index.md new file mode 100644 index 00000000000..a2bb0eca733 --- /dev/null +++ b/doc/development/contributing/verify/index.md @@ -0,0 +1,236 @@ +--- +type: reference, dev +stage: none +group: Verify +--- + +# Contribute to Verify stage codebase + +## What are we working on in Verify? + +Verify stage is working on a comprehensive Continuous Integration platform +integrated into the GitLab product. Our goal is to empower our users to make +great technical and business decisions, by delivering a fast, reliable, secure +platform that verifies assumptions that our users make, and check them against +the criteria defined in CI/CD configuration. They could be unit tests, end-to-end +tests, benchmarking, performance validation, code coverage enforcement, and so on. + +Feedback delivered by GitLab CI/CD makes it possible for our users to make well +informed decisions about technological and business choices they need to make +to succeed. Why is Continuous Integration a mission critical product? + +GitLab CI/CD is our platform to deliver feedback to our users and customers. + +They contribute their continuous integration configuration files +`.gitlab-ci.yml` to describe the questions they want to get answers for. Each +time someone pushes a commit or triggers a pipeline we need to find answers for +very important questions that have been asked in CI/CD configuration. + +Failing to answer these questions or, what might be even worse, providing false +answers, might result in a user making a wrong decision. Such wrong decisions +can have very severe consequences. + +## Core principles of our CI/CD platform + +Data produced by the platform should be: + +1. Accurate. +1. Durable. +1. Accessible. + +The platform itself should be: + +1. Reliable. +1. Secure. +1. Deterministic. +1. Trustworthy. +1. Fast. +1. Simple. + +Since the inception of GitLab CI/CD, we have lived by these principles, +and they serve us and our users well. Some examples of these principles are that: + +- The feedback delivered by GitLab CI/CD and data produced by the platform should be accurate. + If a job fails and we notify a user that it was successful, it can have severe negative consequences. +- Feedback needs to be available when a user needs it and data can not disappear unexpectedly when engineers need it. +- It all doesn’t matter if the platform is not secure and we +are leaking credentials or secrets. +- When a user provides a set of preconditions in a form of CI/CD configuration, the result should be deterministic each time a pipeline runs, because otherwise the platform might not be trustworthy. +- If it is fast, simple to use and has a great UX it will serve our users well. + +## Building things in Verify + +### Measure before you optimize, and make data-informed decisions + +It is very difficult to optimize something that you can not measure. How would you +know if you succeeded, or how significant the success was? If you are working on +a performance or reliability improvement, make sure that you measure things before +you optimize them. + +The best way to measure stuff is to add a Prometheus metric. Counters, gauges, and +histograms are great ways to quickly get approximated results. Unfortunately this +is not the best way to measure tail latency. Prometheus metrics, especially histograms, +are usually approximations. + +If you have to measure tail latency, like how slow something could be or how +large a request payload might be, consider adding custom application logs and +always use structured logging. + +It's useful to use profiling and flamegraphs to understand what the code execution +path truly looks like! + +### Strive for simple solutions, avoid clever solutions + +It is sometimes tempting to use a clever solution to deliver something more +quickly. We want to avoid shipping clever code, because it is usually more +difficult to understand and maintain in the long term. Instead, we want to +focus on boring solutions that make it easier to evolve the codebase and keep the +contribution barrier low. We want to find solutions that are as simple as +possible. + +### Do not confuse boring solutions with easy solutions + +Boring solutions are sometimes confused with easy solutions. Very often the +opposite is true. An easy solution might not be simple - for example, a complex +new library can be included to add a very small functionality that otherwise +could be implemented quickly - it is easier to include this library than to +build this thing, but it would bring a lot of complexity into the product. + +On the other hand, it is also possible to over-engineer a solution when a simple, +well tested, and well maintained library is available. In that case using the +library might make sense. We recognize that we are constantly balancing simple +and easy solutions, and that finding the right balance is important. + +### "Simple" is not mutually exclusive with "flexible" + +Building simple things does not mean that more advanced and flexible solutions +will not be available. A good example here is an expanding complexity of +writing `.gitlab-ci.yml` configuration. For example, you can use a simple +method to define an environment name: + +```yaml +deploy: + environment: production + script: cap deploy +``` + +But the `environment` keyword can be also expanded into another level of +configuration that can offer more flexibility. + +```yaml +deploy: + environment: + name: review/$CI_COMMIT_REF_SLUG + url: https://prod.example.com + script: cap deploy +``` + +This kind of approach shields new users from the complexities of the platform, +but still allows them to go deeper if they need to. This approach can be +applied to many other technical implementations. + +### Make things observable + +GitLab is a DevOps platform. We popularize DevOps because it helps companies +be more efficient and achieve better results. One important component of +DevOps culture is to take ownership over features and code that you are +building. It is very difficult to do that when you don’t know how your features +perform and behave in the production environment. + +This is why we want to make our features and code observable. It +should be written in a way that an author can understand how well or how poorly +the feature or code behaves in the production environment. We usually accomplish +that by introducing the proper mix of Prometheus metrics and application +loggers. + +**TODO** document when to use Prometheus metrics, when to use loggers. Write a +few sentences about histograms and counters. Write a few sentences highlighting +importance of metrics when doing incremental rollouts. + +### Protect customer data + +Making data produced by our CI/CD platform durable is important. We recognize that +data generated in the CI/CD by users and customers is +something important and we must protect it. This data is not only important +because it can contain important information, we also do have compliance and +auditing responsibilities. + +Therefore we must take extra care when we are writing migrations +that permanently removes data from our database, or when we are define +new retention policies. + +As a general rule, when you are writing code that is supposed to remove +data from the database, file system, or object storage, you should get an extra pair +of eyes on your changes. When you are defining a new retention policy, you +should double check with PMs and EMs. + +### Get your changes reviewed + +When your merge request is ready for reviews you must assign +reviewers and then maintainers. Depending on the complexity of a change, you +might want to involve the people that know the most about the codebase area you are +changing. We do have many domain experts in Verify and it is absolutely acceptable to +ask them to review your code when you are not certain if a reviewer or +maintainer assigned by the Reviewer Roulette has enough context about the +change. + +The reviewer roulette offers useful suggestions, but as assigning the right +reviewers is important it should not be done automatically every time. It might +not make sense to assign someone who knows nothing about the area you are +updating, because their feedback might be limited to code style and syntax. +Depending on the complexity and impact of a change, assigning the right people +to review your changes might be very important. + +If you don’t know who to assign, consult `git blame` or ask in the `#verify` +Slack channel (GitLab team members only). + +### Incremental rollouts + +After your merge request is merged by a maintainer, it is time to release it to +users and the wider community. We usually do this with feature flags. +While not every merge request needs a feature flag, most merge +requests in Verify should have feature flags. [**TODO** link to docs about what +needs a feature flag and what doesn’t]. + +If you already follow the advice on this page, you probably already have a +few metrics and perhaps a few loggers added that make your new code observable +in the production environment. You can now use these metrics to incrementally +roll out your changes! + +A typical scenario involves enabling a few features in a few internal projects +while observing your metrics or loggers. Be aware that there might be a +small delay involved in ingesting logs in Elastic or Kibana. After you confirm +the feature works well with internal projects you can start an +incremental rollout for other projects. + +Avoid using "percent of time" incremental rollouts. These are error prone, +especially when you are checking feature flags in a few places in the codebase +and you have not memoized the result of a check in a single place. + +### Do not cause our Universe to implode + +During one of the first GitLab Contributes events we had a discussion about the importance +of keeping CI/CD pipeline, stage, and job statuses accurate. We considered a hypothetical +scenario relating to a software being built by one of our [early customers](https://about.gitlab.com/blog/2016/11/23/gitlab-adoption-growing-at-cern/) + +> What happens if software deployed to the [Large Hadron Collider (LHC)](https://en.wikipedia.org/wiki/Large_Hadron_Collider), +> breaks because of a bug in GitLab CI/CD that showed that a pipeline +> passed, but this data was not accurate and the software deployed was actually +> invalid? A problem like this could cause the LHC to malfunction, which +> could generate a new particle that would then cause the universe to implode. + +That would be quite an undesirable outcome of a small bug in GitLab CI/CD status +processing. Please take extra care when you are working on CI/CD statuses, +we don’t want to implode our Universe! + +This is an extreme and unlikely scenario, but presenting data that is not accurate +can potentially cause a myriad of problems through the +[butterfly effect](https://en.wikipedia.org/wiki/Butterfly_effect). +There are much more likely scenarios that +can have disastrous consequences. GitLab CI/CD is being used by companies +building medical, aviation, and automotive software. Continuous Integration is +a mission critical part of software engineering. + +When you are working on a subsystem for pipeline processing and transitioning +CI/CD statuses, request an additional review from a domain expert and hold +others accountable for doing the same. diff --git a/doc/development/dangerbot.md b/doc/development/dangerbot.md index 8da1f5700e5..9bf0fbe1d78 100644 --- a/doc/development/dangerbot.md +++ b/doc/development/dangerbot.md @@ -121,12 +121,13 @@ to revert the change before merging! #### Adding labels via Danger NOTE: -This is currently applicable to the [`gitlab-org/gitlab`](https://gitlab.com/gitlab-org/gitlab) -project only. +This is applicable to all the projects that use the [`gitlab-dangerfiles` gem](https://rubygems.org/gems/gitlab-dangerfiles). Danger is often used to improve MR hygiene by adding labels. Instead of calling the -API directly in your `Dangerfile`, add the labels to the `project_helper.labels_to_add` array. -The main `Dangerfile` will then take care of adding the labels to the MR with a single API call. +API directly in your `Dangerfile`, add the labels to `helper.labels_to_add` array (with `helper.labels_to_add << label` +or `helper.labels_to_add.concat(array_of_labels)`. +`gitlab-dangerfiles` will then take care of adding the labels to the MR with a single API call after all the rules +have had the chance to add to `helper.labels_to_add`. #### Shared rules and plugins @@ -135,11 +136,30 @@ upstreaming them to the [`gitlab-dangerfiles`](https://gitlab.com/gitlab-org/rub #### Enable Danger on a project -To enable the Dangerfile on another existing GitLab project, run the following -extra steps: +To enable the Dangerfile on another existing GitLab project, complete the following steps: -1. Create a [Project access tokens](../user/project/settings/project_access_tokens.md). -1. Add the token as a CI/CD project variable named `DANGER_GITLAB_API_TOKEN`. +1. Add [`gitlab-dangerfiles`](https://rubygems.org/gems/gitlab-dangerfiles) to your `Gemfile`. +1. Create a `Dangerfile` with the following content: + + ```ruby + require_relative "lib/gitlab-dangerfiles" + + Gitlab::Dangerfiles.for_project(self, &:import_defaults) + ``` + +1. Add the following to your CI/CD configuration: + + ```yaml + include: + - project: 'gitlab-org/quality/pipeline-common' + file: + - '/ci/danger-review.yml' + ``` + +1. If your project is in the `gitlab-org` group, you don't need to set up any token as the `DANGER_GITLAB_API_TOKEN` + variable is available at the group level. If not, follow these last steps: + 1. Create a [Project access tokens](../user/project/settings/project_access_tokens.md). + 1. Add the token as a CI/CD project variable named `DANGER_GITLAB_API_TOKEN`. You should add the ~"Danger bot" label to the merge request before sending it for review. diff --git a/doc/development/database/database_reviewer_guidelines.md b/doc/development/database/database_reviewer_guidelines.md index bc18e606f21..9d5e4821c9f 100644 --- a/doc/development/database/database_reviewer_guidelines.md +++ b/doc/development/database/database_reviewer_guidelines.md @@ -26,7 +26,7 @@ For more information on the database review process, check the [database review ## How to apply for becoming a database reviewer -Team members are encouraged to self-identify as database domain experts and add it to their [team profile](https://gitlab.com/gitlab-com/www-gitlab-com/-/blob/master/data/team.yml) +Team members are encouraged to self-identify as database domain experts, and add it to their profile YAML file: ```yaml projects: @@ -34,10 +34,11 @@ projects: - reviewer database ``` -Assign the MR which adds your expertise to the `team.yml` file to a database maintainer -or the [Database Team's Engineering Manager](https://about.gitlab.com/handbook/engineering/development/enablement/database/). +Create the merge request [using the "Database reviewer" template](https://gitlab.com/gitlab-com/www-gitlab-com/-/blob/master/.gitlab/merge_request_templates/Database%20reviewer.md), +adding your expertise your profile YAML file. Assign to a database maintainer or the +[Database Team's Engineering Manager](https://about.gitlab.com/handbook/engineering/development/enablement/database/). -Once the `team.yml` update is merged, the [Reviewer roulette](../code_review.md#reviewer-roulette) +After the `team.yml` update is merged, the [Reviewer roulette](../code_review.md#reviewer-roulette) may recommend you as a database reviewer. ## Resources for database reviewers diff --git a/doc/development/database/loose_foreign_keys.md b/doc/development/database/loose_foreign_keys.md index d08e90683fe..17a825b4812 100644 --- a/doc/development/database/loose_foreign_keys.md +++ b/doc/development/database/loose_foreign_keys.md @@ -50,6 +50,107 @@ we can: NOTE: For this procedure to work, we must register which tables to clean up asynchronously. +## The `scripts/decomposition/generate-loose-foreign-key` + +We built an automation tool to aid migration of foreign keys into loose foreign keys as part of +decomposition effort. It presents existing keys and allows chosen foreign keys to be automatically +converted into loose foreign keys. This ensures consistency between foreign key and loose foreign +key definitions, and ensures that they are properly tested. + +WARNING: +We strongly advise you to use the automation script for swapping any foreign key to a loose foreign key. + +The tool ensures that all aspects of swapping a foreign key are covered. This includes: + +- Creating a migration to remove a foreign key. +- Updating `db/structure.sql` with the new migration. +- Updating `lib/gitlab/database/gitlab_loose_foreign_keys.yml` to add the new loose foreign key. +- Creating or updating a model's specs to ensure that the loose foreign key is properly supported. +- Creating a new branch, commit, push, and creating a merge request on GitLab.com. +- Creating a merge request template with all the necessary details to validate the safety of the foreign key removal. + +The tool is located at `scripts/decomposition/generate-loose-foreign-key`: + +```shell +$ scripts/decomposition/generate-loose-foreign-key -h + +Usage: scripts/decomposition/generate-loose-foreign-key [options] <filters...> + -c, --cross-schema Show only cross-schema foreign keys + -n, --dry-run Do not execute any commands (dry run) + -b, --[no-]branch Create or not a new branch + -r, --[no-]rspec Create or not a rspecs automatically + -m, --milestone MILESTONE Specify custom milestone (current: 14.8) + -h, --help Prints this help +``` + +For the migration of cross-schema foreign keys, we use the `-c` modifier to show the foreign keys +yet to migrate: + +```shell +$ scripts/decomposition/generate-loose-foreign-key -c +Re-creating current test database +Dropped database 'gitlabhq_test_ee' +Dropped database 'gitlabhq_geo_test_ee' +Created database 'gitlabhq_test_ee' +Created database 'gitlabhq_geo_test_ee' + +Showing cross-schema foreign keys (20): + ID | HAS_LFK | FROM | TO | COLUMN | ON_DELETE + 0 | N | ci_builds | projects | project_id | cascade + 1 | N | ci_job_artifacts | projects | project_id | cascade + 2 | N | ci_pipelines | projects | project_id | cascade + 3 | Y | ci_pipelines | merge_requests | merge_request_id | cascade + 4 | N | external_pull_requests | projects | project_id | cascade + 5 | N | ci_sources_pipelines | projects | project_id | cascade + 6 | N | ci_stages | projects | project_id | cascade + 7 | N | ci_pipeline_schedules | projects | project_id | cascade + 8 | N | ci_runner_projects | projects | project_id | cascade + 9 | Y | dast_site_profiles_pipelines | ci_pipelines | ci_pipeline_id | cascade + 10 | Y | vulnerability_feedback | ci_pipelines | pipeline_id | nullify + 11 | N | ci_variables | projects | project_id | cascade + 12 | N | ci_refs | projects | project_id | cascade + 13 | N | ci_builds_metadata | projects | project_id | cascade + 14 | N | ci_subscriptions_projects | projects | downstream_project_id | cascade + 15 | N | ci_subscriptions_projects | projects | upstream_project_id | cascade + 16 | N | ci_sources_projects | projects | source_project_id | cascade + 17 | N | ci_job_token_project_scope_links | projects | source_project_id | cascade + 18 | N | ci_job_token_project_scope_links | projects | target_project_id | cascade + 19 | N | ci_project_monthly_usages | projects | project_id | cascade + +To match FK write one or many filters to match against FROM/TO/COLUMN: +- scripts/decomposition/generate-loose-foreign-key <filter(s)...> +- scripts/decomposition/generate-loose-foreign-key ci_job_artifacts project_id +- scripts/decomposition/generate-loose-foreign-key dast_site_profiles_pipelines +``` + +The command accepts a list of filters to match from, to, or column for the purpose of the foreign key generation. +For example, run this to swap all foreign keys for `ci_job_token_project_scope_links` for the +decomposed database: + +```shell +scripts/decomposition/generate-loose-foreign-key -c ci_job_token_project_scope_links +``` + +To swap only the `source_project_id` of `ci_job_token_project_scope_links` for the decomposed database, run: + +```shell +scripts/decomposition/generate-loose-foreign-key -c ci_job_token_project_scope_links source_project_id +``` + +To swap all the foreign keys (all having `_id` appended), but not create a new branch (only commit +the changes) and not create rspecs, run: + +```shell +scripts/decomposition/generate-loose-foreign-key -c --no-branch --no-rspec _id +``` + +To swap all foreign keys referencing `projects`, but not create a new branch (only commit the +changes), run: + +```shell +scripts/decomposition/generate-loose-foreign-key -c --no-branch projects +``` + ## Example migration and configuration ### Configure the loose foreign key @@ -148,6 +249,67 @@ end At this point, the setup phase is concluded. The deleted `projects` records should be automatically picked up by the scheduled cleanup worker job. +### Remove the loose foreign key + +When the loose foreign key definition is no longer needed (parent table is removed, or FK is restored), +we need to remove the definition from the YAML file and ensure that we don't leave pending deleted +records in the database. + +1. Remove the loose foreign key definition from the config (`config/gitlab_loose_foreign_keys.yml`). +1. Remove the deletion tracking trigger from the parent table (if the parent table is still there). +1. Remove leftover deleted records from the `loose_foreign_keys_deleted_records` table. + +Migration for removing the trigger: + +```ruby +class UnTrackProjectRecordChanges < Gitlab::Database::Migration[1.0] + include Gitlab::Database::MigrationHelpers::LooseForeignKeyHelpers + + enable_lock_retries! + + def up + untrack_record_deletions(:projects) + end + + def down + track_record_deletions(:projects) + end +end +``` + +With the trigger removal, we prevent further records to be inserted in the `loose_foreign_keys_deleted_records` +table however, there is still a chance for having leftover pending records in the table. These records +must be removed with an inline data migration. + +```ruby +class RemoveLeftoverProjectDeletions < Gitlab::Database::Migration[1.0] + disable_ddl_transaction! + + def up + loop do + result = execute <<~SQL + DELETE FROM "loose_foreign_keys_deleted_records" + WHERE + ("loose_foreign_keys_deleted_records"."partition", "loose_foreign_keys_deleted_records"."id") IN ( + SELECT "loose_foreign_keys_deleted_records"."partition", "loose_foreign_keys_deleted_records"."id" + FROM "loose_foreign_keys_deleted_records" + WHERE + "loose_foreign_keys_deleted_records"."fully_qualified_table_name" = 'public.projects' AND + "loose_foreign_keys_deleted_records"."status" = 1 + LIMIT 100 + ) + SQL + + break if result.cmd_tuples == 0 + end + end + + def down + # no-op + end +end +``` + ## Testing The "`it has loose foreign keys`" shared example can be used to test the presence of the `ON DELETE` trigger and the @@ -234,6 +396,12 @@ We considered using these Rails features as an alternative to foreign keys but t 1. These can lead to severe performance degradation as we load all records from PostgreSQL, loop over them in Ruby, and call individual `DELETE` queries. 1. These can miss data as they only cover the case when the `destroy` method is called directly on the model. There are other cases including `delete_all` and cascading deletes from another parent table that could mean these are missed. +For non-trivial objects that need to clean up data outside the +database (for example, object storage) where you might wish to use `dependent: :destroy`, +see alternatives in +[Avoid `dependent: :nullify` and `dependent: :destroy` across +databases](./multiple_databases.md#avoid-dependent-nullify-and-dependent-destroy-across-databases). + ## Risks of loose foreign keys and possible mitigations In general, the loose foreign keys architecture is eventually consistent and diff --git a/doc/development/database/multiple_databases.md b/doc/development/database/multiple_databases.md index 1338e83070f..c9bbf73be55 100644 --- a/doc/development/database/multiple_databases.md +++ b/doc/development/database/multiple_databases.md @@ -32,7 +32,7 @@ If you are using GDK, you can follow the following steps: 1. On the GDK root directory, run: ```shell - gdk config set gitlab.rails.multiple_databases true + gdk config set gitlab.rails.databases.ci.enabled true ``` 1. Open your `gdk.yml`, and confirm that it has the following lines: @@ -40,7 +40,9 @@ If you are using GDK, you can follow the following steps: ```yaml gitlab: rails: - multiple_databases: true + databases: + ci: + enabled: true ``` 1. Reconfigure GDK: @@ -623,10 +625,14 @@ outcomes when we switch to decomposed, because now you have some queries happening outside the transaction and they may be partially applied while the outer transaction fails, which could lead to surprising bugs. -If you need to do some cleanup after a `destroy` you will need to choose -from some of the options above. If all you need to do is cleanup the child -records themselves from PostgreSQL then you could consider using ["loose foreign -keys"](loose_foreign_keys.md). +For non-trivial objects that need to clean up data outside the +database (for example, object storage), we recommend the setting +[`dependent: :restrict_with_error`](https://guides.rubyonrails.org/association_basics.html#options-for-has-one-dependent). +Such objects should be removed explicitly ahead of time. Using `dependent: :restrict_with_error` +ensures that we forbid destroying the parent object if something is not cleaned up. + +If all you need to do is clean up the child records themselves from PostgreSQL, +consider using [loose foreign keys](loose_foreign_keys.md). ## `config/database.yml` diff --git a/doc/development/database/strings_and_the_text_data_type.md b/doc/development/database/strings_and_the_text_data_type.md index a0dda42fdc7..9674deb4603 100644 --- a/doc/development/database/strings_and_the_text_data_type.md +++ b/doc/development/database/strings_and_the_text_data_type.md @@ -253,6 +253,26 @@ class ValidateTextLimitMigration < Gitlab::Database::Migration[1.0] end ``` +## Increasing a text limit constraint on an existing column + +Increasing text limits on existing database columns can be safely achieved by first adding the new limit (with a different name), +and then dropping the previous limit: + +```ruby +class ChangeMaintainerNoteLimitInCiRunner < Gitlab::Database::Migration[1.0] + disable_ddl_transaction! + + def up + add_text_limit :ci_runners, :maintainer_note, 1024, constraint_name: check_constraint_name(:ci_runners, :maintainer_note, 'max_length_1MB') + remove_text_limit :ci_runners, :maintainer_note, constraint_name: check_constraint_name(:ci_runners, :maintainer_note, 'max_length') + end + + def down + # no-op: Danger of failing if there are records with length(maintainer_note) > 255 + end +end +``` + ## Text limit constraints on large tables If you have to clean up a text column for a really [large table](https://gitlab.com/gitlab-org/gitlab/-/blob/master/rubocop/rubocop-migrations.yml#L3) diff --git a/doc/development/database_review.md b/doc/development/database_review.md index 8e217725a17..4b5845992b9 100644 --- a/doc/development/database_review.md +++ b/doc/development/database_review.md @@ -203,6 +203,12 @@ Include in the MR description: - Order columns based on the [Ordering Table Columns](ordering_table_columns.md) guidelines. - Add foreign keys to any columns pointing to data in other tables, including [an index](migration_style_guide.md#adding-foreign-key-constraints). - Add indexes for fields that are used in statements such as `WHERE`, `ORDER BY`, `GROUP BY`, and `JOIN`s. +- New tables and columns are not necessarily risky, but over time some access patterns are inherently + difficult to scale. To identify these risky patterns in advance, we need to document expectations for + access and size. Include in the MR description answers to these questions: + - What is the anticipated growth for the new table over the next 3 months, 6 months, 1 year? What assumptions are these based on? + - How many reads and writes per hour would you expect this table to have in 3 months, 6 months, 1 year? Under what circumstances are rows updated? What assumptions are these based on? + - Based on the anticipated data volume and access patterns, does the new table pose an availability risk to GitLab.com or self-managed instances? Will the proposed design scale to support the needs of GitLab.com and self-managed customers? #### Preparation when removing columns, tables, indexes, or other structures @@ -245,6 +251,10 @@ Include in the MR description: that post migrations are executed post-deployment in production. - Check [timing guidelines for migrations](migration_style_guide.md#how-long-a-migration-should-take) - Check migrations are reversible and implement a `#down` method +- Check new table migrations: + - Are the stated access patterns and volume reasonable? Do the assumptions they're based on seem sound? Do these patterns pose risks to stability? + - Are the columns [ordered to conserve space](ordering_table_columns.md)? + - Are there foreign keys for references to other tables? - Check data migrations: - Establish a time estimate for execution on GitLab.com. - Depending on timing, data migrations can be placed on regular, post-deploy, or background migrations. diff --git a/doc/development/documentation/graphql_styleguide.md b/doc/development/documentation/graphql_styleguide.md index 5acc8bda6a6..ad19a40a3f5 100644 --- a/doc/development/documentation/graphql_styleguide.md +++ b/doc/development/documentation/graphql_styleguide.md @@ -6,7 +6,7 @@ info: "See the Technical Writers assigned to Development Guidelines: https://abo description: "Writing styles, markup, formatting, and other standards for GraphQL API's GitLab Documentation." --- -# GraphQL API +# Creating a GraphQL example page GraphQL APIs are different from [RESTful APIs](restful_api_styleguide.md). Reference information is generated in our [GraphQL reference](../../api/graphql/reference/index.md). diff --git a/doc/development/documentation/index.md b/doc/development/documentation/index.md index 5cf7bb74549..66d6beb821f 100644 --- a/doc/development/documentation/index.md +++ b/doc/development/documentation/index.md @@ -7,7 +7,7 @@ description: Learn how to contribute to GitLab Documentation. # GitLab Documentation guidelines -The GitLab documentation is [intended as the single source of truth (SSOT)](https://about.gitlab.com/handbook/documentation/) for information about how to configure, use, and troubleshoot GitLab. The documentation contains use cases and usage instructions for every GitLab feature, organized by product area and subject. This includes topics and workflows that span multiple GitLab features, and the use of GitLab with other applications. +The GitLab documentation is [intended as the single source of truth (SSOT)](https://about.gitlab.com/handbook/documentation/) for information about how to configure, use, and troubleshoot GitLab. The documentation contains use cases and usage instructions for every GitLab feature, organized by product area and subject. This includes topics and workflows that span multiple GitLab features and the use of GitLab with other applications. In addition to this page, the following resources can help you craft and contribute to documentation: @@ -55,9 +55,9 @@ docs-only merge requests using the following guide: [Contributions to GitLab docs](workflow.md) are welcome from the entire GitLab community. -To ensure that GitLab docs are current, there are special processes and responsibilities for all [feature changes](workflow.md), that is development work that impacts the appearance, usage, or administration of a feature. +To ensure that the GitLab docs are current, there are special processes and responsibilities for all [feature changes](workflow.md), that is development work that impacts the appearance, usage, or administration of a feature. -However, anyone can contribute [documentation improvements](workflow.md) that are not associated with a feature change. For example, adding a new doc on how to accomplish a use case that's already possible with GitLab or with third-party tools and GitLab. +However, anyone can contribute [documentation improvements](workflow.md) that are not associated with a feature change. For example, adding a new document on how to accomplish a use case that's already possible with GitLab or with third-party tools and GitLab. ## Markdown and styles @@ -87,8 +87,8 @@ belongs to, as well as an information block as described below: - `group`: The [Group](https://about.gitlab.com/company/team/structure/#product-groups) to which the majority of the page's content belongs. - `info`: The following line, which provides direction to contributors regarding - how to contact the Technical Writer associated with the page's Stage and - Group: + how to contact the Technical Writer associated with the page's stage and + group: ```plaintext To determine the technical writer assigned to the Stage/Group @@ -116,7 +116,7 @@ The following metadata should be added when a page is moved to another location: location to which visitors should be redirected for a moved page. [Learn more](redirects.md). - `disqus_identifier`: Identifier for Disqus commenting system. Used to keep - comments with a page that's been moved to a new URL. + comments with a page that has been moved to a new URL. [Learn more](redirects.md#redirections-for-pages-with-disqus-comments). ### Comments metadata @@ -192,8 +192,8 @@ For example: 1. The change shows up in the 14.5 self-managed release, due to missing the release cutoff for 14.4. -The exact cutoff date for each release is flexible, and can be earlier or later -than expected due to holidays, weekends, or other events. In general, MRs merged +The exact cutoff date for each release is flexible, and can be sooner or later +than expected due to holidays, weekends or other events. In general, MRs merged by the 17th should be present in the release on the 22nd, though it is not guaranteed. If it is important that a documentation update is present in that month's release, merge it as early as possible. @@ -209,7 +209,7 @@ with the following conventions: - It's relative to the `doc/` directory in the GitLab repository. - It omits the `.md` extension. -- It doesn't end with a slash (`/`). +- It doesn't end with a forward slash (`/`). The help text follows the [Pajamas guidelines](https://design.gitlab.com/usability/helping-users/#formatting-help-content). @@ -316,7 +316,7 @@ process. This is configured in the `Dangerfile` in the GitLab repository under ## Automatic screenshot generator -You can now set up an automatic screenshot generator to take and compress screenshots, with the +You can now set up an automatic screenshot generator to take and compress screenshots with the help of a configuration file known as **screenshot generator**. ### Use the tool diff --git a/doc/development/documentation/redirects.md b/doc/development/documentation/redirects.md index 8f13048f663..4c748924c67 100644 --- a/doc/development/documentation/redirects.md +++ b/doc/development/documentation/redirects.md @@ -27,7 +27,7 @@ Add a redirect to ensure: - Users see the new page and can update or delete their bookmark. - External sites can update their links, especially sites that have automation that - check for redirecting links. + checks for redirected links. - The documentation site global navigation does not link to a missing page. The links in the global navigation are already tested in the `gitlab-docs` project. @@ -38,26 +38,28 @@ Technical Writers can help with any questions and can review your change. There are two types of redirects: -- Redirect added into the documentation files themselves, for users who +- [Redirect added into the documentation files themselves](#add-a-redirect), for users who view the docs in `/help` on self-managed instances. For example, - [`/help` on GitLab.com](https://gitlab.com/help). -- [GitLab Pages redirects](../../user/project/pages/redirects.md), - for users who view the docs on [`docs.gitlab.com`](https://docs.gitlab.com). + [`/help` on GitLab.com](https://gitlab.com/help). These must be added in the same + MR that renames or moves a doc. Redirects to internal pages expire after three months + and redirects to external pages (starting with `https:`) expire after a year. +- [GitLab Pages redirects](../../user/project/pages/redirects.md), which are added + automatically after redirect files expire. They must not be manually added by + contributors and expire after nine months. Redirects pointing to external sites + are not added to the GitLab Pages redirects. - The Technical Writing team manages the [process](https://gitlab.com/gitlab-org/technical-writing/-/blob/main/.gitlab/issue_templates/tw-monthly-tasks.md) - to regularly update and [clean up the redirects](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/doc/raketasks.md#clean-up-redirects). - If you're a contributor, you may add a new redirect, but you don't need to delete - the old ones. This process is automatic and handled by the Technical - Writing team. +Expired redirect files are removed from the documentation projects by the +[`clean_redirects` Rake task](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/doc/raketasks.md#clean-up-redirects), +as part of the Technical Writing team's [monthly tasks](https://gitlab.com/gitlab-org/technical-writing/-/blob/main/.gitlab/issue_templates/tw-monthly-tasks.md). + +## Add a redirect NOTE: -If the old page you're renaming doesn't exist in a stable branch, skip the -following steps and ask a Technical Writer to add the redirect in -[`redirects.yaml`](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/content/_data/redirects.yaml). -For example, if you add a new page on the 3rd of the month and then rename it before it gets -added in the stable branch on the 18th, the old page will never be part of the internal `/help`. -In that case, you can jump straight to the -[Pages redirect](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/doc/maintenance.md#pages-redirects). +If the renamed page is new, you can sometimes skip the following steps and ask a +Technical Writer to manually add the redirect to [`redirects.yaml`](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/content/_data/redirects.yaml). +For example, if you add a new page and then rename it before it's added to a release +on the 18th. The old page is not in any version's `/help` section, so a technical writer +can jump straight to the [Pages redirect](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/doc/maintenance.md#pages-redirects). To add a redirect: @@ -87,20 +89,13 @@ To add a redirect: bundle exec rake "gitlab:docs:redirect[doc/user/search/old_file.md, https://example.com]" ``` - Alternatively, you can omit the arguments and be asked to enter their values: - - ```shell - bundle exec rake gitlab:docs:redirect - ``` - - If you don't want to use the Rake task, you can use the following template. + - Alternatively, you can omit the arguments and be prompted to enter the values: - Replace the value of `redirect_to` with the new file path and `YYYY-MM-DD` - with the date the file should be removed. + ```shell + bundle exec rake gitlab:docs:redirect + ``` - Redirect files that link to docs in internal documentation projects - are removed after three months. Redirect files that link to external sites are - removed after one year: + If you don't want to use the Rake task, you can use the following template: ```markdown --- @@ -111,9 +106,14 @@ To add a redirect: This document was moved to [another location](../path/to/file/index.md). <!-- This redirect file can be deleted after <YYYY-MM-DD>. --> - <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> + <!-- Redirects that point to other docs in the same project expire in three months. --> + <!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> + <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> ``` + - Replace both instances of `../newpath/to/file/index.md` with the new file path. + - Replace `YYYY-MM-DD` with the expiry date, as explained in the template. + 1. If the documentation page being moved has any Disqus comments, follow the steps described in [Redirections for pages with Disqus comments](#redirections-for-pages-with-disqus-comments). 1. Open a merge request with your changes. If a documentation page diff --git a/doc/development/documentation/styleguide/index.md b/doc/development/documentation/styleguide/index.md index 3e9c0177d48..91e9d0c703d 100644 --- a/doc/development/documentation/styleguide/index.md +++ b/doc/development/documentation/styleguide/index.md @@ -655,52 +655,39 @@ This is overridden by the [documentation-specific punctuation rules](#punctuatio ## Headings -- Add only one H1 in each document, by adding `#` at the beginning of - it (when using Markdown). The `h1` becomes the document `<title>`. -- Start with an `h2` (`##`), and respect the order `h2` > `h3` > `h4` > `h5` > `h6`. - Never skip the hierarchy level, such as `h2` > `h4` -- Avoid putting numbers in headings. Numbers shift, hence documentation anchor - links shift too, which eventually leads to dead links. If you think it is - compelling to add numbers in headings, make sure to at least discuss it with - someone in the Merge Request. -- [Avoid using symbols and special characters](https://gitlab.com/gitlab-org/gitlab-docs/-/issues/84) - in headers. Whenever possible, they should be plain and short text. -- When possible, avoid including words that might change in the future. Changing +In the Markdown document: + +- Add one H1 (`#`) at the start of the page. The `h1` becomes the document `<title>`. +- After the H1, follow the order `h2` > `h3` > `h4` > `h5` > `h6`. +- Do not skip a level. For example: `h2` > `h4`. +- Leave one blank line before and after the heading. + +For the heading text, **do**: + +- Be clear and direct. Make every word count. +- Use active verbs for tasks. For example, `Configure GDK` instead of `Configuring GDK`. +- Talk about what the product does, realistically but from a positive perspective. Instead of + `Limitations`, move the content near other similar information. If you must, you can + use the title `Known issues`. +- Use articles and prepositions. +- Add the [product badge](#product-tier-badges) that corresponds to the license tier. +- Follow [capitalization](#capitalization) guidelines. + +For the heading text, **do not**: + +- Use generic words like `Overview` or `Use cases`. Instead, incorporate + the information under a concept heading. +- Use `How it works`. Incorporate this information under a concept, or use a + noun followed by `workflow`. For example, `Merge request workflow`. +- Use `Important Notes`. Incorporate this information closer to where it belongs. +- Use numbers to indicate steps. If the numbers change, the anchor links changes, + which eventually leads to dead links. If you think you must add numbers in headings, + at least discuss it with a writer in the merge request. +- Use words that might change in the future. Changing a heading changes its anchor URL, which affects other linked pages. -- When introducing a new document, be careful for the headings to be - grammatically and syntactically correct. Mention an [assigned technical writer (TW)](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments) - for review, based upon the [product category](https://about.gitlab.com/handbook/product/categories/). - This is to ensure that no document with wrong heading is going live without an - audit, thus preventing dead links and redirection issues when corrected. -- Use the context provided by parent section headings. That is, don't repeat the parent heading's text in each - subsection's heading. -- Use articles and prepositions in headings where it would make sense in regular text. -- Leave exactly one blank line before and after a heading. -- Do not use links in headings. -- Add the corresponding [product badge](#product-tier-badges) according to the tier the - feature belongs. -- Our documentation site search engine prioritizes words used in headings and - subheadings. Make your subheading titles clear, descriptive, and complete to help - users find the right example, as shown in the section on [heading titles](#heading-titles). -- See [Capitalization](#capitalization) for guidelines on capitalizing headings. - -### Heading titles - -Keep heading titles clear and direct. Make every word count. To accommodate -search engine optimization (SEO), use the imperative, where possible. - -| Do | Don't | -|:--------------------------------------|:------------------------------------------------------------| -| Configure GDK | Configuring GDK | -| GitLab Release and Maintenance Policy | This section covers the GitLab Release and Maintenance Policy | -| Backport to older releases | Backporting to older releases | -| GitLab Pages examples | Examples | - -For guidelines on capitalizing headings, see the section on [capitalization](#capitalization). - -NOTE: -If you change an existing title, be careful. In-page [anchor links](#anchor-links), -links in the GitLab application, and links from external sites can break. +- Repeat text from earlier headings. For example, instead of `Troubleshooting merge requests`, + use `Troubleshooting`. +- Use links. ### Anchor links @@ -1130,6 +1117,17 @@ copy of `https://gitlab.com/gitlab-org/gitlab`, run in a terminal: bin/pngquant compress doc/user/img ``` +### Animated images + +Sometimes an image with animation (such as an animated GIF) +can help the reader understand a complicated interaction with the user interface. + +However, you should use them sparingly and avoid them when you can. +Do not use them to replace written descriptions of processes or the product. + +If you include an animated image, follow the same size and naming conventions we use for images. If the animated image loops, add at least a three +second pause to the end of the loop. + ## Videos Adding GitLab YouTube video tutorials to the documentation is highly diff --git a/doc/development/documentation/styleguide/word_list.md b/doc/development/documentation/styleguide/word_list.md index 2c435cdc69d..c38c6586c3a 100644 --- a/doc/development/documentation/styleguide/word_list.md +++ b/doc/development/documentation/styleguide/word_list.md @@ -89,6 +89,14 @@ For example: Do not use title case **GitLab Agent** or **GitLab Agent for Kubernetes**. +## agent access token + +The token generated when you create an agent for Kubernetes. Use **agent access token**, not: + +- registration token +- secret token +- authentication token + ## allow, enable Try to avoid **allow** and **enable**, unless you are talking about security-related features. @@ -174,8 +182,7 @@ See also [contractions](index.md#contractions). Use one word for **checkbox**. Do not use **check box**. -You **select** (not **check** or **enable**) and **clear** (not **deselect** or **disable**) checkboxes. -For example: +You **select** (not **check** or **enable**) and **clear** (not **deselect** or **disable**) checkboxes. For example: - Select the **Protect environment** checkbox. - Clear the **Protect environment** checkbox. @@ -185,6 +192,8 @@ If you must refer to the checkbox, you can say it is selected or cleared. For ex - Ensure the **Protect environment** checkbox is cleared. - Ensure the **Protect environment** checkbox is selected. +(For `deselect`, [Vale](../testing.md#vale) rule: [`SubstitutionWarning.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/SubstitutionWarning.yml)) + ## checkout, check out Use **check out** as a verb. For the Git command, use `checkout`. @@ -201,10 +210,6 @@ CI/CD is always uppercase. No need to spell it out on first use. Use **CI/CD minutes** instead of **CI minutes**, **pipeline minutes**, **pipeline minutes quota**, or **CI pipeline minutes**. This decision was made in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/342813). -## CI/CD tunnel - -Use lowercase for **tunnel** in **CI/CD tunnel**. - ## click Do not use **click**. Instead, use **select** with buttons, links, menu items, and lists. @@ -256,6 +261,11 @@ Do not use **Developer permissions**. A user who is assigned the Developer role See [the Microsoft style guide](https://docs.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/d/disable-disabled) for guidance on **disable**. Use **inactive** or **off** instead. ([Vale](../testing.md#vale) rule: [`InclusionAbleism.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/InclusionAbleism.yml)) + +## disallow + +Use **prevent** instead of **disallow**. ([Vale](../testing.md#vale) rule: [`Substitutions.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/Substitutions.yml)) + ## dropdown list Use **dropdown list** to refer to the UI element. Do not use **dropdown** without **list** after it. @@ -667,6 +677,10 @@ Do not use [**roles**](#roles) and **permissions** interchangeably. Each user is Permissions are not the same as [**access levels**](#access-level). +## personal access token + +Use lowercase for **personal access token**. + ## please Do not use **please**. For details, see the [Microsoft style guide](https://docs.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/p/please). @@ -951,6 +965,17 @@ One exception: You can use **we recommend** instead of **it is recommended** or Do not use **whitelist**. Another option is **allowlist**. ([Vale](../testing.md#vale) rule: [`InclusionCultural.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/InclusionCultural.yml)) +## yet + +Do not use **yet** when talking about the product or its features. The documentation describes the product as it is today. + +Sometimes you might need to use **yet** when writing a task. If you use +**yet**, ensure the surrounding phrases are written +in present tense, active voice. + +[View guidance about how to write about future features](index.md#promising-features-in-future-versions). +([Vale](../testing.md#vale) rule: [`CurrentStatus.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/CurrentStatus.yml)) + ## you, your, yours Use **you**, **your**, and **yours** instead of [**the user** and **the user's**](#user-users). diff --git a/doc/development/documentation/testing.md b/doc/development/documentation/testing.md index 905b44823c3..49fe0aff3c6 100644 --- a/doc/development/documentation/testing.md +++ b/doc/development/documentation/testing.md @@ -128,7 +128,7 @@ and you should make sure your version matches the version used by GitLab. ## Update linter configuration -[Vale configuration](#vale) and [markdownlint configuration](#markdownlint) is under source control in each +[Vale configuration](#vale) and [markdownlint configuration](#markdownlint) is under source control in each project, so updates must be committed to each project individually. We consider the configuration in the `gitlab` project as the source of truth and that's where all updates should @@ -213,19 +213,19 @@ You can use Vale: #### Vale result types -Vale returns three types of results: `suggestion`, `warning`, and `error`: +Vale returns three types of results: -- **Suggestion**-level results are writing tips and aren't displayed in CI - job output. Suggestions don't break CI. See a list of - [suggestion-level rules](https://gitlab.com/search?utf8=✓&snippets=false&scope=&repository_ref=master&search=path%3Adoc%2F.vale%2Fgitlab+Suggestion%3A&group_id=9970&project_id=278964). -- **Warning**-level results are [Style Guide](styleguide/index.md) violations, aren't displayed in CI - job output, and should contain clear explanations of how to resolve the warning. - Warnings may be technical debt, or can be future error-level test items - (after the Technical Writing team completes its cleanup). Warnings don't break CI. See a list of - [warning-level rules](https://gitlab.com/search?utf8=✓&snippets=false&scope=&repository_ref=master&search=path%3Adoc%2F.vale%2Fgitlab+Warning%3A&group_id=9970&project_id=278964). -- **Error**-level results are Style Guide violations, and should contain clear explanations - of how to resolve the error. Errors break CI and are displayed in CI job output. See a list of - [error-level rules](https://gitlab.com/search?utf8=✓&snippets=false&scope=&repository_ref=master&search=path%3Adoc%2F.vale%2Fgitlab+Error%3A&group_id=9970&project_id=278964). +- **Error** - For branding and trademark issues, and words or phrases with ambiguous meanings. +- **Warning** - For Technical Writing team style preferences. +- **Suggestion** - For basic technical writing tenets and best practices. + +The result types have these attributes: + +| Result type | Displayed in CI/CD job output | Causes CI/CD jobs to fail | Vale rule link | +|--------------|-------------------------------|---------------------------|----------------| +| `error` | **{check-circle}** Yes | **{check-circle}** Yes | [Error-level Vale rules](https://gitlab.com/search?utf8=✓&snippets=false&scope=&repository_ref=master&search=path%3Adoc%2F.vale%2Fgitlab+Error%3A&group_id=9970&project_id=278964) | +| `warning` | **{dotted-circle}** No | **{dotted-circle}** No | [Warning-level Vale rules](https://gitlab.com/search?utf8=✓&snippets=false&scope=&repository_ref=master&search=path%3Adoc%2F.vale%2Fgitlab+Warning%3A&group_id=9970&project_id=278964) | +| `suggestion` | **{dotted-circle}** No | **{dotted-circle}** No | [Suggestion-level Vale rules](https://gitlab.com/search?utf8=✓&snippets=false&scope=&repository_ref=master&search=path%3Adoc%2F.vale%2Fgitlab+Suggestion%3A&group_id=9970&project_id=278964) | #### Vale spelling test @@ -270,12 +270,10 @@ build pipelines: [used (see `variables:` section)](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/.gitlab-ci.yml) when building the `image:docs-lint-markdown`. -1. Install [`vale`](https://github.com/errata-ai/vale/releases). For example, to install using - `brew` for macOS, run: +1. Install [`vale`](https://github.com/errata-ai/vale/releases). To install for: - ```shell - brew install vale - ``` + - macOS using `brew`, run: `brew install vale`. + - Linux, use your distribution's package manager or a [released binary](https://github.com/errata-ai/vale/releases). These tools can be [integrated with your code editor](#configure-editors). @@ -313,6 +311,9 @@ To configure Vale in your editor, install one of the following as appropriate: - Visual Studio Code [`errata-ai.vale-server` extension](https://marketplace.visualstudio.com/items?itemName=errata-ai.vale-server). You can configure the plugin to [display only a subset of alerts](#show-subset-of-vale-alerts). - Vim [ALE plugin](https://github.com/dense-analysis/ale). +- Jetbrains IDEs - No plugin exists, but + [this issue comment](https://github.com/errata-ai/vale-server/issues/39#issuecomment-751714451) + contains tips for configuring an external tool. - Emacs [Flycheck extension](https://github.com/flycheck/flycheck). This requires some configuration: diff --git a/doc/development/ee_features.md b/doc/development/ee_features.md index 17e35d34ec7..5bd830715f5 100644 --- a/doc/development/ee_features.md +++ b/doc/development/ee_features.md @@ -16,6 +16,13 @@ info: To determine the technical writer assigned to the Stage/Group associated w [EE features list](https://about.gitlab.com/features/). <!-- markdownlint-enable MD044 --> +## Act as SaaS + +When developing locally, there are times when you need your instance to act like the SaaS version of the product. +In those instances, you can simulate SaaS by exporting an environment variable as seen below: + +`export GITLAB_SIMULATE_SAAS=1` + ## Act as CE when unlicensed Since the implementation of @@ -437,6 +444,69 @@ resolve when you add the indentation to the equation. EE-specific views should be placed in `ee/app/views/`, using extra sub-directories if appropriate. +#### Using `render_if_exists` + +Instead of using regular `render`, we should use `render_if_exists`, which +doesn't render anything if it cannot find the specific partial. We use this +so that we could put `render_if_exists` in CE, keeping code the same between +CE and EE. + +The advantages of this: + +- Very clear hints about where we're extending EE views while reading CE code. + +The disadvantage of this: + +- If we have typos in the partial name, it would be silently ignored. + +##### Caveats + +The `render_if_exists` view path argument must be relative to `app/views/` and `ee/app/views`. +Resolving an EE template path that is relative to the CE view path doesn't work. + +```haml +- # app/views/projects/index.html.haml + += render_if_exists 'button' # Will not render `ee/app/views/projects/_button` and will quietly fail += render_if_exists 'projects/button' # Will render `ee/app/views/projects/_button` +``` + +#### Using `render_ce` + +For `render` and `render_if_exists`, they search for the EE partial first, +and then CE partial. They would only render a particular partial, not all +partials with the same name. We could take the advantage of this, so that +the same partial path (for example, `shared/issuable/form/default_templates`) could +be referring to the CE partial in CE (that is, +`app/views/shared/issuable/form/_default_templates.html.haml`), while EE +partial in EE (that is, +`ee/app/views/shared/issuable/form/_default_templates.html.haml`). This way, +we could show different things between CE and EE. + +However sometimes we would also want to reuse the CE partial in EE partial +because we might just want to add something to the existing CE partial. We +could workaround this by adding another partial with a different name, but it +would be tedious to do so. + +In this case, we could as well just use `render_ce` which would ignore any EE +partials. One example would be +`ee/app/views/shared/issuable/form/_default_templates.html.haml`: + +```haml +- if @project.feature_available?(:issuable_default_templates) + = render_ce 'shared/issuable/form/default_templates' +- elsif show_promotions? + = render 'shared/promotions/promote_issue_templates' +``` + +In the above example, we can't use +`render 'shared/issuable/form/default_templates'` because it would find the +same EE partial, causing infinite recursion. Instead, we could use `render_ce` +so it ignores any partials in `ee/` and then it would render the CE partial +(that is, `app/views/shared/issuable/form/_default_templates.html.haml`) +for the same path (that is, `shared/issuable/form/default_templates`). This way +we could easily wrap around the CE partial. + ### Code in `lib/gitlab/background_migration/` When you create EE-only background migrations, you have to plan for users that @@ -518,69 +588,6 @@ module EE end ``` -#### Using `render_if_exists` - -Instead of using regular `render`, we should use `render_if_exists`, which -doesn't render anything if it cannot find the specific partial. We use this -so that we could put `render_if_exists` in CE, keeping code the same between -CE and EE. - -The advantages of this: - -- Very clear hints about where we're extending EE views while reading CE code. - -The disadvantage of this: - -- If we have typos in the partial name, it would be silently ignored. - -##### Caveats - -The `render_if_exists` view path argument must be relative to `app/views/` and `ee/app/views`. -Resolving an EE template path that is relative to the CE view path doesn't work. - -```haml -- # app/views/projects/index.html.haml - -= render_if_exists 'button' # Will not render `ee/app/views/projects/_button` and will quietly fail -= render_if_exists 'projects/button' # Will render `ee/app/views/projects/_button` -``` - -#### Using `render_ce` - -For `render` and `render_if_exists`, they search for the EE partial first, -and then CE partial. They would only render a particular partial, not all -partials with the same name. We could take the advantage of this, so that -the same partial path (for example, `shared/issuable/form/default_templates`) could -be referring to the CE partial in CE (that is, -`app/views/shared/issuable/form/_default_templates.html.haml`), while EE -partial in EE (that is, -`ee/app/views/shared/issuable/form/_default_templates.html.haml`). This way, -we could show different things between CE and EE. - -However sometimes we would also want to reuse the CE partial in EE partial -because we might just want to add something to the existing CE partial. We -could workaround this by adding another partial with a different name, but it -would be tedious to do so. - -In this case, we could as well just use `render_ce` which would ignore any EE -partials. One example would be -`ee/app/views/shared/issuable/form/_default_templates.html.haml`: - -```haml -- if @project.feature_available?(:issuable_default_templates) - = render_ce 'shared/issuable/form/default_templates' -- elsif show_promotions? - = render 'shared/promotions/promote_issue_templates' -``` - -In the above example, we can't use -`render 'shared/issuable/form/default_templates'` because it would find the -same EE partial, causing infinite recursion. Instead, we could use `render_ce` -so it ignores any partials in `ee/` and then it would render the CE partial -(that is, `app/views/shared/issuable/form/_default_templates.html.haml`) -for the same path (that is, `shared/issuable/form/default_templates`). This way -we could easily wrap around the CE partial. - ### Code in `lib/` Place EE-specific logic in the top-level `EE` module namespace. Namespace the diff --git a/doc/development/emails.md b/doc/development/emails.md index 5361282334f..b8e390988bd 100644 --- a/doc/development/emails.md +++ b/doc/development/emails.md @@ -13,6 +13,9 @@ If a mailer argument needs to be added or removed, it is important to ensure both backward and forward compatibility. Adhere to the Sidekiq steps for [changing the arguments for a worker](sidekiq/compatibility_across_updates.md#changing-the-arguments-for-a-worker). +The same applies to a new mailer method, or a new mailer. If you introduce either, +follow the steps for [adding new workers](sidekiq/compatibility_across_updates.md#adding-new-workers). + In the following example from [`NotificationService`](https://gitlab.com/gitlab-org/gitlab/-/blob/33ccb22e4fc271dbaac94b003a7a1a2915a13441/app/services/notification_service.rb#L74) adding or removing an argument in this mailer's definition may cause problems during deployment before all Rails and Sidekiq nodes have the updated code. diff --git a/doc/development/event_store.md b/doc/development/event_store.md index c6d553f41cd..b00a824e2eb 100644 --- a/doc/development/event_store.md +++ b/doc/development/event_store.md @@ -262,7 +262,7 @@ module Gitlab end ``` -A worker that is only defined in the EE codebase can subscribe to an event in the same way by +A worker that is only defined in the EE codebase can subscribe to an event in the same way by declaring the subscription in `ee/lib/ee/gitlab/event_store.rb`. Subscriptions are stored in memory when the Rails app is loaded and they are immediately frozen. diff --git a/doc/development/experiment_guide/experimentation.md b/doc/development/experiment_guide/experimentation.md index bb782af80cc..28100564555 100644 --- a/doc/development/experiment_guide/experimentation.md +++ b/doc/development/experiment_guide/experimentation.md @@ -6,4 +6,6 @@ remove_date: '2022-04-13' This document was moved to [another location](gitlab_experiment.md). <!-- This redirect file can be deleted after <2022-04-13>. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/experiment_guide/gitlab_experiment.md b/doc/development/experiment_guide/gitlab_experiment.md index 369690ba86c..78e1f84d701 100644 --- a/doc/development/experiment_guide/gitlab_experiment.md +++ b/doc/development/experiment_guide/gitlab_experiment.md @@ -317,7 +317,7 @@ of tracking an event in Ruby would be: experiment(:pill_color, actor: current_user).track(:clicked) ``` -When you run an experiment with any of the examples so far, an `:assigned` event +When you run an experiment with any of the examples so far, an `:assignment` event is tracked automatically by default. All events that are tracked from an experiment have a special [experiment context](https://gitlab.com/gitlab-org/iglu/-/blob/master/public/schemas/com.gitlab/gitlab_experiment/jsonschema/1-0-3) @@ -448,7 +448,7 @@ The first way is simply by running the experiment. Assuming the experiment has b The second way doesn't run the experiment and is intended to be used if the experiment only needs to surface in the client layer. To accomplish this we can simply `.publish` the experiment. This won't run any logic, but does surface the experiment details in the client layer so they can be utilized there. -An example might be to publish an experiment in a `before_action` in a controller. Assuming we've defined the `PillColorExperiment` class, like we have above, we can surface it to the client by publishing it instead of running it: +An example might be to publish an experiment in a `before_action` in a controller. Assuming we've defined the `PillColorExperiment` class, like we have above, we can surface it to the client by publishing it instead of running it: ```ruby before_action -> { experiment(:pill_color).publish }, only: [:show] diff --git a/doc/development/experiment_guide/index.md b/doc/development/experiment_guide/index.md index 254c136ef79..c34e5eb36dc 100644 --- a/doc/development/experiment_guide/index.md +++ b/doc/development/experiment_guide/index.md @@ -64,3 +64,15 @@ We recommend the following workflow: 1. **If the experiment is a success**, designers add the new icon or illustration to the Pajamas UI kit as part of the cleanup process. Engineers can then add it to the [SVG library](https://gitlab-org.gitlab.io/gitlab-svgs/) and modify the implementation based on the [Frontend Development Guidelines](../fe_guide/icons.md#usage-in-hamlrails-2). + +## Turn off all experiments + +When there is a case on GitLab.com (SaaS) that necessitates turning off all experiments, we have this control. + +You can toggle experiments on SaaS on and off using the `gitlab_experiment` [feature flag](../feature_flags). + +This can be done via chatops: + +- [disable](../feature_flags/controls.md#disabling-feature-flags): `/chatops run feature set gitlab_experiment false` +- [enable](../feature_flags/controls.md#process): `/chatops run feature delete gitlab_experiment` + - This allows the `default_enabled` [value of true in the yml](https://gitlab.com/gitlab-org/gitlab/-/blob/016430f6751b0c34abb24f74608c80a1a8268f20/config/feature_flags/ops/gitlab_experiment.yml#L8) to be honored. diff --git a/doc/development/export_csv.md b/doc/development/export_csv.md index ff827023a50..998e5b1fb3b 100644 --- a/doc/development/export_csv.md +++ b/doc/development/export_csv.md @@ -1,7 +1,7 @@ --- stage: Manage group: Import -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Export to CSV diff --git a/doc/development/fe_guide/content_editor.md b/doc/development/fe_guide/content_editor.md index 139825655e9..2e64f52651e 100644 --- a/doc/development/fe_guide/content_editor.md +++ b/doc/development/fe_guide/content_editor.md @@ -47,7 +47,7 @@ The Content Editor requires two properties: - `renderMarkdown` is an asynchronous function that returns the response (String) of invoking the [Markdown API](../../api/markdown.md). -- `uploadsPath` is a URL that points to a [GitLab upload service](../uploads.md#upload-encodings) +- `uploadsPath` is a URL that points to a [GitLab upload service](../uploads/implementation.md#upload-encodings) with `multipart/form-data` support. See the [`WikiForm.vue`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/javascripts/pages/shared/wikis/components/wiki_form.vue#L207) diff --git a/doc/development/fe_guide/icons.md b/doc/development/fe_guide/icons.md index 3f7490b0221..d107af156db 100644 --- a/doc/development/fe_guide/icons.md +++ b/doc/development/fe_guide/icons.md @@ -88,50 +88,36 @@ Please use the following function inside JS to render an icon: ### Usage in HAML/Rails -To insert a loading spinner in HAML or Rails use the `loading_icon` helper: +To insert a loading spinner in HAML or Rails use the `gl_loading_icon` helper: ```haml -= loading_icon += gl_loading_icon ``` -You can include one or more of the following properties with the `loading_icon` helper, as demonstrated +You can include one or more of the following properties with the `gl_loading_icon` helper, as demonstrated by the examples that follow: -- `container` (optional): wraps the loading icon in a container, which centers the loading icon using the `text-center` CSS property. -- `color` (optional): either `orange` (default), `light`, or `dark`. +- `inline` (optional): uses in an inline element if `true`, otherwise, a block element (default), with the spinner centered. +- `color` (optional): either `dark` (default) or `light`. - `size` (optional): either `sm` (default), `md`, `lg`, or `xl`. -- `css_class` (optional): defaults to an empty string, but can be used for utility classes to fine-tune alignment or spacing. +- `css_class` (optional): defaults to nothing, but can be used for utility classes to fine-tune alignment or spacing. **Example 1:** The following HAML expression generates a loading icon's markup and -centers the icon by wrapping it in a `gl-spinner-container` element. +centers the icon. ```haml -= loading_icon(container: true) -``` - -**Output from example 1:** - -```html -<div class="gl-spinner-container"> - <span class="gl-spinner gl-spinner-orange gl-spinner-sm" aria-label="Loading"></span> -</div> += gl_loading_icon ``` **Example 2:** -The following HAML expression generates a loading icon's markup +The following HAML expression generates an inline loading icon's markup with a custom size. It also appends a margin utility class. ```haml -= loading_icon(size: 'lg', css_class: 'gl-mr-2') -``` - -**Output from example 2:** - -```html -<span class="gl-spinner gl-spinner-orange gl-spinner-lg gl-mr-2" aria-label="Loading"></span> += gl_loading_icon(inline: true, size: 'lg', css_class: 'gl-mr-2') ``` ### Usage in Vue diff --git a/doc/development/fe_guide/style/javascript.md b/doc/development/fe_guide/style/javascript.md index 4a0923ebe19..d04d1879476 100644 --- a/doc/development/fe_guide/style/javascript.md +++ b/doc/development/fe_guide/style/javascript.md @@ -136,7 +136,7 @@ the class name with `js-`. ## ES Module Syntax -For most JavaScript files, use ES module syntax to import or export from modules. +For most JavaScript files, use ES module syntax to import or export from modules. Prefer named exports, as they improve name consistency. ```javascript diff --git a/doc/development/fe_guide/vue3_migration.md b/doc/development/fe_guide/vue3_migration.md index 6e994d5e95d..f174408c946 100644 --- a/doc/development/fe_guide/vue3_migration.md +++ b/doc/development/fe_guide/vue3_migration.md @@ -6,12 +6,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Migration to Vue 3 -Preparations for a Vue 3 migration are tracked in epic [&3174](https://gitlab.com/groups/gitlab-org/-/epics/3174) +The migration from Vue 2 to 3 is tracked in epic [&6252](https://gitlab.com/groups/gitlab-org/-/epics/6252). -In order to prepare for the eventual migration to Vue 3.x, we should not use the following deprecated features in the codebase: - -NOTE: -Our linting rules block the use of these deprecated features. +To ease migration to Vue 3.x, we have added [eslint rules](https://gitlab.com/gitlab-org/frontend/eslint-plugin/-/merge_requests/50) +that prevent us from using the following deprecated features in the codebase. ## Vue filters diff --git a/doc/development/feature_flags/controls.md b/doc/development/feature_flags/controls.md index 6bf5be23ace..f8f03773c12 100644 --- a/doc/development/feature_flags/controls.md +++ b/doc/development/feature_flags/controls.md @@ -72,8 +72,8 @@ group. To enable a feature for 25% of the time, run the following in Slack: ```shell -/chatops run feature set new_navigation_bar 25 --dev -/chatops run feature set new_navigation_bar 25 --staging +/chatops run feature set new_navigation_bar 25 --random --dev +/chatops run feature set new_navigation_bar 25 --random --staging ``` ### Enabling a feature for GitLab.com @@ -121,7 +121,7 @@ command you make so people can understand the change if they need to. To enable a feature for 25% of the time, run the following in Slack: ```shell -/chatops run feature set new_navigation_bar 25 +/chatops run feature set new_navigation_bar 25 --random ``` This sets a feature flag to `true` based on the following formula: diff --git a/doc/development/feature_flags/index.md b/doc/development/feature_flags/index.md index af402713f6e..4b417b26381 100644 --- a/doc/development/feature_flags/index.md +++ b/doc/development/feature_flags/index.md @@ -530,8 +530,9 @@ Feature.remove(:feature_flag_name) ## Feature flags in tests Introducing a feature flag into the codebase creates an additional code path that should be tested. -It is strongly advised to test all code affected by a feature flag, both when **enabled** and **disabled** -to ensure the feature works properly. +It is strongly advised to include automated tests for all code affected by a feature flag, both when **enabled** and **disabled** +to ensure the feature works properly. If automated tests are not included for both states, the functionality associated +with the untested code path should be manually tested before deployment to production. When using the testing environment, all feature flags are enabled by default. diff --git a/doc/development/feature_flags/process.md b/doc/development/feature_flags/process.md index 3fbb207a12b..f98366beb6b 100644 --- a/doc/development/feature_flags/process.md +++ b/doc/development/feature_flags/process.md @@ -6,4 +6,6 @@ remove_date: '2022-03-01' This document was moved to [another location](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/). <!-- This redirect file can be deleted after 2022-03-01. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/features_inside_dot_gitlab.md b/doc/development/features_inside_dot_gitlab.md index 283a0d5d5fb..7b11b541b5a 100644 --- a/doc/development/features_inside_dot_gitlab.md +++ b/doc/development/features_inside_dot_gitlab.md @@ -12,7 +12,7 @@ When implementing new features, please refer to these existing features to avoid - [Custom Dashboards](../operations/metrics/dashboards/index.md#add-a-new-dashboard-to-your-project): `.gitlab/dashboards/`. - [Issue Templates](../user/project/description_templates.md#create-an-issue-template): `.gitlab/issue_templates/`. - [Merge request Templates](../user/project/description_templates.md#create-a-merge-request-template): `.gitlab/merge_request_templates/`. -- [GitLab Agent](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/configuration_repository.md#layout): `.gitlab/agents/`. +- [GitLab agent](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/configuration_repository.md#layout): `.gitlab/agents/`. - [CODEOWNERS](../user/project/code_owners.md#set-up-code-owners): `.gitlab/CODEOWNERS`. - [Route Maps](../ci/review_apps/#route-maps): `.gitlab/route-map.yml`. - [Customize Auto DevOps Helm Values](../topics/autodevops/customize.md#customize-values-for-helm-chart): `.gitlab/auto-deploy-values.yaml`. diff --git a/doc/development/file_storage.md b/doc/development/file_storage.md index d161206f44d..04e2f381c97 100644 --- a/doc/development/file_storage.md +++ b/doc/development/file_storage.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w We use the [CarrierWave](https://github.com/carrierwaveuploader/carrierwave) gem to handle file upload, store and retrieval. -File uploads should be accelerated by workhorse, for details please refer to [uploads development documentation](uploads.md). +File uploads should be accelerated by workhorse, for details please refer to [uploads development documentation](uploads/index.md). There are many places where file uploading is used, according to contexts: diff --git a/doc/development/fips_compliance.md b/doc/development/fips_compliance.md index 0b6d1751668..8fe5af56f9d 100644 --- a/doc/development/fips_compliance.md +++ b/doc/development/fips_compliance.md @@ -69,55 +69,10 @@ installation instructions, including the [advanced instructions for RHEL](https: Note that `asdf` is not used for dependency management because it's essential to use the RedHat-provided Go compiler and other system dependencies. -### Working around broken frontend asset compilation - -A known bug affects asset compilation with FIPS mode enabled: [issue #322883](https://gitlab.com/gitlab-org/gitlab/-/issues/322883). -Until this is resolved, working on frontend issues is not feasible. We can still -work on backend issues by compiling the assets while FIPS is disabled, and -placing GDK into [static asset mode](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/configuration.md#webpack-settings): - -1. Modify your `gdk.yml` to contain the following: - - ```yaml - webpack: - host: 127.0.0.1 - port: 3808 - static: true - ``` - -1. In the GitLab repository, apply this patch to prevent the assets from being - automatically deleted whenever GDK is restarted: - - ```diff - diff --git a/scripts/frontend/webpack_dev_server.js b/scripts/frontend/webpack_dev_server.js - index fbb80c9617d..114720d457c 100755 - --- a/scripts/frontend/webpack_dev_server.js - +++ b/scripts/frontend/webpack_dev_server.js - @@ -15,7 +15,7 @@ const baseConfig = { - // run webpack in compile-once mode and watch for changes - if (STATIC_MODE) { - nodemon({ - - exec: `rm -rf public/assets/webpack ; yarn run webpack && exec ruby -run -e httpd public/ -p ${DEV_SERVER_PORT}`, - + exec: `ruby -run -e httpd public/ -p ${DEV_SERVER_PORT}`, - watch: [ - 'config/webpack.config.js', - 'app/assets/javascripts', - ``` - -1. Run this command in the GitLab repository to generate the asset files - to be served: - - ```shell - bin/rails gitlab:assets:compile - ``` - -Every time you change a frontend asset, you must re-run this command -(with FIPS mode disabled) before seeing the changes. - ### Enable FIPS mode -After the assets are generated, run this command (as root) and restart the -virtual machine: +After GDK and its dependencies are installed, run this command (as +root) and restart the virtual machine: ```shell fips-mode-setup --enable diff --git a/doc/development/foreign_keys.md b/doc/development/foreign_keys.md index a9edbc68a2e..db8367fe5f5 100644 --- a/doc/development/foreign_keys.md +++ b/doc/development/foreign_keys.md @@ -80,6 +80,12 @@ foreign keys to remove the data as this would result in the file system data being left behind. In such a case you should use a service class instead that takes care of removing non database data. +In cases where the relation spans multiple databases you will have even +further problems using `dependent: :destroy` or the above hooks. You can +read more about alternatives at [Avoid `dependent: :nullify` and +`dependent: :destroy` across +databases](database/multiple_databases.md#avoid-dependent-nullify-and-dependent-destroy-across-databases). + ## Alternative primary keys with has_one associations Sometimes a `has_one` association is used to create a one-to-one relationship: diff --git a/doc/development/geo.md b/doc/development/geo.md index 9f5fd674d38..f37901754aa 100644 --- a/doc/development/geo.md +++ b/doc/development/geo.md @@ -438,3 +438,61 @@ old method: If you want to add easy Geo replication of a resource you're working on, check out our [self-service framework](geo/framework.md). + +## Geo development workflow + +### GET:Geo pipeline + +As part of the [package-and-qa](testing_guide/end_to_end/index.md#using-the-package-and-qa-job) pipeline, there is an option to manually trigger a job named `GET:Geo`. This +pipeline uses [GET](https://gitlab.com/gitlab-org/gitlab-environment-toolkit) to spin up a +[1k](../administration/reference_architectures/1k_users.md) Geo installation, +and run the [`gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa) Geo scenario against the instance. +When working on Geo features, it is a good idea to ensure the `qa-geo` job passes in a triggered `GET:Geo pipeline`. + +The pipelines that control the provisioning and teardown of the instance are included in The GitLab Environment Toolkit Configs +[Geo subproject](https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit-configs/Geo). + +When adding new functionality, consider adding new tests to verify the behavior. For steps, +see the [QA documentation](https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa#writing-tests). + +#### Architecture + +The pipeline involves the interaction of multiple different projects: + +- [GitLab](https://gitlab.com/gitlab-org/gitlab) - The [package-and-qa job](testing_guide/end_to_end/index.md#using-the-package-and-qa-job) is launched from merge requests in this project. +- [`omnibus-gitlab`](https://gitlab.com/gitlab-org/omnibus-gitlab) - Builds relevant artifacts containing the changes from the triggering merge request pipeline. +- [GET-Configs/Geo](https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit-configs/Geo) - Coordinates the lifecycle of a short-lived Geo installation that can be evaluated. +- [GET](https://gitlab.com/gitlab-org/gitlab-environment-toolkit) - Contains the necessary logic for creating and destroying Geo installations. Used by `GET-Configs/Geo`. +- [`gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa) - Tool for running automated tests against a GitLab instance. + +```mermaid +flowchart TD; + GET:Geo-->getcg + Provision-->Terraform + Configure-->Ansible + Geo-->Ansible + QA-->gagq + + subgraph "omnibus-gitlab-mirror" + GET:Geo + end + + subgraph getcg [GitLab-environment-toolkit-configs/Geo] + direction LR + Generate-terraform-config-->Provision + Provision-->Generate-ansible-config + Generate-ansible-config-->Configure + Configure-->Geo + Geo-->QA + QA-->Destroy-geo + end + + subgraph get [GitLab Environment Toolkit] + Terraform + Ansible + end + + subgraph GitLab QA + gagq[GitLab QA Geo Scenario] + end +``` diff --git a/doc/development/gitlab_diagram_overview.odg b/doc/development/gitlab_diagram_overview.odg Binary files differdeleted file mode 100644 index 9bfc7313ff4..00000000000 --- a/doc/development/gitlab_diagram_overview.odg +++ /dev/null diff --git a/doc/development/go_guide/go_upgrade.md b/doc/development/go_guide/go_upgrade.md index 889849799bc..a99253b9723 100644 --- a/doc/development/go_guide/go_upgrade.md +++ b/doc/development/go_guide/go_upgrade.md @@ -134,7 +134,7 @@ if you need help finding the correct person or labels: | GitLab Compose Kit | [Issuer Tracker](https://gitlab.com/gitlab-org/gitlab-compose-kit/-/issues) | | GitLab Container Registry | [Issue Tracker](https://gitlab.com/gitlab-org/container-registry) | | GitLab Elasticsearch Indexer | [Issue Tracker](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer/-/issues) | -| GitLab Agent Server (KAS) | [Issue Tracker](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/issues) | +| GitLab agent server for Kubernetes (KAS) | [Issue Tracker](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/issues) | | GitLab Pages | [Issue Tracker](https://gitlab.com/gitlab-org/gitlab-pages/-/issues) | | GitLab Quality Images | [Issue Tracker](https://gitlab.com/gitlab-org/gitlab-build-images/-/issues) | | GitLab Shell | [Issue Tracker](https://gitlab.com/gitlab-org/gitlab-shell/-/issues) | diff --git a/doc/development/gotchas.md b/doc/development/gotchas.md index fbb6b0219aa..d89dbbcf904 100644 --- a/doc/development/gotchas.md +++ b/doc/development/gotchas.md @@ -166,6 +166,18 @@ end Since Active Record is not calling the `.new` method on model classes to instantiate the objects, you should use `expect_next_found_instance_of` or `allow_next_found_instance_of` mock helpers to setup mock on objects returned by Active Record query & finder methods._ +It is also possible to set mocks and expectations for multiple instances of the same Active Record model by using the `expect_next_found_(number)_instances_of` and `allow_next_found_(number)_instances_of` helpers, like so; + +```ruby +expect_next_found_2_instances_of(Project) do |project| + expect(project).to receive(:add_import_job) +end + +allow_next_found_2_instances_of(Project) do |project| + allow(project).to receive(:add_import_job) +end +``` + If we also want to initialize the instance with some particular arguments, we could also pass it like: diff --git a/doc/development/i18n/proofreader.md b/doc/development/i18n/proofreader.md index 76ab00eebfb..7c9777527ef 100644 --- a/doc/development/i18n/proofreader.md +++ b/doc/development/i18n/proofreader.md @@ -98,9 +98,11 @@ are very appreciative of the work done by translators and proofreaders! - Paulo George Gomes Bezerra - [GitLab](https://gitlab.com/paulobezerra), [Crowdin](https://crowdin.com/profile/paulogomes.rep) - André Gama - [GitLab](https://gitlab.com/andregamma), [Crowdin](https://crowdin.com/profile/ToeOficial) - Eduardo Addad de Oliveira - [GitLab](https://gitlab.com/eduardoaddad), [Crowdin](https://crowdin.com/profile/eduardoaddad) + - Horberlan Brito - [GitLab](https://gitlab.com/horberlan), [Crowdin](https://crowdin.com/profile/horberlan) - Romanian - Mircea Pop - [GitLab](https://gitlab.com/eeex), [Crowdin](https://crowdin.com/profile/eex) - Rareș Pița - [GitLab](https://gitlab.com/dlphin), [Crowdin](https://crowdin.com/profile/dlphin) + - Nicolae Liviu - [GitLab](https://gitlab.com/nicklcanada), [Crowdin](https://crowdin.com/profile/nicklcanada) - Russian - Nikita Grylov - [GitLab](https://gitlab.com/nixel2007), [Crowdin](https://crowdin.com/profile/nixel2007) - Alexy Lustin - [GitLab](https://gitlab.com/allustin), [Crowdin](https://crowdin.com/profile/lustin) @@ -110,6 +112,8 @@ are very appreciative of the work done by translators and proofreaders! - Iaroslav Postovalov - [GitLab](https://gitlab.com/CMDR_Tvis), [Crowdin](https://crowdin.com/profile/CMDR_Tvis) - Serbian (Latin and Cyrillic) - Proofreaders needed. +- Sinhalese/Sinhala සිංහල + - හෙළබස (HelaBasa) - [GitLab](https://gitlab.com/helabasa), [Crowdin](https://crowdin.com/profile/helabasa) - Slovak - Proofreaders needed. - Spanish diff --git a/doc/development/img/architecture_simplified.png b/doc/development/img/architecture_simplified.png Binary files differdeleted file mode 100644 index bab673feb4a..00000000000 --- a/doc/development/img/architecture_simplified.png +++ /dev/null diff --git a/doc/development/img/architecture_simplified_v14_9.png b/doc/development/img/architecture_simplified_v14_9.png Binary files differnew file mode 100644 index 00000000000..dcdb9d7b6ae --- /dev/null +++ b/doc/development/img/architecture_simplified_v14_9.png diff --git a/doc/development/img/merge_request_reports_v14_7.png b/doc/development/img/merge_request_reports_v14_7.png Binary files differnew file mode 100644 index 00000000000..282d6f96aa6 --- /dev/null +++ b/doc/development/img/merge_request_reports_v14_7.png diff --git a/doc/development/img/merge_widget_v14_7.png b/doc/development/img/merge_widget_v14_7.png Binary files differnew file mode 100644 index 00000000000..d5e8ed8df52 --- /dev/null +++ b/doc/development/img/merge_widget_v14_7.png diff --git a/doc/development/import_project.md b/doc/development/import_project.md index 86e6e04347c..e910983997c 100644 --- a/doc/development/import_project.md +++ b/doc/development/import_project.md @@ -127,11 +127,11 @@ A project with that name already exists. ##### `Exception: Error importing repository into (namespace) - No space left on device` -The disk has insufficient space to complete the import. +The disk has insufficient space to complete the import. During import, the tarball is cached in your configured `shared_path` directory. Verify the disk has enough free space to accommodate both the cached tarball and the unpacked -project files on disk. +project files on disk. ### Importing via the Rails console diff --git a/doc/development/index.md b/doc/development/index.md index 0501f70b818..048112215fc 100644 --- a/doc/development/index.md +++ b/doc/development/index.md @@ -227,7 +227,7 @@ the [reviewer values](https://about.gitlab.com/handbook/engineering/workflow/rev - [Working with merge request diffs](diffs.md) - [Approval Rules](approval_rules.md) - [Repository mirroring](repository_mirroring.md) -- [File uploads](uploads.md) +- [Uploads development guide](uploads/index.md) - [Auto DevOps development guide](auto_devops.md) - [Renaming features](renaming_features.md) - [Code Intelligence](code_intelligence/index.md) @@ -275,6 +275,7 @@ See [database guidelines](database/index.md). ## Integration guides +- [Integrations development guide](integrations/index.md) - [Jira Connect app](integrations/jira_connect.md) - [Security Scanners](integrations/secure.md) - [Secure Partner Integration](integrations/secure_partner_integration.md) @@ -329,6 +330,10 @@ See [database guidelines](database/index.md). - [CI/CD development documentation](cicd/index.md) - [AppSec development documentation](appsec/index.md) +## Technical Reference by Group + +- [Create: Source Code BE](backend/create_source_code_be/index.md) + ## Other Development guides - [Defining relations between files using projections](projections.md) @@ -338,6 +343,7 @@ See [database guidelines](database/index.md). - [Dashboards for stage groups](stage_group_dashboards.md) - [Preventing transient bugs](transient/prevention-patterns.md) - [GitLab Application SLIs](application_slis/index.md) +- [Spam protection and CAPTCHA development guide](spam_protection_and_captcha/index.md) ## Other GitLab Development Kit (GDK) guides diff --git a/doc/development/insert_into_tables_in_batches.md b/doc/development/insert_into_tables_in_batches.md index cfa0c862471..cd659a3d19b 100644 --- a/doc/development/insert_into_tables_in_batches.md +++ b/doc/development/insert_into_tables_in_batches.md @@ -122,7 +122,7 @@ Large parts of ActiveRecord's persistence API are built around the notion of cal of these callbacks fire in response to model life cycle events such as `save` or `create`. These callbacks cannot be used with bulk insertions, since they are meant to be called for every instance that is saved or created. Since these events do not fire when -records are inserted in bulk, we currently disallow their use. +records are inserted in bulk, we currently prevent their use. The specifics around which callbacks are explicitly allowed are defined in [`BulkInsertSafe`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/concerns/bulk_insert_safe.rb). diff --git a/doc/development/integrations/index.md b/doc/development/integrations/index.md new file mode 100644 index 00000000000..34ac307c98a --- /dev/null +++ b/doc/development/integrations/index.md @@ -0,0 +1,332 @@ +--- +stage: Ecosystem +group: Integrations +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +description: "GitLab's development guidelines for Integrations" +--- + +# Integrations development guide **(FREE)** + +This page provides development guidelines for implementing [GitLab integrations](../../user/project/integrations/index.md), +which are part of our [main Rails project](https://gitlab.com/gitlab-org/gitlab). + +Also see our [direction page](https://about.gitlab.com/direction/ecosystem/integrations/) for an overview of our strategy around integrations. + +This guide is a work in progress. You're welcome to ping `@gitlab-org/ecosystem-stage/integrations` +if you need clarification or spot any outdated information. + +## Add a new integration + +### Define the integration + +1. Add a new model in `app/models/integrations` extending from `Integration`. + - For example, `Integrations::FooBar` in `app/models/integrations/foo_bar.rb`. + - For certain types of integrations, you can also build on these base classes: + - `Integrations::BaseChatNotification` + - `Integrations::BaseIssueTracker` + - `Integrations::BaseMonitoring` + - `Integrations::BaseSlashCommands` + - For integrations that primarily trigger HTTP calls to external services, you can + also use the `Integrations::HasWebHook` concern. This reuses the [webhook functionality](../../user/project/integrations/webhooks.md) + in GitLab through an associated `ServiceHook` model, and automatically records request logs + which can be viewed in the integration settings. +1. Add the integration's underscored name (`'foo_bar'`) to `Integration::INTEGRATION_NAMES`. +1. Add the integration as an association on `Project`: + + ```ruby + has_one :foo_bar_integration, class_name: 'Integrations::FooBar' + ``` + +1. TEMPORARY: Accommodate the current migration to [rename "services" to "integrations"](#rename-services-to-integrations): + - Add the integration's camel-cased name (`'FooBar'`) to `Gitlab::Integrations::StiType::NAMESPACED_INTEGRATIONS`. + +### Define properties + +Integrations can define arbitrary properties to store their configuration with the class method `Integration.prop_accessor`. +The values are stored as a serialized JSON hash in the `integrations.properties` column. + +For example: + +```ruby +module Integrations + class FooBar < Integration + prop_accessor :url + prop_accessor :tags + end +end +``` + +`Integration.prop_accessor` installs accessor methods on the class. Here we would have `#url`, `#url=` and `#url_changed?`, to manage the `url` field. Fields stored in `Integration#properties` should be accessed by these accessors directly on the model, just like other ActiveRecord attributes. + +You should always access the properties through their getters, and not interact with the `properties` hash directly. +You **must not** write to the `properties` hash, you **must** use the generated setter method instead. Direct writes to this +hash are not persisted. + +You should also define validations for all your properties. + +Also refer to the section [Customize the frontend form](#customize-the-frontend-form) below to see how these properties +are exposed in the frontend form for the integration. + +There is an alternative approach using `Integration.data_field`, which you may see in other integrations. +With data fields the values are stored in a separate table per integration. At the moment we don't recommend using this for new integrations. + +### Define trigger events + +Integrations are triggered by calling their `#execute` method in response to events in GitLab, +which gets passed a payload hash with details about the event. + +The supported events have some overlap with [webhook events](../../user/project/integrations/webhook_events.md), +and receive the same payload. You can specify the events you're interested in by overriding +the class method `Integration.supported_events` in your model. + +The following events are supported for integrations: + +| Event type | Default | Value | Trigger +|:-----------------------------------------------------------------------------------------------|:--------|:---------------------|:-- +| Alert event | | `alert` | A a new, unique alert is recorded. +| Commit event | ✓ | `commit` | A commit is created or updated. +| [Deployment event](../../user/project/integrations/webhook_events.md#deployment-events) | | `deployment` | A deployment starts or finishes. +| [Issue event](../../user/project/integrations/webhook_events.md#issue-events) | ✓ | `issue` | An issue is created, updated, or closed. +| [Confidential issue event](../../user/project/integrations/webhook_events.md#issue-events) | ✓ | `confidential_issue` | A confidential issue is created, updated, or closed. +| [Job event](../../user/project/integrations/webhook_events.md#job-events) | | `job` +| [Merge request event](../../user/project/integrations/webhook_events.md#merge-request-events) | ✓ | `merge_request` | A merge request is created, updated, or merged. +| [Comment event](../../user/project/integrations/webhook_events.md#comment-events) | | `comment` | A new comment is added. +| [Confidential comment event](../../user/project/integrations/webhook_events.md#comment-events) | | `confidential_note` | A new comment on a confidential issue is added. +| [Pipeline event](../../user/project/integrations/webhook_events.md#pipeline-events) | | `pipeline` | A pipeline status changes. +| [Push event](../../user/project/integrations/webhook_events.md#push-events) | ✓ | `push` | A push is made to the repository. +| [Tag push event](../../user/project/integrations/webhook_events.md#tag-events) | ✓ | `tag_push` | New tags are pushed to the repository. +| Vulnerability event **(ULTIMATE)** | | `vulnerability` | A new, unique vulnerability is recorded. +| [Wiki page event](../../user/project/integrations/webhook_events.md#wiki-page-events) | ✓ | `wiki_page` | A wiki page is created or updated. + +#### Event examples + +This example defines an integration that responds to `commit` and `merge_request` events: + +```ruby +module Integrations + class FooBar < Integration + def self.supported_events + %w[commit merge_request] + end + end +end +``` + +An integration can also not respond to events, and implement custom functionality some other way: + +```ruby +module Integrations + class FooBar < Integration + def self.supported_events + [] + end + end +end +``` + +### Customize the frontend form + +The frontend form is generated dynamically based on metadata defined in the model. + +By default, the integration form provides: + +- A checkbox to enable or disable the integration. +- Checkboxes for each of the trigger events returned from `Integration#configurable_events`. + +You can also add help text at the top of the form by either overriding `Integration#help`, +or providing a template in `app/views/projects/services/$INTEGRATION_NAME/_help.html.haml`. + +To add your custom properties to the form, you can define the metadata for them in `Integration#fields`. + +This method should return an array of hashes for each field, where the keys can be: + +| Key | Type | Required | Default | Description +|:---------------|:--------|:---------|:-----------------------------|:-- +| `type:` | string | true | | The type of the form field. Can be `text`, `textarea`, `password`, `checkbox`, or `select`. +| `name:` | string | true | | The property name for the form field. This must match a `prop_accessor` [defined on the class](#define-properties). +| `required:` | boolean | false | `false` | Specify if the form field is required or optional. +| `title:` | string | false | Capitalized value of `name:` | The label for the form field. +| `placeholder:` | string | false | | A placeholder for the form field. +| `help:` | string | false | | A help text that displays below the form field. + +#### Additional keys for `type: 'checkbox'` + +| Key | Type | Required | Default | Description +|:------------------|:-------|:---------|:------------------|:-- +| `checkbox_label:` | string | false | Value of `title:` | A custom label that displays next to the checkbox. + +#### Additional keys for `type: 'select'` + +| Key | Type | Required | Default | Description +|:-----------|:------|:---------|:--------|:-- +| `choices:` | array | true | | A nested array of `[label, value]` tuples. + +#### Additional keys for `type: 'password'` + +| Key | Type | Required | Default | Description +|:----------------------------|:-------|:---------|:------------------|:-- +| `non_empty_password_title:` | string | false | Value of `title:` | An alternative label that displays when a value is already stored. +| `non_empty_password_help:` | string | false | Value of `help:` | An alternative help text that displays when a value is already stored. + +#### Frontend form examples + +This example defines a required `url` field, and optional `username` and `password` fields: + +```ruby +module Integrations + class FooBar < Integration + prop_accessor :url, :username, :password + + def fields + [ + { + type: 'text', + name: 'url', + title: s_('FooBarIntegration|Server URL'), + placeholder: 'https://example.com/', + required: true + }, + { + type: 'text', + name: 'username', + title: s_('FooBarIntegration|Username'), + }, + { + type: 'password', + name: 'password', + title: s_('FoobarIntegration|Password' + non_empty_password_title: s_('FooBarIntegration|Enter new password') + } + ] + end + end +end +``` + +### Expose the integration in the API + +#### REST API + +To expose the integration in the [REST API](../../api/integrations.md): + +1. Add the integration's class (`::Integrations::FooBar`) to `API::Helpers::IntegrationsHelpers.integration_classes`. +1. Add all properties that should be exposed to `API::Helpers::IntegrationsHelpers.integrations`. +1. Update the reference documentation in `doc/api/integrations.md`, add a new section for your integration, and document all properties. + +You can also refer to our [REST API style guide](../api_styleguide.md). + +#### GraphQL API + +Integrations use the `Types::Projects::ServiceType` type by default, +which only exposes the `type` and `active` properties. + +To expose additional properties, you can write a class implementing `ServiceType`: + +```ruby +# in app/graphql/types/project/services/foo_bar_service_type.rb +module Types + module Projects + module Services + class FooBarServiceType < BaseObject + graphql_name 'FooBarService' + implements(Types::Projects::ServiceType) + authorize :read_project + + field :frobinity, + GraphQL::Types::Float, + null: true, + description: 'The level of frobinity.' + + field :foo_label, + GraphQL::Types::String, + null: true, + description: 'The foo label to apply.' + end + end + end +end +``` + +Each property you want to expose should have a field defined for it. You can also expose any public instance method of the integration. + +Contact a member of the Integrations team to discuss the best authorization. + +Reference documentation for GraphQL is automatically generated. + +You can also refer to our [GraphQL API style guide](../api_graphql_styleguide.md). + +## Availability of integrations + +By default, integrations are available on the project, group, and instance level. +Most integrations only act in a project context, but can be still configured +from the group and instance levels. + +For some integrations it can make sense to only make it available on the project level. +To do that, the integration must be removed from `Integration::INTEGRATION_NAMES` and +added to `Integration::PROJECT_SPECIFIC_INTEGRATION_NAMES` instead. + +When developing a new integration, we also recommend you gate the availability behind a +[feature flag](../feature_flags/index.md) in `Integration.available_integration_names`. + +## Documentation + +You can provide help text in the integration form, including links to off-site documentation, +as described above in [Customize the frontend form](#customize-the-frontend-form). Refer to +our [usability guidelines](https://design.gitlab.com/usability/helping-users) for help text. + +For more detailed documentation, provide a page in `doc/user/project/integrations`, +and link it from the [Integrations overview](../../user/project/integrations/overview.md). + +You can also refer to our general [documentation guidelines](../documentation/index.md). + +## Testing + +It is often sufficient to add tests for the integration model in `spec/models/integrations`, +and a factory with example settings in `spec/factories/integrations.rb`. + +Each integration is also tested as part of generalized tests. For example, there are feature specs +that verify that the settings form is rendering correctly for all integrations. + +If your integration implements any custom behavior, especially in the frontend, this should be +covered by additional tests. + +You can also refer to our general [testing guidelines](../testing_guide/index.md). + +## Internationalization + +All UI strings should be prepared for translation by following our [internationalization guidelines](../i18n/externalization.md). + +The strings should use the integration name as [namespace](../i18n/externalization.md#namespaces), for example, `s_('FooBarIntegration|My string')`. + +## Ongoing migrations and refactorings + +The Integrations team is in the process of some larger migrations that developers should be aware of. + +### [Rename "services" to "integrations"](https://gitlab.com/groups/gitlab-org/-/epics/2504) + +The "integrations" in GitLab were historically called "services", which frequently caused +confusion with our "service" classes in `app/services`. We sometimes also called +them "project services" because they were initially only available on projects, which is +not the case anymore. + +We decided to change the naming from "services" and "project services" to "integrations". +This refactoring is an ongoing effort, and there are still references to the old names in some places. + +Developers should be especially aware that we still use the old class names for the STI column +`integrations.type`. For example, a class `Integrations::FooBar` still stores +the old name `FooBarService` in the database. This mapping is handled via `Gitlab::Integrations::StiType` +and should be mostly transparent to the rest of the app. + +### [Consolidate integration settings](https://gitlab.com/groups/gitlab-org/-/epics/3955) + +We want to unify the way integration properties are defined. + +## Integration examples + +You can refer to these issues for examples of adding new integrations: + +- [Datadog](https://gitlab.com/gitlab-org/gitlab/-/issues/270123): Metrics collector, similar to the Prometheus integration. +- [EWM/RTC](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/36662): External issue tracker. +- [Shimo](https://gitlab.com/gitlab-org/gitlab/-/issues/343386): External wiki, similar to the Confluence and External Wiki integrations. +- [Webex Teams](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/31543): Chat notifications. +- [ZenTao](https://gitlab.com/gitlab-org/gitlab/-/issues/338178): External issue tracker with custom issue views, similar to the Jira integration. diff --git a/doc/development/integrations/jira_connect.md b/doc/development/integrations/jira_connect.md index cfa1fdba699..5391b2c119e 100644 --- a/doc/development/integrations/jira_connect.md +++ b/doc/development/integrations/jira_connect.md @@ -65,3 +65,52 @@ If the app install failed, you might need to delete `jira_connect_installations` 1. Open the [database console](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/postgresql.md#access-postgresql). 1. Run `TRUNCATE TABLE jira_connect_installations CASCADE;`. + +#### Not authorized to access the file + +If you use Gitpod and you get an error about Jira not being able to access the descriptor file, you might need to make the GDK's port public by following these steps: + +1. Open your GitLab workspace in Gitpod. +1. When the GDK is running, select **Ports** in the bottom-right corner. +1. On the left sidebar, select the port the GDK is listening to (typically `3000`). +1. If the port is marked as private, select the lock icon to make it public. + +## Test the GitLab OAuth authentication flow + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/81126) in GitLab 14.9 [with a flag](../../administration/feature_flags.md) named `jira_connect_oauth`. Disabled by default. + +GitLab for Jira users can authenticate with GitLab using GitLab OAuth. + +WARNING: +This feature is not ready for production use. The feature flag should only be enabled in development. + +The following steps describe setting up an environment to test the GitLab OAuth flow: + +1. Start a Gitpod session and open the rails console. + + ```shell + bundle exec rails console + ``` + +1. Enable the feature flag. + + ```shell + Feature.enable(:jira_connect_oauth) + ``` + +1. On your GitLab instance, go to **Admin > Applications**. +1. Create a new application with the following settings: + - Name: `Jira Connect` + - Redirect URI: `YOUR_GITPOD_INSTANCE/-/jira_connect/oauth_callbacks` + - Scopes: `api` + - Trusted: **No** + - Confidential: **No** +1. Copy the Application ID. +1. Go to [gitpod.io/variables](https://gitpod.io/variables). +1. Create a new variable named `JIRA_CONNECT_OAUTH_CLIENT_ID`, with a scope of `*/*`, and paste the Application ID as the value. + +If you already have an active Gitpod instance, use the following command in the Gitpod terminal to set the environment variable: + +```shell +eval $(gp env -e JIRA_CONNECT_OAUTH_CLIENT_ID=$YOUR_APPLICATION_ID) +``` diff --git a/doc/development/integrations/secure.md b/doc/development/integrations/secure.md index 27a166aebf9..11fb06bd128 100644 --- a/doc/development/integrations/secure.md +++ b/doc/development/integrations/secure.md @@ -329,18 +329,42 @@ You can find the schemas for these scanners here: ### Enable report validation -In GitLab 14.10 and later, report validation against the schemas is enabled. To enable report validation for versions earlier than 14.10, -set [`VALIDATE_SCHEMA`](../../user/application_security/#enable-security-report-validation) to -`"true"`. +> [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/354928) in GitLab 14.9, and planned for removal in GitLab 15.0. +DISCLAIMER: +This page contains information related to upcoming products, features, and functionality. +It is important to note that the information presented is for informational purposes only. +Please do not rely on this information for purchasing or planning purposes. +As with all projects, the items mentioned on this page are subject to change or delay. +The development, release, and timing of any products, features, or functionality remain at the +sole discretion of GitLab Inc. +In GitLab 15.0 and later, report validation is enabled and enforced. Reports that fail validation +are not ingested, and an error message displays on the corresponding pipeline. -Reports that don't pass validation are not ingested by GitLab, and an error message -displays on the corresponding pipeline. +In GitLab 14.10 and later, report validation against the schemas is enabled but not enforced. +Reports that fail validation are ingested but display a warning in the pipeline security tab. -You should ensure that reports generated by the scanner pass validation against the schema version -declared in your reports. GitLab uses the +To enforce report validation for GitLab version 14.10 and earlier, set +[`VALIDATE_SCHEMA`](../../user/application_security/#enable-security-report-validation) to `"true"`. + +### Report validation + +You must ensure that reports generated by the scanner pass validation against the schema version +declared in your reports. Reports that don't pass validation are not ingested by GitLab, and an +error message displays on the corresponding pipeline. + +Reports that use a deprecated version of the secure report schema are ingested but cause a warning +message to display on the corresponding pipeline. If you see this warning, update your +analyzer to use the latest available schemas. + +After the deprecation period for a schema version, the file is removed from GitLab. Reports that +declare removed versions are rejected, and an error message displays on the corresponding pipeline. + +GitLab uses the [`json_schemer`](https://www.rubydoc.info/gems/json_schemer) gem to perform validation. -Ongoing improvements to report validation is tracked [in this epic](https://gitlab.com/groups/gitlab-org/-/epics/6968). +Ongoing improvements to report validation are tracked [in this epic](https://gitlab.com/groups/gitlab-org/-/epics/6968). +In the meantime, you can see which versions are supported in the +[source code](https://gitlab.com/gitlab-org/gitlab/-/blob/08dd756429731a0cca1e27ca9d59eea226398a7d/lib/gitlab/ci/parsers/security/validators/schema_validator.rb#L9-27). ### Report Fields diff --git a/doc/development/internal_api.md b/doc/development/internal_api.md deleted file mode 100644 index 7790a5e23e6..00000000000 --- a/doc/development/internal_api.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'internal_api/index.md' -remove_date: '2022-02-09' ---- - -This document was moved to [another location](internal_api/index.md). - -<!-- This redirect file can be deleted after <2022-02-09>. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/internal_api/index.md b/doc/development/internal_api/index.md index db978253747..ef58d6c2c44 100644 --- a/doc/development/internal_api/index.md +++ b/doc/development/internal_api/index.md @@ -42,7 +42,7 @@ file, and include the token Base64 encoded in a `secret_token` parameter or in the `Gitlab-Shared-Secret` header. NOTE: -The internal API used by GitLab Pages, and GitLab Agent Server (`kas`) uses JSON Web Token (JWT) +The internal API used by GitLab Pages, and GitLab agent server (`kas`) uses JSON Web Token (JWT) authentication, which is different from GitLab Shell. ## Git Authentication @@ -400,25 +400,22 @@ Example response: } ``` -## GitLab Agent endpoints +## GitLab agent endpoints > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/41045) in GitLab 13.4. > - This feature is not deployed on GitLab.com > - It's not recommended for production use. -The following endpoints are used by the GitLab Agent Server (`kas`) +The following endpoints are used by the GitLab agent server (`kas`) for various purposes. These endpoints are all authenticated using JWT. The JWT secret is stored in a file specified in `config/gitlab.yml`. By default, the location is in the root of the GitLab Rails app in a file called `.gitlab_kas_secret`. -WARNING: -The GitLab Agent is under development and is not recommended for production use. +### GitLab agent information -### GitLab Agent information - -Called from GitLab Agent Server (`kas`) to retrieve agent +Called from GitLab agent server (`kas`) to retrieve agent information for the given agent token. This returns the Gitaly connection information for the agent's project in order for `kas` to fetch and update the agent's configuration. @@ -434,9 +431,9 @@ curl --request GET --header "Gitlab-Kas-Api-Request: <JWT token>" \ --header "Authorization: Bearer <agent token>" "http://localhost:3000/api/v4/internal/kubernetes/agent_info" ``` -### GitLab Agent project information +### GitLab agent project information -Called from GitLab Agent Server (`kas`) to retrieve project +Called from GitLab agent server (`kas`) to retrieve project information for the given agent token. This returns the Gitaly connection for the requested project. GitLab `kas` uses this to configure the agent to fetch Kubernetes resources from the project repository to @@ -460,9 +457,9 @@ curl --request GET --header "Gitlab-Kas-Api-Request: <JWT token>" \ --header "Authorization: Bearer <agent token>" "http://localhost:3000/api/v4/internal/kubernetes/project_info?id=7" ``` -### GitLab Agent usage metrics +### GitLab agent usage metrics -Called from GitLab Agent Server (`kas`) to increase the usage +Called from GitLab agent server (`kas`) to increase the usage metric counters. | Attribute | Type | Required | Description | @@ -481,9 +478,9 @@ curl --request POST --header "Gitlab-Kas-Api-Request: <JWT token>" --header "Con --data '{"gitops_sync_count":1}' "http://localhost:3000/api/v4/internal/kubernetes/usage_metrics" ``` -### GitLab Agent alert metrics +### GitLab agent alert metrics -Called from GitLab Agent Server (KAS) to save alerts derived from Cilium on Kubernetes +Called from GitLab agent server (KAS) to save alerts derived from Cilium on Kubernetes Cluster. | Attribute | Type | Required | Description | @@ -505,7 +502,7 @@ curl --request POST --header "Gitlab-Kas-Api-Request: <JWT token>" \ ### Create Starboard vulnerability -Called from the GitLab Agent Server (`kas`) to create a security vulnerability +Called from the GitLab agent server (`kas`) to create a security vulnerability from a Starboard vulnerability report. This request is idempotent. Multiple requests with the same data create a single vulnerability. The response contains the UUID of the created vulnerability finding. @@ -563,7 +560,7 @@ Example response: ### Resolve Starboard vulnerabilities -Called from the GitLab Agent Server (`kas`) to resolve Starboard security vulnerabilities. +Called from the GitLab agent server (`kas`) to resolve Starboard security vulnerabilities. Accepts a list of finding UUIDs and marks all Starboard vulnerabilities not identified by the list as resolved. diff --git a/doc/development/internal_users.md b/doc/development/internal_users.md index 35db2016e1c..95ca593e31e 100644 --- a/doc/development/internal_users.md +++ b/doc/development/internal_users.md @@ -42,3 +42,5 @@ Other examples of internal users: - [Ghost User](../user/profile/account/delete_account.md#associated-records) - [Support Bot](../user/project/service_desk.md#support-bot-user) - Visual Review Bot +- Resource access tokens (including [project access tokens](../user/project/settings/project_access_tokens.md)). + These are implemented as `project_bot` users with a `PersonalAccessToken`. diff --git a/doc/development/kubernetes.md b/doc/development/kubernetes.md index 45c94019c63..a6d9c754838 100644 --- a/doc/development/kubernetes.md +++ b/doc/development/kubernetes.md @@ -136,7 +136,7 @@ Mitigation strategies include: 1. Not allowing redirects to attacker controller resources: [`Kubeclient::KubeClient`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/kubernetes/kube_client.rb#) - can be configured to disallow any redirects by passing in + can be configured to prevent any redirects by passing in `http_max_redirects: 0` as an option. 1. Not exposing error messages: by doing so, we prevent attackers from triggering errors to expose results from diff --git a/doc/development/licensed_feature_availability.md b/doc/development/licensed_feature_availability.md index 629d0027ffe..0de3f94cf70 100644 --- a/doc/development/licensed_feature_availability.md +++ b/doc/development/licensed_feature_availability.md @@ -17,9 +17,8 @@ feature such as [Related issues](../user/project/issues/related_issues.md) or [Service Desk](../user/project/service_desk.md), it should be restricted on namespace scope. -1. Add the feature symbol on `EES_FEATURES`, `EEP_FEATURES`, or `EEU_FEATURES` constants in - `ee/app/models/license.rb`. Note that the prefix `EES` signifies Starter, `EEP` signifies - Premium, and `EEU` signifies Ultimate. +1. Add the feature symbol on `STARTER_FEATURES`, `PREMIUM_FEATURES`, or `ULTIMATE_FEATURES` constants in + `ee/app/models/gitlab_subscriptions/features.rb`. 1. Check using: ```ruby @@ -33,8 +32,8 @@ However, for features such as [Geo](../administration/geo/index.md) and to only a subset of projects or namespaces, the check is made directly in the instance license. -1. Add the feature symbol on `EES_FEATURES`, `EEP_FEATURES` or `EEU_FEATURES` constants in - `ee/app/models/license.rb`. +1. Add the feature symbol to `STARTER_FEATURES`, `PREMIUM_FEATURES` or `ULTIMATE_FEATURES` constants in + `ee/app/models/gitlab_subscriptions/features.rb`. 1. Add the same feature symbol to `GLOBAL_FEATURES`. 1. Check using: diff --git a/doc/development/merge_request_application_and_rate_limit_guidelines.md b/doc/development/merge_request_application_and_rate_limit_guidelines.md new file mode 100644 index 00000000000..94ae126802a --- /dev/null +++ b/doc/development/merge_request_application_and_rate_limit_guidelines.md @@ -0,0 +1,28 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Application and rate limit guidelines + +GitLab, like most large applications, enforces limits within certain features. +The absences of limits can affect security, performance, data, or could even +exhaust the allocated resources for the application. + +Every new feature should have safe usage limits included in its implementation. +Limits are applicable for: + +- System-level resource pools such as API requests, SSHD connections, database connections, storage, and so on. +- Domain-level objects such as CI minutes, groups, sign-in attempts, and so on. + +## When limits are required + +1. Limits are required if the absence of the limit matches severity 1 - 3 in the severity definitions for [limit-related bugs](https://about.gitlab.com/handbook/engineering/quality/issue-triage/#limit-related-bugs). +1. [GitLab application limits](../administration/instance_limits.md) documentation must be updated anytime limits are added, removed, or updated. + +## Additional reading + +- Existing [GitLab application limits](../administration/instance_limits.md) +- Product processes: [introducing application limits](https://about.gitlab.com/handbook/product/product-processes/#introducing-application-limits) +- Development docs: [guide for adding application limits](application_limits.md) diff --git a/doc/development/merge_request_concepts/index.md b/doc/development/merge_request_concepts/index.md new file mode 100644 index 00000000000..f1dab69543a --- /dev/null +++ b/doc/development/merge_request_concepts/index.md @@ -0,0 +1,62 @@ +--- +type: reference, dev +stage: create +group: code_review +info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +--- + +# Merge Request Concepts + +**NOTE**: +The documentation below is the single source of truth for the merge request terminology and functionality. + +## Overview + +The merge request is made up of several different key components and ideas that encompass the overall merge request experience. These concepts sometimes have competing and confusing terminology or overlap with other concepts. The concepts this will cover are: + +1. Merge widget +1. Report widgets +1. Merge checks +1. Approval rules + +### Merge widget + +The merge widget is the component of the merge request where the `merge` button exists: + + + +This area of the merge request is where all of the options and commit messages are defined prior to merging. It also contains information about what is in the merge request, what issues may be closed, and other important information to the merging process. + +### Report widgets + +Reports are widgets within the merge request that report information about changes within the merge request. These widgets provide information to better help the author understand the changes and further improvements to the proposed changes. + +[Design Documentation](https://design.gitlab.com/regions/merge-request-reports) + + + +### Merge checks + +Merge checks are statuses that can either pass or fail and conditionally control the availability of the merge button being available within a merge request. The key distinguishing factor in a merge check is that users **do not** interact with the merge checks inside of the merge request, but are able to influence whether or not the check passes or fails. Results from the check are processed as true/false to determine whether or not a merge request can be merged. Examples include: + +1. merge conflicts +1. pipeline success +1. threads resolution +1. [external status checks](../../user/project/merge_requests/status_checks.md) +1. required approvals + +When all of the required merge checks are satisfied a merge request becomes mergeable. + +### Approvals + +Approval rules specify users that are required to or can optionally approve a merge request based on some kind of organizational policy. When approvals are required, they effectively become a required merge check. The key differentiator between merge checks and approval rules is that users **do** interact with approval rules, by deciding to approve the merge request. + +Additionally, approval settings provide configuration options to define how those approval rules are applied in a merge request. They can set limitations, add requirements, or modify approvals. + +Examples of approval rules and settings include: + +1. [merge request approval rules](../../user/project/merge_requests/approvals/rules.md) +1. [code owner approvals](../../user/project/code_owners.md) +1. [security approvals](../../user/application_security/index.md#security-approvals-in-merge-requests) +1. [prevent editing approval rules](../../user/project/merge_requests/approvals/settings.md#prevent-editing-approval-rules-in-merge-requests)] +1. [remove all approvals when commits are added](../../user/project/merge_requests/approvals/settings.md#remove-all-approvals-when-commits-are-added-to-the-source-branch) diff --git a/doc/development/merge_request_performance_guidelines.md b/doc/development/merge_request_performance_guidelines.md index c8e99e8547f..40f02f4fb6f 100644 --- a/doc/development/merge_request_performance_guidelines.md +++ b/doc/development/merge_request_performance_guidelines.md @@ -446,49 +446,6 @@ that accepts an upper limit of counting rows. In some cases it's desired that badge counters are loaded asynchronously. This can speed up the initial page load and give a better user experience overall. -## Application/misuse limits - -Every new feature should have safe usage quotas introduced. -The quota should be optimised to a level that we consider the feature to -be performant and usable for the user, but **not limiting**. - -**We want the features to be fully usable for the users.** -**However, we want to ensure that the feature continues to perform well if used at its limit** -**and it doesn't cause availability issues.** - -Consider that it's always better to start with some kind of limitation, -instead of later introducing a breaking change that would result in some -workflows breaking. - -The intent is to provide a safe usage pattern for the feature, -as our implementation decisions are optimised for the given data set. -Our feature limits should reflect the optimisations that we introduced. - -The intent of quotas could be different: - -1. We want to provide higher quotas for higher tiers of features: - we want to provide on GitLab.com more capabilities for different tiers, -1. We want to prevent misuse of the feature: someone accidentally creates - 10000 deploy tokens, because of a broken API script, -1. We want to prevent abuse of the feature: someone purposely creates - a 10000 pipelines to take advantage of the system. - -Examples: - -1. Pipeline Schedules: It's very unlikely that user wants to create - more than 50 schedules. - In such cases it's rather expected that this is either misuse - or abuse of the feature. Lack of the upper limit can result - in service degradation as the system tries to process all schedules - assigned the project. - -1. GitLab CI/CD includes: We started with the limit of maximum of 50 nested includes. - We understood that performance of the feature was acceptable at that level. - We received a request from the community that the limit is too small. - We had a time to understand the customer requirement, and implement an additional - fail-safe mechanism (time-based one) to increase the limit 100, and if needed increase it - further without negative impact on availability of the feature and GitLab. - ## Usage of feature flags Each feature that has performance critical elements or has a known performance deficiency @@ -569,7 +526,7 @@ end The usage of shared temporary storage is required if your intent is to persistent file for a disk-based storage, and not Object Storage. -[Workhorse direct_upload](uploads.md#direct-upload) when accepting file +[Workhorse direct_upload](uploads/implementation.md#direct-upload) when accepting file can write it to shared storage, and later GitLab Rails can perform a move operation. The move operation on the same destination is instantaneous. The system instead of performing `copy` operation just re-attaches file into a new place. @@ -593,7 +550,7 @@ that implements a seamless support for Shared and Object Storage-based persisten #### Data access Each feature that accepts data uploads or allows to download them needs to use -[Workhorse direct_upload](uploads.md#direct-upload). It means that uploads needs to be +[Workhorse direct_upload](uploads/implementation.md#direct-upload). It means that uploads needs to be saved directly to Object Storage by Workhorse, and all downloads needs to be served by Workhorse. @@ -605,5 +562,5 @@ can time out, which is especially problematic for slow clients. If clients take to upload/download the processing slot might be killed due to request processing timeout (usually between 30s-60s). -For the above reasons it is required that [Workhorse direct_upload](uploads.md#direct-upload) is implemented +For the above reasons it is required that [Workhorse direct_upload](uploads/implementation.md#direct-upload) is implemented for all file uploads and downloads. diff --git a/doc/development/new_fe_guide/modules/widget_extensions.md b/doc/development/new_fe_guide/modules/widget_extensions.md index 37712cb2cec..d3cd839464d 100644 --- a/doc/development/new_fe_guide/modules/widget_extensions.md +++ b/doc/development/new_fe_guide/modules/widget_extensions.md @@ -128,6 +128,7 @@ mentioned below: variant: '', // Optional: GitLab UI badge variant, defaults to info }, actions: [], // Optional: Action button for row + children: [], // Optional: Child content to render, structure matches the same structure } ``` diff --git a/doc/development/packages.md b/doc/development/packages.md index 38c1b941eaf..35a93c77c7f 100644 --- a/doc/development/packages.md +++ b/doc/development/packages.md @@ -151,7 +151,7 @@ During this phase, the idea is to collect as much information as possible about 1. Empty file structure (API file, base service for this package) 1. Authentication system for "logging in" to the package manager 1. Identify metadata and create applicable tables - 1. Workhorse route for [object storage direct upload](uploads.md#direct-upload) + 1. Workhorse route for [object storage direct upload](uploads/implementation.md#direct-upload) 1. Endpoints required for upload/publish 1. Endpoints required for install/download 1. Endpoints required for required actions @@ -210,7 +210,7 @@ File uploads should be handled by GitLab Workhorse using object accelerated uplo the workhorse proxy that checks all incoming requests to GitLab intercept the upload request, upload the file, and forward a request to the main GitLab codebase only containing the metadata and file location rather than the file itself. An overview of this process can be found in the -[development documentation](uploads.md#direct-upload). +[development documentation](uploads/implementation.md#direct-upload). In terms of code, this means a route must be added to the [GitLab Workhorse project](https://gitlab.com/gitlab-org/gitlab-workhorse) for each upload endpoint being added @@ -272,7 +272,7 @@ features must be implemented when the feature flag is removed. - File format guards (only accept valid file formats for the package type) - Name regex with validation - Version regex with validation -- Workhorse route for [accelerated](uploads.md#how-to-add-a-new-upload-route) uploads +- Workhorse route for [accelerated](uploads/working_with_uploads.md) uploads - Background workers for extracting package metadata (if applicable) - Documentation (how to use the feature) - API Documentation (individual endpoints with curl examples) diff --git a/doc/development/performance.md b/doc/development/performance.md index b5294c8359d..1e3e0570206 100644 --- a/doc/development/performance.md +++ b/doc/development/performance.md @@ -7,7 +7,38 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Performance Guidelines This document describes various guidelines to follow to ensure good and -consistent performance of GitLab. +consistent performance of GitLab. Refer to the [Index](#performance-documentation) section below to navigate to Performance-related pages. + +## Performance Documentation + +- General: + - [Solving performance issues](#workflow) + - [Handbook performance page](https://about.gitlab.com/handbook/engineering/performance/) + - [Merge request performance guidelines](../development/merge_request_performance_guidelines.md) +- Backend: + - [Tooling](#tooling) + - Database: + - [Query performance guidelines](../development/query_performance.md) + - [Pagination performance guidelines](../development/database/pagination_performance_guidelines.md) + - [Keyset pagination performance](../development/database/keyset_pagination.md#performance) + - [Troubleshooting import/export performance issues](../development/import_export.md#troubleshooting-performance-issues) + - [Pipelines performance in the `gitlab` project](../development/pipelines.md#performance) +- Frontend: + - [Performance guidelines](../development/fe_guide/performance.md) + - [Performance dashboards and monitoring guidelines](../development/new_fe_guide/development/performance.md) + - [Browser performance testing guidelines](../user/project/merge_requests/browser_performance_testing.md) + - [`gdk measure` and `gdk measure-workflow`](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/gdk_commands.md#measure-performance) +- QA: + - [Load performance testing](../user/project/merge_requests/load_performance_testing.md) + - [GitLab Performance Tool project](https://gitlab.com/gitlab-org/quality/performance) + - [Review apps performance metrics](../development/testing_guide/review_apps.md#performance-metrics) +- Monitoring & Overview: + - [GitLab performance monitoring](../administration/monitoring/performance/index.md) + - [Development department performance indicators](https://about.gitlab.com/handbook/engineering/development/performance-indicators/) + - [Service measurement](../development/service_measurement.md) +- Self-managed administration and customer-focused: + - [File system performance benchmarking](../administration/operations/filesystem_benchmarking.md) + - [Sidekiq performance troubleshooting](../administration/troubleshooting/sidekiq.md) ## Workflow @@ -95,8 +126,13 @@ end This however leads to the question: how many iterations should we run to get meaningful statistics? -The benchmark-ips Gem basically takes care of all this and much more, and as a -result of this should be used instead of the `Benchmark` module. +The [`benchmark-ips`](https://github.com/evanphx/benchmark-ips) +gem takes care of all this and much more. You should therefore use it instead of the `Benchmark` +module. + +The GitLab Gemfile also contains the [`benchmark-memory`](https://github.com/michaelherold/benchmark-memory) +gem, which works similarly to the `benchmark` and `benchmark-ips` gems. However, `benchmark-memory` +instead returns the memory size, objects, and strings allocated and retained during the benchmark. In short: @@ -110,7 +146,7 @@ In short: - If you must write a benchmark use the benchmark-ips Gem instead of Ruby's `Benchmark` module. -## Profiling +## Profiling with Stackprof By collecting snapshots of process state at regular intervals, profiling allows you to see where time is spent in a process. The @@ -124,15 +160,36 @@ frequency (for example, 100hz, that is 100 stacks per second). This type of prof has quite a low (albeit non-zero) overhead and is generally considered to be safe for production. -### Development - A profiler can be a very useful tool during development, even if it does run *in an unrepresentative environment*. In particular, a method is not necessarily troublesome just because it's executed many times, or takes a long time to execute. Profiles are tools you can use to better understand what is happening in an application - using that information wisely is up to you! -Keeping that in mind, to create a profile, identify (or create) a spec that +There are multiple ways to create a profile with Stackprof. + +### Wrapping a code block + +To profile a specific code block, you can wrap that block in a `Stackprof.run` call: + +```ruby +StackProf.run(mode: :wall, out: 'tmp/stackprof-profiling.dump') do + #... +end +``` + +This creates a `.dump` file that you can [read](#reading-a-stackprof-profile). +For all available options, see the [Stackprof documentation](https://github.com/tmm1/stackprof#all-options). + +### Performance bar + +With the [Performance bar](../administration/monitoring/performance/performance_bar.md), +you have the option to profile a request using Stackprof and immediately output the results to a +[Speedscope flamegraph](profiling.md#speedscope-flamegraphs). + +### RSpec profiling with Stackprof + +To create a profile from a spec, identify (or create) a spec that exercises the troublesome code path, then run it using the `bin/rspec-stackprof` helper, for example: @@ -161,89 +218,10 @@ Finished in 18.19 seconds (files took 4.8 seconds to load) 187 (1.1%) 187 (1.1%) block (4 levels) in class_attribute ``` -You can limit the specs that are run by passing any arguments `rspec` would +You can limit the specs that are run by passing any arguments `RSpec` would normally take. -The output is sorted by the `Samples` column by default. This is the number of -samples taken where the method is the one currently being executed. The `Total` -column shows the number of samples taken where the method, or any of the methods -it calls, were being executed. - -To create a graphical view of the call stack: - -```shell -stackprof tmp/project_policy_spec.rb.dump --graphviz > project_policy_spec.dot -dot -Tsvg project_policy_spec.dot > project_policy_spec.svg -``` - -To load the profile in [KCachegrind](https://kcachegrind.github.io/): - -```shell -stackprof tmp/project_policy_spec.rb.dump --callgrind > project_policy_spec.callgrind -kcachegrind project_policy_spec.callgrind # Linux -qcachegrind project_policy_spec.callgrind # Mac -``` - -For flame graphs, enable raw collection first. Note that raw -collection can generate a very large file, so increase the `INTERVAL`, or -run on a smaller number of specs for smaller file size: - -```shell -RAW=true bin/rspec-stackprof spec/policies/group_member_policy_spec.rb -``` - -You can then generate, and view the resultant flame graph. It might take a -while to generate based on the output file size: - -```shell -# Generate -stackprof --flamegraph tmp/group_member_policy_spec.rb.dump > group_member_policy_spec.flame - -# View -stackprof --flamegraph-viewer=group_member_policy_spec.flame -``` - -It may be useful to zoom in on a specific method, for example: - -```shell -$ stackprof tmp/project_policy_spec.rb.dump --method warm_asset_cache - -TestEnv#warm_asset_cache (/Users/lupine/dev/gitlab.com/gitlab-org/gitlab-development-kit/gitlab/spec/support/test_env.rb:164) - samples: 0 self (0.0%) / 6288 total (36.9%) - callers: - 6288 ( 100.0%) block (2 levels) in <top (required)> - callees (6288 total): - 6288 ( 100.0%) Capybara::RackTest::Driver#visit - code: - | 164 | def warm_asset_cache - | 165 | return if warm_asset_cache? - | 166 | return unless defined?(Capybara) - | 167 | - 6288 (36.9%) | 168 | Capybara.current_session.driver.visit '/' - | 169 | end -$ stackprof tmp/project_policy_spec.rb.dump --method BasePolicy#abilities -BasePolicy#abilities (/Users/lupine/dev/gitlab.com/gitlab-org/gitlab-development-kit/gitlab/app/policies/base_policy.rb:79) - samples: 0 self (0.0%) / 50 total (0.3%) - callers: - 25 ( 50.0%) BasePolicy.abilities - 25 ( 50.0%) BasePolicy#collect_rules - callees (50 total): - 25 ( 50.0%) ProjectPolicy#rules - 25 ( 50.0%) BasePolicy#collect_rules - code: - | 79 | def abilities - | 80 | return RuleSet.empty if @user && @user.blocked? - | 81 | return anonymous_abilities if @user.nil? - 50 (0.3%) | 82 | collect_rules { rules } - | 83 | end -``` - -Since the profile includes the work done by the test suite as well as the -application code, these profiles can be used to investigate slow tests as well. -However, for smaller runs (like this example), this means that the cost of -setting up the test suite tends to dominate. - -### Production +### Using Stackprof in production Stackprof can also be used to profile production workloads. @@ -274,8 +252,8 @@ the timeout. Once profiling stops, the profile is written out to disk at `$STACKPROF_FILE_PREFIX/stackprof.$PID.$RAND.profile`. It can then be inspected -further via the `stackprof` command line tool, as described in the previous -section. +further through the `stackprof` command line tool, as described in the +[Reading a Stackprof profile section](#reading-a-stackprof-profile). Currently supported profiling targets are: @@ -295,14 +273,85 @@ For Sidekiq, the signal can be sent to the `sidekiq-cluster` process via `pkill -USR2 bin/sidekiq-cluster`, which forwards the signal to all Sidekiq children. Alternatively, you can also select a specific PID of interest. -Production profiles can be especially noisy. It can be helpful to visualize them -as a [flame graph](https://github.com/brendangregg/FlameGraph). This can be done -via: +### Reading a Stackprof profile + +The output is sorted by the `Samples` column by default. This is the number of samples taken where +the method is the one currently executed. The `Total` column shows the number of samples taken where +the method (or any of the methods it calls) is executed. + +To create a graphical view of the call stack: ```shell -bundle exec stackprof --stackcollapse /tmp/stackprof.55769.c6c3906452.profile | flamegraph.pl > flamegraph.svg +stackprof tmp/project_policy_spec.rb.dump --graphviz > project_policy_spec.dot +dot -Tsvg project_policy_spec.dot > project_policy_spec.svg ``` +To load the profile in [KCachegrind](https://kcachegrind.github.io/): + +```shell +stackprof tmp/project_policy_spec.rb.dump --callgrind > project_policy_spec.callgrind +kcachegrind project_policy_spec.callgrind # Linux +qcachegrind project_policy_spec.callgrind # Mac +``` + +You can also generate and view the resultant flame graph. To view a flame graph that +`bin/rspec-stackprof` creates, you must set the `RAW` environment variable to `true` when running +`bin/rspec-stackprof`. + +It might take a while to generate based on the output file size: + +```shell +# Generate +stackprof --flamegraph tmp/group_member_policy_spec.rb.dump > group_member_policy_spec.flame + +# View +stackprof --flamegraph-viewer=group_member_policy_spec.flame +``` + +To export the flame graph to an SVG file, use [Brendan Gregg's FlameGraph tool](https://github.com/brendangregg/FlameGraph): + +```shell +stackprof --stackcollapse /tmp/group_member_policy_spec.rb.dump | flamegraph.pl > flamegraph.svg +``` + +It's also possible to view flame graphs through [speedscope](https://github.com/jlfwong/speedscope). +You can do this when using the [performance bar](profiling.md#speedscope-flamegraphs) +and when [profiling code blocks](https://github.com/jlfwong/speedscope/wiki/Importing-from-stackprof-(ruby)). +This option isn't supported by `bin/rspec-stackprof`. + +You can profile speciific methods by using `--method method_name`: + +```shell +$ stackprof tmp/project_policy_spec.rb.dump --method access_allowed_to + +ProjectPolicy#access_allowed_to? (/Users/royzwambag/work/gitlab-development-kit/gitlab/app/policies/project_policy.rb:793) + samples: 0 self (0.0%) / 578 total (0.7%) + callers: + 397 ( 68.7%) block (2 levels) in <class:ProjectPolicy> + 95 ( 16.4%) block in <class:ProjectPolicy> + 86 ( 14.9%) block in <class:ProjectPolicy> + callees (578 total): + 399 ( 69.0%) ProjectPolicy#team_access_level + 141 ( 24.4%) Project::GeneratedAssociationMethods#project_feature + 30 ( 5.2%) DeclarativePolicy::Base#can? + 8 ( 1.4%) Featurable#access_level + code: + | 793 | def access_allowed_to?(feature) + 141 (0.2%) | 794 | return false unless project.project_feature + | 795 | + 8 (0.0%) | 796 | case project.project_feature.access_level(feature) + | 797 | when ProjectFeature::DISABLED + | 798 | false + | 799 | when ProjectFeature::PRIVATE + 429 (0.5%) | 800 | can?(:read_all_resources) || team_access_level >= ProjectFeature.required_minimum_access_level(feature) + | 801 | else +``` + +When using Stackprof to profile specs, the profile includes the work done by the test suite and the +application code. You can therefore use these profiles to investigate slow tests as well. However, +for smaller runs (like this example), this means that the cost of setting up the test suite tends to +dominate. + ## RSpec profiling The GitLab development environment also includes the @@ -459,11 +508,14 @@ The `mem_*` values represent different aspects of how objects and memory are all We can use `memory_profiler` for profiling. -The [`memory_profiler`](https://github.com/SamSaffron/memory_profiler) gem is already present in the GitLab `Gemfile`, -you just need to require it: +The [`memory_profiler`](https://github.com/SamSaffron/memory_profiler) +gem is already present in the GitLab `Gemfile`. It's also available in the [performance bar](../administration/monitoring/performance/performance_bar.md) +for the current URL. + +To use the memory profiler directly in your code, use `require` to add it: ```ruby -require 'sidekiq/testing' +require 'memory_profiler' report = MemoryProfiler.report do # Code you want to profile @@ -473,10 +525,17 @@ output = File.open('/tmp/profile.txt','w') report.pretty_print(output) ``` -The report breaks down 2 key concepts: +The report shows the retained and allocated memory grouped by gem, file, location, and class. The +memory profiler also performs a string analysis that shows how often a string is allocated and +retained. -- Retained: long lived memory use and object count retained due to the execution of the code block. -- Allocated: all object allocation and memory allocation during code block. +#### Retained versus allocated + +- Retained memory: long-lived memory use and object count retained due to the execution of the code + block. This has a direct impact on memory and the garbage collector. +- Allocated memory: all object allocation and memory allocation during the code block. This might + have minimal impact on memory, but substantial impact on performance. The more objects you + allocate, the more work is being done and the slower the application is. As a general rule, **retained** is always smaller than or equal to **allocated**. @@ -512,6 +571,32 @@ Fragmented Ruby heap snapshot could look like this: Memory fragmentation could be reduced by tuning GC parameters [as described in this post](https://www.speedshop.co/2017/12/04/malloc-doubles-ruby-memory.html). This should be considered as a tradeoff, as it may affect overall performance of memory allocation and GC cycles. +### Derailed Benchmarks + +`derailed_benchmarks` is a [gem](https://github.com/zombocom/derailed_benchmarks) +described as "A series of things you can use to benchmark a Rails or Ruby app." +We include `derailed_benchmarks` in our `Gemfile`. + +We run `derailed exec perf:mem` in every pipeline with a `test` stage, in a job +called `memory-on-boot`. ([Read an example job.](https://gitlab.com/gitlab-org/gitlab/-/jobs/2144695684).) +You may find the results: + +- On the merge request **Overview** tab, in the merge request reports area, in the + **Metrics Reports** [dropdown list](../ci/metrics_reports.md). +- In the `memory-on-boot` artifacts for a full report and a dependency breakdown. + +`derailed_benchmarks` also provides other methods to investigate memory. To learn more, +refer to the [gem documentation](https://github.com/zombocom/derailed_benchmarks#running-derailed-exec). +Most of the methods (`derailed exec perf:*`) attempt to boot your Rails app in a +`production` environment and run benchmarks against it. +It is possible both in GDK and GCK: + +- For GDK, follow the + [the instructions](https://github.com/zombocom/derailed_benchmarks#running-in-production-locally) + on the gem page. You must do similar for Redis configurations to avoid errors. +- GCK includes `production` configuration sections + [out of the box](https://gitlab.com/gitlab-org/gitlab-compose-kit#running-production-like). + ## Importance of Changes When working on performance improvements, it's important to always ask yourself @@ -612,7 +697,7 @@ end ## String Freezing -In recent Ruby versions calling `freeze` on a String leads to it being allocated +In recent Ruby versions calling `.freeze` on a String leads to it being allocated only once and re-used. For example, on Ruby 2.3 or later this only allocates the "foo" String once: @@ -626,6 +711,12 @@ Depending on the size of the String and how frequently it would be allocated (before the `.freeze` call was added), this _may_ make things faster, but this isn't guaranteed. +Freezing strings saves memory, as every allocated string uses at least one `RVALUE_SIZE` bytes (40 +bytes on x64) of memory. + +You can use the [memory profiler](#using-memory-profiler) +to see which strings are allocated often and could potentially benefit from a `.freeze`. + Strings are frozen by default in Ruby 3.0. To prepare our codebase for this eventuality, we are adding the following header to all Ruby files: diff --git a/doc/development/permissions.md b/doc/development/permissions.md index a5d211a5d2e..47aebc2f4d2 100644 --- a/doc/development/permissions.md +++ b/doc/development/permissions.md @@ -9,6 +9,23 @@ info: To determine the technical writer assigned to the Stage/Group associated w There are multiple types of permissions across GitLab, and when implementing anything that deals with permissions, all of them should be considered. +## Instance + +### User types + +Each user can be one of the following types: + +- Regular. +- External - access to groups and projects only if direct member. +- [Internal users](internal_users.md) - system created. +- [Auditor](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/policies/ee/base_policy.rb#L9): + - No access to projects or groups settings menu. + - No access to Admin Area. + - Read-only access to everything else. +- [Administrator](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/policies/base_policy.rb#L6) - read-write access. + +See the [permissions page](../user/permissions.md) for details on how each user type is used. + ## Groups and Projects ### General permissions @@ -38,7 +55,7 @@ Additionally, the following project features can have different visibility level - Issues - Repository - - Merge Request + - Merge request - Forks - Pipelines - Analytics @@ -124,9 +141,9 @@ into different features like Merge Requests and CI flow. | View | License information | Dependency list, License Compliance | Can view repository | | View | Dependency information | Dependency list, License Compliance | Can view repository | | View | Vulnerabilities information | Dependency list | Can view security findings | -| View | Black/Whitelisted licenses for the project | License Compliance, Merge request | Can view repository | -| View | Security findings | Merge Request, CI job page, Pipeline security tab | Can read the project and CI jobs | -| View | Vulnerability feedback | Merge Request | Can read security findings | +| View | Black/Whitelisted licenses for the project | License Compliance, merge request | Can view repository | +| View | Security findings | merge request, CI job page, Pipeline security tab | Can read the project and CI jobs | +| View | Vulnerability feedback | merge request | Can read security findings | | View | Dependency List page | Project | Can access Dependency information | | View | License Compliance page | Project | Can access License information| diff --git a/doc/development/pipelines.md b/doc/development/pipelines.md index f3a4f47eb22..2aef0e10314 100644 --- a/doc/development/pipelines.md +++ b/doc/development/pipelines.md @@ -37,7 +37,7 @@ flowchart LR subgraph backend be["Backend code"]--tested with-->rspec end - + be--generates-->fixtures["frontend fixtures"] fixtures--used in-->jest ``` @@ -69,7 +69,7 @@ In addition, there are a few circumstances where we would always run the full RS - when the `pipeline:run-all-rspec` label is set on the merge request - when the merge request is created by an automation (e.g. Gitaly update or MR targeting a stable branch) - when the merge request is created in a security mirror -- when any CI config file is changed (i.e. `.gitlab-ci.yml` or `.gitlab/ci/**/*`) +- when any CI configuration file is changed (i.e. `.gitlab-ci.yml` or `.gitlab/ci/**/*`) ### Jest minimal jobs @@ -85,7 +85,7 @@ In addition, there are a few circumstances where we would always run the full Je - when the `pipeline:run-all-jest` label is set on the merge request - when the merge request is created by an automation (e.g. Gitaly update or MR targeting a stable branch) - when the merge request is created in a security mirror -- when any CI config file is changed (i.e. `.gitlab-ci.yml` or `.gitlab/ci/**/*`) +- when any CI configuration file is changed (i.e. `.gitlab-ci.yml` or `.gitlab/ci/**/*`) - when any frontend "core" file is changed (i.e. `package.json`, `yarn.lock`, `babel.config.js`, `jest.config.*.js`, `config/helpers/**/*.js`) - when any vendored JavaScript file is changed (i.e. `vendor/assets/javascripts/**/*`) - when any backend file is changed ([see the patterns list for details](https://gitlab.com/gitlab-org/gitlab/-/blob/3616946936c1adbd9e754c1bd06f86ba670796d8/.gitlab/ci/rules.gitlab-ci.yml#L205-216)) @@ -194,6 +194,14 @@ We keep track of retried tests in the `$RETRIED_TESTS_REPORT_FILE` file saved as See the [experiment issue](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/1148). +### Single database testing + +By default, all tests run with [multiple databases](database/multiple_databases.md). + +We also run tests with a single database in nightly scheduled pipelines, and in merge requests that touch database-related files. + +If you want to force tests to run with a single database, you can add the `pipeline:run-single-db` label to the merge request. + ### Monitoring The GitLab test suite is [monitored](performance.md#rspec-profiling) for the `main` branch, and any branch @@ -218,7 +226,7 @@ of `gitlab-org/gitlab-foss`. These jobs are only created in the following cases: - when the `pipeline:run-as-if-foss` label is set on the merge request - when the merge request is created in the `gitlab-org/security/gitlab` project -- when any CI config file is changed (i.e. `.gitlab-ci.yml` or `.gitlab/ci/**/*`) +- when any CI configuration file is changed (i.e. `.gitlab-ci.yml` or `.gitlab/ci/**/*`) The `* as-if-foss` jobs are run in addition to the regular EE-context jobs. They have the `FOSS_ONLY='1'` variable set and get the `ee/` folder removed before the tests start running. @@ -247,7 +255,7 @@ appending `-jh` to the branch name. If a corresponding JH branch is found, rather than from the default branch. NOTE: -For now, CI will try to fetch the branch on the [GitLab JH mirror](https://gitlab.com/gitlab-org/gitlab-jh/gitlab), so it might take some time for the new JH branch to propagate to the mirror. +For now, CI will try to fetch the branch on the [GitLab JH mirror](https://gitlab.com/gitlab-org/gitlab-jh-mirrors/gitlab), so it might take some time for the new JH branch to propagate to the mirror. ## `undercover` RSpec test @@ -263,6 +271,16 @@ In the event of an emergency, or false positive from this job, add the `pipeline:skip-undercoverage` label to the merge request to allow this job to fail. +You can disable the `undercover` code coverage check by wrapping the desired block of code in `# :nocov:` lines: + +```ruby +# :nocov: +def some_method + # code coverage for this method will be skipped +end +# :nocov: +``` + ## PostgreSQL versions testing Our test suite runs against PG12 as GitLab.com runs on PG12 and @@ -291,6 +309,21 @@ We follow the [PostgreSQL versions shipped with Omnibus GitLab](../administratio | PG11 | `nightly` | `nightly` | `nightly` | `nightly` | `nightly` | `nightly` | | PG13 | `nightly` | `nightly` | `nightly` | `nightly` | `nightly` | `nightly` | +## Redis versions testing + +Our test suite runs against Redis 6 as GitLab.com runs on Redis 6 and +[Omnibus defaults to Redis 6 for new installs and upgrades](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/config/software/redis.rb). + +We do run our test suite against Redis 5 on `nightly` scheduled pipelines, specifically when running backward-compatible and forward-compatible PostgreSQL jobs. + +### Current versions testing + +| Where? | Redis version | +| ------ | ------------------ | +| MRs | 6 | +| `default branch` (non-scheduled pipelines) | 6 | +| `nightly` scheduled pipelines | 5 | + ## Pipelines types for merge requests In general, pipelines for an MR fall into one of the following types (from shorter to longer), depending on the changes made in the MR: @@ -531,6 +564,8 @@ that are scoped to a single [configuration keyword](../ci/yaml/index.md#job-keyw | `.use-pg11-ee` | Same as `.use-pg11` but also use an `elasticsearch` service (see [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml) for the specific version of the service). | | `.use-pg12` | Allows a job to use the `postgres` 12 and `redis` services (see [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml) for the specific versions of the services). | | `.use-pg12-ee` | Same as `.use-pg12` but also use an `elasticsearch` service (see [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml) for the specific version of the service). | +| `.use-pg13` | Allows a job to use the `postgres` 13 and `redis` services (see [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml) for the specific versions of the services). | +| `.use-pg13-ee` | Same as `.use-pg13` but also use an `elasticsearch` service (see [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml) for the specific version of the service). | | `.use-kaniko` | Allows a job to use the `kaniko` tool to build Docker images. | | `.as-if-foss` | Simulate the FOSS project by setting the `FOSS_ONLY='1'` CI/CD variable. | | `.use-docker-in-docker` | Allows a job to use Docker in Docker. | @@ -614,6 +649,20 @@ otherwise. If you want a running pipeline to finish even if you push new commits to a merge request, be sure to start the `dont-interrupt-me` job before pushing. +### Git fetch caching + +Because GitLab.com uses the [pack-objects cache](../administration/gitaly/configure_gitaly.md#pack-objects-cache), +concurrent Git fetches of the same pipeline ref are deduplicated on +the Gitaly server (always) and served from cache (when available). + +This works well for the following reasons: + +- The pack-objects cache is enabled on all Gitaly servers on GitLab.com. +- The CI/CD [Git strategy setting](../ci/pipelines/settings.md#choose-the-default-git-strategy) for `gitlab-org/gitlab` is **Git clone**, + causing all jobs to fetch the same data, which maximizes the cache hit ratio. +- We use [shallow clone](../ci/pipelines/settings.md#limit-the-number-of-changes-fetched-during-clone) to avoid downloading the full Git + history for every job. + ### Caching strategy 1. All jobs must only pull caches by default. @@ -647,19 +696,31 @@ request, be sure to start the `dont-interrupt-me` job before pushing. We limit the artifacts that are saved and retrieved by jobs to the minimum in order to reduce the upload/download time and costs, as well as the artifacts storage. -### Git fetch caching +### Components caching -Because GitLab.com uses the [pack-objects cache](../administration/gitaly/configure_gitaly.md#pack-objects-cache), -concurrent Git fetches of the same pipeline ref are deduplicated on -the Gitaly server (always) and served from cache (when available). +Some external components (currently only GitLab Workhorse) of GitLab need to be built from source as a preliminary step for running tests. -This works well for the following reasons: +In [this MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/79766), we introduced a new `build-components` job that: -- The pack-objects cache is enabled on all Gitaly servers on GitLab.com. -- The CI/CD [Git strategy setting](../ci/pipelines/settings.md#choose-the-default-git-strategy) for `gitlab-org/gitlab` is **Git clone**, - causing all jobs to fetch the same data, which maximizes the cache hit ratio. -- We use [shallow clone](../ci/pipelines/settings.md#limit-the-number-of-changes-fetched-during-clone) to avoid downloading the full Git - history for every job. +- runs automatically for all GitLab.com `gitlab-org/gitlab` scheduled pipelines +- runs automatically for any `master` commit that touches the `workhorse/` folder +- is manual for GitLab.com's `gitlab-org`'s MRs + +This job tries to download a generic package that contains GitLab Workhorse binaries needed in the GitLab test suite (under `tmp/tests/gitlab-workhorse`). + +- If the package URL returns a 404: + 1. It runs `scripts/setup-test-env`, so that the GitLab Workhorse binaries are built. + 1. It then creates an archive which contains the binaries and upload it [as a generic package](https://gitlab.com/gitlab-org/gitlab/-/packages/). +- Otherwise, if the package already exists, it exit the job successfully. + +We also changed the `setup-test-env` job to: + +1. First download the GitLab Workhorse generic package build and uploaded by `build-components`. +1. If the package is retrieved successfully, its content is placed in the right folder (i.e. `tmp/tests/gitlab-workhorse`), preventing the building of the binaries when `scripts/setup-test-env` is run later on. +1. If the package URL returns a 404, the behavior doesn't change compared to the current one: the GitLab Workhorse binaries are built as part of `scripts/setup-test-env`. + +NOTE: +The version of the package is the workhorse tree SHA (i.e. `git rev-parse HEAD:workhorse`). ### Pre-clone step diff --git a/doc/development/product_qualified_lead_guide/index.md b/doc/development/product_qualified_lead_guide/index.md index f9d18bacecd..6943f931d79 100644 --- a/doc/development/product_qualified_lead_guide/index.md +++ b/doc/development/product_qualified_lead_guide/index.md @@ -35,6 +35,7 @@ The hand-raise lead form accepts the following parameters via provide or inject. ```javascript provide: { + small, user: { namespaceId, userName, @@ -43,9 +44,18 @@ The hand-raise lead form accepts the following parameters via provide or inject. companyName, glmContent, }, + ctaTracking: { + action, + label, + property, + value, + experiment, + }, }, ``` +The `ctaTracking` parameters follow [the `data-track` attributes](../snowplow/implementation.md#data-track-attributes) for implementing Snowplow tracking. The provided tracking attributes are attached to the button inside the `HandRaiseLeadButton` component, which triggers the hand-raise lead modal when selected. + ### Monitor the lead location When embedding a new hand raise form, use a unique `glmContent` or `glm_content` field that is different to any existing values. @@ -82,13 +92,80 @@ The flow of a PQL lead is as follows: 1. Marketo does scoring and sends the form to Salesforce. 1. Our Sales team uses Salesforce to connect to the leads. +### Trial lead flow + +#### Trial lead flow on GitLab.com + +```mermaid +sequenceDiagram + Trial Frontend Forms ->>TrialsController#create_lead: GitLab.com frontend sends [lead] to backend + TrialsController#create_lead->>CreateLeadService: [lead] + TrialsController#create_lead->>ApplyTrialService: [lead] Apply the trial + CreateLeadService->>SubscriptionPortalClient#generate_trial(sync_to_gl=false): [lead] Creates customer account on CustomersDot + ApplyTrialService->>SubscriptionPortalClient#generate_trial(sync_to_gl=true): [lead] Asks CustomersDot to apply the trial on namespace + SubscriptionPortalClient#generate_trial(sync_to_gl=false)->>CustomersDot|TrialsController#create(sync_to_gl=false): GitLab.com sends [lead] to CustomersDot + SubscriptionPortalClient#generate_trial(sync_to_gl=true)->>CustomersDot|TrialsController#create(sync_to_gl=true): GitLab.com asks CustomersDot to apply the trial + + +``` + +#### Trial lead flow on CustomersDot (sync_to_gl) + +```mermaid +sequenceDiagram + CustomersDot|TrialsController#create->>HostedPlans|CreateTrialService#execute: Save [lead] to leads table for monitoring purposes + HostedPlans|CreateTrialService#execute->>BaseTrialService#create_account: Creates a customer record in customers table + HostedPlans|CreateTrialService#create_platypus_lead->>PlatypusLogLeadService: Creates a platypus lead + HostedPlans|CreateTrialService#create_platypus_lead->>Platypus|CreateLeadWorker: Async worker to submit [lead] to Platypus + Platypus|CreateLeadWorker->>Platypus|CreateLeadService: [lead] + Platypus|CreateLeadService->>PlatypusApp#post: [lead] + PlatypusApp#post->>Platypus: [lead] is sent to Platypus +``` + +#### Applying the trial to a namespace on CustomersDot + +```mermaid +sequenceDiagram + HostedPlans|CreateTrialService->load_namespace#Gitlab api/namespaces: Load namespace details + HostedPlans|CreateTrialService->create_order#: Creates an order in orders table + HostedPlans|CreateTrialService->create_trial_history#: Creates a record in trial_histories table +``` + +### Hand raise lead flow + +#### Hand raise flow on GitLab.com + +```mermaid +sequenceDiagram + HandRaiseForm Vue Component->>TrialsController#create_hand_raise_lead: GitLab.com frontend sends [lead] to backend + TrialsController#create_hand_raise_lead->>CreateHandRaiseLeadService: [lead] + CreateHandRaiseLeadService->>SubscriptionPortalClient: [lead] + SubscriptionPortalClient->>CustomersDot|TrialsController#create_hand_raise_lead: GitLab.com sends [lead] to CustomersDot +``` + +#### Hand raise flow on CustomersDot + +```mermaid +sequenceDiagram + CustomersDot|TrialsController#create_hand_raise_lead->>PlatypusLogLeadService: Save [lead] to leads table for monitoring purposes + CustomersDot|TrialsController#create_hand_raise_lead->>Platypus|CreateLeadWorker: Async worker to submit [lead] to Platypus + Platypus|CreateLeadWorker->>Platypus|CreateLeadService: [lead] + Platypus|CreateLeadService->>PlatypusApp#post: [lead] + PlatypusApp#post->>Platypus: [lead] is sent to Platypus +``` + +### PQL flow after Platypus for all lead types + +```mermaid +sequenceDiagram + Platypus->>Workato: [lead] + Workato->>Marketo: [lead] + Marketo->>Salesforce(SFDC): [lead] +``` + ## Monitor and manually test leads - Check the application and Sidekiq logs on `gitlab.com` and CustomersDot to monitor leads. - Check the `leads` table in CustomersDot. - Set up staging credentials for Platypus, and track the leads on the [Platypus Dashboard](https://staging.ci.nexus.gitlabenvironment.cloud/admin/queues/queue/new-lead-queue). - Ask for access to the Marketo Sandbox and validate the leads there. - -## Trials - -Trials follow the same flow as the PQL leads. diff --git a/doc/development/profiling.md b/doc/development/profiling.md index deb743569c5..a3142b06e12 100644 --- a/doc/development/profiling.md +++ b/doc/development/profiling.md @@ -91,6 +91,73 @@ printer = RubyProf::CallStackPrinter.new(result) printer.print(File.open('/tmp/profile.html', 'w')) ``` +### Stackprof support + +By default, `Gitlab::Profiler.profile` uses a tracing profiler called [`ruby-prof`](https://ruby-prof.github.io/). However, sampling profilers +[run faster and use less memory](https://jvns.ca/blog/2017/12/17/how-do-ruby---python-profilers-work-/), so they might be preferred. + +You can switch to [Stackprof](https://github.com/tmm1/stackprof) (a sampling profiler) to generate a profile by passing `sampling_mode: true`. +Pass in a `profiler_options` hash to configure the output file (`out`) of the sampling data. For example: + +```ruby +Gitlab::Profiler.profile('/gitlab-org/gitlab-test', user: User.first, sampling_mode: true, profiler_options: { out: 'tmp/profile.dump' }) +``` + +You can get a summary of where time was spent by running Stackprof against the sampling data. For example: + +```shell +stackprof tmp/profile.dump +``` + +Example sampling data: + +```plaintext +================================== + Mode: wall(1000) + Samples: 8745 (6.92% miss rate) + GC: 1399 (16.00%) +================================== + TOTAL (pct) SAMPLES (pct) FRAME + 1022 (11.7%) 1022 (11.7%) Sprockets::PathUtils#stat + 957 (10.9%) 957 (10.9%) (marking) + 493 (5.6%) 493 (5.6%) Sprockets::PathUtils#entries + 576 (6.6%) 471 (5.4%) Mustermann::AST::Translator#decorator_for + 439 (5.0%) 439 (5.0%) (sweeping) + 630 (7.2%) 241 (2.8%) Sprockets::Cache::FileStore#get + 208 (2.4%) 208 (2.4%) ActiveSupport::FileUpdateChecker#watched + 206 (2.4%) 206 (2.4%) Digest::Instance#file + 544 (6.2%) 176 (2.0%) Sprockets::Cache::FileStore#safe_open + 176 (2.0%) 176 (2.0%) ActiveSupport::FileUpdateChecker#max_mtime + 268 (3.1%) 147 (1.7%) ActiveRecord::ConnectionAdapters::PostgreSQLAdapter#exec_no_cache + 140 (1.6%) 140 (1.6%) ActiveSupport::BacktraceCleaner#add_gem_filter + 116 (1.3%) 116 (1.3%) Bootsnap::CompileCache::ISeq.storage_to_output + 160 (1.8%) 113 (1.3%) Gem::Version#<=> + 109 (1.2%) 109 (1.2%) block in <main> + 108 (1.2%) 108 (1.2%) Gem::Version.new + 131 (1.5%) 105 (1.2%) Sprockets::EncodingUtils#unmarshaled_deflated + 1166 (13.3%) 82 (0.9%) Mustermann::RegexpBased#initialize + 82 (0.9%) 78 (0.9%) FileUtils.touch + 72 (0.8%) 72 (0.8%) Sprockets::Manifest.compile_match_filter + 71 (0.8%) 70 (0.8%) Grape::Router#compile! + 91 (1.0%) 65 (0.7%) ActiveRecord::ConnectionAdapters::PostgreSQL::DatabaseStatements#query + 93 (1.1%) 64 (0.7%) ActionDispatch::Journey::Path::Pattern::AnchoredRegexp#accept + 59 (0.7%) 59 (0.7%) Mustermann::AST::Translator.dispatch_table + 62 (0.7%) 59 (0.7%) Rails::BacktraceCleaner#initialize + 2492 (28.5%) 49 (0.6%) Sprockets::PathUtils#stat_directory + 242 (2.8%) 49 (0.6%) Gitlab::Instrumentation::RedisBase.add_call_details + 47 (0.5%) 47 (0.5%) URI::RFC2396_Parser#escape + 46 (0.5%) 46 (0.5%) #<Class:0x00000001090c2e70>#__setobj__ + 44 (0.5%) 44 (0.5%) Sprockets::Base#normalize_logical_path +``` + +You can also generate flamegraphs: + +```shell +stackprof --d3-flamegraph tmp/profile.dump > flamegraph.html +``` + +See [the Stackprof documentation](https://github.com/tmm1/stackprof) for more details. + ## Speedscope flamegraphs You can generate a flamegraph for a particular URL by selecting a flamegraph sampling mode button in the performance bar or by adding the `performance_bar=flamegraph` parameter to the request. diff --git a/doc/development/rake_tasks.md b/doc/development/rake_tasks.md index 5c8e2d5fc55..1e9367ecee4 100644 --- a/doc/development/rake_tasks.md +++ b/doc/development/rake_tasks.md @@ -196,6 +196,16 @@ One way to generate the initial list is to run the Rake task `rubocop:todo:gener bundle exec rake rubocop:todo:generate ``` +To generate TODO list for specific RuboCop rules, pass them comma-seperated as +argument to the Rake task: + +```shell +bundle exec rake 'rubocop:todo:generate[Gitlab/NamespacedClass,Lint/Syntax]' +bundle exec rake rubocop:todo:generate\[Gitlab/NamespacedClass,Lint/Syntax\] +``` + +Some shells require brackets to be escaped or quoted. + See [Resolving RuboCop exceptions](contributing/style_guides.md#resolving-rubocop-exceptions) on how to proceed from here. @@ -219,14 +229,14 @@ To update the Emoji aliases file (used for Emoji autocomplete), run the following: ```shell -bundle exec rake gemojione:aliases +bundle exec rake tanuki_emoji:aliases ``` To update the Emoji digests file (used for Emoji autocomplete), run the following: ```shell -bundle exec rake gemojione:digests +bundle exec rake tanuki_emoji:digests ``` This updates the file `fixtures/emojis/digests.json` based on the currently @@ -235,7 +245,7 @@ available Emoji. To generate a sprite file containing all the Emoji, run: ```shell -bundle exec rake gemojione:sprite +bundle exec rake tanuki_emoji:sprite ``` If new emoji are added, the sprite sheet may change size. To compensate for diff --git a/doc/development/redis/new_redis_instance.md b/doc/development/redis/new_redis_instance.md index dcd79be0e5c..96f860f3890 100644 --- a/doc/development/redis/new_redis_instance.md +++ b/doc/development/redis/new_redis_instance.md @@ -112,7 +112,7 @@ while and there are no issues, we can proceed. ### Proposed solution: Migrate data by using MultiStore with the fallback strategy -We need a way to migrate users to a new Redis store without causing any inconveniences from UX perspective. +We need a way to migrate users to a new Redis store without causing any inconveniences from UX perspective. We also want the ability to fall back to the "old" Redis instance if something goes wrong with the new instance. Migration Requirements: @@ -129,13 +129,13 @@ We need to write data into both Redis instances (old + new). We read from the new instance, but we need to fall back to the old instance when pre-fetching from the new dedicated Redis instance that failed. We need to log any issues or exceptions with a new instance, but still fall back to the old instance. -The proposed migration strategy is to implement and use the [MultiStore](https://gitlab.com/gitlab-org/gitlab/-/blob/fcc42e80ed261a862ee6ca46b182eee293ae60b6/lib/gitlab/redis/multi_store.rb). -We used this approach with [adding new dedicated Redis instance for session keys](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/579). +The proposed migration strategy is to implement and use the [MultiStore](https://gitlab.com/gitlab-org/gitlab/-/blob/fcc42e80ed261a862ee6ca46b182eee293ae60b6/lib/gitlab/redis/multi_store.rb). +We used this approach with [adding new dedicated Redis instance for session keys](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/579). Also MultiStore comes with corresponding [specs](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/lib/gitlab/redis/multi_store_spec.rb). The MultiStore looks like a `redis-rb ::Redis` instance. -In the new Redis instance class you added in [Step 1](#step-1-support-configuring-the-new-instance), +In the new Redis instance class you added in [Step 1](#step-1-support-configuring-the-new-instance), override the [Redis](https://gitlab.com/gitlab-org/gitlab/-/blob/fcc42e80ed261a862ee6ca46b182eee293ae60b6/lib/gitlab/redis/sessions.rb#L20-28) method from the `::Gitlab::Redis::Wrapper` ```ruby @@ -177,7 +177,7 @@ bin/feature-flag use_primary_store_as_default_for_foo ``` By enabling `use_primary_and_secondary_stores_for_foo` feature flag, our `Gitlab::Redis::Foo` will use `MultiStore` to write to both new Redis instance -and the [old (fallback-instance)](#fallback-instance). +and the [old (fallback-instance)](#fallback-instance). If we fail to fetch data from the new instance, we will fallback and read from the old Redis instance. We can monitor logs for `Gitlab::Redis::MultiStore::ReadFromPrimaryError`, and also the Prometheus counter `gitlab_redis_multi_store_read_fallback_total`. @@ -218,7 +218,7 @@ When a command outside of the supported list is used, `method_missing` will pass This ensures that anything unexpected behaves like it would before. NOTE: -By tracking `gitlab_redis_multi_store_method_missing_total` counter and `Gitlab::Redis::MultiStore::MethodMissingError`, +By tracking `gitlab_redis_multi_store_method_missing_total` counter and `Gitlab::Redis::MultiStore::MethodMissingError`, a developer will need to add an implementation for missing Redis commands before proceeding with the migration. ##### Errors diff --git a/doc/development/secure_coding_guidelines.md b/doc/development/secure_coding_guidelines.md index d34b12c6361..10f6c22e54a 100644 --- a/doc/development/secure_coding_guidelines.md +++ b/doc/development/secure_coding_guidelines.md @@ -253,12 +253,27 @@ the mitigations for a new feature. - [More details](https://dev.gitlab.org/gitlab/gitlabhq/-/merge_requests/2530/diffs) -#### Feature-specific mitigations +#### URL blocker & validation libraries + +[`Gitlab::UrlBlocker`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/url_blocker.rb) can be used to validate that a +provided URL meets a set of constraints. Importantly, when `dns_rebind_protection` is `true`, the method returns a known-safe URI where the hostname +has been replaced with an IP address. This prevents DNS rebinding attacks, because the DNS record has been resolved. However, if we ignore this returned +value, we **will not** be protected against DNS rebinding. -For situations in which an allowlist or GitLab:HTTP cannot be used, it will be necessary to implement mitigations directly in the feature. It is best to validate the destination IP addresses themselves, not just domain names, as DNS can be controlled by the attacker. Below are a list of mitigations that should be implemented. +This is the case with validators such as the `AddressableUrlValidator` (called with `validates :url, addressable_url: {opts}` or `public_url: {opts}`). +Validation errors are only raised when validations are called, for example when a record is created or saved. If we ignore the value returned by the validation +when persisting the record, **we need to recheck** its validity before using it. You can learn more about [Time of Check to Time of Use bugs](#time-of-check-to-time-of-use-bugs) in a later section +of these guidelines. + +#### Feature-specific mitigations There are many tricks to bypass common SSRF validations. If feature-specific mitigations are necessary, they should be reviewed by the AppSec team, or a developer who has worked on SSRF mitigations previously. +For situations in which you can't use an allowlist or GitLab:HTTP, you must implement mitigations +directly in the feature. It's best to validate the destination IP addresses themselves, not just +domain names, as the attacker can control DNS. Below is a list of mitigations that you should +implement. + - Block connections to all localhost addresses - `127.0.0.1/8` (IPv4 - note the subnet mask) - `::1` (IPv6) @@ -270,9 +285,36 @@ There are many tricks to bypass common SSRF validations. If feature-specific mit - `169.254.0.0/16` - In particular, for GCP: `metadata.google.internal` -> `169.254.169.254` - For HTTP connections: Disable redirects or validate the redirect destination -- To mitigate DNS rebinding attacks, validate and use the first IP address received +- To mitigate DNS rebinding attacks, validate and use the first IP address received. + +See [`url_blocker_spec.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/lib/gitlab/url_blocker_spec.rb) for examples of SSRF payloads. See [time of check to time of use bugs](#time-of-check-to-time-of-use-bugs) to learn more about DNS rebinding's class of bug. -See [`url_blocker_spec.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/lib/gitlab/url_blocker_spec.rb) for examples of SSRF payloads +Don't rely on methods like `.start_with?` when validating a URL, or make assumptions about which +part of a string maps to which part of a URL. Use the `URI` class to parse the string, and validate +each component (scheme, host, port, path, and so on). Attackers can create valid URLs which look +safe, but lead to malicious locations. + +```ruby +user_supplied_url = "https://my-safe-site.com@my-evil-site.com" # Content before an @ in a URL is usually for basic authentication +user_supplied_url.start_with?("https://my-safe-site.com") # Don't trust with start_with? for URLs! +=> true +URI.parse(user_supplied_url).host +=> "my-evil-site.com" + +user_supplied_url = "https://my-safe-site.com-my-evil-site.com" +user_supplied_url.start_with?("https://my-safe-site.com") # Don't trust with start_with? for URLs! +=> true +URI.parse(user_supplied_url).host +=> "my-safe-site.com-my-evil-site.com" + +# Here's an example where we unsafely attempt to validate a host while allowing for +# subdomains +user_supplied_url = "https://my-evil-site-my-safe-site.com" +user_supplied_host = URI.parse(user_supplied_url).host +=> "my-evil-site-my-safe-site.com" +user_supplied_host.end_with?("my-safe-site.com") # Don't trust with end_with? +=> true +``` ## XSS guidelines @@ -448,8 +490,7 @@ parameter when using `check_allowed_absolute_path!()`. To use a combination of both checks, follow the example below: ```ruby -path = Gitlab::Utils.check_path_traversal!(path) -Gitlab::Utils.check_allowed_absolute_path!(path, path_allowlist) +Gitlab::Utils.check_allowed_absolute_path_and_path_traversal!(path, path_allowlist) ``` In the REST API, we have the [`FilePath`](https://gitlab.com/gitlab-org/security/gitlab/-/blob/master/lib/api/validations/validators/file_path.rb) @@ -1120,3 +1161,52 @@ func printZipContents(src string) error { return nil } ``` + +## Time of check to time of use bugs + +Time of check to time of use, or TOCTOU, is a class of error which occur when the state of something changes unexpectedly partway during a process. +More specifically, it's when the property you checked and validated has changed when you finally get around to using that property. + +These types of bugs are often seen in environments which allow multi-threading and concurrency, like filesystems and distributed web applications; these are a type of race condition. TOCTOU also occurs when state is checked and stored, then after a period of time that state is relied on without re-checking its accuracy and/or validity. + +### Examples + +**Example 1:** you have a model which accepts a URL as input. When the model is created you verify that the URL's host resolves to a public IP address, to prevent attackers making internal network calls. But DNS records can change ([DNS rebinding](#server-side-request-forgery-ssrf)]). An attacker updates the DNS record to `127.0.0.1`, and when your code resolves those URL's host it results in sending a potentially malicious request to a server on the internal network. The property was valid at the "time of check", but invalid and malicious at "time of use". + +GitLab-specific example can be found in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/214401) where, although `Gitlab::UrlBlocker.validate!` was called, the returned value was not used. This made it vulnerable to TOCTOU bug and SSRF protection bypass through [DNS rebinding](#server-side-request-forgery-ssrf). The fix was to [use the validated IP address](https://gitlab.com/gitlab-org/gitlab/-/commit/7af8abd4df9a98f7a1ae7c4ec9840d0a7a8c684d). + +**Example 2:** you have a feature which schedules jobs. When the user schedules the job, they have permission to do so. But imagine if, between the time they schedule the job and the time it is run, their permissions are restricted. Unless you re-check permissions at time of use, you could inadvertently allow unauthorized activity. + +**Example 3:** you need to fetch a remote file, and perform a `HEAD` request to get and validate the content length and content type. When you subsequently make a `GET` request, though, the file delivered is a different size or different file type. (This is stretching the definition of TOCTOU, but things _have_ changed between time of check and time of use). + +**Example 4:** you allow users to upvote a comment if they haven't already. The server is multi-threaded, and you aren't using transactions or an applicable database index. By repeatedly clicking upvote in quick succession a malicious user is able to add multiple upvotes: the requests arrive at the same time, the checks run in parallel and confirm that no upvote exists yet, and so each upvote is written to the database. + +Here's some pseudocode showing an example of a potential TOCTOU bug: + +```ruby +def upvote(comment, user) + # The time between calling .exists? and .create can lead to TOCTOU, + # particularly if .create is a slow method, or runs in a background job + if Upvote.exists?(comment: comment, user: user) + return + else + Upvote.create(comment: comment, user: user) + end +end +``` + +### Prevention & defense + +- Assume values will change between the time you validate them and the time you use them. +- Perform checks as close to execution time as possible. +- Perform checks after your operation completes. +- Use your framework's validations and database features to impose constraints and atomic reads and writes. +- Read about [Server Side Request Forgery (SSRF) and DNS rebinding](#server-side-request-forgery-ssrf) + +An example of well implemented `Gitlab::UrlBlocker.validate!` call that prevents TOCTOU bug: + +1. [Preventing DNS rebinding in Gitea importer](https://gitlab.com/gitlab-org/gitlab/-/commit/7af8abd4df9a98f7a1ae7c4ec9840d0a7a8c684d) + +### Resources + +- [CWE-367: Time-of-check Time-of-use (TOCTOU) Race Condition](https://cwe.mitre.org/data/definitions/367.html) diff --git a/doc/development/service_ping/implement.md b/doc/development/service_ping/implement.md index 3a1e4c6d87b..25e841e113b 100644 --- a/doc/development/service_ping/implement.md +++ b/doc/development/service_ping/implement.md @@ -16,7 +16,7 @@ Service Ping consists of two kinds of data: To implement a new metric in Service Ping, follow these steps: 1. [Implement the required counter](#types-of-counters) -1. [Name and place the metric](#name-and-place-the-metric) +1. [Name and place the metric](metrics_dictionary.md#metric-key_path) 1. [Test counters manually using your Rails console](#test-counters-manually-using-your-rails-console) 1. [Generate the SQL query](#generate-the-sql-query) 1. [Optimize queries with `#database-lab`](#optimize-queries-with-database-lab) @@ -26,13 +26,12 @@ To implement a new metric in Service Ping, follow these steps: 1. [Verify your metric](#verify-your-metric) 1. [Set up and test Service Ping locally](#set-up-and-test-service-ping-locally) -NOTE: -When you add or change a Service Metric, you must migrate metrics to [instrumentation classes](metrics_instrumentation.md). -For information about the progress on migrating Service ping metrics, see this [epic](https://gitlab.com/groups/gitlab-org/-/epics/5547). - ## Instrumentation classes -We recommend you use [instrumentation classes](metrics_instrumentation.md) in `usage_data.rb` where possible. +NOTE: +Implementing metrics directly in `usage_data.rb` is deprecated. +When you add or change a Service Ping Metric, you must migrate metrics to [instrumentation classes](metrics_instrumentation.md). +For information about the progress on migrating Service Ping metrics, see this [epic](https://gitlab.com/groups/gitlab-org/-/epics/5547). For example, we have the following instrumentation class: `lib/gitlab/usage/metrics/instrumentations/count_boards_metric.rb`. @@ -45,7 +44,7 @@ boards: add_metric('CountBoardsMetric', time_frame: 'all'), ## Types of counters -There are several types of counters in `usage_data.rb`: +There are several types of counters for metrics: - **[Batch counters](#batch-counters)**: Used for counts and sums. - **[Redis counters](#redis-counters):** Used for in-memory counts. @@ -72,64 +71,39 @@ you must add a specialized index on the columns involved in a counter. #### Ordinary batch counters -Simple count of a given `ActiveRecord_Relation`, does a non-distinct batch count, smartly reduces `batch_size`, and handles errors. -Handles the `ActiveRecord::StatementInvalid` error. +Create a new [database metrics](metrics_instrumentation.md#database-metrics) instrumentation class with `count` operation for a given `ActiveRecord_Relation` Method: ```ruby -count(relation, column = nil, batch: true, start: nil, finish: nil) +add_metric('CountIssuesMetric', time_frame: 'all') ``` -Arguments: - -- `relation` the ActiveRecord_Relation to perform the count -- `column` the column to perform the count on, by default is the primary key -- `batch`: default `true` to use batch counting -- `start`: custom start of the batch counting to avoid complex min calculations -- `end`: custom end of the batch counting to avoid complex min calculations - Examples: -```ruby -count(User.active) -count(::Clusters::Cluster.aws_installed.enabled, :cluster_id) -count(::Clusters::Cluster.aws_installed.enabled, :cluster_id, start: ::Clusters::Cluster.minimum(:id), finish: ::Clusters::Cluster.maximum(:id)) -``` +Examples using `usage_data.rb` have been [deprecated](usage_data.md). We recommend to use [instrumentation classes](metrics_instrumentation.md). #### Distinct batch counters -Distinct count of a given `ActiveRecord_Relation` on given column, a distinct batch count, smartly reduces `batch_size`, and handles errors. -Handles the `ActiveRecord::StatementInvalid` error. +Create a new [database metrics](metrics_instrumentation.md#database-metrics) instrumentation class with `distinct_count` operation for a given `ActiveRecord_Relation`. Method: ```ruby -distinct_count(relation, column = nil, batch: true, batch_size: nil, start: nil, finish: nil) +add_metric('CountUsersAssociatingMilestonesToReleasesMetric', time_frame: 'all') ``` -Arguments: - -- `relation`: the ActiveRecord_Relation to perform the count -- `column`: the column to perform the distinct count, by default is the primary key -- `batch`: default `true` to use batch counting -- `batch_size`: if none set it uses default value 10000 from `Gitlab::Database::BatchCounter` -- `start`: custom start of the batch counting to avoid complex min calculations -- `end`: custom end of the batch counting to avoid complex min calculations - WARNING: Counting over non-unique columns can lead to performance issues. For more information, see the [iterating tables in batches](../iterating_tables_in_batches.md) guide. Examples: -```ruby -distinct_count(::Project, :creator_id) -distinct_count(::Note.with_suggestions.where(time_period), :author_id, start: ::User.minimum(:id), finish: ::User.maximum(:id)) -distinct_count(::Clusters::Applications::CertManager.where(time_period).available.joins(:cluster), 'clusters.user_id') -``` +Examples using `usage_data.rb` have been [deprecated](usage_data.md). We recommend to use [instrumentation classes](metrics_instrumentation.md). #### Sum batch operation +There is no support for `sum` for database metrics. + Sum the values of a given ActiveRecord_Relation on given column and handles errors. Handles the `ActiveRecord::StatementInvalid` error @@ -686,29 +660,6 @@ We return fallback values in these cases: | Timeouts, general failures | -1 | | Standard errors in counters | -2 | -## Name and place the metric - -Add the metric in one of the top-level keys: - -- `settings`: for settings related metrics. -- `counts_weekly`: for counters that have data for the most recent 7 days. -- `counts_monthly`: for counters that have data for the most recent 28 days. -- `counts`: for counters that have data for all time. - -### How to get a metric name suggestion - -The metric YAML generator can suggest a metric name for you. -To generate a metric name suggestion, first instrument the metric at the provided `key_path`. -Then, generate the metric's YAML definition and -return to the instrumentation and update it. - -1. Add the metric instrumentation to `lib/gitlab/usage_data.rb` inside one - of the [top-level keys](#name-and-place-the-metric), using any name you choose. -1. Run the [metrics YAML generator](metrics_dictionary.md#metrics-definition-and-validation). -1. Use the metric name suggestion to select a suitable metric name. -1. Update the instrumentation you created in the first step and change the metric name to the suggested name. -1. Update the metric's YAML definition with the correct `key_path`. - ## Test counters manually using your Rails console ```ruby diff --git a/doc/development/service_ping/index.md b/doc/development/service_ping/index.md index 86e70cc8bbc..6878fd1bf28 100644 --- a/doc/development/service_ping/index.md +++ b/doc/development/service_ping/index.md @@ -6,7 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Service Ping Guide **(FREE SELF)** -> Introduced in GitLab Ultimate 11.2, more statistics. +> - Introduced in GitLab Ultimate 11.2, more statistics. +> - In GitLab 14.1, [renamed from Usage Ping to Service Ping](https://gitlab.com/groups/gitlab-org/-/epics/5990). In 14.0 and earlier, use the Usage Ping documentation for the Rails commands appropriate to your version. Service Ping is a GitLab process that collects and sends a weekly payload to GitLab. The payload provides important high-level data that helps our product, support, @@ -68,7 +69,10 @@ Because of these limitations we recommend you: > Introduced in GitLab 14.1. -In GitLab versions 14.1 and later, free self-managed users running [GitLab EE](../ee_features.md) can receive paid features by registering with GitLab and sending us activity data through Service Ping. Features introduced here do not remove the feature from its paid tier. Users can continue to access the features in a paid tier without sharing usage data. +In GitLab versions 14.1 and later, GitLab Free customers with a self-managed instance running +[GitLab EE](../ee_features.md) can receive paid features by registering with GitLab and sending us +activity data through Service Ping. Features introduced here do not remove the feature from its paid +tier. Users can continue to access the features in a paid tier without sharing usage data. #### Features available in 14.1 and later @@ -209,17 +213,17 @@ sequenceDiagram - `uuid` - GitLab instance unique identifier - `hostname` - GitLab instance hostname -- `version` - GitLab instance current versions +- `version` - GitLab instance current versions - `elapsed` - Amount of time which passed since Service Ping report process started and moment of error occurrence - `message` - Error message <pre> <code> { - "uuid"=>"02333324-1cd7-4c3b-a45b-a4993f05fb1d", - "hostname"=>"127.0.0.1", - "version"=>"14.7.0-pre", - "elapsed"=>0.006946, + "uuid"=>"02333324-1cd7-4c3b-a45b-a4993f05fb1d", + "hostname"=>"127.0.0.1", + "version"=>"14.7.0-pre", + "elapsed"=>0.006946, "message"=>'PG::UndefinedColumn: ERROR: column \"non_existent_attribute\" does not exist\nLINE 1: SELECT COUNT(non_existent_attribute) FROM \"issues\" /*applica...' } </code> @@ -576,7 +580,7 @@ skip_db_write: ServicePing::SubmitService.new(skip_db_write: true).execute ``` -## Manually upload Service Ping payload +## Manually upload Service Ping payload > [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/7388) in GitLab 14.8 with a flag named `admin_application_settings_service_usage_data_center`. Disabled by default. @@ -596,7 +600,7 @@ To upload payload manually: ## Monitoring -Service Ping reporting process state is monitored with [internal SiSense dashboard](https://app.periscopedata.com/app/gitlab/968489/Product-Intelligence---Service-Ping-Health). +Service Ping reporting process state is monitored with [internal SiSense dashboard](https://app.periscopedata.com/app/gitlab/968489/Product-Intelligence---Service-Ping-Health). ## Troubleshooting diff --git a/doc/development/service_ping/metrics_dictionary.md b/doc/development/service_ping/metrics_dictionary.md index 93eec4efabd..6884844da3f 100644 --- a/doc/development/service_ping/metrics_dictionary.md +++ b/doc/development/service_ping/metrics_dictionary.md @@ -47,13 +47,58 @@ Each metric is defined in a separate YAML file consisting of a number of fields: | `distribution` | yes | `array`; may be set to one of `ce, ee` or `ee`. The [distribution](https://about.gitlab.com/handbook/marketing/strategic-marketing/tiers/#definitions) where the tracked feature is available. | | `performance_indicator_type` | no | `array`; may be set to one of [`gmau`, `smau`, `paid_gmau`, or `umau`](https://about.gitlab.com/handbook/business-technology/data-team/data-catalog/xmau-analysis/). | | `tier` | yes | `array`; may contain one or a combination of `free`, `premium` or `ultimate`. The [tier]( https://about.gitlab.com/handbook/marketing/strategic-marketing/tiers/) where the tracked feature is available. This should be verbose and contain all tiers where a metric is available. | -| `milestone` | no | The milestone when the metric is introduced. | +| `milestone` | no | The milestone when the metric is introduced and when it's available to self-managed instances with the official GitLab release. | | `milestone_removed` | no | The milestone when the metric is removed. | -| `introduced_by_url` | no | The URL to the merge request that introduced the metric. | +| `introduced_by_url` | no | The URL to the merge request that introduced the metric to be available for self-managed instances. | | `repair_issue_url` | no | The URL of the issue that was created to repair a metric with a `broken` status. | | `options` | no | `object`: options information needed to calculate the metric value. | | `skip_validation` | no | This should **not** be set. [Used for imported metrics until we review, update and make them valid](https://gitlab.com/groups/gitlab-org/-/epics/5425). | +### Metric key_path + +The `key_path` of the metric is the location in the JSON Service Ping payload. + +The `key_path` could be composed from multiple parts separated by `.` and it must be unique. + +We recommend to add the metric in one of the top-level keys: + +- `settings`: for settings related metrics. +- `counts_weekly`: for counters that have data for the most recent 7 days. +- `counts_monthly`: for counters that have data for the most recent 28 days. +- `counts`: for counters that have data for all time. + +NOTE: +We can't control what the metric's `key_path` is, because some of them are generated dynamically in `usage_data.rb`. +For example, see [Redis HLL metrics](implement.md#redis-hll-counters). + +### Metric name + +To improve metric discoverability by a wider audience, each metric with +instrumentation added at an appointed `key_path` receives a `name` attribute +filled with the name suggestion, corresponding to the metric `data_source` and instrumentation. +Metric name suggestions can contain two types of elements: + +1. **User input prompts**: enclosed by angle brackets (`< >`), these pieces should be replaced or + removed when you create a metrics YAML file. +1. **Fixed suggestion**: plaintext parts generated according to well-defined algorithms. + They are based on underlying instrumentation, and must not be changed. + +For a metric name to be valid, it must not include any prompt, and fixed suggestions +must not be changed. + +#### Generate a metric name suggestion + +The metric YAML generator can suggest a metric name for you. +To generate a metric name suggestion, first instrument the metric at the provided `key_path`. +Then, generate the metric's YAML definition and +return to the instrumentation and update it. + +1. Add the metric instrumentation class to `lib/gitlab/usage/metrics/instrumentations/`. +1. Add the metric logic in the instrumentation class. +1. Run the [metrics YAML generator](metrics_dictionary.md#metrics-definition-and-validation). +1. Use the metric name suggestion to select a suitable metric name. +1. Update the metric's YAML definition with the correct `key_path`. + ### Metric statuses Metric definitions can have one of the following statuses: @@ -81,21 +126,6 @@ which has a related schema in `/config/metrics/objects_schemas/topology_schema.j - `all`: The metric data applies for the whole time the metric has been active (all-time interval). For example, the following metric counts all users that create issues: `/config/metrics/counts_all/20210216181115_issues.yml`. - `none`: The metric collects a type of data that's not tracked over time, such as settings and configuration information. Therefore, a time interval is not applicable. For example, `uuid` has no time interval applicable: `config/metrics/license/20210201124933_uuid.yml`. -### Metric name - -To improve metric discoverability by a wider audience, each metric with -instrumentation added at an appointed `key_path` receives a `name` attribute -filled with the name suggestion, corresponding to the metric `data_source` and instrumentation. -Metric name suggestions can contain two types of elements: - -1. **User input prompts**: Enclosed by `<>`, these pieces should be replaced or - removed when you create a metrics YAML file. -1. **Fixed suggestion**: Plaintext parts generated according to well-defined algorithms. - They are based on underlying instrumentation, and should not be changed. - -For a metric name to be valid, it must not include any prompt, and no fixed suggestions -should be changed. - ### Data category We use the following categories to classify a metric: diff --git a/doc/development/service_ping/metrics_instrumentation.md b/doc/development/service_ping/metrics_instrumentation.md index c98b0df92aa..c684d9d12ef 100644 --- a/doc/development/service_ping/metrics_instrumentation.md +++ b/doc/development/service_ping/metrics_instrumentation.md @@ -57,6 +57,50 @@ module Gitlab end ``` +### Ordinary batch counters Example + +```ruby +module Gitlab + module Usage + module Metrics + module Instrumentations + class CountIssuesMetric < DatabaseMetric + operation :count + + start { Issue.minimum(:id) } + finish { Issue.maximum(:id) } + + relation { Issue } + end + end + end + end +end +``` + +### Distinct batch counters Example + +```ruby +# frozen_string_literal: true + +module Gitlab + module Usage + module Metrics + module Instrumentations + class CountUsersAssociatingMilestonesToReleasesMetric < DatabaseMetric + operation :distinct_count, column: :author_id + + relation { Release.with_milestones } + + start { Release.minimum(:author_id) } + finish { Release.maximum(:author_id) } + end + end + end + end +end +``` + ## Redis metrics [Example of a merge request that adds a `Redis` metric](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66582). diff --git a/doc/development/service_ping/review_guidelines.md b/doc/development/service_ping/review_guidelines.md index 137e11608cf..ee2d8f4f4a1 100644 --- a/doc/development/service_ping/review_guidelines.md +++ b/doc/development/service_ping/review_guidelines.md @@ -51,12 +51,13 @@ are regular backend changes. #### The Product Intelligence **reviewer** should - Perform a first-pass review on the merge request and suggest improvements to the author. -- Check the [metrics location](implement.md#name-and-place-the-metric) in +- Check the [metrics location](metrics_dictionary.md#metric-key_path) in the Service Ping JSON payload. -- Suggest that the author checks the [naming suggestion](implement.md#how-to-get-a-metric-name-suggestion) while +- Suggest that the author checks the [naming suggestion](metrics_dictionary.md#generate-a-metric-name-suggestion) while generating the metric's YAML definition. - Add the `~database` label and ask for a [database review](../database_review.md) for metrics that are based on Database. +- Add `~Data Warehouse::Impact Check` for any database metric that has a query change. Changes in queries can affect [data operations](https://about.gitlab.com/handbook/business-technology/data-team/how-we-work/triage/#gitlabcom-db-structure-changes). - For tracking using Redis HLL (HyperLogLog): - Check the Redis slot. - Check if a [feature flag is needed](implement.md#recommendations). diff --git a/doc/development/service_ping/usage_data.md b/doc/development/service_ping/usage_data.md new file mode 100644 index 00000000000..a25ad5f62be --- /dev/null +++ b/doc/development/service_ping/usage_data.md @@ -0,0 +1,70 @@ +--- +stage: Growth +group: Product Intelligence +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Usage Data Metrics guide + +This guide describes deprecated usage for metrics in `usage_data.rb`. + +NOTE: +Implementing metrics direct in `usage_data.rb` is deprecated, We recommend you use [instrumentation classes](metrics_instrumentation.md). + +## Ordinary batch counters + +Simple count of a given `ActiveRecord_Relation`, does a non-distinct batch count, smartly reduces `batch_size`, and handles errors. +Handles the `ActiveRecord::StatementInvalid` error. + +Method: + +```ruby +count(relation, column = nil, batch: true, start: nil, finish: nil) +``` + +Arguments: + +- `relation` the ActiveRecord_Relation to perform the count +- `column` the column to perform the count on, by default is the primary key +- `batch`: default `true` to use batch counting +- `start`: custom start of the batch counting to avoid complex min calculations +- `end`: custom end of the batch counting to avoid complex min calculations + +Examples: + +```ruby +count(User.active) +count(::Clusters::Cluster.aws_installed.enabled, :cluster_id) +count(::Clusters::Cluster.aws_installed.enabled, :cluster_id, start: ::Clusters::Cluster.minimum(:id), finish: ::Clusters::Cluster.maximum(:id)) +``` + +## Distinct batch counters + +Distinct count of a given `ActiveRecord_Relation` on given column, a distinct batch count, smartly reduces `batch_size`, and handles errors. +Handles the `ActiveRecord::StatementInvalid` error. + +Method: + +```ruby +distinct_count(relation, column = nil, batch: true, batch_size: nil, start: nil, finish: nil) +``` + +Arguments: + +- `relation`: the ActiveRecord_Relation to perform the count +- `column`: the column to perform the distinct count, by default is the primary key +- `batch`: default `true` to use batch counting +- `batch_size`: if none set it uses default value 10000 from `Gitlab::Database::BatchCounter` +- `start`: custom start of the batch counting to avoid complex min calculations +- `end`: custom end of the batch counting to avoid complex min calculations + +WARNING: +Counting over non-unique columns can lead to performance issues. For more information, see the [iterating tables in batches](../iterating_tables_in_batches.md) guide. + +Examples: + +```ruby +distinct_count(::Project, :creator_id) +distinct_count(::Note.with_suggestions.where(time_period), :author_id, start: ::User.minimum(:id), finish: ::User.maximum(:id)) +distinct_count(::Clusters::Applications::CertManager.where(time_period).available.joins(:cluster), 'clusters.user_id') +``` diff --git a/doc/development/shared_files.md b/doc/development/shared_files.md index e26464a5d80..4f13bb80761 100644 --- a/doc/development/shared_files.md +++ b/doc/development/shared_files.md @@ -11,5 +11,5 @@ servers in `shared/`, using a shared storage solution like NFS. Although this is some GitLab installations, it must not be the only file storage option for a given feature. This is because [cloud-native GitLab installations do not support it](architecture.md#adapting-existing-and-introducing-new-components). -Our [uploads documentation](uploads.md) describes how to handle file storage in +Our [uploads documentation](uploads/index.md) describes how to handle file storage in such a way that it supports both options: direct disk access and object storage. diff --git a/doc/development/sidekiq/idempotent_jobs.md b/doc/development/sidekiq/idempotent_jobs.md index 4b201e22ca9..38db22f8467 100644 --- a/doc/development/sidekiq/idempotent_jobs.md +++ b/doc/development/sidekiq/idempotent_jobs.md @@ -75,7 +75,7 @@ job executed, the first job would do nothing. GitLab supports two deduplication strategies: -- `until_executing` +- `until_executing`, which is the default strategy - `until_executed` More [deduplication strategies have been @@ -190,6 +190,7 @@ that can tolerate some duplication. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/69372) in GitLab 14.3. > - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/338350) in GitLab 14.4. > - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/338350) in GitLab 14.6. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/346598) in GitLab 14.9. [Feature flag preserve_latest_wal_locations_for_idempotent_jobs](https://gitlab.com/gitlab-org/gitlab/-/issues/346598) removed. The deduplication always take into account the latest binary replication pointer, not the first one. This happens because we drop the same job scheduled for the second time and the Write-Ahead Log (WAL) is lost. @@ -199,10 +200,3 @@ To support both deduplication and maintaining data consistency with load balanci we are preserving the latest WAL location for idempotent jobs in Redis. This way we are always comparing the latest binary replication pointer, making sure that we read from the replica that is fully caught up. - -FLAG: -On self-managed GitLab, by default this feature is available. To hide the feature, ask an administrator to -[disable the feature flag](../../administration/feature_flags.md) named `preserve_latest_wal_locations_for_idempotent_jobs`. - -This feature flag is related to GitLab development and is not intended to be used by GitLab administrators, though. -On GitLab.com, this feature is available. diff --git a/doc/development/sidekiq/worker_attributes.md b/doc/development/sidekiq/worker_attributes.md index d681e17a053..3bd6d313e2c 100644 --- a/doc/development/sidekiq/worker_attributes.md +++ b/doc/development/sidekiq/worker_attributes.md @@ -259,7 +259,7 @@ these scenarios, since `:always` should be considered the exception, not the rul To allow for reads to be served from replicas, we added two additional consistency modes: `:sticky` and `:delayed`. When you declare either `:sticky` or `:delayed` consistency, workers become eligible for database -load-balancing. +load-balancing. In both cases, if the replica is not up-to-date and the time from scheduling the job was less than the minimum delay interval, the jobs sleep up to the minimum delay interval (0.8 seconds). This gives the replication process time to finish. diff --git a/doc/development/sidekiq_style_guide.md b/doc/development/sidekiq_style_guide.md index 3756dd6b598..1b5e7addf29 100644 --- a/doc/development/sidekiq_style_guide.md +++ b/doc/development/sidekiq_style_guide.md @@ -6,4 +6,6 @@ remove_date: '2022-04-13' This document was moved to [another location](sidekiq/index.md). <!-- This redirect file can be deleted after <2022-04-13>. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/snowplow/implementation.md b/doc/development/snowplow/implementation.md index d35413cfd5f..6061a1d4cd2 100644 --- a/doc/development/snowplow/implementation.md +++ b/doc/development/snowplow/implementation.md @@ -47,10 +47,7 @@ To implement tracking for HAML or Vue templates, add a [`data-track` attribute]( The following example shows `data-track-*` attributes assigned to a button: ```haml -%button.btn{ data: { track: { action: "click_button", label: "template_preview", property: "my-template" } } } - -// or -// %button.btn{ data: { track_action: "click_button", track_label: "template_preview", track_property: "my-template" } } +%button.btn{ data: { track_action: "click_button", track_label: "template_preview", track_property: "my-template" } } ``` ```html @@ -69,7 +66,7 @@ The following example shows `data-track-*` attributes assigned to a button: | `data-track-action` | true | Action the user is taking. Clicks must be prepended with `click` and activations must be prepended with `activate`. For example, focusing a form field is `activate_form_input` and clicking a button is `click_button`. Replaces `data-track-event`, which was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/290962) in GitLab 13.11. | | `data-track-label` | false | The specific element or object to act on. This can be: the label of the element, for example, a tab labeled 'Create from template' for `create_from_template`; a unique identifier if no text is available, for example, `groups_dropdown_close` for closing the Groups dropdown in the top bar; or the name or title attribute of a record being created. | | `data-track-property` | false | Any additional property of the element, or object being acted on. | -| `data-track-value` | false | Describes a numeric value (decimal) directly related to the event. This could be the value of an input. For example, `10` when clicking `internal` visibility. If omitted, this is the element's `value` property or `undefined`. For checkboxes, the default value is the element's checked attribute or `0` when unchecked. | +| `data-track-value` | false | Describes a numeric value (decimal) directly related to the event. This could be the value of an input. For example, `10` when clicking `internal` visibility. If omitted, this is the element's `value` property or `undefined`. For checkboxes, the default value is the element's checked attribute or `0` when unchecked. The value is parsed as numeric before sendind the event. | | `data-track-extra` | false | A key-value pair object passed as a valid JSON string. This attribute is added to the `extra` property in our [`gitlab_standard`](schemas.md#gitlab_standard) schema. | | `data-track-context` | false | To append a custom context object, passed as a valid JSON string. | @@ -520,7 +517,7 @@ To install and run Snowplow Micro, complete these steps to modify the ### Troubleshoot To control content security policy warnings when using an external host, modify `config/gitlab.yml` -to allow or disallow them. To allow them, add the relevant host for `connect_src`. For example, for +to allow or prevent them. To allow them, add the relevant host for `connect_src`. For example, for `https://snowplow.trx.gitlab.net`: ```yaml diff --git a/doc/development/spam_protection_and_captcha/exploratory_testing.md b/doc/development/spam_protection_and_captcha/exploratory_testing.md new file mode 100644 index 00000000000..e508265cf83 --- /dev/null +++ b/doc/development/spam_protection_and_captcha/exploratory_testing.md @@ -0,0 +1,360 @@ +--- +stage: Manage +group: Authentication and Authorization +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exploratory testing of CAPTCHAs + +You can reliably test CAPTCHA on review apps, and in your local development environment (GDK). +You can always: + +- Force a reCAPTCHA to appear where it is supported. +- Force a checkbox to display, instead of street sign images to find and select. + +To set up testing, follow the configuration on this page. + +## Use appropriate test data + +Make sure you are testing a scenario which has spam/CAPTCHA enabled. For example: +make sure you are editing a _public_ snippet, as only public snippets are checked for spam. + +## Enable feature flags + +Enable any relevant feature flag, if the spam/CAPTCHA support is behind a feature flag. + +## Set up Akismet and reCAPTCHA + +1. To set up reCAPTCHA: + 1. Review the [GitLab reCAPTCHA documentation](../../integration/recaptcha.md). + 1. Get Google's official test reCAPTCHA credentials using the instructions from + [Google's reCAPTCHA documentation](https://developers.google.com/recaptcha/docs/faq#id-like-to-run-automated-tests-with-recaptcha.-what-should-i-do). + 1. For **Site key**, use: `6LeIxAcTAAAAAJcZVRqyHh71UMIEGNQ_MXjiZKhI` + 1. For **Secret key**, use: `6LeIxAcTAAAAAGG-vFI1TnRWxMZNFuojJ4WifJWe` + 1. Go to **Admin -> Settings -> Reporting** settings: `http://gdk.test:3000/admin/application_settings/reporting#js-spam-settings` + 1. Select **Enable reCAPTCHA**. Enabling for login is not required unless you are testing that feature. + 1. Enter the **Site key** and **Secret key**. +1. To set up Akismet: + 1. Review the [GitLab documentation on Akismet](../../integration/akismet.md). + 1. Get an Akismet API key. You can sign up for [a testing key from Akismet](https://akismet.com). + You must enter your local host (such as`gdk.test`) and email when signing up. + 1. Go to GitLab Akismet settings page, for example: + `http://gdk.test:3000/admin/application_settings/reporting#js-spam-settings` + 1. Enable Akismet and enter your Akismet **API key**. +1. To force an Akismet false-positive spam check, refer to the + [Akismet API documentation](https://akismet.com/development/api/#comment-check) and + [Akismet Getting Started documentation](https://docs.akismet.com/getting-started/confirm/) for more details: + 1. You can use `akismet-guaranteed-spam@example.com` as the author email to force spam using the following steps: + 1. Go to user email settings: `http://gdk.test:3000/-/profile/emails` + 1. Add `akismet-guaranteed-spam@example.com` as a secondary email for the administrator user. + 1. Confirm it in the Rails console: `bin/rails c` -> `User.find_by_username('root').emails.last.confirm` + 1. Switch this verified email to be your primary email: + 1. Go to **Avatar dropdown list -> Edit Profile -> Main Settings**. + 1. For **Email**, enter `akismet-guaranteed-spam@example.com` to replace `admin@example.com`. + 1. Select **Update Profile Settings** to save your changes. + +## Test in the web UI + +After you have all the above configuration in place, you can test CAPTCHAs. Test +in an area of the application which already has CAPTCHA support, such as: + +- Creating or editing an issue. +- Creating or editing a public snippet. Only **public** snippets are checked for spam. + +## Test in a development environment + +After you force Spam Flagging + CAPTCHA using the steps above, you can test the +behavior with any spam-protected model/controller action. + +### Test with CAPTCHA enabled (CONDITIONAL_ALLOW verdict) + +If CAPTCHA is enabled in these areas, you must solve the CAPTCHA popup modal before you can resubmit the form: + +- **Admin -> Settings -> Reporting -> Spam** +- **Anti-bot Protection -> Enable reCAPTCHA** + +<!-- vale gitlab.Substitutions = NO --> + +### Testing with CAPTCHA disabled ("DISALLOW" verdict) + +<!-- vale gitlab.Substitutions = YES --> + +If CAPTCHA is disabled in **Admin -> Settings -> Reporting -> Spam** and **Anti-bot Protection -> Enable reCAPTCHA**, +no CAPTCHA popup displays. You are prevented from submitting the form at all. + +### HTML page to render reCAPTCHA + +NOTE: +If you use **Google's official test reCAPTCHA credentials** listed in +[Set up Akismet and reCAPTCHA](#set-up-akismet-and-recaptcha), the +CAPTCHA response string does not matter. It can be any string. If you use a +real, valid key pair, you must solve the CAPTCHA to obtain a +valid CAPTCHA response to use. You can do this once only, and only before it expires. + +To directly test the GraphQL API via [GraphQL Explorer](http://gdk.test:3000/-/graphql-explorer), +get a reCAPTCHA response string via this form: `public/recaptcha.html` (`http://gdk.test:3000/recaptcha.html`): + +```html +<html> +<head> + <title>reCAPTCHA demo: Explicit render after an onload callback</title> + <script type="text/javascript"> + var onloadCallback = function() { + grecaptcha.render('html_element', { + 'sitekey' : '6Ld05AsaAAAAAMsm1yTUp4qsdFARN15rQJPPqv6i' + }); + }; + function onSubmit() { + window.document.getElementById('recaptchaResponse').innerHTML = grecaptcha.getResponse(); + return false; + } + </script> +</head> +<body> +<form onsubmit="return onSubmit()"> + <div id="html_element"></div> + <br> + <input type="submit" value="Submit"> +</form> +<div> + <h1>recaptchaResponse:</h1> + <div id="recaptchaResponse"></div> +</div> +<script src="https://www.google.com/recaptcha/api.js?onload=onloadCallback&render=explicit" + async defer> +</script> +</body> +</html> +``` + +## Spam/CAPTCHA API exploratory testing examples + +These sections describe the steps needed to perform manual exploratory testing of +various scenarios of the Spam and CAPTCHA behavior for the REST and GraphQL APIs. + +For the prerequisites, you must: + +1. Perform all the steps listed above to enable Spam and CAPTCHA in the development environment, + and force form submissions to require a CAPTCHA. +1. Ensure you have created an HTML page to render CAPTCHA under the `/public` directory, + with a page that contains a form to manually generate a valid CAPTCHA response string. + If you use **Google's official test reCAPTCHA credentials** listed in + [Set up Akismet and reCAPTCHA](#set-up-akismet-and-recaptcha), the contents of the + CAPTCHA response string don't matter. +1. Go to **Admin -> Settings -> Reporting -> Spam and Anti-bot protection**. +1. Select or clear **Enable reCAPTCHA** and **Enable Akismet** according to your + scenario's needs. + +The following examples use snippet creation as an example. You could also use +snippet updates, issue creation, or issue updates. Issues and snippets are the +only models with full Spam and CAPTCHA support. + +### Initial setup + +1. Create an API token. +1. Export it in your terminal for the REST commands: `export PRIVATE_TOKEN=<your_api_token>` +1. Ensure you are logged into GitLab development environment at `localhost:3000` before using GraphiQL explorer, + because it uses your logged-in user as authorization for running GraphQL queries. +1. For the GraphQL examples, use the GraphiQL explorer at `http://localhost:3000/-/graphql-explorer`. +1. Use the `--include` (`-i`) option to `curl` to print the HTTP response headers, including the status code. + +### Scenario: Akismet and CAPTCHA enabled + +In this example, Akismet and CAPTCHA are enabled: + +1. [Initial request](#initial-request). + +<!-- TODO in future edit + +Some example videos: + +- REST API: + + + +GraphQL API: + + + +--> + +#### Initial request + +This initial request fails because no CAPTCHA response is provided. + +REST request: + +```shell +curl --request POST --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" "http://localhost:3000/api/v4/snippets?title=Title&file_name=FileName&content=Content&visibility=public" +``` + +REST response: + +```shell +{"needs_captcha_response":true,"spam_log_id":42,"captcha_site_key":"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX","message":{"error":"Your snippet has been recognized as spam. Please, change the content or solve the reCAPTCHA to proceed."}} +``` + +GraphQL request: + +```graphql +mutation { + createSnippet(input: { + title: "Title" + visibilityLevel: public + blobActions: [ + { + action: create + filePath: "BlobPath" + content: "BlobContent" + } + ] + }) { + snippet { + id + title + } + errors + } +} +``` + +GraphQL response: + +```json +{ + "data": { + "createSnippet": null + }, + "errors": [ + { + "message": "Request denied. Solve CAPTCHA challenge and retry", + "locations": [ + { + "line": 22, + "column": 5 + } + ], + "path": [ + "createSnippet" + ], + "extensions": { + "needs_captcha_response": true, + "spam_log_id": 140, + "captcha_site_key": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" + } + } + ] +} +``` + +#### Second request + +This request succeeds because a CAPTCHA response is provided. + +REST request: + +```shell +export CAPTCHA_RESPONSE="<CAPTCHA response obtained from HTML page to render CAPTCHA>" +export SPAM_LOG_ID="<spam_log_id obtained from initial REST response>" +curl --request POST --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" --header "X-GitLab-Captcha-Response: $CAPTCHA_RESPONSE" --header "X-GitLab-Spam-Log-Id: $SPAM_LOG_ID" "http://localhost:3000/api/v4/snippets?title=Title&file_name=FileName&content=Content&visibility=public" +``` + +REST response: + +```shell +{"id":42,"title":"Title","description":null,"visibility":"public", "other_fields": "..."} +``` + +GraphQL request: + +NOTE: +The GitLab GraphiQL implementation doesn't allow passing of headers, so we must write +this as a `curl` query. Here, `--data-binary` is used to properly handle escaped double quotes +in the JSON-embedded query. + +```shell +export CAPTCHA_RESPONSE="<CAPTCHA response obtained from HTML page to render CAPTCHA>" +export SPAM_LOG_ID="<spam_log_id obtained from initial REST response>" +curl --include "http://localhost:3000/api/graphql" --header "Authorization: Bearer $PRIVATE_TOKEN" --header "Content-Type: application/json" --header "X-GitLab-Captcha-Response: $CAPTCHA_RESPONSE" --header "X-GitLab-Spam-Log-Id: $SPAM_LOG_ID" --request POST --data-binary '{"query": "mutation {createSnippet(input: {title: \"Title\" visibilityLevel: public blobActions: [ { action: create filePath: \"BlobPath\" content: \"BlobContent\" } ] }) { snippet { id title } errors }}"}' +``` + +GraphQL response: + +```json +{"data":{"createSnippet":{"snippet":{"id":"gid://gitlab/PersonalSnippet/42","title":"Title"},"errors":[]}}} +``` + +### Scenario: Akismet enabled, CAPTCHA disabled + +For this scenario, ensure you clear **Enable reCAPTCHA** in the Admin Area settings as described above. +If CAPTCHA is not enabled, any request flagged as potential spam fails with no chance to resubmit, +even if it could otherwise be resubmitted if CAPTCHA were enabled and successfully solved. + +The REST request is the same as if CAPTCHA was enabled: + +```shell +curl --request POST --header "PRIVATE-TOKEN: $PRIVATE_TOKEN" "http://localhost:3000/api/v4/snippets?title=Title&file_name=FileName&content=Content&visibility=public" +``` + +REST response: + +```shell +{"message":{"error":"Your snippet has been recognized as spam and has been discarded."}} +``` + +GraphQL request: + +```graphql +mutation { + createSnippet(input: { + title: "Title" + visibilityLevel: public + blobActions: [ + { + action: create + filePath: "BlobPath" + content: "BlobContent" + } + ] + }) { + snippet { + id + title + } + errors + } +} +``` + +GraphQL response: + +```json +{ + "data": { + "createSnippet": null + }, + "errors": [ + { + "message": "Request denied. Spam detected", + "locations": [ + { + "line": 22, + "column": 5 + } + ], + "path": [ + "createSnippet" + ], + "extensions": { + "spam": true + } + } + ] +} +``` + +### Scenario: allow_possible_spam feature flag enabled + +With the `allow_possible_spam` feature flag enabled, the API returns a 200 response. Any +valid request is successful and no CAPTCHA is presented, even if the request is considered +spam. diff --git a/doc/development/spam_protection_and_captcha/graphql_api.md b/doc/development/spam_protection_and_captcha/graphql_api.md new file mode 100644 index 00000000000..b47e3f84320 --- /dev/null +++ b/doc/development/spam_protection_and_captcha/graphql_api.md @@ -0,0 +1,66 @@ +--- +stage: Manage +group: Authentication and Authorization +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# GraphQL API spam protection and CAPTCHA support + +If the model can be modified via the GraphQL API, you must also add support to all of the +relevant GraphQL mutations which may modify spammable or spam-related attributes. This +definitely includes the `Create` and `Update` mutations, but may also include others, such as those +related to changing a model's confidential/public flag. + +## Add support to the GraphQL mutations + +This implementation is very similar to the controller implementation. You create a `spam_params` +instance based on the request, and pass it to the relevant Service class constructor. + +The three main differences from the controller implementation are: + +1. Use `include Mutations::SpamProtection` instead of `...JsonFormatActionsSupport`. +1. Obtain the request from the context via `context[:request]` when creating the `SpamParams` + instance. +1. After you create or updated the `Spammable` model instance, call `#check_spam_action_response!` + and pass it the model instance. This call will: + 1. Perform the necessary spam checks on the model. + 1. If spam is detected: + - Raise a `GraphQL::ExecutionError` exception. + - Include the relevant information added as error fields to the response via the `extensions:` parameter. + For more details on these fields, refer to the section on + [Spam and CAPTCHA support in the GraphQL API](../../api/graphql/index.md#resolve-mutations-detected-as-spam). + + NOTE: + If you use the standard ApolloLink or Axios interceptor CAPTCHA support described + above, the field details are unimportant. They become important if you + attempt to use the GraphQL API directly to process a failed check for potential spam, and + resubmit the request with a solved CAPTCHA response. + +For example: + +```ruby +module Mutations + module Widgets + class Create < BaseMutation + include Mutations::SpamProtection + + def resolve(args) + spam_params = ::Spam::SpamParams.new_from_request(request: context[:request]) + + service_response = ::Widgets::CreateService.new( + project: project, + current_user: current_user, + params: args, + spam_params: spam_params + ).execute + + widget = service_response.payload[:widget] + check_spam_action_response!(widget) + + # If possible spam wasdetected, an exception would have been thrown by + # `#check_spam_action_response!`, so the normal resolve return logic can follow below. + end + end + end +end +``` diff --git a/doc/development/spam_protection_and_captcha/index.md b/doc/development/spam_protection_and_captcha/index.md new file mode 100644 index 00000000000..9b195df536d --- /dev/null +++ b/doc/development/spam_protection_and_captcha/index.md @@ -0,0 +1,53 @@ +--- +stage: Manage +group: Authentication and Authorization +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Spam protection and CAPTCHA + +This guide provides an overview of how to add spam protection and CAPTCHA support to new areas of the +GitLab application. + +## Add spam protection and CAPTCHA support to a new area + +To add this support, you must implement the following areas as applicable: + +1. [Model and Services](model_and_services.md): The basic prerequisite + changes to the backend code which are required to add spam or CAPTCHA API and UI support + for a feature which does not yet have support. +1. REST API (Supported, documentation coming soon): The changes needed to add + spam or CAPTCHA support to Grape REST API endpoints. Refer to the related + [REST API documentation](../../api/index.md#resolve-requests-detected-as-spam). +1. [GraphQL API](graphql_api.md): The changes needed to add spam or CAPTCHA support to GraphQL + mutations. Refer to the related + [GraphQL API documentation](../../api/graphql/index.md#resolve-mutations-detected-as-spam). +1. [Web UI](web_ui.md): The various possible scenarios encountered when adding + spam/CAPTCHA support to the web UI, depending on whether the UI is JavaScript API-based (Vue or + plain JavaScript) or HTML-form (HAML) based. + +You should also perform manual exploratory testing of the new feature. Refer to +[Exploratory testing](exploratory_testing.md) for more information. + +## Spam-related model and API fields + +Multiple levels of spam flagging determine how spam is handled. These levels are referenced in +[`Spam::SpamConstants`](https://gitlab.com/gitlab-org/gitlab/blob/master/app/services/spam/spam_constants.rb#L4-4), +and used various places in the application, such as +[`Spam::SpamActionService#perform_spam_service_check`](https://gitlab.com/gitlab-org/gitlab/blob/d7585b56c9e7dc69414af306d82906e28befe7da/app/services/spam/spam_action_service.rb#L61-61). + +The possible values include: + +- `BLOCK_USER` +- `DISALLOW` +- `CONDITIONAL_ALLOW` +- `OVERRIDE_VIA_ALLOW_POSSIBLE_SPAM` +- `ALLOW` +- `NOOP` + +## Related topics + +- [Spam and CAPTCHA support in the GraphQL API](../../api/graphql/index.md#resolve-mutations-detected-as-spam) +- [Spam and CAPTCHA support in the REST API](../../api/index.md#resolve-requests-detected-as-spam) +- [reCAPTCHA Spam and Anti-bot Protection](../../integration/recaptcha.md) +- [Akismet and Spam Logs](../../integration/akismet.md) diff --git a/doc/development/spam_protection_and_captcha/model_and_services.md b/doc/development/spam_protection_and_captcha/model_and_services.md new file mode 100644 index 00000000000..d0519cba68f --- /dev/null +++ b/doc/development/spam_protection_and_captcha/model_and_services.md @@ -0,0 +1,155 @@ +--- +stage: Manage +group: Authentication and Authorization +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Model and services spam protection and CAPTCHA support + +Before adding any spam or CAPTCHA support to the REST API, GraphQL API, or Web UI, you must +first add the necessary support to: + +1. The backend ActiveRecord models. +1. The services layer. + +All or most of the following changes are required, regardless of the type of spam or CAPTCHA request +implementation you are supporting. Some newer features which are completely based on the GraphQL API +may not have any controllers, and don't require you to add the `mark_as_spam` action to the controller. + +To do this: + +1. [Add `Spammable` support to the ActiveRecord model](#add-spammable-support-to-the-activerecord-model). +1. [Add support for the `mark_as_spam` action to the controller](#add-support-for-the-mark_as_spam-action-to-the-controller). +1. [Add a call to SpamActionService to the execute method of services](#add-a-call-to-spamactionservice-to-the-execute-method-of-services). + +## Add `Spammable` support to the ActiveRecord model + +1. Include the `Spammable` module in the model class: + + ```ruby + include Spammable + ``` + +1. Add: `attr_spammable` to indicate which fields can be checked for spam. Up to + two fields per model are supported: a "`title`" and a "`description`". You can + designate which fields to consider the "`title`" or "`description`". For example, + this line designates the `content` field as the `description`: + + ```ruby + attr_spammable :content, spam_description: true + ``` + +1. Add a `#check_for_spam?` method implementation: + + ```ruby + def check_for_spam?(user:) + # Return a boolean result based on various applicable checks, which may include + # which attributes have changed, the type of user, whether the data is publicly + # visible, and other criteria. This may vary based on the type of model, and + # may change over time as spam checking requirements evolve. + end + ``` + + Refer to other existing `Spammable` models' + implementations of this method for examples of the required logic checks. + +## Add support for the `mark_as_spam` action to the controller + +The `SpammableActions::AkismetMarkAsSpamAction` module adds support for a `#mark_as_spam` action +to a controller. This controller allows administrators to manage spam for the associated +`Spammable` model in the [Spam Log section](../../integration/akismet.md) of the Admin Area page. + +1. Include the `SpammableActions::AkismetMarkAsSpamAction` module in the controller. + + ```ruby + include SpammableActions::AkismetMarkAsSpamAction + ``` + +1. Add a `#spammable_path` method implementation. The spam administration page redirects + to this page after edits. Refer to other existing controllers' implementations + of this method for examples of the type of path logic required. In general, it should + be the `#show` action for the `Spammable` model's controller. + + ```ruby + def spammable_path + widget_path(widget) + end + ``` + +NOTE: +There may be other changes needed to controllers, depending on how the feature is +implemented. See [Web UI](web_ui.md) for more details. + +## Add a call to SpamActionService to the execute method of services + +This approach applies to any service which can persist spammable attributes: + +1. In the relevant Create or Update service under `app/services`, pass in a populated + `Spam::SpamParams` instance. (Refer to instructions later on in this page.) +1. Use it and the `Spammable` model instance to execute a `Spam::SpamActionService` instance. +1. If the spam check fails: + - An error is added to the model, which causes it to be invalid and prevents it from being saved. + - The `needs_recaptcha` property is set to `true`. + + These changes to the model enable it for handling by the subsequent backend and frontend CAPTCHA logic. + +Make these changes to each relevant service: + +1. Change the constructor to take a `spam_params:` argument as a required named argument. + + Using named arguments for the constructor helps you identify all the calls to + the constructor that need changing. It's less risky because the interpreter raises + type errors unless the caller is changed to pass the `spam_params` argument. + If you use an IDE (such as RubyMine) which supports this, your + IDE flags it as an error in the editor. + +1. In the constructor, set the `@spam_params` instance variable from the `spam_params` constructor + argument. Add an `attr_reader: :spam_params` in the `private` section of the class. + +1. In the `execute` method, add a call to execute the `Spam::SpamActionService`. + (You can also use `before_create` or `before_update`, if the service + uses that pattern.) This method uses named arguments, so its usage is clear if + you refer to existing examples. However, two important considerations exist: + 1. The `SpamActionService` must be executed _after_ all necessary changes are made to + the unsaved (and dirty) `Spammable` model instance. This ordering ensures + spammable attributes exist to be spam-checked. + 1. The `SpamActionService` must be executed _before_ the model is checked for errors and + attempting a `save`. If potential spam is detected in the model's changed attributes, we must prevent a save. + +```ruby +module Widget + class CreateService < ::Widget::BaseService + # NOTE: We require the spam_params and do not default it to nil, because + # spam_checking is likely to be necessary. However, if there is not a request available in scope + # in the caller (for example, a note created via email) and the required arguments to the + # SpamParams constructor are not otherwise available, spam_params: must be explicitly passed as nil. + def initialize(project:, current_user: nil, params: {}, spam_params:) + super(project: project, current_user: current_user, params: params) + + @spam_params = spam_params + end + + def execute + widget = Widget::BuildService.new(project, current_user, params).execute + + # More code that may manipulate dirty model before it is spam checked. + + # NOTE: do this AFTER the spammable model is instantiated, but BEFORE + # it is validated or saved. + Spam::SpamActionService.new( + spammable: widget, + spam_params: spam_params, + user: current_user, + # Or `action: :update` for a UpdateService or service for an existing model. + action: :create + ).execute + + # Possibly more code related to saving model, but should not change any attributes. + + widget.save + end + + private + + attr_reader :spam_params +``` diff --git a/doc/development/spam_protection_and_captcha/web_ui.md b/doc/development/spam_protection_and_captcha/web_ui.md new file mode 100644 index 00000000000..6aa01f401bd --- /dev/null +++ b/doc/development/spam_protection_and_captcha/web_ui.md @@ -0,0 +1,196 @@ +--- +stage: Manage +group: Authentication and Authorization +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Web UI spam protection and CAPTCHA support + +The approach for adding spam protection and CAPTCHA support to a new UI area of the GitLab application +depends upon how the existing code is implemented. + +## Supported scenarios of request submissions + +Three different scenarios are supported. Two are used with JavaScript XHR/Fetch requests +for either Apollo or Axios, and one is used only with standard HTML form requests: + +1. A JavaScript-based submission (possibly via Vue) + 1. Using Apollo (GraphQL API via Fetch/XHR request) + 1. Using Axios (REST API via Fetch/XHR request) +1. A standard HTML form submission (HTML request) + +Some parts of the implementation depend upon which of these scenarios you must support. + +## Implementation tasks specific to JavaScript XHR/Fetch requests + +Two approaches are fully supported: + +1. Apollo, using the GraphQL API. +1. Axios, using either the GraphQL API. + +The spam and CAPTCHA-related data communication between the frontend and backend requires no +additional fields being added to the models. Instead, communication is handled: + +- Through custom header values in the request. +- Through top-level JSON fields in the response. + +The spam and CAPTCHA-related logic is also cleanly abstracted into reusable modules and helper methods +which can wrap existing logic, and only alter the existing flow if potential spam +is detected or a CAPTCHA display is needed. This approach allows the spam and CAPTCHA +support to be easily added to new areas of the application with minimal changes to +existing logic. In the case of the frontend, potentially **zero** changes are needed! + +On the frontend, this is handled abstractly and transparently using `ApolloLink` for Apollo, and an +Axios interceptor for Axios. The CAPTCHA display is handled by a standard GitLab UI / Pajamas modal +component. You can find all the relevant frontend code under `app/assets/javascripts/captcha`. + +However, even though the actual handling of the request interception and +modal is transparent, without any mandatory changes to the involved JavaScript or Vue components +for the form or page, changes in request or error handling may be required. Changes are needed +because the existing behavior may not work correctly: for example, if a failed or cancelled +CAPTCHA display interrupts the normal request flow or UI updates. +Careful exploratory testing of all scenarios is important to uncover any potential +problems. + +This sequence diagram illustrates the normal CAPTCHA flow for JavaScript XHR/Fetch requests +on the frontend: + +```mermaid +sequenceDiagram + participant U as User + participant V as Vue/JS Application + participant A as ApolloLink or Axios Interceptor + participant G as GitLab API + U->>V: Save model + V->>A: Request + A->>G: Request + G--xA: Response with error and spam/CAPTCHA related fields + A->>U: CAPTCHA presented in modal + U->>A: CAPTCHA solved to obtain valid CAPTCHA response + A->>G: Request with valid CAPTCHA response and SpamLog ID in headers + G-->>A: Response with success + A-->>V: Response with success +``` + +The backend is also cleanly abstracted via mixin modules and helper methods. The three main +changes required to the relevant backend controller actions (normally just `create`/`update`) are: + +1. Create a `SpamParams` parameter object instance based on the request, using the simple static + `#new_from_request` factory method. This method takes a request, and returns a `SpamParams` instance. +1. Pass the created `SpamParams` instance as the `spam_params` named argument to the + Service class constructor, which you should have already added. If the spam check indicates + the changes to the model are possibly spam, then: + - An error is added to the model. + - The `needs_recaptcha` property on the model is set to true. +1. Wrap the existing controller action return value (rendering or redirecting) in a block passed to + a `#with_captcha_check_json_format` helper method, which transparently handles: + 1. Check if CAPTCHA is enabled, and if so, proceeding with the next step. + 1. Checking if there the model contains an error, and the `needs_recaptcha` flag is true. + - If yes: Add the appropriate spam or CAPTCHA fields to the JSON response, and return + a `409 - Conflict` HTTP status code. + - If no (if CAPTCHA is disabled or if no spam was detected): The normal request return + logic passed in the block is run. + +Thanks to the abstractions, it's more straightforward to implement than it is to explain it. +You don't have to worry much about the hidden details! + +Make these changes: + +## Add support to the controller actions + +If the feature's frontend submits directly to controller actions, and does not only use the GraphQL +API, then you must add support to the appropriate controllers. + +The action methods may be directly in the controller class, or they may be abstracted +to a module included in the controller class. Our example uses a module. The +only difference when directly modifying the controller: +`extend ActiveSupport::Concern` is not required. + +```ruby +module WidgetsActions + # NOTE: This `extend` probably already exists, but it MUST be moved to occur BEFORE all + # `include` statements. Otherwise, confusing bugs may occur in which the methods + # in the included modules cannot be found. + extend ActiveSupport::Concern + + include SpammableActions::CaptchaCheck::JsonFormatActionsSupport + + def create + spam_params = ::Spam::SpamParams.new_from_request(request: request) + widget = ::Widgets::CreateService.new( + project: project, + current_user: current_user, + params: params, + spam_params: spam_params + ).execute + + respond_to do |format| + format.json do + with_captcha_check_json_format do + # The action's existing `render json: ...` (or wrapper method) and related logic. Possibly + # including different rendering cases if the model is valid or not. It's all wrapped here + # within the `with_captcha_check_json_format` block. For example: + if widget.valid? + render json: serializer.represent(widget) + else + render json: { errors: widget.errors.full_messages }, status: :unprocessable_entity + end + end + end + end + end +end +``` + +## Implementation tasks specific to HTML form requests + +Some areas of the application have not been converted to use the GraphQL API via +a JavaScript client, but instead rely on standard Rails HAML form submissions via an +`HTML` MIME type request. In these areas, the action returns a pre-rendered HTML (HAML) page +as the response body. Unfortunately, in this case +[it is not possible](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66427#note_636989204) +to use any of the JavaScript-based frontend support as described above. Instead we must use an +alternate approach which handles the rendering of the CAPTCHA form via a HAML template. + +Everything is still cleanly abstracted, and the implementation in the backend +controllers is virtually identical to the JavaScript/JSON based approach. Replace the +word `JSON` with `HTML` (using the appropriate case) in the module names and helper methods. + +The action methods might be directly in the controller, or they +might be in a module. In this example, they are directly in the +controller, and we also do an `update` method instead of `create`: + +```ruby +class WidgetsController < ApplicationController + include SpammableActions::CaptchaCheck::HtmlFormatActionsSupport + + def update + # Existing logic to find the `widget` model instance... + + spam_params = ::Spam::SpamParams.new_from_request(request: request) + ::Widgets::UpdateService.new( + project: project, + current_user: current_user, + params: params, + spam_params: spam_params + ).execute(widget) + + respond_to do |format| + format.html do + if widget.valid? + # NOTE: `spammable_path` is required by the `SpammableActions::AkismetMarkAsSpamAction` + # module, and it should have already been implemented on this controller according to + # the instructions above. It is reused here to avoid duplicating the route helper call. + redirect_to spammable_path + else + # If we got here, there were errors on the model instance - from a failed spam check + # and/or other validation errors on the model. Either way, we'll re-render the form, + # and if a CAPTCHA render is necessary, it will be automatically handled by + # `with_captcha_check_html_format` + with_captcha_check_html_format { render :edit } + end + end + end + end +end +``` diff --git a/doc/development/testing_guide/end_to_end/best_practices.md b/doc/development/testing_guide/end_to_end/best_practices.md index 405ff40ef6a..e0f6cbe632d 100644 --- a/doc/development/testing_guide/end_to_end/best_practices.md +++ b/doc/development/testing_guide/end_to_end/best_practices.md @@ -17,13 +17,16 @@ In case custom inflection logic is needed, custom inflectors are added in the [q ## Link a test to its test case Every test should have a corresponding test case in the [GitLab project Test Cases](https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases) as well as a results issue in the [Quality Test Cases project](https://gitlab.com/gitlab-org/quality/testcases/-/issues). -It's recommended that you reuse the issue created to plan the test as the results issue. If a test case or results issue does not already exist you -can create them yourself. Alternatively, you can run the test in a pipeline that has reporting -enabled and the test-case reporter will automatically create a new test case and/or results issue and link the results issue to it's corresponding test case. +If a test case issue does not yet exist you can create one yourself. To do so, create a new +issue in the [Test Cases](https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases) GitLab project +with a placeholder title. After the test case URL is linked to a test in the code, when the test is +run in a pipeline that has reporting enabled, the `report-results` script automatically updates the +test case and the results issue. +If a results issue does not yet exist, the `report-results` script automatically creates one and +links it to its corresponding test case. -Whether you create a new test case or one is created automatically, you will need to manually add -a `testcase` RSpec metadata tag. In most cases, a single test will be associated with a single test case - ([see below for exceptions](#exceptions)). +To link a test case to a test in the code, you must manually add a `testcase` RSpec metadata tag. +In most cases, a single test is associated with a single test case. For example: @@ -41,7 +44,7 @@ RSpec.describe 'Stage' do end ``` -### Exceptions +### For shared tests Most tests are defined by a single line of a `spec` file, which is why those tests can be linked to a single test case via the `testcase` tag. @@ -54,24 +57,19 @@ multiple tests, including: - Templated tests. - Tests in shared examples that include more than one example. -In those and similar cases we can't assign a single `testcase` tag and so we rely on the test-case -reporter to programmatically determine the correct test case based on the name and description of -the test. In such cases, the test-case reporter will automatically create a test case and/or results issue -the first time the test runs, if none exist already. +In those and similar cases we need to include the test case link by other means. -In such a case, if you create the test case or results issue yourself or want to reuse an existing issue, -you must use this [end-to-end test issue template](https://gitlab.com/gitlab-org/quality/testcases/-/blob/master/.gitlab/issue_templates/End-to-end%20Test.md) -to format the issue description. (Note you must copy/paste this for test cases as templates aren't currently available.) - -To illustrate, there are two tests in the shared examples in [`qa/specs/features/ee/browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/47b17db82c38ab704a23b5ba5d296ea0c6a732c8/qa/qa/specs/features/ee/browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb): +To illustrate, there are two tests in the shared examples in [`qa/specs/features/ee/browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb): ```ruby -shared_examples 'only user with access pushes and merges' do - it 'unselected maintainer user fails to push' do +shared_examples 'unselected maintainer' do |testcase| + it 'user fails to push', testcase: testcase do ... end +end - it 'selected developer user pushes and merges' do +shared_examples 'selected developer' do |testcase| + it 'user pushes and merges', testcase: testcase do ... end end @@ -84,112 +82,22 @@ RSpec.describe 'Create' do describe 'Restricted protected branch push and merge' do context 'when only one user is allowed to merge and push to a protected branch' do ... - it_behaves_like 'only user with access pushes and merges' - end - end -end -``` - -There would be two associated test cases, one for each shared example, with the following content: - -[Test 1 Test Case](https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/347774): - -````markdown -```markdown -Title: browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb | Create Restricted -protected branch push and merge when only one user is allowed to merge and push to a protected -branch behaves like only user with access pushes and merges selecte... - -Description: -### Full description - -Create Restricted protected branch push and merge when only one user is allowed to merge and push -to a protected branch behaves like only user with access pushes and merges selected developer user -pushes and merges - -### File path - -./qa/specs/features/ee/browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb - -### DO NOT EDIT BELOW THIS LINE - -Active and historical test results: - -https://gitlab.com/gitlab-org/quality/testcases/-/issues/2177 - -``` -```` - -[Test 1 Results Issue](https://gitlab.com/gitlab-org/quality/testcases/-/issues/2177): - -````markdown -```markdown -Title: browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb | Create Restricted -protected branch push and merge when only one user is allowed to merge and push to a protected -branch behaves like only user with access pushes and merges selecte... - -Description: -### Full description - -Create Restricted protected branch push and merge when only one user is allowed to merge and push -to a protected branch behaves like only user with access pushes and merges selected developer user -pushes and merges - -### File path - -./qa/specs/features/ee/browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb -``` -```` - -[Test 2 Test Case](https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/347775): - -````markdown -```markdown -Title: browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb | Create Restricted -protected branch push and merge when only one user is allowed to merge and push to a protected -branch behaves like only user with access pushes and merges unselec... - -Description: -### Full description - -Create Restricted protected branch push and merge when only one user is allowed to merge and push -to a protected branch behaves like only user with access pushes and merges unselected maintainer -user fails to push - -### File path - -./qa/specs/features/ee/browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb - -### DO NOT EDIT BELOW THIS LINE - -Active and historical test results: + it_behaves_like 'unselected maintainer', 'https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/347775' + it_behaves_like 'selected developer', 'https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/347774' + end -https://gitlab.com/gitlab-org/quality/testcases/-/issues/2176 + context 'when only one group is allowed to merge and push to a protected branch' do + ... + it_behaves_like 'unselected maintainer', 'https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/347772' + it_behaves_like 'selected developer', 'https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/347773' + end + end +end ``` -```` - -[Test 2 Results Issue](https://gitlab.com/gitlab-org/quality/testcases/-/issues/2176): - -````markdown -```markdown -Title: browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb | Create Restricted -protected branch push and merge when only one user is allowed to merge and push to a protected -branch behaves like only user with access pushes and merges unselec... - -Description: -### Full description - -Create Restricted protected branch push and merge when only one user is allowed to merge and push -to a protected branch behaves like only user with access pushes and merges unselected maintainer -user fails to push -### File path - -./qa/specs/features/ee/browser_ui/3_create/repository/restrict_push_protected_branch_spec.rb -``` -```` +We recommend creating four associated test cases, two for each shared example. ## Prefer API over UI @@ -518,3 +426,12 @@ Neither problem is present if we create a custom negatable matcher because the ` would be used, which would wait only as long as necessary for the job to disappear. Lastly, negatable matchers are preferred over using matchers of the form `have_no_*` because it's a common and familiar practice to negate matchers using `not_to`. If we facilitate that practice by adding negatable matchers, we make it easier for subsequent test authors to write efficient tests. + +## Use logger over puts + +We currently use Rails `logger` to handle logs in both GitLab QA application and end-to-end tests. +This provides additional functionalities when compared with `puts`, such as: + +- Ability to specify the logging level. +- Ability to tag similar logs. +- Auto-formatting log messages. diff --git a/doc/development/testing_guide/end_to_end/feature_flags.md b/doc/development/testing_guide/end_to_end/feature_flags.md index 238b1ccd8f9..c3e3f117c2b 100644 --- a/doc/development/testing_guide/end_to_end/feature_flags.md +++ b/doc/development/testing_guide/end_to_end/feature_flags.md @@ -130,7 +130,7 @@ You can set this variable inside the `fabricate_via_api` call. For a consistent - Add the word `activated` to the end of a variable's name. - Inside the `initialize` method, set the variable's default value. -For example: +For example: ```ruby def initialize diff --git a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md index 07f7e9582ee..f9b505a8271 100644 --- a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md +++ b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md @@ -20,7 +20,7 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec | `:github` | The test requires a GitHub personal access token. | | `:group_saml` | The test requires a GitLab instance that has SAML SSO enabled at the group level. Interacts with an external SAML identity provider. Paired with the `:orchestrated` tag. | | `:instance_saml` | The test requires a GitLab instance that has SAML SSO enabled at the instance level. Interacts with an external SAML identity provider. Paired with the `:orchestrated` tag. | -| `:integrations` | This aims to test the available [integrations](../../../user/project/integrations/overview.md#integrations-listing). The test requires Docker to be installed in the run context. It will provision the containers and can be run against a local instance or using the `gitlab-qa` scenario `Test::Integration::Integrations` | +| `:integrations` | This aims to test the available [integrations](../../../user/project/integrations/overview.md#integrations-listing). The test requires Docker to be installed in the run context. It will provision the containers and can be run against a local instance or using the `gitlab-qa` scenario `Test::Integration::Integrations` | | `:service_ping_disabled` | The test interacts with the GitLab configuration service ping at the instance level to turn admin setting service ping checkbox on or off. This tag will have the test run only in the `service_ping_disabled` job and must be paired with the `:orchestrated` and `:requires_admin` tags. | | `:jira` | The test requires a Jira Server. [GitLab-QA](https://gitlab.com/gitlab-org/gitlab-qa) provisions the Jira Server in a Docker container when the `Test::Integration::Jira` test scenario is run. | `:kubernetes` | The test includes a GitLab instance that is configured to be run behind an SSH tunnel, allowing a TLS-accessible GitLab. This test also includes provisioning of at least one Kubernetes cluster to test against. _This tag is often be paired with `:orchestrated`._ | @@ -42,6 +42,7 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec | `:requires_praefect` | The test requires that the GitLab instance uses [Gitaly Cluster](../../../administration/gitaly/praefect.md) (a.k.a. Praefect) as the repository storage . It's assumed to be used by default but if not the test can be skipped by setting `QA_CAN_TEST_PRAEFECT` to `false`. | | `:runner` | The test depends on and sets up a GitLab Runner instance, typically to run a pipeline. | | `:skip_live_env` | The test is excluded when run against live deployed environments such as Staging, Canary, and Production. | +| `:skip_fips_env` | The test is excluded when run against an environment in FIPS mode. | | `:skip_signup_disabled` | The test uses UI to sign up a new user and is skipped in any environment that does not allow new user registration via the UI. | | `:smoke` | The test belongs to the test suite which verifies basic functionality of a GitLab instance.| | `:smtp` | The test requires a GitLab instance to be configured to use an SMTP server. Tests SMTP notification email delivery from GitLab by using MailHog. | diff --git a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md index 7fb95769fc2..49a9124253d 100644 --- a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md +++ b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md @@ -169,135 +169,6 @@ docker run \ Where `interface_ip_address` is your local network's interface IP, which you can find with the `ifconfig` command. The same would apply to GDK running with the instance address as `localhost` too. -## Guide to run and debug Monitor tests - -### How to set up - -To run the Monitor tests locally, against the GDK, please follow the preparation steps below: - -1. Complete the [Prerequisites](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/auto_devops/index.md#prerequisites-for-gitlab-team-members-only), at least through step 5. Note that the monitor tests do not require permissions to work with GKE because they use [k3s as a Kubernetes cluster provider](https://github.com/rancher/k3s). -1. The test setup deploys the app in a Kubernetes cluster, using the Auto DevOps deployment strategy. -To enable Auto DevOps in GDK, follow the [associated setup](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/auto_devops/index.md#setup) instructions. If you have problems, review the [troubleshooting guide](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/auto_devops/tips_and_troubleshooting.md) or reach out to the `#gdk` channel in the internal GitLab Slack. -1. Do [secure your GitLab instance](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/auto_devops/index.md#secure-your-gitlab-instance) since it is now publicly accessible on `https://[YOUR-PORT].qa-tunnel.gitlab.info`. -1. Install the Kubernetes command line tool known as `kubectl`. Use the [official installation instructions](https://kubernetes.io/docs/tasks/tools/). - -You might see NGINX issues when you run `gdk start` or `gdk restart`. In that case, run `sft login` to revalidate your credentials and regain access the QA Tunnel. - -### How to run - -Navigate to the folder in `/your-gdk/gitlab/qa` and issue the command: - -```shell -QA_DEBUG=true WEBDRIVER_HEADLESS=false GITLAB_ADMIN_USERNAME=rootusername GITLAB_ADMIN_PASSWORD=rootpassword GITLAB_QA_ACCESS_TOKEN=your_token_here GITLAB_QA_ADMIN_ACCESS_TOKEN=your_token_here CLUSTER_API_URL=https://kubernetes.docker.internal:6443 bundle exec bin/qa Test::Instance::All https://[YOUR-PORT].qa-tunnel.gitlab.info/ -- qa/specs/features/browser_ui/8_monitor/all_monitor_core_features_spec.rb --tag kubernetes --tag orchestrated --tag requires_admin -``` - -The following includes more information on the command: - -- `QA_DEBUG` - Set to `true` to verbosely log page object actions. -- `WEBDRIVER_HEADLESS` - When running locally, set to `false` to allow browser tests to be visible - watch your tests being run. -- `GITLAB_ADMIN_USERNAME` - Administrator username to use when adding a license. -- `GITLAB_ADMIN_PASSWORD` - Administrator password to use when adding a license. -- `GITLAB_QA_ACCESS_TOKEN` and `GITLAB_QA_ADMIN_ACCESS_TOKEN` - A valid personal access token with the `api` scope. This is used for API access during tests, and is used in the version that staging is currently running. The `ADMIN_ACCESS_TOKEN` is from a user with administrator access. Used for API access as an administrator during tests. -- `CLUSTER_API_URL` - Use the address `https://kubernetes.docker.internal:6443` . This address is used to enable the cluster to be network accessible while deploying using Auto DevOps. -- `https://[YOUR-PORT].qa-tunnel.gitlab.info/` - The address of your local GDK -- `qa/specs/features/browser_ui/8_monitor/all_monitor_core_features_spec.rb` - The path to the monitor core specs -- `--tag` - the meta-tags used to filter the specs correctly - -At the moment of this writing, there are two specs which run monitor tests: - -- `qa/specs/features/browser_ui/8_monitor/all_monitor_core_features_spec.rb` - has the specs of features in GitLab Free -- `qa/specs/features/ee/browser_ui/8_monitor/all_monitor_features_spec.rb` - has the specs of features for paid GitLab (Enterprise Edition) - -### How to debug - -The monitor tests follow this setup flow: - -1. Creates a k3s cluster on your local machine. -1. Creates a project that has Auto DevOps enabled and uses an Express template (NodeJS) for the app to be deployed. -1. Associates the created cluster to the project and installs GitLab Runner, Prometheus and Ingress which are the needed components for a successful deployment. -1. Creates a CI pipeline with 2 jobs (`build` and `production`) to deploy the app on the Kubernetes cluster. -1. Goes to Operation > Metrics menu to verify data is being received and the app is being monitored successfully. - -The test requires a number of components. The setup requires time to collect the metrics of a real deployment. -The complexity of the setup may lead to problems unrelated to the app. The following sections include common strategies to debug possible issues. - -#### Deployment with Auto DevOps - -When debugging issues in the CI or locally in the CLI, open the Kubernetes job in the pipeline. -In the job log window, click on the top right icon labeled as *"Show complete raw"* to reveal raw job logs. -You can now search through the logs for *Job log*, which matches delimited sections like this one: - -```shell -------- Job log: ------- -``` - -A Job log is a subsection within these logs, related to app deployment. We use two jobs: `build` and `production`. -You can find the root causes of deployment failures in these logs, which can compromise the entire test. -If a `build` job fails, the `production` job doesn't run, and the test fails. - -The long test setup does not take screenshots of failures, which is a known [issue](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/270). -However, if the spec fails (after a successful deployment) then you should be able to find screenshots which display the feature failure. -To access them in CI, go to the main job log window, look on the left side panel's Job artifacts section, and click Browse. - -#### Common issues - -**Container Registry** - -When enabling Auto DevOps in the GDK, you may see issues with the Container Registry, which stores -images of the app to be deployed. - -You can access the Registry is available by opening an existing project. On the left hand menu, -select `Packages & Registries > Container Registries`. If the Registry is available, this page should load normally. - -Also, the Registry should be running in Docker: - -```shell -$ docker ps - -CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES -f035f339506c registry.gitlab.com/gitlab-org/build/cng/gitlab-container-registry:v2.9.1-gitlab "/bin/sh -c 'exec /b…" 3 hours ago Up 3 hours 0.0.0.0:5000->5000/tcp jovial_proskuriakova -``` - -The `gdk status` command shows if the registry is running: - -```shell -run: ./services/registry: (pid 2662) 10875s, normally down; run: log: (pid 65148) 177993s -run: ./services/tunnel_gitlab: (pid 2650) 10875s, normally down; run: log: (pid 65154) 177993s -run: ./services/tunnel_registry: (pid 2651) 10875s, normally down; run: log: (pid 65155) 177993s -``` - -Also, restarting Docker and then, on the Terminal, issue the command -`docker login https://[YOUR-REGISTRY-PORT].qa-tunnel.gitlab.info:443` and use -the GDK credentials to sign in. Note that the Registry port and GDK port aren't -the same. When configuring Auto DevOps in GDK, the `gdk reconfigure` command -outputs the port of the Registry: - -```shell -********************************************* -Tunnel URLs - -GitLab: https://[PORT].qa-tunnel.gitlab.info -Registry: https://[PORT].qa-tunnel.gitlab.info -********************************************* -``` - -These Tunnel URLs are used by the QA SSH Tunnel generated when enabling Auto DevOps on the GDK. - -**Pod Eviction** - -Pod eviction happens when a node in a Kubernetes cluster is running out of memory or disk. After many local deployments this issue can happen. The UI shows that installing Prometheus, GitLab Runner and Ingress failed. How to be sure it is an Eviction? While the test is running, open another Terminal window and debug the current Kubernetes cluster by `kubectl get pods --all-namespaces`. If you observe that Pods have *Evicted status* such as the install-runner here: - -```shell -$ kubectl get pods --all-namespaces - -NAMESPACE NAME READY STATUS RESTARTS AGE -gitlab-managed-apps install-ingress 0/1 Pending 0 25s -gitlab-managed-apps install-prometheus 0/1 Pending 0 12s -gitlab-managed-apps install-runner 0/1 Evicted 0 75s -``` - -You can free some memory with either of the following commands: `docker prune system` or `docker prune volume`. - ## Geo tests Geo end-to-end tests can run locally against a [Geo GDK setup](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/geo.md) or on Geo spun up in Docker containers. diff --git a/doc/development/testing_guide/flaky_tests.md b/doc/development/testing_guide/flaky_tests.md index 09fc8f4d33e..3c8df7d3416 100644 --- a/doc/development/testing_guide/flaky_tests.md +++ b/doc/development/testing_guide/flaky_tests.md @@ -117,14 +117,17 @@ reproduction. - [Don't wait for AJAX when no AJAX request is fired](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/30461): <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10454> - [Bis](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/34647): <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12626> -#### PhantomJS / WebKit related issues - -- Memory is through the roof! (Load images but block images requests!): <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12003> - #### Capybara expectation times out - [Test imports a project (via Sidekiq) that is growing over time, leading to timeouts when the import takes longer than 60 seconds](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/22599) +### Hanging specs + +If a spec hangs, it might be caused by a [bug in Rails](https://github.com/rails/rails/issues/34310): + +- <https://gitlab.com/gitlab-org/gitlab/-/merge_requests/81112> +- <https://gitlab.com/gitlab-org/gitlab/-/issues/337039> + ## Resources - [Flaky Tests: Are You Sure You Want to Rerun Them?](https://semaphoreci.com/blog/2017/04/20/flaky-tests.html) diff --git a/doc/development/uploads.md b/doc/development/uploads.md index 14ecf38e21c..1860f898a26 100644 --- a/doc/development/uploads.md +++ b/doc/development/uploads.md @@ -1,430 +1,9 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: 'uploads/index.md' +remove_date: '2022-04-30' --- -# Uploads development documentation +This document was moved to [another location](uploads/index.md). -[GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) has special rules for handling uploads. -We process the upload in Workhorse to prevent occupying a Ruby process on I/O operations and because it is cheaper. -This process can also directly upload to object storage. - -## The problem description - -The following graph explains machine boundaries in a scalable GitLab installation. Without any Workhorse optimization in place, we can expect incoming requests to follow the numbers on the arrows. - -```mermaid -graph TB - subgraph "load balancers" - LB(Proxy) - end - - subgraph "Shared storage" - nfs(NFS) - end - - subgraph "redis cluster" - r(persisted redis) - end - LB-- 1 -->Workhorse - - subgraph "web or API fleet" - Workhorse-- 2 -->rails - end - rails-- "3 (write files)" -->nfs - rails-- "4 (schedule a job)" -->r - - subgraph sidekiq - s(sidekiq) - end - s-- "5 (fetch a job)" -->r - s-- "6 (read files)" -->nfs -``` - -We have three challenges here: performance, availability, and scalability. - -### Performance - -Rails process are expensive in terms of both CPU and memory. Ruby [global interpreter lock](https://en.wikipedia.org/wiki/Global_interpreter_lock) adds to cost too because the Ruby process spends time on I/O operations on step 3 causing incoming requests to pile up. - -In order to improve this, [disk buffered upload](#disk-buffered-upload) was implemented. With this, Rails no longer deals with writing uploaded files to disk. - -```mermaid -graph TB - subgraph "load balancers" - LB(HA Proxy) - end - - subgraph "Shared storage" - nfs(NFS) - end - - subgraph "redis cluster" - r(persisted redis) - end - LB-- 1 -->Workhorse - - subgraph "web or API fleet" - Workhorse-- "3 (without files)" -->rails - end - Workhorse -- "2 (write files)" -->nfs - rails-- "4 (schedule a job)" -->r - - subgraph sidekiq - s(sidekiq) - end - s-- "5 (fetch a job)" -->r - s-- "6 (read files)" -->nfs -``` - -### Availability - -There's also an availability problem in this setup, NFS is a [single point of failure](https://en.wikipedia.org/wiki/Single_point_of_failure). - -To address this problem an HA object storage can be used and it's supported by [direct upload](#direct-upload) - -### Scalability - -Scaling NFS is outside of our support scope, and NFS is not a part of cloud native installations. - -All features that require Sidekiq and do not use direct upload doesn't work without NFS. In Kubernetes, machine boundaries translate to PODs, and in this case the uploaded file is written into the POD private disk. Since Sidekiq POD cannot reach into other pods, the operation fails to read it. - -## How to select the proper level of acceleration? - -Selecting the proper acceleration is a tradeoff between speed of development and operational costs. - -We can identify three major use-cases for an upload: - -1. **storage:** if we are uploading for storing a file (like artifacts, packages, or discussion attachments). In this case [direct upload](#direct-upload) is the proper level as it's the less resource-intensive operation. Additional information can be found on [File Storage in GitLab](file_storage.md). -1. **in-controller/synchronous processing:** if we allow processing **small files** synchronously, using [disk buffered upload](#disk-buffered-upload) may speed up development. -1. **Sidekiq/asynchronous processing:** Asynchronous processing must implement [direct upload](#direct-upload), the reason being that it's the only way to support Cloud Native deployments without a shared NFS. - -For more details about currently broken feature see [epic &1802](https://gitlab.com/groups/gitlab-org/-/epics/1802). - -### Handling repository uploads - -Some features involves Git repository uploads without using a regular Git client. -Some examples are uploading a repository file from the web interface and [design management](../user/project/issues/design_management.md). - -Those uploads requires the rails controller to act as a Git client in lieu of the user. -Those operation falls into _in-controller/synchronous processing_ category, but we have no warranties on the file size. - -In case of a LFS upload, the file pointer is committed synchronously, but file upload to object storage is performed asynchronously with Sidekiq. - -## Upload encodings - -By upload encoding we mean how the file is included within the incoming request. - -We have three kinds of file encoding in our uploads: - -1. <i class="fa fa-check-circle"></i> **multipart**: `multipart/form-data` is the most common, a file is encoded as a part of a multipart encoded request. -1. <i class="fa fa-check-circle"></i> **body**: some APIs uploads files as the whole request body. -1. <i class="fa fa-times-circle"></i> **JSON**: some JSON APIs upload files as base64-encoded strings. This requires a change to GitLab Workhorse, - which is tracked [in this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/325068). - -## Uploading technologies - -By uploading technologies we mean how all the involved services interact with each other. - -GitLab supports 3 kinds of uploading technologies, here follows a brief description with a sequence diagram for each one. Diagrams are not meant to be exhaustive. - -### Rack Multipart upload - -This is the default kind of upload, and it's the most expensive in terms of resources. - -In this case, Workhorse is unaware of files being uploaded and acts as a regular proxy. - -When a multipart request reaches the rails application, `Rack::Multipart` leaves behind temporary files in `/tmp` and uses valuable Ruby process time to copy files around. - -```mermaid -sequenceDiagram - participant c as Client - participant w as Workhorse - participant r as Rails - - activate c - c ->>+w: POST /some/url/upload - w->>+r: POST /some/url/upload - - r->>r: save the incoming file on /tmp - r->>r: read the file for processing - - r-->>-c: request result - deactivate c - deactivate w -``` - -### Disk buffered upload - -This kind of upload avoids wasting resources caused by handling upload writes to `/tmp` in rails. - -This optimization is not active by default on REST API requests. - -When enabled, Workhorse looks for files in multipart MIME requests, uploading -any it finds to a temporary file on shared storage. The MIME data in the request -is replaced with the path to the corresponding file before it is forwarded to -Rails. - -To prevent abuse of this feature, Workhorse signs the modified request with a -special header, stating which entries it modified. Rails ignores any -unsigned path entries. - -```mermaid -sequenceDiagram - participant c as Client - participant w as Workhorse - participant r as Rails - participant s as NFS - - activate c - c ->>+w: POST /some/url/upload - - w->>+s: save the incoming file on a temporary location - s-->>-w: request result - - w->>+r: POST /some/url/upload - Note over w,r: file was replaced with its location<br>and other metadata - - opt requires async processing - r->>+redis: schedule a job - redis-->>-r: job is scheduled - end - - r-->>-c: request result - deactivate c - w->>-w: cleanup - - opt requires async processing - activate sidekiq - sidekiq->>+redis: fetch a job - redis-->>-sidekiq: job - - sidekiq->>+s: read file - s-->>-sidekiq: file - - sidekiq->>sidekiq: process file - - deactivate sidekiq - end -``` - -### Direct upload - -This is the more advanced acceleration technique we have in place. - -Workhorse asks Rails for temporary pre-signed object storage URLs and directly uploads to object storage. - -In this setup, an extra Rails route must be implemented in order to handle authorization. Examples of this can be found in: - -- [`Projects::LfsStorageController`](https://gitlab.com/gitlab-org/gitlab/-/blob/cc723071ad337573e0360a879cbf99bc4fb7adb9/app/controllers/projects/lfs_storage_controller.rb) - and [its routes](https://gitlab.com/gitlab-org/gitlab/-/blob/cc723071ad337573e0360a879cbf99bc4fb7adb9/config/routes/git_http.rb#L31-32). -- [API endpoints for uploading packages](packages.md#file-uploads). - -Direct upload falls back to _disk buffered upload_ when `direct_upload` is disabled inside the [object storage setting](../administration/uploads.md#object-storage-settings). -The answer to the `/authorize` call contains only a file system path. - -```mermaid -sequenceDiagram - participant c as Client - participant w as Workhorse - participant r as Rails - participant os as Object Storage - - activate c - c ->>+w: POST /some/url/upload - - w ->>+r: POST /some/url/upload/authorize - Note over w,r: this request has an empty body - r-->>-w: presigned OS URL - - w->>+os: PUT file - Note over w,os: file is stored on a temporary location. Rails select the destination - os-->>-w: request result - - w->>+r: POST /some/url/upload - Note over w,r: file was replaced with its location<br>and other metadata - - r->>+os: move object to final destination - os-->>-r: request result - - opt requires async processing - r->>+redis: schedule a job - redis-->>-r: job is scheduled - end - - r-->>-c: request result - deactivate c - w->>-w: cleanup - - opt requires async processing - activate sidekiq - sidekiq->>+redis: fetch a job - redis-->>-sidekiq: job - - sidekiq->>+os: get object - os-->>-sidekiq: file - - sidekiq->>sidekiq: process file - - deactivate sidekiq - end -``` - -## How to add a new upload route - -In this section, we describe how to add a new upload route [accelerated](#uploading-technologies) by Workhorse for [body and multipart](#upload-encodings) encoded uploads. - -Upload routes belong to one of these categories: - -1. Rails controllers: uploads handled by Rails controllers. -1. Grape API: uploads handled by a Grape API endpoint. -1. GraphQL API: uploads handled by a GraphQL resolve function. - -WARNING: -GraphQL uploads do not support [direct upload](#direct-upload) yet. Depending on the use case, the feature may not work on installations without NFS (like GitLab.com or Kubernetes installations). Uploading to object storage inside the GraphQL resolve function may result in timeout errors. For more details please follow [issue #280819](https://gitlab.com/gitlab-org/gitlab/-/issues/280819). - -### Update Workhorse for the new route - -For both the Rails controller and Grape API uploads, Workhorse has to be updated in order to get the -support for the new upload route. - -1. Open a new issue in the [Workhorse tracker](https://gitlab.com/gitlab-org/gitlab-workhorse/-/issues/new) describing precisely the new upload route: - - The route's URL. - - The [upload encoding](#upload-encodings). - - If possible, provide a dump of the upload request. -1. Implement and get the MR merged for this issue above. -1. Ask the Maintainers of [Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) to create a new release. You can do that in the MR - directly during the maintainer review or ask for it in the `#workhorse` Slack channel. -1. Bump the [Workhorse version file](https://gitlab.com/gitlab-org/gitlab/-/blob/master/GITLAB_WORKHORSE_VERSION) - to the version you have from the previous points, or bump it in the same merge request that contains - the Rails changes (see [Implementing the new route with a Rails controller](#implementing-the-new-route-with-a-rails-controller) or [Implementing the new route with a Grape API endpoint](#implementing-the-new-route-with-a-grape-api-endpoint) below). - -### Implementing the new route with a Rails controller - -For a Rails controller upload, we usually have a [multipart](#upload-encodings) upload and there are a -few things to do: - -1. The upload is available under the parameter name you're using. For example, it could be an `artifact` - or a nested parameter such as `user[avatar]`. Let's say that we have the upload under the - `file` parameter, reading `params[:file]` should get you an [`UploadedFile`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/uploaded_file.rb) instance. -1. Generally speaking, it's a good idea to check if the instance is from the [`UploadedFile`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/uploaded_file.rb) class. For example, see how we checked -[that the parameter is indeed an `UploadedFile`](https://gitlab.com/gitlab-org/gitlab/-/commit/ea30fe8a71bf16ba07f1050ab4820607b5658719#51c0cc7a17b7f12c32bc41cfab3649ff2739b0eb_79_77). - -WARNING: -**Do not** call `UploadedFile#from_params` directly! Do not build an [`UploadedFile`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/uploaded_file.rb) -instance using `UploadedFile#from_params`! This method can be unsafe to use depending on the `params` -passed. Instead, use the [`UploadedFile`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/uploaded_file.rb) -instance that [`multipart.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/middleware/multipart.rb) -builds automatically for you. - -### Implementing the new route with a Grape API endpoint - -For a Grape API upload, we can have [body or a multipart](#upload-encodings) upload. Things are slightly more complicated: two endpoints are needed. One for the -Workhorse pre-upload authorization and one for accepting the upload metadata from Workhorse: - -1. Implement an endpoint with the URL + `/authorize` suffix that will: - - Check that the request is coming from Workhorse with the `require_gitlab_workhorse!` from the [API helpers](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/helpers.rb). - - Check user permissions. - - Set the status to `200` with `status 200`. - - Set the content type with `content_type Gitlab::Workhorse::INTERNAL_API_CONTENT_TYPE`. - - Use your dedicated `Uploader` class (let's say that it's `FileUploader`) to build the response with `FileUploader.workhorse_authorize(params)`. -1. Implement the endpoint for the upload request that will: - - Require all the `UploadedFile` objects as parameters. - - For example, if we expect a single parameter `file` to be an [`UploadedFile`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/uploaded_file.rb) instance, -use `requires :file, type: ::API::Validations::Types::WorkhorseFile`. - - Body upload requests have their upload available under the parameter `file`. - - Check that the request is coming from Workhorse with the `require_gitlab_workhorse!` from the -[API helpers](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/helpers.rb). - - Check the user permissions. - - The remaining code of the processing. This is where the code must be reading the parameter (for -our example, it would be `params[:file]`). - -WARNING: -**Do not** call `UploadedFile#from_params` directly! Do not build an [`UploadedFile`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/uploaded_file.rb) -object using `UploadedFile#from_params`! This method can be unsafe to use depending on the `params` -passed. Instead, use the [`UploadedFile`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/uploaded_file.rb) -object that [`multipart.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/middleware/multipart.rb) -builds automatically for you. - -### Document Object Storage buckets and CarrierWave integration - -When using Object Storage, GitLab expects each kind of upload to maintain its own bucket in the respective -Object Storage destination. Moreover, the integration with CarrierWave is not used all the time. -The [Object Storage Working Group](https://about.gitlab.com/company/team/structure/working-groups/object-storage/) -is investigating an approach that unifies Object Storage buckets into a single one and removes CarrierWave -so as to simplify implementation and administration of uploads. - -Therefore, document new uploads here by slotting them into the following tables: - -- [Feature bucket details](#feature-bucket-details) -- [CarrierWave integration](#carrierwave-integration) - -#### Feature bucket details - -| Feature | Upload technology | Uploader | Bucket structure | -|------------------------------------------|-------------------|-----------------------|-----------------------------------------------------------------------------------------------------------| -| Job artifacts | `direct upload` | `workhorse` | `/artifacts/<proj_id_hash>/<date>/<job_id>/<artifact_id>` | -| Pipeline artifacts | `carrierwave` | `sidekiq` | `/artifacts/<proj_id_hash>/pipelines/<pipeline_id>/artifacts/<artifact_id>` | -| Live job traces | `fog` | `sidekiq` | `/artifacts/tmp/builds/<job_id>/chunks/<chunk_index>.log` | -| Job traces archive | `carrierwave` | `sidekiq` | `/artifacts/<proj_id_hash>/<date>/<job_id>/<artifact_id>/job.log` | -| Autoscale runner caching | N/A | `gitlab-runner` | `/gitlab-com-[platform-]runners-cache/???` | -| Backups | N/A | `s3cmd`, `awscli`, or `gcs` | `/gitlab-backups/???` | -| Git LFS | `direct upload` | `workhorse` | `/lsf-objects/<lfs_obj_oid[0:2]>/<lfs_obj_oid[2:2]>` | -| Design management files | `disk buffering` | `rails controller` | `/lsf-objects/<lfs_obj_oid[0:2]>/<lfs_obj_oid[2:2]>` | -| Design management thumbnails | `carrierwave` | `sidekiq` | `/uploads/design_management/action/image_v432x230/<model_id>` | -| Generic file uploads | `direct upload` | `workhorse` | `/uploads/@hashed/[0:2]/[2:4]/<hash1>/<hash2>/file` | -| Generic file uploads - personal snippets | `direct upload` | `workhorse` | `/uploads/personal_snippet/<snippet_id>/<filename>` | -| Global appearance settings | `disk buffering` | `rails controller` | `/uploads/appearance/...` | -| Topics | `disk buffering` | `rails controller` | `/uploads/projects/topic/...` | -| Avatar images | `direct upload` | `workhorse` | `/uploads/[user,group,project]/avatar/<model_id>` | -| Import/export | `direct upload` | `workhorse` | `/uploads/import_export_upload/???` | -| GitLab Migration | `carrierwave` | `sidekiq` | `/uploads/bulk_imports/???` | -| MR diffs | `carrierwave` | `sidekiq` | `/external-diffs/merge_request_diffs/mr-<mr_id>/diff-<diff_id>` | -| Package manager archives | `direct upload` | `sidekiq` | `/packages/<proj_id_hash>/packages/<pkg_segment>/files/<pkg_file_id>` | -| Package manager archives | `direct upload` | `sidekiq` | `/packages/<container_id_hash>/debian_*_component_file/<component_file_id>` | -| Package manager archives | `direct upload` | `sidekiq` | `/packages/<container_id_hash>/debian_*_distribution/<distribution_file_id>` | -| Container image cache (?) | `direct upload` | `workhorse` | `/dependency-proxy/<group_id_hash>/dependency_proxy/<group_id>/files/<proxy_id>/<blob_id or manifest_id>` | -| Terraform state files | `carrierwave` | `rails controller` | `/terraform/<proj_id_hash>/<terraform_state_id>` | -| Pages content archives | `carrierwave` | `sidekiq` | `/gitlab-gprd-pages/<proj_id_hash>/pages_deployments/<deployment_id>/` | -| Secure Files | `carrierwave` | `sidekiq` | `/ci-secure-files/<proj_id_hash>/secure_files/<secure_file_id>/` | - -#### CarrierWave integration - -| File | Carrierwave usage | Categorized | -|---------------------------------------------------------|----------------------------------------------------------------------------------|---------------------| -| `app/models/project.rb` | `include Avatarable` | :white_check_mark: | -| `app/models/projects/topic.rb` | `include Avatarable` | :white_check_mark: | -| `app/models/group.rb` | `include Avatarable` | :white_check_mark: | -| `app/models/user.rb` | `include Avatarable` | :white_check_mark: | -| `app/models/terraform/state_version.rb` | `include FileStoreMounter` | :white_check_mark: | -| `app/models/ci/job_artifact.rb` | `include FileStoreMounter` | :white_check_mark: | -| `app/models/ci/pipeline_artifact.rb` | `include FileStoreMounter` | :white_check_mark: | -| `app/models/pages_deployment.rb` | `include FileStoreMounter` | :white_check_mark: | -| `app/models/lfs_object.rb` | `include FileStoreMounter` | :white_check_mark: | -| `app/models/dependency_proxy/blob.rb` | `include FileStoreMounter` | :white_check_mark: | -| `app/models/dependency_proxy/manifest.rb` | `include FileStoreMounter` | :white_check_mark: | -| `app/models/packages/composer/cache_file.rb` | `include FileStoreMounter` | :white_check_mark: | -| `app/models/packages/package_file.rb` | `include FileStoreMounter` | :white_check_mark: | -| `app/models/concerns/packages/debian/component_file.rb` | `include FileStoreMounter` | :white_check_mark: | -| `ee/app/models/issuable_metric_image.rb` | `include FileStoreMounter` | | -| `ee/app/models/vulnerabilities/remediation.rb` | `include FileStoreMounter` | | -| `ee/app/models/vulnerabilities/export.rb` | `include FileStoreMounter` | | -| `app/models/packages/debian/project_distribution.rb` | `include Packages::Debian::Distribution` | :white_check_mark: | -| `app/models/packages/debian/group_distribution.rb` | `include Packages::Debian::Distribution` | :white_check_mark: | -| `app/models/packages/debian/project_component_file.rb` | `include Packages::Debian::ComponentFile` | :white_check_mark: | -| `app/models/packages/debian/group_component_file.rb` | `include Packages::Debian::ComponentFile` | :white_check_mark: | -| `app/models/merge_request_diff.rb` | `mount_uploader :external_diff, ExternalDiffUploader` | :white_check_mark: | -| `app/models/note.rb` | `mount_uploader :attachment, AttachmentUploader` | :white_check_mark: | -| `app/models/appearance.rb` | `mount_uploader :logo, AttachmentUploader` | :white_check_mark: | -| `app/models/appearance.rb` | `mount_uploader :header_logo, AttachmentUploader` | :white_check_mark: | -| `app/models/appearance.rb` | `mount_uploader :favicon, FaviconUploader` | :white_check_mark: | -| `app/models/project.rb` | `mount_uploader :bfg_object_map, AttachmentUploader` | | -| `app/models/import_export_upload.rb` | `mount_uploader :import_file, ImportExportUploader` | :white_check_mark: | -| `app/models/import_export_upload.rb` | `mount_uploader :export_file, ImportExportUploader` | :white_check_mark: | -| `app/models/ci/deleted_object.rb` | `mount_uploader :file, DeletedObjectUploader` | | -| `app/models/design_management/action.rb` | `mount_uploader :image_v432x230, DesignManagement::DesignV432x230Uploader` | :white_check_mark: | -| `app/models/concerns/packages/debian/distribution.rb` | `mount_uploader :signed_file, Packages::Debian::DistributionReleaseFileUploader` | :white_check_mark: | -| `app/models/bulk_imports/export_upload.rb` | `mount_uploader :export_file, ExportUploader` | :white_check_mark: | -| `ee/app/models/user_permission_export_upload.rb` | `mount_uploader :file, AttachmentUploader` | | -| `app/models/ci/secure_file.rb` | `include FileStoreMounter` | | +<!-- This redirect file can be deleted after 2022-04-30. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/uploads/background.md b/doc/development/uploads/background.md new file mode 100644 index 00000000000..e68e4127b57 --- /dev/null +++ b/doc/development/uploads/background.md @@ -0,0 +1,154 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Uploads guide: Why GitLab uses custom upload logic + +This page is for developers trying to better understand the history behind GitLab uploads and the +technical challenges associated with uploads. + +## Problem description + +GitLab and [GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) use special rules for handling file uploads, +because in an ordinary Rails application file uploads can become expensive as files grow in size. +Rails often sacrifices performance to provide a better developer experience, including how it handles +`multipart/form-post` uploads. In any Rack server, Rails applications included, when such a request arrives at the application server, +several things happen: + +1. A [Rack middleware](https://github.com/rack/rack/blob/main/lib/rack/multipart.rb) intercepts the request and parses the request body. +1. The middleware writes each file in the multipart request to a temporary directory on disk. +1. A `params` hash is constructed with entries pointing to the respective files on disk. +1. A Rails controller acts on the file contents. + +While this is convenient for developers, it is costly for the Ruby server process to buffer large files on disk. +Because of Ruby's [global interpreter lock](https://en.wikipedia.org/wiki/Global_interpreter_lock), +only a single thread of execution of a given Ruby process can be on CPU. This means the amount of CPU +time spent doing this is not available to other worker threads serving user requests. +Buffering files to disk also means spending more time in I/O routines and mode switches, which are expensive operations. + +The following diagram shows how GitLab handled such a request prior to putting optimizations in place. + +```mermaid +graph TB + subgraph "load balancers" + LB(Proxy) + end + + subgraph "Shared storage" + nfs(NFS) + end + + subgraph "redis cluster" + r(persisted redis) + end + LB-- 1 -->Workhorse + + subgraph "web or API fleet" + Workhorse-- 2 -->rails + end + rails-- "3 (write files)" -->nfs + rails-- "4 (schedule a job)" -->r + + subgraph sidekiq + s(sidekiq) + end + s-- "5 (fetch a job)" -->r + s-- "6 (read files)" -->nfs +``` + +We went through two major iterations of our uploads architecture to improve on these problems: + +1. [Moving disk buffering to Workhorse.](#moving-disk-buffering-to-workhorse) +1. [Uploading to Object Storage from Workhorse.](#moving-to-object-storage-and-direct-uploads) + +### Moving disk buffering to Workhorse + +To address the performance issues resulting from buffering files in Ruby, we moved this logic to Workhorse instead, +our reverse proxy fronting the GitLab Rails application. +Workhorse is written in Go, and is much better at dealing with stream processing and I/O than Rails. + +There are two parts to this implementation: + +1. In Workhorse, a request handler detects `multipart/form-data` content in an incoming user request. + If such a request is detected, Workhorse hijacks the request body before forwarding it to Rails. + Workhorse writes all files to disk, rewrites the multipart form fields to point to the new locations, signs the + request, then forwards it to Rails. +1. In Rails, a [custom multipart Rack middleware](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/middleware/multipart.rb) + identifies any signed multipart requests coming from Workhorse and prepares the `params` hash Rails + would expect, now pointing to the files cached by Workhorse. This makes it a drop-in replacement for `Rack::Multipart`. + +The diagram below shows how GitLab handles such a request today: + +```mermaid +graph TB + subgraph "load balancers" + LB(HA Proxy) + end + + subgraph "Shared storage" + nfs(NFS) + end + + subgraph "redis cluster" + r(persisted redis) + end + LB-- 1 -->Workhorse + + subgraph "web or API fleet" + Workhorse-- "3 (without files)" -->rails + end + Workhorse -- "2 (write files)" -->nfs + rails-- "4 (schedule a job)" -->r + + subgraph sidekiq + s(sidekiq) + end + s-- "5 (fetch a job)" -->r + s-- "6 (read files)" -->nfs +``` + +While this "one-size-fits-all" solution greatly improves performance for multipart uploads without compromising +developer ergonomics, it severely limits GitLab [availability](#availability-challenges) +and [scalability](#scalability-challenges). + +#### Availability challenges + +Moving file buffering to Workhorse addresses the immediate performance problems stemming from Ruby not being good at +handling large file uploads. However, a remaining issue of this solution is its reliance on attached storage, +whether via ordinary hard drives or network attached storage like NFS. +NFS is a [single point of failure](https://en.wikipedia.org/wiki/Single_point_of_failure), and is unsuitable for +deploying GitLab in highly available, cloud native environments. + +#### Scalability challenges + +NFS is not a part of cloud native installations, such as those running in Kubernetes. +In Kubernetes, machine boundaries translate to pods, and without network-attached storage, disk-buffered uploads +must be written directly to the pod's file system. + +Using disk buffering presents us with a scalability challenge here. If Workhorse can only +write files to a pod's private file system, then these files are inaccessible outside of this particular pod. +With disk buffering, a Rails controller will accept a file upload and enqueue it for upload in a Sidekiq +background job. Therefore, Sidekiq requires access to these files. +However, in a cloud native environment all Sidekiq instances run on separate pods, so they are +not able to access files buffered to disk on a web server pod. + +Therefore, all features that involve Sidekiq uploading disk-buffered files severely limit the scalability of GitLab. + +## Moving to object storage and direct uploads + +To address these availability and scalability problems, +instead of buffering files to disk, we have added support for uploading files directly +from Workhorse to a given destination. While it remains possible to upload to local or network-attached storage +this way, you should use a highly available +[object store](https://en.wikipedia.org/wiki/Object_storage), +such as AWS S3, Google GCS, or Azure, for scalability reasons. + +With direct uploads, Workhorse does not buffer files to disk. Instead, it first authorizes the request with +the Rails application to find out where to upload it, then streams the file directly to its ultimate destination. + +To learn more about how disk buffering and direct uploads are implemented, see: + +- [How uploads work technically](implementation.md) +- [Adding new uploads](working_with_uploads.md) diff --git a/doc/development/uploads/implementation.md b/doc/development/uploads/implementation.md new file mode 100644 index 00000000000..13a875cd1af --- /dev/null +++ b/doc/development/uploads/implementation.md @@ -0,0 +1,190 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Uploads guide: How uploads work technically + +This page is for developers trying to better understand what kinds of uploads exist in GitLab and how they are implemented. + +## Kinds of uploads and how to choose between them + +We can identify three major use-cases for an upload: + +1. **storage:** if we are uploading for storing a file (like artifacts, packages, or discussion attachments). In this case [direct upload](#direct-upload) is the proper level as it's the less resource-intensive operation. Additional information can be found on [File Storage in GitLab](../file_storage.md). +1. **in-controller/synchronous processing:** if we allow processing **small files** synchronously, using [disk buffered upload](#disk-buffered-upload) may speed up development. +1. **Sidekiq/asynchronous processing:** Asynchronous processing must implement [direct upload](#direct-upload), the reason being that it's the only way to support Cloud Native deployments without a shared NFS. + +Selecting the proper acceleration is a tradeoff between speed of development and operational costs. + +For more details about currently broken feature see [epic &1802](https://gitlab.com/groups/gitlab-org/-/epics/1802). + +### Handling repository uploads + +Some features involves Git repository uploads without using a regular Git client. +Some examples are uploading a repository file from the web interface and [design management](../../user/project/issues/design_management.md). + +Those uploads requires the rails controller to act as a Git client in lieu of the user. +Those operation falls into _in-controller/synchronous processing_ category, but we have no warranties on the file size. + +In case of a LFS upload, the file pointer is committed synchronously, but file upload to object storage is performed asynchronously with Sidekiq. + +## Upload encodings + +By upload encoding we mean how the file is included within the incoming request. + +We have three kinds of file encoding in our uploads: + +1. <i class="fa fa-check-circle"></i> **multipart**: `multipart/form-data` is the most common, a file is encoded as a part of a multipart encoded request. +1. <i class="fa fa-check-circle"></i> **body**: some APIs uploads files as the whole request body. +1. <i class="fa fa-times-circle"></i> **JSON**: some JSON APIs upload files as base64-encoded strings. This requires a change to GitLab Workhorse, + which is tracked [in this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/325068). + +## Uploading technologies + +By uploading technologies we mean how all the involved services interact with each other. + +GitLab supports 3 kinds of uploading technologies, here follows a brief description with a sequence diagram for each one. Diagrams are not meant to be exhaustive. + +### Rack Multipart upload + +This is the default kind of upload, and it's the most expensive in terms of resources. + +In this case, Workhorse is unaware of files being uploaded and acts as a regular proxy. + +When a multipart request reaches the rails application, `Rack::Multipart` leaves behind temporary files in `/tmp` and uses valuable Ruby process time to copy files around. + +```mermaid +sequenceDiagram + participant c as Client + participant w as Workhorse + participant r as Rails + + activate c + c ->>+w: POST /some/url/upload + w->>+r: POST /some/url/upload + + r->>r: save the incoming file on /tmp + r->>r: read the file for processing + + r-->>-c: request result + deactivate c + deactivate w +``` + +### Disk buffered upload + +This kind of upload avoids wasting resources caused by handling upload writes to `/tmp` in rails. + +This optimization is not active by default on REST API requests. + +When enabled, Workhorse looks for files in multipart MIME requests, uploading +any it finds to a temporary file on shared storage. The MIME data in the request +is replaced with the path to the corresponding file before it is forwarded to +Rails. + +To prevent abuse of this feature, Workhorse signs the modified request with a +special header, stating which entries it modified. Rails ignores any +unsigned path entries. + +```mermaid +sequenceDiagram + participant c as Client + participant w as Workhorse + participant r as Rails + participant s as NFS + + activate c + c ->>+w: POST /some/url/upload + + w->>+s: save the incoming file on a temporary location + s-->>-w: request result + + w->>+r: POST /some/url/upload + Note over w,r: file was replaced with its location<br>and other metadata + + opt requires async processing + r->>+redis: schedule a job + redis-->>-r: job is scheduled + end + + r-->>-c: request result + deactivate c + w->>-w: cleanup + + opt requires async processing + activate sidekiq + sidekiq->>+redis: fetch a job + redis-->>-sidekiq: job + + sidekiq->>+s: read file + s-->>-sidekiq: file + + sidekiq->>sidekiq: process file + + deactivate sidekiq + end +``` + +### Direct upload + +This is the more advanced acceleration technique we have in place. + +Workhorse asks Rails for temporary pre-signed object storage URLs and directly uploads to object storage. + +In this setup, an extra Rails route must be implemented in order to handle authorization. Examples of this can be found in: + +- [`Projects::LfsStorageController`](https://gitlab.com/gitlab-org/gitlab/-/blob/cc723071ad337573e0360a879cbf99bc4fb7adb9/app/controllers/projects/lfs_storage_controller.rb) + and [its routes](https://gitlab.com/gitlab-org/gitlab/-/blob/cc723071ad337573e0360a879cbf99bc4fb7adb9/config/routes/git_http.rb#L31-32). +- [API endpoints for uploading packages](../packages.md#file-uploads). + +Direct upload falls back to _disk buffered upload_ when `direct_upload` is disabled inside the [object storage setting](../../administration/uploads.md#object-storage-settings). +The answer to the `/authorize` call contains only a file system path. + +```mermaid +sequenceDiagram + participant c as Client + participant w as Workhorse + participant r as Rails + participant os as Object Storage + + activate c + c ->>+w: POST /some/url/upload + + w ->>+r: POST /some/url/upload/authorize + Note over w,r: this request has an empty body + r-->>-w: presigned OS URL + + w->>+os: PUT file + Note over w,os: file is stored on a temporary location. Rails select the destination + os-->>-w: request result + + w->>+r: POST /some/url/upload + Note over w,r: file was replaced with its location<br>and other metadata + + r->>+os: move object to final destination + os-->>-r: request result + + opt requires async processing + r->>+redis: schedule a job + redis-->>-r: job is scheduled + end + + r-->>-c: request result + deactivate c + w->>-w: cleanup + + opt requires async processing + activate sidekiq + sidekiq->>+redis: fetch a job + redis-->>-sidekiq: job + + sidekiq->>+os: get object + os-->>-sidekiq: file + + sidekiq->>sidekiq: process file + + deactivate sidekiq + end +``` diff --git a/doc/development/uploads/index.md b/doc/development/uploads/index.md new file mode 100644 index 00000000000..c486f2d3689 --- /dev/null +++ b/doc/development/uploads/index.md @@ -0,0 +1,14 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Uploads development guide + +Uploads are an integral part of many GitLab features. To understand how GitLab handles uploads, refer to +the following pages: + +- [Why GitLab uses custom upload logic.](background.md) +- [How uploads work technically.](implementation.md) +- [How to add new uploads.](working_with_uploads.md) diff --git a/doc/development/uploads/working_with_uploads.md b/doc/development/uploads/working_with_uploads.md new file mode 100644 index 00000000000..99c04888804 --- /dev/null +++ b/doc/development/uploads/working_with_uploads.md @@ -0,0 +1,163 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Uploads guide: Adding new uploads + +In this section, we describe how to add a new upload route [accelerated](implementation.md#uploading-technologies) by Workhorse for [body and multipart](implementation.md#upload-encodings) encoded uploads. + +Upload routes belong to one of these categories: + +1. Rails controllers: uploads handled by Rails controllers. +1. Grape API: uploads handled by a Grape API endpoint. +1. GraphQL API: uploads handled by a GraphQL resolve function. + +WARNING: +GraphQL uploads do not support [direct upload](implementation.md#direct-upload) yet. Depending on the use case, the feature may not work on installations without NFS (like GitLab.com or Kubernetes installations). Uploading to object storage inside the GraphQL resolve function may result in timeout errors. For more details please follow [issue #280819](https://gitlab.com/gitlab-org/gitlab/-/issues/280819). + +## Update Workhorse for the new route + +For both the Rails controller and Grape API uploads, Workhorse has to be updated in order to get the +support for the new upload route. + +1. Open a new issue in the [Workhorse tracker](https://gitlab.com/gitlab-org/gitlab-workhorse/-/issues/new) describing precisely the new upload route: + - The route's URL. + - The [upload encoding](implementation.md#upload-encodings). + - If possible, provide a dump of the upload request. +1. Implement and get the MR merged for this issue above. +1. Ask the Maintainers of [Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) to create a new release. You can do that in the MR + directly during the maintainer review or ask for it in the `#workhorse` Slack channel. +1. Bump the [Workhorse version file](https://gitlab.com/gitlab-org/gitlab/-/blob/master/GITLAB_WORKHORSE_VERSION) + to the version you have from the previous points, or bump it in the same merge request that contains + the Rails changes (see [Implementing the new route with a Rails controller](#implementing-the-new-route-with-a-rails-controller) or [Implementing the new route with a Grape API endpoint](#implementing-the-new-route-with-a-grape-api-endpoint) below). + +## Implementing the new route with a Rails controller + +For a Rails controller upload, we usually have a [multipart](implementation.md#upload-encodings) upload and there are a +few things to do: + +1. The upload is available under the parameter name you're using. For example, it could be an `artifact` + or a nested parameter such as `user[avatar]`. Let's say that we have the upload under the + `file` parameter, reading `params[:file]` should get you an [`UploadedFile`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/uploaded_file.rb) instance. +1. Generally speaking, it's a good idea to check if the instance is from the [`UploadedFile`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/uploaded_file.rb) class. For example, see how we checked +[that the parameter is indeed an `UploadedFile`](https://gitlab.com/gitlab-org/gitlab/-/commit/ea30fe8a71bf16ba07f1050ab4820607b5658719#51c0cc7a17b7f12c32bc41cfab3649ff2739b0eb_79_77). + +WARNING: +**Do not** call `UploadedFile#from_params` directly! Do not build an [`UploadedFile`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/uploaded_file.rb) +instance using `UploadedFile#from_params`! This method can be unsafe to use depending on the `params` +passed. Instead, use the [`UploadedFile`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/uploaded_file.rb) +instance that [`multipart.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/middleware/multipart.rb) +builds automatically for you. + +## Implementing the new route with a Grape API endpoint + +For a Grape API upload, we can have [body or a multipart](implementation.md#upload-encodings) upload. Things are slightly more complicated: two endpoints are needed. One for the +Workhorse pre-upload authorization and one for accepting the upload metadata from Workhorse: + +1. Implement an endpoint with the URL + `/authorize` suffix that will: + - Check that the request is coming from Workhorse with the `require_gitlab_workhorse!` from the [API helpers](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/helpers.rb). + - Check user permissions. + - Set the status to `200` with `status 200`. + - Set the content type with `content_type Gitlab::Workhorse::INTERNAL_API_CONTENT_TYPE`. + - Use your dedicated `Uploader` class (let's say that it's `FileUploader`) to build the response with `FileUploader.workhorse_authorize(params)`. +1. Implement the endpoint for the upload request that will: + - Require all the `UploadedFile` objects as parameters. + - For example, if we expect a single parameter `file` to be an [`UploadedFile`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/uploaded_file.rb) instance, +use `requires :file, type: ::API::Validations::Types::WorkhorseFile`. + - Body upload requests have their upload available under the parameter `file`. + - Check that the request is coming from Workhorse with the `require_gitlab_workhorse!` from the +[API helpers](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/helpers.rb). + - Check the user permissions. + - The remaining code of the processing. This is where the code must be reading the parameter (for +our example, it would be `params[:file]`). + +WARNING: +**Do not** call `UploadedFile#from_params` directly! Do not build an [`UploadedFile`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/uploaded_file.rb) +object using `UploadedFile#from_params`! This method can be unsafe to use depending on the `params` +passed. Instead, use the [`UploadedFile`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/uploaded_file.rb) +object that [`multipart.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/middleware/multipart.rb) +builds automatically for you. + +## Document Object Storage buckets and CarrierWave integration + +When using Object Storage, GitLab expects each kind of upload to maintain its own bucket in the respective +Object Storage destination. Moreover, the integration with CarrierWave is not used all the time. +The [Object Storage Working Group](https://about.gitlab.com/company/team/structure/working-groups/object-storage/) +is investigating an approach that unifies Object Storage buckets into a single one and removes CarrierWave +so as to simplify implementation and administration of uploads. + +Therefore, document new uploads here by slotting them into the following tables: + +- [Feature bucket details](#feature-bucket-details) +- [CarrierWave integration](#carrierwave-integration) + +### Feature bucket details + +| Feature | Upload technology | Uploader | Bucket structure | +|------------------------------------------|-------------------|-----------------------|-----------------------------------------------------------------------------------------------------------| +| Job artifacts | `direct upload` | `workhorse` | `/artifacts/<proj_id_hash>/<date>/<job_id>/<artifact_id>` | +| Pipeline artifacts | `carrierwave` | `sidekiq` | `/artifacts/<proj_id_hash>/pipelines/<pipeline_id>/artifacts/<artifact_id>` | +| Live job traces | `fog` | `sidekiq` | `/artifacts/tmp/builds/<job_id>/chunks/<chunk_index>.log` | +| Job traces archive | `carrierwave` | `sidekiq` | `/artifacts/<proj_id_hash>/<date>/<job_id>/<artifact_id>/job.log` | +| Autoscale runner caching | N/A | `gitlab-runner` | `/gitlab-com-[platform-]runners-cache/???` | +| Backups | N/A | `s3cmd`, `awscli`, or `gcs` | `/gitlab-backups/???` | +| Git LFS | `direct upload` | `workhorse` | `/lsf-objects/<lfs_obj_oid[0:2]>/<lfs_obj_oid[2:2]>` | +| Design management files | `disk buffering` | `rails controller` | `/lsf-objects/<lfs_obj_oid[0:2]>/<lfs_obj_oid[2:2]>` | +| Design management thumbnails | `carrierwave` | `sidekiq` | `/uploads/design_management/action/image_v432x230/<model_id>` | +| Generic file uploads | `direct upload` | `workhorse` | `/uploads/@hashed/[0:2]/[2:4]/<hash1>/<hash2>/file` | +| Generic file uploads - personal snippets | `direct upload` | `workhorse` | `/uploads/personal_snippet/<snippet_id>/<filename>` | +| Global appearance settings | `disk buffering` | `rails controller` | `/uploads/appearance/...` | +| Topics | `disk buffering` | `rails controller` | `/uploads/projects/topic/...` | +| Avatar images | `direct upload` | `workhorse` | `/uploads/[user,group,project]/avatar/<model_id>` | +| Import/export | `direct upload` | `workhorse` | `/uploads/import_export_upload/???` | +| GitLab Migration | `carrierwave` | `sidekiq` | `/uploads/bulk_imports/???` | +| MR diffs | `carrierwave` | `sidekiq` | `/external-diffs/merge_request_diffs/mr-<mr_id>/diff-<diff_id>` | +| Package manager archives | `direct upload` | `sidekiq` | `/packages/<proj_id_hash>/packages/<pkg_segment>/files/<pkg_file_id>` | +| Package manager archives | `direct upload` | `sidekiq` | `/packages/<container_id_hash>/debian_*_component_file/<component_file_id>` | +| Package manager archives | `direct upload` | `sidekiq` | `/packages/<container_id_hash>/debian_*_distribution/<distribution_file_id>` | +| Container image cache (?) | `direct upload` | `workhorse` | `/dependency-proxy/<group_id_hash>/dependency_proxy/<group_id>/files/<proxy_id>/<blob_id or manifest_id>` | +| Terraform state files | `carrierwave` | `rails controller` | `/terraform/<proj_id_hash>/<terraform_state_id>` | +| Pages content archives | `carrierwave` | `sidekiq` | `/gitlab-gprd-pages/<proj_id_hash>/pages_deployments/<deployment_id>/` | +| Secure Files | `carrierwave` | `sidekiq` | `/ci-secure-files/<proj_id_hash>/secure_files/<secure_file_id>/` | + +### CarrierWave integration + +| File | Carrierwave usage | Categorized | +|---------------------------------------------------------|----------------------------------------------------------------------------------|---------------------| +| `app/models/project.rb` | `include Avatarable` | :white_check_mark: | +| `app/models/projects/topic.rb` | `include Avatarable` | :white_check_mark: | +| `app/models/group.rb` | `include Avatarable` | :white_check_mark: | +| `app/models/user.rb` | `include Avatarable` | :white_check_mark: | +| `app/models/terraform/state_version.rb` | `include FileStoreMounter` | :white_check_mark: | +| `app/models/ci/job_artifact.rb` | `include FileStoreMounter` | :white_check_mark: | +| `app/models/ci/pipeline_artifact.rb` | `include FileStoreMounter` | :white_check_mark: | +| `app/models/pages_deployment.rb` | `include FileStoreMounter` | :white_check_mark: | +| `app/models/lfs_object.rb` | `include FileStoreMounter` | :white_check_mark: | +| `app/models/dependency_proxy/blob.rb` | `include FileStoreMounter` | :white_check_mark: | +| `app/models/dependency_proxy/manifest.rb` | `include FileStoreMounter` | :white_check_mark: | +| `app/models/packages/composer/cache_file.rb` | `include FileStoreMounter` | :white_check_mark: | +| `app/models/packages/package_file.rb` | `include FileStoreMounter` | :white_check_mark: | +| `app/models/concerns/packages/debian/component_file.rb` | `include FileStoreMounter` | :white_check_mark: | +| `ee/app/models/issuable_metric_image.rb` | `include FileStoreMounter` | | +| `ee/app/models/vulnerabilities/remediation.rb` | `include FileStoreMounter` | | +| `ee/app/models/vulnerabilities/export.rb` | `include FileStoreMounter` | | +| `app/models/packages/debian/project_distribution.rb` | `include Packages::Debian::Distribution` | :white_check_mark: | +| `app/models/packages/debian/group_distribution.rb` | `include Packages::Debian::Distribution` | :white_check_mark: | +| `app/models/packages/debian/project_component_file.rb` | `include Packages::Debian::ComponentFile` | :white_check_mark: | +| `app/models/packages/debian/group_component_file.rb` | `include Packages::Debian::ComponentFile` | :white_check_mark: | +| `app/models/merge_request_diff.rb` | `mount_uploader :external_diff, ExternalDiffUploader` | :white_check_mark: | +| `app/models/note.rb` | `mount_uploader :attachment, AttachmentUploader` | :white_check_mark: | +| `app/models/appearance.rb` | `mount_uploader :logo, AttachmentUploader` | :white_check_mark: | +| `app/models/appearance.rb` | `mount_uploader :header_logo, AttachmentUploader` | :white_check_mark: | +| `app/models/appearance.rb` | `mount_uploader :favicon, FaviconUploader` | :white_check_mark: | +| `app/models/project.rb` | `mount_uploader :bfg_object_map, AttachmentUploader` | | +| `app/models/import_export_upload.rb` | `mount_uploader :import_file, ImportExportUploader` | :white_check_mark: | +| `app/models/import_export_upload.rb` | `mount_uploader :export_file, ImportExportUploader` | :white_check_mark: | +| `app/models/ci/deleted_object.rb` | `mount_uploader :file, DeletedObjectUploader` | | +| `app/models/design_management/action.rb` | `mount_uploader :image_v432x230, DesignManagement::DesignV432x230Uploader` | :white_check_mark: | +| `app/models/concerns/packages/debian/distribution.rb` | `mount_uploader :signed_file, Packages::Debian::DistributionReleaseFileUploader` | :white_check_mark: | +| `app/models/bulk_imports/export_upload.rb` | `mount_uploader :export_file, ExportUploader` | :white_check_mark: | +| `ee/app/models/user_permission_export_upload.rb` | `mount_uploader :file, AttachmentUploader` | | +| `app/models/ci/secure_file.rb` | `include FileStoreMounter` | | diff --git a/doc/development/value_stream_analytics/img/object_hierarchy_example_V14_10.png b/doc/development/value_stream_analytics/img/object_hierarchy_example_V14_10.png Binary files differindex 0e3c760fa3c..9c4d4ae7718 100644 --- a/doc/development/value_stream_analytics/img/object_hierarchy_example_V14_10.png +++ b/doc/development/value_stream_analytics/img/object_hierarchy_example_V14_10.png diff --git a/doc/development/workspaces/index.md b/doc/development/workspace/index.md index b36c79e84d7..b01a7826b3d 100644 --- a/doc/development/workspaces/index.md +++ b/doc/development/workspace/index.md @@ -4,19 +4,19 @@ type: index, dev stage: none group: Development info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" -description: "Development Guidelines: learn about workspaces when developing GitLab." +description: "Development Guidelines: learn about workspace when developing GitLab." --- -# Workspaces +# Workspace -The [Workspaces initiative](../../user/workspace/index.md) focuses on reaching feature parity between +The [Workspace initiative](../../user/workspace/index.md) focuses on reaching feature parity between SaaS and self-managed installations. ## Consolidate groups and projects - [Architecture blueprint](../../architecture/blueprints/consolidating_groups_and_projects/index.md) -One facet of the workspaces initiative is to consolidate groups and projects, +One facet of the workspace initiative is to consolidate groups and projects, addressing the feature disparity between them. Some features, such as epics, are only available at the group level. Some features, such as issues, are only available at the project level. Other features, such as milestones, are available to both groups |
