diff options
Diffstat (limited to 'doc')
32 files changed, 722 insertions, 236 deletions
diff --git a/doc/.vale/gitlab/Admin.yml b/doc/.vale/gitlab/Admin.yml index a2477026560..987dbfdbd09 100644 --- a/doc/.vale/gitlab/Admin.yml +++ b/doc/.vale/gitlab/Admin.yml @@ -8,6 +8,6 @@ extends: substitution message: 'Verify this use of the word "admin". Can it be updated to "administration", "administrator", "administer", or "Admin Area"?' link: https://docs.gitlab.com/ee/development/documentation/styleguide/index.html level: suggestion -ignorecase: true +ignorecase: false swap: - 'admin ?\w*': '(?:Admin (Area|Mode)|[Aa]dminist(ration|rator|rators|er|rative))' + '[Aa]dmin ?\w*': '(?:Admin( Area| Mode)?|[Aa]dminist(ration|rator|rators|er|rative))' diff --git a/doc/administration/database_load_balancing.md b/doc/administration/database_load_balancing.md index 9c1ed9b3477..e9f989c96ea 100644 --- a/doc/administration/database_load_balancing.md +++ b/doc/administration/database_load_balancing.md @@ -113,7 +113,7 @@ Some background jobs can use database replicas to read application state. This allows to offload the primary database. Load balancing is disabled by default in Sidekiq. When enabled, we can define -[the data consistency](../development/sidekiq_style_guide.md#job-data-consistency) +[the data consistency](../development/sidekiq_style_guide.md#job-data-consistency-strategies) requirements for a specific job. To enable it, define the `ENABLE_LOAD_BALANCING_FOR_SIDEKIQ` variable to the environment, as shown below. diff --git a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md index b43825092b7..bb0287df596 100644 --- a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md +++ b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md @@ -147,7 +147,7 @@ and they will assist you with any issues you are having. You can also use `gitlab-rake`, instead of `/usr/local/bin/gitlab-rake`. -- Troubleshooting **Infrastructure > Kubernetes** integration: +- Troubleshooting **Infrastructure > Kubernetes clusters** integration: - Check the output of `kubectl get events -w --all-namespaces`. - Check the logs of pods within `gitlab-managed-apps` namespace. diff --git a/doc/api/graphql/reference/index.md b/doc/api/graphql/reference/index.md index 0069d109e1e..e84769fa568 100644 --- a/doc/api/graphql/reference/index.md +++ b/doc/api/graphql/reference/index.md @@ -14459,9 +14459,9 @@ List limit metric setting. | Value | Description | | ----- | ----------- | -| <a id="listlimitmetricall_metrics"></a>`all_metrics` | | -| <a id="listlimitmetricissue_count"></a>`issue_count` | | -| <a id="listlimitmetricissue_weights"></a>`issue_weights` | | +| <a id="listlimitmetricall_metrics"></a>`all_metrics` | Limit list by number and total weight of issues. | +| <a id="listlimitmetricissue_count"></a>`issue_count` | Limit list by number of issues. | +| <a id="listlimitmetricissue_weights"></a>`issue_weights` | Limit list by total weight of issues. | ### `MeasurementIdentifier` diff --git a/doc/api/group_milestones.md b/doc/api/group_milestones.md index cf542a9da0b..40d51f90b5a 100644 --- a/doc/api/group_milestones.md +++ b/doc/api/group_milestones.md @@ -144,6 +144,11 @@ Parameters: | `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | | `milestone_id` | integer | yes | The ID of a group milestone | +Currently, this API endpoint doesn't return issues from any subgroups. +If you want to get all the milestones' issues, you can instead use the +[List issues API](issues.md#list-issues) and filter for a +particular milestone (for example, `GET /issues?milestone=1.0.0&state=opened`). + ## Get all merge requests assigned to a single milestone Gets all merge requests assigned to a single group milestone. diff --git a/doc/ci/runners/README.md b/doc/ci/runners/README.md index b493da993ca..45113c480c7 100644 --- a/doc/ci/runners/README.md +++ b/doc/ci/runners/README.md @@ -21,7 +21,7 @@ which depends on your [subscription plan](../../subscriptions/gitlab_com/index.m Linux shared runners on GitLab.com run in autoscale mode and are powered by Google Cloud Platform. -Autoscaling means reduced queue times to spin up CI/CD jobs, and isolated VMs for each project, thus maximizing security. These shared runners are available for users and customers on GitLab.com. +Autoscaling means reduced queue times to spin up CI/CD jobs, and isolated VMs for each job, thus maximizing security. These shared runners are available for users and customers on GitLab.com. GitLab offers Ultimate tier capabilities and included CI/CD minutes per group per month for our [Open Source](https://about.gitlab.com/solutions/open-source/join/), [Education](https://about.gitlab.com/solutions/education/), and [Startups](https://about.gitlab.com/solutions/startups/) programs. For private projects, GitLab offers various [plans](https://about.gitlab.com/pricing/), starting with a Free tier. diff --git a/doc/development/database/database_reviewer_guidelines.md b/doc/development/database/database_reviewer_guidelines.md index de131ddffbc..16734dada13 100644 --- a/doc/development/database/database_reviewer_guidelines.md +++ b/doc/development/database/database_reviewer_guidelines.md @@ -52,7 +52,7 @@ that require a more in-depth discussion between the database reviewers and maint - [Database Office Hours Agenda](https://docs.google.com/document/d/1wgfmVL30F8SdMg-9yY6Y8djPSxWNvKmhR5XmsvYX1EI/edit). - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [YouTube playlist with past recordings](https://www.youtube.com/playlist?list=PL05JrBw4t0Kp-kqXeiF7fF7cFYaKtdqXM). -You should also join the [#database-labs](../understanding_explain_plans.md#database-lab) +You should also join the [#database-lab](../understanding_explain_plans.md#database-lab-engine) Slack channel and get familiar with how to use Joe, the Slackbot that provides developers with their own clone of the production database. diff --git a/doc/development/database/keyset_pagination.md b/doc/development/database/keyset_pagination.md new file mode 100644 index 00000000000..e30c3cc8832 --- /dev/null +++ b/doc/development/database/keyset_pagination.md @@ -0,0 +1,251 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Keyset pagination + +The keyset pagination library can be used in HAML-based views and the REST API within the GitLab project. + +You can read about keyset pagination and how it compares to the offset based pagination on our [pagination guidelines](pagination_guidelines.md) page. + +## API overview + +### Synopsis + +Keyset pagination with `ActiveRecord` in Rails controllers: + +```ruby +cursor = params[:cursor] # this is nil when the first page is requested +paginator = Project.order(:created_at).keyset_paginate(cursor: cursor, per_page: 20) + +paginator.each do |project| + puts project.name # prints maximum 20 projects +end +``` + +### Usage + +This library adds a single method to ActiveRecord relations: [`#keyset_paginate`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/initializers/active_record_keyset_pagination.rb). + +This is similar in spirit (but not in implementation) to Kaminari's `paginate` method. + +Keyset pagination works without any configuration for simple ActiveRecord queries: + +- Order by one column. +- Order by two columns, where the last column is the primary key. + +The library can detect nullable and non-distinct columns and based on these, it will add extra ordering using the primary key. This is necessary because keyset pagination expects distinct order by values: + +```ruby +Project.order(:created_at).keyset_paginate.records # ORDER BY created_at, id + +Project.order(:name).keyset_paginate.records # ORDER BY name, id + +Project.order(:created_at, id: :desc).keyset_paginate.records # ORDER BY created_at, id + +Project.order(created_at: :asc, id: :desc).keyset_paginate.records # ORDER BY created_at, id DESC +``` + +The `keyset_paginate` method returns [a special paginator object](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/pagination/keyset/paginator.rb) which contains the loaded records and additional information for requesting various pages. + +The method accepts the following keyword arguments: + +- `cursor` - Encoded order by column values for requesting the next page (can be `nil`). +- `per_page` - Number of records to load per page (default 20). +- `keyset_order_options` - Extra options for building the keyset paginated database query, see an example for `UNION` queries in the performance section (optional). + +The paginator object has the following methods: + +- `records` - Returns the records for the current page. +- `has_next_page?` - Tells whether there is a next page. +- `has_previous_page?` - Tells whether there is a previous page. +- `cursor_for_next_page` - Encoded values as `String` for requesting the next page (can be `nil`). +- `cursor_for_previous_page` - Encoded values as `String` for requesting the previous page (can be `nil`). +- `cursor_for_first_page` - Encoded values as `String` for requesting the first page. +- `cursor_for_last_page` - Encoded values as `String` for requesting the last page. +- The paginator objects includes the `Enumerable` module and delegates the enumerable functionality to the `records` method/array. + +Example for getting the first and the second page: + +```ruby +paginator = Project.order(:name).keyset_paginate + +paginator.to_a # same as .records + +cursor = paginator.cursor_for_next_page # encoded column attributes for the next page + +paginator = Project.order(:name).keyset_paginate(cursor: cursor).records # loading the next page +``` + +Since keyset pagination does not support page numbers, we are restricted to go to the following pages: + +- Next page +- Previous page +- Last page +- First page + +#### Usage in Rails with HAML views + +Consider the following controller action, where we list the projects ordered by name: + +```ruby +def index + @projects = Project.order(:name).keyset_paginate(cursor: params[:cursor]) +end +``` + +In the HAML file, we can render the records: + +```ruby +- if @projects.any? + - @projects.each do |project| + .project-container + = project.name + + = keyset_paginate @projects +``` + +## Performance + +The performance of the keyset pagination depends on the database index configuration and the number of columns we use in the `ORDER BY` clause. + +In case we order by the primary key (`id`), then the generated queries will be efficient since the primary key is covered by a database index. + +When two or more columns are used in the `ORDER BY` clause, it's advised to check the generated database query and make sure that the correct index configuration is used. More information can be found on the [pagination guideline page](pagination_guidelines.md#index-coverage). + +NOTE: +While the query performance of the first page might look good, the second page (where the cursor attributes are used in the query) might yield poor performance. It's advised to always verify the performance of both queries: first page and second page. + +Example database query with tie-breaker (`id`) column: + +```sql +SELECT "issues".* +FROM "issues" +WHERE (("issues"."id" > 99 + AND "issues"."created_at" = '2021-02-16 11:26:17.408466') + OR ("issues"."created_at" > '2021-02-16 11:26:17.408466') + OR ("issues"."created_at" IS NULL)) +ORDER BY "issues"."created_at" DESC NULLS LAST, "issues"."id" DESC +LIMIT 20 +``` + +`OR` queries are difficult to optimize in PostgreSQL, we generally advise using [`UNION` queries](../sql.md#use-unions) instead. The keyset pagination library can generate efficient `UNION` when multiple columns are present in the `ORDER BY` clause. This is triggered when we specify the `use_union_optimization: true` option in the options passed to `Relation#keyset_paginate`. + +Example: + +```ruby +# Triggers a simple query for the first page. +paginator1 = Project.order(:created_at, id: :desc).keyset_paginate(per_page: 2, keyset_order_options: { use_union_optimization: true }) + +cursor = paginator1.cursor_for_next_page + +# Triggers UNION query for the second page +paginator2 = Project.order(:created_at, id: :desc).keyset_paginate(per_page: 2, cursor: cursor, keyset_order_options: { use_union_optimization: true }) + +puts paginator2.records.to_a # UNION query +``` + +## Complex order configuration + +Common `ORDER BY` configurations will be handled by the `keyset_paginate` method automatically so no manual configuration is needed. There are a few edge cases where order object configuration is necessary: + +- `NULLS LAST` ordering. +- Function-based ordering. +- Ordering with a custom tie-breaker column, like `iid`. + +These order objects can be defined in the model classes as normal ActiveRecord scopes, there is no special behavior that prevents using these scopes elsewhere (kaminari, background jobs). + +### `NULLS LAST` ordering + +Consider the following scope: + +```ruby +scope = Issue.where(project_id: 10).order(Gitlab::Database.nulls_last_order('relative_position', 'DESC')) +# SELECT "issues".* FROM "issues" WHERE "issues"."project_id" = 10 ORDER BY relative_position DESC NULLS LAST + +scope.keyset_paginate # raises: Gitlab::Pagination::Keyset::Paginator::UnsupportedScopeOrder: The order on the scope does not support keyset pagination +``` + +The `keyset_paginate` method raises an error because the order value on the query is a custom SQL string and not an [`Arel`](https://www.rubydoc.info/gems/arel) AST node. The keyset library cannot automatically infer configuration values from these kinds of queries. + +To make keyset pagination work, we need to configure custom order objects, to do so, we need to collect information about the order columns: + +- `relative_position` can have duplicated values since no unique index is present. +- `relative_position` can have null values because we don't have a not null constraint on the column. For this, we need to determine where will we see NULL values, at the beginning of the resultset or the end (`NULLS LAST`). +- Keyset pagination requires distinct order columns, so we'll need to add the primary key (`id`) to make the order distinct. +- Jumping to the last page and paginating backwards actually reverses the `ORDER BY` clause. For this, we'll need to provide the reversed `ORDER BY` clause. + +Example: + +```ruby +order = Gitlab::Pagination::Keyset::Order.build([ + # The attributes are documented in the `lib/gitlab/pagination/keyset/column_order_definition.rb` file + Gitlab::Pagination::Keyset::ColumnOrderDefinition.new( + attribute_name: 'relative_position', + column_expression: Issue.arel_table[:relative_position], + order_expression: Gitlab::Database.nulls_last_order('relative_position', 'DESC'), + reversed_order_expression: Gitlab::Database.nulls_first_order('relative_position', 'ASC'), + nullable: :nulls_last, + order_direction: :desc, + distinct: false + ), + Gitlab::Pagination::Keyset::ColumnOrderDefinition.new( + attribute_name: 'id', + order_expression: Issue.arel_table[:id].asc, + nullable: :not_nullable, + distinct: true + ) +]) + +scope = Issue.where(project_id: 10).order(order) # or reorder() + +scope.keyset_paginate.records # works +``` + +### Function-based ordering + +In the following example, we multiply the `id` by 10 and ordering by that value. Since the `id` column is unique, we need to define only one column: + +```ruby +order = Gitlab::Pagination::Keyset::Order.build([ + Gitlab::Pagination::Keyset::ColumnOrderDefinition.new( + attribute_name: 'id_times_ten', + order_expression: Arel.sql('id * 10').asc, + nullable: :not_nullable, + order_direction: :asc, + distinct: true, + add_to_projections: true + ) +]) + +paginator = Issue.where(project_id: 10).order(order).keyset_paginate(per_page: 5) +puts paginator.records.map(&:id_times_ten) + +cursor = paginator.cursor_for_next_page + +paginator = Issue.where(project_id: 10).order(order).keyset_paginate(cursor: cursor, per_page: 5) +puts paginator.records.map(&:id_times_ten) +``` + +The `add_to_projections` flag tells the paginator to expose the column expression in the `SELECT` clause. This is necessary because the keyset pagination needs to somehow extract the last value from the records to request the next page. + +### `iid` based ordering + +When ordering issues, the database ensures that we'll have distinct `iid` values within a project. Ordering by one column is enough to make the pagination work if the `project_id` filter is present: + +```ruby +order = Gitlab::Pagination::Keyset::Order.build([ + Gitlab::Pagination::Keyset::ColumnOrderDefinition.new( + attribute_name: 'iid', + order_expression: Issue.arel_table[:iid].asc, + nullable: :not_nullable, + distinct: true + ) +]) + +scope = Issue.where(project_id: 10).order(order) + +scope.keyset_paginate.records # works +``` diff --git a/doc/development/database/pagination_guidelines.md b/doc/development/database/pagination_guidelines.md index 3308ebfcaae..ce656851f86 100644 --- a/doc/development/database/pagination_guidelines.md +++ b/doc/development/database/pagination_guidelines.md @@ -58,9 +58,7 @@ It's not possible to make all filter and sort combinations performant, so we sho ### Prepare for scaling -Offset-based pagination is the easiest way to paginate over records, however, it does not scale well for large tables. As a long-term solution, keyset pagination is preferred. The tooling around keyset pagination is not as mature as for offset pagination so currently, it's easier to start with offset pagination and then switch to keyset pagination. - -To avoid losing functionality and maintaining backward compatibility when switching pagination methods, it's advised to consider the following approach in the design phase: +Offset-based pagination is the easiest way to paginate over records, however, it does not scale well for large database tables. As a long-term solution, [keyset pagination](keyset_pagination.md) is preferred. Switching between offset and keyset pagination is generally straightforward and can be done without affecting the end-user if the following conditions are met: - Avoid presenting total counts, prefer limit counts. - Example: count maximum 1001 records, and then on the UI show 1000+ if the count is 1001, show the actual number otherwise. @@ -304,7 +302,22 @@ LIMIT 20 ##### Tooling -Using keyset pagination outside of GraphQL is not straightforward. We have the low-level blocks for building keyset pagination database queries, however, the usage in application code is still not streamlined yet. +A generic keyset pagination library is available within the GitLab project which can most of the cases easly replace the existing, kaminari based pagination with significant performance improvements when dealing with large datasets. + +Example: + +```ruby +# first page +paginator = Project.order(:created_at, :id).keyset_paginate(per_page: 20) +puts paginator.to_a # records + +# next page +cursor = paginator.cursor_for_next_page +paginator = Project.order(:created_at, :id).keyset_paginate(cursor: cursor, per_page: 20) +puts paginator.to_a # records +``` + +For a comprehensive overview, take a look at the [keyset pagination guide](keyset_pagination.md) page. #### Performance diff --git a/doc/development/documentation/styleguide/index.md b/doc/development/documentation/styleguide/index.md index 225db273cb6..7787366dbf4 100644 --- a/doc/development/documentation/styleguide/index.md +++ b/doc/development/documentation/styleguide/index.md @@ -216,15 +216,15 @@ to update. Put files for a specific product area into the related folder: -| Directory | What belongs here | +| Directory | Contents | |:----------------------|:------------------| -| `doc/user/` | User related documentation. Anything that can be done in the GitLab user interface goes here, including usage of the `/admin` interface. | +| `doc/user/` | Documentation for users. Anything that can be done in the GitLab user interface goes here, including usage of the `/admin` interface. | | `doc/administration/` | Documentation that requires the user to have access to the server where GitLab is installed. Administrator settings in the GitLab user interface are under `doc/user/admin_area/`. | -| `doc/api/` | API-related documentation. | +| `doc/api/` | Documentation for the API. | | `doc/development/` | Documentation related to the development of GitLab, whether contributing code or documentation. Related process and style guides should go here. | | `doc/legal/` | Legal documents about contributing to GitLab. | -| `doc/install/` | Contains instructions for installing GitLab. | -| `doc/update/` | Contains instructions for updating GitLab. | +| `doc/install/` | Instructions for installing GitLab. | +| `doc/update/` | Instructions for updating GitLab. | | `doc/topics/` | Indexes per topic (`doc/topics/topic_name/index.md`): all resources for that topic. | ### Work with directories and files @@ -300,11 +300,17 @@ Do not include the same information in multiple places. ## Language -GitLab documentation should be clear and easy to understand. +GitLab documentation should be clear and easy to understand. Avoid unnecessary words. -- Be clear, concise, and stick to the goal of the documentation. +- Be clear, concise, and stick to the goal of the topic. - Write in US English with US grammar. (Tested in [`British.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/British.yml).) - Use [inclusive language](#inclusive-language). +- Rewrite to avoid wordiness: + - there is + - there are + - enables you to + - in order to + - because of the fact that ### Capitalization diff --git a/doc/development/fe_guide/graphql.md b/doc/development/fe_guide/graphql.md index 870605c82f4..844ef2156d9 100644 --- a/doc/development/fe_guide/graphql.md +++ b/doc/development/fe_guide/graphql.md @@ -842,6 +842,70 @@ Keep in mind, this means your app will not batch queries. Once subscriptions are mature, this process can be replaced by using them and we can remove the separate link library and return to batching queries. +#### Subscriptions + +We use [subscriptions](https://www.apollographql.com/docs/react/data/subscriptions/) to receive real-time updates from GraphQL API via websockets. Currently, the number of existing subscriptions is limited, you can check a list of available ones in [GraphqiQL explorer](https://gitlab.com/-/graphql-explorer) + +**NOTE:** +We cannot test subscriptions using GraphiQL, because they require an ActionCable client, which GraphiQL does not support at the moment. + +Subscriptions don't require any additional configuration of Apollo Client instance, you can use them in the application right away. To distinguish subscriptions from queries and mutations, we recommend naming them with `.subscription.graphql` extension: + +```graphql +// ~/sidebar/queries/issuable_assignees.subscription.graphql + +subscription issuableAssigneesUpdated($issuableId: IssuableID!) { + issuableAssigneesUpdated(issuableId: $issuableId) { + ... on Issue { + assignees { + nodes { + ...User + status { + availability + } + } + } + } + } +} +``` + +When using GraphQL subscriptions in Vue application, we recommend updating existing Apollo query results with [subscribeToMore](https://apollo.vuejs.org/guide/apollo/subscriptions.html#subscribe-to-more) option: + +```javascript +import issuableAssigneesSubscription from '~/sidebar/queries/issuable_assignees.subscription.graphql' + +apollo: { + issuable: { + query() { + return assigneesQueries[this.issuableType].query; + }, + subscribeToMore: { + // Specify the subscription that will update the query + document() { + return issuableAssigneesSubscription; + }, + variables() { + return { + issuableId: convertToGraphQLId(this.issuableClass, this.issuableId), + }; + }, + // Describe how subscription should update the query + updateQuery(prev, { subscriptionData }) { + if (prev && subscriptionData?.data?.issuableAssigneesUpdated) { + const data = produce(prev, (draftData) => { + draftData.workspace.issuable.assignees.nodes = + subscriptionData.data.issuableAssigneesUpdated.assignees.nodes; + }); + return data; + } + return prev; + }, + }, + }, +}, +``` + ### Testing #### Generating the GraphQL schema diff --git a/doc/development/query_performance.md b/doc/development/query_performance.md index 87e26cf42df..3ff36c7d005 100644 --- a/doc/development/query_performance.md +++ b/doc/development/query_performance.md @@ -38,8 +38,8 @@ cache, or what PostgreSQL calls shared buffers. This is the "warm cache" query. When analyzing an [`EXPLAIN` plan](understanding_explain_plans.md), you can see the difference not only in the timing, but by looking at the output for `Buffers` -by running your explain with `EXPLAIN(analyze, buffers)`. The [#database-lab](understanding_explain_plans.md#database-lab) -tool will automatically include these options. +by running your explain with `EXPLAIN(analyze, buffers)`. [Database Lab](understanding_explain_plans.md#database-lab-engine) +will automatically include these options. If you are making a warm cache query, you will only see the `shared hits`. diff --git a/doc/development/sidekiq_style_guide.md b/doc/development/sidekiq_style_guide.md index c87870b088c..7bc3ecf002f 100644 --- a/doc/development/sidekiq_style_guide.md +++ b/doc/development/sidekiq_style_guide.md @@ -155,7 +155,7 @@ A job scheduled for an idempotent worker is [deduplicated](#deduplication) when an unstarted job with the same arguments is already in the queue. WARNING: -For [data consistency jobs](#job-data-consistency), the deduplication is not compatible with the +For [data consistency jobs](#job-data-consistency-strategies), the deduplication is not compatible with the `data_consistency` attribute set to `:sticky` or `:delayed`. The reason for this is that deduplication always takes into account the latest binary replication pointer into account, not the first one. There is an [open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/325291) to improve this. @@ -462,18 +462,56 @@ If we expect an increase of **less than 5%**, then no further action is needed. Otherwise, please ping `@gitlab-org/scalability` on the merge request and ask for a review. -## Job data consistency +## Job data consistency strategies -In order to utilize [Sidekiq read-only database replicas capabilities](../administration/database_load_balancing.md#enable-the-load-balancer-for-sidekiq), -set the `data_consistency` attribute of the job to `:always`, `:sticky`, or `:delayed`. +In GitLab 13.11 and earlier, Sidekiq workers would always send database queries to the primary +database node, +both for reads and writes. This ensured that data integrity +is both guaranteed and immediate, since in a single-node scenario it is impossible to encounter +stale reads even for workers that read their own writes. +If a worker writes to the primary, but reads from a replica, however, the possibility +of reading a stale record is non-zero due to replicas potentially lagging behind the primary. + +When the number of jobs that rely on the database increases, ensuring immediate data consistency +can put unsustainable load on the primary database server. We therefore added the ability to use +[database load-balancing in Sidekiq workers](../administration/database_load_balancing.md#enable-the-load-balancer-for-sidekiq). +By configuring a worker's `data_consistency` field, we can then allow the scheduler to target read replicas +under several strategies outlined below. + +## Trading immediacy for reduced primary load + +Not requiring immediate data consistency allows developers to decide to either: + +- Ensure immediately consistent reads, but increase load on the primary database. +- Prefer read replicas to add relief to the primary, but increase the likelihood of stale reads that have to be retried. + +By default, any worker has a data consistency requirement of `:always`, so, as before, all +database operations target the primary. To allow for reads to be served from replicas instead, we +added two additional consistency modes: `:sticky` and `:delayed`. + +When you declare either `:sticky` or `:delayed` consistency, workers become eligible for database +load-balancing. In both cases, jobs are enqueued with a short delay. +This minimizes the likelihood of replication lag after a write. + +The difference is in what happens when there is replication lag after the delay: `sticky` workers +switch over to the primary right away, whereas `delayed` workers fail fast and are retried once. +If they still encounter replication lag, they also switch to the primary instead. +**If your worker never performs any writes, it is strongly advised to apply one of these consistency settings, +since it will never need to rely on the primary database node.** + +The table below shows the `data_consistency` attribute and its values, ordered by the degree to which +they prefer read replicas and will wait for replicas to catch up: | **Data Consistency** | **Description** | |--------------|-----------------------------| -| `:always` | The job is required to use the primary database (default). | -| `:sticky` | The job uses a replica as long as possible. It switches to primary either on write or long replication lag. It should be used on jobs that require to be executed as fast as possible. | -| `:delayed` | The job always uses replica, but switches to primary on write. The job is delayed if there's a long replication lag. If the replica is not up-to-date with the next retry, it switches to the primary. It should be used on jobs where we are fine to delay the execution of a given job due to their importance such as expire caches, execute hooks, etc. | +| `:always` | The job is required to use the primary database (default). It should be used for workers that primarily perform writes or that have very strict requirements around reading their writes without suffering any form of delay. | +| `:sticky` | The job prefers replicas, but switches to the primary for writes or when encountering replication lag. It should be used for jobs that require to be executed as fast as possible but can sustain a small initial queuing delay. | +| `:delayed` | The job prefers replicas, but switches to the primary for writes. When encountering replication lag before the job starts, the job is retried once. If the replica is still not up to date on the next retry, it switches to the primary. It should be used for jobs where delaying execution further typically does not matter, such as cache expiration or web hooks execution. | + +In all cases workers read either from a replica that is fully caught up, +or from the primary node, so data consistency is always ensured. -To set a data consistency for a job, use the `data_consistency` class method: +To set a data consistency for a worker, use the `data_consistency` class method: ```ruby class DelayedWorker @@ -499,8 +537,8 @@ When `feature_flag` is disabled, the job defaults to `:always`, which means that The `feature_flag` property does not allow the use of [feature gates based on actors](../development/feature_flags/index.md). This means that the feature flag cannot be toggled only for particular -projects, groups, or users, but instead, you can safely use [percentage of time rollout](../development/feature_flags/index.md). -Note that since we check the feature flag on both Sidekiq client and server, rolling out a 10% of the time, +projects, groups, or users, but instead, you can safely use [percentage of time rollout](../development/feature_flags/index.md). +Note that since we check the feature flag on both Sidekiq client and server, rolling out a 10% of the time, will likely results in 1% (0.1 [from client]*0.1 [from server]) of effective jobs using replicas. Example: @@ -515,15 +553,6 @@ class DelayedWorker end ``` -### Delayed job execution - -Scheduling workers that utilize [Sidekiq read-only database replicas capabilities](#job-data-consistency), -(workers with `data_consistency` attribute set to `:sticky` or `:delayed`), -by calling `SomeWorker.perform_async` results in a worker performing in the future (1 second in the future). - -This way, the replica has a chance to catch up, and the job will likely use the replica. -For workers with `data_consistency` set to `:delayed`, it can also reduce the number of retried jobs. - ## Jobs with External Dependencies Most background jobs in the GitLab application communicate with other GitLab diff --git a/doc/development/understanding_explain_plans.md b/doc/development/understanding_explain_plans.md index 66dc1fef31a..f9d1e7e2eee 100644 --- a/doc/development/understanding_explain_plans.md +++ b/doc/development/understanding_explain_plans.md @@ -198,13 +198,39 @@ Here we can see that our filter has to remove 65,677 rows, and that we use 208,846 buffers. Each buffer in PostgreSQL is 8 KB (8192 bytes), meaning our above node uses *1.6 GB of buffers*. That's a lot! +Keep in mind that some statistics are per-loop averages, while others are total values: + +| Field name | Value type | +| --- | --- | +| Actual Total Time | per-loop average | +| Actual Rows | per-loop average | +| Buffers Shared Hit | total value | +| Buffers Shared Read | total value | +| Buffers Shared Dirtied | total value | +| Buffers Shared Written | total value | +| I/O Read Time | total value | +| I/O Read Write | total value | + +For example: + +```sql + -> Index Scan using users_pkey on public.users (cost=0.43..3.44 rows=1 width=1318) (actual time=0.025..0.025 rows=1 loops=888) + Index Cond: (users.id = issues.author_id) + Buffers: shared hit=3543 read=9 + I/O Timings: read=17.760 write=0.000 +``` + +Here we can see that this node used 3552 buffers (3543 + 9), returned 888 rows (`888 * 1`), and the actual duration was 22.2 milliseconds (`888 * 0.025`). +17.76 milliseconds of the total duration was spent in reading from disk, to retrieve data that was not in the cache. + ## Node types There are quite a few different types of nodes, so we only cover some of the more common ones here. A full list of all the available nodes and their descriptions can be found in -the [PostgreSQL source file `plannodes.h`](https://gitlab.com/postgres/postgres/blob/master/src/include/nodes/plannodes.h) +the [PostgreSQL source file `plannodes.h`](https://gitlab.com/postgres/postgres/blob/master/src/include/nodes/plannodes.h). +pgMustard's [EXPLAIN docs](https://www.pgmustard.com/docs/explain) also offer detailed look into nodes and their fields. ### Seq Scan @@ -441,7 +467,7 @@ When optimizing a query, we usually need to reduce the amount of data we're dealing with. Indexes are the way to work with fewer pages (buffers) to get the result, so, during optimization, look at the number of buffers used (read and hit), and work on reducing these numbers. Reduced timing will be the consequence of reduced -buffer numbers. [#database-lab](#database-lab) guarantees that the plan is structurally +buffer numbers. [Database Lab Engine](#database-lab-engine) guarantees that the plan is structurally identical to production (and overall number of buffers is the same as on production), but difference in cache state and I/O speed may lead to different timings. @@ -617,7 +643,7 @@ If we look at the plan we also see our costs are very low: Index Scan using projects_pkey on projects (cost=0.43..3.45 rows=1 width=4) (actual time=0.049..0.050 rows=1 loops=145) ``` -Here our cost is only 3.45, and it only takes us 0.050 milliseconds to do so. +Here our cost is only 3.45, and it takes us 7.25 milliseconds to do so (0.05 * 145). The next index scan is a bit more expensive: ```sql @@ -681,64 +707,26 @@ There are a few ways to get the output of a query plan. Of course you can directly run the `EXPLAIN` query in the `psql` console, or you can follow one of the other options below. -### Rails console +### Database Lab Engine -Using the [`activerecord-explain-analyze`](https://github.com/6/activerecord-explain-analyze) -you can directly generate the query plan from the Rails console: +GitLab team members can use [Database Lab Engine](https://gitlab.com/postgres-ai/database-lab), and the companion +SQL optimization tool - [Joe Bot](https://gitlab.com/postgres-ai/joe). -```ruby -pry(main)> require 'activerecord-explain-analyze' -=> true -pry(main)> Project.where('build_timeout > ?', 3600).explain(analyze: true) - Project Load (1.9ms) SELECT "projects".* FROM "projects" WHERE (build_timeout > 3600) - ↳ (pry):12 -=> EXPLAIN for: SELECT "projects".* FROM "projects" WHERE (build_timeout > 3600) -Seq Scan on public.projects (cost=0.00..2.17 rows=1 width=742) (actual time=0.040..0.041 rows=0 loops=1) - Output: id, name, path, description, created_at, updated_at, creator_id, namespace_id, ... - Filter: (projects.build_timeout > 3600) - Rows Removed by Filter: 14 - Buffers: shared hit=2 -Planning time: 0.411 ms -Execution time: 0.113 ms -``` +Database Lab Engine provides developers with their own clone of the production database, while Joe Bot helps with exploring execution plans. -### ChatOps +Joe Bot is available in the [`#database-lab`](https://gitlab.slack.com/archives/CLJMDRD8C) channel on Slack, +and through its [web interface](https://console.postgres.ai/gitlab/joe-instances). -[GitLab team members can also use our ChatOps solution, available in Slack using the -`/chatops` slash command](chatops_on_gitlabcom.md). -You can use ChatOps to get a query plan by running the following: +With Joe Bot you can execute DDL statements (like creating indexes, tables, and columns) and get query plans for `SELECT`, `UPDATE`, and `DELETE` statements. -```sql -/chatops run explain SELECT COUNT(*) FROM projects WHERE visibility_level IN (0, 20) -``` +For example, in order to test new index on a column that is not existing on production yet, you can do the following: -Visualising the plan using <https://explain.depesz.com/> is also supported: +Create the column: ```sql -/chatops run explain --visual SELECT COUNT(*) FROM projects WHERE visibility_level IN (0, 20) +exec ALTER TABLE projects ADD COLUMN last_at timestamp without time zone ``` -Quoting the query is not necessary. - -For more information about the available options, run: - -```sql -/chatops run explain --help -``` - -### `#database-lab` - -Another tool GitLab team members can use is a chatbot powered by [Joe](https://gitlab.com/postgres-ai/joe) -which uses [Database Lab](https://gitlab.com/postgres-ai/database-lab) to instantly provide developers -with their own clone of the production database. - -Joe is available in the -[`#database-lab`](https://gitlab.slack.com/archives/CLJMDRD8C) channel on Slack. - -Unlike ChatOps, it gives you a way to execute DDL statements (like creating indexes and tables) and get query plan not only for `SELECT` but also `UPDATE` and `DELETE`. - -For example, in order to test new index you can do the following: - Create the index: ```sql @@ -769,18 +757,67 @@ For more information about the available options, run: help ``` +The web interface comes with the following execution plan visualizers included: + +- [Depesz](https://explain.depesz.com/) +- [PEV2](https://github.com/dalibo/pev2) +- [FlameGraph](https://github.com/mgartner/pg_flame) + #### Tips & Tricks -The database connection is now maintained during your whole session, so you can use `exec set ...` for any session variables (such as `enable_seqscan` or `work_mem`). These settings will be applied to all subsequent commands until you reset them. +The database connection is now maintained during your whole session, so you can use `exec set ...` for any session variables (such as `enable_seqscan` or `work_mem`). These settings will be applied to all subsequent commands until you reset them. For example you can disable parallel queries with + +```sql +exec SET max_parallel_workers_per_gather = 0 +``` + +### Rails console + +Using the [`activerecord-explain-analyze`](https://github.com/6/activerecord-explain-analyze) +you can directly generate the query plan from the Rails console: + +```ruby +pry(main)> require 'activerecord-explain-analyze' +=> true +pry(main)> Project.where('build_timeout > ?', 3600).explain(analyze: true) + Project Load (1.9ms) SELECT "projects".* FROM "projects" WHERE (build_timeout > 3600) + ↳ (pry):12 +=> EXPLAIN for: SELECT "projects".* FROM "projects" WHERE (build_timeout > 3600) +Seq Scan on public.projects (cost=0.00..2.17 rows=1 width=742) (actual time=0.040..0.041 rows=0 loops=1) + Output: id, name, path, description, created_at, updated_at, creator_id, namespace_id, ... + Filter: (projects.build_timeout > 3600) + Rows Removed by Filter: 14 + Buffers: shared hit=2 +Planning time: 0.411 ms +Execution time: 0.113 ms +``` + +### ChatOps + +[GitLab team members can also use our ChatOps solution, available in Slack using the +`/chatops` slash command](chatops_on_gitlabcom.md). + +NOTE: +While ChatOps is still available, the recommended way to generate execution plans is to use [Database Lab Engine](#database-lab-engine). -It is also possible to use transactions. This may be useful when you are working on statements that modify the data, for example INSERT, UPDATE, and DELETE. The `explain` command will perform `EXPLAIN ANALYZE`, which executes the statement. In order to run each `explain` starting from a clean state you can wrap it in a transaction, for example: +You can use ChatOps to get a query plan by running the following: ```sql -exec BEGIN +/chatops run explain SELECT COUNT(*) FROM projects WHERE visibility_level IN (0, 20) +``` -explain UPDATE some_table SET some_column = TRUE +Visualising the plan using <https://explain.depesz.com/> is also supported: + +```sql +/chatops run explain --visual SELECT COUNT(*) FROM projects WHERE visibility_level IN (0, 20) +``` -exec ROLLBACK +Quoting the query is not necessary. + +For more information about the available options, run: + +```sql +/chatops run explain --help ``` ## Further reading diff --git a/doc/development/usage_ping/index.md b/doc/development/usage_ping/index.md index 95dc4f2979a..de6a234e20c 100644 --- a/doc/development/usage_ping/index.md +++ b/doc/development/usage_ping/index.md @@ -24,11 +24,32 @@ More links: ## What is Usage Ping? -- GitLab sends a weekly payload containing usage data to GitLab Inc. Usage Ping provides high-level data to help our product, support, and sales teams. It does not send any project names, usernames, or any other specific data. The information from the usage ping is not anonymous, it is linked to the hostname of the instance. Sending usage ping is optional, and any instance can disable analytics. -- The usage data is primarily composed of row counts for different tables in the instance's database. By comparing these counts month over month (or week over week), we can get a rough sense for how an instance is using the different features in the product. In addition to counts, other facts - that help us classify and understand GitLab installations are collected. -- Usage ping is important to GitLab as we use it to calculate our Stage Monthly Active Users (SMAU) which helps us measure the success of our stages and features. -- While usage ping is enabled, GitLab gathers data from the other instances and can show usage statistics of your instance to your users. +Usage Ping is a process in GitLab that collects and sends a weekly payload to GitLab Inc. +The payload provides important high-level data that helps our product, support, +and sales teams understand how GitLab is used. For example, the data helps to: + +- Compare counts month over month (or week over week) to get a rough sense for how an instance uses + different product features. +- Collect other facts that help us classify and understand GitLab installations. +- Calculate our Stage Monthly Active Users (SMAU), which helps to measure the success of our stages + and features. + +Usage Ping information is not anonymous. It's linked to the instance's hostname. However, it does +not contain project names, usernames, or any other specific data. + +Sending a Usage Ping payload is optional and can be [disabled](#disable-usage-ping) on any instance. +When Usage Ping is enabled, GitLab gathers data from the other instances +and can show your instance's usage statistics to your users. + +### Terminology + +We use the following terminology to describe the Usage Ping components: + +- **Usage Ping**: the process that collects and generates a JSON payload. +- **Usage data**: the contents of the Usage Ping JSON payload. This includes metrics. +- **Metrics**: primarily made up of row counts for different tables in an instance's database. Each + metric has a corresponding [metric definition](metrics_dictionary.md#metrics-definition-and-validation) + in a YAML file. ### Why should we enable Usage Ping? diff --git a/doc/user/admin_area/analytics/usage_trends.md b/doc/user/admin_area/analytics/usage_trends.md index 49c81b1a965..9c09b62f8af 100644 --- a/doc/user/admin_area/analytics/usage_trends.md +++ b/doc/user/admin_area/analytics/usage_trends.md @@ -17,7 +17,10 @@ This feature might not be available to you. Check the **version history** note a Usage Trends gives you an overview of how much data your instance contains, and how quickly this volume is changing over time. -To see Usage Trends, go to **Admin Area > Analytics > Usage Trends**. +To see Usage Trends: + +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Analytics > Usage Trends**. ## Total counts diff --git a/doc/user/admin_area/settings/email.md b/doc/user/admin_area/settings/email.md index 5deb71698ff..e637408328d 100644 --- a/doc/user/admin_area/settings/email.md +++ b/doc/user/admin_area/settings/email.md @@ -20,8 +20,9 @@ The logo in the header of some emails can be customized, see the [logo customiza The additional text appears at the bottom of any email and can be used for legal/auditing/compliance reasons. -1. Go to **Admin Area > Settings > Preferences** (`/admin/application_settings/preferences`). -1. Expand the **Email** section. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > Preferences** (`/admin/application_settings/preferences`). +1. Expand **Email**. 1. Enter your text in the **Additional text** field. 1. Click **Save**. @@ -34,8 +35,9 @@ This configuration option sets the email hostname for [private commit emails](.. In order to change this option: -1. Go to **Admin Area > Settings > Preferences** (`/admin/application_settings/preferences`). -1. Expand the **Email** section. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > Preferences** (`/admin/application_settings/preferences`). +1. Expand **Email**. 1. Enter the desired hostname in the **Custom hostname (for private commit emails)** field. 1. Select **Save changes**. diff --git a/doc/user/admin_area/settings/img/file_template_admin_area.png b/doc/user/admin_area/settings/img/file_template_admin_area.png Binary files differdeleted file mode 100644 index 269d997e1d9..00000000000 --- a/doc/user/admin_area/settings/img/file_template_admin_area.png +++ /dev/null diff --git a/doc/user/admin_area/settings/img/file_template_admin_area_v14_0.png b/doc/user/admin_area/settings/img/file_template_admin_area_v14_0.png Binary files differnew file mode 100644 index 00000000000..33fce8a2b77 --- /dev/null +++ b/doc/user/admin_area/settings/img/file_template_admin_area_v14_0.png diff --git a/doc/user/admin_area/settings/instance_template_repository.md b/doc/user/admin_area/settings/instance_template_repository.md index c8a4c2866ca..8a796435ef8 100644 --- a/doc/user/admin_area/settings/instance_template_repository.md +++ b/doc/user/admin_area/settings/instance_template_repository.md @@ -23,7 +23,7 @@ To select a project to serve as the custom template repository: 1. In the left sidebar, select **Settings > Templates**. 1. Select the project: -  +  1. Add custom templates to the selected repository. diff --git a/doc/user/admin_area/settings/rate_limit_on_issues_creation.md b/doc/user/admin_area/settings/rate_limit_on_issues_creation.md index 3acfb636a13..ef2b8ad80cd 100644 --- a/doc/user/admin_area/settings/rate_limit_on_issues_creation.md +++ b/doc/user/admin_area/settings/rate_limit_on_issues_creation.md @@ -10,7 +10,13 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/28129) in GitLab 12.10. This setting allows you to rate limit the requests to the issue creation endpoint. -You can change its value in **Admin Area > Settings > Network > Issues Rate Limits**. +To can change its value: + +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > Network**. +1. Expand **Issues Rate Limits**. +1. Under **Max requests per minute per user**, enter the new value. +1. Select **Save changes**. For example, if you set a limit of 300, requests using the [Projects::IssuesController#create](https://gitlab.com/gitlab-org/gitlab/raw/master/app/controllers/projects/issues_controller.rb) diff --git a/doc/user/admin_area/settings/rate_limit_on_notes_creation.md b/doc/user/admin_area/settings/rate_limit_on_notes_creation.md index 67a97d26b34..193f39542cf 100644 --- a/doc/user/admin_area/settings/rate_limit_on_notes_creation.md +++ b/doc/user/admin_area/settings/rate_limit_on_notes_creation.md @@ -13,9 +13,11 @@ This setting allows you to rate limit the requests to the note creation endpoint To change the note creation rate limit: -1. Go to **Admin Area > Settings > Network**. -1. Expand the **Notes Rate Limits** section. -1. Enter the new value. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > Network**. +1. Expand **Notes Rate Limits**. +1. Under **Max requests per minute per user**, enter the new value. +1. Optional. Under **List of users to be excluded from the limit**, list users to be excluded fromt the limit. 1. Select **Save changes**. This limit is: diff --git a/doc/user/application_security/container_scanning/index.md b/doc/user/application_security/container_scanning/index.md index 8e83ade5608..323a064c3e4 100644 --- a/doc/user/application_security/container_scanning/index.md +++ b/doc/user/application_security/container_scanning/index.md @@ -11,10 +11,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: Versions of GitLab prior to 14.0 used Clair as the default container scanning engine. GitLab 14.0 -replaces Clair with Trivy and removes Clair from the product. If you run container scanning with the -default settings, GitLab switches you seamlessly and automatically to Trivy in GitLab 14.0. However, -if you customized the variables in your container scanning job, you should review the -[migration guide](#migrating-from-clair-to-trivy) and make any necessary updates. +removes Clair from the product and replaces it with two new scanners. If you +run container scanning with the default settings, GitLab switches you seamlessly and automatically +to Trivy in GitLab 14.0. However, if you customized the variables in your container scanning job, +you should review the [migration guide](#change-scanners) +and make any necessary updates. Your application's Docker image may itself be based on Docker images that contain known vulnerabilities. By including an extra job in your pipeline that scans for those vulnerabilities and @@ -23,6 +24,7 @@ displays them in a merge request, you can use GitLab to audit your Docker-based GitLab provides integration with open-source tools for vulnerability static analysis in containers: - [Trivy](https://github.com/aquasecurity/trivy) +- [Grype](https://github.com/anchore/grype) To integrate GitLab with security scanners other than those listed here, see [Security scanner integration](../../../development/integrations/secure.md). @@ -79,8 +81,10 @@ Other changes: - GitLab 13.9 [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/322656) integration with [Trivy](https://github.com/aquasecurity/trivy) by upgrading `CS_MAJOR_VERSION` from `3` to `4`. - GitLab 14.0 [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/61850) - integration with [Trivy](https://github.com/aquasecurity/trivy) - as the default for container scanning. + an integration with [Trivy](https://github.com/aquasecurity/trivy) + as the default for container scanning, and also [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/326279) + an integration with [Grype](https://github.com/anchore/grype) + as an alternative scanner. To include the `Container-Scanning.gitlab-ci.yml` template (GitLab 11.9 and later), add the following to your `.gitlab-ci.yml` file: @@ -151,7 +155,7 @@ You can [configure](#customizing-the-container-scanning-settings) analyzers by u | `ADDITIONAL_CA_CERT_BUNDLE` | `""` | Bundle of CA certs that you want to trust. See [Using a custom SSL CA certificate authority](#using-a-custom-ssl-ca-certificate-authority) for more details. | All | | `CI_APPLICATION_REPOSITORY` | `$CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG` | Docker repository URL for the image to be scanned. | All | | `CI_APPLICATION_TAG` | `$CI_COMMIT_SHA` | Docker repository tag for the image to be scanned. | All | -| `CS_ANALYZER_IMAGE` | `$SECURE_ANALYZERS_PREFIX/$CS_PROJECT:$CS_MAJOR_VERSION` | Docker image of the analyzer. | All | +| `CS_ANALYZER_IMAGE` | `registry.gitlab.com/security-products/container-scanning:4` | Docker image of the analyzer. | All | | `CS_DOCKER_INSECURE` | `"false"` | Allow access to secure Docker registries using HTTPS without validating the certificates. | All | | `CS_REGISTRY_INSECURE` | `"false"` | Allow access to insecure registries (HTTP only). Should only be set to `true` when testing the image locally. | Trivy. The registry must listen on port `80/tcp`. | | `CS_SEVERITY_THRESHOLD` | `UNKNOWN` | Severity level threshold. The scanner outputs vulnerabilities with severity level higher than or equal to this threshold. Supported levels are Unknown, Low, Medium, High, and Critical. | Trivy | @@ -165,6 +169,7 @@ You can [configure](#customizing-the-container-scanning-settings) analyzers by u Support depends on the scanner: +- [Grype](https://github.com/anchore/grype#grype) - [Trivy](https://aquasecurity.github.io/trivy/latest/vuln-detection/os/) (Default). ### Overriding the container scanning template @@ -189,7 +194,18 @@ GitLab 13.0 and later doesn't support [`only` and `except`](../../../ci/yaml/REA When overriding the template, you must use [`rules`](../../../ci/yaml/README.md#rules) instead. -### Migrating from Clair to Trivy +### Change scanners + +The container-scanning analyzer can use different scanners, depending on the value of the +`CS_ANALYZER_IMAGE` configuration variable. + +The following options are available: + +| Scanner name | `CS_ANALYZER_IMAGE` | +| ------------ | ------------------- | +| Default ([Trivy](https://github.com/aquasecurity/trivy)) | `registry.gitlab.com/security-products/container-scanning:4` | +| [Grype](https://github.com/anchore/grype) | `registry.gitlab.com/security-products/container-scanning/grype:4` | +| Trivy | `registry.gitlab.com/security-products/container-scanning/trivy:4` | If you're migrating from a GitLab 13.x release to a GitLab 14.x release and have customized the `container_scanning` job or its CI variables, you might need to perform these migration steps in @@ -214,17 +230,16 @@ your CI file: complete list of supported variables, see [available variables](#available-cicd-variables). 1. Make any [necessary customizations](#customizing-the-container-scanning-settings) - to the `Trivy` scanner. We recommend that you minimize such customizations, as they might require + to the chosen scanner. We recommend that you minimize such customizations, as they might require changes in future GitLab major releases. 1. Trigger a new run of a pipeline that includes the `container_scanning` job. Inspect the job output and ensure that the log messages do not mention Clair. -**Troubleshooting** - +NOTE: Prior to the GitLab 14.0 release, any variable defined under the scope `container_scanning` is not -considered for the Trivy scanner. Verify that all variables for Trivy are -either defined as a global variable, or under `container_scanning`. +considered for scanners other than Clair. In GitLab 14.0 and later, all variables can be defined +either as a global variable or under `container_scanning`. ### Using a custom SSL CA certificate authority @@ -362,14 +377,17 @@ Support for custom certificate authorities was introduced in the following versi | Scanner | Version | | -------- | ------- | | `Trivy` | [4.0.0](https://gitlab.com/gitlab-org/security-products/analyzers/container-scanning/-/releases/4.0.0) | +| `Grype` | [4.3.0](https://gitlab.com/gitlab-org/security-products/analyzers/container-scanning/-/releases/4.3.0) | #### Make GitLab container scanning analyzer images available inside your Docker registry -For container scanning, import the following default images from `registry.gitlab.com` into your +For container scanning, import the following images from `registry.gitlab.com` into your [local Docker container registry](../../packages/container_registry/index.md): ```plaintext -registry.gitlab.com/security-products/container-scanning +registry.gitlab.com/security-products/container-scanning:4 +registry.gitlab.com/security-products/container-scanning/grype:4 +registry.gitlab.com/security-products/container-scanning/trivy:4 ``` The process for importing Docker images into a local offline Docker registry depends on @@ -410,13 +428,13 @@ following `.gitlab-yml.ci` example as a template. ```yaml variables: SOURCE_IMAGE: registry.gitlab.com/security-products/container-scanning:4 - TARGET_IMAGE: $CI_REGISTRY/$CI_PROJECT_PATH/gitlab-container-scanning + TARGET_IMAGE: $CI_REGISTRY/namespace/gitlab-container-scanning image: docker:stable update-scanner-image: services: - - docker:19-dind + - docker:dind script: - docker pull $SOURCE_IMAGE - docker tag $SOURCE_IMAGE $TARGET_IMAGE diff --git a/doc/user/application_security/sast/index.md b/doc/user/application_security/sast/index.md index c11e367a688..e80807b31bf 100644 --- a/doc/user/application_security/sast/index.md +++ b/doc/user/application_security/sast/index.md @@ -510,8 +510,8 @@ Some analyzers can be customized with CI/CD variables. | `SBT_PATH` | SpotBugs | Path to the `sbt` executable. | | `FAIL_NEVER` | SpotBugs | Set to `1` to ignore compilation failure. | | `SAST_GOSEC_CONFIG` | Gosec | **{warning}** **[Removed](https://gitlab.com/gitlab-org/gitlab/-/issues/328301)** in GitLab 14.0 - use custom rulesets instead. Path to configuration for Gosec (optional). | -| `PHPCS_SECURITY_AUDIT_PHP_EXTENSIONS` | phpcs-security-audit | Comma separated list of additional PHP Extensions. | -| `SAST_DISABLE_BABEL` | NodeJsScan | Disable Babel processing for the NodeJsScan scanner. Set to `true` to disable Babel processing. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/33065) in GitLab 13.2. +| `PHPCS_SECURITY_AUDIT_PHP_EXTENSIONS` | phpcs-security-audit | Comma separated list of additional PHP Extensions. | +| `SAST_DISABLE_BABEL` | NodeJsScan | **{warning}** **[Removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/64025)** in GitLab 13.5 | | `SAST_SEMGREP_METRICS` | Semgrep | Set to `"false"` to disable sending anonymized scan metrics to [r2c](https://r2c.dev/). Default: `true`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/330565) in GitLab 14.0. | #### Custom CI/CD variables diff --git a/doc/user/gitlab_com/index.md b/doc/user/gitlab_com/index.md index 223d3363186..f371de30b88 100644 --- a/doc/user/gitlab_com/index.md +++ b/doc/user/gitlab_com/index.md @@ -11,15 +11,15 @@ This page contains information about the settings that are used on ## SSH host keys fingerprints -Below are the fingerprints for GitLab.com's SSH host keys. The first time you connect -to a GitLab.com repository, one of these keys is displayed in the output. +Below are the fingerprints for GitLab.com's SSH host keys. The first time you +connect to a GitLab.com repository, one of these keys is displayed in the output. -| Algorithm | MD5 (deprecated) | SHA256 | -| --------- | --- | ------- | -| DSA (deprecated) | `7a:47:81:3a:ee:89:89:64:33:ca:44:52:3d:30:d4:87` | `p8vZBUOR0XQz6sYiaWSMLmh0t9i8srqYKool/Xfdfqw` | -| ECDSA | `f1:d0:fb:46:73:7a:70:92:5a:ab:5d:ef:43:e2:1c:35` | `HbW3g8zUjNSksFbqTiUWPWg2Bq1x8xdGUrliXFzSnUw` | -| ED25519 | `2e:65:6a:c8:cf:bf:b2:8b:9a:bd:6d:9f:11:5c:12:16` | `eUXGGm1YGsMAS7vkcx6JOJdOGHPem5gQp4taiCfCLB8` | -| RSA | `b6:03:0e:39:97:9e:d0:e7:24:ce:a3:77:3e:01:42:09` | `ROQFvPThGrW4RuWLoL9tq9I9zJ42fK4XywyRtbOz/EQ` | +| Algorithm | MD5 (deprecated) | SHA256 | +|------------------|------------------|---------| +| ED25519 | `2e:65:6a:c8:cf:bf:b2:8b:9a:bd:6d:9f:11:5c:12:16` | `eUXGGm1YGsMAS7vkcx6JOJdOGHPem5gQp4taiCfCLB8` | +| RSA | `b6:03:0e:39:97:9e:d0:e7:24:ce:a3:77:3e:01:42:09` | `ROQFvPThGrW4RuWLoL9tq9I9zJ42fK4XywyRtbOz/EQ` | +| DSA (deprecated) | `7a:47:81:3a:ee:89:89:64:33:ca:44:52:3d:30:d4:87` | `p8vZBUOR0XQz6sYiaWSMLmh0t9i8srqYKool/Xfdfqw` | +| ECDSA | `f1:d0:fb:46:73:7a:70:92:5a:ab:5d:ef:43:e2:1c:35` | `HbW3g8zUjNSksFbqTiUWPWg2Bq1x8xdGUrliXFzSnUw` | ## SSH `known_hosts` entries @@ -34,32 +34,40 @@ gitlab.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAA ## Mail configuration -GitLab.com sends emails from the `mg.gitlab.com` domain via [Mailgun](https://www.mailgun.com/) and has -its own dedicated IP address (`192.237.158.143`). +GitLab.com sends emails from the `mg.gitlab.com` domain by using [Mailgun](https://www.mailgun.com/), +and has its own dedicated IP address (`192.237.158.143`). -NOTE: The IP address for `mg.gitlab.com` is subject to change at any time. ## Backups [See our backup strategy](https://about.gitlab.com/handbook/engineering/infrastructure/production/#backups). -There are several ways to perform backups of your content on GitLab.com. +To back up an entire project on GitLab.com, you can export it either: -Projects can be backed up in their entirety by exporting them either [through the UI](../project/settings/import_export.md) or [API](../../api/project_import_export.md#schedule-an-export), the latter of which can be used to programmatically upload exports to a storage platform such as AWS S3. +- [Through the UI](../project/settings/import_export.md). +- [Through the API](../../api/project_import_export.md#schedule-an-export). You + can also use the API to programmatically upload exports to a storage platform, + such as Amazon S3. -With exports, be sure to take note of [what is and is not](../project/settings/import_export.md#exported-contents), included in a project export. +With exports, be aware of [what is and is not](../project/settings/import_export.md#exported-contents) +included in a project export. -Since GitLab is built on Git, you can back up **just** the repository of a project by [cloning](../../gitlab-basics/start-using-git.md#clone-a-repository) it to another machine. Similarly, if you need to back up just the wiki of a repository it can also be cloned and all files uploaded to that wiki are included [if they were uploaded after 2020-08-22](../project/wiki/index.md#create-a-new-wiki-page). +GitLab is built on Git, so you can back up just the repository of a project by +[cloning](../../gitlab-basics/start-using-git.md#clone-a-repository) it to +another computer. +Similarly, you can clone a project's wiki to back it up. All files +[uploaded after August 22, 2020](../project/wiki/index.md#create-a-new-wiki-page) +are included when cloning. ## Alternative SSH port -GitLab.com can be reached via a [different SSH port](https://about.gitlab.com/blog/2016/02/18/gitlab-dot-com-now-supports-an-alternate-git-plus-ssh-port/) for `git+ssh`. +GitLab.com can be reached by using a [different SSH port](https://about.gitlab.com/blog/2016/02/18/gitlab-dot-com-now-supports-an-alternate-git-plus-ssh-port/) for `git+ssh`. -| Setting | Value | -| --------- | ------------------- | -| `Hostname` | `altssh.gitlab.com` | -| `Port` | `443` | +| Setting | Value | +|------------|---------------------| +| `Hostname` | `altssh.gitlab.com` | +| `Port` | `443` | An example `~/.ssh/config` is the following: @@ -76,26 +84,26 @@ Host gitlab.com Below are the settings for [GitLab Pages](https://about.gitlab.com/stages-devops-lifecycle/pages/). -| Setting | GitLab.com | Default | -| --------------------------- | ---------------- | ------------- | -| Domain name | `gitlab.io` | - | -| IP address | `35.185.44.232` | - | -| Custom domains support | yes | no | -| TLS certificates support | yes | no | -| Maximum size (compressed) | 1G | 100M | +| Setting | GitLab.com | Default | +|---------------------------|------------------------|------------------------| +| Domain name | `gitlab.io` | - | +| IP address | `35.185.44.232` | - | +| Custom domains support | **{check-circle}** Yes | **{dotted-circle}** No | +| TLS certificates support | **{check-circle}** Yes | **{dotted-circle}** No | +| Maximum size (compressed) | 1 GB | 100 MB | -NOTE: -The maximum size of your Pages site is regulated by the artifacts maximum size +The maximum size of your Pages site is regulated by the artifacts maximum size, which is part of [GitLab CI/CD](#gitlab-cicd). ## GitLab CI/CD Below are the current settings regarding [GitLab CI/CD](../../ci/README.md). -Any settings or feature limits not listed here are using the defaults listed in the related documentation. +Any settings or feature limits not listed here are using the defaults listed in +the related documentation. -| Setting | GitLab.com | Default | -| ----------- | ----------------- | ------------- | -| Artifacts maximum size (compressed) | 1G | 100M | +| Setting | GitLab.com | Default | +|-------------------------------------|------------|---------| +| Artifacts maximum size (compressed) | 1 GB | 100 MB | | Artifacts [expiry time](../../ci/yaml/README.md#artifactsexpire_in) | From June 22, 2020, deleted after 30 days unless otherwise specified (artifacts created before that date have no expiry). | deleted after 30 days unless otherwise specified | | Scheduled Pipeline Cron | `*/5 * * * *` | `3-59/10 * * * *` | | [Max jobs in active pipelines](../../administration/instance_limits.md#number-of-jobs-in-active-pipelines) | `500` for Free tier, unlimited otherwise | Unlimited | @@ -107,19 +115,22 @@ Any settings or feature limits not listed here are using the defaults listed in ## Account and limit settings -GitLab.com has the following [account limits](../admin_area/settings/account_and_limit_settings.md) enabled. If a setting is not listed, it is set to the default value. +GitLab.com has the following [account limits](../admin_area/settings/account_and_limit_settings.md) +enabled. If a setting is not listed, it is set to the default value. -If you are near -or over the repository size limit, you can [reduce your repository size with Git](../project/repository/reducing_the_repo_size_using_git.md). +If you are near or over the repository size limit, you can +[reduce your repository size with Git](../project/repository/reducing_the_repo_size_using_git.md). -| Setting | GitLab.com | Default | -| ----------- | ----------- | ------------- | +| Setting | GitLab.com | Default | +|-------------------------------|------------|---------| | [Repository size including LFS](../admin_area/settings/account_and_limit_settings.md#repository-size-limit) | 10 GB | Unlimited | -| Maximum import size | 5 GB | Unlimited ([Modified](https://gitlab.com/gitlab-org/gitlab/-/issues/251106) from 50MB to unlimited in GitLab 13.8. | -| Maximum attachment size | 10 MB | 10 MB | +| Maximum import size | 5 GB | Unlimited ([Modified](https://gitlab.com/gitlab-org/gitlab/-/issues/251106) from 50MB to unlimited in GitLab 13.8. | +| Maximum attachment size | 10 MB | 10 MB | NOTE: -`git push` and GitLab project imports are limited to 5 GB per request through Cloudflare. Git LFS and imports other than a file upload are not affected by this limit. +`git push` and GitLab project imports are limited to 5 GB per request through +Cloudflare. Git LFS and imports other than a file upload are not affected by +this limit. ## IP range @@ -129,17 +140,16 @@ from those IPs and allow them. GitLab.com is fronted by Cloudflare. For incoming connections to GitLab.com you might need to allow CIDR blocks of Cloudflare ([IPv4](https://www.cloudflare.com/ips-v4) and [IPv6](https://www.cloudflare.com/ips-v6)). -For outgoing connections from CI/CD runners we are not providing static IP addresses. -All our runners are deployed into Google Cloud Platform (GCP) - any IP based -firewall can be configured by looking up all +For outgoing connections from CI/CD runners, we are not providing static IP +addresses. All GitLab runners are deployed into Google Cloud Platform (GCP). Any +IP-based firewall can be configured by looking up all [IP address ranges or CIDR blocks for GCP](https://cloud.google.com/compute/docs/faq#find_ip_range). ## Hostname list -To configure allow-lists in local HTTP(S) proxies, or other -web-blocking software that govern end-user machines, -pages on GitLab.com will attempt to load content from -the following hostnames: +Add these hostnames when you configure allow-lists in local HTTP(S) proxies, +or other web-blocking software that governs end-user computers. Pages on +GitLab.com load content from these hostnames: - `gitlab.com` - `*.gitlab.com` @@ -147,19 +157,18 @@ the following hostnames: - `*.gitlab.io` - `*.gitlab.net` -Documentation and Company pages served over `docs.gitlab.com` -and `about.gitlab.com` will attempt to also load certain page -content directly from common public CDN hostnames. +Documentation and Company pages served over `docs.gitlab.com` and `about.gitlab.com` +also load certain page content directly from common public CDN hostnames. ## Webhooks The following limits apply for [Webhooks](../project/integrations/webhooks.md): -| Setting | GitLab.com | Default | -| ------- | ---------- | ------- | -| [Webhook rate limit](../../administration/instance_limits.md#webhook-rate-limit) | `120` calls per minute for Free tier, unlimited for all paid tiers | Unlimited -| [Number of webhooks](../../administration/instance_limits.md#number-of-webhooks) | `100` per-project, `50` per-group | `100` per-project, `50` per-group -| Maximum payload size | `25 MB` | `25 MB` +| Setting | GitLab.com | Default | +|----------------------|------------|---------| +| [Webhook rate limit](../../administration/instance_limits.md#webhook-rate-limit) | `120` calls per minute for GitLab Free, unlimited for GitLab Premium and GitLab Ultimate | Unlimited | +| [Number of webhooks](../../administration/instance_limits.md#number-of-webhooks) | `100` per project, `50` per group | `100` per project, `50` per group | +| Maximum payload size | 25 MB | 25 MB | ## Shared runners @@ -172,15 +181,15 @@ For more information, see [choosing a runner](../../ci/runners/README.md). GitLab.com runs [Sidekiq](https://sidekiq.org) with arguments `--timeout=4 --concurrency=4` and the following environment variables: -| Setting | GitLab.com | Default | -|-------- |----------- |-------- | -| `SIDEKIQ_DAEMON_MEMORY_KILLER` | - | `1` | -| `SIDEKIQ_MEMORY_KILLER_MAX_RSS` | `2000000` | `2000000` | -| `SIDEKIQ_MEMORY_KILLER_HARD_LIMIT_RSS` | - | - | -| `SIDEKIQ_MEMORY_KILLER_CHECK_INTERVAL` | - | `3` | -| `SIDEKIQ_MEMORY_KILLER_GRACE_TIME` | - | `900` | -| `SIDEKIQ_MEMORY_KILLER_SHUTDOWN_WAIT` | - | `30` | -| `SIDEKIQ_LOG_ARGUMENTS` | `1` | `1` | +| Setting | GitLab.com | Default | +|----------------------------------------|------------|-----------| +| `SIDEKIQ_DAEMON_MEMORY_KILLER` | - | `1` | +| `SIDEKIQ_MEMORY_KILLER_MAX_RSS` | `2000000` | `2000000` | +| `SIDEKIQ_MEMORY_KILLER_HARD_LIMIT_RSS` | - | - | +| `SIDEKIQ_MEMORY_KILLER_CHECK_INTERVAL` | - | `3` | +| `SIDEKIQ_MEMORY_KILLER_GRACE_TIME` | - | `900` | +| `SIDEKIQ_MEMORY_KILLER_SHUTDOWN_WAIT` | - | `30` | +| `SIDEKIQ_LOG_ARGUMENTS` | `1` | `1` | NOTE: The `SIDEKIQ_MEMORY_KILLER_MAX_RSS` setting is `16000000` on Sidekiq import @@ -228,11 +237,8 @@ The list of GitLab.com specific settings (and their defaults) is as follows: | `idle_in_transaction_session_timeout` | 60s | 60s | Some of these settings are in the process being adjusted. For example, the value -for `shared_buffers` is quite high and as such we are looking into adjusting it. -More information on this particular change can be found at -<https://gitlab.com/gitlab-com/infrastructure/-/issues/1555>. An up to date list -of proposed changes can be found at -<https://gitlab.com/gitlab-com/infrastructure/-/issues?scope=all&state=opened&label_name[]=database&label_name[]=change>. +for `shared_buffers` is quite high, and we are +[considering adjusting it](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/4985). ## Puma @@ -282,16 +288,18 @@ paths that exceed 10 requests per **minute** per IP address. See the source below for which paths are protected. This includes user creation, user confirmation, user sign in, and password reset. -[User and IP rate limits](../admin_area/settings/user_and_ip_rate_limits.md#response-headers) includes a list of the headers responded to blocked requests. +[User and IP rate limits](../admin_area/settings/user_and_ip_rate_limits.md#response-headers) +includes a list of the headers responded to blocked requests. See [Protected Paths](../admin_area/settings/protected_paths.md) for more details. ### IP blocks IP blocks can occur when GitLab.com receives unusual traffic from a single -IP address that the system views as potentially malicious, based on rate limit -settings. After the unusual traffic ceases, the IP address is automatically -released depending on the type of block, as described in a following section. +IP address that the system views as potentially malicious. This can be based on +rate limit settings. After the unusual traffic ceases, the IP address is +automatically released depending on the type of block, as described in a +following section. If you receive a `403 Forbidden` error for all requests to GitLab.com, check for any automated processes that may be triggering a block. For @@ -309,8 +317,8 @@ This applies only to Git requests and container registry (`/jwt/auth`) requests This limit: - Is reset by requests that authenticate successfully. For example, 29 - failed authentication requests followed by 1 successful request, followed by 29 - more failed authentication requests would not trigger a ban. + failed authentication requests followed by 1 successful request, followed by + 29 more failed authentication requests would not trigger a ban. - Does not apply to JWT requests authenticated by `gitlab-ci-token`. No response headers are provided. @@ -326,33 +334,42 @@ doesn't return the following headers: ### Visibility settings -On GitLab.com, projects, groups, and snippets created -As of GitLab 12.2 (July 2019), projects, groups, and snippets have the -[**Internal** visibility](../../public_access/public_access.md#internal-projects) setting [disabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/12388). +If created before GitLab 12.2 (July 2019), these items have the +[Internal visibility](../../public_access/public_access.md#internal-projects) +setting [disabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/12388): + +- Projects +- Groups +- Snippets ### SSH maximum number of connections -GitLab.com defines the maximum number of concurrent, unauthenticated SSH connections by -using the [MaxStartups setting](http://man.openbsd.org/sshd_config.5#MaxStartups). -If more than the maximum number of allowed connections occur concurrently, they are -dropped and users get +GitLab.com defines the maximum number of concurrent, unauthenticated SSH +connections by using the [MaxStartups setting](http://man.openbsd.org/sshd_config.5#MaxStartups). +If more than the maximum number of allowed connections occur concurrently, they +are dropped and users get [an `ssh_exchange_identification` error](../../topics/git/troubleshooting_git.md#ssh_exchange_identification-error). ### Import/export -To help avoid abuse, project and group imports, exports, and export downloads are rate limited. See [Project import/export rate limits](../../user/project/settings/import_export.md#rate-limits) and [Group import/export rate limits](../../user/group/settings/import_export.md#rate-limits) for details. +To help avoid abuse, project and group imports, exports, and export downloads +are rate limited. See [Project import/export rate limits](../../user/project/settings/import_export.md#rate-limits) and [Group import/export rate limits](../../user/group/settings/import_export.md#rate-limits) +for details. ### Non-configurable limits -See [non-configurable limits](../../security/rate_limits.md#non-configurable-limits) for information on -rate limits that are not configurable, and therefore also used on GitLab.com. +See [non-configurable limits](../../security/rate_limits.md#non-configurable-limits) +for information on rate limits that are not configurable, and therefore also +used on GitLab.com. ## GitLab.com Logging -We use [Fluentd](https://gitlab.com/gitlab-com/runbooks/tree/master/logging/doc#fluentd) to parse our logs. Fluentd sends our logs to -[Stackdriver Logging](https://gitlab.com/gitlab-com/runbooks/tree/master/logging/doc#stackdriver) and [Cloud Pub/Sub](https://gitlab.com/gitlab-com/runbooks/tree/master/logging/doc#cloud-pubsub). -Stackdriver is used for storing logs long-term in Google Cold Storage (GCS). Cloud Pub/Sub -is used to forward logs to an [Elastic cluster](https://gitlab.com/gitlab-com/runbooks/tree/master/logging/doc#elastic) using [`pubsubbeat`](https://gitlab.com/gitlab-com/runbooks/tree/master/logging/doc#pubsubbeat-vms). +We use [Fluentd](https://gitlab.com/gitlab-com/runbooks/tree/master/logging/doc#fluentd) +to parse our logs. Fluentd sends our logs to +[Stackdriver Logging](https://gitlab.com/gitlab-com/runbooks/tree/master/logging/doc#stackdriver) +and [Cloud Pub/Sub](https://gitlab.com/gitlab-com/runbooks/tree/master/logging/doc#cloud-pubsub). +Stackdriver is used for storing logs long-term in Google Cold Storage (GCS). +Cloud Pub/Sub is used to forward logs to an [Elastic cluster](https://gitlab.com/gitlab-com/runbooks/tree/master/logging/doc#elastic) using [`pubsubbeat`](https://gitlab.com/gitlab-com/runbooks/tree/master/logging/doc#pubsubbeat-vms). You can view more information in our runbooks such as: diff --git a/doc/user/group/epics/epic_boards.md b/doc/user/group/epics/epic_boards.md index 2f9dc27d87f..c31b0c7f78a 100644 --- a/doc/user/group/epics/epic_boards.md +++ b/doc/user/group/epics/epic_boards.md @@ -25,7 +25,7 @@ To view an epic board, in a group, select **Epics > Boards**. Prerequisites: -- A minimum of [Reporter](../../permissions.md#group-members-permissions) access to a group in GitLab. +- You must have at least the [Reporter role](../../permissions.md#group-members-permissions) for a group. To create a new epic board: @@ -49,7 +49,7 @@ To change these options later, [edit the board](#edit-the-scope-of-an-epic-board Prerequisites: -- A minimum of [Reporter](../../permissions.md#group-members-permissions) access to a group in GitLab. +- You must have at least the [Reporter role](../../permissions.md#group-members-permissions) for a group. - A minimum of two boards present in a group. To delete the active epic board: @@ -73,7 +73,7 @@ To delete the active epic board: Prerequisites: -- A minimum of [Reporter](../../permissions.md#group-members-permissions) access to a group in GitLab. +- You must have at least the [Reporter role](../../permissions.md#group-members-permissions) for a group. To create a new list: @@ -90,7 +90,7 @@ list view that's removed. You can always create it again later if you need. Prerequisites: -- A minimum of [Reporter](../../permissions.md#group-members-permissions) access to a group in GitLab. +- You must have at least the [Reporter role](../../permissions.md#group-members-permissions) for a group. To remove a list from an epic board: @@ -120,7 +120,7 @@ You can move epics and lists by dragging them. Prerequisites: -- A minimum of [Reporter](../../permissions.md#group-members-permissions) access to a group in GitLab. +- You must have at least the [Reporter role](../../permissions.md#group-members-permissions) for a group. To move an epic, select the epic card and drag it to another position in its current list or into another list. Learn about possible effects in [Dragging epics between lists](#dragging-epics-between-lists). @@ -143,7 +143,7 @@ and the target list. Prerequisites: -- A minimum of [Reporter](../../permissions.md#group-members-permissions) access to a group in GitLab. +- You must have at least the [Reporter role](../../permissions.md#group-members-permissions) for a group. To edit the scope of an epic board: diff --git a/doc/user/packages/container_registry/index.md b/doc/user/packages/container_registry/index.md index 9d65c5d37ad..d6e86e64e78 100644 --- a/doc/user/packages/container_registry/index.md +++ b/doc/user/packages/container_registry/index.md @@ -488,6 +488,10 @@ Cleanup policies can be run on all projects, with these exceptions: Feature.disable(:container_expiration_policies_historic_entry, Project.find(<project id>)) ``` +WARNING: +For performance reasons, enabled cleanup policies are automatically disabled for projects on +GitLab.com that don't have a container image. + ### How the cleanup policy works The cleanup policy collects all tags in the Container Registry and excludes tags diff --git a/doc/user/project/clusters/index.md b/doc/user/project/clusters/index.md index 97296d22dd9..8dd8ed52dd7 100644 --- a/doc/user/project/clusters/index.md +++ b/doc/user/project/clusters/index.md @@ -31,7 +31,7 @@ Besides integration at the project level, Kubernetes clusters can also be integrated at the [group level](../../group/clusters/index.md) or [GitLab instance level](../../instance/clusters/index.md). -To view your project level Kubernetes clusters, navigate to **Infrastructure > Kubernetes** +To view your project level Kubernetes clusters, navigate to **Infrastructure > Kubernetes clusters** from your project. On this page, you can [add a new cluster](#adding-and-removing-clusters) and view information about your existing clusters, such as: @@ -187,7 +187,7 @@ your cluster. This can cause deployment jobs to fail. To clear the cache: -1. Navigate to your project's **Infrastructure > Kubernetes** page, and select your cluster. +1. Navigate to your project's **Infrastructure > Kubernetes clusters** page, and select your cluster. 1. Expand the **Advanced settings** section. 1. Click **Clear cluster cache**. diff --git a/doc/user/project/description_templates.md b/doc/user/project/description_templates.md index d2897c7310e..711d7f561e4 100644 --- a/doc/user/project/description_templates.md +++ b/doc/user/project/description_templates.md @@ -116,12 +116,13 @@ Only instance administrators can set instance-level templates. To set the instance-level description template repository: -1. Select the **Admin Area** icon (**{admin}**). -1. Go to **Settings > Templates**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > Templates**. +1. Expand **Templates** 1. From the dropdown, select your template project as the template repository at instance level. 1. Select **Save changes**. - + Learn more about [instance template repository](../admin_area/settings/instance_template_repository.md). diff --git a/doc/user/project/merge_requests/approvals/settings.md b/doc/user/project/merge_requests/approvals/settings.md index 97e4b7da396..b72a4125d0e 100644 --- a/doc/user/project/merge_requests/approvals/settings.md +++ b/doc/user/project/merge_requests/approvals/settings.md @@ -34,7 +34,7 @@ on merge requests, you can disable this setting: 1. Select the **Prevent users from modifying MR approval rules in merge requests** checkbox. 1. Select **Save changes**. -TODO This change affects all open merge requests. +This change affects all open merge requests. ## Reset approvals on push diff --git a/doc/user/project/time_tracking.md b/doc/user/project/time_tracking.md index 3c9b0341661..b7fd14ae74b 100644 --- a/doc/user/project/time_tracking.md +++ b/doc/user/project/time_tracking.md @@ -109,8 +109,15 @@ Default conversion rates are 1mo = 4w, 1w = 5d and 1d = 8h. > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/29469/) in GitLab 12.1. -In GitLab self-managed instances, the display of time units can be limited to -hours through the option in **Admin Area > Settings > Preferences** under **Localization**. +In GitLab self-managed instances, you can limit the display of time units to +hours. +To do so: + +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > Preferences**. +1. Expand **Localization**. +1. Under **Time tracking**, select the **Limit display of time tracking units to hours** checkbox. +1. Select **Save changes**. With this option enabled, `75h` is displayed instead of `1w 4d 3h`. diff --git a/doc/user/shortcuts.md b/doc/user/shortcuts.md index 6abbb128f49..37e89dd54db 100644 --- a/doc/user/shortcuts.md +++ b/doc/user/shortcuts.md @@ -81,7 +81,7 @@ relatively quickly to work, and they take you to another page in the project. | <kbd>g</kbd> + <kbd>j</kbd> | Go to the CI/CD jobs list (**CI/CD > Jobs**). | | <kbd>g</kbd> + <kbd>l</kbd> | Go to the project metrics (**Monitor > Metrics**). | | <kbd>g</kbd> + <kbd>e</kbd> | Go to the project environments (**Deployments > Environments**). | -| <kbd>g</kbd> + <kbd>k</kbd> | Go to the project Kubernetes cluster integration page (**Infrastructure > Kubernetes**). Note that you must have at least [`maintainer` permissions](permissions.md) to access this page. | +| <kbd>g</kbd> + <kbd>k</kbd> | Go to the project Kubernetes cluster integration page (**Infrastructure > Kubernetes clusters**). Note that you must have at least [`maintainer` permissions](permissions.md) to access this page. | | <kbd>g</kbd> + <kbd>s</kbd> | Go to the project snippets list (**Snippets**). | | <kbd>g</kbd> + <kbd>w</kbd> | Go to the project wiki (**Wiki**), if enabled. | |
