diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2022-03-18 20:02:30 +0000 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2022-03-18 20:02:30 +0000 |
| commit | 41fe97390ceddf945f3d967b8fdb3de4c66b7dea (patch) | |
| tree | 9c8d89a8624828992f06d892cd2f43818ff5dcc8 /doc/administration | |
| parent | 0804d2dc31052fb45a1efecedc8e06ce9bc32862 (diff) | |
| download | gitlab-ce-41fe97390ceddf945f3d967b8fdb3de4c66b7dea.tar.gz | |
Add latest changes from gitlab-org/gitlab@14-9-stable-eev14.9.0-rc42
Diffstat (limited to 'doc/administration')
70 files changed, 1113 insertions, 365 deletions
diff --git a/doc/administration/audit_event_streaming.md b/doc/administration/audit_event_streaming.md index 3bdc67e5a69..ca59dff7ef4 100644 --- a/doc/administration/audit_event_streaming.md +++ b/doc/administration/audit_event_streaming.md @@ -25,6 +25,24 @@ GitLab can stream a single event more than once to the same destination. Use the WARNING: Event streaming destinations will receive **all** audit event data, which could include sensitive information. Make sure you trust the destination endpoint. +### Add event streaming destination using GitLab UI + +Users with at least the Owner role of a group can add event streaming destinations for it: + +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Security & Compliance > Audit events** +1. On the main area, select **Streams** tab. + - When the destination list is empty, select **Add stream** activate edit mode and add a new destination. + - When the destination list is not empty, select **{plus}** under the **Streams** tab to activate edit mode. +1. Enter the endpoint you wish to add and select **Add**. + +Event streaming is enabled if: + +- No warning is shown. +- The added endpoint is displayed in the UI. + +### Add event streaming destination using the API + To enable event streaming, a group owner must add a new event streaming destination using the `externalAuditEventDestinationCreate` mutation in the GraphQL API. @@ -34,8 +52,8 @@ mutation { errors externalAuditEventDestination { destinationUrl - group { verificationToken + group { name } } @@ -50,7 +68,17 @@ Event streaming is enabled if: ## List currently enabled streaming destinations -Group owners can view a list of event streaming destinations at any time using the `externalAuditEventDesinations` query type. +### List currently enabled streaming destination using GitLab UI + +Users with at least the Owner role of a group can list event streaming destinations: + +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Security & Compliance > Audit events** +1. On the main area, select **Streams** tab. + +### List currently enabled streaming destinations using the API + +Group owners can view a list of event streaming destinations at any time using the `externalAuditEventDestinations` query type. ```graphql query { @@ -69,6 +97,45 @@ query { If the resulting list is empty, then audit event streaming is not enabled for that group. +## Delete currently enabled streaming destination + +Group Owners can delete event streaming destinations at any time using the `deleteAuditEventDestinations` mutation type. + +### Delete currently enabled streaming using GitLab UI + +Uses with at least the Owner role of a group can delete event streaming destination. + +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Security & Compliance > Audit events** +1. On the main area, select **Streams** tab. +1. Select **{remove}** at the right side of each item. + +The external streaming destination is delete when: + +- No warning is shown. +- The deleted endpoint is not displayed in the UI. + +### Delete currently enabled streaming using the API + +You can delete an event streaming destination by specifying an ID. Get the required ID by [listing the details](audit_event_streaming.md#list-currently-enabled-streaming-destinations-using-the-api) of event streaming destinations. + +```graphql + +mutation{ + externalAuditEventDestinationDestroy(input: { id: destination }) { + errors + } +} + +``` + +Destination is deleted if: + +- The returned `errors` object is empty. +- The API responds with `200 OK`. + +When the last destination is successfully deleted, event streaming is disabled for the group. + ## Verify event authenticity > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345424) in GitLab 14.8. @@ -78,3 +145,228 @@ token is generated when the event destination is created and cannot be changed. Each streamed event contains a random alphanumeric identifier for the `X-Gitlab-Event-Streaming-Token` HTTP header that can be verified against the destination's value when [listing streaming destinations](#list-currently-enabled-streaming-destinations). + +## Audit event streaming on Git operations + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332747) in GitLab 14.9 [with a flag](../administration/feature_flags.md) named `audit_event_streaming_git_operations`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](feature_flags.md) named `audit_event_streaming_git_operations`. On GitLab.com, this feature is not available. + +Streaming audit events can be sent when signed-in users push or pull a project's remote Git repositories: + +- [Using SSH](../ssh/index.md). +- Using HTTP or HTTPS. +- Using the **Download** button (**{download}**) in GitLab UI. + +Audit events are not captured for users that are not signed in. For example, when downloading a public project. + +To configure streaming audit events for Git operations, see [Add a new event streaming destination](#add-a-new-event-streaming-destination). + +### Request headers + +Request headers are formatted as follows: + +```plaintext +POST /logs HTTP/1.1 +Host: <DESTINATION_HOST> +Content-Type: application/x-www-form-urlencoded +X-Gitlab-Event-Streaming-Token: <DESTINATION_TOKEN> +``` + +### Example responses for SSH events + +Fetch: + +```json +{ + "id": 1, + "author_id": 1, + "entity_id": 29, + "entity_type": "Project", + "details": { + "author_name": "Administrator", + "target_id": 29, + "target_type": "Project", + "target_details": "example-project", + "custom_message": { + "protocol": "ssh", + "action": "git-upload-pack" + }, + "ip_address": "127.0.0.1", + "entity_path": "example-group/example-project" + }, + "ip_address": "127.0.0.1", + "author_name": "Administrator", + "entity_path": "example-group/example-project", + "target_details": "example-project", + "created_at": "2022-02-23T06:21:05.283Z", + "target_type": "Project", + "target_id": 29 +} +``` + +Push: + +```json +{ + "id": 1, + "author_id": 1, + "entity_id": 29, + "entity_type": "Project", + "details": { + "author_name": "Administrator", + "target_id": 29, + "target_type": "Project", + "target_details": "example-project", + "custom_message": { + "protocol": "ssh", + "action": "git-receive-pack" + }, + "ip_address": "127.0.0.1", + "entity_path": "example-group/example-project" + }, + "ip_address": "127.0.0.1", + "author_name": "Administrator", + "entity_path": "example-group/example-project", + "target_details": "example-project", + "created_at": "2022-02-23T06:23:08.746Z", + "target_type": "Project", + "target_id": 29 +} +``` + +### Example responses for HTTP and HTTPS events + +Fetch: + +```json +{ + "id": 1, + "author_id": 1, + "entity_id": 29, + "entity_type": "Project", + "details": { + "author_name": "Administrator", + "target_id": 29, + "target_type": "Project", + "target_details": "example-project", + "custom_message": { + "protocol": "http", + "action": "git-upload-pack" + }, + "ip_address": "127.0.0.1", + "entity_path": "example-group/example-project" + }, + "ip_address": "127.0.0.1", + "author_name": "Administrator", + "entity_path": "example-group/example-project", + "target_details": "example-project", + "created_at": "2022-02-23T06:25:43.938Z", + "target_type": "Project", + "target_id": 29 +} +``` + +Push: + +```json +{ + "id": 1, + "author_id": 1, + "entity_id": 29, + "entity_type": "Project", + "details": { + "author_name": "Administrator", + "target_id": 29, + "target_type": "Project", + "target_details": "example-project", + "custom_message": { + "protocol": "http", + "action": "git-receive-pack" + }, + "ip_address": "127.0.0.1", + "entity_path": "example-group/example-project" + }, + "ip_address": "127.0.0.1", + "author_name": "Administrator", + "entity_path": "example-group/example-project", + "target_details": "example-project", + "created_at": "2022-02-23T06:26:29.294Z", + "target_type": "Project", + "target_id": 29 +} +``` + +### Example responses for events from GitLab UI download button + +Fetch: + +```json +{ + "id": 1, + "author_id": 99, + "entity_id": 29, + "entity_type": "Project", + "details": { + "custom_message": "Repository Download Started", + "author_name": "example_username", + "target_id": 29, + "target_type": "Project", + "target_details": "example-group/example-project", + "ip_address": "127.0.0.1", + "entity_path": "example-group/example-project" + }, + "ip_address": "127.0.0.1", + "author_name": "example_username", + "entity_path": "example-group/example-project", + "target_details": "example-group/example-project", + "created_at": "2022-02-23T06:27:17.873Z", + "target_type": "Project", + "target_id": 29 +} +``` + +## Audit event streaming on merge request approval actions + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/271162) in GitLab 14.9. + +Stream audit events that relate to merge approval actions performed within a project. + +### Request headers + +Request headers are formatted as follows: + +```plaintext +POST /logs HTTP/1.1 +Host: <DESTINATION_HOST> +Content-Type: application/x-www-form-urlencoded +X-Gitlab-Event-Streaming-Token: <DESTINATION_TOKEN> +``` + +### Example request body + +```json +{ + "id": 1, + "author_id": 1, + "entity_id": 6, + "entity_type": "Project", + "details": { + "author_name": "example_username", + "target_id": 20, + "target_type": "MergeRequest", + "target_details": "merge request title", + "custom_message": "Approved merge request", + "ip_address": "127.0.0.1", + "entity_path": "example-group/example-project" + }, + "ip_address": "127.0.0.1", + "author_name": "example_username", + "entity_path": "example-group/example-project", + "target_details": "merge request title", + "created_at": "2022-03-09T06:53:11.181Z", + "target_type": "MergeRequest", + "target_id": 20 +} +``` diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md index d4902a18cac..955f7a6a830 100644 --- a/doc/administration/audit_events.md +++ b/doc/administration/audit_events.md @@ -44,17 +44,22 @@ There are two kinds of events logged: - Instance events scoped to the whole GitLab instance, used by your Compliance team to perform formal audits. +NOTE: +Some events are recorded and available only as [streaming audit events](audit_event_streaming.md). + ### Impersonation data > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/536) in GitLab 13.0. When a user is being [impersonated](../user/admin_area/index.md#user-impersonation), their actions are logged as audit events as usual, with two additional details: -1. Usual audit events include information about the impersonating administrator. These are visible in their respective Audit Event pages depending on their type (Group/Project/User). -1. Extra audit events are recorded for the start and stop of the administrator's impersonation session. These are visible in - the: +1. Usual audit events include information about the impersonating administrator. These audit events are visible in their + respective audit event pages depending on their type (group, project, or user). +1. Extra audit events are recorded for the start and stop of the administrator's impersonation session. These audit events + are visible in the: - Instance audit events. - - Group audit events for all groups the user belongs to (GitLab 14.8 and later). This is limited to 20 groups for performance reasons. + - Group audit events for all groups the user belongs to (GitLab 14.8 and later). For performance reasons, group audit + events are limited to the oldest 20 groups to which you belong.  @@ -107,6 +112,8 @@ From there, you can see the following actions: - Compliance framework created, updated, or deleted. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/340649) in GitLab 14.5. - Event streaming destination created, updated, or deleted. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/344664) in GitLab 14.6. - Instance administrator started or stopped impersonation of a group member. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/300961) in GitLab 14.8. +- Group deploy token was successfully created, revoked, or deleted. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/353452) in GitLab 14.9. +- Failed attempt to create a group deploy token. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/353452) in GitLab 14.9. Group events can also be accessed via the [Group Audit Events API](../api/audit_events.md#group-audit-events) @@ -158,6 +165,8 @@ From there, you can see the following actions: - Allowing force push to protected branch changed ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/338873) in GitLab 14.3) - Code owner approval requirement on merge requests targeting protected branch changed ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/338873) in GitLab 14.3) - Users and groups allowed to merge and push to protected branch added or removed ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/338873) in GitLab 14.3) +- Project deploy token was successfully created, revoked or deleted ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/353451) in GitLab 14.9) +- Failed attempt to create a project deploy token ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/353451) in GitLab 14.9) Project events can also be accessed via the [Project Audit Events API](../api/audit_events.md#project-audit-events). diff --git a/doc/administration/auditor_users.md b/doc/administration/auditor_users.md index 12e222290e7..7c89f34ce61 100644 --- a/doc/administration/auditor_users.md +++ b/doc/administration/auditor_users.md @@ -1,6 +1,6 @@ --- -stage: Enablement -group: Distribution +stage: Manage +group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/auth/atlassian.md b/doc/administration/auth/atlassian.md index 7c6aee2c2da..6be53922a5a 100644 --- a/doc/administration/auth/atlassian.md +++ b/doc/administration/auth/atlassian.md @@ -13,14 +13,14 @@ To enable the Atlassian OmniAuth provider for passwordless authentication you mu 1. Go to <https://developer.atlassian.com/console/myapps/> and sign-in with the Atlassian account to administer the application. -1. Click **Create a new app**. -1. Choose an App Name, such as 'GitLab', and click **Create**. +1. Select **Create a new app**. +1. Choose an App Name, such as 'GitLab', and select **Create**. 1. Note the `Client ID` and `Secret` for the [GitLab configuration](#gitlab-configuration) steps. -1. On the left sidebar under **APIS AND FEATURES**, click **OAuth 2.0 (3LO)**. -1. Enter the GitLab callback URL using the format `https://gitlab.example.com/users/auth/atlassian_oauth2/callback` and click **Save changes**. -1. Click **+ Add** in the left sidebar under **APIS AND FEATURES**. -1. Click **Add** for **Jira platform REST API** and then **Configure**. -1. Click **Add** next to the following scopes: +1. On the left sidebar under **APIS AND FEATURES**, select **OAuth 2.0 (3LO)**. +1. Enter the GitLab callback URL using the format `https://gitlab.example.com/users/auth/atlassian_oauth2/callback` and select **Save changes**. +1. Select **+ Add** in the left sidebar under **APIS AND FEATURES**. +1. Select **Add** for **Jira platform REST API** and then **Configure**. +1. Select **Add** next to the following scopes: - **View Jira issue data** - **View user profiles** - **Create and manage issues** @@ -73,6 +73,6 @@ To enable the Atlassian OmniAuth provider for passwordless authentication you mu 1. Save the configuration file. 1. [Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) or [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect if you installed GitLab via Omnibus or from source respectively. -On the sign-in page there should now be an Atlassian icon below the regular sign in form. Click the icon to begin the authentication process. +On the sign-in page there should now be an Atlassian icon below the regular sign in form. Select the icon to begin the authentication process. If everything goes right, the user is signed in to GitLab using their Atlassian credentials. diff --git a/doc/administration/auth/ldap/ldap-troubleshooting.md b/doc/administration/auth/ldap/ldap-troubleshooting.md index 06fe579e101..b8391bec72f 100644 --- a/doc/administration/auth/ldap/ldap-troubleshooting.md +++ b/doc/administration/auth/ldap/ldap-troubleshooting.md @@ -89,14 +89,14 @@ established but GitLab doesn't show you LDAP users in the output, one of the following is most likely true: - The `bind_dn` user doesn't have enough permissions to traverse the user tree. -- The user(s) don't fall under the [configured `base`](index.md#configure-ldap). -- The [configured `user_filter`](index.md#set-up-ldap-user-filter) blocks access to the user(s). +- The users don't fall under the [configured `base`](index.md#configure-ldap). +- The [configured `user_filter`](index.md#set-up-ldap-user-filter) blocks access to the users. In this case, you con confirm which of the above is true using [ldapsearch](#ldapsearch) with the existing LDAP configuration in your `/etc/gitlab/gitlab.rb`. -#### User(s) cannot sign-in +#### Users cannot sign-in A user can have trouble signing in for any number of reasons. To get started, here are some questions to ask yourself: @@ -284,7 +284,7 @@ If you don't find a particular user's GitLab email in the output, then that user hasn't signed in with LDAP yet. Next, GitLab searches its `identities` table for the existing -link between this user and the configured LDAP provider(s): +link between this user and the configured LDAP providers: ```sql Identity Load (0.9ms) SELECT "identities".* FROM "identities" WHERE "identities"."user_id" = 20 AND (provider LIKE 'ldap%') LIMIT 1 @@ -334,7 +334,7 @@ Gitlab::Auth::Ldap::Person.find_by_uid('<uid>', adapter) ### Group memberships **(PREMIUM SELF)** -#### Membership(s) not granted +#### Memberships not granted Sometimes you may think a particular user should be added to a GitLab group via LDAP group sync, but for some reason it's not happening. You can check several @@ -348,7 +348,7 @@ things to debug the situation. 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Overview > Users**. 1. Search for the user. - 1. Open the user by clicking their name. Do not click **Edit**. + 1. Open the user by selecting their name. Do not select **Edit**. 1. Select the **Identities** tab. There should be an LDAP identity with an LDAP DN as the 'Identifier'. If not, this user hasn't signed in with LDAP yet and must do so first. @@ -558,7 +558,7 @@ aren't blocked or unable to access their accounts. NOTE: The following script requires that any new accounts with the new -email address are removed first. This is because emails have to be unique in GitLab. +email address are removed first. Email addresses must be unique in GitLab. Go to the [rails console](#rails-console) and then run: diff --git a/doc/administration/cicd.md b/doc/administration/cicd.md index a7bd07d5d38..5e061320c5b 100644 --- a/doc/administration/cicd.md +++ b/doc/administration/cicd.md @@ -71,6 +71,27 @@ Plan.default.actual_limits.update!(ci_needs_size_limit: 100) To disable directed acyclic graphs (DAG), set the limit to `0`. +## Change maximum scheduled pipeline frequency + +[Scheduled pipelines](../ci/pipelines/schedules.md) can be configured with any [cron value](../topics/cron/index.md), +but they do not always run exactly when scheduled. An internal process, called the +_pipeline schedule worker_, queues all the scheduled pipelines, but does not +run continuously. The worker runs on its own schedule, and scheduled pipelines that +are ready to start are only queued the next time the worker runs. Scheduled pipelines +can't run more frequently than the worker. + +The default frequency of the pipeline schedule worker is `3-59/10 * * * *` (every ten minutes, +starting with `0:03`, `0:13`, `0:23`, and so on). The default frequency for GitLab.com +is listed in the [GitLab.com settings](../user/gitlab_com/index.md#gitlab-cicd). + +To change the frequency of the pipeline schedule worker: + +1. Edit the `gitlab_rails['pipeline_schedule_worker_cron']` value in your instance's `gitlab.rb` file. +1. [Reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +For example, to set the maximum frequency of pipelines to twice a day, set `pipeline_schedule_worker_cron` +to a cron value of `0 */12 * * *` (`00:00` and `12:00` every day). + <!-- ## Troubleshooting Include any troubleshooting steps that you can foresee. If you know beforehand what issues diff --git a/doc/administration/clusters/kas.md b/doc/administration/clusters/kas.md index 192b636e246..e5c371b9d40 100644 --- a/doc/administration/clusters/kas.md +++ b/doc/administration/clusters/kas.md @@ -4,34 +4,33 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Install the GitLab Agent Server for Kubernetes (KAS) **(FREE SELF)** +# Install the GitLab agent server for Kubernetes (KAS) **(FREE SELF)** -> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3834) in GitLab 13.10, the GitLab Agent Server (KAS) became available on GitLab.com under `wss://kas.gitlab.com`. +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3834) in GitLab 13.10, the GitLab agent server (KAS) became available on GitLab.com at `wss://kas.gitlab.com`. > - [Moved](https://gitlab.com/groups/gitlab-org/-/epics/6290) from GitLab Premium to GitLab Free in 14.5. -The GitLab Agent Server for Kubernetes is a GitLab backend service dedicated to -managing the [GitLab Agent for Kubernetes](../../user/clusters/agent/index.md). +The agent server is a component you install together with GitLab. It is required to +manage the [GitLab agent for Kubernetes](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent). -The KAS acronym refers to the former name, Kubernetes Agent Server. +The KAS acronym refers to the former name, `Kubernetes agent server`. -The KAS is already installed and available in GitLab.com under `wss://kas.gitlab.com`. -This document describes how to install a KAS for GitLab self-managed instances. +The agent server for Kubernetes is installed and available on GitLab.com at `wss://kas.gitlab.com`. +If you use self-managed GitLab, you must install an agent server or specify an external installation. ## Installation options -As a GitLab administrator of self-managed instances, you can install KAS according to your GitLab -installation method: +As a GitLab administrator, you can install the agent server: -- For [Omnibus installations](#install-kas-with-omnibus). -- For [GitLab Helm Chart installations](#install-kas-with-the-gitlab-helm-chart). +- For [Omnibus installations](#for-omnibus). +- For [GitLab Helm Chart installations](#for-gitlab-helm-chart). -You can also opt to use an [external KAS](#use-an-external-kas-installation). +Or, you can [use an external agent server](#use-an-external-installation). -### Install KAS with Omnibus +### For Omnibus For [Omnibus](https://docs.gitlab.com/omnibus/) package installations: -1. Edit `/etc/gitlab/gitlab.rb` to enable the Agent Server: +1. To enable the agent server, edit `/etc/gitlab/gitlab.rb`: ```ruby gitlab_kas['enable'] = true @@ -39,49 +38,49 @@ For [Omnibus](https://docs.gitlab.com/omnibus/) package installations: 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -To configure any additional options related to your KAS, -refer to the **Enable GitLab KAS** section of the +For additional configuration options, see the **Enable GitLab KAS** section of the [`gitlab.rb.template`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/files/gitlab-config-template/gitlab.rb.template). -### Install KAS with the GitLab Helm Chart +### For GitLab Helm Chart -For GitLab [Helm Chart](https://docs.gitlab.com/charts/) -installations, you must set `global.kas.enabled` to `true`. -For example, in a shell with `helm` and `kubectl` -installed, run: +For GitLab [Helm Chart](https://docs.gitlab.com/charts/) installations: -```shell -helm repo add gitlab https://charts.gitlab.io/ -helm repo update -helm upgrade --install gitlab gitlab/gitlab \ - --timeout 600s \ - --set global.hosts.domain=<YOUR_DOMAIN> \ - --set global.hosts.externalIP=<YOUR_IP> \ - --set certmanager-issuer.email=<YOUR_EMAIL> \ - --set global.kas.enabled=true # <-- without this, KAS will not be installed -``` +1. Set `global.kas.enabled` to `true`. For example, in a shell with `helm` and `kubectl` + installed, run: -To configure KAS, use a `gitlab.kas` sub-section in your `values.yaml` file: + ```shell + helm repo add gitlab https://charts.gitlab.io/ + helm repo update + helm upgrade --install gitlab gitlab/gitlab \ + --timeout 600s \ + --set global.hosts.domain=<YOUR_DOMAIN> \ + --set global.hosts.externalIP=<YOUR_IP> \ + --set certmanager-issuer.email=<YOUR_EMAIL> \ + --set global.kas.enabled=true # <-- without this setting, the agent server will not be installed + ``` -```yaml -gitlab: - kas: - # put your KAS custom options here -``` +1. To configure the agent server, use a `gitlab.kas` sub-section in your `values.yaml` file: + + ```yaml + gitlab: + kas: + # put your custom options here + ``` For details, see [how to use the GitLab-KAS chart](https://docs.gitlab.com/charts/charts/gitlab/kas/). -### Use an external KAS installation +### Use an external installation > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/299850) in GitLab 13.10. -Besides installing KAS with GitLab, you can opt to configure GitLab to use an external KAS. +Instead of installing the agent server, you can configure GitLab to use an external agent server. -For GitLab instances installed through the GitLab Helm Chart, see [how to configure your external KAS](https://docs.gitlab.com/charts/charts/globals.html#external-kas). +If you used the GitLab Helm Chart to install GitLab, see +[how to configure your external agent server](https://docs.gitlab.com/charts/charts/globals.html#external-kas). -For GitLab instances installed through Omnibus packages: +If you used the Omnibus packages: -1. Edit `/etc/gitlab/gitlab.rb` adding the paths to your external KAS: +1. Edit `/etc/gitlab/gitlab.rb` and add the paths to your external agent server: ```ruby gitlab_kas['enable'] = false @@ -96,7 +95,7 @@ For GitLab instances installed through Omnibus packages: ## Troubleshooting -If you have issues while using the GitLab Agent Server for Kubernetes, view the +If you have issues while using the agent server for Kubernetes, view the service logs by running the following command: ```shell @@ -105,9 +104,9 @@ kubectl logs -f -l=app=kas -n <YOUR-GITLAB-NAMESPACE> In Omnibus GitLab, find the logs in `/var/log/gitlab/gitlab-kas/`. -You can also [troubleshoot issues with individual Agents](../../user/clusters/agent/troubleshooting.md). +You can also [troubleshoot issues with individual agents](../../user/clusters/agent/troubleshooting.md). -### KAS logs - GitOps: failed to get project information +### GitOps: failed to get project information If you get the following error message: @@ -115,11 +114,11 @@ If you get the following error message: {"level":"warn","time":"2020-10-30T08:37:26.123Z","msg":"GitOps: failed to get project info","agent_id":4,"project_id":"root/kas-manifest001","error":"error kind: 0; status: 404"} ``` -It means that the specified manifest project `root/kas-manifest001` -doesn't exist or the manifest project is private. To fix it, make sure the project path is correct -and its visibility is [set to public](../../public_access/public_access.md). +The project specified by the manifest (`root/kas-manifest001`) +doesn't exist or the project where the manifest is kept is private. To fix this issue, +ensure the project path is correct and that the project's visibility is [set to public](../../public_access/public_access.md). -### KAS logs - Configuration file not found +### Configuration file not found If you get the following error message: @@ -127,29 +126,29 @@ If you get the following error message: time="2020-10-29T04:44:14Z" level=warning msg="Config: failed to fetch" agent_id=2 error="configuration file not found: \".gitlab/agents/test-agent/config.yaml\ ``` -It means that the path to the configuration project is incorrect, -or the path to `config.yaml` inside the project is not valid. +The path is incorrect for either: + +- The repository where the agent was registered. +- The agent configuration file. -To fix this, ensure that the paths to the configuration repository and to the `config.yaml` file -are correct. +To fix this issue, ensure that the paths are correct. -### KAS logs - `dial tcp <GITLAB_INTERNAL_IP>:443: connect: connection refused` +### `dial tcp <GITLAB_INTERNAL_IP>:443: connect: connection refused` -If you are running a self-managed GitLab instance and: +If you are running self-managed GitLab and: - The instance isn't running behind an SSL-terminating proxy. - The instance doesn't have HTTPS configured on the GitLab instance itself. - The instance's hostname resolves locally to its internal IP address. -You may see the following error when the KAS tries to connect to the GitLab API: +When the agent server tries to connect to the GitLab API, the following error might occur: ```json {"level":"error","time":"2021-08-16T14:56:47.289Z","msg":"GetAgentInfo()","correlation_id":"01FD7QE35RXXXX8R47WZFBAXTN","grpc_service":"gitlab.agent.reverse_tunnel.rpc.ReverseTunnel","grpc_method":"Connect","error":"Get \"https://gitlab.example.com/api/v4/internal/kubernetes/agent_info\": dial tcp 172.17.0.4:443: connect: connection refused"} ``` -To fix this for [Omnibus](https://docs.gitlab.com/omnibus/) package installations, -set the following parameter in `/etc/gitlab/gitlab.rb` -(replacing `gitlab.example.com` with your GitLab instance's hostname): +To fix this issue for [Omnibus](https://docs.gitlab.com/omnibus/) package installations, +set the following parameter in `/etc/gitlab/gitlab.rb`. Replace `gitlab.example.com` with your GitLab instance's hostname: ```ruby gitlab_kas['gitlab_address'] = 'http://gitlab.example.com' diff --git a/doc/administration/database_load_balancing.md b/doc/administration/database_load_balancing.md deleted file mode 100644 index 92b8342f251..00000000000 --- a/doc/administration/database_load_balancing.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'postgresql/database_load_balancing.md' -remove_date: '2022-02-19' ---- - -This file was moved to [another location](postgresql/database_load_balancing.md). - -<!-- This redirect file can be deleted after <2022-02-19>. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/docs_self_host.md b/doc/administration/docs_self_host.md index 3a4511e5aa4..8464c35c3bb 100644 --- a/doc/administration/docs_self_host.md +++ b/doc/administration/docs_self_host.md @@ -18,7 +18,7 @@ To host the GitLab product documentation, you can use: - Your own web server After you create a website by using one of these methods, you redirect the UI links -in the product to point to your website. +in the product to point to your website. NOTE: The website you create must be hosted under a subdirectory that matches diff --git a/doc/administration/encrypted_configuration.md b/doc/administration/encrypted_configuration.md index 9224def4a5a..baa6e967507 100644 --- a/doc/administration/encrypted_configuration.md +++ b/doc/administration/encrypted_configuration.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Distribution -info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers" +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" type: reference --- diff --git a/doc/administration/feature_flags.md b/doc/administration/feature_flags.md index 85a7304b5e8..aebd6eb1d4c 100644 --- a/doc/administration/feature_flags.md +++ b/doc/administration/feature_flags.md @@ -26,7 +26,7 @@ Features behind flags can be gradually rolled out, typically: 1. The feature becomes enabled by default. 1. The feature flag is removed. -These features can be enabled and disabled to allow or disallow users to use +These features can be enabled and disabled to allow or prevent users from using them. It can be done by GitLab administrators with access to GitLab Rails console. diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index f7367ef863b..1725c283396 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -10,7 +10,7 @@ Geo replicates your database, your Git repositories, and few other assets, but there are some [limitations](../index.md#limitations). WARNING: -Disaster recovery for multi-secondary configurations is in **Alpha**. +Disaster recovery for multi-secondary configurations is in [**Alpha**](../../../policy/alpha-beta-support.md#alpha-features). For the latest updates, check the [Disaster Recovery epic for complete maturity](https://gitlab.com/groups/gitlab-org/-/epics/3574). Multi-secondary configurations require the complete re-synchronization and re-configuration of all non-promoted secondaries and causes downtime. diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md index b207be47aa1..9b3e0ecb427 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md @@ -6,7 +6,7 @@ type: howto --- WARNING: -This runbook is in **alpha**. For complete, production-ready documentation, see the +This runbook is in [**Alpha**](../../../../policy/alpha-beta-support.md#alpha-features). For complete, production-ready documentation, see the [disaster recovery documentation](../index.md). # Disaster Recovery (Geo) promotion runbooks **(PREMIUM SELF)** diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md index 5a6f9eb8be7..fa30b11fe32 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md @@ -6,7 +6,7 @@ type: howto --- WARNING: -This runbook is in **alpha**. For complete, production-ready documentation, see the +This runbook is in [**Alpha**](../../../../policy/alpha-beta-support.md#alpha-features). For complete, production-ready documentation, see the [disaster recovery documentation](../index.md). # Disaster Recovery (Geo) promotion runbooks **(PREMIUM SELF)** diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md index c4164284e97..1b80e91c9c4 100644 --- a/doc/administration/geo/index.md +++ b/doc/administration/geo/index.md @@ -23,7 +23,7 @@ to clone and fetch large repositories, speeding up development. For a video introduction to Geo, see [Introduction to GitLab Geo - GitLab Features](https://www.youtube.com/watch?v=-HDLxSjEh6w). -To make sure you're using the right version of the documentation, navigate to [the Geo page on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/administration/geo/index.md) and choose the appropriate release from the **Switch branch/tag** dropdown. For example, [`v13.7.6-ee`](https://gitlab.com/gitlab-org/gitlab/-/blob/v13.7.6-ee/doc/administration/geo/index.md). +To make sure you're using the right version of the documentation, go to [the Geo page on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/administration/geo/index.md) and choose the appropriate release from the **Switch branch/tag** dropdown list. For example, [`v13.7.6-ee`](https://gitlab.com/gitlab-org/gitlab/-/blob/v13.7.6-ee/doc/administration/geo/index.md). Geo uses a set of defined terms that are described in the [Geo Glossary](glossary.md). Be sure to familiarize yourself with those terms. @@ -148,14 +148,14 @@ NOTE: When using HTTP or HTTPS proxying, your load balancer must be configured to pass through the `Connection` and `Upgrade` hop-by-hop headers. See the [web terminal](../integration/terminal.md) integration guide for more details. NOTE: -When using HTTPS protocol for port 443, you need to add an SSL certificate to the load balancers. +When using HTTPS protocol for port 443, you must add an SSL certificate to the load balancers. If you wish to terminate SSL at the GitLab application server instead, use TCP protocol. #### Internal URL HTTP requests from any Geo secondary site to the primary Geo site use the Internal URL of the primary Geo site. If this is not explicitly defined in the primary Geo site settings in the Admin Area, the -public URL of the primary site will be used. +public URL of the primary site is used. To update the internal URL of the primary Geo site: @@ -187,7 +187,7 @@ Because the replicated database instance is read-only, we need this additional d This daemon: - Reads a log of events replicated by the **primary** site to the **secondary** database instance. -- Updates the Geo Tracking Database instance with changes that need to be executed. +- Updates the Geo Tracking Database instance with changes that must be executed. When something is marked to be updated in the tracking database instance, asynchronous jobs running on the **secondary** site execute the required operations and update the state. @@ -198,9 +198,9 @@ This new architecture allows GitLab to be resilient to connectivity issues betwe WARNING: This list of limitations only reflects the latest version of GitLab. If you are using an older version, extra limitations may be in place. -- Pushing directly to a **secondary** site redirects (for HTTP) or proxies (for SSH) the request to the **primary** site instead of [handling it directly](https://gitlab.com/gitlab-org/gitlab/-/issues/1381), except when using Git over HTTP with credentials embedded within the URI. For example, `https://user:password@secondary.tld`. +- Pushing directly to a **secondary** site redirects (for HTTP) or proxies (for SSH) the request to the **primary** site instead of [handling it directly](https://gitlab.com/gitlab-org/gitlab/-/issues/1381), except when using Git over HTTP with credentials embedded in the URI. For example, `https://user:password@secondary.tld`. - The **primary** site has to be online for OAuth login to happen. Existing sessions and Git are not affected. Support for the **secondary** site to use an OAuth provider independent from the primary is [being planned](https://gitlab.com/gitlab-org/gitlab/-/issues/208465). -- The installation takes multiple manual steps that together can take about an hour depending on circumstances. We are working on improving this experience. See [Omnibus GitLab issue #2978](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/2978) for details. +- The installation takes multiple manual steps that together can take about an hour depending on circumstances. Consider using [the GitLab Environment Toolkit](https://gitlab.com/gitlab-org/gitlab-environment-toolkit) to deploy and operate production GitLab instances based on our [Reference Architectures](../reference_architectures/index.md), including automation of common daily tasks. We are planning to [improve Geo's installation even further](https://gitlab.com/groups/gitlab-org/-/epics/1465). - Real-time updates of issues/merge requests (for example, via long polling) doesn't work on the **secondary** site. - GitLab Runners cannot register with a **secondary** site. Support for this is [planned for the future](https://gitlab.com/gitlab-org/gitlab/-/issues/3294). - [Selective synchronization](replication/configuration.md#selective-synchronization) only limits what repositories and files are replicated. The entire PostgreSQL data is still replicated. Selective synchronization is not built to accommodate compliance / export control use cases. @@ -210,13 +210,25 @@ This list of limitations only reflects the latest version of GitLab. If you are There is a complete list of all GitLab [data types](replication/datatypes.md) and [existing support for replication and verification](replication/datatypes.md#limitations-on-replicationverification). +### View replication data on the primary site + +If you try to view replication data on the primary site, you receive a warning that this may be inconsistent: + +> Viewing projects and designs data from a primary site is not possible when using a unified URL. Visit the secondary site directly. + +The only way to view projects replication data for a particular secondary site is to visit that secondary site directly. For example, `https://<IP of your secondary site>/admin/geo/replication/projects`. +An [epic exists](https://gitlab.com/groups/gitlab-org/-/epics/4623) to fix this limitation. + +The only way to view designs replication data for a particular secondary site is to visit that secondary site directly. For example, `https://<IP of your secondary site>/admin/geo/replication/designs`. +An [epic exists](https://gitlab.com/groups/gitlab-org/-/epics/4624) to fix this limitation. + ## Setup instructions For setup instructions, see [Setting up Geo](setup/index.md). ## Post-installation documentation -After installing GitLab on the **secondary** site(s) and performing the initial configuration, see the following documentation for post-installation information. +After installing GitLab on the **secondary** sites and performing the initial configuration, see the following documentation for post-installation information. ### Configuring Geo @@ -224,7 +236,7 @@ For information on configuring Geo, see [Geo configuration](replication/configur ### Updating Geo -For information on how to update your Geo site(s) to the latest GitLab version, see [Updating the Geo sites](replication/updating_the_geo_sites.md). +For information on how to update your Geo sites to the latest GitLab version, see [Updating the Geo sites](replication/updating_the_geo_sites.md). ### Pausing and resuming replication @@ -237,8 +249,8 @@ secondary. If the site is paused, be sure to resume before promoting. This issue has been fixed in GitLab 13.4 and later. WARNING: -Pausing and resuming of replication is currently only supported for Geo installations using an -Omnibus GitLab-managed database. External databases are currently not supported. +Pausing and resuming of replication is only supported for Geo installations using an +Omnibus GitLab-managed database. External databases are not supported. In some circumstances, like during [upgrades](replication/updating_the_geo_sites.md) or a [planned failover](disaster_recovery/planned_failover.md), it is desirable to pause replication between the primary and secondary. diff --git a/doc/administration/geo/replication/object_storage.md b/doc/administration/geo/replication/object_storage.md index 3a10d3bad58..06f25fa15af 100644 --- a/doc/administration/geo/replication/object_storage.md +++ b/doc/administration/geo/replication/object_storage.md @@ -35,7 +35,7 @@ To have: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10586) in GitLab 12.4. WARNING: -This is a [**beta** feature](https://about.gitlab.com/handbook/product/#beta) and is not ready yet for production use at any scale. The main limitations are a lack of testing at scale and no verification of any replicated data. +This is a [**Beta** feature](../../../policy/alpha-beta-support.md#beta-features) and is not ready yet for production use at any scale. The main limitations are a lack of testing at scale and no verification of any replicated data. **Secondary** sites can replicate files stored on the **primary** site regardless of whether they are stored on the local file system or in object storage. diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index 1371e5d84c8..8f55ce99787 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -268,7 +268,7 @@ sudo gitlab-rake gitlab:geo:check GitLab Geo is available ... no Try fixing it: - Upload a new license that includes the GitLab Geo feature + Add a new license that includes the GitLab Geo feature For more information see: https://about.gitlab.com/features/gitlab-geo/ GitLab Geo is enabled ... Exception: PG::UndefinedTable: ERROR: relation "geo_nodes" does not exist @@ -1043,6 +1043,25 @@ To resolve this issue: the **primary** node using IPv4 in the `/etc/hosts` file. Alternatively, you should [enable IPv6 on the **primary** node](https://docs.gitlab.com/omnibus/settings/nginx.html#setting-the-nginx-listen-address-or-addresses). +### Secondary site returns 502 errors with Geo proxying + +When [Geo proxying for secondary sites](../secondary_proxy/index.md) is enabled, and the secondary site user interface returns +502 errors, it is possible that the response header proxied from the primary site is too large. + +Check the NGINX logs for errors similar to this example: + +```plaintext +2022/01/26 00:02:13 [error] 26641#0: *829148 upstream sent too big header while reading response header from upstream, client: 1.2.3.4, server: geo.staging.gitlab.com, request: "POST /users/sign_in HTTP/2.0", upstream: "http://unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket:/users/sign_in", host: "geo.staging.gitlab.com", referrer: "https://geo.staging.gitlab.com/users/sign_in" +``` + +To resolve this issue: + +1. Set `nginx['proxy_custom_buffer_size'] = '8k'` in `/etc/gitlab.rb` on all web nodes on the secondary site. +1. Reconfigure the **secondary** using `sudo gitlab-ctl reconfigure`. + +If you still get this error, you can further increase the buffer size by repeating the steps above +and changing the `8k` size, for example by doubling it to `16k`. + ### Geo Admin Area shows 'Unknown' for health status and 'Request failed with status code 401' If using a load balancer, ensure that the load balancer's URL is set as the `external_url` in the diff --git a/doc/administration/geo/replication/using_a_geo_server.md b/doc/administration/geo/replication/using_a_geo_server.md deleted file mode 100644 index 62562a1149d..00000000000 --- a/doc/administration/geo/replication/using_a_geo_server.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: '../../geo/replication/usage.md' -remove_date: '2022-02-01' ---- - -This document was moved to [another location](../../geo/replication/usage.md). - -<!-- This redirect file can be deleted after 2022-02-01 --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/geo/secondary_proxy/index.md b/doc/administration/geo/secondary_proxy/index.md index c8dcbb81312..768adab9101 100644 --- a/doc/administration/geo/secondary_proxy/index.md +++ b/doc/administration/geo/secondary_proxy/index.md @@ -140,6 +140,8 @@ for details. To use TLS certificates with Let's Encrypt, you can manually point the domain to one of the Geo sites, generate the certificate, then copy it to all other sites. +- [Viewing projects and designs data from a primary site is not possible when using a unified URL](../index.md#view-replication-data-on-the-primary-site). + ## Behavior of secondary sites when the primary Geo site is down Considering that web traffic is proxied to the primary, the behavior of the secondary sites differs when the primary diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md index a8bebef8f44..9c917be123e 100644 --- a/doc/administration/geo/setup/database.md +++ b/doc/administration/geo/setup/database.md @@ -192,7 +192,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o for more details. NOTE: - If you need to use `0.0.0.0` or `*` as the listen_address, you also need to add + If you need to use `0.0.0.0` or `*` as the listen_address, you also must add `127.0.0.1/32` to the `postgresql['md5_auth_cidr_addresses']` setting, to allow Rails to connect through `127.0.0.1`. For more information, see [omnibus-5258](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5258). @@ -378,7 +378,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o 1. Configure PostgreSQL: This step is similar to how we configured the **primary** instance. - We need to enable this, even if using a single node. + We must enable this, even if using a single node. Edit `/etc/gitlab/gitlab.rb` and add the following, replacing the IP addresses with addresses appropriate to your network configuration: @@ -407,7 +407,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ``` For external PostgreSQL instances, see [additional instructions](external_database.md). - If you bring a former **primary** node back online to serve as a **secondary** node, then you also need to remove `roles(['geo_primary_role'])` or `geo_primary_role['enable'] = true`. + If you bring a former **primary** node back online to serve as a **secondary** node, then you also must remove `roles(['geo_primary_role'])` or `geo_primary_role['enable'] = true`. 1. Reconfigure GitLab for the changes to take effect: @@ -472,7 +472,7 @@ data before running `pg_basebackup`. initial replication to take under an hour. - Pass `--sslmode=disable` to skip PostgreSQL TLS authentication altogether (for example, you know the network path is secure, or you are using a site-to-site - VPN). This is **not** safe over the public Internet! + VPN). It is **not** safe over the public Internet! - You can read more details about each `sslmode` in the [PostgreSQL documentation](https://www.postgresql.org/docs/12/libpq-ssl.html#LIBPQ-SSL-PROTECTION); the instructions above are carefully written to ensure protection against @@ -480,7 +480,7 @@ data before running `pg_basebackup`. - Change the `--slot-name` to the name of the replication slot to be used on the **primary** database. The script attempts to create the replication slot automatically if it does not exist. - - If you're repurposing an old server into a Geo **secondary** node, you need to + - If you're repurposing an old server into a Geo **secondary** node, you must add `--force` to the command line. - When not in a production machine you can disable backup step if you really sure this is what you want by adding `--skip-backup` @@ -547,7 +547,7 @@ FATAL: could not connect to the primary server: FATAL: password authentication On all GitLab Geo **secondary** servers: -1. The first step isn't necessary from a configuration perspective, since the hashed `'sql_replication_password'` +1. The first step isn't necessary from a configuration perspective, because the hashed `'sql_replication_password'` is not used on the GitLab Geo **secondary**. However in the event that **secondary** needs to be promoted to the GitLab Geo **primary**, make sure to match the `'sql_replication_password'` in the secondary server configuration. @@ -612,7 +612,7 @@ and other database best practices. ##### Step 1. Configure Patroni permanent replication slot on the primary site -To set up database replication with Patroni on a secondary node, we need to +To set up database replication with Patroni on a secondary node, we must configure a _permanent replication slot_ on the primary node's Patroni cluster, and ensure password authentication is used. @@ -678,7 +678,7 @@ Leader instance**: ##### Step 2. Configure the internal load balancer on the primary site To avoid reconfiguring the Standby Leader on the secondary site whenever a new -Leader is elected on the primary site, we need to set up a TCP internal load +Leader is elected on the primary site, we must set up a TCP internal load balancer which gives a single endpoint for connecting to the Patroni cluster's Leader. @@ -860,7 +860,7 @@ For each Patroni instance on the secondary site: ### Migrating from repmgr to Patroni -1. Before migrating, it is recommended that there is no replication lag between the primary and secondary sites and that replication is paused. In GitLab 13.2 and later, you can pause and resume replication with `gitlab-ctl geo-replication-pause` and `gitlab-ctl geo-replication-resume` on a Geo secondary database node. +1. Before migrating, we recommend that there is no replication lag between the primary and secondary sites and that replication is paused. In GitLab 13.2 and later, you can pause and resume replication with `gitlab-ctl geo-replication-pause` and `gitlab-ctl geo-replication-resume` on a Geo secondary database node. 1. Follow the [instructions to migrate repmgr to Patroni](../../postgresql/replication_and_failover.md#switching-from-repmgr-to-patroni). When configuring Patroni on each primary site database node, add `patroni['replication_slots'] = { '<slot_name>' => 'physical' }` to `gitlab.rb` where `<slot_name>` is the name of the replication slot for your Geo secondary. This ensures that Patroni recognizes the replication slot as permanent and not drop it upon restarting. 1. If database replication to the secondary was paused before migration, resume replication once Patroni is confirmed working on the primary. @@ -869,7 +869,7 @@ to `gitlab.rb` where `<slot_name>` is the name of the replication slot for your Before the introduction of Patroni, Geo had no Omnibus support for HA setups on the secondary node. -With Patroni it's now possible to support that. In order to migrate the existing PostgreSQL to Patroni: +With Patroni it's now possible to support that. To migrate the existing PostgreSQL to Patroni: 1. Make sure you have a Consul cluster setup on the secondary (similar to how you set it up on the primary). 1. [Configure a permanent replication slot](#step-1-configure-patroni-permanent-replication-slot-on-the-primary-site). @@ -894,7 +894,7 @@ A production-ready and secure setup requires at least three Consul nodes, two Patroni nodes and one PgBouncer node on the secondary site. Because of [omnibus-6587](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6587), Consul can't track multiple -services, so these need to be different than the nodes used for the Standby Cluster database. +services, so these must be different than the nodes used for the Standby Cluster database. Be sure to use [password credentials](../../postgresql/replication_and_failover.md#database-authorization-for-patroni) and other database best practices. @@ -1037,7 +1037,7 @@ For each node running the `gitlab-rails`, `sidekiq`, and `geo-logcursor` service sudo -i ``` -1. Edit `/etc/gitlab/gitlab.rb` and add the following attributes. You may have other attributes set, but the following need to be set. +1. Edit `/etc/gitlab/gitlab.rb` and add the following attributes. You may have other attributes set, but the following must be set. ```ruby # Tracking database settings diff --git a/doc/administration/geo/setup/index.md b/doc/administration/geo/setup/index.md index 7d365f73101..04a341aa822 100644 --- a/doc/administration/geo/setup/index.md +++ b/doc/administration/geo/setup/index.md @@ -10,7 +10,7 @@ type: howto These instructions assume you have a working instance of GitLab. They guide you through: 1. Making your existing instance the **primary** site. -1. Adding **secondary** site(s). +1. Adding **secondary** sites. WARNING: The steps below should be followed in the order they appear. **Make sure the GitLab version is the same on all sites.** @@ -19,15 +19,15 @@ The steps below should be followed in the order they appear. **Make sure the Git If you installed GitLab using the Omnibus packages (highly recommended): -1. [Install GitLab Enterprise Edition](https://about.gitlab.com/install/) on the node(s) that will serve as the **secondary** site. Do not create an account or log in to the new **secondary** site. -1. [Upload the GitLab License](../../../user/admin_area/license.md) on the **primary** site to unlock Geo. The license must be for [GitLab Premium](https://about.gitlab.com/pricing/) or higher. +1. [Install GitLab Enterprise Edition](https://about.gitlab.com/install/) on the nodes that will serve as the **secondary** site. Do not create an account or log in to the new **secondary** site. +1. [Add the GitLab License](../../../user/admin_area/license.md) on the **primary** site to unlock Geo. The license must be for [GitLab Premium](https://about.gitlab.com/pricing/) or higher. 1. [Set up the database replication](database.md) (`primary (read-write) <-> secondary (read-only)` topology). -1. [Configure fast lookup of authorized SSH keys in the database](../../operations/fast_ssh_key_lookup.md). This step is required and needs to be done on **both** the **primary** and **secondary** site(s). -1. [Configure GitLab](../replication/configuration.md) to set the **primary** and **secondary** site(s). -1. Optional: [Configure a secondary LDAP server](../../auth/ldap/index.md) for the **secondary** site(s). See [notes on LDAP](../index.md#ldap). +1. [Configure fast lookup of authorized SSH keys in the database](../../operations/fast_ssh_key_lookup.md). This step is required and needs to be done on **both** the **primary** and **secondary** sites. +1. [Configure GitLab](../replication/configuration.md) to set the **primary** and **secondary** sites. +1. Optional: [Configure a secondary LDAP server](../../auth/ldap/index.md) for the **secondary** sites. See [notes on LDAP](../index.md#ldap). 1. Follow the [Using a Geo Site](../replication/usage.md) guide. 1. [Configure Geo secondary proxying](../secondary_proxy/index.md) to use a single, unified URL for all Geo sites. This step is recommended to accelerate most read requests while transparently proxying writes to the primary Geo site. ## Post-installation documentation -After installing GitLab on the **secondary** site(s) and performing the initial configuration, see the [following documentation for post-installation information](../index.md#post-installation-documentation). +After installing GitLab on the **secondary** sites and performing the initial configuration, see the [following documentation for post-installation information](../index.md#post-installation-documentation). diff --git a/doc/administration/get_started.md b/doc/administration/get_started.md index f3b4400c8c3..e6a2e833af3 100644 --- a/doc/administration/get_started.md +++ b/doc/administration/get_started.md @@ -39,8 +39,8 @@ Get started: - Create a [project](../user/project/working_with_projects.md#create-a-project). - Create a [group](../user/group/index.md#create-a-group). - [Add members](../user/group/index.md#add-users-to-a-group) to the group. -- Create a [subgroup](../user/group/subgroups/index.md#creating-a-subgroup). -- [Add members](../user/group/subgroups/index.md#membership) to the subgroup. +- Create a [subgroup](../user/group/subgroups/index.md#create-a-subgroup). +- [Add members](../user/group/subgroups/index.md#subgroup-membership) to the subgroup. - Enable [external authorization control](../user/admin_area/settings/external_authorization.md#configuration). **More resources** @@ -49,7 +49,7 @@ Get started: - Sync group memberships [by using LDAP](../administration/auth/ldap/ldap_synchronization.md#group-sync). - Manage user access with inherited permissions. Use up to 20 levels of subgroups to organize both teams and projects. - Learn more about [inherited permissions](../user/project/members/index.md#inherited-membership). - - View [nested category examples](../user/group/subgroups/index.md#overview). + - View an [example](../user/group/subgroups/index.md). ## Import projects diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index ea24e901943..f88a1aa2b4d 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -13,7 +13,7 @@ It is used by GitLab to read and write Git data. Gitaly is present in every GitLab installation and coordinates Git repository storage and retrieval. Gitaly can be: -- A simple background service operating on a single instance Omnibus GitLab (all of +- A background service operating on a single instance Omnibus GitLab (all of GitLab on one machine). - Separated onto its own instance and configured in a full cluster configuration, depending on scaling and availability requirements. @@ -71,7 +71,7 @@ the current status of these issues, please refer to the referenced issues and ep | Gitaly Cluster + Geo - Issues retrying failed syncs | If Gitaly Cluster is used on a Geo secondary site, repositories that have failed to sync could continue to fail when Geo tries to resync them. Recovering from this state requires assistance from support to run manual steps. Work is in-progress to update Gitaly Cluster to [identify repositories with a unique and persistent identifier](https://gitlab.com/gitlab-org/gitaly/-/issues/3485), which is expected to resolve the issue. | No known solution at this time. | | Database inconsistencies due to repository access outside of Gitaly Cluster's control | Operations that write to the repository storage that do not go through normal Gitaly Cluster methods can cause database inconsistencies. These can include (but are not limited to) snapshot restoration for cluster node disks, node upgrades which modify files under Git control, or any other disk operation that may touch repository storage external to GitLab. The Gitaly team is actively working to provide manual commands to [reconcile the Praefect database with the repository storage](https://gitlab.com/groups/gitlab-org/-/epics/6723). | Don't directly change repositories on any Gitaly Cluster node at this time. | | Praefect unable to insert data into the database due to migrations not being applied after an upgrade | If the database is not kept up to date with completed migrations, then the Praefect node is unable to perform normal operation. | Make sure the Praefect database is up and running with all migrations completed (For example: `/opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-migrate-status` should show a list of all applied migrations). Consider [requesting live upgrade assistance](https://about.gitlab.com/support/scheduling-live-upgrade-assistance.html) so your upgrade plan can be reviewed by support. | -| Restoring a Gitaly Cluster node from a snapshot in a running cluster | Because the Gitaly Cluster runs with consistent state, introducing a single node that is behind will result in the cluster not being able to reconcile the nodes data and other nodes data | Don't restore a single Gitaly Cluster node from a backup snapshot. If you need to restore from backup, it's best to snapshot all Gitaly Cluster nodes at the same time and take a database dump of the Praefect database. | +| Restoring a Gitaly Cluster node from a snapshot in a running cluster | Because the Gitaly Cluster runs with consistent state, introducing a single node that is behind will result in the cluster not being able to reconcile the nodes data and other nodes data | Don't restore a single Gitaly Cluster node from a backup snapshot. If you must restore from backup, it's best to snapshot all Gitaly Cluster nodes at the same time and take a database dump of the Praefect database. | ### Snapshot backup and recovery limitations @@ -179,7 +179,7 @@ In this example: - Repositories are stored on a virtual storage called `storage-1`. - Three Gitaly nodes provide `storage-1` access: `gitaly-1`, `gitaly-2`, and `gitaly-3`. - The three Gitaly nodes share data in three separate hashed storage locations. -- The [replication factor](#replication-factor) is `3`. There are three copies maintained +- The [replication factor](#replication-factor) is `3`. Three copies are maintained of each repository. The availability objectives for Gitaly clusters assuming a single node failure are: @@ -237,7 +237,7 @@ Engineering support for NFS for Git repositories is deprecated. Technical suppor is not well suited to Git workloads which are CPU and IOPS sensitive. Specifically: -- Git is sensitive to file system latency. Even simple operations require many +- Git is sensitive to file system latency. Some operations require many read operations. Operations that are fast on block storage can become an order of magnitude slower. This significantly impacts GitLab application performance. - NFS performance optimizations that prevent the performance gap between @@ -386,8 +386,7 @@ them back to direct Gitaly storage: 1. Create and configure a new [Gitaly server](configure_gitaly.md#run-gitaly-on-its-own-server). 1. [Move the repositories](../operations/moving_repositories.md#move-repositories) - to the newly created storage. There are different possibilities to move them - by shard or by group, this gives you the opportunity to spread them over + to the newly created storage. You can move them by shard or by group, which gives you the opportunity to spread them over multiple Gitaly servers. ## Monitor Gitaly and Gitaly Cluster @@ -509,7 +508,7 @@ The following metrics are available from the `/metrics` endpoint: They reflect configuration defined for this instance of Praefect. - `gitaly_praefect_replication_latency_bucket`, a histogram measuring the amount of time it takes - for replication to complete once the replication job starts. Available in GitLab 12.10 and later. + for replication to complete after the replication job starts. Available in GitLab 12.10 and later. - `gitaly_praefect_replication_delay_bucket`, a histogram measuring how much time passes between when the replication job is created and when it starts. Available in GitLab 12.10 and later. - `gitaly_praefect_node_latency_bucket`, a histogram measuring the latency in Gitaly returning @@ -542,7 +541,7 @@ You can also monitor the [Praefect logs](../logs.md#praefect-logs). The following metrics are available from the `/db_metrics` endpoint: - `gitaly_praefect_unavailable_repositories`, the number of repositories that have no healthy, up to date replicas. -- `gitaly_praefect_read_only_repositories`, the number of repositories in read-only mode within a virtual storage. +- `gitaly_praefect_read_only_repositories`, the number of repositories in read-only mode in a virtual storage. This metric is available for backwards compatibility reasons. `gitaly_praefect_unavailable_repositories` is more accurate. - `gitaly_praefect_replication_queue_depth`, the number of jobs in the replication queue. diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index 8f17835b8a3..d7a06994b6c 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -616,7 +616,7 @@ Note the following: - IP address, you must add it as a Subject Alternative Name to the certificate. - When running Praefect sub-commands such as `dial-nodes` and `list-untracked-repositories` from the command line with [Gitaly TLS enabled](configure_gitaly.md#enable-tls-support), you must set the `SSL_CERT_DIR` or `SSL_CERT_FILE` - environment variable so that the Gitaly certificate is trusted. For example: + environment variable so that the Gitaly certificate is trusted. For example: ```shell sudo SSL_CERT_DIR=/etc/gitlab/trusted_certs /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dial-nodes @@ -1196,6 +1196,9 @@ You can configure: current assignments: gitaly-1, gitaly-2 ``` +If `default_replication_factor` is unset, the repositories are always replicated on every node defined in `virtual_storages`. If a new +node is introduced to the virtual storage, both new and existing repositories are replicated to the node automatically. + ## Automatic failover and primary election strategies Praefect regularly checks the health of each Gitaly node. This is used to automatically fail over diff --git a/doc/administration/gitaly/recovery.md b/doc/administration/gitaly/recovery.md index 9210c8f08d3..a7166f7e62e 100644 --- a/doc/administration/gitaly/recovery.md +++ b/doc/administration/gitaly/recovery.md @@ -76,7 +76,7 @@ The following parameters are available: some assigned copies that are not available. NOTE: -`dataloss` is still in beta and the output format is subject to change. +`dataloss` is still in [Beta](../../policy/alpha-beta-support.md#beta-features) and the output format is subject to change. To check for repositories with outdated primaries or for unavailable repositories, run: @@ -377,7 +377,7 @@ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.t The `track-repository` Praefect sub-command adds repositories on disk to the Praefect database to be tracked. ```shell -sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml track-repository -virtual-storage <virtual-storage> -repository <repository> -replicate-immediately +sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml track-repository -virtual-storage <virtual-storage> -authoritative-storage <storage-name> -repository <repository> -replicate-immediately ``` - `-virtual-storage` is the virtual storage the repository is located in. Virtual storages are configured in `/etc/gitlab/gitlab.rb` under `praefect['virtual_storages]` and looks like the following: diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md index ff136b44340..516af4ca469 100644 --- a/doc/administration/gitaly/troubleshooting.md +++ b/doc/administration/gitaly/troubleshooting.md @@ -554,7 +554,7 @@ Is [some cases](index.md#known-issues) the Praefect database can get out of sync a given repository is fully synced on all nodes, run the [`gitlab:praefect:replicas` Rake task](../raketasks/praefect.md#replica-checksums) that checksums the repository on all Gitaly nodes. -The [Praefect dataloss](recovery.md#check-for-data-loss) command only checks the state of the repo in the Praefect database, and cannot +The [Praefect dataloss](recovery.md#check-for-data-loss) command only checks the state of the repository in the Praefect database, and cannot be relied to detect sync problems in this scenario. ### Relation does not exist errors @@ -610,3 +610,45 @@ Possible solutions: - Provision larger VMs to gain access to larger network traffic allowances. - Use your cloud service's monitoring and logging to check that the Praefect nodes are not exhausting their traffic allowances. + +## Profiling Gitaly + +Gitaly exposes several of Golang's built-in performance profiling tools on the Prometheus listen port. For example, if Prometheus is listening +on port `9236` of the GitLab server: + +- Get a list of running `goroutines` and their backtraces: + + ```shell + curl --output goroutines.txt "http://<gitaly_server>:9236/debug/pprof/goroutine?debug=2" + ``` + +- Run a CPU profile for 30 seconds: + + ```shell + curl --output cpu.bin "http://<gitaly_server>:9236/debug/pprof/profile" + ``` + +- Profile heap memory usage: + + ```shell + curl --output heap.bin "http://<gitaly_server>:9236/debug/pprof/heap" + ``` + +- Record a 5 second execution trace. This will impact Gitaly's performance while running: + + ```shell + curl --output trace.bin "http://<gitaly_server>:9236/debug/pprof/trace?seconds=5" + ``` + +On a host with `go` installed, the CPU profile and heap profile can be viewed in a browser: + +```shell +go tool pprof -http=:8001 cpu.bin +go tool pprof -http=:8001 heap.bin +``` + +Execution traces can be viewed by running: + +```shell +go tool trace heap.bin +``` diff --git a/doc/administration/incoming_email.md b/doc/administration/incoming_email.md index cd1f39b1295..caade32b7c2 100644 --- a/doc/administration/incoming_email.md +++ b/doc/administration/incoming_email.md @@ -68,11 +68,16 @@ this method only supports replies, and not the other features of [incoming email ## Accepted headers -Email is processed correctly when a configured email address is present in one of the following headers: +> Accepting `Received` headers [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/81489) in GitLab 14.9 [with a flag](feature_flags.md) named `use_received_header_for_incoming_emails`. Enabled by default. + +Email is processed correctly when a configured email address is present in one of the following headers +(sorted in the order they are checked): - `To` +- `References` - `Delivered-To` - `Envelope-To` or `X-Envelope-To` +- `Received` In GitLab 14.6 and later, [Service Desk](../user/project/service_desk.md) also checks accepted headers. @@ -84,6 +89,9 @@ However, it might not include the configured GitLab email address if: - The address was included when using "Reply all". - The email was forwarded. +The `Received` header can contain multiple email addresses. These are checked in the order that they appear. +The first match is used. + ## Rejected headers To prevent unwanted issue creation from automatic email systems, GitLab ignores all incoming email diff --git a/doc/administration/index.md b/doc/administration/index.md index bd6549fca80..50dc2bc905a 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -35,7 +35,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Installing GitLab on Amazon Web Services (AWS)](../install/aws/index.md): Set up GitLab on Amazon AWS. - [Geo](geo/index.md): Replicate your GitLab instance to other geographic locations as a read-only fully operational version. - [Disaster Recovery](geo/disaster_recovery/index.md): Quickly fail-over to a different site with minimal effort in a disaster situation. -- [Add License](../user/admin_area/license.md): Upload a license at install time to unlock features that are in paid tiers of GitLab. +- [Add License](../user/admin_area/license.md): Add a license at install time to unlock features that are in paid tiers of GitLab. ### Configuring GitLab @@ -68,7 +68,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Elasticsearch](../integration/elasticsearch.md): Enable Elasticsearch to empower Advanced Search. Use when you deal with a huge amount of data. - [External Classification Policy Authorization](../user/admin_area/settings/external_authorization.md) -- [Upload a license](../user/admin_area/license.md): Upload a license to unlock +- [Add a license](../user/admin_area/license.md): Add a license to unlock features that are in paid tiers of GitLab. - [Admin Area](../user/admin_area/index.md): for self-managed instance-wide configuration and maintenance. diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md index 614ab5278c5..11a34f5b5f8 100644 --- a/doc/administration/instance_limits.md +++ b/doc/administration/instance_limits.md @@ -153,6 +153,17 @@ Set the limit to `0` to disable it. - **Default rate limit**: Disabled (unlimited). +### Search rate limit + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/80631) in GitLab 14.9 + +This setting limits global search requests. + +| Limit | Default (requests per minute) | +|-------------------------|-------------------------------| +| Authenticated user | 30 | +| Unauthenticated user | 10 | + ## Gitaly concurrency limit Clone traffic can put a large strain on your Gitaly service. To prevent such workloads from overwhelming your Gitaly server, you can set concurrency limits in Gitaly's configuration file. @@ -208,13 +219,18 @@ When the number exceeds the limit the page displays an alert and links to a pagi > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/51401) in GitLab 11.10. -The number of pipelines that can be created in a single push is 4. -This limit prevents the accidental creation of pipelines when `git push --all` -or `git push --mirror` is used. +When pushing multiple changes with a single Git push, like multiple tags or branches, +only four tag or branch pipelines can be triggered. This limit prevents the accidental +creation of a large number of pipelines when using `git push --all` or `git push --mirror`. + +[Merge request pipelines](../ci/pipelines/merge_request_pipelines.md) are not limited. +If the Git push updates multiple merge requests at the same time, a merge request pipeline +can trigger for every updated merge request. -This limit does not affect any of the updated merge request pipelines. -All updated merge requests have a pipeline created when using -[merge request pipelines](../ci/pipelines/merge_request_pipelines.md). +To remove the limit so that any number of pipelines can trigger for a single Git push event, +administrators can enable the `git_push_create_all_pipelines` [feature flag](feature_flags.md). +Enabling this feature flag is not recommended, as it can cause excessive load on the GitLab +instance if too many changes are pushed at once and a flood of pipelines are created accidentally. ## Retention of activity history @@ -558,7 +574,7 @@ Plan.default.actual_limits.update!(ci_max_artifact_size_junit: 10) ### Number of files per GitLab Pages web-site -The total number of file entries (including directories and symlinks) is limited to `100000` per +The total number of file entries (including directories and symlinks) is limited to `200,000` per GitLab Pages website. This is the default limit for all [GitLab self-managed and SaaS plans](https://about.gitlab.com/pricing/). @@ -800,7 +816,7 @@ Reports that go over the 20 MB limit aren't loaded. Affected reports: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/8638) in GitLab 13.3. You can set a limit on the content of repository files that are indexed in -Elasticsearch. Any files larger than this limit only index the file name. +Elasticsearch. Any files larger than this limit only index the filename. The file content is neither indexed nor searchable. Setting a limit helps reduce the memory usage of the indexing processes and @@ -942,3 +958,7 @@ varies by file type: If a branch is merged while open merge requests still point to it, GitLab can retarget merge requests pointing to the now-merged branch. To learn more, read [Branch retargeting on merge](../user/project/merge_requests/getting_started.md#branch-retargeting-on-merge). + +## CDN-based limits on GitLab.com + +In addition to application-based limits, GitLab.com is configured to use Cloudflare's standard DDoS protection and Spectrum to protect Git over SSH. Cloudflare terminates client TLS connections but is not application aware and cannot be used for limits tied to users or groups. Cloudflare page rules and rate limits are configured with Terraform. These configurations are [not public](https://about.gitlab.com/handbook/communication/#not-public) because they include security and abuse implementations that detect malicious activities and making them public would undermine those operations. diff --git a/doc/administration/integration/plantuml.md b/doc/administration/integration/plantuml.md index 94fef89d966..5a9699b3a39 100644 --- a/doc/administration/integration/plantuml.md +++ b/doc/administration/integration/plantuml.md @@ -21,7 +21,7 @@ blocks to an HTML image tag, with the source pointing to the PlantUML instance. diagram delimiters `@startuml`/`@enduml` aren't required, as these are replaced by the `plantuml` block: -- **Markdown** +- **Markdown** files with the extension `.md`: ````markdown ```plantuml @@ -30,7 +30,10 @@ by the `plantuml` block: ``` ```` -- **AsciiDoc** + For additional acceptable extensions, review the + [`languages.yaml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/vendor/languages.yml#L3174) file. + +- **AsciiDoc** files with the extension `.asciidoc`, `.adoc`, or `.asc`: ```plaintext [plantuml, format="png", id="myDiagram", width="200px"] @@ -224,6 +227,16 @@ these steps: gitlab_rails['env'] = { 'PLANTUML_ENCODING' => 'deflate' } ``` + In GitLab Helm chart, you can set it by adding a variable to the + [global.extraEnv](https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/charts/globals.md#extraenv) + section, like this: + + ```yaml + global: + extraEnv: + PLANTUML_ENCODING: deflate + ``` + - For GitLab versions 13.1 and later, PlantUML integration now [requires a header prefix in the URL](https://github.com/plantuml/plantuml/issues/117#issuecomment-6235450160) to distinguish different encoding types. diff --git a/doc/administration/integration/terminal.md b/doc/administration/integration/terminal.md index f570c9b559f..792afc6c3d7 100644 --- a/doc/administration/integration/terminal.md +++ b/doc/administration/integration/terminal.md @@ -11,6 +11,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: This feature was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. +- Read more about the non-deprecated [Web Terminals accessible through the Web IDE](../../user/project/web_ide/index.md). +- Read more about the non-deprecated [Web Terminals accessible from a running CI job](../../ci/interactive_web_terminal/index.md). + +--- + With the introduction of the [Kubernetes integration](../../user/infrastructure/clusters/index.md), GitLab can store and use credentials for a Kubernetes cluster. GitLab uses these credentials to provide access to diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index 8f824fcefb3..544a5973052 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -307,6 +307,33 @@ To migrate back to local storage: ## Expiring artifacts +> [In GitLab 14.6](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/76504), we improved the performance of removing expired artifacts, introduced [with a flag](feature_flags.md) named `ci_destroy_all_expired_service`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to +[enable the feature flag](feature_flags.md) named `ci_destroy_all_expired_service`. The feature is not ready for +production use. +On GitLab.com, this feature is not available. + +### Removing expired job artifacts on GitLab self-managed instances + +In the process of migrating old artifacts for our SaaS customers, we are working to resolve any potential unrecoverable data loss for self-managed customers for artifacts that they may not want deleted yet. Before we can use the more performant way of cleaning up expired artifacts, we need to do some remediation to make sure customers don't lose their data, which is part of our effort in [the relevant epic](https://gitlab.com/groups/gitlab-org/-/epics/7097). + +Two options are available: + +- If you don't need any artifacts created before 2020-06-23, an Administrator can enable the worker for removing expired CI/CD artifacts: + + ```ruby + Feature.enable(:ci_destroy_all_expired_service) + ``` + +- If you want to keep any artifacts (including job logs) before 2020-06-23, follow the [progress of the migration effort](https://gitlab.com/groups/gitlab-org/-/epics/7097) where we work on a resolution to have this flag fully enabled in a future release. + +Alternatively, Administrators can also run commands in the Rails console to +[delete artifacts from completed jobs prior to a specific date](#delete-job-artifacts-from-jobs-completed-before-a-specific-date). + +### Usage details + If [`artifacts:expire_in`](../ci/yaml/index.md#artifactsexpire_in) is used to set an expiry for the artifacts, they are marked for deletion right after that date passes. Otherwise, they expire per the [default artifacts expiration setting](../user/admin_area/settings/continuous_integration.md). @@ -387,6 +414,37 @@ space, and in some cases, manually delete job artifacts to reclaim disk space. One possible first step is to [clean up _orphaned_ artifact files](../raketasks/cleanup.md#remove-orphan-artifact-files). +#### List projects and builds with artifacts with a specific expiration (or no expiration) + +Using a [Rails console](operations/rails_console.md), you can find projects that have job artifacts with either: + +- No expiration date. +- An expiration date more than 7 days in the future. + +Similar to [deleting artifacts](#delete-job-artifacts-from-jobs-completed-before-a-specific-date), use the following example time frames +and alter them as needed: + +- `7.days.from_now` +- `10.days.from_now` +- `2.weeks.from_now` +- `3.months.from_now` + +Each of the following scripts also limits the search to 50 results with `.limit(50)`, but this number can also be changed as needed: + +```ruby +# Find builds & projects with artifacts that never expire +builds_with_artifacts_that_never_expire = Ci::Build.with_downloadable_artifacts.where(artifacts_expire_at: nil).limit(50) +builds_with_artifacts_that_never_expire.find_each do |build| + puts "Build with id #{build.id} has artifacts that don't expire and belongs to project #{build.project.full_path}" +end + +# Find builds & projects with artifacts that expire after 7 days from today +builds_with_artifacts_that_expire_in_a_week = Ci::Build.with_downloadable_artifacts.where('artifacts_expire_at > ?', 7.days.from_now).limit(50) +builds_with_artifacts_that_expire_in_a_week.find_each do |build| + puts "Build with id #{build.id} has artifacts that expire at #{build.artifacts_expire_at} and belongs to project #{build.project.full_path}" +end +``` + #### List projects by total size of job artifacts stored List the top 20 projects, sorted by the total size of job artifacts stored, by @@ -435,7 +493,7 @@ list = arts.order(size: :desc).limit(50).each do |art| end ``` -To change the number of projects listed, change the number in `limit(50)`. +To change the number of job artifacts listed, change the number in `limit(50)`. #### Delete job artifacts from jobs completed before a specific date @@ -572,3 +630,15 @@ review: ``` In both cases, you might need to add `region` to the job artifact [object storage configuration](#connection-settings). + +### Job artifact upload fails with `500 Internal Server Error (Missing file)` + +Bucket names that include folder paths are not supported with [consolidated object storage](object_storage.md#consolidated-object-storage-configuration). +For example, `bucket/path`. If a bucket name has a path in it, you might receive an error similar to: + +```plaintext +WARNING: Uploading artifacts as "archive" to coordinator... POST https://gitlab.example.com/api/v4/jobs/job_id/artifacts?artifact_format=zip&artifact_type=archive&expire_in=1+day: 500 Internal Server Error (Missing file) +FATAL: invalid argument +``` + +If a job artifact fails to upload with the above error when using consolidated object storage, make sure you are [using separate buckets](object_storage.md#use-separate-buckets) for each data type. diff --git a/doc/administration/logs.md b/doc/administration/logs.md index c4e9642d048..22e13270179 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -1049,7 +1049,10 @@ For example: ## Mattermost Logs -For Omnibus GitLab installations, Mattermost logs are in `/var/log/gitlab/mattermost/mattermost.log`. +For Omnibus GitLab installations, Mattermost logs are in these locations: + +- `/var/log/gitlab/mattermost/mattermost.log` +- `/var/log/gitlab/mattermost/current` ## Workhorse Logs @@ -1095,9 +1098,9 @@ For Omnibus GitLab installations, GitLab Monitor logs are in `/var/log/gitlab/gi For Omnibus GitLab installations, GitLab Exporter logs are in `/var/log/gitlab/gitlab-exporter/`. -## GitLab Agent Server +## GitLab agent server -For Omnibus GitLab installations, GitLab Agent Server logs are +For Omnibus GitLab installations, GitLab agent server logs are in `/var/log/gitlab/gitlab-kas/`. ## Praefect Logs diff --git a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md index 911d76a89e3..09cef51d14c 100644 --- a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md +++ b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md @@ -8,10 +8,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32351) in GitLab 12.7 [with a flag](../../feature_flags.md) named `self_monitoring_project`. Disabled by default. > - Generally available in GitLab 12.8. [Feature flag `self_monitoring_project`](https://gitlab.com/gitlab-org/gitlab/-/issues/198511) removed. -> - [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/348909) in GitLab 14.8. +> - [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/348909) in GitLab 14.9. Planned for removal in GitLab 15.0. WARNING: -This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/348909) in GitLab 14.8. +This feature is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/348909) +for use in GitLab 14.9, and is planned for removal in GitLab 15.0. GitLab provides administrators insights into the health of their GitLab instance. diff --git a/doc/administration/monitoring/index.md b/doc/administration/monitoring/index.md index df655053723..381c807dbc5 100644 --- a/doc/administration/monitoring/index.md +++ b/doc/administration/monitoring/index.md @@ -24,6 +24,3 @@ Explore our features to monitor your GitLab instance: provide health check information when probed. - [`nginx_status`](https://docs.gitlab.com/omnibus/settings/nginx.html#enablingdisabling-nginx_status): Monitor your NGINX server status. -- [Auto Monitoring](../../topics/autodevops/stages.md#auto-monitoring): Automated - monitoring for your application's server and response metrics, provided by - [Auto DevOps](../../topics/autodevops/index.md). diff --git a/doc/administration/monitoring/performance/img/performance_bar_request_selector_warning.png b/doc/administration/monitoring/performance/img/performance_bar_request_selector_warning.png Binary files differdeleted file mode 100644 index 3872c8b41b2..00000000000 --- a/doc/administration/monitoring/performance/img/performance_bar_request_selector_warning.png +++ /dev/null diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md index e93dbd3a2d6..bc311f73dfe 100644 --- a/doc/administration/monitoring/performance/performance_bar.md +++ b/doc/administration/monitoring/performance/performance_bar.md @@ -30,8 +30,7 @@ From left to right, the performance bar displays: is enabled. It shows which server role was used for the query. "Primary" means that the query was sent to the read/write primary server. "Replica" means it was sent to a read-only replica. - - **Config name**: shows up only when the - `GITLAB_MULTIPLE_DATABASE_METRICS` environment variable is set. This is + - **Configuration name**: this is used to distinguish between different databases configured for different GitLab features. The name shown is the same name used to configure database connections in GitLab. @@ -48,12 +47,13 @@ From left to right, the performance bar displays: - **External HTTP calls**: the time taken (in milliseconds) and the total number of external calls to other systems. Click to display a modal window with more details. -- **Load timings** of the page: if your browser supports load timings (Chromium - and Chrome) several values in milliseconds, separated by slashes. +- **Load timings** of the page: if your browser supports load timings, several + values in milliseconds, separated by slashes. Click to display a modal window with more details. The values, from left to right: - **Backend**: time needed for the base page to load. - [**First Contentful Paint**](https://web.dev/first-contentful-paint/): - Time until something was visible to the user. + Time until something was visible to the user. Displays `NaN` if your browser does not + support this feature. - [**DomContentLoaded**](https://developers.google.com/web/fundamentals/performance/critical-rendering-path/measure-crp) Event. - **Total number of requests** the page loaded. - **Memory**: the amount of memory consumed and objects allocated during the selected request. @@ -64,6 +64,9 @@ From left to right, the performance bar displays: can be added by its full URL (authenticated as the current user), or by the value of its `X-Request-Id` header. - **Download**: a link to download the raw JSON used to generate the Performance Bar reports. +- **Memory Report**: a link that generates a + [memory profiling](../../../development/performance.md#using-memory-profiler) + report of the current URL. - **Flamegraph** with mode: a link to generate a [flamegraph](../../../development/profiling.md#speedscope-flamegraphs) of the current URL with the selected [Stackprof mode](https://github.com/tmm1/stackprof#sampling): - The **Wall** mode samples every *interval* of the time on a clock on a wall. The interval is set to `10100` microseconds. @@ -91,18 +94,14 @@ For non-administrators to display the performance bar, it must be ## Request warnings +> [Warning icon in the request selector removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/82187) in GitLab 14.9. + Requests that exceed predefined limits display a warning **{warning}** icon and explanation next to the metric. In this example, the Gitaly call duration exceeded the threshold.  -If any requests on the current page generated warnings, the warning icon displays -next to the **Requests** selector menu. In this selector menu, an exclamation `(!)` -appears next to requests with warnings. - - - ## Enable the performance bar for non-administrators The performance bar is disabled by default for non-administrators. To enable it diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index 9d90c4f3cc7..22f7419be9a 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -301,7 +301,7 @@ Prometheus has several custom flags to configure local storage: - `storage.tsdb.retention.time`: when to remove old data. Defaults to `15d`. Overrides `storage.tsdb.retention` if this flag is set to anything other than the default. -- `storage.tsdb.retention.size`: [EXPERIMENTAL] the maximum number of bytes of storage blocks to +- `storage.tsdb.retention.size`: (experimental) the maximum number of bytes of storage blocks to retain. The oldest data is removed first. Defaults to `0` (disabled). This flag is experimental and may change in future releases. Units supported: `B`, `KB`, `MB`, `GB`, `TB`, `PB`, `EB`. For example, `512MB`. diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md index 420f2191ef2..8ea4ad9019e 100644 --- a/doc/administration/object_storage.md +++ b/doc/administration/object_storage.md @@ -62,7 +62,7 @@ Using the consolidated object storage configuration has a number of advantages: - It enables the use of [encrypted S3 buckets](#encrypted-s3-buckets). - It [uploads files to S3 with proper `Content-MD5` headers](https://gitlab.com/gitlab-org/gitlab-workhorse/-/issues/222). -Because [direct upload mode](../development/uploads.md#direct-upload) +Because [direct upload mode](../development/uploads/implementation.md#direct-upload) must be enabled, only the following providers can be used: - [Amazon S3-compatible providers](#s3-compatible-connection-settings) @@ -598,6 +598,7 @@ See the following additional guides: 1. Configure [database lookup of SSH keys](operations/fast_ssh_key_lookup.md) to eliminate the need for a shared `authorized_keys` file. 1. [Prevent local disk usage for job logs](job_logs.md#prevent-local-disk-usage). +1. [Disable Pages local storage](pages/index.md#disable-pages-local-storage). ## Warnings, limitations, and known issues diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md index dca99879cc3..8877b1266e0 100644 --- a/doc/administration/operations/fast_ssh_key_lookup.md +++ b/doc/administration/operations/fast_ssh_key_lookup.md @@ -182,7 +182,7 @@ file for the environment, since it isn't generated dynamically. ## Troubleshooting If your SSH traffic is [slow](https://github.com/linux-pam/linux-pam/issues/270) -or causing high CPU load, be sure to check the size of `/var/log/btmp`, and ensure it is rotated on a regular basis. +or causing high CPU load, be sure to check the size of `/var/log/btmp`, and ensure it is rotated on a regular basis or after reaching a certain size. If this file is very large, GitLab SSH fast lookup can cause the bottleneck to be hit more frequently, thus decreasing performance even further. If you are able to, you may consider disabling [`UsePAM` in your `sshd_config`](https://linux.die.net/man/5/sshd_config) to avoid reading `/var/log/btmp` altogether. diff --git a/doc/administration/operations/puma.md b/doc/administration/operations/puma.md index c7df8249ae4..9e828b39f46 100644 --- a/doc/administration/operations/puma.md +++ b/doc/administration/operations/puma.md @@ -4,45 +4,19 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Puma **(FREE SELF)** +# Configure the bundled Puma instance of the GitLab package **(FREE SELF)** -Puma is a simple, fast, multi-threaded, and highly concurrent HTTP 1.1 server for -Ruby applications. It's the default GitLab web server since GitLab 13.0 -and has replaced Unicorn. From GitLab 14.0, Unicorn is no longer supported. +Puma is a fast, multi-threaded, and highly concurrent HTTP 1.1 server for +Ruby applications. It runs the core Rails application that provides the user-facing +features of GitLab. -NOTE: -Starting with GitLab 13.0, Puma is the default web server and Unicorn has been disabled. -In GitLab 14.0, Unicorn was removed from the Linux package and only Puma is available. - -## Configure Puma - -To configure Puma: - -1. Determine suitable Puma worker and thread [settings](../../install/requirements.md#puma-settings). -1. If you're switching from Unicorn, [convert any custom settings to Puma](#convert-unicorn-settings-to-puma). -1. For multi-node deployments, configure the load balancer to use the - [readiness check](../load_balancer.md#readiness-check). -1. Reconfigure GitLab so the above changes take effect: - - ```shell - sudo gitlab-ctl reconfigure - ``` - -For Helm-based deployments, see the -[`webservice` chart documentation](https://docs.gitlab.com/charts/charts/gitlab/webservice/index.html). - -For more details about the Puma configuration, see the -[Puma documentation](https://github.com/puma/puma#configuration). +## Reducing memory use -## Puma Worker Killer +To reduce memory use, Puma forks worker processes. Each time a worker is created, +it shares memory with the primary process. The worker uses additional memory only +when it changes or adds to its memory pages. -Puma forks worker processes as part of a strategy to reduce memory use. - -Each time a worker is created, it shares memory with the primary process and -only uses additional memory when it makes changes or additions to its memory pages. - -Memory use by workers therefore increases over time, and Puma Worker Killer is the -mechanism that recovers this memory. +Memory use increases over time, but you can use Puma Worker Killer to recover memory. By default: @@ -50,6 +24,8 @@ By default: exceeds a [memory limit](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/cluster/puma_worker_killer_initializer.rb). - Rolling restarts of Puma workers are performed every 12 hours. +### Change the memory limit setting + To change the memory limit setting: 1. Edit `/etc/gitlab/gitlab.rb`: @@ -58,26 +34,28 @@ To change the memory limit setting: puma['per_worker_max_memory_mb'] = 1024 ``` -1. Reconfigure GitLab for the changes to take effect: +1. Reconfigure GitLab: ```shell sudo gitlab-ctl reconfigure ``` -There are costs associated with killing and replacing workers including -reduced capacity to run GitLab, and CPU that is consumed -restarting the workers. `per_worker_max_memory_mb` should be set to a -higher value if the worker killer is replacing workers too often. +When workers are killed and replaced, capacity to run GitLab is reduced, +and CPU is consumed. Set `per_worker_max_memory_mb` to a higher value if the worker killer +is replacing workers too often. -Worker count is calculated based on CPU cores, so a small GitLab deployment +Worker count is calculated based on CPU cores. A small GitLab deployment with 4-8 workers may experience performance issues if workers are being restarted -frequently, once or more per minute. This is too often. +too often (once or more per minute). A higher value of `1200` or more would be beneficial if the server has free memory. -The worker killer checks every 20 seconds, and can be monitored using -[the Puma log](../logs.md#puma_stdoutlog) `/var/log/gitlab/puma/puma_stdout.log`. -For example, for GitLab 13.5: +### Monitor worker memory + +The worker killer checks memory every 20 seconds. + +To monitor the worker killer, use [the Puma log](../logs.md#puma_stdoutlog) `/var/log/gitlab/puma/puma_stdout.log`. +For example: ```plaintext PumaWorkerKiller: Out of memory. 4 workers consuming total: 4871.23828125 MB @@ -88,9 +66,9 @@ From this output: - The formula that calculates the maximum memory value results in workers being killed before they reach the `per_worker_max_memory_mb` value. -- The default values for the formula before GitLab 13.5 were 550MB for the primary - and `per_worker_max_memory_mb` specified 850MB for each worker. -- As of GitLab 13.5 the values are primary: 800MB, worker: 1024MB. +- In GitLab 13.4 and earlier, the default values for the formula were 550MB for the primary + and 850MB for each worker. +- In GitLab 13.5 and later, the values are primary: 800MB, worker: 1024MB. - The threshold for workers to be killed is set at 98% of the limit: ```plaintext @@ -102,16 +80,15 @@ From this output: Increasing the maximum to `1200`, for example, would set a `max: 5488 MB` value. -Workers use additional memory on top of the shared memory, how much +Workers use additional memory on top of the shared memory. The amount of memory depends on a site's use of GitLab. -## Worker timeout +## Change the worker timeout -A [timeout of 60 seconds](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/initializers/rack_timeout.rb) -is used when Puma is enabled. +The default Puma [timeout is 60 seconds](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/initializers/rack_timeout.rb). NOTE: -Unlike Unicorn, the `puma['worker_timeout']` setting does not set the maximum request duration. +The `puma['worker_timeout']` setting does not set the maximum request duration. To change the worker timeout to 600 seconds: @@ -123,26 +100,38 @@ To change the worker timeout to 600 seconds: } ``` -1. Reconfigure GitLab for the changes to take effect: +1. Reconfigure GitLab: ```shell sudo gitlab-ctl reconfigure ``` -## Memory-constrained environments +## Disable Puma clustered mode in memory-constrained environments In a memory-constrained environment with less than 4GB of RAM available, consider disabling Puma -[Clustered mode](https://github.com/puma/puma#clustered-mode). +[clustered mode](https://github.com/puma/puma#clustered-mode). -Configuring Puma by setting the amount of `workers` to `0` could reduce memory usage by hundreds of MB. -For details on Puma worker and thread settings, see the [Puma requirements](../../install/requirements.md#puma-settings). +Set the number of `workers` to `0` to reduce memory usage by hundreds of MB: + +1. Edit `/etc/gitlab/gitlab.rb`: -Unlike in a Clustered mode, which is set up by default, only a single Puma process would serve the application. + ```ruby + puma['worker_processes'] = 0 + ``` -The downside of running Puma with such configuration is the reduced throughput, which could be -considered as a fair tradeoff in a memory-constraint environment. +1. Reconfigure GitLab: -When running Puma in Single mode, some features are not supported: + ```shell + sudo gitlab-ctl reconfigure + ``` + +Unlike in a clustered mode, which is set up by default, only a single Puma process would serve the application. +For details on Puma worker and thread settings, see the [Puma requirements](../../install/requirements.md#puma-settings). + +The downside of running Puma in this configuration is the reduced throughput, which can be +considered a fair tradeoff in a memory-constrained environment. + +When running Puma in single mode, some features are not supported: - [Phased restart](https://gitlab.com/gitlab-org/gitlab/-/issues/300665) - [Puma Worker Killer](https://gitlab.com/gitlab-org/gitlab/-/issues/300664) @@ -151,22 +140,23 @@ To learn more, visit [epic 5303](https://gitlab.com/groups/gitlab-org/-/epics/53 ## Performance caveat when using Puma with Rugged -For deployments where NFS is used to store Git repository, we allow GitLab to use -[direct Git access](../gitaly/index.md#direct-access-to-git-in-gitlab) to improve performance using +For deployments where NFS is used to store Git repositories, GitLab uses +[direct Git access](../gitaly/index.md#direct-access-to-git-in-gitlab) to improve performance by using [Rugged](https://github.com/libgit2/rugged). Rugged usage is automatically enabled if direct Git access [is available](../gitaly/index.md#how-it-works) -and Puma is running single threaded, unless it is disabled by -[feature flags](../../development/gitaly.md#legacy-rugged-code). +and Puma is running single threaded, unless it is disabled by a +[feature flag](../../development/gitaly.md#legacy-rugged-code). -MRI Ruby uses a GVL. This allows MRI Ruby to be multi-threaded, but running at -most on a single core. Since Rugged can use a thread for long periods of -time (due to intensive I/O operations of Git access), this can starve other threads -that might be processing requests. This is not a case for Unicorn or Puma running -in a single thread mode, as concurrently at most one request is being processed. +MRI Ruby uses a Global VM Lock (GVL). GVL allows MRI Ruby to be multi-threaded, but running at +most on a single core. -We are actively working on removing Rugged usage. Even though performance without Rugged +Git includes intensive I/O operations. When Rugged uses a thread for a long period of time, +other threads that might be processing requests can starve. Puma running in single thread mode +does not have this issue, because concurrently at most one request is being processed. + +GitLab is working to remove Rugged usage. Even though performance without Rugged is acceptable today, in some cases it might be still beneficial to run with it. Given the caveat of running Rugged with multi-threaded Puma, and acceptable @@ -177,55 +167,70 @@ This default behavior may not be the optimal configuration in some situations. I plays an important role in your deployment, we suggest you benchmark to find the optimal configuration: -- The safest option is to start with single-threaded Puma. When working with - Rugged, single-threaded Puma works the same as Unicorn. -- To force Rugged to be used with multi-threaded Puma, you can use - [feature flags](../../development/gitaly.md#legacy-rugged-code). +- The safest option is to start with single-threaded Puma. +- To force Rugged to be used with multi-threaded Puma, you can use a + [feature flag](../../development/gitaly.md#legacy-rugged-code). -## Convert Unicorn settings to Puma +## Switch from Unicorn to Puma NOTE: -Starting with GitLab 13.0, Puma is the default web server and Unicorn has been -disabled by default. In GitLab 14.0, Unicorn was removed from the Linux package -and only Puma is available. +For Helm-based deployments, see the +[`webservice` chart documentation](https://docs.gitlab.com/charts/charts/gitlab/webservice/index.html). + +Starting with GitLab 13.0, Puma is the default web server and Unicorn has been disabled. +In GitLab 14.0, [Unicorn was removed](../../update/removals.md#unicorn-in-gitlab-self-managed) +from the Linux package and is no longer supported. -Puma has a multi-thread architecture which uses less memory than a multi-process +Puma has a multi-thread architecture that uses less memory than a multi-process application server like Unicorn. On GitLab.com, we saw a 40% reduction in memory -consumption. Most Rails applications requests normally include a proportion of I/O wait time. +consumption. Most Rails application requests normally include a proportion of I/O wait time. -During I/O wait time MRI Ruby releases the GVL (Global VM Lock) to other threads. +During I/O wait time, MRI Ruby releases the GVL to other threads. Multi-threaded Puma can therefore still serve more requests than a single process. When switching to Puma, any Unicorn server configuration will _not_ carry over automatically, due to differences between the two application servers. -The table below summarizes which Unicorn configuration keys correspond to those -in Puma when using the Linux package, and which ones have no corresponding counterpart. - -| Unicorn | Puma | -| ------------------------------------ | ---------------------------------- | -| `unicorn['enable']` | `puma['enable']` | -| `unicorn['worker_timeout']` | `puma['worker_timeout']` | -| `unicorn['worker_processes']` | `puma['worker_processes']` | -| n/a | `puma['ha']` | -| n/a | `puma['min_threads']` | -| n/a | `puma['max_threads']` | -| `unicorn['listen']` | `puma['listen']` | -| `unicorn['port']` | `puma['port']` | -| `unicorn['socket']` | `puma['socket']` | -| `unicorn['pidfile']` | `puma['pidfile']` | -| `unicorn['tcp_nopush']` | n/a | -| `unicorn['backlog_socket']` | n/a | -| `unicorn['somaxconn']` | `puma['somaxconn']` | -| n/a | `puma['state_path']` | -| `unicorn['log_directory']` | `puma['log_directory']` | -| `unicorn['worker_memory_limit_min']` | n/a | -| `unicorn['worker_memory_limit_max']` | `puma['per_worker_max_memory_mb']` | -| `unicorn['exporter_enabled']` | `puma['exporter_enabled']` | -| `unicorn['exporter_address']` | `puma['exporter_address']` | -| `unicorn['exporter_port']` | `puma['exporter_port']` | - -## Puma exporter - -You can use the Puma exporter to measure various Puma metrics. For more information, see -[Puma exporter](../monitoring/prometheus/puma_exporter.md). +To switch from Unicorn to Puma: + +1. Determine suitable Puma [worker and thread settings](../../install/requirements.md#puma-settings). +1. Convert any custom Unicorn settings to Puma. + + The table below summarizes which Unicorn configuration keys correspond to those + in Puma when using the Linux package, and which ones have no corresponding counterpart. + + | Unicorn | Puma | + | ------------------------------------ | ---------------------------------- | + | `unicorn['enable']` | `puma['enable']` | + | `unicorn['worker_timeout']` | `puma['worker_timeout']` | + | `unicorn['worker_processes']` | `puma['worker_processes']` | + | n/a | `puma['ha']` | + | n/a | `puma['min_threads']` | + | n/a | `puma['max_threads']` | + | `unicorn['listen']` | `puma['listen']` | + | `unicorn['port']` | `puma['port']` | + | `unicorn['socket']` | `puma['socket']` | + | `unicorn['pidfile']` | `puma['pidfile']` | + | `unicorn['tcp_nopush']` | n/a | + | `unicorn['backlog_socket']` | n/a | + | `unicorn['somaxconn']` | `puma['somaxconn']` | + | n/a | `puma['state_path']` | + | `unicorn['log_directory']` | `puma['log_directory']` | + | `unicorn['worker_memory_limit_min']` | n/a | + | `unicorn['worker_memory_limit_max']` | `puma['per_worker_max_memory_mb']` | + | `unicorn['exporter_enabled']` | `puma['exporter_enabled']` | + | `unicorn['exporter_address']` | `puma['exporter_address']` | + | `unicorn['exporter_port']` | `puma['exporter_port']` | + +1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +1. Optional. For multi-node deployments, configure the load balancer to use the + [readiness check](../load_balancer.md#readiness-check). + +## Related topics + +- [Use the Puma exporter to measure various Puma metrics](../monitoring/prometheus/puma_exporter.md) diff --git a/doc/administration/package_information/deprecated_os.md b/doc/administration/package_information/deprecated_os.md deleted file mode 100644 index 1f6fe0fce5d..00000000000 --- a/doc/administration/package_information/deprecated_os.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'supported_os.md' -remove_date: '2022-02-18' ---- - -This document was moved to [another location](supported_os.md). - -<!-- This redirect file can be deleted after <2022-02-18>. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/package_information/index.md b/doc/administration/package_information/index.md index 2781f789409..c908f526569 100644 --- a/doc/administration/package_information/index.md +++ b/doc/administration/package_information/index.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Package information **(FREE SELF)** diff --git a/doc/administration/package_information/licensing.md b/doc/administration/package_information/licensing.md index 02358c66993..a7bf5c52d7b 100644 --- a/doc/administration/package_information/licensing.md +++ b/doc/administration/package_information/licensing.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Package Licensing **(FREE SELF)** diff --git a/doc/administration/package_information/omnibus_packages.md b/doc/administration/package_information/omnibus_packages.md index 115d6c394ad..21e3f5711ba 100644 --- a/doc/administration/package_information/omnibus_packages.md +++ b/doc/administration/package_information/omnibus_packages.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Omnibus based packages and images **(FREE SELF)** diff --git a/doc/administration/package_information/postgresql_versions.md b/doc/administration/package_information/postgresql_versions.md index 97a35fb29ed..e707cb70187 100644 --- a/doc/administration/package_information/postgresql_versions.md +++ b/doc/administration/package_information/postgresql_versions.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # PostgreSQL versions shipped with Omnibus GitLab **(FREE SELF)** diff --git a/doc/administration/package_information/signed_packages.md b/doc/administration/package_information/signed_packages.md index fb605f8d5be..d7bcfa113ff 100644 --- a/doc/administration/package_information/signed_packages.md +++ b/doc/administration/package_information/signed_packages.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Package Signatures **(FREE SELF)** diff --git a/doc/administration/package_information/supported_os.md b/doc/administration/package_information/supported_os.md index 71ebc4d3647..547b8bf8658 100644 --- a/doc/administration/package_information/supported_os.md +++ b/doc/administration/package_information/supported_os.md @@ -27,6 +27,7 @@ The following lists the currently supported OSs and their possible EOL dates. | SLES 12 | GitLab EE 9.0.0 | x86_64 | Oct 2027 | <https://www.suse.com/lifecycle/> | | Ubuntu 18.04 | GitLab CE / GitLab EE 10.7.0 | amd64 | April 2023 | <https://wiki.ubuntu.com/Releases> | | Ubuntu 20.04 | GitLab CE / GitLab EE 13.2.0 | amd64, arm64 | April 2025 | <https://wiki.ubuntu.com/Releases> | +| Amazon Linux 2 | GitLab CE / GitLab EE 14.9.0 | amd64, arm64 | June 2023 | <https://aws.amazon.com/amazon-linux-2/faqs> | | Raspbian Buster | GitLab CE 12.2.0 | armhf | 2022 | <https://wiki.debian.org/DebianReleases#Production_Releases> | NOTE: diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md index 33a5311709f..7bf7362c68b 100644 --- a/doc/administration/packages/container_registry.md +++ b/doc/administration/packages/container_registry.md @@ -457,6 +457,22 @@ To configure the `s3` storage driver in Omnibus: - `pathstyle` should be set to true to use `host/bucket_name/object` style paths instead of `bucket_name.host/object`. [Set to false for AWS S3](https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/). + You can set a rate limit on connections to S3 to avoid 503 errors from the S3 API. To do this, + set `maxrequestspersecond` to a number within the [S3 request rate threshold](https://aws.amazon.com/premiumsupport/knowledge-center/s3-503-within-request-rate-prefix/): + + ```ruby + registry['storage'] = { + 's3' => { + 'accesskey' => 's3-access-key', + 'secretkey' => 's3-secret-key-for-access-key', + 'bucket' => 'your-s3-bucket', + 'region' => 'your-s3-region', + 'regionendpoint' => 'your-s3-regionendpoint', + 'maxrequestspersecond' => 100 + } + } + ``` + 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. **Installations from source** @@ -1704,3 +1720,38 @@ What does this mean? This strongly suggests that the S3 user does not have the r [permissions to perform a HEAD request](https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html). The solution: check the [IAM permissions again](https://docs.docker.com/registry/storage-drivers/s3/). Once the right permissions were set, the error goes away. + +### Missing `gitlab-registry.key` prevents container repository deletion + +If you disable your GitLab instance's Container Registry and try to remove a project that has +container repositories, the following error occurs: + +```plaintext +Errno::ENOENT: No such file or directory @ rb_sysopen - /var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key +``` + +In this case, follow these steps: + +1. Temporarily enable the instance-wide setting for the Container Registry in your `gitlab.rb`: + + ```ruby + gitlab_rails['registry_enabled'] = true + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) + for the changes to take effect. +1. Try the removal again. + +If you still can't remove the repository using the common methods, you can use the +[GitLab Rails console](../troubleshooting/navigating_gitlab_via_rails_console.md) +to remove the project by force: + +```ruby +# Path to the project you'd like to remove +prj = Project.find_by_full_path(<project_path>) + +# The following will delete the project's container registry, so be sure to double-check the path beforehand! +if prj.has_container_registry_tags? + prj.container_repositories.each { |p| p.destroy } +end +``` diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index f000824711f..56ccf70d169 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -547,6 +547,7 @@ archive. You can modify the cache behavior by changing the following configurati | `zip_cache_cleanup` | The interval at which archives are cleaned from memory if they have already expired. Default is 30s. | | `zip_cache_refresh` | The time interval in which an archive is extended in memory if accessed before `zip_cache_expiration`. This works together with `zip_cache_expiration` to determine if an archive is extended in memory. See the [example below](#zip-cache-refresh-example) for important details. Default is 30s. | | `zip_open_timeout` | The maximum time allowed to open a ZIP archive. Increase this time for big archives or slow network connections, as doing so may affect the latency of serving Pages. Default is 30s. | +| `zip_http_client_timeout` | The maximum time for the ZIP HTTP client. Default is 30m. | #### ZIP cache refresh example @@ -1387,15 +1388,20 @@ This can happen if your `gitlab-secrets.json` file is out of date between GitLab Pages. Follow steps 8-10 of [Running GitLab Pages on a separate server](#running-gitlab-pages-on-a-separate-server), in all of your GitLab Pages instances. -### Intermittent 502 errors when using an AWS Network Load Balancer and GitLab Pages is running on multiple application servers +### Intermittent 502 errors when using an AWS Network Load Balancer and GitLab Pages Connections will time out when using a Network Load Balancer with client IP preservation enabled and [the request is looped back to the source server](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-troubleshooting.html#loopback-timeout). This can happen to GitLab instances with multiple servers -running both the core GitLab application and GitLab Pages. +running both the core GitLab application and GitLab Pages. This can also happen when a single +container is running both the core GitLab application and GitLab Pages. AWS [recommends using an IP target type](https://aws.amazon.com/premiumsupport/knowledge-center/target-connection-fails-load-balancer/) to resolve this issue. +Turning off [client IP preservation](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-target-groups.html#client-ip-preservation) +may resolve this issue when the core GitLab application and GitLab Pages run on the same host or +container. + ### 500 error with `securecookie: failed to generate random iv` and `Failed to save the session` This problem most likely results from an [out-dated operating system](../package_information/supported_os.md#os-versions-that-are-no-longer-supported). diff --git a/doc/administration/postgresql/external.md b/doc/administration/postgresql/external.md index 67c5448a8a0..f4e4c7f8bef 100644 --- a/doc/administration/postgresql/external.md +++ b/doc/administration/postgresql/external.md @@ -21,7 +21,7 @@ If you use a cloud-managed service, or provide your own PostgreSQL instance: 1. If you are using a cloud-managed service, you may need to grant additional roles to your `gitlab` user: - Amazon RDS requires the [`rds_superuser`](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.html#Appendix.PostgreSQL.CommonDBATasks.Roles) role. - - Azure Database for PostgreSQL requires the [`azure_pg_admin`](https://docs.microsoft.com/en-us/azure/postgresql/howto-create-users#how-to-create-additional-admin-users-in-azure-database-for-postgresql) role. + - Azure Database for PostgreSQL requires the [`azure_pg_admin`](https://docs.microsoft.com/en-us/azure/postgresql/howto-create-users#how-to-create-additional-admin-users-in-azure-database-for-postgresql) role. Azure Database for PostgreSQL - Flexible Server requires [allow-listing extensions before they can be installed](https://docs.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-extensions#how-to-use-postgresql-extensions). - Google Cloud SQL requires the [`cloudsqlsuperuser`](https://cloud.google.com/sql/docs/postgres/users#default-users) role. This is for the installation of extensions during installation and upgrades. As an alternative, diff --git a/doc/administration/raketasks/doctor.md b/doc/administration/raketasks/doctor.md index bed3cdcbcfe..457077462a6 100644 --- a/doc/administration/raketasks/doctor.md +++ b/doc/administration/raketasks/doctor.md @@ -6,4 +6,6 @@ remove_date: '2022-03-04' This document was moved to [another location](check.md#verify-database-values-can-be-decrypted-using-the-current-secrets). <!-- This redirect file can be deleted after 2022-03-04. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index eb1127b5e99..fcce44f62b2 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -1363,6 +1363,7 @@ To configure the Praefect nodes, on each one: alertmanager['enable'] = false grafana['enable'] = false gitlab_exporter['enable'] = false + gitlab_kas['enable'] = false nginx['enable'] = false # Praefect Configuration @@ -1503,6 +1504,7 @@ On each node: alertmanager['enable'] = false grafana['enable'] = false gitlab_exporter['enable'] = false + gitlab_kas['enable'] = false nginx['enable'] = false # Prevent database migrations from running on upgrade automatically @@ -1524,6 +1526,11 @@ On each node: # Gitaly Auth Token # Should be the same as praefect_internal_token gitaly['auth_token'] = '<praefect_internal_token>' + + # Gitaly Pack-objects cache + # Recommended to be enabled for improved performance but can notably increase disk I/O + # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info + gitaly['pack_objects_cache_enabled'] = true ``` 1. Append the following to `/etc/gitlab/gitlab.rb` for each respective server: @@ -1631,7 +1638,7 @@ To configure Praefect with TLS: ```ruby git_data_dirs({ "default" => { - "gitaly_address" => 'tls://LOAD_BALANCER_SERVER_ADDRESS:2305', + "gitaly_address" => 'tls://LOAD_BALANCER_SERVER_ADDRESS:3305', "gitaly_token" => 'PRAEFECT_EXTERNAL_TOKEN' } }) @@ -1675,6 +1682,7 @@ To configure the Sidekiq nodes, on each one: alertmanager['enable'] = false grafana['enable'] = false gitlab_exporter['enable'] = false + gitlab_kas['enable'] = false nginx['enable'] = false # External URL diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index 86819024eeb..c08fe985b40 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -1367,6 +1367,7 @@ To configure the Praefect nodes, on each one: alertmanager['enable'] = false grafana['enable'] = false gitlab_exporter['enable'] = false + gitlab_kas['enable'] = false nginx['enable'] = false # Praefect Configuration @@ -1507,6 +1508,7 @@ On each node: alertmanager['enable'] = false grafana['enable'] = false gitlab_exporter['enable'] = false + gitlab_kas['enable'] = false nginx['enable'] = false # Prevent database migrations from running on upgrade automatically @@ -1528,6 +1530,11 @@ On each node: # Gitaly Auth Token # Should be the same as praefect_internal_token gitaly['auth_token'] = '<praefect_internal_token>' + + # Gitaly Pack-objects cache + # Recommended to be enabled for improved performance but can notably increase disk I/O + # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info + gitaly['pack_objects_cache_enabled'] = true ``` 1. Append the following to `/etc/gitlab/gitlab.rb` for each respective server: @@ -1635,7 +1642,7 @@ To configure Praefect with TLS: ```ruby git_data_dirs({ "default" => { - "gitaly_address" => 'tls://LOAD_BALANCER_SERVER_ADDRESS:2305', + "gitaly_address" => 'tls://LOAD_BALANCER_SERVER_ADDRESS:3305', "gitaly_token" => 'PRAEFECT_EXTERNAL_TOKEN' } }) @@ -1679,6 +1686,7 @@ To configure the Sidekiq nodes, on each one: alertmanager['enable'] = false grafana['enable'] = false gitlab_exporter['enable'] = false + gitlab_kas['enable'] = false nginx['enable'] = false # External URL diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index f6c484b08b1..6f6c02c309a 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -286,11 +286,6 @@ further configuration steps. ```ruby # Disable all components except PostgreSQL related ones roles(['postgres_role']) - prometheus['enable'] = false - alertmanager['enable'] = false - pgbouncer_exporter['enable'] = false - redis_exporter['enable'] = false - gitlab_exporter['enable'] = false # Set the network addresses that the exporters used for monitoring will listen on node_exporter['listen_address'] = '0.0.0.0:9100' @@ -365,19 +360,7 @@ Omnibus: ```ruby ## Enable Redis - redis['enable'] = true - - # Avoid running unnecessary services on the Redis server - gitaly['enable'] = false - postgresql['enable'] = false - puma['enable'] = false - sidekiq['enable'] = false - gitlab_workhorse['enable'] = false - prometheus['enable'] = false - alertmanager['enable'] = false - grafana['enable'] = false - gitlab_exporter['enable'] = false - nginx['enable'] = false + roles(["redis_master_role"]) redis['bind'] = '0.0.0.0' redis['port'] = 6379 @@ -481,6 +464,7 @@ To configure the Gitaly server, on the server node you want to use for Gitaly: alertmanager['enable'] = false grafana['enable'] = false gitlab_exporter['enable'] = false + gitlab_kas['enable'] = false nginx['enable'] = false # Prevent database migrations from running on upgrade automatically diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index 587303a1f8f..76f81e65580 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -1307,6 +1307,7 @@ To configure the Praefect nodes, on each one: alertmanager['enable'] = false grafana['enable'] = false gitlab_exporter['enable'] = false + gitlab_kas['enable'] = false nginx['enable'] = false # Praefect Configuration @@ -1447,6 +1448,7 @@ On each node: alertmanager['enable'] = false grafana['enable'] = false gitlab_exporter['enable'] = false + gitlab_kas['enable'] = false nginx['enable'] = false # Prevent database migrations from running on upgrade automatically @@ -1468,6 +1470,11 @@ On each node: # Gitaly Auth Token # Should be the same as praefect_internal_token gitaly['auth_token'] = '<praefect_internal_token>' + + # Gitaly Pack-objects cache + # Recommended to be enabled for improved performance but can notably increase disk I/O + # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info + gitaly['pack_objects_cache_enabled'] = true ``` 1. Append the following to `/etc/gitlab/gitlab.rb` for each respective server: @@ -1575,7 +1582,7 @@ To configure Praefect with TLS: ```ruby git_data_dirs({ "default" => { - "gitaly_address" => 'tls://LOAD_BALANCER_SERVER_ADDRESS:2305', + "gitaly_address" => 'tls://LOAD_BALANCER_SERVER_ADDRESS:3305', "gitaly_token" => 'PRAEFECT_EXTERNAL_TOKEN' } }) @@ -1621,6 +1628,7 @@ To configure the Sidekiq nodes, one each one: alertmanager['enable'] = false grafana['enable'] = false gitlab_exporter['enable'] = false + gitlab_kas['enable'] = false nginx['enable'] = false # External URL diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index f4bf232d548..dfa963d1ad0 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -1376,6 +1376,7 @@ To configure the Praefect nodes, on each one: alertmanager['enable'] = false grafana['enable'] = false gitlab_exporter['enable'] = false + gitlab_kas['enable'] = false nginx['enable'] = false # Praefect Configuration @@ -1516,6 +1517,7 @@ On each node: alertmanager['enable'] = false grafana['enable'] = false gitlab_exporter['enable'] = false + gitlab_kas['enable'] = false nginx['enable'] = false # Prevent database migrations from running on upgrade automatically @@ -1537,6 +1539,11 @@ On each node: # Gitaly Auth Token # Should be the same as praefect_internal_token gitaly['auth_token'] = '<praefect_internal_token>' + + # Gitaly Pack-objects cache + # Recommended to be enabled for improved performance but can notably increase disk I/O + # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info + gitaly['pack_objects_cache_enabled'] = true ``` 1. Append the following to `/etc/gitlab/gitlab.rb` for each respective server: @@ -1644,7 +1651,7 @@ To configure Praefect with TLS: ```ruby git_data_dirs({ "default" => { - "gitaly_address" => 'tls://LOAD_BALANCER_SERVER_ADDRESS:2305', + "gitaly_address" => 'tls://LOAD_BALANCER_SERVER_ADDRESS:3305', "gitaly_token" => 'PRAEFECT_EXTERNAL_TOKEN' } }) @@ -1688,6 +1695,7 @@ To configure the Sidekiq nodes, on each one: alertmanager['enable'] = false grafana['enable'] = false gitlab_exporter['enable'] = false + gitlab_kas['enable'] = false nginx['enable'] = false # External URL diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index 4014ec04904..f2463afbf3b 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -1305,6 +1305,7 @@ To configure the Praefect nodes, on each one: alertmanager['enable'] = false grafana['enable'] = false gitlab_exporter['enable'] = false + gitlab_kas['enable'] = false nginx['enable'] = false # Praefect Configuration @@ -1445,6 +1446,7 @@ On each node: alertmanager['enable'] = false grafana['enable'] = false gitlab_exporter['enable'] = false + gitlab_kas['enable'] = false nginx['enable'] = false # Prevent database migrations from running on upgrade automatically @@ -1466,6 +1468,11 @@ On each node: # Gitaly Auth Token # Should be the same as praefect_internal_token gitaly['auth_token'] = '<praefect_internal_token>' + + # Gitaly Pack-objects cache + # Recommended to be enabled for improved performance but can notably increase disk I/O + # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info + gitaly['pack_objects_cache_enabled'] = true ``` 1. Append the following to `/etc/gitlab/gitlab.rb` for each respective server: @@ -1573,7 +1580,7 @@ To configure Praefect with TLS: ```ruby git_data_dirs({ "default" => { - "gitaly_address" => 'tls://LOAD_BALANCER_SERVER_ADDRESS:2305', + "gitaly_address" => 'tls://LOAD_BALANCER_SERVER_ADDRESS:3305', "gitaly_token" => 'PRAEFECT_EXTERNAL_TOKEN' } }) @@ -1617,6 +1624,7 @@ To configure the Sidekiq nodes, one each one: alertmanager['enable'] = false grafana['enable'] = false gitlab_exporter['enable'] = false + gitlab_kas['enable'] = false nginx['enable'] = false # External URL diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index 815155866e8..3fcd6d7ae4e 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -103,12 +103,14 @@ The following table details the testing done against the reference architectures <style> table.test-coverage td { + border-top: 1px solid #dbdbdb; border-left: 1px solid #dbdbdb; border-right: 1px solid #dbdbdb; border-bottom: 1px solid #dbdbdb; } table.test-coverage th { + border-top: 1px solid #dbdbdb; border-left: 1px solid #dbdbdb; border-right: 1px solid #dbdbdb; border-bottom: 1px solid #dbdbdb; @@ -131,7 +133,6 @@ table.test-coverage th { <th scope="col">Omnibus</th> <th scope="col">Cloud Native Hybrid</th> <th scope="col">Omnibus</th> - <th scope="col">Cloud Native Hybrid</th> </tr> <tr> <th scope="row">1k</th> @@ -140,7 +141,6 @@ table.test-coverage th { <td></td> <td></td> <td></td> - <td></td> </tr> <tr> <th scope="row">2k</th> @@ -149,7 +149,6 @@ table.test-coverage th { <td></td> <td></td> <td></td> - <td></td> </tr> <tr> <th scope="row">3k</th> @@ -158,7 +157,6 @@ table.test-coverage th { <td></td> <td></td> <td></td> - <td></td> </tr> <tr> <th scope="row">5k</th> @@ -167,7 +165,6 @@ table.test-coverage th { <td></td> <td></td> <td></td> - <td></td> </tr> <tr> <th scope="row">10k</th> @@ -176,7 +173,6 @@ table.test-coverage th { <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/10k">Ad-Hoc (inc Cloud Services)</a></td> <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/10k-Cloud-Native-Hybrid">Ad-Hoc</a></td> <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/10k">Ad-Hoc</a></td> - <td></td> </tr> <tr> <th scope="row">25k</th> @@ -185,7 +181,6 @@ table.test-coverage th { <td></td> <td></td> <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/25k">Ad-Hoc</a></td> - <td></td> </tr> <tr> <th scope="row">50k</th> @@ -194,7 +189,6 @@ table.test-coverage th { <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/50k">Ad-Hoc (inc Cloud Services)</a></td> <td></td> <td></td> - <td></td> </tr> </table> @@ -218,7 +212,6 @@ The Standard Reference Architectures are designed to be platform agnostic, with <th scope="col">Omnibus</th> <th scope="col">Cloud Native Hybrid</th> <th scope="col">Omnibus</th> - <th scope="col">Cloud Native Hybrid</th> </tr> <tr> <th scope="row">1k</th> @@ -227,7 +220,6 @@ The Standard Reference Architectures are designed to be platform agnostic, with <td></td> <td></td> <td></td> - <td></td> </tr> <tr> <th scope="row">2k</th> @@ -236,7 +228,6 @@ The Standard Reference Architectures are designed to be platform agnostic, with <td></td> <td></td> <td></td> - <td></td> </tr> <tr> <th scope="row">3k</th> @@ -245,7 +236,6 @@ The Standard Reference Architectures are designed to be platform agnostic, with <td></td> <td></td> <td></td> - <td></td> </tr> <tr> <th scope="row">5k</th> @@ -254,7 +244,6 @@ The Standard Reference Architectures are designed to be platform agnostic, with <td></td> <td></td> <td></td> - <td></td> </tr> <tr> <th scope="row">10k</th> @@ -263,7 +252,6 @@ The Standard Reference Architectures are designed to be platform agnostic, with <td></td> <td></td> <td></td> - <td></td> </tr> <tr> <th scope="row">25k</th> @@ -272,7 +260,6 @@ The Standard Reference Architectures are designed to be platform agnostic, with <td></td> <td></td> <td></td> - <td></td> </tr> <tr> <th scope="row">50k</th> @@ -281,10 +268,92 @@ The Standard Reference Architectures are designed to be platform agnostic, with <td></td> <td></td> <td></td> + </tr> +</table> + +### Recommended cloud providers and services + +NOTE: +The following lists are non exhaustive. Generally, other cloud providers not listed +here will likely work with the same specs, but this hasn't been validated. +Additionally, when it comes to other cloud provider services not listed here, +it's advised to be cautious as each implementation can be notably different +and should be tested thoroughly before production use. + +Through testing and real life usage, the Reference Architectures are validated and supported on the following cloud providers: + +<table> +<thead> + <tr> + <th>Reference Architecture</th> + <th>GCP</th> + <th>AWS</th> + <th>Azure</th> + <th>Bare Metal</th> + </tr> +</thead> +<tbody> + <tr> + <td>Omnibus</td> + <td>✅</td> + <td>✅</td> + <td>✅</td> + <td>✅</td> + </tr> + <tr> + <td>Cloud Native Hybrid</td> + <td>✅</td> + <td>✅</td> + <td></td> + <td></td> + </tr> +</tbody> +</table> + +Additionally, the following cloud provider services are validated and supported for use as part of the Reference Architectures: + +<table> +<thead> + <tr> + <th>Cloud Service</th> + <th>GCP</th> + <th>AWS</th> + <th>Bare Metal</th> + </tr> +</thead> +<tbody> + <tr> + <td>Object Storage</td> + <td>✅ <a href="https://cloud.google.com/storage" target="_blank">Cloud Storage</a></td> + <td>✅ <a href="https://aws.amazon.com/s3/" target="_blank">S3</a></td> + <td>✅ <a href="https://min.io/" target="_blank">MinIO</a></td> + </tr> + <tr> + <td>Database</td> + <td>✅ <a href="https://cloud.google.com/sql" target="_blank" rel="noopener noreferrer">Cloud SQL</a></td> + <td>✅ <a href="https://aws.amazon.com/rds/" target="_blank" rel="noopener noreferrer">RDS</a></td> + <td></td> + </tr> + <tr> + <td>Redis</td> + <td></td> + <td>✅ <a href="https://aws.amazon.com/elasticache/" target="_blank" rel="noopener noreferrer">Elasticache</a></td> <td></td> </tr> +</tbody> </table> +The following specific cloud provider services have been found to have issues in terms of either functionality or performance. As such, they either have caveats that should be considered or are not recommended: + +- [Azure Blob Storage](https://azure.microsoft.com/en-gb/services/storage/blobs/) has been found to have performance limits that can impact production use at certain times. For larger Reference Architectures the service may not be sufficient for production use and an alternative is recommended for use instead. +- [Azure Database for PostgreSQL Server](https://azure.microsoft.com/en-gb/services/postgresql/#overview) (Single / Flexible) is not recommended for use due to notable performance issues or missing functionality. +- [AWS Aurora Database](https://aws.amazon.com/rds/aurora) is not recommended due to compatibility issues. + +NOTE: +As a general rule we unfortunately don't recommend Azure Services at this time. +If required, we advise thorough testing is done at your intended scale +over a sustained period to validate if the service is suitable. + ## Availability Components GitLab comes with the following components for your use, listed from least to diff --git a/doc/administration/reply_by_email.md b/doc/administration/reply_by_email.md index 055f40ead64..13fa8450996 100644 --- a/doc/administration/reply_by_email.md +++ b/doc/administration/reply_by_email.md @@ -47,6 +47,8 @@ following headers, in this order: 1. `References` header 1. `Delivered-To` header 1. `Envelope-To` header +1. `X-Envelope-To` header +1. `Received` header If it finds a reply key, it leaves your reply as a comment on the entity the notification was about (issue, merge request, commit...). diff --git a/doc/administration/repository_checks.md b/doc/administration/repository_checks.md index 0591f5b3c40..6dd8faac65b 100644 --- a/doc/administration/repository_checks.md +++ b/doc/administration/repository_checks.md @@ -32,6 +32,12 @@ Instead of checking repositories manually, GitLab can be configured to run the c When enabled, GitLab periodically runs a repository check on all project repositories and wiki repositories to detect possible data corruption. A project is checked no more than once per month. +Administrators can configure the frequency of repository checks. To edit the frequency: + +- For Omnibus GitLab installations, edit `gitlab_rails['repository_check_worker_cron']` in + `/etc/gitlab/gitlab.rb`. +- For source-based installations, edit `[gitlab.cron_jobs.repository_check_worker]` in + `/home/git/gitlab/config/gitlab.yml`. If any projects fail their repository checks, all GitLab administrators receive an email notification of the situation. By default, this notification is sent out once a week at midnight at diff --git a/doc/administration/restart_gitlab.md b/doc/administration/restart_gitlab.md index 88909b2b9d8..44bfa153123 100644 --- a/doc/administration/restart_gitlab.md +++ b/doc/administration/restart_gitlab.md @@ -9,6 +9,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w Depending on how you installed GitLab, there are different methods to restart its services. +NOTE: +A short downtime is expected for all methods. + ## Omnibus installations If you have used the [Omnibus packages](https://about.gitlab.com/install/) to install GitLab, then @@ -90,8 +93,8 @@ application that powers Omnibus GitLab, makes sure that all things like director permissions, and services are in place and in the same shape that they were initially shipped. -It also restarts GitLab components where needed, if any of their -configuration files have changed. +It also [restarts GitLab components](#how-to-restart-gitlab) +where needed, if any of their configuration files have changed. If you manually edit any files in `/var/opt/gitlab` that are managed by Chef, running reconfigure reverts the changes AND restarts the services that diff --git a/doc/administration/sidekiq.md b/doc/administration/sidekiq.md index fea6ac9e554..b6c1c6a704e 100644 --- a/doc/administration/sidekiq.md +++ b/doc/administration/sidekiq.md @@ -217,6 +217,73 @@ To enable health checks for Sidekiq: sudo gitlab-ctl reconfigure ``` +## Configure LDAP and user or group synchronization + +If you use LDAP for user and group management, you must add the LDAP configuration to your Sidekiq node as well as the LDAP +synchronization worker. If the LDAP configuration and LDAP synchronization worker are not applied to your Sidekiq node, +users and groups are not automatically synchronized. + +For more information about configuring LDAP for GitLab, see: + +- [GitLab LDAP configuration documentation](auth/ldap/index.md#configure-ldap) +- [LDAP synchronization documentation](auth/ldap/ldap_synchronization.md#adjust-ldap-user-sync-schedule) + +To enable LDAP with the synchronization worker for Sidekiq: + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['ldap_enabled'] = true + gitlab_rails['prevent_ldap_sign_in'] = false + gitlab_rails['ldap_servers'] = { + 'main' => { + 'label' => 'LDAP', + 'host' => 'ldap.mydomain.com', + 'port' => 389, + 'uid' => 'sAMAccountName', + 'encryption' => 'simple_tls', + 'verify_certificates' => true, + 'bind_dn' => '_the_full_dn_of_the_user_you_will_bind_with', + 'password' => '_the_password_of_the_bind_user', + 'tls_options' => { + 'ca_file' => '', + 'ssl_version' => '', + 'ciphers' => '', + 'cert' => '', + 'key' => '' + }, + 'timeout' => 10, + 'active_directory' => true, + 'allow_username_or_email_login' => false, + 'block_auto_created_users' => false, + 'base' => 'dc=example,dc=com', + 'user_filter' => '', + 'attributes' => { + 'username' => ['uid', 'userid', 'sAMAccountName'], + 'email' => ['mail', 'email', 'userPrincipalName'], + 'name' => 'cn', + 'first_name' => 'givenName', + 'last_name' => 'sn' + }, + 'lowercase_usernames' => false, + + # Enterprise Edition only + # https://docs.gitlab.com/ee/administration/auth/ldap/ldap_synchronization.html + 'group_base' => '', + 'admin_group' => '', + 'external_groups' => [], + 'sync_ssh_keys' => false + } + } + gitlab_rails['ldap_sync_worker_cron'] = "0 */12 * * *" + ``` + +1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + ## Related topics - [Extra Sidekiq processes](operations/extra_sidekiq_processes.md) diff --git a/doc/administration/terraform_state.md b/doc/administration/terraform_state.md index d1113125c4e..5e8f7cb18bb 100644 --- a/doc/administration/terraform_state.md +++ b/doc/administration/terraform_state.md @@ -141,10 +141,10 @@ You can optionally track progress and verify that all packages migrated successf - `sudo gitlab-rails dbconsole` for Omnibus GitLab instances. - `sudo -u git -H psql -d gitlabhq_production` for source-installed instances. -Verify `objectstg` below (where `store=2`) has count of all states: +Verify `objectstg` below (where `file_store=2`) has count of all states: ```shell -gitlabhq_production=# SELECT count(*) AS total, sum(case when store = '1' then 1 else 0 end) AS filesystem, sum(case when store = '2' then 1 else 0 end) AS objectstg FROM terraform_states; +gitlabhq_production=# SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM terraform_states; total | filesystem | objectstg ------+------------+----------- diff --git a/doc/administration/troubleshooting/debug.md b/doc/administration/troubleshooting/debug.md index 744e12d4da1..0520ce470cb 100644 --- a/doc/administration/troubleshooting/debug.md +++ b/doc/administration/troubleshooting/debug.md @@ -225,7 +225,7 @@ gitlab_rails['env'] = { ``` For source installations, set the environment variable. -Refer to [Puma Worker timeout](../operations/puma.md#worker-timeout). +Refer to [Puma Worker timeout](../operations/puma.md#change-the-worker-timeout). [Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab for the changes to take effect. diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index 265c5278fd6..1c948771f5b 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -771,6 +771,26 @@ group.require_two_factor_authentication=false group.save ``` +### Check and toggle a feature for all projects in a group + +```ruby +projects = Group.find_by_name('_group_name').projects +projects.each do |p| + state = p.<feature-name>? + + if state + puts "#{p.name} has <feature-name> already enabled. Skipping..." + else + puts "#{p.name} didn't have <feature-name> enabled. Enabling..." + p.project_feature.update!(builds_access_level: ProjectFeature::PRIVATE) + end +end +``` + +To find features that can be toggled, run `pp p.project_feature`. +Available permission levels are listed in +[concerns/featurable.rb](https://gitlab.com/gitlab-org/gitlab/blob/master/app/models/concerns/featurable.rb). + ## Authentication ### Re-enable standard web sign-in form @@ -1023,7 +1043,7 @@ This is needed for example in a known edge-case with ### Remove licenses -To clean up the [License History table](../../user/admin_area/license.md#view-license-details-and-history): +To clean up the [License History table](../../user/admin_area/license_file.md#view-license-details-and-history): ```ruby TYPE = :trial? diff --git a/doc/administration/troubleshooting/img/AzureAD-scim_provisioning.png b/doc/administration/troubleshooting/img/AzureAD-scim_provisioning.png Binary files differindex b8edcfa31c2..b2c385151a6 100644 --- a/doc/administration/troubleshooting/img/AzureAD-scim_provisioning.png +++ b/doc/administration/troubleshooting/img/AzureAD-scim_provisioning.png diff --git a/doc/administration/troubleshooting/linux_cheat_sheet.md b/doc/administration/troubleshooting/linux_cheat_sheet.md index b66e88a8d60..913437c2d77 100644 --- a/doc/administration/troubleshooting/linux_cheat_sheet.md +++ b/doc/administration/troubleshooting/linux_cheat_sheet.md @@ -14,7 +14,7 @@ having an issue with GitLab, you may want to check your [support options](https: first, before attempting to use this information. WARNING: -If you administer GitLab you are expected to know these commands for your distribution +It is [beyond the scope of GitLab Support to assist in systems administration](https://about.gitlab.com/support/statement-of-support.html#training). GitLab administrators are expected to know these commands for their distribution of choice. If you are a GitLab Support Engineer, consider this a cross-reference to translate `yum` -> `apt-get` and the like. diff --git a/doc/administration/troubleshooting/postgresql.md b/doc/administration/troubleshooting/postgresql.md index 47fd424b1fd..d0ed3c5c12a 100644 --- a/doc/administration/troubleshooting/postgresql.md +++ b/doc/administration/troubleshooting/postgresql.md @@ -107,6 +107,10 @@ This section is for links to information elsewhere in the GitLab documentation. To fix this, see [Backup and restore a non-packaged PostgreSQL database](https://docs.gitlab.com/omnibus/settings/database.html#backup-and-restore-a-non-packaged-postgresql-database). +- Deploying PostgreSQL on Azure Database for PostgreSQL - Flexible Server may result in an error stating `extension "btree_gist" is not allow-listed for "azure_pg_admin" users in Azure Database for PostgreSQL` + + To resolve the above error, [allow-list the extension](https://docs.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-extensions#how-to-use-postgresql-extensions) prior to install. + ## Support topics ### Database deadlocks diff --git a/doc/administration/uploads.md b/doc/administration/uploads.md index 55c3f85bfb9..aba7b5fc318 100644 --- a/doc/administration/uploads.md +++ b/doc/administration/uploads.md @@ -67,7 +67,7 @@ For source installations the following settings are nested under `uploads:` and |---------|-------------|---------| | `enabled` | Enable/disable object storage | `false` | | `remote_directory` | The bucket name where Uploads will be stored| | -| `direct_upload` | Set to `true` to remove Puma from the Upload path. Workhorse handles the actual Artifact Upload to Object Storage while Puma does minimal processing to keep track of the upload. There is no need for local shared storage. The option may be removed if support for a single storage type for all files is introduced. Read more on [direct upload](../development/uploads.md#direct-upload). | `false` | +| `direct_upload` | Set to `true` to remove Puma from the Upload path. Workhorse handles the actual Artifact Upload to Object Storage while Puma does minimal processing to keep track of the upload. There is no need for local shared storage. The option may be removed if support for a single storage type for all files is introduced. Read more on [direct upload](../development/uploads/implementation.md#direct-upload). | `false` | | `background_upload` | Set to `false` to disable automatic upload. Option may be removed once upload is direct to S3 (if `direct_upload` is set to `true` it will override `background_upload`) | `true` | | `proxy_download` | Set to `true` to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` | | `connection` | Various connection options described below | | |
