diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2020-10-21 07:08:36 +0000 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2020-10-21 07:08:36 +0000 |
| commit | 48aff82709769b098321c738f3444b9bdaa694c6 (patch) | |
| tree | e00c7c43e2d9b603a5a6af576b1685e400410dee /doc/administration | |
| parent | 879f5329ee916a948223f8f43d77fba4da6cd028 (diff) | |
| download | gitlab-ce-48aff82709769b098321c738f3444b9bdaa694c6.tar.gz | |
Add latest changes from gitlab-org/gitlab@13-5-stable-eev13.5.0-rc42
Diffstat (limited to 'doc/administration')
137 files changed, 3468 insertions, 2442 deletions
diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md index 099346b2b0b..ac972e2e33e 100644 --- a/doc/administration/audit_events.md +++ b/doc/administration/audit_events.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w GitLab offers a way to view the changes made within the GitLab server for owners and administrators on a [paid plan](https://about.gitlab.com/pricing/). GitLab system administrators can also take advantage of the logs located on the -filesystem. See [the logs system documentation](logs.md) for more details. +file system. See [the logs system documentation](logs.md) for more details. ## Overview @@ -108,7 +108,7 @@ Server-wide audit logging introduces the ability to observe user actions across the entire instance of your GitLab server, making it easy to understand who changed what and when for audit purposes. -To view the server-wide admin log, visit **Admin Area > Monitoring > Audit Log**. +To view the server-wide administrator log, visit **Admin Area > Monitoring > Audit Log**. In addition to the group and project events, the following user actions are also recorded: @@ -126,6 +126,7 @@ recorded: - User was added ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/251) in GitLab 12.8) - User was blocked via Admin Area ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/251) in GitLab 12.8) - User was blocked via API ([introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25872) in GitLab 12.9) +- Failed second-factor authentication attempt ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/16826) in GitLab 13.5) It's possible to filter particular actions by choosing an audit data type from the filter dropdown box. You can further filter by specific group, project, or user @@ -151,7 +152,7 @@ on adding these events into GitLab: The current architecture of audit events is not prepared to receive a very high amount of records. It may make the user interface for your project or audit logs very busy, and the disk space consumed by the -`audit_events` PostgreSQL table will increase considerably. It's disabled by default +`audit_events` PostgreSQL table may increase considerably. It's disabled by default to prevent performance degradations on GitLab instances with very high Git write traffic. In an upcoming release, Audit Logs for Git push events will be enabled @@ -172,6 +173,7 @@ the steps bellow. ```ruby Feature.enable(:repository_push_audit_event) + ``` ## Export to CSV **(PREMIUM ONLY)** @@ -183,6 +185,7 @@ the steps bellow. CAUTION: **Warning:** This feature might not be available to you. Check the **version history** note above for details. +If available, you can enable it with a [feature flag](#enable-or-disable-audit-log-export-to-csv). Export to CSV allows customers to export the current filter view of your audit log as a CSV file, diff --git a/doc/administration/auditor_users.md b/doc/administration/auditor_users.md index ace210183b2..c41065abd17 100644 --- a/doc/administration/auditor_users.md +++ b/doc/administration/auditor_users.md @@ -1,34 +1,39 @@ -# Auditor users **(PREMIUM ONLY)** +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- ->[Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/998) in [GitLab Premium](https://about.gitlab.com/pricing/) 8.17. +# Auditor users **(PREMIUM ONLY)** Auditor users are given read-only access to all projects, groups, and other resources on the GitLab instance. ## Overview -Auditor users can have full access to their own resources (projects, groups, -snippets, etc.), and read-only access to **all** other resources, except the -Admin Area. To put another way, they are just regular users (who can be added -to projects, create personal snippets, create milestones on their groups, etc.) -who also happen to have read-only access to all projects on the system that -they haven't been explicitly [given access](../user/permissions.md) to. +Auditor users are able to have both full access to their own resources +(including projects, groups, and snippets) and read-only access to _all_ other +resources, except the [Admin Area](../user/admin_area/index.md). These user +accounts are regular users who can be added to projects, create personal +snippets, and create milestones on their groups, while also having read-only +access to all projects on the server to which they haven't been explicitly +[given access](../user/permissions.md). The Auditor role is _not_ a read-only version of the Admin role. Auditor users -will not be able to access the project/group settings pages, or the Admin Area. +can't access the project or group settings pages, or the Admin Area. -To sum up, assuming you have logged-in as an Auditor user: +Assuming you have signed in as an Auditor user: - For a project the Auditor is not member of, the Auditor should have - read-only access. If the project is public or internal, they would have the - same access as the users that are not members of that project/group. + read-only access. If the project is public or internal, they have the same + access as users that aren't members of that project or group. - For a project the Auditor owns, the Auditor should have full access to everything. -- For a project the Auditor has been added to as a member, the Auditor should - have the same access as the [permissions](../user/permissions.md) they were given to. For example, if - they were added as a Developer, they could then push commits or comment on - issues. -- The Auditor cannot view the Admin Area, or perform any admin actions. +- For a project to which the Auditor is added as a member, the Auditor should + have the same access as their given [permissions](../user/permissions.md). + For example, if they were added as a Developer, they can push commits or + comment on issues. +- The Auditor can't view the Admin Area, or perform any admin actions. For more information about what an Auditor can or can't do, see the [Permissions and restrictions of an Auditor user](#permissions-and-restrictions-of-an-auditor-user) @@ -36,33 +41,37 @@ section. ## Use cases -1. Your compliance department wants to run tests against the entire GitLab base - to ensure users are complying with password, credit card, and other sensitive - data policies. With Auditor users, this can be achieved very easily without - resulting to tactics like giving a user admin rights or having to use the API - to add them to all projects. -1. If particular users need visibility or access to most of all projects in - your GitLab instance, instead of manually adding the user to all projects, - you can simply create an Auditor user and share the credentials with those - that you want to grant access to. +The following use cases describe some situations where Auditor users could be +helpful: + +- Your compliance department wants to run tests against the entire GitLab base + to ensure users are complying with password, credit card, and other sensitive + data policies. With Auditor users, this can be achieved very without having + to give them user admin rights or using the API to add them to all projects. +- If particular users need visibility or access to most of all projects in + your GitLab instance, instead of manually adding the user to all projects, + you can create an Auditor user and then share the credentials with those users + to which you want to grant access. ## Adding an Auditor user +To create a new Auditor user: + 1. Create a new user or edit an existing one by navigating to - **Admin Area > Users**. You will find the option of the access level under + **Admin Area > Users**. You will find the option of the access level in the 'Access' section.  -1. Click **Save changes** or **Create user** for the changes to take effect. +1. Select **Save changes** or **Create user** for the changes to take effect. -To revoke the Auditor permissions from a user, simply make them a Regular user -following the same steps as above. +To revoke Auditor permissions from a user, make them a regular user by +following the previous steps. ## Permissions and restrictions of an Auditor user An Auditor user should be able to access all projects and groups of a GitLab -instance, with the following permissions/restrictions: +instance, with the following permissions and restrictions: - Has read-only access to the API - Can access projects that are: @@ -70,15 +79,15 @@ instance, with the following permissions/restrictions: - Public - Internal - Can read all files in a repository -- Can read issues / MRs +- Can read issues and MRs - Can read project snippets - Cannot be Admin and Auditor at the same time - Cannot access the Admin Area -- In a group / project they're not a member of: +- In a group or project they're not a member of: - Cannot access project settings - Cannot access group settings - Cannot commit to repository - - Cannot create / comment on issues / MRs - - Cannot create/modify files from the Web UI + - Cannot create or comment on issues and MRs + - Cannot create or modify files from the Web UI - Cannot merge a merge request - Cannot create project snippets diff --git a/doc/administration/auth/README.md b/doc/administration/auth/README.md index 926a4abab7d..cf82454cfd2 100644 --- a/doc/administration/auth/README.md +++ b/doc/administration/auth/README.md @@ -18,7 +18,7 @@ providers: - [Azure](../../integration/azure.md) - [Bitbucket Cloud](../../integration/bitbucket.md) - [CAS](../../integration/cas.md) -- [Crowd](../../integration/crowd.md) +- [Crowd](crowd.md) - [Facebook](../../integration/facebook.md) - [GitHub](../../integration/github.md) - [GitLab.com](../../integration/gitlab.md) diff --git a/doc/administration/auth/ldap/index.md b/doc/administration/auth/ldap/index.md index 1dac098ec0c..3df85babc94 100644 --- a/doc/administration/auth/ldap/index.md +++ b/doc/administration/auth/ldap/index.md @@ -12,6 +12,7 @@ GitLab integrates with LDAP to support user authentication. This integration works with most LDAP-compliant directory servers, including: - Microsoft Active Directory + - [Microsoft Active Directory Trusts](https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2008-R2-and-2008/cc771568(v=ws.10)) are not supported. - Apple Open Directory - Open LDAP - 389 Server @@ -21,9 +22,6 @@ Users added through LDAP take a [licensed seat](../../../subscriptions/self_mana GitLab Enterprise Editions (EE) include enhanced integration, including group membership syncing as well as multiple LDAP servers support. -NOTE: **Note:** -[Microsoft Active Directory Trusts](https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2008-R2-and-2008/cc771568(v=ws.10)) are not supported. - ## Overview [LDAP](https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol) @@ -55,9 +53,8 @@ are already logged in or are using Git over SSH will still be able to access GitLab for up to one hour. Manually block the user in the GitLab Admin Area to immediately block all access. -NOTE: **Note:** GitLab Enterprise Edition Starter supports a -[configurable sync time](#adjusting-ldap-user-sync-schedule). +[configurable sync time](#adjusting-ldap-user-sync-schedule). **(STARTER)** ## Git password authentication **(CORE ONLY)** @@ -100,7 +97,6 @@ library. `start_tls` corresponds to StartTLS, not to be confused with regular TL Normally, if you specify `simple_tls` it will be on port 636, while `start_tls` (StartTLS) would be on port 389. `plain` also operates on port 389. Removed values: `tls` was replaced with `start_tls` and `ssl` was replaced with `simple_tls`. -NOTE: **Note:** LDAP users must have an email address set, regardless of whether it is used to sign-in. ### Example Configurations **(CORE ONLY)** @@ -120,7 +116,6 @@ gitlab_rails['ldap_servers'] = { 'verify_certificates' => true, 'bind_dn' => '_the_full_dn_of_the_user_you_will_bind_with', 'password' => '_the_password_of_the_bind_user', - 'encryption' => 'plain', 'verify_certificates' => true, 'tls_options' => { 'ca_file' => '', @@ -430,8 +425,7 @@ gitlab_rails['ldap_servers'] = { } ``` -NOTE: **Note:** -Any number of LDAP servers can be configured. However, make sure to use a unique naming convention for the `label` section of each entry as this will be the display name of the tab shown on the sign-in page. +If you configure multiple LDAP servers, use a unique naming convention for the `label` section of each entry. That label is used as the display name of the tab shown on the sign-in page. ## User sync **(STARTER ONLY)** @@ -445,11 +439,10 @@ The process executes the following access checks: blocked/disabled state). This will only be checked if `active_directory: true` is set in the LDAP configuration. -NOTE: **Note:** In Active Directory, a user is marked as disabled/blocked if the user account control attribute (`userAccountControl:1.2.840.113556.1.4.803`) -has bit 2 set. See <https://ctovswild.com/2009/09/03/bitmask-searches-in-ldap/> -for more information. +has bit 2 set. +For more information, see <https://ctovswild.com/2009/09/03/bitmask-searches-in-ldap/> The user will be set to `ldap_blocked` state in GitLab if the above conditions fail. This means the user will not be able to sign-in or push/pull code. @@ -460,8 +453,10 @@ The process will also update the following user information: - If `sync_ssh_keys` is set, SSH public keys. - If Kerberos is enabled, Kerberos identity. -NOTE: **Note:** -The LDAP sync process updates existing users while new users are created on first sign in. +The LDAP sync process: + +- Updates existing users. +- Creates new users on first sign in. ### Adjusting LDAP user sync schedule **(STARTER ONLY)** @@ -469,11 +464,13 @@ NOTE: **Note:** These are cron formatted values. You can use a crontab generator to create these values, for example <http://www.crontabgenerator.com/>. -By default, GitLab will run a worker once per day at 01:30 a.m. server time to +By default, GitLab runs a worker once per day at 01:30 a.m. server time to check and update GitLab users against LDAP. You can manually configure LDAP user sync times by setting the -following configuration values. The example below shows how to set LDAP user +following configuration values, in cron format. If needed, you can +use a [crontab generator](http://crontabgenerator.com). +The example below shows how to set LDAP user sync to run once every 12 hours at the top of the hour. **Omnibus installations** @@ -512,7 +509,7 @@ GitLab group membership to be automatically updated based on LDAP group members. The `group_base` configuration should be a base LDAP 'container', such as an 'organization' or 'organizational unit', that contains LDAP groups that should be available to GitLab. For example, `group_base` could be -`ou=groups,dc=example,dc=com`. In the config file it will look like the +`ou=groups,dc=example,dc=com`. In the configuration file it will look like the following. **Omnibus configuration** @@ -617,14 +614,12 @@ To enable it you need to: ### Adjusting LDAP group sync schedule **(STARTER ONLY)** -NOTE: **Note:** -These are cron formatted values. You can use a crontab generator to create -these values, for example [Crontab Generator](http://www.crontabgenerator.com/). - By default, GitLab runs a group sync process every hour, on the hour. +The values shown are in cron format. If needed, you can use a +[Crontab Generator](http://www.crontabgenerator.com). CAUTION: **Important:** -It's recommended that you do not start the sync process too frequently as this +Do not start the sync process too frequently as this could lead to multiple syncs running concurrently. This is primarily a concern for installations with a large number of LDAP users. Please review the [LDAP group sync benchmark metrics](#benchmarks) to see how @@ -727,7 +722,8 @@ Other LDAP servers should work, too. Active Directory also supports nested groups. Group sync will recursively resolve membership if `active_directory: true` is set in the configuration file. -NOTE: **Note:** +##### Nested group memberships + Nested group memberships are resolved only if the nested group is found within the configured `group_base`. For example, if GitLab sees a nested group with DN `cn=nested_group,ou=special_groups,dc=example,dc=com` but diff --git a/doc/administration/auth/ldap/ldap-troubleshooting.md b/doc/administration/auth/ldap/ldap-troubleshooting.md index 3d3ac124ac4..c6558bf1791 100644 --- a/doc/administration/auth/ldap/ldap-troubleshooting.md +++ b/doc/administration/auth/ldap/ldap-troubleshooting.md @@ -682,7 +682,7 @@ The rails console is a valuable tool to help debug LDAP problems. It allows you directly interact with the application by running commands and seeing how GitLab responds to them. -Please refer to [this guide](../../troubleshooting/debug.md#starting-a-rails-console-session) +Please refer to [this guide](../../operations/rails_console.md#starting-a-rails-console-session) for instructions on how to use the rails console. #### Enable debug output diff --git a/doc/administration/auth/smartcard.md b/doc/administration/auth/smartcard.md index 0ecf3ca090d..a696d0499a4 100644 --- a/doc/administration/auth/smartcard.md +++ b/doc/administration/auth/smartcard.md @@ -1,4 +1,7 @@ --- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- diff --git a/doc/administration/compliance.md b/doc/administration/compliance.md index 237261f2567..44270283308 100644 --- a/doc/administration/compliance.md +++ b/doc/administration/compliance.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Compliance features You can configure the following GitLab features to help ensure that your GitLab instance meets common compliance standards. Click a feature name for further documentation. diff --git a/doc/administration/consul.md b/doc/administration/consul.md index ae22d8bd4d0..4eed020c284 100644 --- a/doc/administration/consul.md +++ b/doc/administration/consul.md @@ -1,4 +1,7 @@ --- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- diff --git a/doc/administration/database_load_balancing.md b/doc/administration/database_load_balancing.md index 36ef905fd90..e88f88a0427 100644 --- a/doc/administration/database_load_balancing.md +++ b/doc/administration/database_load_balancing.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Database Load Balancing **(PREMIUM ONLY)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/1283) in [GitLab Premium](https://about.gitlab.com/pricing/) 9.0. @@ -128,7 +134,7 @@ production: disconnect_timeout: 120 ``` -Here the `discover:` section specifies the configuration details to use for +Here, the `discover:` section specifies the configuration details to use for service discovery. ### Configuration @@ -139,7 +145,7 @@ The following options can be set: |----------------------|---------------------------------------------------------------------------------------------------|-----------| | `nameserver` | The nameserver to use for looking up the DNS record. | localhost | | `record` | The record to look up. This option is required for service discovery to work. | | -| `record_type` | Optional record type to look up, this can be either A or SRV (since GitLab 12.3) | A | +| `record_type` | Optional record type to look up, this can be either A or SRV (GitLab 12.3 and later) | A | | `port` | The port of the nameserver. | 8600 | | `interval` | The minimum time in seconds between checking the DNS record. | 60 | | `disconnect_timeout` | The time in seconds after which an old connection is closed, after the list of hosts was updated. | 120 | diff --git a/doc/administration/environment_variables.md b/doc/administration/environment_variables.md index d48a47e9645..25bfc3c132d 100644 --- a/doc/administration/environment_variables.md +++ b/doc/administration/environment_variables.md @@ -1,77 +1,75 @@ --- -stage: Verify -group: Continuous Integration +stage: Enablement +group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- -# Environment Variables +# Environment variables GitLab exposes certain environment variables which can be used to override their defaults values. -People usually configure GitLab via `/etc/gitlab/gitlab.rb` for Omnibus +People usually configure GitLab with `/etc/gitlab/gitlab.rb` for Omnibus installations, or `gitlab.yml` for installations from source. -Below you will find the supported environment variables which you can use to -override certain values. +You can use the following environment variables to override certain values: ## Supported environment variables -Variable | Type | Description --------- | ---- | ----------- -`ENABLE_BOOTSNAP` | string | Enables Bootsnap for speeding up initial Rails boot (`1` to enable) -`GITLAB_CDN_HOST` | string | Sets the base URL for a CDN to serve static assets (e.g. `//mycdnsubdomain.fictional-cdn.com`) -`GITLAB_ROOT_PASSWORD` | string | Sets the password for the `root` user on installation -`GITLAB_HOST` | string | The full URL of the GitLab server (including `http://` or `https://`) -`RAILS_ENV` | string | The Rails environment; can be one of `production`, `development`, `staging` or `test` -`DATABASE_URL` | string | The database URL; is of the form: `postgresql://localhost/blog_development` -`GITLAB_EMAIL_FROM` | string | The e-mail address used in the "From" field in e-mails sent by GitLab -`GITLAB_EMAIL_DISPLAY_NAME` | string | The name used in the "From" field in e-mails sent by GitLab -`GITLAB_EMAIL_REPLY_TO` | string | The e-mail address used in the "Reply-To" field in e-mails sent by GitLab -`GITLAB_EMAIL_SUBJECT_SUFFIX` | string | The e-mail subject suffix used in e-mails sent by GitLab -`GITLAB_UNICORN_MEMORY_MIN` | integer | The minimum memory threshold (in bytes) for the Unicorn worker killer -`GITLAB_UNICORN_MEMORY_MAX` | integer | The maximum memory threshold (in bytes) for the Unicorn worker killer -`GITLAB_SHARED_RUNNERS_REGISTRATION_TOKEN` | string | Sets the initial registration token used for runners -`UNSTRUCTURED_RAILS_LOG` | string | Enables the unstructured log in addition to JSON logs (defaults to `true`) +| Variable | Type | Description | +|--------------------------------------------|---------|---------------------------------------------------------------------------------------------------------| +| `DATABASE_URL` | string | The database URL; is of the form: `postgresql://localhost/blog_development`. | +| `ENABLE_BOOTSNAP` | string | Enables Bootsnap for speeding up initial Rails boot (`1` to enable). | +| `GITLAB_CDN_HOST` | string | Sets the base URL for a CDN to serve static assets (for example, `//mycdnsubdomain.fictional-cdn.com`). | +| `GITLAB_EMAIL_DISPLAY_NAME` | string | The name used in the **From** field in emails sent by GitLab. | +| `GITLAB_EMAIL_FROM` | string | The email address used in the **From** field in emails sent by GitLab. | +| `GITLAB_EMAIL_REPLY_TO` | string | The email address used in the **Reply-To** field in emails sent by GitLab. | +| `GITLAB_EMAIL_SUBJECT_SUFFIX` | string | The email subject suffix used in emails sent by GitLab. | +| `GITLAB_HOST` | string | The full URL of the GitLab server (including `http://` or `https://`). | +| `GITLAB_ROOT_PASSWORD` | string | Sets the password for the `root` user on installation. | +| `GITLAB_SHARED_RUNNERS_REGISTRATION_TOKEN` | string | Sets the initial registration token used for runners. | +| `GITLAB_UNICORN_MEMORY_MAX` | integer | The maximum memory threshold (in bytes) for the [unicorn-worker-killer](operations/unicorn.md#unicorn-worker-killer). | +| `GITLAB_UNICORN_MEMORY_MIN` | integer | The minimum memory threshold (in bytes) for the [unicorn-worker-killer](operations/unicorn.md#unicorn-worker-killer). | +| `RAILS_ENV` | string | The Rails environment; can be one of `production`, `development`, `staging`, or `test`. | +| `UNSTRUCTURED_RAILS_LOG` | string | Enables the unstructured log in addition to JSON logs (defaults to `true`). | ## Complete database variables -The recommended way of specifying your database connection information is to set -the `DATABASE_URL` environment variable. This variable only holds connection -information (`adapter`, `database`, `username`, `password`, `host` and `port`), -but not behavior information (`encoding`, `pool`). If you don't want to use -`DATABASE_URL` and/or want to set database behavior information, you will have -to either: +The recommended method for specifying your database connection information is +to set the `DATABASE_URL` environment variable. This variable contains +connection information (`adapter`, `database`, `username`, `password`, `host`, +and `port`), but no behavior information (`encoding` or `pool`). If you don't +want to use `DATABASE_URL`, or want to set database behavior information, +either: -- copy our template file: `cp config/database.yml.env config/database.yml`, or -- set a value for some `GITLAB_DATABASE_XXX` variables +- Copy the template file, `cp config/database.yml.env config/database.yml`. +- Set a value for some `GITLAB_DATABASE_XXX` variables. The list of `GITLAB_DATABASE_XXX` variables that you can set is: -Variable | Default value | Overridden by `DATABASE_URL`? --------- | ------------- | ----------------------------- -`GITLAB_DATABASE_ADAPTER` | `postgresql` | Yes -`GITLAB_DATABASE_DATABASE` | `gitlab_#{ENV['RAILS_ENV']` | Yes -`GITLAB_DATABASE_USERNAME` | `root` | Yes -`GITLAB_DATABASE_PASSWORD` | None | Yes -`GITLAB_DATABASE_HOST` | `localhost` | Yes -`GITLAB_DATABASE_PORT` | `5432` | Yes -`GITLAB_DATABASE_ENCODING` | `unicode` | No -`GITLAB_DATABASE_POOL` | `10` | No +| Variable | Default value | Overridden by `DATABASE_URL`? | +|-----------------------------|--------------------------------|-------------------------------| +| `GITLAB_DATABASE_ADAPTER` | `postgresql` | **{check-circle}** Yes | +| `GITLAB_DATABASE_DATABASE` | `gitlab_#{ENV['RAILS_ENV']` | **{check-circle}** Yes | +| `GITLAB_DATABASE_ENCODING` | `unicode` | **{dotted-circle}** No | +| `GITLAB_DATABASE_HOST` | `localhost` | **{check-circle}** Yes | +| `GITLAB_DATABASE_PASSWORD` | _none_ | **{check-circle}** Yes | +| `GITLAB_DATABASE_POOL` | `10` | **{dotted-circle}** No | +| `GITLAB_DATABASE_PORT` | `5432` | **{check-circle}** Yes | +| `GITLAB_DATABASE_USERNAME` | `root` | **{check-circle}** Yes | ## Adding more variables -We welcome merge requests to make more settings configurable via variables. -Please make changes in the `config/initializers/1_settings.rb` file and stick -to the naming scheme `GITLAB_#{name in 1_settings.rb in upper case}`. +We welcome merge requests to make more settings configurable by using variables. +Make changes to the `config/initializers/1_settings.rb` file, and use the +naming scheme `GITLAB_#{name in 1_settings.rb in upper case}`. ## Omnibus configuration -To set environment variables, follow [these -instructions](https://docs.gitlab.com/omnibus/settings/environment-variables.html). +To set environment variables, follow [these instructions](https://docs.gitlab.com/omnibus/settings/environment-variables.html). It's possible to preconfigure the GitLab Docker image by adding the environment variable `GITLAB_OMNIBUS_CONFIG` to the `docker run` command. -For more information see the [Pre-configure Docker container](https://docs.gitlab.com/omnibus/docker/#pre-configure-docker-container) -section in the Omnibus documentation. +For more information, see the [Pre-configure Docker container](https://docs.gitlab.com/omnibus/docker/#pre-configure-docker-container) +section of the Omnibus GitLab documentation. diff --git a/doc/administration/feature_flags.md b/doc/administration/feature_flags.md index a8a14063f26..4129677f134 100644 --- a/doc/administration/feature_flags.md +++ b/doc/administration/feature_flags.md @@ -65,7 +65,7 @@ For installations from the source: sudo -u git -H bundle exec rails console -e production ``` -For details, see [starting a Rails console session](troubleshooting/debug.md#starting-a-rails-console-session). +For details, see [starting a Rails console session](operations/rails_console.md#starting-a-rails-console-session). ### Enable or disable the feature @@ -79,10 +79,10 @@ To enable a feature, run: Feature.enable(:<feature flag>) ``` -Example, to enable Evidence Collection: +Example, to enable a fictional feature flag named `my_awesome_feature`: ```ruby -Feature.enable(:release_evidence_collection) +Feature.enable(:my_awesome_feature) ``` To disable a feature, run: @@ -91,10 +91,10 @@ To disable a feature, run: Feature.disable(:<feature flag>) ``` -Example, to disable Evidence Collection: +Example, to disable a fictional feature flag named `my_awesome_feature`: ```ruby -Feature.disable(:release_evidence_collection) +Feature.disable(:my_awesome_feature) ``` Some feature flags can be enabled or disabled on a per project basis: @@ -112,18 +112,18 @@ Feature.enable(:product_analytics, Project.find(1234)) `Feature.enable` and `Feature.disable` always return `nil`, this is not an indication that the command failed: ```ruby -irb(main):001:0> Feature.enable(:release_evidence_collection) +irb(main):001:0> Feature.enable(:my_awesome_feature) => nil ``` -To check if a flag is enabled or disabled you can use `Feature.enabled?` or `Feature.disabled?`: +To check if a flag is enabled or disabled you can use `Feature.enabled?` or `Feature.disabled?`. For example, for a fictional feature flag named `my_awesome_feature`: ```ruby -Feature.enable(:release_evidence_collection) +Feature.enable(:my_awesome_feature) => nil -Feature.enabled?(:release_evidence_collection) +Feature.enabled?(:my_awesome_feature) => true -Feature.disabled?(:release_evidence_collection) +Feature.disabled?(:my_awesome_feature) => false ``` diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index 8862776ee1b..32b3558a51f 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -11,11 +11,11 @@ Geo replicates your database, your Git repositories, and few other assets. We will support and replicate more data in the future, that will enable you to failover with minimal effort, in a disaster situation. -See [Geo current limitations](../index.md#current-limitations) for more information. +See [Geo limitations](../index.md#limitations) for more information. CAUTION: **Warning:** Disaster recovery for multi-secondary configurations is in **Alpha**. -For the latest updates, check the [Disaster Recovery epic for complete maturity](https://gitlab.com/groups/gitlab-org/-/epics/590). +For the latest updates, check the [Disaster Recovery epic for complete maturity](https://gitlab.com/groups/gitlab-org/-/epics/590). Multi-secondary configurations require the complete re-synchronization and re-configuration of all non-promoted secondaries and will cause downtime. @@ -84,8 +84,8 @@ must disable the **primary** node. single recommendation. You may need to: - Reconfigure the load balancers. - - Change DNS records (for example, point the primary DNS record to the **secondary** - node in order to stop usage of the **primary** node). + - Change DNS records (for example, point the primary DNS record to the + **secondary** node to stop usage of the **primary** node). - Stop the virtual servers. - Block traffic through a firewall. - Revoke object storage permissions from the **primary** node. @@ -98,16 +98,16 @@ must disable the **primary** node. Note the following when promoting a secondary: -- If replication was paused on the secondary node, for example as a part of upgrading, - while you were running a version of GitLab lower than 13.4, you _must_ - [enable the node via the database](../replication/troubleshooting.md#while-promoting-the-secondary-i-got-an-error-activerecordrecordinvalid) +- If replication was paused on the secondary node (for example as a part of + upgrading, while you were running a version of GitLab earlier than 13.4), you + _must_ [enable the node by using the database](../replication/troubleshooting.md#message-activerecordrecordinvalid-validation-failed-enabled-geo-primary-node-cannot-be-disabled) before proceeding. - A new **secondary** should not be added at this time. If you want to add a new **secondary**, do this after you have completed the entire process of promoting the **secondary** to the **primary**. - If you encounter an `ActiveRecord::RecordInvalid: Validation failed: Name has already been taken` - error during this process, please read - [the troubleshooting advice](../replication/troubleshooting.md#fixing-errors-during-a-failover-or-when-promoting-a-secondary-to-a-primary-node). + error message during this process, for more information, see this + [troubleshooting advice](../replication/troubleshooting.md#fixing-errors-during-a-failover-or-when-promoting-a-secondary-to-a-primary-node). #### Promoting a **secondary** node running on a single machine @@ -120,6 +120,10 @@ Note the following when promoting a secondary: 1. Edit `/etc/gitlab/gitlab.rb` to reflect its new status as **primary** by removing any lines that enabled the `geo_secondary_role`: + Users of GitLab 13.5 or later can skip this step, due to the appropriate + roles being enabled or disabled during the promotion in the following + step. + ```ruby ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. geo_secondary_role['enable'] = true @@ -129,7 +133,10 @@ Note the following when promoting a secondary: ``` 1. Promote the **secondary** node to the **primary** node. - + +DANGER: **Danger:** +In GitLab 13.2 and later versions, promoting a secondary node to a primary while the secondary is paused fails. We are [investigating the issue](https://gitlab.com/gitlab-org/gitlab/-/issues/225173). Do not pause replication before promoting a secondary. If the node is paused, please resume before promoting. + To promote the secondary node to primary along with preflight checks: ```shell @@ -139,7 +146,7 @@ Note the following when promoting a secondary: If you have already run the [preflight checks](planned_failover.md#preflight-checks) separately or don't want to run them, you can skip preflight checks with: ```shell - gitlab-ctl promote-to-primary-node --skip-preflight-check + gitlab-ctl promote-to-primary-node --skip-preflight-checks ``` You can also promote the secondary node to primary **without any further confirmation**, even when preflight checks fail: @@ -159,6 +166,9 @@ conjunction with multiple servers, as it can only perform changes on a **secondary** with only a single machine. Instead, you must do this manually. +DANGER: **Danger:** +In GitLab 13.2 and later versions, promoting a secondary node to a primary while the secondary is paused fails. We are [investigating the issue](https://gitlab.com/gitlab-org/gitlab/-/issues/225173). Do not pause replication before promoting a secondary. If the node is paused, please resume before promoting. + 1. SSH in to the database node in the **secondary** and trigger PostgreSQL to promote to read-write: @@ -201,6 +211,9 @@ an external PostgreSQL database, as it can only perform changes on a **secondary node with GitLab and the database on the same machine. As a result, a manual process is required: +DANGER: **Danger:** +In GitLab 13.2 and later versions, promoting a secondary node to a primary while the secondary is paused fails. We are [investigating the issue](https://gitlab.com/gitlab-org/gitlab/-/issues/225173). Do not pause replication before promoting a secondary. If the node is paused, please resume before promoting. + 1. Promote the replica database associated with the **secondary** site. This will set the database to read-write: - Amazon RDS - [Promoting a Read Replica to Be a Standalone DB Instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html#USER_ReadRepl.Promote) diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md index 9b9c386652c..1238c4d8e2a 100644 --- a/doc/administration/geo/disaster_recovery/planned_failover.md +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -27,7 +27,7 @@ have a high degree of confidence in being able to perform them accurately. ## Not all data is automatically replicated -If you are using any GitLab features that Geo [doesn't support](../index.md#current-limitations), +If you are using any GitLab features that Geo [doesn't support](../index.md#limitations), you must make separate provisions to ensure that the **secondary** node has an up-to-date copy of any data associated with that feature. This may extend the required scheduled maintenance period significantly. diff --git a/doc/administration/geo/disaster_recovery/promotion_runbook.md b/doc/administration/geo/disaster_recovery/promotion_runbook.md index fb2353513df..7eb6ef01aee 100644 --- a/doc/administration/geo/disaster_recovery/promotion_runbook.md +++ b/doc/administration/geo/disaster_recovery/promotion_runbook.md @@ -1,269 +1,5 @@ --- -stage: Enablement -group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers -type: howto +redirect_to: runbooks/planned_failover_single_node.md --- -CAUTION: **Caution:** -This runbook is in **alpha**. For complete, production-ready documentation, see the -[disaster recovery documentation](index.md). - -# Disaster Recovery (Geo) promotion runbooks **(PREMIUM ONLY)** - -## Geo planned failover runbook 1 - -| Component | Configuration | -| ----------- | --------------- | -| PostgreSQL | Omnibus-managed | -| Geo site | Single-node | -| Secondaries | One | - -This runbook will guide you through a planned failover of a single-node Geo site -with one secondary. The following general architecture is assumed: - -```mermaid -graph TD - subgraph main[Geo deployment] - subgraph Primary[Primary site] - Node_1[(GitLab node)] - end - subgraph Secondary1[Secondary site] - Node_2[(GitLab node)] - end - end -``` - -This guide will result in the following: - -1. An offline primary. -1. A promoted secondary that is now the new primary. - -What is not covered: - -1. Re-adding the old **primary** as a secondary. -1. Adding a new secondary. - -### Preparation - -NOTE: **Note:** -Before following any of those steps, make sure you have `root` access to the -**secondary** to promote it, since there isn't provided an automated way to -promote a Geo replica and perform a failover. - -On the **secondary** node, navigate to the **Admin Area > Geo** dashboard to -review its status. Replicated objects (shown in green) should be close to 100%, -and there should be no failures (shown in red). If a large proportion of -objects aren't yet replicated (shown in gray), consider giving the node more -time to complete. - - - -If any objects are failing to replicate, this should be investigated before -scheduling the maintenance window. After a planned failover, anything that -failed to replicate will be **lost**. - -You can use the -[Geo status API](../../../api/geo_nodes.md#retrieve-project-sync-or-verification-failures-that-occurred-on-the-current-node) -to review failed objects and the reasons for failure. -A common cause of replication failures is the data being missing on the -**primary** node - you can resolve these failures by restoring the data from backup, -or removing references to the missing data. - -The maintenance window won't end until Geo replication and verification is -completely finished. To keep the window as short as possible, you should -ensure these processes are close to 100% as possible during active use. - -If the **secondary** node is still replicating data from the **primary** node, -follow these steps to avoid unnecessary data loss: - -1. Until a [read-only mode](https://gitlab.com/gitlab-org/gitlab/-/issues/14609) - is implemented, updates must be prevented from happening manually to the - **primary**. Note that your **secondary** node still needs read-only - access to the **primary** node during the maintenance window: - - 1. At the scheduled time, using your cloud provider or your node's firewall, block - all HTTP, HTTPS and SSH traffic to/from the **primary** node, **except** for your IP and - the **secondary** node's IP. - - For instance, you can run the following commands on the **primary** node: - - ```shell - sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 22 -j ACCEPT - sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 22 -j ACCEPT - sudo iptables -A INPUT --destination-port 22 -j REJECT - - sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 80 -j ACCEPT - sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 80 -j ACCEPT - sudo iptables -A INPUT --tcp-dport 80 -j REJECT - - sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 443 -j ACCEPT - sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 443 -j ACCEPT - sudo iptables -A INPUT --tcp-dport 443 -j REJECT - ``` - - From this point, users will be unable to view their data or make changes on the - **primary** node. They will also be unable to log in to the **secondary** node. - However, existing sessions will work for the remainder of the maintenance period, and - public data will be accessible throughout. - - 1. Verify the **primary** node is blocked to HTTP traffic by visiting it in browser via - another IP. The server should refuse connection. - - 1. Verify the **primary** node is blocked to Git over SSH traffic by attempting to pull an - existing Git repository with an SSH remote URL. The server should refuse - connection. - - 1. On the **primary** node, disable non-Geo periodic background jobs by navigating - to **Admin Area > Monitoring > Background Jobs > Cron**, clicking `Disable All`, - and then clicking `Enable` for the `geo_sidekiq_cron_config_worker` cron job. - This job will re-enable several other cron jobs that are essential for planned - failover to complete successfully. - -1. Finish replicating and verifying all data: - - CAUTION: **Caution:** - Not all data is automatically replicated. Read more about - [what is excluded](planned_failover.md#not-all-data-is-automatically-replicated). - - 1. If you are manually replicating any - [data not managed by Geo](../replication/datatypes.md#limitations-on-replicationverification), - trigger the final replication process now. - 1. On the **primary** node, navigate to **Admin Area > Monitoring > Background Jobs > Queues** - and wait for all queues except those with `geo` in the name to drop to 0. - These queues contain work that has been submitted by your users; failing over - before it is completed will cause the work to be lost. - 1. On the **primary** node, navigate to **Admin Area > Geo** and wait for the - following conditions to be true of the **secondary** node you are failing over to: - - All replication meters to each 100% replicated, 0% failures. - - All verification meters reach 100% verified, 0% failures. - - Database replication lag is 0ms. - - The Geo log cursor is up to date (0 events behind). - - 1. On the **secondary** node, navigate to **Admin Area > Monitoring > Background Jobs > Queues** - and wait for all the `geo` queues to drop to 0 queued and 0 running jobs. - 1. On the **secondary** node, use [these instructions](../../raketasks/check.md) - to verify the integrity of CI artifacts, LFS objects, and uploads in file - storage. - - At this point, your **secondary** node will contain an up-to-date copy of everything the - **primary** node has, meaning nothing will be lost when you fail over. - -1. In this final step, you need to permanently disable the **primary** node. - - CAUTION: **Caution:** - When the **primary** node goes offline, there may be data saved on the **primary** node - that has not been replicated to the **secondary** node. This data should be treated - as lost if you proceed. - - TIP: **Tip:** - If you plan to [update the **primary** domain DNS record](index.md#step-4-optional-updating-the-primary-domain-dns-record), - you may wish to lower the TTL now to speed up propagation. - - When performing a failover, we want to avoid a split-brain situation where - writes can occur in two different GitLab instances. So to prepare for the - failover, you must disable the **primary** node: - - - If you have SSH access to the **primary** node, stop and disable GitLab: - - ```shell - sudo gitlab-ctl stop - ``` - - Prevent GitLab from starting up again if the server unexpectedly reboots: - - ```shell - sudo systemctl disable gitlab-runsvdir - ``` - - NOTE: **Note:** - (**CentOS only**) In CentOS 6 or older, there is no easy way to prevent GitLab from being - started if the machine reboots isn't available (see [Omnibus GitLab issue #3058](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3058)). - It may be safest to uninstall the GitLab package completely with `sudo yum remove gitlab-ee`. - - NOTE: **Note:** - (**Ubuntu 14.04 LTS**) If you are using an older version of Ubuntu - or any other distribution based on the Upstart init system, you can prevent GitLab - from starting if the machine reboots as `root` with - `initctl stop gitlab-runsvvdir && echo 'manual' > /etc/init/gitlab-runsvdir.override && initctl reload-configuration`. - - - If you do not have SSH access to the **primary** node, take the machine offline and - prevent it from rebooting. Since there are many ways you may prefer to accomplish - this, we will avoid a single recommendation. You may need to: - - - Reconfigure the load balancers. - - Change DNS records (for example, point the **primary** DNS record to the **secondary** - node in order to stop usage of the **primary** node). - - Stop the virtual servers. - - Block traffic through a firewall. - - Revoke object storage permissions from the **primary** node. - - Physically disconnect a machine. - -### Promoting the **secondary** node - -Note the following when promoting a secondary: - -- A new **secondary** should not be added at this time. If you want to add a new - **secondary**, do this after you have completed the entire process of promoting - the **secondary** to the **primary**. -- If you encounter an `ActiveRecord::RecordInvalid: Validation failed: Name has already been taken` - error during this process, read - [the troubleshooting advice](../replication/troubleshooting.md#fixing-errors-during-a-failover-or-when-promoting-a-secondary-to-a-primary-node). - -To promote the secondary node: - -1. SSH in to your **secondary** node and login as root: - - ```shell - sudo -i - ``` - -1. Edit `/etc/gitlab/gitlab.rb` to reflect its new status as **primary** by - removing any lines that enabled the `geo_secondary_role`: - - ```ruby - ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. - geo_secondary_role['enable'] = true - - ## In 11.5+ documentation, the role was enabled as follows. Remove this line. - roles ['geo_secondary_role'] - ``` - -1. Run the following command to list out all preflight checks and automatically - check if replication and verification are complete before scheduling a planned - failover to ensure the process will go smoothly: - - ```shell - gitlab-ctl promotion-preflight-checks - ``` - -1. Promote the **secondary**: - - ```shell - gitlab-ctl promote-to-primary-node - ``` - - If you have already run the [preflight checks](planned_failover.md#preflight-checks) - or don't want to run them, you can skip them: - - ```shell - gitlab-ctl promote-to-primary-node --skip-preflight-check - ``` - - You can also promote the secondary node to primary **without any further confirmation**, even when preflight checks fail: - - ```shell - sudo gitlab-ctl promote-to-primary-node --force - ``` - -1. Verify you can connect to the newly promoted **primary** node using the URL used - previously for the **secondary** node. - - If successful, the **secondary** node has now been promoted to the **primary** node. - -### Next steps - -To regain geographic redundancy as quickly as possible, you should -[add a new **secondary** node](../setup/index.md). To -do that, you can re-add the old **primary** as a new secondary and bring it back -online. +This document was moved to [another location](runbooks/planned_failover_single_node.md). diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md new file mode 100644 index 00000000000..1e3bac0b354 --- /dev/null +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md @@ -0,0 +1,274 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: howto +--- + +CAUTION: **Caution:** +This runbook is in **alpha**. For complete, production-ready documentation, see the +[disaster recovery documentation](../index.md). + +# Disaster Recovery (Geo) promotion runbooks **(PREMIUM ONLY)** + +## Geo planned failover for a multi-node configuration + +| Component | Configuration | +|-------------|-----------------| +| PostgreSQL | Omnibus-managed | +| Geo site | Multi-node | +| Secondaries | One | + +This runbook will guide you through a planned failover of a multi-node Geo site +with one secondary. The following [2000 user reference architecture](../../../../administration/reference_architectures/2k_users.md) is assumed: + +```mermaid +graph TD + subgraph main[Geo deployment] + subgraph Primary[Primary site, multi-node] + Node_1[Rails node 1] + Node_2[Rails node 2] + Node_3[PostgreSQL node] + Node_4[Gitaly node] + Node_5[Redis node] + Node_6[Monitoring node] + end + subgraph Secondary[Secondary site, multi-node] + Node_7[Rails node 1] + Node_8[Rails node 2] + Node_9[PostgreSQL node] + Node_10[Gitaly node] + Node_11[Redis node] + Node_12[Monitoring node] + end + end +``` + +The load balancer node and optional NFS server are omitted for clarity. + +This guide will result in the following: + +1. An offline primary. +1. A promoted secondary that is now the new primary. + +What is not covered: + +1. Re-adding the old **primary** as a secondary. +1. Adding a new secondary. + +### Preparation + +NOTE: **Note:** +Before following any of those steps, make sure you have `root` access to the +**secondary** to promote it, since there isn't provided an automated way to +promote a Geo replica and perform a failover. + +On the **secondary** node, navigate to the **Admin Area > Geo** dashboard to +review its status. Replicated objects (shown in green) should be close to 100%, +and there should be no failures (shown in red). If a large proportion of +objects aren't yet replicated (shown in gray), consider giving the node more +time to complete. + + + +If any objects are failing to replicate, this should be investigated before +scheduling the maintenance window. After a planned failover, anything that +failed to replicate will be **lost**. + +You can use the +[Geo status API](../../../../api/geo_nodes.md#retrieve-project-sync-or-verification-failures-that-occurred-on-the-current-node) +to review failed objects and the reasons for failure. +A common cause of replication failures is the data being missing on the +**primary** node - you can resolve these failures by restoring the data from backup, +or removing references to the missing data. + +The maintenance window won't end until Geo replication and verification is +completely finished. To keep the window as short as possible, you should +ensure these processes are close to 100% as possible during active use. + +If the **secondary** node is still replicating data from the **primary** node, +follow these steps to avoid unnecessary data loss: + +1. Until a [read-only mode](https://gitlab.com/gitlab-org/gitlab/-/issues/14609) + is implemented, updates must be prevented from happening manually to the + **primary**. Note that your **secondary** node still needs read-only + access to the **primary** node during the maintenance window: + + 1. At the scheduled time, using your cloud provider or your node's firewall, block + all HTTP, HTTPS and SSH traffic to/from the **primary** node, **except** for your IP and + the **secondary** node's IP. + + For instance, you can run the following commands on the **primary** node: + + ```shell + sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 22 -j ACCEPT + sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 22 -j ACCEPT + sudo iptables -A INPUT --destination-port 22 -j REJECT + + sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 80 -j ACCEPT + sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 80 -j ACCEPT + sudo iptables -A INPUT --tcp-dport 80 -j REJECT + + sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 443 -j ACCEPT + sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 443 -j ACCEPT + sudo iptables -A INPUT --tcp-dport 443 -j REJECT + ``` + + From this point, users will be unable to view their data or make changes on the + **primary** node. They will also be unable to log in to the **secondary** node. + However, existing sessions will work for the remainder of the maintenance period, and + public data will be accessible throughout. + + 1. Verify the **primary** node is blocked to HTTP traffic by visiting it in browser via + another IP. The server should refuse connection. + + 1. Verify the **primary** node is blocked to Git over SSH traffic by attempting to pull an + existing Git repository with an SSH remote URL. The server should refuse + connection. + + 1. On the **primary** node, disable non-Geo periodic background jobs by navigating + to **Admin Area > Monitoring > Background Jobs > Cron**, clicking `Disable All`, + and then clicking `Enable` for the `geo_sidekiq_cron_config_worker` cron job. + This job will re-enable several other cron jobs that are essential for planned + failover to complete successfully. + +1. Finish replicating and verifying all data: + + CAUTION: **Caution:** + Not all data is automatically replicated. Read more about + [what is excluded](../planned_failover.md#not-all-data-is-automatically-replicated). + + 1. If you are manually replicating any + [data not managed by Geo](../../replication/datatypes.md#limitations-on-replicationverification), + trigger the final replication process now. + 1. On the **primary** node, navigate to **Admin Area > Monitoring > Background Jobs > Queues** + and wait for all queues except those with `geo` in the name to drop to 0. + These queues contain work that has been submitted by your users; failing over + before it is completed will cause the work to be lost. + 1. On the **primary** node, navigate to **Admin Area > Geo** and wait for the + following conditions to be true of the **secondary** node you are failing over to: + - All replication meters to each 100% replicated, 0% failures. + - All verification meters reach 100% verified, 0% failures. + - Database replication lag is 0ms. + - The Geo log cursor is up to date (0 events behind). + + 1. On the **secondary** node, navigate to **Admin Area > Monitoring > Background Jobs > Queues** + and wait for all the `geo` queues to drop to 0 queued and 0 running jobs. + 1. On the **secondary** node, use [these instructions](../../../raketasks/check.md) + to verify the integrity of CI artifacts, LFS objects, and uploads in file + storage. + + At this point, your **secondary** node will contain an up-to-date copy of everything the + **primary** node has, meaning nothing will be lost when you fail over. + +1. In this final step, you need to permanently disable the **primary** node. + + CAUTION: **Caution:** + When the **primary** node goes offline, there may be data saved on the **primary** node + that has not been replicated to the **secondary** node. This data should be treated + as lost if you proceed. + + TIP: **Tip:** + If you plan to [update the **primary** domain DNS record](../index.md#step-4-optional-updating-the-primary-domain-dns-record), + you may wish to lower the TTL now to speed up propagation. + + When performing a failover, we want to avoid a split-brain situation where + writes can occur in two different GitLab instances. So to prepare for the + failover, you must disable the **primary** node: + + - If you have SSH access to the **primary** node, stop and disable GitLab: + + ```shell + sudo gitlab-ctl stop + ``` + + Prevent GitLab from starting up again if the server unexpectedly reboots: + + ```shell + sudo systemctl disable gitlab-runsvdir + ``` + + NOTE: **Note:** + (**CentOS only**) In CentOS 6 or older, there is no easy way to prevent GitLab from being + started if the machine reboots isn't available (see [Omnibus GitLab issue #3058](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3058)). + It may be safest to uninstall the GitLab package completely with `sudo yum remove gitlab-ee`. + + NOTE: **Note:** + (**Ubuntu 14.04 LTS**) If you are using an older version of Ubuntu + or any other distribution based on the Upstart init system, you can prevent GitLab + from starting if the machine reboots as `root` with + `initctl stop gitlab-runsvvdir && echo 'manual' > /etc/init/gitlab-runsvdir.override && initctl reload-configuration`. + + - If you do not have SSH access to the **primary** node, take the machine offline and + prevent it from rebooting. Since there are many ways you may prefer to accomplish + this, we will avoid a single recommendation. You may need to: + + - Reconfigure the load balancers. + - Change DNS records (for example, point the **primary** DNS record to the + **secondary** node to stop using the **primary** node). + - Stop the virtual servers. + - Block traffic through a firewall. + - Revoke object storage permissions from the **primary** node. + - Physically disconnect a machine. + +### Promoting the **secondary** node + +NOTE: **Note:** +A new **secondary** should not be added at this time. If you want to add a new +**secondary**, do this after you have completed the entire process of promoting +the **secondary** to the **primary**. + +CAUTION: **Caution:** +If you encounter an `ActiveRecord::RecordInvalid: Validation failed: Name has already been taken` error during this process, read +[the troubleshooting advice](../../replication/troubleshooting.md#fixing-errors-during-a-failover-or-when-promoting-a-secondary-to-a-primary-node). + +The `gitlab-ctl promote-to-primary-node` command cannot be used yet in +conjunction with multiple servers, as it can only +perform changes on a **secondary** with only a single machine. Instead, you must +do this manually. + +DANGER: **Danger:** +In GitLab 13.2 and later versions, promoting a secondary node to a primary while the secondary is paused fails. We are [investigating the issue](https://gitlab.com/gitlab-org/gitlab/-/issues/225173). Do not pause replication before promoting a secondary. If the node is paused, please resume before promoting. + +1. SSH in to the PostgreSQL node in the **secondary** and trigger PostgreSQL to + promote to read-write: + + ```shell + sudo gitlab-pg-ctl promote + ``` + + In GitLab 12.8 and earlier, see [Message: `sudo: gitlab-pg-ctl: command not found`](../../replication/troubleshooting.md#message-sudo-gitlab-pg-ctl-command-not-found). + +1. Edit `/etc/gitlab/gitlab.rb` on every machine in the **secondary** to + reflect its new status as **primary** by removing any lines that enabled the + `geo_secondary_role`: + + ```ruby + ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. + geo_secondary_role['enable'] = true + + ## In 11.5+ documentation, the role was enabled as follows. Remove this line. + roles ['geo_secondary_role'] + ``` + + After making these changes [Reconfigure GitLab](../../../restart_gitlab.md#omnibus-gitlab-reconfigure) each + machine so the changes take effect. + +1. Promote the **secondary** to **primary**. SSH into a single Rails node + server and execute: + + ```shell + sudo gitlab-rake geo:set_secondary_as_primary + ``` + +1. Verify you can connect to the newly promoted **primary** using the URL used + previously for the **secondary**. + +1. Success! The **secondary** has now been promoted to **primary**. + +### Next steps + +To regain geographic redundancy as quickly as possible, you should +[add a new **secondary** node](../../setup/index.md). To +do that, you can re-add the old **primary** as a new secondary and bring it back +online. diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md new file mode 100644 index 00000000000..5e847030077 --- /dev/null +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md @@ -0,0 +1,269 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: howto +--- + +CAUTION: **Caution:** +This runbook is in **alpha**. For complete, production-ready documentation, see the +[disaster recovery documentation](../index.md). + +# Disaster Recovery (Geo) promotion runbooks **(PREMIUM ONLY)** + +## Geo planned failover for a single-node configuration + +| Component | Configuration | +|-------------|-----------------| +| PostgreSQL | Omnibus-managed | +| Geo site | Single-node | +| Secondaries | One | + +This runbook will guide you through a planned failover of a single-node Geo site +with one secondary. The following general architecture is assumed: + +```mermaid +graph TD + subgraph main[Geo deployment] + subgraph Primary[Primary site] + Node_1[(GitLab node)] + end + subgraph Secondary1[Secondary site] + Node_2[(GitLab node)] + end + end +``` + +This guide will result in the following: + +1. An offline primary. +1. A promoted secondary that is now the new primary. + +What is not covered: + +1. Re-adding the old **primary** as a secondary. +1. Adding a new secondary. + +### Preparation + +NOTE: **Note:** +Before following any of those steps, make sure you have `root` access to the +**secondary** to promote it, since there isn't provided an automated way to +promote a Geo replica and perform a failover. + +On the **secondary** node, navigate to the **Admin Area > Geo** dashboard to +review its status. Replicated objects (shown in green) should be close to 100%, +and there should be no failures (shown in red). If a large proportion of +objects aren't yet replicated (shown in gray), consider giving the node more +time to complete. + + + +If any objects are failing to replicate, this should be investigated before +scheduling the maintenance window. After a planned failover, anything that +failed to replicate will be **lost**. + +You can use the +[Geo status API](../../../../api/geo_nodes.md#retrieve-project-sync-or-verification-failures-that-occurred-on-the-current-node) +to review failed objects and the reasons for failure. +A common cause of replication failures is the data being missing on the +**primary** node - you can resolve these failures by restoring the data from backup, +or removing references to the missing data. + +The maintenance window won't end until Geo replication and verification is +completely finished. To keep the window as short as possible, you should +ensure these processes are close to 100% as possible during active use. + +If the **secondary** node is still replicating data from the **primary** node, +follow these steps to avoid unnecessary data loss: + +1. Until a [read-only mode](https://gitlab.com/gitlab-org/gitlab/-/issues/14609) + is implemented, updates must be prevented from happening manually to the + **primary**. Note that your **secondary** node still needs read-only + access to the **primary** node during the maintenance window: + + 1. At the scheduled time, using your cloud provider or your node's firewall, block + all HTTP, HTTPS and SSH traffic to/from the **primary** node, **except** for your IP and + the **secondary** node's IP. + + For instance, you can run the following commands on the **primary** node: + + ```shell + sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 22 -j ACCEPT + sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 22 -j ACCEPT + sudo iptables -A INPUT --destination-port 22 -j REJECT + + sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 80 -j ACCEPT + sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 80 -j ACCEPT + sudo iptables -A INPUT --tcp-dport 80 -j REJECT + + sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 443 -j ACCEPT + sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 443 -j ACCEPT + sudo iptables -A INPUT --tcp-dport 443 -j REJECT + ``` + + From this point, users will be unable to view their data or make changes on the + **primary** node. They will also be unable to log in to the **secondary** node. + However, existing sessions will work for the remainder of the maintenance period, and + public data will be accessible throughout. + + 1. Verify the **primary** node is blocked to HTTP traffic by visiting it in browser via + another IP. The server should refuse connection. + + 1. Verify the **primary** node is blocked to Git over SSH traffic by attempting to pull an + existing Git repository with an SSH remote URL. The server should refuse + connection. + + 1. On the **primary** node, disable non-Geo periodic background jobs by navigating + to **Admin Area > Monitoring > Background Jobs > Cron**, clicking `Disable All`, + and then clicking `Enable` for the `geo_sidekiq_cron_config_worker` cron job. + This job will re-enable several other cron jobs that are essential for planned + failover to complete successfully. + +1. Finish replicating and verifying all data: + + CAUTION: **Caution:** + Not all data is automatically replicated. Read more about + [what is excluded](../planned_failover.md#not-all-data-is-automatically-replicated). + + 1. If you are manually replicating any + [data not managed by Geo](../../replication/datatypes.md#limitations-on-replicationverification), + trigger the final replication process now. + 1. On the **primary** node, navigate to **Admin Area > Monitoring > Background Jobs > Queues** + and wait for all queues except those with `geo` in the name to drop to 0. + These queues contain work that has been submitted by your users; failing over + before it is completed will cause the work to be lost. + 1. On the **primary** node, navigate to **Admin Area > Geo** and wait for the + following conditions to be true of the **secondary** node you are failing over to: + - All replication meters to each 100% replicated, 0% failures. + - All verification meters reach 100% verified, 0% failures. + - Database replication lag is 0ms. + - The Geo log cursor is up to date (0 events behind). + + 1. On the **secondary** node, navigate to **Admin Area > Monitoring > Background Jobs > Queues** + and wait for all the `geo` queues to drop to 0 queued and 0 running jobs. + 1. On the **secondary** node, use [these instructions](../../../raketasks/check.md) + to verify the integrity of CI artifacts, LFS objects, and uploads in file + storage. + + At this point, your **secondary** node will contain an up-to-date copy of everything the + **primary** node has, meaning nothing will be lost when you fail over. + +1. In this final step, you need to permanently disable the **primary** node. + + CAUTION: **Caution:** + When the **primary** node goes offline, there may be data saved on the **primary** node + that has not been replicated to the **secondary** node. This data should be treated + as lost if you proceed. + + TIP: **Tip:** + If you plan to [update the **primary** domain DNS record](../index.md#step-4-optional-updating-the-primary-domain-dns-record), + you may wish to lower the TTL now to speed up propagation. + + When performing a failover, we want to avoid a split-brain situation where + writes can occur in two different GitLab instances. So to prepare for the + failover, you must disable the **primary** node: + + - If you have SSH access to the **primary** node, stop and disable GitLab: + + ```shell + sudo gitlab-ctl stop + ``` + + Prevent GitLab from starting up again if the server unexpectedly reboots: + + ```shell + sudo systemctl disable gitlab-runsvdir + ``` + + NOTE: **Note:** + (**CentOS only**) In CentOS 6 or older, there is no easy way to prevent GitLab from being + started if the machine reboots isn't available (see [Omnibus GitLab issue #3058](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3058)). + It may be safest to uninstall the GitLab package completely with `sudo yum remove gitlab-ee`. + + NOTE: **Note:** + (**Ubuntu 14.04 LTS**) If you are using an older version of Ubuntu + or any other distribution based on the Upstart init system, you can prevent GitLab + from starting if the machine reboots as `root` with + `initctl stop gitlab-runsvvdir && echo 'manual' > /etc/init/gitlab-runsvdir.override && initctl reload-configuration`. + + - If you do not have SSH access to the **primary** node, take the machine offline and + prevent it from rebooting. Since there are many ways you may prefer to accomplish + this, we will avoid a single recommendation. You may need to: + + - Reconfigure the load balancers. + - Change DNS records (for example, point the **primary** DNS record to the + **secondary** node to stop using the **primary** node). + - Stop the virtual servers. + - Block traffic through a firewall. + - Revoke object storage permissions from the **primary** node. + - Physically disconnect a machine. + +### Promoting the **secondary** node + +Note the following when promoting a secondary: + +- A new **secondary** should not be added at this time. If you want to add a new + **secondary**, do this after you have completed the entire process of promoting + the **secondary** to the **primary**. +- If you encounter an `ActiveRecord::RecordInvalid: Validation failed: Name has already been taken` + error during this process, read + [the troubleshooting advice](../../replication/troubleshooting.md#fixing-errors-during-a-failover-or-when-promoting-a-secondary-to-a-primary-node). + +To promote the secondary node: + +1. SSH in to your **secondary** node and login as root: + + ```shell + sudo -i + ``` + +1. Edit `/etc/gitlab/gitlab.rb` to reflect its new status as **primary** by + removing any lines that enabled the `geo_secondary_role`: + + ```ruby + ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. + geo_secondary_role['enable'] = true + + ## In 11.5+ documentation, the role was enabled as follows. Remove this line. + roles ['geo_secondary_role'] + ``` + +1. Run the following command to list out all preflight checks and automatically + check if replication and verification are complete before scheduling a planned + failover to ensure the process will go smoothly: + + ```shell + gitlab-ctl promotion-preflight-checks + ``` + +1. Promote the **secondary**: + + ```shell + gitlab-ctl promote-to-primary-node + ``` + + If you have already run the [preflight checks](../planned_failover.md#preflight-checks) + or don't want to run them, you can skip them: + + ```shell + gitlab-ctl promote-to-primary-node --skip-preflight-check + ``` + + You can also promote the secondary node to primary **without any further confirmation**, even when preflight checks fail: + + ```shell + sudo gitlab-ctl promote-to-primary-node --force + ``` + +1. Verify you can connect to the newly promoted **primary** node using the URL used + previously for the **secondary** node. + + If successful, the **secondary** node has now been promoted to the **primary** node. + +### Next steps + +To regain geographic redundancy as quickly as possible, you should +[add a new **secondary** node](../../setup/index.md). To +do that, you can re-add the old **primary** as a new secondary and bring it back +online. diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md index 6fdf213ac78..8767940816b 100644 --- a/doc/administration/geo/index.md +++ b/doc/administration/geo/index.md @@ -39,7 +39,7 @@ Implementing Geo provides the following benefits: In addition, it: -- Can be used for cloning and fetching projects, in addition to reading any data available in the GitLab web interface (see [current limitations](#current-limitations)). +- Can be used for cloning and fetching projects, in addition to reading any data available in the GitLab web interface (see [limitations](#limitations)). - Overcomes slow connections between distant offices, saving time by improving speed for distributed teams. - Helps reducing the loading time for automated tasks, custom integrations, and internal workflows. - Can quickly fail over to a **secondary** node in a [disaster recovery](disaster_recovery/index.md) scenario. @@ -67,9 +67,9 @@ Keep in mind that: - **Secondary** nodes talk to the **primary** node to: - Get user data for logins (API). - Replicate repositories, LFS Objects, and Attachments (HTTPS + JWT). -- Since GitLab Premium 10.0, the **primary** node no longer talks to **secondary** nodes to notify for changes (API). +- In GitLab Premium 10.0 and later, the **primary** node no longer talks to **secondary** nodes to notify for changes (API). - Pushing directly to a **secondary** node (for both HTTP and SSH, including Git LFS) was [introduced](https://about.gitlab.com/releases/2018/09/22/gitlab-11-3-released/) in [GitLab Premium](https://about.gitlab.com/pricing/#self-managed) 11.3. -- There are [limitations](#current-limitations) in the current implementation. +- There are [limitations](#limitations) when using Geo. ### Architecture @@ -195,6 +195,9 @@ For information on how to update your Geo nodes to the latest GitLab version, se > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/35913) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.2. +DANGER: **Danger:** +In GitLab 13.2 and later versions, promoting a secondary node to a primary while the secondary is paused fails. We are [investigating the issue](https://gitlab.com/gitlab-org/gitlab/-/issues/225173). Do not pause replication before promoting a secondary. If the node is paused, please resume before promoting. + In some circumstances, like during [upgrades](replication/updating_the_geo_nodes.md) or a [planned failover](disaster_recovery/planned_failover.md), it is desirable to pause replication between the primary and secondary. Pausing and resuming replication is done via a command line tool from the secondary node. @@ -247,7 +250,7 @@ For more information on removing a Geo node, see [Removing **secondary** Geo nod To find out how to disable Geo, see [Disabling Geo](replication/disable_geo.md). -## Current limitations +## Limitations CAUTION: **Caution:** This list of limitations only reflects the latest version of GitLab. If you are using an older version, extra limitations may be in place. @@ -261,6 +264,7 @@ This list of limitations only reflects the latest version of GitLab. If you are - Object pools for forked project deduplication work only on the **primary** node, and are duplicated on the **secondary** node. - [External merge request diffs](../merge_request_diffs.md) will not be replicated if they are on-disk, and viewing merge requests will fail. However, external MR diffs in object storage **are** supported. The default configuration (in-database) does work. - GitLab Runners cannot register with a **secondary** node. Support for this is [planned for the future](https://gitlab.com/gitlab-org/gitlab/-/issues/3294). +- Geo **secondary** nodes can not be configured to [use high-availability configurations of PostgreSQL](https://gitlab.com/groups/gitlab-org/-/epics/2536). ### Limitations on replication/verification @@ -278,7 +282,7 @@ For answers to common questions, see the [Geo FAQ](replication/faq.md). ## Log files -Since GitLab 9.5, Geo stores structured log messages in a `geo.log` file. For Omnibus installations, this file is at `/var/log/gitlab/gitlab-rails/geo.log`. +In GitLab 9.5 and later, Geo stores structured log messages in a `geo.log` file. For Omnibus installations, this file is at `/var/log/gitlab/gitlab-rails/geo.log`. This file contains information about when Geo attempts to sync repositories and files. Each line in the file contains a separate JSON entry that can be ingested into. For example, Elasticsearch or Splunk. diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index 166a724f9c1..d1fa2d503be 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -23,30 +23,34 @@ We currently distinguish between three different data types: See the list below of each feature or component we replicate, its corresponding data type, replication, and verification methods: -| Type | Feature / component | Replication method | Verification method | -|:---------|:----------------------------------------------|:--------------------------------------|:-----------------------| -| Database | Application data in PostgreSQL | Native | Native | -| Database | Redis | _N/A_ (*1*) | _N/A_ | -| Database | Elasticsearch | Native | Native | -| Database | Personal snippets | PostgreSQL Replication | PostgreSQL Replication | -| Database | Project snippets | PostgreSQL Replication | PostgreSQL Replication | -| Database | SSH public keys | PostgreSQL Replication | PostgreSQL Replication | -| Git | Project repository | Geo with Gitaly | Gitaly Checksum | -| Git | Project wiki repository | Geo with Gitaly | Gitaly Checksum | -| Git | Project designs repository | Geo with Gitaly | Gitaly Checksum | -| Git | Object pools for forked project deduplication | Geo with Gitaly | _Not implemented_ | -| Blobs | User uploads _(filesystem)_ | Geo with API | _Not implemented_ | -| Blobs | User uploads _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | -| Blobs | LFS objects _(filesystem)_ | Geo with API | _Not implemented_ | -| Blobs | LFS objects _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | -| Blobs | CI job artifacts _(filesystem)_ | Geo with API | _Not implemented_ | -| Blobs | CI job artifacts _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | -| Blobs | Archived CI build traces _(filesystem)_ | Geo with API | _Not implemented_ | -| Blobs | Archived CI build traces _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | -| Blobs | Container registry _(filesystem)_ | Geo with API/Docker API | _Not implemented_ | -| Blobs | Container registry _(object storage)_ | Geo with API/Managed/Docker API (*2*) | _Not implemented_ | -| Blobs | Package registry _(filesystem)_ | Geo with API | _Not implemented_ | -| Blobs | Package registry _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | +| Type | Feature / component | Replication method | Verification method | +|:---------|:------------------------------------------------|:--------------------------------------|:-----------------------| +| Database | Application data in PostgreSQL | Native | Native | +| Database | Redis | _N/A_ (*1*) | _N/A_ | +| Database | Elasticsearch | Native | Native | +| Database | Personal snippets | PostgreSQL Replication | PostgreSQL Replication | +| Database | Project snippets | PostgreSQL Replication | PostgreSQL Replication | +| Database | SSH public keys | PostgreSQL Replication | PostgreSQL Replication | +| Git | Project repository | Geo with Gitaly | Gitaly Checksum | +| Git | Project wiki repository | Geo with Gitaly | Gitaly Checksum | +| Git | Project designs repository | Geo with Gitaly | Gitaly Checksum | +| Git | Object pools for forked project deduplication | Geo with Gitaly | _Not implemented_ | +| Blobs | User uploads _(filesystem)_ | Geo with API | _Not implemented_ | +| Blobs | User uploads _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | +| Blobs | LFS objects _(filesystem)_ | Geo with API | _Not implemented_ | +| Blobs | LFS objects _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | +| Blobs | CI job artifacts _(filesystem)_ | Geo with API | _Not implemented_ | +| Blobs | CI job artifacts _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | +| Blobs | Archived CI build traces _(filesystem)_ | Geo with API | _Not implemented_ | +| Blobs | Archived CI build traces _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | +| Blobs | Container registry _(filesystem)_ | Geo with API/Docker API | _Not implemented_ | +| Blobs | Container registry _(object storage)_ | Geo with API/Managed/Docker API (*2*) | _Not implemented_ | +| Blobs | Package registry _(filesystem)_ | Geo with API | _Not implemented_ | +| Blobs | Package registry _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | +| Blobs | Versioned Terraform State _(filesystem)_ | Geo with API | _Not implemented_ | +| Blobs | Versioned Terraform State _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | +| Blobs | External Merge Request Diffs _(filesystem)_ | Geo with API | _Not implemented_ | +| Blobs | External Merge Request Diffs _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | - (*1*): Redis replication can be used as part of HA with Redis sentinel. It's not used between Geo nodes. - (*2*): Object storage replication can be performed by Geo or by your object storage provider/appliance @@ -160,39 +164,34 @@ replicating data from those features will cause the data to be **lost**. If you wish to use those features on a **secondary** node, or to execute a failover successfully, you must replicate their data using some other means. -| Feature | Replicated (added in GitLab version) | Verified (added in GitLab version) | Notes | -|:------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------|:----------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------| -| Application data in PostgreSQL | **Yes** (10.2) | **Yes** (10.2) | | -| Project repository | **Yes** (10.2) | **Yes** (10.7) | | -| Project wiki repository | **Yes** (10.2) | **Yes** (10.7) | | -| Project designs repository | **Yes** (12.7) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/32467) | | -| Uploads | **Yes** (10.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Verified only on transfer, or manually (*1*) | -| LFS objects | **Yes** (10.2) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8922) | Verified only on transfer, or manually (*1*). Unavailable for new LFS objects in 11.11.x and 12.0.x (*2*). | -| CI job artifacts (other than traces) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Verified only manually (*1*) | -| Archived traces | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Verified only on transfer, or manually (*1*) | -| Personal snippets | **Yes** (10.2) | **Yes** (10.2) | | -| [Versioned snippets](../../../user/snippets.md#versioned-snippets) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2810) | | -| Project snippets | **Yes** (10.2) | **Yes** (10.2) | | -| Object pools for forked project deduplication | **Yes** | No | | -| [Server-side Git hooks](../../server_hooks.md) | No | No | | -| [Elasticsearch integration](../../../integration/elasticsearch.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/1186) | No | | -| [GitLab Pages](../../pages/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/589) | No | | -| [Container Registry](../../packages/container_registry.md) | **Yes** (12.3) | No | | -| [NPM Registry](../../../user/packages/npm_registry/index.md) | **Yes** (13.2) | No | Behind feature flag `geo_package_file_replication`, enabled by default | -| [Maven Repository](../../../user/packages/maven_repository/index.md) | **Yes** (13.2) | No | Behind feature flag `geo_package_file_replication`, enabled by default | -| [Conan Repository](../../../user/packages/conan_repository/index.md) | **Yes** (13.2) | No | Behind feature flag `geo_package_file_replication`, enabled by default | -| [NuGet Repository](../../../user/packages/nuget_repository/index.md) | **Yes** (13.2) | No | Behind feature flag `geo_package_file_replication`, enabled by default | -| [PyPi Repository](../../../user/packages/pypi_repository/index.md) | **Yes** (13.2) | No | Behind feature flag `geo_package_file_replication`, enabled by default | -| [Composer Repository](../../../user/packages/composer_repository/index.md) | **Yes** (13.2) | No | Behind feature flag `geo_package_file_replication`, enabled by default | -| [External merge request diffs](../../merge_request_diffs.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/33817) | No | | -| [Terraform State](../../terraform_state.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/3112)(*3*) | No | | -| [Vulnerability Export](../../../user/application_security/security_dashboard/#export-vulnerabilities) | [No](https://gitlab.com/groups/gitlab-org/-/epics/3111)(*3*) | No | | -| Content in object storage | **Yes** (12.4) | No | | - -- (*1*): The integrity can be verified manually using - [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing - the output between them. -- (*2*): GitLab versions 11.11.x and 12.0.x are affected by [a bug that prevents any new - LFS objects from replicating](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). -- (*3*): If you are using Object Storage, the replication can be performed by the - Object Storage provider if supported. Please see [Geo with Object Storage](object_storage.md) +| Feature | Replicated (added in GitLab version) | Verified (added in GitLab version) | Object Storage replication (please see [Geo with Object Storage](object_storage.md)) | Notes | +|:---------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------|:----------------------------------------------------------|:-------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| [Application data in PostgreSQL](../../postgresql/index.md) | **Yes** (10.2) | **Yes** (10.2) | No | | +| [Project repository](../../..//user/project/repository/) | **Yes** (10.2) | **Yes** (10.7) | No | | +| [Project wiki repository](../../../user/project/wiki/) | **Yes** (10.2) | **Yes** (10.7) | No | | +| [Uploads](../../uploads.md) | **Yes** (10.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | No | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing the output between them. | +| [LFS objects](../../lfs/index.md) | **Yes** (10.2) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8922) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing the output between them. GitLab versions 11.11.x and 12.0.x are affected by [a bug that prevents any new LFS objects from replicating](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). | +| [Personal snippets](../../../user/snippets.md#personal-snippets) | **Yes** (10.2) | **Yes** (10.2) | No | | +| [Project snippets](../../../user/snippets.md#project-snippets) | **Yes** (10.2) | **Yes** (10.2) | No | | +| [CI job artifacts (other than Job Logs)](../../../ci/pipelines/job_artifacts.md) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Via Object Storage provider if supported. Native Geo support (Beta) . | Verified only manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing the output between them | +| [Job logs](../../job_logs.md) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing the output between them | +| [Object pools for forked project deduplication](../../../development/git_object_deduplication.md) | **Yes** | No | No | | +| [Container Registry](../../packages/container_registry.md) | **Yes** (12.3) | No | No | Disabled by default. See [instructions](./docker_registry.md) to enable. | +| [Content in object storage (beta)](object_storage.md) | **Yes** (12.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/13845) | No | | +| [Project designs repository](../../../user/project/issues/design_management.md) | **Yes** (12.7) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/32467) | Via Object Storage provider if supported. Native Geo support (Beta). | | +| [NPM Registry](../../../user/packages/npm_registry/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | +| [Maven Repository](../../../user/packages/maven_repository/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | +| [Conan Repository](../../../user/packages/conan_repository/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | +| [NuGet Repository](../../../user/packages/nuget_repository/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | +| [PyPI Repository](../../../user/packages/pypi_repository/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | +| [Composer Repository](../../../user/packages/composer_repository/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | +| [Generic packages](../../../user/packages/generic_packages/index.md) | **Yes** (13.5) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | +| [Versioned Terraform State](../../terraform_state.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_terraform_state_version_replication`, enabled by default | +| [External merge request diffs](../../merge_request_diffs.md) | **Yes** (13.5) | No | Behind feature flag `geo_merge_request_diff_replication`, enabled by default | | +| [Versioned snippets](../../../user/snippets.md#versioned-snippets) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2810) | No | | +| [Server-side Git hooks](../../server_hooks.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1867) | No | No | | +| [Elasticsearch integration](../../../integration/elasticsearch.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/1186) | No | No | | +| [GitLab Pages](../../pages/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/589) | No | No | | +| [CI Pipeline Artifacts](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/pipeline_artifact.rb) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Persists additional artifacts after a pipeline completes | +| [Dependency proxy images](../../../user/packages/dependency_proxy/index.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/259694) | No | No | Blocked on [Geo: Secondary Mimicry](https://gitlab.com/groups/gitlab-org/-/epics/1528). Note that replication of this cache is not needed for Disaster Recovery purposes because it can be recreated from external sources. | +| [Vulnerability Export](../../../user/application_security/security_dashboard/#export-vulnerabilities) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Not planned because they are ephemeral and sensitive. They can be regenerated on demand. | diff --git a/doc/administration/geo/replication/disable_geo.md b/doc/administration/geo/replication/disable_geo.md index aed8e5fc3bc..14a11d9c1e3 100644 --- a/doc/administration/geo/replication/disable_geo.md +++ b/doc/administration/geo/replication/disable_geo.md @@ -29,7 +29,7 @@ anymore on these nodes. You can follow our docs to [remove your secondary Geo no If the current node that you want to keep using is a secondary node, you need to first promote it to primary. You can use our steps on [how to promote a secondary node](../disaster_recovery/#step-3-promoting-a-secondary-node) -in order to do that. +to do that. ## Remove the primary node from the UI diff --git a/doc/administration/geo/replication/faq.md b/doc/administration/geo/replication/faq.md index 3892d73b465..f7f391b360e 100644 --- a/doc/administration/geo/replication/faq.md +++ b/doc/administration/geo/replication/faq.md @@ -67,3 +67,7 @@ That's totally fine. We use HTTP(s) to fetch repository changes from the **prima ## Is this possible to set up a Docker Registry for a **secondary** node that mirrors the one on the **primary** node? Yes. See [Docker Registry for a **secondary** node](docker_registry.md). + +## Can I login to a secondary node? + +Yes, but secondary nodes receive all authentication data (like user accounts and logins) from the primary instance. This means you will be re-directed to the primary for authentication and routed back afterwards. diff --git a/doc/administration/geo/replication/geo_validation_tests.md b/doc/administration/geo/replication/geo_validation_tests.md index 8247b8c6336..c28688930b5 100644 --- a/doc/administration/geo/replication/geo_validation_tests.md +++ b/doc/administration/geo/replication/geo_validation_tests.md @@ -114,6 +114,22 @@ The following are GitLab upgrade validation tests we performed. The following are PostgreSQL upgrade validation tests we performed. +### September 2020 + +[Verify PostgreSQL 12 upgrade for Geo installations](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5454): + +- Description: With PostgreSQL 12 available as an opt-in version in GitLab 13.3, we tested upgrading + existing Geo installations from PostgreSQL 11 to 12. We also re-tested fresh installations of GitLab + with Geo after fixes were made to support PostgreSQL 12. These tests were done using a + [nightly build](https://packages.gitlab.com/gitlab/nightly-builds/packages/ubuntu/bionic/gitlab-ee_13.3.6+rnightly.169516.d5209202-0_amd64.deb) + of GitLab 13.4. +- Outcome: Tests were successful for Geo deployments with a single database node on the primary and secondary. + We encountered known issues with repmgr and Patroni managed PostgreSQL clusters on the Geo primary. Using + PostgreSQL 12 with a database cluster on the primary is not recommended until the issues are resolved. +- Known issues for PostgreSQL clusters: + - [Ensure Patroni detects PostgreSQL update](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5423) + - [Allow configuring permanent replication slots in patroni](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5628) + ### August 2020 [Verify Geo installation with PostgreSQL 12](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5453): @@ -168,5 +184,4 @@ The following are additional validation tests we performed. [Test Gitaly Cluster on a Geo Deployment](https://gitlab.com/gitlab-org/gitlab/-/issues/223210): - Description: Tested a Geo deployment with Gitaly clusters configured on both the primary and secondary Geo sites. Triggered automatic Gitaly cluster failover on the primary Geo site, and ran end-to-end Geo tests. Then triggered Gitaly cluster failover on the secondary Geo site, and re-ran the end-to-end Geo tests. - - Outcome: Successful end-to-end tests before and after Gitaly cluster failover on the primary site, and before and after Gitaly cluster failover on the secondary site. diff --git a/doc/administration/high_availability/img/geo-ha-diagram.png b/doc/administration/geo/replication/img/geo-ha-diagram.png Binary files differindex da5d612827c..da5d612827c 100644 --- a/doc/administration/high_availability/img/geo-ha-diagram.png +++ b/doc/administration/geo/replication/img/geo-ha-diagram.png diff --git a/doc/administration/geo/replication/multiple_servers.md b/doc/administration/geo/replication/multiple_servers.md index cba41c375a3..7d65d2165c5 100644 --- a/doc/administration/geo/replication/multiple_servers.md +++ b/doc/administration/geo/replication/multiple_servers.md @@ -13,7 +13,7 @@ described, it is possible to adapt these instructions to your needs. ## Architecture overview - + _[diagram source - GitLab employees only](https://docs.google.com/drawings/d/1z0VlizKiLNXVVVaERFwgsIOuEgjcUqDTWPdQYsE7Z4c/edit)_ @@ -133,7 +133,7 @@ Configure the following services, again using the non-Geo multi-node documentation: - [Configuring Redis for GitLab](../../redis/replication_and_failover.md#example-configuration-for-the-gitlab-application) for multiple nodes. -- [Gitaly](../../high_availability/gitaly.md), which will store data that is +- [Gitaly](../../gitaly/index.md), which will store data that is synchronized from the **primary** node. NOTE: **Note:** @@ -422,7 +422,6 @@ application servers above, with some changes to run only the `sidekiq` service: ## alertmanager['enable'] = false consul['enable'] = false - geo_logcursor['enable'] = false gitaly['enable'] = false gitlab_exporter['enable'] = false gitlab_workhorse['enable'] = false diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index f6d6f39fb19..b62c5c6f460 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -386,6 +386,15 @@ This happens when you have added IP addresses without a subnet mask in `postgres To fix this, add the subnet mask in `/etc/gitlab/gitlab.rb` under `postgresql['md5_auth_cidr_addresses']` to respect the CIDR format (i.e. `1.2.3.4/32`). +### Message: `Found data in the gitlabhq_production database!` when running `gitlab-ctl replicate-geo-database` + +This happens if data is detected in the `projects` table. When one or more projects are detected, the operation +is aborted to prevent accidental data loss. To bypass this message, pass the `--force` option to the command. + +In GitLab 13.4, a seed project is added when GitLab is first installed. This makes it necessary to pass `--force` even +on a new Geo secondary node. There is an [issue to account for seed projects](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5618) +when checking the database. + ### Very large repositories never successfully synchronize on the **secondary** node GitLab places a timeout on all repository clones, including project imports @@ -483,8 +492,8 @@ to start again from scratch, there are a few steps that can help you: gitlab-ctl start geo-postgresql ``` - Reconfigure in order to recreate the folders and make sure permissions and ownership - are correctly + Reconfigure to recreate the folders and make sure permissions and ownership + are correct: ```shell gitlab-ctl reconfigure @@ -605,46 +614,18 @@ or `gitlab-ctl promote-to-primary-node`, either: ### Message: ActiveRecord::RecordInvalid: Validation failed: Enabled Geo primary node cannot be disabled -This error may occur if you have paused replication from the original primary node before attempting to promote this node. -To double check this, you can do the following: - -- Get the current secondary node's ID using: - - ```shell - sudo gitlab-rails runner 'puts GeoNode.current_node.id' - ``` - -- Double check that the node is active through the database by running the following - on the secondary node, replacing `ID_FROM_ABOVE`: - - ```shell - sudo gitlab-rails dbconsole - - SELECT enabled FROM geo_nodes WHERE id = ID_FROM_ABOVE; - ``` - -- If the above returned `f` it means that the replication was paused. - You can re-enable it through an `UPDATE` statement in the database: - - ```shell - sudo gitlab-rails dbconsole - - UPDATE geo_nodes SET enabled = 't' WHERE id = ID_FROM_ABOVE; - ``` - -### While Promoting the secondary, I got an error `ActiveRecord::RecordInvalid` - If you disabled a secondary node, either with the [replication pause task](../index.md#pausing-and-resuming-replication) -(13.2) or via the UI (13.1 and earlier), you must first re-enable the -node before you can continue. This is fixed in 13.4. +(13.2) or by using the user interface (13.1 and earlier), you must first +re-enable the node before you can continue. This is fixed in 13.4. -From `gitlab-psql`, execute the following, replacing `<your secondary url>` -with the URL for your secondary server starting with `http` or `https` and ending with a `/`. +Run the following command, replacing `https://<secondary url>/` with the URL +for your secondary server, using either `http` or `https`, and ensuring that you +end the URL with a slash (`/`): ```shell -SECONDARY_URL="https://<secondary url>/" -DATABASE_NAME="gitlabhq_production" -sudo gitlab-psql -d "$DATABASE_NAME" -c "UPDATE geo_nodes SET enabled = true WHERE url = '$SECONDARY_URL';" +sudo gitlab-rails dbconsole + +UPDATE geo_nodes SET enabled = true WHERE url = 'https://<secondary url>/' AND enabled = false;" ``` This should update 1 row. diff --git a/doc/administration/geo/replication/updating_the_geo_nodes.md b/doc/administration/geo/replication/updating_the_geo_nodes.md index b78aeb06ebf..1af2b8d0b88 100644 --- a/doc/administration/geo/replication/updating_the_geo_nodes.md +++ b/doc/administration/geo/replication/updating_the_geo_nodes.md @@ -21,14 +21,17 @@ Updating Geo nodes involves performing: NOTE: **Note:** These general update steps are not intended for [high-availability deployments](https://docs.gitlab.com/omnibus/update/README.html#multi-node--ha-deployment), and will cause downtime. If you want to avoid downtime, consider using [zero downtime updates](https://docs.gitlab.com/omnibus/update/README.html#zero-downtime-updates). +DANGER: **Danger:** +In GitLab 13.2 and later versions, promoting a secondary node to a primary while the secondary is paused fails. We are [investigating the issue](https://gitlab.com/gitlab-org/gitlab/-/issues/225173). Do not pause replication before promoting a secondary. If the node is paused, please resume before promoting. + To update the Geo nodes when a new GitLab version is released, update **primary** and all **secondary** nodes: 1. **Optional:** [Pause replication on each **secondary** node.](../index.md#pausing-and-resuming-replication) 1. Log into the **primary** node. -1. [Update GitLab on the **primary** node using Omnibus](https://docs.gitlab.com/omnibus/update/README.html). +1. [Update GitLab on the **primary** node using Omnibus's Geo-specific steps](https://docs.gitlab.com/omnibus/update/README.html#geo-deployment). 1. Log into each **secondary** node. -1. [Update GitLab on each **secondary** node using Omnibus](https://docs.gitlab.com/omnibus/update/README.html). +1. [Update GitLab on each **secondary** node using Omnibus's Geo-specific steps](https://docs.gitlab.com/omnibus/update/README.html#geo-deployment). 1. If you paused replication in step 1, [resume replication on each **secondary**](../index.md#pausing-and-resuming-replication) 1. [Test](#check-status-after-updating) **primary** and **secondary** nodes, and check version in each. diff --git a/doc/administration/geo/replication/version_specific_updates.md b/doc/administration/geo/replication/version_specific_updates.md index 1ae246e3e61..85c869eff92 100644 --- a/doc/administration/geo/replication/version_specific_updates.md +++ b/doc/administration/geo/replication/version_specific_updates.md @@ -397,7 +397,7 @@ existing repositories was added in GitLab 10.1. ## Updating to GitLab 10.0 -Since GitLab 10.0, we require all **Geo** systems to [use SSH key lookups via +In GitLab 10.0 and later, we require all **Geo** systems to [use SSH key lookups via the database](../../operations/fast_ssh_key_lookup.md) to avoid having to maintain consistency of the `authorized_keys` file for SSH access. Failing to do this will prevent users from being able to clone via SSH. @@ -447,8 +447,8 @@ Omnibus is the following: > **IMPORTANT**: With GitLab 9.0, the PostgreSQL version is updated to 9.6 and manual steps are -required in order to update the **secondary** nodes and keep the Streaming -Replication working. Downtime is required, so plan ahead. +required to update the **secondary** nodes and keep the Streaming Replication +working. Downtime is required, so plan ahead. The following steps apply only if you update from a 8.17 GitLab version to 9.0+. For previous versions, update to GitLab 8.17 first before attempting to @@ -611,9 +611,9 @@ is prepended with the relevant node for better clarity: ### Update tracking database on **secondary** node -After updating a **secondary** node, you might need to run migrations on -the tracking database. The tracking database was added in GitLab 9.1, -and it is required since 10.0. +After updating a **secondary** node, you might need to run migrations on the +tracking database. The tracking database was added in GitLab 9.1, and is +required in GitLab 10.0 and later. 1. Run database migrations on tracking database: diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md index aefa8a0e399..09b9c71aeb7 100644 --- a/doc/administration/geo/setup/database.md +++ b/doc/administration/geo/setup/database.md @@ -17,9 +17,10 @@ NOTE: **Note:** The stages of the setup process must be completed in the documented order. Before attempting the steps in this stage, [complete all prior stages](../setup/index.md#using-omnibus-gitlab). -This document describes the minimal steps you have to take in order to -replicate your **primary** GitLab database to a **secondary** node's database. You may -have to change some values according to your database setup, how big it is, etc. +This document describes the minimal steps you have to take to replicate your +**primary** GitLab database to a **secondary** node's database. You may have to +change some values, based on attributes including your database's setup and +size. You are encouraged to first read through all the steps before executing them in your testing/production environment. @@ -433,6 +434,11 @@ data before running `pg_basebackup`. NOTE: **Note:** Replication slot names must only contain lowercase letters, numbers, and the underscore character. + NOTE: **Note:** + In GitLab 13.4, a seed project is added when GitLab is first installed. This makes it necessary to pass `--force` even + on a new Geo secondary node. There is an [issue to account for seed projects](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5618) + when checking the database. + When prompted, enter the _plaintext_ password you set up for the `gitlab_replicator` user in the first step. diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index e6b137bac29..59a6f2596da 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -90,10 +90,10 @@ When running Gitaly on its own server, note the following regarding GitLab versi leveraged for redundancy on block-level Git data, but only has to be mounted on the Gitaly servers. - From GitLab 11.8 to 12.2, it is possible to use Elasticsearch in a Gitaly setup that doesn't use - NFS. In order to use Elasticsearch in these versions, the + NFS. To use Elasticsearch in these versions, the [repository indexer](../../integration/elasticsearch.md#elasticsearch-repository-indexer) must be enabled in your GitLab configuration. -- [Since GitLab 12.3](https://gitlab.com/gitlab-org/gitlab/-/issues/6481), the new indexer is +- [In GitLab 12.3 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/6481), the new indexer is the default and no configuration is required. ### Network architecture @@ -317,7 +317,7 @@ disable enforcement. For more information, see the documentation on configuring ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -1. Run `sudo /opt/gitlab/embedded/service/gitlab-shell/bin/check -config /opt/gitlab/embedded/service/gitlab-shell/config.yml` +1. Run `sudo /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml` to confirm that Gitaly can perform callbacks to the GitLab internal API. **For installations from source** @@ -364,7 +364,7 @@ disable enforcement. For more information, see the documentation on configuring ``` 1. Save the files and [restart GitLab](../restart_gitlab.md#installations-from-source). -1. Run `sudo -u git /home/git/gitlab-shell/bin/check -config /home/git/gitlab-shell/config.yml` +1. Run `sudo -u git /home/git/gitaly/gitaly-hooks check /home/git/gitaly/config.toml` to confirm that Gitaly can perform callbacks to the GitLab internal API. ### Configure Gitaly clients @@ -382,10 +382,10 @@ if previously enabled manually. Gitaly makes the following assumptions: - Your `gitaly1.internal` Gitaly server can be reached at `gitaly1.internal:8075` from your Gitaly - clients, and that Gitaly server can read and write to `/mnt/gitlab/default` and + clients, and that Gitaly server can read, write, and set permissions on `/mnt/gitlab/default` and `/mnt/gitlab/storage1`. - Your `gitaly2.internal` Gitaly server can be reached at `gitaly2.internal:8075` from your Gitaly - clients, and that Gitaly server can read and write to `/mnt/gitlab/storage2`. + clients, and that Gitaly server can read, write, and set permissions on `/mnt/gitlab/storage2`. - Your `gitaly1.internal` and `gitaly2.internal` Gitaly servers can reach each other. You can't define Gitaly servers with some as a local Gitaly server @@ -406,8 +406,8 @@ server (with `gitaly_address`) unless you setup with special ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -1. Run `sudo gitlab-rake gitlab:gitaly:check` to confirm the Gitaly client can connect to Gitaly - servers. +1. Run `sudo gitlab-rake gitlab:gitaly:check` on the Gitaly client (for example, the + Rails application) to confirm it can connect to Gitaly servers. 1. Tail the logs to see the requests: ```shell @@ -424,17 +424,17 @@ server (with `gitaly_address`) unless you setup with special storages: default: gitaly_address: tcp://gitaly1.internal:8075 - path: /some/dummy/path + path: /some/local/path storage1: gitaly_address: tcp://gitaly1.internal:8075 - path: /some/dummy/path + path: /some/local/path storage2: gitaly_address: tcp://gitaly2.internal:8075 - path: /some/dummy/path + path: /some/local/path ``` NOTE: **Note:** - `/some/dummy/path` should be set to a local folder that exists, however no data will be stored in + `/some/local/path` should be set to a local folder that exists, however no data will be stored in this folder. This will no longer be necessary after [this issue](https://gitlab.com/gitlab-org/gitaly/-/issues/1282) is resolved. @@ -482,6 +482,14 @@ git_data_dirs({ 'storage1' => { 'gitaly_address' => 'tcp://gitlab.internal:8075', 'path' => '/mnt/gitlab/git-data' }, 'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' }, }) + +# Make Gitaly accept connections on all network interfaces +gitaly['listen_addr'] = "0.0.0.0:8075" + +# Or for TLS +gitaly['tls_listen_addr'] = "0.0.0.0:9999" +gitaly['certificate_path'] = "/etc/gitlab/ssl/cert.pem" +gitaly['key_path'] = "/etc/gitlab/ssl/key.pem" ``` `path` can only be included for storage shards on the local Gitaly server. @@ -532,20 +540,12 @@ corresponding to each Gitaly server must be installed on that Gitaly server. Additionally, the certificate (or its certificate authority) must be installed on all: -- Gitaly servers, including the Gitaly server using the certificate. +- Gitaly servers. - Gitaly clients that communicate with it. -The process is documented in the -[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates) -and repeated below. - Note the following: -- The certificate must specify the address you use to access the Gitaly server. If you are: - - Addressing the Gitaly server by a hostname, you can either use the Common Name field for this, - or add it as a Subject Alternative Name. - - Addressing the Gitaly server by its IP address, you must add it as a Subject Alternative Name to - the certificate. [gRPC does not support using an IP address as Common Name in a certificate](https://github.com/grpc/grpc/issues/2691). +- The certificate must specify the address you use to access the Gitaly server. You must add the hostname or IP address as a Subject Alternative Name to the certificate. - You can configure Gitaly servers with both an unencrypted listening address `listen_addr` and an encrypted listening address `tls_listen_addr` at the same time. This allows you to gradually transition from unencrypted to encrypted traffic if necessary. @@ -631,17 +631,17 @@ To configure Gitaly with TLS: storages: default: gitaly_address: tls://gitaly1.internal:9999 - path: /some/dummy/path + path: /some/local/path storage1: gitaly_address: tls://gitaly1.internal:9999 - path: /some/dummy/path + path: /some/local/path storage2: gitaly_address: tls://gitaly2.internal:9999 - path: /some/dummy/path + path: /some/local/path ``` NOTE: **Note:** - `/some/dummy/path` should be set to a local folder that exists, however no data will be stored + `/some/local/path` should be set to a local folder that exists, however no data will be stored in this folder. This will no longer be necessary after [Gitaly issue #1282](https://gitlab.com/gitlab-org/gitaly/-/issues/1282) is resolved. @@ -711,6 +711,15 @@ Gitaly Go process. Some examples of things that are implemented in `gitaly-ruby` - RPCs that deal with wikis. - RPCs that create commits on behalf of a user, such as merge commits. +We recommend: + +- At least 300MB memory per worker. +- No more than one worker per core. + +NOTE: **Note:** +`gitaly-ruby` is planned to be eventually removed. To track progress, see the +[Remove the Gitaly-Ruby sidecar](https://gitlab.com/groups/gitlab-org/-/epics/2862) epic. + ### Configure number of `gitaly-ruby` workers `gitaly-ruby` has much less capacity than Gitaly implemented in Go. If your Gitaly server has to handle lots of @@ -1021,6 +1030,9 @@ The second facet presents the only real solution. For this, we developed ## Troubleshooting Gitaly +Check [Gitaly timeouts](../../user/admin_area/settings/gitaly_timeouts.md) when troubleshooting +Gitaly. + ### Checking versions when using standalone Gitaly servers When using standalone Gitaly servers, you must make sure they are the same version @@ -1242,13 +1254,6 @@ unset http_proxy unset https_proxy ``` -### Gitaly not listening on new address after reconfiguring - -When updating the `gitaly['listen_addr']` or `gitaly['prometheus_listen_addr']` -values, Gitaly may continue to listen on the old address after a `sudo gitlab-ctl reconfigure`. - -When this occurs, performing a `sudo gitlab-ctl restart` will resolve the issue. This will no longer be necessary after [this issue](https://gitlab.com/gitlab-org/gitaly/-/issues/2521) is resolved. - ### Permission denied errors appearing in Gitaly logs when accessing repositories from a standalone Gitaly server If this error occurs even though file permissions are correct, it's likely that diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index 876904a2093..d091ae5895a 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -182,7 +182,7 @@ failure. For greater fault tolerance, the following options are available: - For Geo instances, either: - Set up a separate [PostgreSQL instance](https://www.postgresql.org/docs/11/high-availability.html). - Use a cloud-managed PostgreSQL service. AWS - [Relational Database Service](https://aws.amazon.com/rds/)) is recommended. + [Relational Database Service](https://aws.amazon.com/rds/) is recommended. To complete this section you will need: @@ -356,7 +356,7 @@ application server, or a Gitaly node. If you want to use a TLS client certificate, the options below can be used: ```ruby - # Connect to PostreSQL using a TLS client certificate + # Connect to PostgreSQL using a TLS client certificate # praefect['database_sslcert'] = '/path/to/client-cert' # praefect['database_sslkey'] = '/path/to/client-key' @@ -547,14 +547,14 @@ To configure Praefect with TLS: storages: default: gitaly_address: tls://praefect1.internal:3305 - path: /some/dummy/path + path: /some/local/path storage1: gitaly_address: tls://praefect2.internal:3305 - path: /some/dummy/path + path: /some/local/path ``` NOTE: **Note:** - `/some/dummy/path` should be set to a local folder that exists, however no + `/some/local/path` should be set to a local folder that exists, however no data will be stored in this folder. This will no longer be necessary after [this issue](https://gitlab.com/gitlab-org/gitaly/-/issues/1282) is resolved. @@ -873,10 +873,10 @@ Particular attention should be shown to: gitlab-ctl reconfigure ``` -1. Verify each `gitlab-shell` on each Gitaly node can reach GitLab. On each Gitaly node run: +1. Verify on each Gitaly node the Git Hooks can reach GitLab. On each Gitaly node run: ```shell - /opt/gitlab/embedded/service/gitlab-shell/bin/check -config /opt/gitlab/embedded/service/gitlab-shell/config.yml + /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml ``` 1. Verify that GitLab can reach Praefect: @@ -943,16 +943,17 @@ cluster. ## Distributed reads > - Introduced in GitLab 13.1 in [beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha-beta-ga) with feature flag `gitaly_distributed_reads` set to disabled. -> - [Made generally available](https://gitlab.com/gitlab-org/gitaly/-/issues/2951) in GitLab 13.3. +> - [Made generally available and enabled by default](https://gitlab.com/gitlab-org/gitaly/-/issues/2951) in GitLab 13.3. +> - [Disabled by default](https://gitlab.com/gitlab-org/gitaly/-/issues/3178) in GitLab 13.5. Praefect supports distribution of read operations across Gitaly nodes that are configured for the virtual node. -The feature is enabled by default. To disable distributed reads, the `gitaly_distributed_reads` -[feature flag](../feature_flags.md) must be disabled in a Ruby console: +The feature is disabled by default. To enable distributed reads, the `gitaly_distributed_reads` +[feature flag](../feature_flags.md) must be enabled in a Ruby console: ```ruby -Feature.disable(:gitaly_distributed_reads) +Feature.enable(:gitaly_distributed_reads) ``` If enabled, all RPCs marked with `ACCESSOR` option like @@ -993,6 +994,8 @@ information, see the [strong consistency epic](https://gitlab.com/groups/gitlab- To enable strong consistency: +- In GitLab 13.5, you must use Git v2.28.0 or higher on Gitaly nodes to enable + strong consistency. - In GitLab 13.4 and later, the strong consistency voting strategy has been improved. Instead of requiring all nodes to agree, only the primary and half of the secondaries need to agree. This strategy is enabled by default. To diff --git a/doc/administration/gitaly/reference.md b/doc/administration/gitaly/reference.md index 0c211c220d7..53001b946d8 100644 --- a/doc/administration/gitaly/reference.md +++ b/doc/administration/gitaly/reference.md @@ -138,8 +138,8 @@ Most of the time we use `git cat-file --batch` processes for that. For better performance, Gitaly can re-use these `git cat-file` processes across RPC calls. Previously used processes are kept around in a ["Git cat-file cache"](https://about.gitlab.com/blog/2019/07/08/git-performance-on-nfs/#enter-cat-file-cache). -In order to control how much system resources this uses, we have a maximum number -of cat-file processes that can go into the cache. +To control how much system resources this uses, we have a maximum number of +cat-file processes that can go into the cache. The default limit is 100 `cat-file`s, which constitute a pair of `git cat-file --batch` and `git cat-file --batch-check` processes. If diff --git a/doc/administration/high_availability/README.md b/doc/administration/high_availability/README.md deleted file mode 100644 index d36b029cbb3..00000000000 --- a/doc/administration/high_availability/README.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -redirect_to: ../reference_architectures/index.md ---- - -# Reference Architectures - -This document was moved to [another location](../reference_architectures/index.md). diff --git a/doc/administration/high_availability/alpha_database.md b/doc/administration/high_availability/alpha_database.md deleted file mode 100644 index 99c28e5c7a6..00000000000 --- a/doc/administration/high_availability/alpha_database.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: 'database.md' ---- - -This document was moved to [another location](../postgresql/index.md). diff --git a/doc/administration/high_availability/consul.md b/doc/administration/high_availability/consul.md deleted file mode 100644 index 362d6ee8ba7..00000000000 --- a/doc/administration/high_availability/consul.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: ../consul.md ---- - -This document was moved to [another location](../consul.md). diff --git a/doc/administration/high_availability/database.md b/doc/administration/high_availability/database.md deleted file mode 100644 index 784e496d10e..00000000000 --- a/doc/administration/high_availability/database.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: '../postgresql/index.md' ---- - -This document was moved to [another location](../postgresql/index.md). diff --git a/doc/administration/high_availability/gitaly.md b/doc/administration/high_availability/gitaly.md deleted file mode 100644 index a1e8fe3b488..00000000000 --- a/doc/administration/high_availability/gitaly.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: ../gitaly/index.md ---- - -This document was moved to [another location](../gitaly/index.md). diff --git a/doc/administration/high_availability/gitlab.md b/doc/administration/high_availability/gitlab.md deleted file mode 100644 index 748373c8941..00000000000 --- a/doc/administration/high_availability/gitlab.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: ../reference_architectures/index.md ---- - -This document was moved to [another location](../reference_architectures/index.md). diff --git a/doc/administration/high_availability/img/fully-distributed.png b/doc/administration/high_availability/img/fully-distributed.png Binary files differdeleted file mode 100644 index c3cd2bf24f0..00000000000 --- a/doc/administration/high_availability/img/fully-distributed.png +++ /dev/null diff --git a/doc/administration/high_availability/img/horizontal.png b/doc/administration/high_availability/img/horizontal.png Binary files differdeleted file mode 100644 index 75d08e1097a..00000000000 --- a/doc/administration/high_availability/img/horizontal.png +++ /dev/null diff --git a/doc/administration/high_availability/img/hybrid.png b/doc/administration/high_availability/img/hybrid.png Binary files differdeleted file mode 100644 index 8dd9923e597..00000000000 --- a/doc/administration/high_availability/img/hybrid.png +++ /dev/null diff --git a/doc/administration/high_availability/load_balancer.md b/doc/administration/high_availability/load_balancer.md deleted file mode 100644 index 5cedc4e11ae..00000000000 --- a/doc/administration/high_availability/load_balancer.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: ../load_balancer.md ---- - -This document was moved to [another location](../load_balancer.md). diff --git a/doc/administration/high_availability/monitoring_node.md b/doc/administration/high_availability/monitoring_node.md deleted file mode 100644 index 76bcf6d0d40..00000000000 --- a/doc/administration/high_availability/monitoring_node.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: ../monitoring/prometheus/index.md ---- - -This document was moved to [another location](../monitoring/prometheus/index.md). diff --git a/doc/administration/high_availability/nfs.md b/doc/administration/high_availability/nfs.md deleted file mode 100644 index e3342fa0813..00000000000 --- a/doc/administration/high_availability/nfs.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: ../nfs.md ---- - -This document was moved to [another location](../nfs.md). diff --git a/doc/administration/high_availability/nfs_host_client_setup.md b/doc/administration/high_availability/nfs_host_client_setup.md deleted file mode 100644 index e3342fa0813..00000000000 --- a/doc/administration/high_availability/nfs_host_client_setup.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: ../nfs.md ---- - -This document was moved to [another location](../nfs.md). diff --git a/doc/administration/high_availability/object_storage.md b/doc/administration/high_availability/object_storage.md deleted file mode 100644 index eeb730d3cc7..00000000000 --- a/doc/administration/high_availability/object_storage.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: '../object_storage.md' ---- - -This document was moved to [another location](../object_storage.md). diff --git a/doc/administration/high_availability/pgbouncer.md b/doc/administration/high_availability/pgbouncer.md deleted file mode 100644 index 44f4aa37651..00000000000 --- a/doc/administration/high_availability/pgbouncer.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: ../postgresql/pgbouncer.md ---- - -This document was moved to [another location](../postgresql/pgbouncer.md). diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md deleted file mode 100644 index 2b5771f49f2..00000000000 --- a/doc/administration/high_availability/redis.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: ../redis/index.md ---- - -This document was moved to [another location](../redis/index.md). diff --git a/doc/administration/high_availability/redis_source.md b/doc/administration/high_availability/redis_source.md deleted file mode 100644 index 75496638979..00000000000 --- a/doc/administration/high_availability/redis_source.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: ../redis/replication_and_failover_external.md ---- - -This document was moved to [another location](../redis/replication_and_failover_external.md). diff --git a/doc/administration/high_availability/sidekiq.md b/doc/administration/high_availability/sidekiq.md deleted file mode 100644 index ac92ae2eaaa..00000000000 --- a/doc/administration/high_availability/sidekiq.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -redirect_to: ../sidekiq.md ---- - -This document was moved to [another location](../sidekiq.md). diff --git a/doc/administration/housekeeping.md b/doc/administration/housekeeping.md index 4110f8b7646..c784c788b31 100644 --- a/doc/administration/housekeeping.md +++ b/doc/administration/housekeeping.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Housekeeping > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/3041) in GitLab 8.4. @@ -28,6 +34,9 @@ the `pushes_since_gc` value is 200 a `git gc` will be run. `git add`. - `git repack` ([man page](https://mirrors.edge.kernel.org/pub/software/scm/git/docs/git-repack.html)) re-organize existing packs into a single, more efficient pack. +Housekeeping will also [remove unreferenced LFS files](../raketasks/cleanup.md#remove-unreferenced-lfs-files) +from your project on the same schedule as the `git gc` operation, freeing up storage space for your project. + You can find this option under your project's **Settings > General > Advanced**.  diff --git a/doc/administration/img/export_audit_log_v13_4.png b/doc/administration/img/export_audit_log_v13_4.png Binary files differindex 1b404b5742c..e4ba330b8a9 100644 --- a/doc/administration/img/export_audit_log_v13_4.png +++ b/doc/administration/img/export_audit_log_v13_4.png diff --git a/doc/administration/incoming_email.md b/doc/administration/incoming_email.md index c0c03044225..bd075e86a15 100644 --- a/doc/administration/incoming_email.md +++ b/doc/administration/incoming_email.md @@ -82,15 +82,15 @@ instead of the regular password for the mailbox. To set up a basic Postfix mail server with IMAP access on Ubuntu, follow the [Postfix setup documentation](reply_by_email_postfix_setup.md). -### Security Concerns +### Security concerns -WARNING: **WARNING:** +CAUTION: **Caution:** Be careful when choosing the domain used for receiving incoming email. -For the sake of example, suppose your top-level company domain is `hooli.com`. +For example, suppose your top-level company domain is `hooli.com`. All employees in your company have an email address at that domain via Google Apps, and your company's private Slack instance requires a valid `@hooli.com` -email address in order to sign up. +email address to sign up. If you also host a public-facing GitLab instance at `hooli.com` and set your incoming email domain to `hooli.com`, an attacker could abuse the "Create new @@ -112,7 +112,7 @@ See GitLab issue [#30366](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/303 for a real-world example of this exploit. CAUTION: **Caution:** -Be sure to use a mail server that has been configured to reduce +Use a mail server that has been configured to reduce spam. A Postfix mail server that is running on a default configuration, for example, can result in abuse. All messages received on the configured mailbox will be processed diff --git a/doc/administration/index.md b/doc/administration/index.md index a6448fcf64f..07aa3b50bc6 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -1,8 +1,11 @@ --- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers description: 'Learn how to install, configure, update, and maintain your GitLab instance.' --- -# Administrator Docs **(CORE ONLY)** +# Administrator documentation **(CORE ONLY)** Learn how to administer your self-managed GitLab instance. @@ -12,18 +15,16 @@ GitLab has two product distributions available through [different subscriptions] - The open core [GitLab Enterprise Edition (EE)](https://gitlab.com/gitlab-org/gitlab). You can [install either GitLab CE or GitLab EE](https://about.gitlab.com/install/ce-or-ee/). -However, the features you'll have access to depend on the subscription you choose -(Core, Starter, Premium, or Ultimate). +However, the features you have access to depend on your chosen [subscription](https://about.gitlab.com/pricing/). -NOTE: **Note:** -GitLab Community Edition installations only have access to Core features. +GitLab Community Edition installations have access only to Core features. -GitLab.com is administered by GitLab, Inc., therefore, only GitLab team members have -access to its admin configurations. If you're a GitLab.com user, please check the -[user documentation](../user/index.md). +Non-administrator users can't access GitLab administration tools and settings. -NOTE: **Note:** -Non-administrator users don’t have access to GitLab administration tools and settings. +GitLab.com is administered by GitLab, Inc., and only GitLab team members have +access to its administration tools and settings. Users of GitLab.com should +instead refer to the [User documentation](../user/index.md) for GitLab +configuration and usage documentation. ## Installing and maintaining GitLab @@ -52,8 +53,10 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [GitLab Pages configuration](pages/index.md): Enable and configure GitLab Pages. - [GitLab Pages configuration for GitLab source installations](pages/source.md): Enable and configure GitLab Pages on [source installations](../install/installation.md#installation-from-source). - [Uploads administration](uploads.md): Configure GitLab uploads storage. -- [Environment variables](environment_variables.md): Supported environment variables that can be used to override their defaults values in order to configure GitLab. -- [Plugins](plugins.md): With custom plugins, GitLab administrators can introduce custom integrations without modifying GitLab's source code. +- [Environment variables](environment_variables.md): Supported environment + variables that can be used to override their default values to configure + GitLab. +- [Plugins](file_hooks.md): With custom plugins, GitLab administrators can introduce custom integrations without modifying GitLab's source code. - [Enforcing Terms of Service](../user/admin_area/settings/terms.md) - [Third party offers](../user/admin_area/settings/third_party_offers.md) - [Compliance](compliance.md): A collection of features from across the application that you may configure to help ensure that your GitLab instance and DevOps workflow meet compliance standards. @@ -82,6 +85,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Operations](operations/index.md): Keeping GitLab up and running (clean up Redis sessions, moving repositories, Sidekiq MemoryKiller, Puma). - [Restart GitLab](restart_gitlab.md): Learn how to restart GitLab and its components. - [Invalidate Markdown cache](invalidate_markdown_cache.md): Invalidate any cached Markdown. +- [Instance review](instance_review.md): Request a free review of your GitLab instance. #### Updating GitLab @@ -113,7 +117,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Kerberos authentication](../integration/kerberos.md) **(STARTER ONLY)** - See also other [authentication](../topics/authentication/index.md#gitlab-administrators) topics (for example, enforcing 2FA). - [Email users](../tools/email.md): Email GitLab users from within GitLab. **(STARTER ONLY)** -- [User Cohorts](../user/admin_area/user_cohorts.md): Display the monthly cohorts of new users and their activities over time. +- [User Cohorts](../user/admin_area/analytics/user_cohorts.md): Display the monthly cohorts of new users and their activities over time. - [Audit logs and events](audit_events.md): View the changes made within the GitLab server for: - Groups and projects. **(STARTER)** - Instances. **(PREMIUM ONLY)** diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md index abd98002934..cb37be8d9dd 100644 --- a/doc/administration/instance_limits.md +++ b/doc/administration/instance_limits.md @@ -1,4 +1,7 @@ --- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- @@ -163,7 +166,7 @@ There is a limit when embedding metrics in GFM for performance reasons. On GitLab.com, the [maximum number of webhooks and their size](../user/gitlab_com/index.md#webhooks) per project, and per group, is limited. To set this limit on a self-managed installation, run the following in the -[GitLab Rails console](troubleshooting/debug.md#starting-a-rails-console-session): +[GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): ```ruby # If limits don't exist for the default plan, you can create one with: @@ -203,7 +206,7 @@ support keyset-based pagination. More information about pagination options can b found in the [API docs section on pagination](../api/README.md#pagination). To set this limit on a self-managed installation, run the following in the -[GitLab Rails console](troubleshooting/debug.md#starting-a-rails-console-session): +[GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): ```ruby # If limits don't exist for the default plan, you can create one with: @@ -238,7 +241,7 @@ will fail with a `job_activity_limit_exceeded` error. This limit is disabled by default. To set this limit on a self-managed installation, run the following in the -[GitLab Rails console](troubleshooting/debug.md#starting-a-rails-console-session): +[GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): ```ruby # If limits don't exist for the default plan, you can create one with: @@ -264,7 +267,7 @@ limit, the subscription will be considered invalid. - On [GitLab Starter](https://about.gitlab.com/pricing/#self-managed) tier or higher self-managed installations, this limit is defined for the `default` plan that affects all projects. To set this limit on a self-managed installation, run the following in the -[GitLab Rails console](troubleshooting/debug.md#starting-a-rails-console-session): +[GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): ```ruby Plan.default.actual_limits.update!(ci_project_subscriptions: 500) @@ -290,7 +293,7 @@ or higher tiers), this limit is defined for the `default` plan that affects all projects. By default, there is no limit. To set this limit on a self-managed installation, run the following in the -[GitLab Rails console](troubleshooting/debug.md#starting-a-rails-console-session): +[GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): ```ruby Plan.default.actual_limits.update!(ci_pipeline_schedules: 100) @@ -308,7 +311,7 @@ On self-managed instances this limit is defined for the `default` plan. By defau this limit is set to `25`. To update this limit to a new value on a self-managed installation, run the following in the -[GitLab Rails console](troubleshooting/debug.md#starting-a-rails-console-session): +[GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): ```ruby Plan.default.actual_limits.update!(ci_instance_level_variables: 30) @@ -333,6 +336,7 @@ setting is used: | Artifact limit name | Default value | |---------------------------------------------|---------------| | `ci_max_artifact_size_accessibility` | 0 | +| `ci_max_artifact_size_api_fuzzing` | 0 | | `ci_max_artifact_size_archive` | 0 | | `ci_max_artifact_size_browser_performance` | 0 | | `ci_max_artifact_size_cluster_applications` | 0 | @@ -360,7 +364,7 @@ setting is used: | `ci_max_artifact_size_trace` | 0 | For example, to set the `ci_max_artifact_size_junit` limit to 10MB on a self-managed -installation, run the following in the [GitLab Rails console](troubleshooting/debug.md#starting-a-rails-console-session): +installation, run the following in the [GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): ```ruby Plan.default.actual_limits.update!(ci_max_artifact_size_junit: 10) @@ -480,7 +484,7 @@ indexed](#maximum-file-size-indexed)). - For self-managed installations it is unlimited by default This limit can be configured for self-managed installations when [enabling -Elasticsearch](../integration/elasticsearch.md#enabling-elasticsearch). +Elasticsearch](../integration/elasticsearch.md#enabling-advanced-search). NOTE: **Note:** Set the limit to `0` to disable it. @@ -527,13 +531,17 @@ More information can be found in the [Push event activities limit and bulk push > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/218017) in GitLab 13.4. -On GitLab.com, the maximum file size for a package that's uploaded to the [GitLab Package Registry](../user/packages/package_registry/index.md) -is 5 gigabytes. +On GitLab.com, the maximum file size for a package that's uploaded to the [GitLab Package Registry](../user/packages/package_registry/index.md) varies by format: -Limits are set per package type. +- Conan: 3GB +- Generic: 5GB +- Maven: 3GB +- NPM: 500MB +- NuGet: 500MB +- PyPI: 3GB To set this limit on a self-managed installation, run the following in the -[GitLab Rails console](troubleshooting/debug.md#starting-a-rails-console-session): +[GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): ```ruby # File size limit is stored in bytes @@ -552,6 +560,12 @@ Plan.default.actual_limits.update!(maven_max_file_size: 100.megabytes) # For PyPI Packages Plan.default.actual_limits.update!(pypi_max_file_size: 100.megabytes) + +# For Debian Packages +Plan.default.actual_limits.update!(debian_max_file_size: 100.megabytes) + +# For Generic Packages +Plan.default.actual_limits.update!(generic_packages_max_file_size: 100.megabytes) ``` Set the limit to `0` to allow any file size. diff --git a/doc/administration/instance_review.md b/doc/administration/instance_review.md index 8ea4bff252e..7eadb54804b 100644 --- a/doc/administration/instance_review.md +++ b/doc/administration/instance_review.md @@ -1,13 +1,25 @@ +--- +stage: Growth +group: Conversion +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Instance Review **(CORE ONLY)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/6995) in [GitLab Core](https://about.gitlab.com/pricing/) 11.3. -If you are running a medium size instance of GitLab Core edition you are qualified for a free Instance Review. You can find the button in the User menu. +If you are running a medium size instance (50+ users) of +[GitLab Core](https://about.gitlab.com/pricing/) edition, you are qualified for a +free Instance Review. - +1. Sign in as a user with Admin [permissions](../user/permissions.md). +1. In the top menu, click your user icon, and select + **Get a free instance review**: -When you click the button you will be redirected to a form with prefilled data obtained from your instance. +  -Once you submit the data to GitLab Inc. you can see the initial report. +1. GitLab redirects you to a form with prefilled data obtained from your instance. +1. Click **Submit** to see the initial report. -Additionally you will be contacted by our team for further review which should help you to improve your usage of GitLab. +A GitLab team member will contact you for further review, to provide suggestions +that will help you improve your usage of GitLab. diff --git a/doc/administration/integration/plantuml.md b/doc/administration/integration/plantuml.md index 49ea59d239c..5bdea9d8843 100644 --- a/doc/administration/integration/plantuml.md +++ b/doc/administration/integration/plantuml.md @@ -132,7 +132,7 @@ stop; You need to enable PlantUML integration from Settings under Admin Area. To do that, login with an Admin account and do following: -- In GitLab, go to **Admin Area > Settings > Integrations**. +- In GitLab, go to **Admin Area > Settings > General**. - Expand the **PlantUML** section. - Check **Enable PlantUML** checkbox. - Set the PlantUML instance as `https://gitlab.example.com/-/plantuml/`. diff --git a/doc/administration/integration/terminal.md b/doc/administration/integration/terminal.md index c363bd30543..3c53535d856 100644 --- a/doc/administration/integration/terminal.md +++ b/doc/administration/integration/terminal.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Web terminals > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/7690) in GitLab 8.15. diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index 2a79923b793..8069b12e0b9 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -99,8 +99,13 @@ artifacts, you can use an object storage like AWS S3 instead. This configuration relies on valid AWS credentials to be configured already. Use an object storage option like AWS S3 to store job artifacts. +If you configure GitLab to store artifacts on object storage, you may also want to +[eliminate local disk usage for job logs](job_logs.md#prevent-local-disk-usage). +In both cases, job logs are archived and moved to object storage when the job completes. + DANGER: **Danger:** -If you configure GitLab to store CI logs and artifacts on object storage, you must also enable [incremental logging](job_logs.md#new-incremental-logging-architecture). Otherwise, job logs will disappear or not be saved. +In a multi-server setup you must use one of the options to +[eliminate local disk usage for job logs](job_logs.md#prevent-local-disk-usage), or job logs could be lost. [Read more about using object storage with GitLab](object_storage.md). @@ -117,9 +122,9 @@ For source installations the following settings are nested under `artifacts:` an |---------|-------------|---------| | `enabled` | Enable/disable object storage | `false` | | `remote_directory` | The bucket name where Artifacts will be stored| | -| `direct_upload` | Set to true to enable direct upload of Artifacts without the need of local shared storage. Option may be removed once we decide to support only single storage for all files. | `false` | -| `background_upload` | Set to false to disable automatic upload. Option may be removed once upload is direct to S3 | `true` | -| `proxy_download` | Set to true to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` | +| `direct_upload` | Set to `true` to enable direct upload of Artifacts without the need of local shared storage. Option may be removed once we decide to support only single storage for all files. | `false` | +| `background_upload` | Set to `false` to disable automatic upload. Option may be removed once upload is direct to S3 | `true` | +| `proxy_download` | Set to `true` to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` | | `connection` | Various connection options described below | | #### Connection settings @@ -203,9 +208,9 @@ _The artifacts are stored by default in enabled: true object_store: enabled: true - remote_directory: "artifacts" # The bucket name + remote_directory: "artifacts" # The bucket name connection: - provider: AWS # Only AWS supported at the moment + provider: AWS # Only AWS supported at the moment aws_access_key_id: AWS_ACCESS_KEY_ID aws_secret_access_key: AWS_SECRET_ACCESS_KEY region: eu-central-1 @@ -316,9 +321,9 @@ _The uploads are stored by default in **In Omnibus installations:** -In order to migrate back to local storage: +To migrate back to local storage: -1. Set both `direct_upload` and `background_upload` to false in `gitlab.rb`, under the artifacts object storage settings. +1. Set both `direct_upload` and `background_upload` to `false` in `gitlab.rb`, under the artifacts object storage settings. 1. [Reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure). 1. Run `gitlab-rake gitlab:artifacts:migrate_to_local`. 1. Disable object_storage for artifacts in `gitlab.rb`: @@ -369,7 +374,7 @@ default artifacts expiration setting, which you can find in the [CI/CD Admin set > Introduced in GitLab 10.3. -To disable [the dependencies validation](../ci/yaml/README.md#when-a-dependent-job-will-fail), +To disable [the dependencies validation](../ci/yaml/README.md#when-a-dependent-job-fails), you can enable the `ci_disable_validates_dependencies` feature flag from a Rails console. **In Omnibus installations:** @@ -419,10 +424,10 @@ generated by [GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse). that are located in the artifacts archive itself. The metadata file is in a binary format, with additional Gzip compression. -GitLab does not extract the artifacts archive in order to save space, memory -and disk I/O. It instead inspects the metadata file which contains all the -relevant information. This is especially important when there is a lot of -artifacts, or an archive is a very large file. +GitLab doesn't extract the artifacts archive to save space, memory, and disk +I/O. It instead inspects the metadata file which contains all the relevant +information. This is especially important when there is a lot of artifacts, or +an archive is a very large file. When clicking on a specific file, [GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) extracts it from the archive and the download begins. This implementation saves space, diff --git a/doc/administration/job_logs.md b/doc/administration/job_logs.md index c34035e3c0c..c89ffb8da8b 100644 --- a/doc/administration/job_logs.md +++ b/doc/administration/job_logs.md @@ -65,6 +65,15 @@ job logs are automatically migrated to it along with the other job artifacts. See "Phase 4: uploading" in [Data flow](#data-flow) to learn about the process. +## Prevent local disk usage + +If you want to avoid any local disk usage for job logs, +you can do so using one of the following options: + +- Enable the [beta incremental logging](#new-incremental-logging-architecture) feature. +- Set the [job logs location](#changing-the-job-logs-local-location) + to an NFS drive. + ## How to remove job logs There isn't a way to automatically expire old job logs, but it's safe to remove diff --git a/doc/administration/lfs/index.md b/doc/administration/lfs/index.md index 5c1a9519a35..a360542ac44 100644 --- a/doc/administration/lfs/index.md +++ b/doc/administration/lfs/index.md @@ -6,14 +6,16 @@ type: reference, howto disqus_identifier: 'https://docs.gitlab.com/ee/workflow/lfs/lfs_administration.html' --- -# GitLab Git Large File Storage (LFS) Administration +# GitLab Git Large File Storage (LFS) Administration **(CORE ONLY)** + +> - Git LFS is supported in GitLab starting with version 8.2. +> - Support for object storage, such as AWS S3, was introduced in 10.0. +> - LFS is enabled in GitLab self-managed instances by default. Documentation on how to use Git LFS are under [Managing large binary files with Git LFS doc](../../topics/git/lfs/index.md). ## Requirements -- Git LFS is supported in GitLab starting with version 8.2. -- Support for object storage, such as AWS S3, was introduced in 10.0. - Users need to install [Git LFS client](https://git-lfs.github.com) version 1.0.1 and up. ## Configuration @@ -41,6 +43,8 @@ gitlab_rails['lfs_enabled'] = false gitlab_rails['lfs_storage_path'] = "/mnt/storage/lfs-objects" ``` +After you update settings in `/etc/gitlab/gitlab.rb`, make sure to run [Omnibus GitLab reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure). + ### Configuration for installations from source In `config/gitlab.yml`: diff --git a/doc/administration/libravatar.md b/doc/administration/libravatar.md index beecd9e4783..a92e6fade03 100644 --- a/doc/administration/libravatar.md +++ b/doc/administration/libravatar.md @@ -1,4 +1,7 @@ --- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: howto --- @@ -72,10 +75,9 @@ Then run `sudo gitlab-ctl reconfigure` for the changes to take effect. missing images for user email addresses that are not found on the Libravatar service. -In order to use a set other than `identicon`, replace the `&d=identicon` -portion of the URL with another supported set. -For example, you can use the `retro` set, in which case the URL would look like: -`plain_url: "http://cdn.libravatar.org/avatar/%{hash}?s=%{size}&d=retro"` +To use a set other than `identicon`, replace the `&d=identicon` portion of the +URL with another supported set. For example, you can use the `retro` set, in +which case the URL would look like: `plain_url: "http://cdn.libravatar.org/avatar/%{hash}?s=%{size}&d=retro"` ## Usage examples for Microsoft Office 365 @@ -84,8 +86,8 @@ Note that this service requires a login, so this use case is most useful in a corporate installation where all users have access to Office 365. ```ruby -gitlab_rails['gravatar_plain_url'] = 'http://outlook.office365.com/owa/service.svc/s/GetPersonaPhoto?email=%{email}&size=HR120x120' -gitlab_rails['gravatar_ssl_url'] = 'https://outlook.office365.com/owa/service.svc/s/GetPersonaPhoto?email=%{email}&size=HR120x120' +gitlab_rails['gravatar_plain_url'] = 'http://outlook.office.com/owa/service.svc/s/GetPersonaPhoto?email=%{email}&size=HR120x120' +gitlab_rails['gravatar_ssl_url'] = 'https://outlook.office.com/owa/service.svc/s/GetPersonaPhoto?email=%{email}&size=HR120x120' ``` <!-- ## Troubleshooting diff --git a/doc/administration/load_balancer.md b/doc/administration/load_balancer.md index fe534f30f66..ae4fa83662a 100644 --- a/doc/administration/load_balancer.md +++ b/doc/administration/load_balancer.md @@ -1,4 +1,7 @@ --- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- diff --git a/doc/administration/logs.md b/doc/administration/logs.md index fcd6264dafd..c9e0cca807e 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: APM +group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- @@ -100,9 +100,9 @@ The ActionCable connection or channel class is used as the `controller`. ```json { - "method":{}, - "path":{}, - "format":{}, + "method":null, + "path":null, + "format":null, "controller":"IssuesChannel", "action":"subscribe", "status":200, @@ -363,8 +363,7 @@ This file lives in `/var/log/gitlab/gitlab-rails/git_json.log` for Omnibus GitLab packages or in `/home/git/gitlab/log/git_json.log` for installations from source. -NOTE: **Note:** -After 12.2, this file was renamed from `githost.log` to +After GitLab version 12.2, this file was renamed from `githost.log` to `git_json.log` and stored in JSON format. GitLab has to interact with Git repositories, but in some rare cases @@ -599,7 +598,6 @@ installations from source. ## Unicorn Logs -NOTE: **Note:** Starting with GitLab 13.0, Puma is the default web server used in GitLab all-in-one package based installations as well as GitLab Helm chart deployments. @@ -674,10 +672,8 @@ This log records: - Information whenever [Rack Attack](../security/rack_attack.md) registers an abusive request. - Requests over the [Rate Limit](../user/admin_area/settings/rate_limits_on_raw_endpoints.md) on raw endpoints. - [Protected paths](../user/admin_area/settings/protected_paths.md) abusive requests. - -NOTE: **Note:** -In GitLab versions [12.3](https://gitlab.com/gitlab-org/gitlab/-/issues/29239) and greater, user ID and username are also -recorded on this log, if available. +- In GitLab versions [12.3](https://gitlab.com/gitlab-org/gitlab/-/issues/29239) and greater, + user ID and username, if available. ## `graphql_json.log` @@ -967,19 +963,9 @@ When [troubleshooting](troubleshooting/index.md) issues that aren't localized to previously listed components, it's helpful to simultaneously gather multiple logs and statistics from a GitLab instance. -### GitLabSOS - -If performance degradations or cascading errors occur that can't readily be attributed to one -of the previously listed GitLab components, [GitLabSOS](https://gitlab.com/gitlab-com/support/toolbox/gitlabsos/) -can provide a perspective spanning all of Omnibus GitLab. For more details and instructions -to run it, see [the GitLabSOS documentation](https://gitlab.com/gitlab-com/support/toolbox/gitlabsos/#gitlabsos). - -NOTE: **Note:** -GitLab Support likes to use this custom-made tool. - ### Briefly tail the main logs -If the bug or error is readily reproducible bug or error, save the main GitLab logs +If the bug or error is readily reproducible, save the main GitLab logs [to a file](troubleshooting/linux_cheat_sheet.md#files--dirs) while reproducing the problem once or more times: @@ -989,6 +975,16 @@ sudo gitlab-ctl tail | tee /tmp/<case-ID-and-keywords>.log Conclude the log gathering with <kbd>Ctrl</kbd> + <kbd>C</kbd>. +### GitLabSOS + +If performance degradations or cascading errors occur that can't readily be attributed to one +of the previously listed GitLab components, [GitLabSOS](https://gitlab.com/gitlab-com/support/toolbox/gitlabsos/) +can provide a perspective spanning all of Omnibus GitLab. For more details and instructions +to run it, see [the GitLabSOS documentation](https://gitlab.com/gitlab-com/support/toolbox/gitlabsos/#gitlabsos). + +NOTE: **Note:** +GitLab Support likes to use this custom-made tool. + ### Fast-stats [Fast-stats](https://gitlab.com/gitlab-com/support/toolbox/fast-stats) is a tool diff --git a/doc/administration/merge_request_diffs.md b/doc/administration/merge_request_diffs.md index 3f4cd6e2751..3c093d44780 100644 --- a/doc/administration/merge_request_diffs.md +++ b/doc/administration/merge_request_diffs.md @@ -17,12 +17,11 @@ By default, merge request diffs are stored in the database, in a table named `merge_request_diff_files`. Larger installations may find this table grows too large, in which case, switching to external storage is recommended. -## Using external storage - Merge request diffs can be stored on disk, or in object storage. In general, it -is better to store the diffs in the database than on disk. +is better to store the diffs in the database than on disk. A compromise is available +that only [stores outdated diffs](#alternative-in-database-storage) outside of database. -To enable external storage of merge request diffs, follow the instructions below. +## Using external storage **In Omnibus installations:** @@ -68,16 +67,40 @@ To enable external storage of merge request diffs, follow the instructions below ## Using object storage -CAUTION: **WARNING:** - Currently migrating to object storage is **non-reversible** +CAUTION: **Warning:** +Currently migrating to object storage is **non-reversible** Instead of storing the external diffs on disk, we recommended the use of an object store like AWS S3 instead. This configuration relies on valid AWS credentials to be configured already. +**In Omnibus installations:** + +1. Edit `/etc/gitlab/gitlab.rb` and add the following line: + + ```ruby + gitlab_rails['external_diffs_enabled'] = true + ``` + +1. Set [object storage settings](#object-storage-settings). +1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +**In installations from source:** + +1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following + lines: + + ```yaml + external_diffs: + enabled: true + ``` + +1. Set [object storage settings](#object-storage-settings). +1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. + [Read more about using object storage with GitLab](object_storage.md). -## Object Storage Settings +### Object Storage Settings NOTE: **Note:** In GitLab 13.2 and later, we recommend using the @@ -92,12 +115,12 @@ then `object_store:`. On Omnibus installations, they are prefixed by |---------|-------------|---------| | `enabled` | Enable/disable object storage | `false` | | `remote_directory` | The bucket name where external diffs will be stored| | -| `direct_upload` | Set to true to enable direct upload of external diffs without the need of local shared storage. Option may be removed once we decide to support only single storage for all files. | `false` | -| `background_upload` | Set to false to disable automatic upload. Option may be removed once upload is direct to S3 | `true` | -| `proxy_download` | Set to true to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` | +| `direct_upload` | Set to `true` to enable direct upload of external diffs without the need of local shared storage. Option may be removed once we decide to support only single storage for all files. | `false` | +| `background_upload` | Set to `false` to disable automatic upload. Option may be removed once upload is direct to S3 | `true` | +| `proxy_download` | Set to `true` to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` | | `connection` | Various connection options described below | | -### S3 compatible connection settings +#### S3 compatible connection settings See [the available connection settings for different providers](object_storage.md#connection-settings). diff --git a/doc/administration/monitoring/github_imports.md b/doc/administration/monitoring/github_imports.md index 0d79684c951..64f45001438 100644 --- a/doc/administration/monitoring/github_imports.md +++ b/doc/administration/monitoring/github_imports.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: APM +group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- diff --git a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md index efe31997b25..fcaf9aaaaee 100644 --- a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md +++ b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: APM +group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- @@ -9,17 +9,18 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32351) in GitLab 12.7, behind a disabled feature flag (`self_monitoring_project`). > - The feature flag was removed and the Self Monitoring Project was [made generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/198511) in GitLab 12.8. -GitLab has been adding the ability for administrators to see insights into the health of -their GitLab instance. In order to surface this experience in a native way, similar to how -you would interact with an application deployed via GitLab, a base project called -"GitLab self monitoring" with -[internal visibility](../../../public_access/public_access.md#internal-projects) will be -added under a group called "GitLab Instance Administrators" specifically created for -visualizing and configuring the monitoring of your GitLab instance. +GitLab has been adding the ability for administrators to see insights into the +health of their GitLab instance. To surface this experience in a native way +(similar to how you would interact with an application deployed using GitLab), +a base project called "GitLab self monitoring" with +[internal visibility](../../../public_access/public_access.md#internal-projects) +will be added under a group called "GitLab Instance Administrators" +specifically created for visualizing and configuring the monitoring of your +GitLab instance. -All administrators at the time of creation of the project and group will be added -as maintainers of the group and project, and as an admin, you'll be able to add new -members to the group in order to give them maintainer access to the project. +All administrators at the time of creation of the project and group will be +added as maintainers of the group and project, and as an admin, you'll be able +to add new members to the group to give them maintainer access to the project. This project is used to self monitor your GitLab instance. The metrics dashboard of the project shows some basic resource usage charts, such as CPU and memory usage @@ -74,7 +75,8 @@ you should ## Taking action on Prometheus alerts **(ULTIMATE)** You can [add a webhook](../../../operations/metrics/alerts.md#external-prometheus-instances) -to the Prometheus configuration in order for GitLab to receive notifications of any alerts. +to the Prometheus configuration for GitLab to receive notifications of any +alerts. Once the webhook is setup, you can [take action on incoming alerts](../../../operations/metrics/alerts.md#trigger-actions-from-alerts). diff --git a/doc/administration/monitoring/index.md b/doc/administration/monitoring/index.md index a54c25450c6..fbd4c1aa3dd 100644 --- a/doc/administration/monitoring/index.md +++ b/doc/administration/monitoring/index.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: APM +group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- diff --git a/doc/administration/monitoring/ip_whitelist.md b/doc/administration/monitoring/ip_whitelist.md index 862a9368be8..480a4ea3598 100644 --- a/doc/administration/monitoring/ip_whitelist.md +++ b/doc/administration/monitoring/ip_whitelist.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: APM +group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- diff --git a/doc/administration/monitoring/performance/gitlab_configuration.md b/doc/administration/monitoring/performance/gitlab_configuration.md index d09dabab40d..6677eb73664 100644 --- a/doc/administration/monitoring/performance/gitlab_configuration.md +++ b/doc/administration/monitoring/performance/gitlab_configuration.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: APM +group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- diff --git a/doc/administration/monitoring/performance/grafana_configuration.md b/doc/administration/monitoring/performance/grafana_configuration.md index 136a2749e80..77ed5d442e6 100644 --- a/doc/administration/monitoring/performance/grafana_configuration.md +++ b/doc/administration/monitoring/performance/grafana_configuration.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: APM +group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- @@ -17,7 +17,6 @@ or Grafana supplies package repositories (Yum/Apt) for easy installation. See [Grafana installation documentation](https://grafana.com/docs/grafana/latest/installation/) for detailed steps. -NOTE: **Note:** Before starting Grafana for the first time, set the admin user and password in `/etc/grafana/grafana.ini`. If you don't, the default password is `admin`. @@ -50,7 +49,6 @@ JSON file individually: 1. After the dashboard is imported, click the **Save dashboard** icon in the top bar:  - NOTE: **Note:** If you don't save the dashboard after importing it, the dashboard is removed when you navigate away from the page. diff --git a/doc/administration/monitoring/performance/img/performance_bar.png b/doc/administration/monitoring/performance/img/performance_bar.png Binary files differindex be06e0b2f94..380e2060b24 100644 --- a/doc/administration/monitoring/performance/img/performance_bar.png +++ b/doc/administration/monitoring/performance/img/performance_bar.png diff --git a/doc/administration/monitoring/performance/img/performance_bar_sql_queries.png b/doc/administration/monitoring/performance/img/performance_bar_sql_queries.png Binary files differindex 6b571f4e85c..ef74e0a3b6e 100644 --- a/doc/administration/monitoring/performance/img/performance_bar_sql_queries.png +++ b/doc/administration/monitoring/performance/img/performance_bar_sql_queries.png diff --git a/doc/administration/monitoring/performance/index.md b/doc/administration/monitoring/performance/index.md index 6f22327e499..68b89b837ac 100644 --- a/doc/administration/monitoring/performance/index.md +++ b/doc/administration/monitoring/performance/index.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: APM +group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- @@ -11,7 +11,7 @@ GitLab comes with its own application performance measuring system as of GitLab Community and Enterprise editions. Apart from this introduction, you are advised to read through the following -documents in order to understand and properly configure GitLab Performance Monitoring: +documents to understand and properly configure GitLab Performance Monitoring: - [GitLab Configuration](gitlab_configuration.md) - [Prometheus documentation](../prometheus/index.md) diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md index e247ec3708c..45dc4730cab 100644 --- a/doc/administration/monitoring/performance/performance_bar.md +++ b/doc/administration/monitoring/performance/performance_bar.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: APM +group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- @@ -15,7 +15,7 @@ From left to right, it displays: - **Current Host**: the current host serving the page. - **Database queries**: the time taken (in milliseconds) and the total number - of database queries, displayed in the format `00ms / 00pg`. Click to display + of database queries, displayed in the format `00ms / 00 (00 cached) pg`. Click to display a modal window with more details:  - **Gitaly calls**: the time taken (in milliseconds) and the total number of diff --git a/doc/administration/monitoring/performance/request_profiling.md b/doc/administration/monitoring/performance/request_profiling.md index 5746b95eb44..6e03404fd26 100644 --- a/doc/administration/monitoring/performance/request_profiling.md +++ b/doc/administration/monitoring/performance/request_profiling.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: APM +group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- @@ -25,7 +25,6 @@ To profile a request: curl --header 'X-Profile-Token: <token>' --header 'X-Profile-Mode: <mode>' "https://gitlab.example.com/group/project" ``` - NOTE: **Note:** Profiled requests can take longer than usual. After the request completes, you can view the profiling output from the diff --git a/doc/administration/monitoring/prometheus/gitlab_exporter.md b/doc/administration/monitoring/prometheus/gitlab_exporter.md index 686ed14ba42..971dafb4ba2 100644 --- a/doc/administration/monitoring/prometheus/gitlab_exporter.md +++ b/doc/administration/monitoring/prometheus/gitlab_exporter.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: APM +group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index ae31a3db023..adb1f719f3c 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: APM +group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- @@ -91,8 +91,8 @@ The following metrics are available: | `gitlab_transaction_rails_queue_duration_total` | Counter | 9.4 | Measures latency between GitLab Workhorse forwarding a request to Rails | `controller`, `action` | | `gitlab_transaction_view_duration_total` | Counter | 9.4 | Duration for views | `controller`, `action`, `view` | | `gitlab_view_rendering_duration_seconds` | Histogram | 10.2 | Duration for views (histogram) | `controller`, `action`, `view` | -| `http_requests_total` | Counter | 9.4 | Rack request count | `method` | -| `http_request_duration_seconds` | Histogram | 9.4 | HTTP response time from rack middleware | `method`, `status` | +| `http_requests_total` | Counter | 9.4 | Rack request count | `method`, `status` | +| `http_request_duration_seconds` | Histogram | 9.4 | HTTP response time from rack middleware | `method` | | `gitlab_transaction_db_count_total` | Counter | 13.1 | Counter for total number of SQL calls | `controller`, `action` | | `gitlab_transaction_db_write_count_total` | Counter | 13.1 | Counter for total number of write SQL calls | `controller`, `action` | | `gitlab_transaction_db_cached_count_total` | Counter | 13.1 | Counter for total number of cached SQL calls | `controller`, `action` | @@ -113,6 +113,8 @@ The following metrics are available: | `action_cable_pool_largest_size` | Gauge | 13.4 | Largest number of worker threads observed so far in ActionCable thread pool | `server_mode` | | `action_cable_pool_pending_tasks` | Gauge | 13.4 | Number of tasks waiting to be executed in ActionCable thread pool | `server_mode` | | `action_cable_pool_tasks_total` | Gauge | 13.4 | Total number of tasks executed in ActionCable thread pool | `server_mode` | +| `gitlab_issuable_fast_count_by_state_total` | Counter | 13.5 | Total number of row count operations on issue/merge request list pages | | +| `gitlab_issuable_fast_count_by_state_failures_total` | Counter | 13.5 | Number of soft-failed row count operations on issue/merge request list pages | | ## Metrics controlled by a feature flag @@ -122,6 +124,8 @@ The following metrics can be controlled by feature flags: |:---------------------------------------------------------------|:-------------------------------------------------------------------| | `gitlab_method_call_duration_seconds` | `prometheus_metrics_method_instrumentation` | | `gitlab_view_rendering_duration_seconds` | `prometheus_metrics_view_instrumentation` | +| `gitlab_issuable_fast_count_by_state_total` | `soft_fail_count_by_state` | +| `gitlab_issuable_fast_count_by_state_failures_total` | `soft_fail_count_by_state` | ## Sidekiq metrics @@ -184,12 +188,12 @@ configuration option in `gitlab.yml`. These metrics are served from the | `geo_package_files_synced` | Gauge | 13.3 | Number of syncable package files synced on secondary | `url` | | `geo_package_files_failed` | Gauge | 13.3 | Number of syncable package files failed to sync on secondary | `url` | | `geo_package_files_registry` | Gauge | 13.3 | Number of package files in the registry | `url` | -| `geo_terraform_states` | Gauge | 13.3 | Number of terraform states on primary | `url` | -| `geo_terraform_states_checksummed` | Gauge | 13.3 | Number of terraform states checksummed on primary | `url` | -| `geo_terraform_states_checksum_failed` | Gauge | 13.3 | Number of terraform states failed to calculate the checksum on primary | `url` | -| `geo_terraform_states_synced` | Gauge | 13.3 | Number of syncable terraform states synced on secondary | `url` | -| `geo_terraform_states_failed` | Gauge | 13.3 | Number of syncable terraform states failed to sync on secondary | `url` | -| `geo_terraform_states_registry` | Gauge | 13.3 | Number of terraform states in the registry | `url` | +| `geo_terraform_state_versions` | Gauge | 13.5 | Number of terraform state versions on primary | `url` | +| `geo_terraform_state_versions_checksummed` | Gauge | 13.5 | Number of terraform state versions checksummed on primary | `url` | +| `geo_terraform_state_versions_checksum_failed` | Gauge | 13.5 | Number of terraform state versions failed to calculate the checksum on primary | `url` | +| `geo_terraform_state_versions_synced` | Gauge | 13.5 | Number of syncable terraform state versions synced on secondary | `url` | +| `geo_terraform_state_versions_failed` | Gauge | 13.5 | Number of syncable terraform state versions failed to sync on secondary | `url` | +| `geo_terraform_state_versions_registry` | Gauge | 13.5 | Number of terraform state versions in the registry | `url` | | `global_search_bulk_cron_queue_size` | Gauge | 12.10 | Number of database records waiting to be synchronized to Elasticsearch | | | `global_search_awaiting_indexing_queue_size` | Gauge | 13.2 | Number of database updates waiting to be synchronized to Elasticsearch while indexing is paused | | | `geo_merge_request_diffs` | Gauge | 13.4 | Number of merge request diffs on primary | `url` | @@ -204,6 +208,9 @@ configuration option in `gitlab.yml`. These metrics are served from the | `geo_snippet_repositories_synced` | Gauge | 13.4 | Number of syncable snippets synced on secondary | `url` | | `geo_snippet_repositories_failed` | Gauge | 13.4 | Number of syncable snippets failed on secondary | `url` | | `geo_snippet_repositories_registry` | Gauge | 13.4 | Number of syncable snippets in the registry | `url` | +| `limited_capacity_worker_running_jobs` | Gauge | 13.5 | Number of running jobs | `worker` | +| `limited_capacity_worker_max_running_jobs` | Gauge | 13.5 | Maximum number of running jobs | `worker` | +| `limited_capacity_worker_remaining_work_count` | Gauge | 13.5 | Number of jobs waiting to be enqueued | `worker` | ## Database load balancing metrics **(PREMIUM ONLY)** diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index 7d93e9797be..63231996dcc 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: APM +group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- @@ -32,7 +32,7 @@ dashboard tool like [Grafana](https://grafana.com). ## Configuring Prometheus NOTE: **Note:** -For installations from source, you'll have to install and configure it yourself. +For installations from source, you must install and configure it yourself. Prometheus and its exporters are on by default, starting with GitLab 9.0. Prometheus will run as the `gitlab-prometheus` user and listen on @@ -60,9 +60,9 @@ it's not recommended to change the port Prometheus listens on, as this might affect or conflict with other services running on the GitLab server. Proceed at your own risk. -In order to access Prometheus from outside the GitLab server you will need to -set a FQDN or IP in `prometheus['listen_address']`. -To change the address/port that Prometheus listens on: +To access Prometheus from outside the GitLab server, set an FQDN or IP in +`prometheus['listen_address']`. To change the address/port that Prometheus +listens on: 1. Edit `/etc/gitlab/gitlab.rb` 1. Add or find and uncomment the following line: @@ -179,7 +179,7 @@ The next step is to tell all the other nodes where the monitoring node is: take effect. NOTE: **Note:** -Once monitoring using Service Discovery is enabled with `consul['monitoring_service_discovery'] = true`, +After monitoring using Service Discovery is enabled with `consul['monitoring_service_discovery'] = true`, ensure that `prometheus['scrape_configs']` is not set in `/etc/gitlab/gitlab.rb`. Setting both `consul['monitoring_service_discovery'] = true` and `prometheus['scrape_configs']` in `/etc/gitlab/gitlab.rb` will result in errors. @@ -312,7 +312,6 @@ To use an external Prometheus server: You can visit `http://localhost:9090` for the dashboard that Prometheus offers by default. -NOTE: **Note:** If SSL has been enabled on your GitLab instance, you may not be able to access Prometheus on the same browser as GitLab if using the same FQDN due to [HSTS](https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security). We plan to [provide access via GitLab](https://gitlab.com/gitlab-org/multi-user-prometheus), but in the interim there are diff --git a/doc/administration/monitoring/prometheus/node_exporter.md b/doc/administration/monitoring/prometheus/node_exporter.md index 07f6b8b8e1e..dae1f02b196 100644 --- a/doc/administration/monitoring/prometheus/node_exporter.md +++ b/doc/administration/monitoring/prometheus/node_exporter.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: APM +group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- @@ -10,7 +10,7 @@ The [node exporter](https://github.com/prometheus/node_exporter) enables you to various machine resources such as memory, disk and CPU utilization. NOTE: **Note:** -For installations from source you'll have to install and configure it yourself. +For installations from source you must install and configure it yourself. To enable the node exporter: diff --git a/doc/administration/monitoring/prometheus/pgbouncer_exporter.md b/doc/administration/monitoring/prometheus/pgbouncer_exporter.md index 62d0bf684b6..4554bc06401 100644 --- a/doc/administration/monitoring/prometheus/pgbouncer_exporter.md +++ b/doc/administration/monitoring/prometheus/pgbouncer_exporter.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: APM +group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- @@ -12,7 +12,7 @@ The [PgBouncer exporter](https://github.com/prometheus-community/pgbouncer_expor you to measure various [PgBouncer](https://www.pgbouncer.org/) metrics. NOTE: **Note:** -For installations from source you'll have to install and configure it yourself. +For installations from source you must install and configure it yourself. To enable the PgBouncer exporter: diff --git a/doc/administration/monitoring/prometheus/postgres_exporter.md b/doc/administration/monitoring/prometheus/postgres_exporter.md index e3fff45fce3..9eb9ba3c59f 100644 --- a/doc/administration/monitoring/prometheus/postgres_exporter.md +++ b/doc/administration/monitoring/prometheus/postgres_exporter.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: APM +group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w The [PostgreSQL Server Exporter](https://github.com/wrouesnel/postgres_exporter) allows you to export various PostgreSQL metrics. NOTE: **Note:** -For installations from source you will have to install and configure it yourself. +For installations from source you must install and configure it yourself. To enable the PostgreSQL Server Exporter: @@ -20,7 +20,6 @@ To enable the PostgreSQL Server Exporter: postgres_exporter['enable'] = true ``` - NOTE: **Note:** If PostgreSQL Server Exporter is configured on a separate node, make sure that the local address is [listed in `trust_auth_cidr_addresses`](../../postgresql/replication_and_failover.md#network-information) or the exporter will not be able to connect to the database. diff --git a/doc/administration/monitoring/prometheus/redis_exporter.md b/doc/administration/monitoring/prometheus/redis_exporter.md index b7c66959349..16a758c9804 100644 --- a/doc/administration/monitoring/prometheus/redis_exporter.md +++ b/doc/administration/monitoring/prometheus/redis_exporter.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: APM +group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- @@ -11,7 +11,7 @@ various [Redis](https://redis.io) metrics. For more information on what is expor [read the upstream documentation](https://github.com/oliver006/redis_exporter/blob/master/README.md#whats-exported). NOTE: **Note:** -For installations from source you'll have to install and configure it yourself. +For installations from source you must install and configure it yourself. To enable the Redis exporter: diff --git a/doc/administration/monitoring/prometheus/registry_exporter.md b/doc/administration/monitoring/prometheus/registry_exporter.md index 3d28b26b685..3bf4db8a665 100644 --- a/doc/administration/monitoring/prometheus/registry_exporter.md +++ b/doc/administration/monitoring/prometheus/registry_exporter.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: APM +group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- diff --git a/doc/administration/nfs.md b/doc/administration/nfs.md index b54f05ad536..fabbfb2552e 100644 --- a/doc/administration/nfs.md +++ b/doc/administration/nfs.md @@ -1,4 +1,7 @@ --- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- @@ -296,6 +299,25 @@ Having multiple NFS mounts will require manually making sure the data directorie are empty before attempting a restore. Read more about the [restore prerequisites](../raketasks/backup_restore.md). +## Testing NFS + +Once you've set up the NFS server and client, you can verify NFS is configured correctly +by testing the following commands: + +```shell +sudo mkdir /gitlab-nfs/test-dir +sudo chown git /gitlab-nfs/test-dir +sudo chgrp gitlab-www /gitlab-nfs/test-dir +sudo chgrp root /gitlab-nfs/test-dir +sudo chmod 2755 /gitlab-nfs/test-dir +sudo -u git mkdir /gitlab-nfs/test-dir/test2 +sudo -u git chmod 2755 /gitlab-nfs/test-dir/test2 +sudo ls -lah /gitlab-nfs/test-dir/test2 +sudo -u git rm -r /gitlab-nfs/test-dir +``` + +Any `Operation not permitted` errors means you should investigate your NFS server export options. + ## NFS in a Firewalled Environment If the traffic between your NFS server and NFS client(s) is subject to port filtering @@ -317,6 +339,28 @@ sudo ufw allow from <client_ip_address> to any port nfs ## Known issues +### Upgrade to Gitaly Cluster or disable caching if experiencing data loss + +CAUTION: **Caution:** +From GitLab 13.0, using NFS for Git repositories is deprecated. In GitLab 14.0, +support for NFS for Git repositories is scheduled to be removed. Upgrade to +[Gitaly Cluster](gitaly/praefect.md) as soon as possible. + +Customers and users have reported data loss on high-traffic repositories when using NFS for Git repositories. +For example, we have seen [inconsistent updates after a push](https://gitlab.com/gitlab-org/gitaly/-/issues/2589). The problem may be partially mitigated by adjusting caching using the following NFS client mount options: + +| Setting | Description | +| ------- | ----------- | +| `lookupcache=positive` | Tells the NFS client to honor `positive` cache results but invalidates any `negative` cache results. Negative cache results cause problems with Git. Specifically, a `git push` can fail to register uniformly across all NFS clients. The negative cache causes the clients to 'remember' that the files did not exist previously. +| `actimeo=0` | Sets the time to zero that the NFS client caches files and directories before requesting fresh information from a server. | +| `noac` | Tells the NFS client not to cache file attributes and forces application writes to become synchronous so that local changes to a file become visible on the server immediately. | + +CAUTION: **Caution:** +The `actimeo=0` and `noac` options both result in a significant reduction in performance, possibly leading to timeouts. +You may be able to avoid timeouts and data loss using `actimeo=0` and `lookupcache=positive` _without_ `noac`, however +we expect the performance reduction will still be significant. As noted above, we strongly recommend upgrading to +[Gitaly Cluster](gitaly/praefect.md) as soon as possible. + ### Avoid using AWS's Elastic File System (EFS) GitLab strongly recommends against using AWS Elastic File System (EFS). diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md index 39365ffe404..8b788e6d91d 100644 --- a/doc/administration/object_storage.md +++ b/doc/administration/object_storage.md @@ -1,4 +1,7 @@ --- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- @@ -17,7 +20,7 @@ GitLab has been tested on a number of object storage providers: - [Google Cloud Storage](https://cloud.google.com/storage) - [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces/) - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) -- [Openstack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) +- [OpenStack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) - On-premises hardware and appliances from various storage vendors. - MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. @@ -369,7 +372,7 @@ Here are the valid connection parameters for Rackspace Cloud, provided by | `rackspace_username` | The username of the Rackspace account with access to the container | `joe.smith` | | `rackspace_api_key` | The API key of the Rackspace account with access to the container | `ABC123DEF456ABC123DEF456ABC123DE` | | `rackspace_region` | The Rackspace storage region to use, a three letter code from the [list of service access endpoints](https://developer.rackspace.com/docs/cloud-files/v1/general-api-info/service-access/) | `iad` | -| `rackspace_temp_url_key` | The private key you have set in the Rackspace API for temporary URLs. Read more [here](https://developer.rackspace.com/docs/cloud-files/v1/use-cases/public-access-to-your-cloud-files-account/#tempurl) | `ABC123DEF456ABC123DEF456ABC123DE` | +| `rackspace_temp_url_key` | The private key you have set in the Rackspace API for [temporary URLs](https://developer.rackspace.com/docs/cloud-files/v1/use-cases/public-access-to-your-cloud-files-account/#tempurl). | `ABC123DEF456ABC123DEF456ABC123DE` | NOTE: **Note:** Regardless of whether the container has public access enabled or disabled, Fog will @@ -383,7 +386,7 @@ and comparing the output of the returned headers. The following YAML shows how the `object_store` section defines object-specific configuration block and how the `enabled` and -`proxy_download` flags can be overriden. The `bucket` is the only +`proxy_download` flags can be overridden. The `bucket` is the only required parameter within each type: ```yaml @@ -503,13 +506,13 @@ supported by consolidated configuration form, refer to the following guides: |Object storage type|Supported by consolidated configuration?| |-------------------|----------------------------------------| | [Backups](../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage)|No| -| [Job artifacts](job_artifacts.md#using-object-storage) and [incremental logging](job_logs.md#new-incremental-logging-architecture) | Yes | +| [Job artifacts](job_artifacts.md#using-object-storage) including archived job logs | Yes | | [LFS objects](lfs/index.md#storing-lfs-objects-in-remote-object-storage) | Yes | | [Uploads](uploads.md#using-object-storage) | Yes | | [Container Registry](packages/container_registry.md#use-object-storage) (optional feature) | No | | [Merge request diffs](merge_request_diffs.md#using-object-storage) | Yes | | [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | -| [Packages](packages/index.md#using-object-storage) (optional feature) **(PREMIUM ONLY)** | Yes | +| [Packages](packages/index.md#using-object-storage) (optional feature) | Yes | | [Dependency Proxy](packages/dependency_proxy.md#using-object-storage) (optional feature) **(PREMIUM ONLY)** | Yes | | [Pseudonymizer](pseudonymizer.md#configuration) (optional feature) **(ULTIMATE ONLY)** | No | | [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | @@ -520,12 +523,13 @@ supported by consolidated configuration form, refer to the following guides: If you're working to [scale out](reference_architectures/index.md) your GitLab implementation, or add fault tolerance and redundancy, you may be looking at removing dependencies on block or network filesystems. -See the following guides and +See the following additional guides and [note that Pages requires disk storage](#gitlab-pages-requires-nfs): 1. Make sure the [`git` user home directory](https://docs.gitlab.com/omnibus/settings/configuration.html#moving-the-home-directory-for-a-user) is on local disk. 1. Configure [database lookup of SSH keys](operations/fast_ssh_key_lookup.md) to eliminate the need for a shared `authorized_keys` file. +1. [Prevent local disk usage for job logs](job_logs.md#prevent-local-disk-usage). ## Warnings, limitations, and known issues @@ -569,7 +573,8 @@ The dependency on disk storage also prevents Pages being deployed using the ### Incremental logging is required for CI to use object storage If you configure GitLab to use object storage for CI logs and artifacts, -[you must also enable incremental logging](job_artifacts.md#using-object-storage). +you can avoid [local disk usage for job logs](job_logs.md#data-flow) by enabling +[beta incremental logging](job_logs.md#new-incremental-logging-architecture). ### Proxy Download @@ -685,7 +690,7 @@ in the `storage_options` configuration section: | Setting | Description | |-------------------------------------|-------------| -| `server_side_encryption` | Encryption mode (AES256 or aws:kms) | +| `server_side_encryption` | Encryption mode (`AES256` or `aws:kms`) | | `server_side_encryption_kms_key_id` | Amazon Resource Name. Only needed when `aws:kms` is used in `server_side_encryption`. See the [Amazon documentation on using KMS encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html) | As with the case for default encryption, these options only work when diff --git a/doc/administration/operations/cleaning_up_redis_sessions.md b/doc/administration/operations/cleaning_up_redis_sessions.md index d2aec2f7c47..a94f88439f2 100644 --- a/doc/administration/operations/cleaning_up_redis_sessions.md +++ b/doc/administration/operations/cleaning_up_redis_sessions.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Cleaning up stale Redis sessions Since version 6.2, GitLab stores web user sessions as key-value pairs in Redis. diff --git a/doc/administration/operations/extra_sidekiq_processes.md b/doc/administration/operations/extra_sidekiq_processes.md index e589ecc4216..76dc9bf5510 100644 --- a/doc/administration/operations/extra_sidekiq_processes.md +++ b/doc/administration/operations/extra_sidekiq_processes.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Run multiple Sidekiq processes **(CORE ONLY)** GitLab allows you to start multiple Sidekiq processes. diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md index 6cd393be330..88ef69b29f2 100644 --- a/doc/administration/operations/fast_ssh_key_lookup.md +++ b/doc/administration/operations/fast_ssh_key_lookup.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Fast lookup of authorized SSH keys in the database > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/1631) in [GitLab Starter](https://about.gitlab.com/pricing/) 9.3. @@ -114,7 +120,6 @@ This is a brief overview. Please refer to the above instructions for more contex 1. Enable writes to the `authorized_keys` file in Application Settings 1. Remove the `AuthorizedKeysCommand` lines from `/etc/ssh/sshd_config` or from `/assets/sshd_config` if you are using Omnibus Docker. 1. Reload `sshd`: `sudo service sshd reload` -1. Remove the `/opt/gitlab-shell/authorized_keys` file ## Compiling a custom version of OpenSSH for CentOS 6 diff --git a/doc/administration/operations/filesystem_benchmarking.md b/doc/administration/operations/filesystem_benchmarking.md index 64afd1b44f3..c55f253b772 100644 --- a/doc/administration/operations/filesystem_benchmarking.md +++ b/doc/administration/operations/filesystem_benchmarking.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Filesystem Performance Benchmarking Filesystem performance has a big impact on overall GitLab performance, diff --git a/doc/administration/operations/index.md b/doc/administration/operations/index.md index 45b8e5ad448..aaeb8c723d0 100644 --- a/doc/administration/operations/index.md +++ b/doc/administration/operations/index.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Performing Operations in GitLab Keep your GitLab instance up and running smoothly. diff --git a/doc/administration/operations/moving_repositories.md b/doc/administration/operations/moving_repositories.md index 4763c012538..94eea1e25b8 100644 --- a/doc/administration/operations/moving_repositories.md +++ b/doc/administration/operations/moving_repositories.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Moving repositories managed by GitLab Sometimes you need to move all repositories managed by GitLab to diff --git a/doc/administration/operations/puma.md b/doc/administration/operations/puma.md index f5b09d7a978..2d53a790428 100644 --- a/doc/administration/operations/puma.md +++ b/doc/administration/operations/puma.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Switching to Puma As of GitLab 12.9, [Puma](https://github.com/puma/puma) has replaced [Unicorn](https://yhbt.net/unicorn/) diff --git a/doc/administration/operations/rails_console.md b/doc/administration/operations/rails_console.md new file mode 100644 index 00000000000..9db9a885baf --- /dev/null +++ b/doc/administration/operations/rails_console.md @@ -0,0 +1,144 @@ +# The Rails Console + +The [Rails console](https://guides.rubyonrails.org/command_line.html#rails-console). +provides a way to interact with your GitLab instance from the command line. + +CAUTION: **Caution:** +The Rails console interacts directly with GitLab. In many cases, +there are no handrails to prevent you from permanently modifying, corrupting +or destroying production data. If you would like to explore the Rails console +with no consequences, you are strongly advised to do so in a test environment. + +The Rails console is for GitLab system administrators who are troubleshooting +a problem or need to retrieve some data that can only be done through direct +access of the GitLab application. + +## Starting a Rails console session + +**For Omnibus installations** + +```shell +sudo gitlab-rails console +``` + +**For installations from source** + +```shell +sudo -u git -H bundle exec rails console -e production +``` + +**For Kubernetes deployments** + +The console is in the task-runner pod. Refer to our [Kubernetes cheat sheet](../troubleshooting/kubernetes_cheat_sheet.md#gitlab-specific-kubernetes-information) for details. + +To exit the console, type: `quit`. + +## Output Rails console session history + +Enter the following command on the rails console to display +your command history. + +```ruby +puts Readline::HISTORY.to_a +``` + +You can then copy it to your clipboard and save for future reference. + +## Using the Rails Runner + +If you need to run some Ruby code in the context of your GitLab production +environment, you can do so using the [Rails Runner](https://guides.rubyonrails.org/command_line.html#rails-runner). +When executing a script file, the script must be accessible by the `git` user. + +When the command or script completes, the Rails Runner process finishes. +It is useful for running within other scripts or cron jobs for example. + +**For Omnibus installations** + +```shell +sudo gitlab-rails runner "RAILS_COMMAND" + +# Example with a two-line Ruby script +sudo gitlab-rails runner "user = User.first; puts user.username" + +# Example with a ruby script file (make sure to use the full path) +sudo gitlab-rails runner /path/to/script.rb +``` + +**For installations from source** + +```shell +sudo -u git -H bundle exec rails runner -e production "RAILS_COMMAND" + +# Example with a two-line Ruby script +sudo -u git -H bundle exec rails runner -e production "user = User.first; puts user.username" + +# Example with a ruby script file (make sure to use the full path) +sudo -u git -H bundle exec rails runner -e production /path/to/script.rb +``` + +Rails Runner does not produce the same output as the console. + +If you set a variable on the console, the console will generate useful debug output +such as the variable contents or properties of referenced entity: + +```ruby +irb(main):001:0> user = User.first +=> #<User id:1 @root> +``` + +Rails Runner does not do this: you have to be explicit about generating +output: + +```shell +$ sudo gitlab-rails runner "user = User.first" +$ sudo gitlab-rails runner "user = User.first; puts user.username ; puts user.id" +root +1 +``` + +Some basic knowledge of Ruby will be very useful. Try [this +30-minute tutorial](https://try.ruby-lang.org/) for a quick introduction. +Rails experience is helpful but not essential. + +### Troubleshooting Rails Runner + +The `gitlab-rails` command executes Rails Runner using a non-root account and group, by default: `git:git`. + +If the non-root account cannot find the Ruby script filename passed to `gitlab-rails runner` +you may get a syntax error, not an error that the file couldn't be accessed. + +A common reason for this is that the script has been put in the root account's home directory. + +`runner` tries to parse the path and file parameter as Ruby code. + +For example: + +```plaintext +[root ~]# echo 'puts "hello world"' > ./helloworld.rb +[root ~]# sudo gitlab-rails runner ./helloworld.rb +Please specify a valid ruby command or the path of a script to run. +Run 'rails runner -h' for help. + +/opt/gitlab/..../runner_command.rb:45: syntax error, unexpected '.' +./helloworld.rb +^ +[root ~]# sudo gitlab-rails runner /root/helloworld.rb +Please specify a valid ruby command or the path of a script to run. +Run 'rails runner -h' for help. + +/opt/gitlab/..../runner_command.rb:45: unknown regexp options - hllwrld +[root ~]# mv ~/helloworld.rb /tmp +[root ~]# sudo gitlab-rails runner /tmp/helloworld.rb +hello world +``` + +A meaningful error should be generated if the directory can be accessed, but the file cannot: + +```plaintext +[root ~]# chmod 400 /tmp/helloworld.rb +[root ~]# sudo gitlab-rails runner /tmp/helloworld.rb +Traceback (most recent call last): + [traceback removed] +/opt/gitlab/..../runner_command.rb:42:in `load': cannot load such file -- /tmp/helloworld.rb (LoadError) +``` diff --git a/doc/administration/operations/sidekiq_memory_killer.md b/doc/administration/operations/sidekiq_memory_killer.md index d1ff98a0079..d99468411e3 100644 --- a/doc/administration/operations/sidekiq_memory_killer.md +++ b/doc/administration/operations/sidekiq_memory_killer.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Sidekiq MemoryKiller The GitLab Rails application code suffers from memory leaks. For web requests @@ -8,7 +14,7 @@ MemoryKiller applies the same approach to the Sidekiq processes used by GitLab to process background jobs. Unlike puma-worker-killer, which is enabled by default for all GitLab -installations since GitLab 13.0, the Sidekiq MemoryKiller is enabled by default +installations of GitLab 13.0 and later, the Sidekiq MemoryKiller is enabled by default _only_ for Omnibus packages. The reason for this is that the MemoryKiller relies on runit to restart Sidekiq after a memory-induced shutdown and GitLab installations from source do not all use runit or an equivalent. diff --git a/doc/administration/operations/ssh_certificates.md b/doc/administration/operations/ssh_certificates.md index c81eb15955d..7cbd8c74f90 100644 --- a/doc/administration/operations/ssh_certificates.md +++ b/doc/administration/operations/ssh_certificates.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # User lookup via OpenSSH's AuthorizedPrincipalsCommand > [Available in](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/19911) GitLab diff --git a/doc/administration/operations/unicorn.md b/doc/administration/operations/unicorn.md index a1593c3a6c3..80dafde14aa 100644 --- a/doc/administration/operations/unicorn.md +++ b/doc/administration/operations/unicorn.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Understanding Unicorn and unicorn-worker-killer NOTE: **Note:** diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md index 74af5c8149b..56b7f01e1ad 100644 --- a/doc/administration/packages/container_registry.md +++ b/doc/administration/packages/container_registry.md @@ -10,15 +10,13 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - Container Registry manifest `v1` support was added in GitLab 8.9 to support > Docker versions earlier than 1.10. -NOTE: **Note:** -This document is the administrator's guide. To learn how to use GitLab Container -Registry, see the [user documentation](../../user/packages/container_registry/index.md). +With the GitLab Container Registry, every project can have its +own space to store Docker images. -With the Container Registry integrated into GitLab, every project can have its -own space to store its Docker images. +Read more about the Docker Registry in [the Docker documentation](https://docs.docker.com/registry/introduction/). -You can read more about the Docker Registry at -<https://docs.docker.com/registry/introduction/>. +This document is the administrator's guide. To learn how to use the GitLab Container +Registry, see the [user documentation](../../user/packages/container_registry/index.md). ## Enable the Container Registry @@ -37,9 +35,8 @@ Otherwise, the Container Registry is not enabled. To enable it: - You can configure it for your [GitLab domain](#configure-container-registry-under-an-existing-gitlab-domain), or - You can configure it for [a different domain](#configure-container-registry-under-its-own-domain). -NOTE: **Note:** The Container Registry works under HTTPS by default. You can use HTTP -but it's not recommended and is out of the scope of this document. +but it's not recommended and is beyond the scope of this document. Read the [insecure Registry documentation](https://docs.docker.com/registry/insecure/) if you want to implement this. @@ -47,12 +44,12 @@ if you want to implement this. If you have installed GitLab from source: -1. You will have to [install Registry](https://docs.docker.com/registry/deploying/) by yourself. -1. After the installation is complete, you will have to configure the Registry's - settings in `gitlab.yml` in order to enable it. -1. Use the sample NGINX configuration file that is found under +1. You must [install Registry](https://docs.docker.com/registry/deploying/) by yourself. +1. After the installation is complete, to enable it, you must configure the Registry's + settings in `gitlab.yml`. +1. Use the sample NGINX configuration file from under [`lib/support/nginx/registry-ssl`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/support/nginx/registry-ssl) and edit it to match the - `host`, `port` and TLS certs paths. + `host`, `port`, and TLS certificate paths. The contents of `gitlab.yml` are: @@ -67,21 +64,20 @@ registry: issuer: gitlab-issuer ``` -where: +Where: | Parameter | Description | | --------- | ----------- | | `enabled` | `true` or `false`. Enables the Registry in GitLab. By default this is `false`. | -| `host` | The host URL under which the Registry will run and the users will be able to use. | -| `port` | The port under which the external Registry domain will listen on. | -| `api_url` | The internal API URL under which the Registry is exposed to. It defaults to `http://localhost:5000`. | +| `host` | The host URL under which the Registry runs and users can use. | +| `port` | The port the external Registry domain listens on. | +| `api_url` | The internal API URL under which the Registry is exposed. It defaults to `http://localhost:5000`. | | `key` | The private key location that is a pair of Registry's `rootcertbundle`. Read the [token auth configuration documentation](https://docs.docker.com/registry/configuration/#token). | | `path` | This should be the same directory like specified in Registry's `rootdirectory`. Read the [storage configuration documentation](https://docs.docker.com/registry/configuration/#storage). This path needs to be readable by the GitLab user, the web-server user and the Registry user. Read more in [#configure-storage-for-the-container-registry](#configure-storage-for-the-container-registry). | | `issuer` | This should be the same value as configured in Registry's `issuer`. Read the [token auth configuration documentation](https://docs.docker.com/registry/configuration/#token). | -NOTE: **Note:** A Registry init file is not shipped with GitLab if you install it from source. -Hence, [restarting GitLab](../restart_gitlab.md#installations-from-source) will not restart the Registry should +Hence, [restarting GitLab](../restart_gitlab.md#installations-from-source) does not restart the Registry should you modify its settings. Read the upstream documentation on how to achieve that. At the **absolute** minimum, make sure your [Registry configuration](https://docs.docker.com/registry/configuration/#auth) @@ -98,37 +94,34 @@ auth: ``` CAUTION: **Caution:** -If `auth` is not set up, users will be able to pull Docker images without authentication. +If `auth` is not set up, users can pull Docker images without authentication. ## Container Registry domain configuration There are two ways you can configure the Registry's external domain. Either: -- [Use the existing GitLab domain](#configure-container-registry-under-an-existing-gitlab-domain) where in that case - the Registry will have to listen on a port and reuse GitLab's TLS certificate, +- [Use the existing GitLab domain](#configure-container-registry-under-an-existing-gitlab-domain). + The Registry listens on a port and reuses GitLab's TLS certificate. - [Use a completely separate domain](#configure-container-registry-under-its-own-domain) with a new TLS certificate for that domain. -Since the container Registry requires a TLS certificate, in the end it all boils -down to how easy or pricey it is to get a new one. +Because the Container Registry requires a TLS certificate, cost may be a factor. -Please take this into consideration before configuring the Container Registry +Take this into consideration before configuring the Container Registry for the first time. ### Configure Container Registry under an existing GitLab domain If the Registry is configured to use the existing GitLab domain, you can -expose the Registry on a port so that you can reuse the existing GitLab TLS +expose the Registry on a port. This way you can reuse the existing GitLab TLS certificate. -Assuming that the GitLab domain is `https://gitlab.example.com` and the port the -Registry is exposed to the outside world is `5050`, here is what you need to set +If the GitLab domain is `https://gitlab.example.com` and the port to the outside world is `5050`, here is what you need to set in `gitlab.rb` or `gitlab.yml` if you are using Omnibus GitLab or installed GitLab from source respectively. -NOTE: **Note:** -Be careful to choose a port different than the one that Registry listens to (`5000` by default), -otherwise you will run into conflicts. +Ensure you choose a port different than the one that Registry listens to (`5000` by default), +otherwise conflicts occur. **Omnibus GitLab installations** @@ -139,7 +132,7 @@ otherwise you will run into conflicts. registry_external_url 'https://gitlab.example.com:5050' ``` - Note how the `registry_external_url` is listening on HTTPS under the + The `registry_external_url` is listening on HTTPS under the existing GitLab URL, but on a different port. If your TLS certificate is not in `/etc/gitlab/ssl/gitlab.example.com.crt` @@ -160,7 +153,6 @@ otherwise you will run into conflicts. openssl s_client -showcerts -servername gitlab.example.com -connect gitlab.example.com:5050 > cacert.pem ``` -NOTE: **Note:** If your certificate provider provides the CA Bundle certificates, append them to the TLS certificate file. **Installations from source** @@ -187,12 +179,11 @@ docker login gitlab.example.com:5050 ### Configure Container Registry under its own domain -If the Registry is configured to use its own domain, you will need a TLS -certificate for that specific domain (e.g., `registry.example.com`) or maybe +When the Registry is configured to use its own domain, you need a TLS +certificate for that specific domain (for example, `registry.example.com`). You might need a wildcard certificate if hosted under a subdomain of your existing GitLab -domain (e.g., `registry.gitlab.example.com`). +domain, for example, `registry.gitlab.example.com`. -NOTE: **Note:** As well as manually generated SSL certificates (explained here), certificates automatically generated by Let's Encrypt are also [supported in Omnibus installs](https://docs.gitlab.com/omnibus/settings/ssl.html#host-services). @@ -210,19 +201,19 @@ Let's assume that you want the container Registry to be accessible at chmod 600 /etc/gitlab/ssl/registry.gitlab.example.com.* ``` -1. Once the TLS certificate is in place, edit `/etc/gitlab/gitlab.rb` with: +1. After the TLS certificate is in place, edit `/etc/gitlab/gitlab.rb` with: ```ruby registry_external_url 'https://registry.gitlab.example.com' ``` - Note how the `registry_external_url` is listening on HTTPS. + The `registry_external_url` is listening on HTTPS. 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -If you have a [wildcard certificate](https://en.wikipedia.org/wiki/Wildcard_certificate), you need to specify the path to the -certificate in addition to the URL, in this case `/etc/gitlab/gitlab.rb` will -look like: +If you have a [wildcard certificate](https://en.wikipedia.org/wiki/Wildcard_certificate), you must specify the path to the +certificate in addition to the URL, in this case `/etc/gitlab/gitlab.rb` +looks like: ```ruby registry_nginx['ssl_certificate'] = "/etc/gitlab/ssl/certificate.pem" @@ -252,9 +243,8 @@ docker login registry.gitlab.example.com ## Disable Container Registry site-wide -NOTE: **Note:** -Disabling the Registry in the Rails GitLab application as set by the following -steps, will not remove any existing Docker images. This is handled by the +When you disable the Registry by following these steps, you do not +remove any existing Docker images. This is handled by the Registry application itself. **Omnibus GitLab** @@ -281,7 +271,7 @@ Registry application itself. ## Disable Container Registry for new projects site-wide -If the Container Registry is enabled, then it will be available on all new +If the Container Registry is enabled, then it should be available on all new projects. To disable this function and let the owners of a project to enable the Container Registry by themselves, follow the steps below. @@ -317,7 +307,7 @@ the Container Registry by themselves, follow the steps below. You can configure the Container Registry to use various storage backends by configuring a storage driver. By default the GitLab Container Registry -is configured to use the [filesystem driver](#use-filesystem) +is configured to use the [file system driver](#use-file-system) configuration. The different supported drivers are: @@ -331,15 +321,14 @@ The different supported drivers are: | swift | OpenStack Swift Object Storage | | oss | Aliyun OSS | -NOTE: **Note:** -Although most S3 compatible services (like [MinIO](https://min.io/)) should work with the registry, we only guarantee support for AWS S3. Because we cannot assert the correctness of third-party S3 implementations, we can debug issues, but we cannot patch the registry unless an issue is reproducible against an AWS S3 bucket. +Although most S3 compatible services (like [MinIO](https://min.io/)) should work with the Container Registry, we only guarantee support for AWS S3. Because we cannot assert the correctness of third-party S3 implementations, we can debug issues, but we cannot patch the registry unless an issue is reproducible against an AWS S3 bucket. Read more about the individual driver's configuration options in the [Docker Registry docs](https://docs.docker.com/registry/configuration/#storage). -### Use filesystem +### Use file system -If you want to store your images on the filesystem, you can change the storage +If you want to store your images on the file system, you can change the storage path for the Container Registry, follow the steps below. This path is accessible to: @@ -347,8 +336,7 @@ This path is accessible to: - The user running the Container Registry daemon. - The user running GitLab. -CAUTION: **Warning:** -You should confirm that all GitLab, Registry and web server users +All GitLab, Registry, and web server users must have access to this directory. **Omnibus GitLab installations** @@ -387,13 +375,10 @@ driver for the Container Registry. [Read more about using object storage with GitLab](../object_storage.md). CAUTION: **Warning:** -GitLab will not backup Docker images that are not stored on the -filesystem. Remember to enable backups with your object storage provider if +GitLab does not back up Docker images that are not stored on the +file system. Enable backups with your object storage provider if desired. -NOTE: **Note:** -`regionendpoint` is only required when configuring an S3 compatible service such as MinIO. It takes a URL such as `http://127.0.0.1:9000`. - **Omnibus GitLab installations** To configure the `s3` storage driver in Omnibus: @@ -412,14 +397,14 @@ To configure the `s3` storage driver in Omnibus: } ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + - `regionendpoint` is only required when configuring an S3 compatible service such as MinIO. It takes a URL such as `http://127.0.0.1:9000`. + - `your-s3-bucket` should be the name of a bucket that exists, and can't include subdirectories. -NOTE: **Note:** -`your-s3-bucket` should only be the name of a bucket that exists, and can't include subdirectories. +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. **Installations from source** -Configuring the storage driver is done in your registry configuration YML file created +Configuring the storage driver is done in the registry configuration YML file created when you [deployed your Docker registry](https://docs.docker.com/registry/deploying/). `s3` storage driver example: @@ -438,8 +423,7 @@ storage: enabled: true ``` -NOTE: **Note:** -`your-s3-bucket` should only be the name of a bucket that exists, and can't include subdirectories. +`your-s3-bucket` should be the name of a bucket that exists, and can't include subdirectories. #### Migrate to object storage without downtime @@ -451,7 +435,7 @@ you can pull from the Container Registry, but you cannot push. 1. Optional: To reduce the amount of data to be migrated, run the [garbage collection tool without downtime](#performing-garbage-collection-without-downtime). 1. This example uses the `aws` CLI. If you haven't configured the CLI before, you have to configure your credentials by running `sudo aws configure`. - Because a non-admin user likely can't access the Container Registry folder, + Because a non-administrator user likely can't access the Container Registry folder, ensure you use `sudo`. To check your credential configuration, run [`ls`](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/ls.html) to list all buckets. @@ -483,14 +467,14 @@ you can pull from the Container Registry, but you cannot push. sudo aws --endpoint-url https://your-object-storage-backend.com s3 sync registry s3://mybucket --delete --dryrun ``` - After verifying the command is going to perform as expected, remove the + After verifying the command performs as expected, remove the [`--dryrun`](https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html) flag and run the command. DANGER: **Danger:** The [`--delete`](https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html) - flag will delete files that exist in the destination but not in the source. - Make sure not to swap the source and destination, or you will delete all data in the Registry. + flag deletes files that exist in the destination but not in the source. + If you swap the source and destination, all data in the Registry is deleted. 1. Verify all Container Registry files have been uploaded to object storage by looking at the file count returned by these two commands: @@ -560,15 +544,11 @@ However, this behavior is undesirable for registries used by internal hosts that ### Storage limitations Currently, there is no storage limitation, which means a user can upload an -infinite amount of Docker images with arbitrary sizes. This setting will be +infinite amount of Docker images with arbitrary sizes. This setting should be configurable in future releases. ## Change the registry's internal port -NOTE: **Note:** -This is not to be confused with the port that GitLab itself uses to expose -the Registry to the world. - The Registry server listens on localhost at port `5000` by default, which is the address for which the Registry server should accept connections. In the examples below we set the Registry's port to `5001`. @@ -589,7 +569,7 @@ In the examples below we set the Registry's port to `5001`. [`http:addr`](https://docs.docker.com/registry/configuration/#http) value: ```yaml - http + http: addr: localhost:5001 ``` @@ -603,9 +583,8 @@ on how to achieve that. ## Use an external container registry with GitLab as an auth endpoint -NOTE: **Note:** -In using an external container registry, some features associated with the -container registry may be unavailable or have [inherent risks](./../../user/packages/container_registry/index.md#use-with-external-container-registries) +If you use an external container registry, some features associated with the +container registry may be unavailable or have [inherent risks](./../../user/packages/container_registry/index.md#use-with-external-container-registries). **Omnibus GitLab** @@ -619,13 +598,12 @@ You can use GitLab as an auth endpoint with an external container registry. gitlab_rails['registry_issuer'] = "omnibus-gitlab-issuer" ``` - NOTE: **Note:** `gitlab_rails['registry_enabled'] = true` is needed to enable GitLab's Container Registry features and authentication endpoint. GitLab's bundled - Container Registry service will not be started even with this enabled. + Container Registry service does not start, even with this enabled. 1. A certificate-key pair is required for GitLab and the external container - registry to communicate securely. You will need to create a certificate-key + registry to communicate securely. You need to create a certificate-key pair, configuring the external container registry with the public certificate and configuring GitLab with the private key. To do that, add the following to `/etc/gitlab/gitlab.rb`: @@ -641,11 +619,10 @@ You can use GitLab as an auth endpoint with an external container registry. gitlab_rails['registry_key_path'] = "/custom/path/to/registry-key.key" ``` - NOTE: **Note:** - The file specified at `registry_key_path` gets populated with the - content specified by `internal_key`, each time reconfigure is executed. If - no file is specified, Omnibus GitLab will default it to - `/var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key` and will populate + Each time reconfigure is executed, the file specified at `registry_key_path` + gets populated with the content specified by `internal_key`. If + no file is specified, Omnibus GitLab defaults it to + `/var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key` and populates it. 1. To change the container registry URL displayed in the GitLab Container @@ -686,8 +663,7 @@ response to events happening within the registry. Read more about the Container Registry notifications configuration options in the [Docker Registry notifications documentation](https://docs.docker.com/registry/notifications/). -NOTE: **Note:** -Multiple endpoints can be configured for the Container Registry. +You can configure multiple endpoints for the Container Registry. **Omnibus GitLab installations** @@ -761,28 +737,16 @@ project.container_repositories.find_each do |repo| end ``` -NOTE: **Note:** You can also [run cleanup on a schedule](../../user/packages/container_registry/index.md#cleanup-policy). ## Container Registry garbage collection -NOTE: **Note:** -The garbage collection tools are only available when you've installed GitLab -via an Omnibus package or the [cloud native chart](https://docs.gitlab.com/charts/charts/registry/#garbage-collection). - -DANGER: **Danger:** -By running the built-in garbage collection command, it will cause downtime to -the Container Registry. If you run this command on an instance in an environment -where one of your other instances is still writing to the Registry storage, -referenced manifests will be removed. To avoid that, make sure Registry is set to -[read-only mode](#performing-garbage-collection-without-downtime) before proceeding. - Container Registry can use considerable amounts of disk space. To clear up some unused layers, the registry includes a garbage collect command. GitLab offers a set of APIs to manipulate the Container Registry and aid the process of removing unused tags. Currently, this is exposed using the API, but in the future, -these controls will be migrated to the GitLab interface. +these controls should migrate to the GitLab interface. Project maintainers can [delete Container Registry tags in bulk](../../api/container_registry.md#delete-registry-repository-tags-in-bulk) @@ -791,6 +755,15 @@ it only unlinks tags from manifests and image blobs. To recycle the Container Registry data in the whole GitLab instance, you can use the built-in command provided by `gitlab-ctl`. +Prerequisites: + +- You must have installed GitLab by using an Omnibus package or the + [cloud native chart](https://docs.gitlab.com/charts/charts/registry/#garbage-collection). +- You must set the Registry to [read-only mode](#performing-garbage-collection-without-downtime). + Running garbage collection causes downtime for the Container Registry. When you run this command + on an instance in an environment where another instances is still writing to the Registry storage, + referenced manifests are removed. + ### Understanding the content-addressable layers Consider the following example, where you first build the image: @@ -818,15 +791,14 @@ no longer directly accessible via the `:latest` tag. > [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/987) in Omnibus GitLab 8.12. -There are a couple of considerations you need to note before running the -built-in command: +Before you run the built-in command, note the following: -- The built-in command will stop the registry before it starts the garbage collection. +- The built-in command stops the registry before it starts the garbage collection. - The garbage collect command takes some time to complete, depending on the amount of data that exists. -- If you changed the location of registry configuration file, you will need to +- If you changed the location of registry configuration file, you must specify its path. -- After the garbage collection is done, the registry should start up automatically. +- After the garbage collection is done, the registry should start automatically. If you did not change the default location of the configuration file, run: @@ -882,7 +854,6 @@ During this time, you will be able to pull from the Container Registry, but you will not be able to push. -NOTE: **Note:** By default, the [registry storage path](#configure-storage-for-the-container-registry) is `/var/opt/gitlab/gitlab-rails/shared/registry`. @@ -1065,7 +1036,7 @@ Start with a value between `25000000` (25MB) and `50000000` (50MB). s3: accesskey: 'AKIAKIAKI' secretkey: 'secret123' - bucket: 'gitlab-registry-bucket-AKIAKIAKI' + bucket: 'gitlab-registry-bucket-AKIAKIAKI' chunksize: 25000000 ``` @@ -1166,12 +1137,7 @@ curl localhost:5001/debug/vars ### Advanced Troubleshooting -NOTE: **Note:** -The following section is only recommended for experts. - -Sometimes it's not obvious what is wrong, and you may need to dive deeper into -the communication between the Docker client and the Registry to find out -what's wrong. We will use a concrete example in the past to illustrate how to +We will use a concrete example in the past to illustrate how to diagnose a problem with the S3 setup. #### Unexpected 403 error during push diff --git a/doc/administration/packages/dependency_proxy.md b/doc/administration/packages/dependency_proxy.md index 2f9cfecc9d4..fba3d51f741 100644 --- a/doc/administration/packages/dependency_proxy.md +++ b/doc/administration/packages/dependency_proxy.md @@ -11,7 +11,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w GitLab can be utilized as a dependency proxy for a variety of common package managers. This is the administration documentation. If you want to learn how to use the -dependency proxies, see the [user guide](../../user/group/dependency_proxy/index.md). +dependency proxies, see the [user guide](../../user/packages/dependency_proxy/index.md). ## Enabling the Dependency Proxy feature @@ -135,28 +135,28 @@ This section describes the earlier configuration format. ## ## The location where build dependency_proxy are stored (default: shared/dependency_proxy). ## - #storage_path: shared/dependency_proxy + # storage_path: shared/dependency_proxy object_store: enabled: false - remote_directory: dependency_proxy # The bucket name. - #direct_upload: false # Use Object Storage directly for uploads instead of background uploads if enabled (Default: false). - #background_upload: true # Temporary option to limit automatic upload (Default: true). - #proxy_download: false # Passthrough all downloads via GitLab instead of using Redirects to Object Storage. + remote_directory: dependency_proxy # The bucket name. + # direct_upload: false # Use Object Storage directly for uploads instead of background uploads if enabled (Default: false). + # background_upload: true # Temporary option to limit automatic upload (Default: true). + # proxy_download: false # Passthrough all downloads via GitLab instead of using Redirects to Object Storage. connection: + ## + ## If the provider is AWS S3, use the following + ## + provider: AWS + region: us-east-1 + aws_access_key_id: AWS_ACCESS_KEY_ID + aws_secret_access_key: AWS_SECRET_ACCESS_KEY ## - ## If the provider is AWS S3, uncomment the following + ## If the provider is other than AWS (an S3-compatible one), comment out the previous 4 lines and use the following instead: ## - #provider: AWS - #region: us-east-1 - #aws_access_key_id: AWS_ACCESS_KEY_ID - #aws_secret_access_key: AWS_SECRET_ACCESS_KEY - ## - ## If the provider is other than AWS (an S3-compatible one), uncomment the following - ## - #host: 's3.amazonaws.com' # default: s3.amazonaws.com. - #aws_signature_version: 4 # For creation of signed URLs. Set to 2 if provider does not support v4. - #endpoint: 'https://s3.amazonaws.com' # Useful for S3-compliant services such as DigitalOcean Spaces. - #path_style: false # If true, use 'host/bucket_name/object' instead of 'bucket_name.host/object'. + # host: 's3.amazonaws.com' # default: s3.amazonaws.com. + # aws_signature_version: 4 # For creation of signed URLs. Set to 2 if provider does not support v4. + # endpoint: 'https://s3.amazonaws.com' # Useful for S3-compliant services such as DigitalOcean Spaces. + # path_style: false # If true, use 'host/bucket_name/object' instead of 'bucket_name.host/object'. ``` 1. [Restart GitLab](../restart_gitlab.md#installations-from-source "How to restart GitLab") for the changes to take effect. diff --git a/doc/administration/packages/index.md b/doc/administration/packages/index.md index 1061f3c33db..0b3a7ae9fa5 100644 --- a/doc/administration/packages/index.md +++ b/doc/administration/packages/index.md @@ -14,13 +14,14 @@ The Packages feature allows GitLab to act as a repository for the following: | Software repository | Description | Available in GitLab version | | ------------------- | ----------- | --------------------------- | -| [PyPi Repository](../../user/packages/pypi_repository/index.md) | The GitLab PyPi Repository enables every project in GitLab to have its own space to store [PyPi](https://pypi.org/) packages. | 12.10+ | +| [PyPI Repository](../../user/packages/pypi_repository/index.md) | The GitLab PyPI Repository enables every project in GitLab to have its own space to store [PyPI](https://pypi.org/) packages. | 12.10+ | | [Composer Repository](../../user/packages/composer_repository/index.md) | The GitLab Composer Repository enables every project in GitLab to have its own space to store [Composer](https://getcomposer.org/) packages. | 13.1+ | | [NuGet Repository](../../user/packages/nuget_repository/index.md) | The GitLab NuGet Repository enables every project in GitLab to have its own space to store [NuGet](https://www.nuget.org/) packages. | 12.8+ | | [Conan Repository](../../user/packages/conan_repository/index.md) | The GitLab Conan Repository enables every project in GitLab to have its own space to store [Conan](https://conan.io/) packages. | 12.4+ | | [Maven Repository](../../user/packages/maven_repository/index.md) | The GitLab Maven Repository enables every project in GitLab to have its own space to store [Maven](https://maven.apache.org/) packages. | 11.3+ | | [NPM Registry](../../user/packages/npm_registry/index.md) | The GitLab NPM Registry enables every project in GitLab to have its own space to store [NPM](https://www.npmjs.com/) packages. | 11.7+ | | [Go Proxy](../../user/packages/go_proxy/index.md) | The Go proxy for GitLab enables every project in GitLab to be fetched with the [Go proxy protocol](https://proxy.golang.org/). | 13.1+ | +| [Generic packages](../../user/packages/generic_packages/index.md) | Store arbitrary files, like release binaries. | 13.5+ | Don't you see your package management system supported yet? Please consider contributing @@ -142,33 +143,33 @@ We recommend using the [consolidated object storage settings](../object_storage. 1. Edit the `packages` section in `config/gitlab.yml` (uncomment where necessary): ```yaml - packages: - enabled: true + packages: + enabled: true + ## + ## The location where build packages are stored (default: shared/packages). + ## + # storage_path: shared/packages + object_store: + enabled: false + remote_directory: packages # The bucket name. + # direct_upload: false # Use Object Storage directly for uploads instead of background uploads if enabled (Default: false). + # background_upload: true # Temporary option to limit automatic upload (Default: true). + # proxy_download: false # Passthrough all downloads via GitLab instead of using Redirects to Object Storage. + connection: ## - ## The location where build packages are stored (default: shared/packages). + ## If the provider is AWS S3, use the following: ## - #storage_path: shared/packages - object_store: - enabled: false - remote_directory: packages # The bucket name. - #direct_upload: false # Use Object Storage directly for uploads instead of background uploads if enabled (Default: false). - #background_upload: true # Temporary option to limit automatic upload (Default: true). - #proxy_download: false # Passthrough all downloads via GitLab instead of using Redirects to Object Storage. - connection: - ## - ## If the provider is AWS S3, uncomment the following - ## - #provider: AWS - #region: us-east-1 - #aws_access_key_id: AWS_ACCESS_KEY_ID - #aws_secret_access_key: AWS_SECRET_ACCESS_KEY - ## - ## If the provider is other than AWS (an S3-compatible one), uncomment the following - ## - #host: 's3.amazonaws.com' # default: s3.amazonaws.com. - #aws_signature_version: 4 # For creation of signed URLs. Set to 2 if provider does not support v4. - #endpoint: 'https://s3.amazonaws.com' # Useful for S3-compliant services such as DigitalOcean Spaces. - #path_style: false # If true, use 'host/bucket_name/object' instead of 'bucket_name.host/object'. + provider: AWS + region: us-east-1 + aws_access_key_id: AWS_ACCESS_KEY_ID + aws_secret_access_key: AWS_SECRET_ACCESS_KEY + ## + ## If the provider is other than AWS (an S3-compatible one), comment out the previous 4 lines and use the following instead: + ## + # host: 's3.amazonaws.com' # default: s3.amazonaws.com. + # aws_signature_version: 4 # For creation of signed URLs. Set to 2 if provider does not support v4. + # endpoint: 'https://s3.amazonaws.com' # Useful for S3-compliant services such as DigitalOcean Spaces. + # path_style: false # If true, use 'host/bucket_name/object' instead of 'bucket_name.host/object'. ``` 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index 3c0030be629..9f72293a730 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -53,8 +53,14 @@ supporting custom domains a secondary IP is not needed. Before proceeding with the Pages configuration, you will need to: -1. Have an exclusive root domain for serving GitLab Pages. Note that you cannot - use a subdomain of your GitLab's instance domain. +1. Have a domain for Pages that is not a subdomain of your GitLab's instance domain. + + | GitLab domain | Pages domain | Does it work? | + | :---: | :---: | :---: | + | `example.com` | `example.io` | **{check-circle}** Yes | + | `example.com` | `pages.example.com` | **{dotted-circle}** No | + | `gitlab.example.com` | `pages.example.com` | **{check-circle}** Yes | + 1. Configure a **wildcard DNS record**. 1. (Optional) Have a **wildcard certificate** for that domain if you decide to serve Pages under HTTPS. @@ -235,7 +241,7 @@ control over how the Pages daemon runs and serves content in your environment. | `pages_path` | The directory on disk where pages are stored, defaults to `GITLAB-RAILS/shared/pages`. | `pages_nginx[]` | | | `enable` | Include a virtual host `server{}` block for Pages inside NGINX. Needed for NGINX to proxy traffic back to the Pages daemon. Set to `false` if the Pages daemon should directly receive all requests, for example, when using [custom domains](index.md#custom-domains). -| `FF_ENABLE_REDIRECTS` | Feature flag to enable redirects. See the [redirects documentation](../../user/project/pages/redirects.md#enable-or-disable-redirects) for more info. | +| `FF_ENABLE_REDIRECTS` | Feature flag to disable redirects (enabled by default). Read the [redirects documentation](../../user/project/pages/redirects.md#disable-redirects) for more info. | --- @@ -370,8 +376,8 @@ Pages access control is disabled by default. To enable it: 1. Users can now configure it in their [projects' settings](../../user/project/pages/pages_access_control.md). NOTE: **Important:** -For multi-node setups, in order for this setting to be effective, it has to be applied -to all the App nodes as well as the Sidekiq nodes. +For this setting to be effective with multi-node setups, it has to be applied to +all the App nodes and Sidekiq nodes. #### Disabling public access to all Pages websites @@ -397,8 +403,7 @@ redeployed. This issue will be resolved by ### Running behind a proxy Like the rest of GitLab, Pages can be used in those environments where external -internet connectivity is gated by a proxy. In order to use a proxy for GitLab -pages: +internet connectivity is gated by a proxy. To use a proxy for GitLab Pages: 1. Configure in `/etc/gitlab/gitlab.rb`: @@ -425,10 +430,6 @@ Authority (CA) in the system certificate store. For Omnibus, this is fixed by [installing a custom CA in Omnibus GitLab](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). -## Enable redirects - -In GitLab Pages, you can [enable the redirects feature](../../user/project/pages/redirects.md#enable-or-disable-redirects) to configure rules to forward one URL to another using HTTP redirects. - ## Activate verbose logging for daemon Verbose logging was [introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/2533) in @@ -508,7 +509,8 @@ To override the global maximum pages size for a specific group: ## Running GitLab Pages on a separate server -You can run the GitLab Pages daemon on a separate server in order to decrease the load on your main application server. +You can run the GitLab Pages daemon on a separate server to decrease the load on +your main application server. To configure GitLab Pages on a separate server: @@ -595,7 +597,7 @@ database encryption. Proceed with caution. ```ruby pages_external_url "http://<pages_server_URL>" gitlab_pages['enable'] = false - gitlab_rails['pages_enabled']=false + pages_nginx['enable'] = false gitlab_rails['pages_path'] = "/mnt/pages" ``` @@ -779,3 +781,16 @@ For example, if there is a connection timeout: ```plaintext error="failed to connect to internal Pages API: Get \"https://gitlab.example.com:3000/api/v4/internal/pages/status\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" ``` + +### 500 error with `securecookie: failed to generate random iv` and `Failed to save the session` + +This problem most likely results from an [out-dated operating system](https://docs.gitlab.com/omnibus/package-information/deprecated_os.html). +The [Pages daemon uses the `securecookie` library](https://gitlab.com/search?group_id=9970&project_id=734943&repository_ref=master&scope=blobs&search=securecookie&snippets=false) to get random strings via [crypto/rand in Go](https://golang.org/pkg/crypto/rand/#pkg-variables). +This requires the `getrandom` syscall or `/dev/urandom` to be available on the host OS. +Upgrading to an [officially supported operating system](https://about.gitlab.com/install/) is recommended. + +### The requested scope is invalid, malformed, or unknown + +This problem comes from the permissions of the GitLab Pages OAuth application. To fix it, go to +**Admin > Applications > GitLab Pages** and edit the application. Under **Scopes**, ensure that the +`api` scope is selected and save your changes. diff --git a/doc/administration/pages/source.md b/doc/administration/pages/source.md index 662817e7411..87217b141a4 100644 --- a/doc/administration/pages/source.md +++ b/doc/administration/pages/source.md @@ -347,10 +347,6 @@ world. Custom domains and TLS are supported. 1. Restart NGINX 1. [Restart GitLab](../restart_gitlab.md#installations-from-source) -## Enable redirects - -In GitLab Pages, you can [enable the redirects feature](../../user/project/pages/redirects.md#enable-or-disable-redirects) to configure rules to forward one URL to another using HTTP redirects. - ## NGINX caveats NOTE: **Note:** diff --git a/doc/administration/postgresql/external.md b/doc/administration/postgresql/external.md index 632b68fb014..4a164c66578 100644 --- a/doc/administration/postgresql/external.md +++ b/doc/administration/postgresql/external.md @@ -1,3 +1,9 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Configure GitLab using an external PostgreSQL service If you're hosting GitLab on a cloud provider, you can optionally use a diff --git a/doc/administration/postgresql/index.md b/doc/administration/postgresql/index.md index 2720d8e696b..c7ec46db654 100644 --- a/doc/administration/postgresql/index.md +++ b/doc/administration/postgresql/index.md @@ -1,11 +1,14 @@ --- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- # Configuring PostgreSQL for scaling In this section, you'll be guided through configuring a PostgreSQL database to -be used with GitLab in one of our [Scalable and Highly Available Setups](../reference_architectures/index.md). +be used with GitLab in one of our [reference architectures](../reference_architectures/index.md). There are essentially three setups to choose from. ## PostgreSQL replication and failover with Omnibus GitLab **(PREMIUM ONLY)** diff --git a/doc/administration/postgresql/pgbouncer.md b/doc/administration/postgresql/pgbouncer.md index 9db3e017359..7760b197267 100644 --- a/doc/administration/postgresql/pgbouncer.md +++ b/doc/administration/postgresql/pgbouncer.md @@ -1,4 +1,7 @@ --- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- @@ -145,6 +148,35 @@ ote_pid | tls (1 row) ``` +## Procedure for bypassing PgBouncer + +Some database changes have to be done directly, and not through PgBouncer. This includes database restores and GitLab upgrades (because of the database migrations). + +1. To find the primary node, run the following on a database node: + + ```shell + sudo gitlab-ctl repmgr cluster show + ``` + +1. Edit `/etc/gitlab/gitlab.rb` on the application node you're performing the task on, and update + `gitlab_rails['db_host']` and `gitlab_rails['db_port']` with the database + primary's host and port. + +1. Run reconfigure: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +Once you've performed the tasks or procedure, switch back to using PgBouncer: + +1. Change back `/etc/gitlab/gitlab.rb` to point to PgBouncer. +1. Run reconfigure: + + ```shell + sudo gitlab-ctl reconfigure + ``` + ## Troubleshooting In case you are experiencing any issues connecting through PgBouncer, the first diff --git a/doc/administration/postgresql/replication_and_failover.md b/doc/administration/postgresql/replication_and_failover.md index bc2af167e6c..0a40b61fd3c 100644 --- a/doc/administration/postgresql/replication_and_failover.md +++ b/doc/administration/postgresql/replication_and_failover.md @@ -1,10 +1,16 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # PostgreSQL replication and failover with Omnibus GitLab **(PREMIUM ONLY)** -This document will focus only on configuration supported with [GitLab Premium](https://about.gitlab.com/pricing/), using the Omnibus GitLab package. -If you are a Community Edition or Starter user, consider using a cloud hosted solution. -This document will not cover installations from source. +This document focuses on configuration supported with [GitLab Premium](https://about.gitlab.com/pricing/), using the Omnibus GitLab package. +If you're a Community Edition or Starter user, consider using a cloud hosted solution. +This document doesn't cover installations from source. -If a setup with replication and failover is not what you were looking for, see +If a setup with replication and failover isn't what you were looking for, see the [database configuration document](https://docs.gitlab.com/omnibus/settings/database.html) for the Omnibus GitLab packages. @@ -87,9 +93,9 @@ information. #### Network information -PostgreSQL does not listen on any network interface by default. It needs to know -which IP address to listen on in order to be accessible to other services. -Similarly, PostgreSQL access is controlled based on the network source. +PostgreSQL doesn't listen on any network interface by default. It needs to know +which IP address to listen on to be accessible to other services. Similarly, +PostgreSQL access is controlled based on the network source. This is why you will need: @@ -419,7 +425,7 @@ Check the [Troubleshooting section](#troubleshooting) before proceeding. 1. Make sure you collect [`CONSUL_SERVER_NODES`](#consul-information), [`CONSUL_PASSWORD_HASH`](#consul-information), and [`PGBOUNCER_PASSWORD_HASH`](#pgbouncer-information) before executing the next step. -1. One each node, edit the `/etc/gitlab/gitlab.rb` config file and replace values noted in the `# START user configuration` section as below: +1. One each node, edit the `/etc/gitlab/gitlab.rb` configuration file and replace values noted in the `# START user configuration` section as below: ```ruby # Disable all components except PgBouncer and Consul agent @@ -1263,6 +1269,11 @@ with: sudo gitlab-ctl stop patroni ``` +Note that stopping or restarting Patroni service on the leader node will trigger the automatic failover. If you +want to signal Patroni to reload its configuration or restart PostgreSQL process without triggering the failover, you +must use the `reload` or `restart` sub-commands of `gitlab-ctl patroni` instead. These two sub-commands are wrappers of +the same `patronictl` commands. + ### Manual failover procedure for Patroni While Patroni supports automatic failover, you also have the ability to perform @@ -1348,3 +1359,93 @@ You can switch an exiting database cluster to use Patroni instead of repmgr with 1. Repeat the last two steps for all replica nodes. `gitlab.rb` should look the same on all nodes. 1. Optional: You can remove `gitlab_repmgr` database and role on the primary. + +### Upgrading PostgreSQL major version in a Patroni cluster + +As of GitLab 13.3, PostgreSQL 11.7 and 12.3 are both shipped with Omnibus GitLab. GitLab still +uses PostgreSQL 11 by default. Therefore `gitlab-ctl pg-upgrade` does not automatically upgrade +to PostgreSQL 12. If you want to upgrade to PostgreSQL 12, you must ask for it explicitly. + +CAUTION: **Warning:** +The procedure for upgrading PostgreSQL in a Patroni cluster is different than when upgrading using repmgr. +The following outlines the key differences and important considerations that need to be accounted for when +upgrading PostgreSQL. + +Here are a few key facts that you must consider before upgrading PostgreSQL: + +- The main point is that you will have to **shut down the Patroni cluster**. This means that your + GitLab deployment will be down for the duration of database upgrade or, at least, as long as your leader + node is upgraded. This can be **a significant downtime depending on the size of your database**. + +- Upgrading PostgreSQL creates a new data directory with a new control data. From Patroni's perspective + this is a new cluster that needs to be bootstrapped again. Therefore, as part of the upgrade procedure, + the cluster state, which is stored in Consul, will be wiped out. Once the upgrade is completed, Patroni + will be instructed to bootstrap a new cluster. **Note that this will change your _cluster ID_**. + +- The procedures for upgrading leader and replicas are not the same. That is why it is important to use the + right procedure on each node. + +- Upgrading a replica node **deletes the data directory and resynchronizes it** from the leader using the + configured replication method (currently `pg_basebackup` is the only available option). It might take some + time for replica to catch up with the leader, depending on the size of your database. + +- An overview of the upgrade procedure is outlined in [Patoni's documentation](https://patroni.readthedocs.io/en/latest/existing_data.html#major-upgrade-of-postgresql-version). + You can still use `gitlab-ctl pg-upgrade` which implements this procedure with a few adjustments. + +Considering these, you should carefully plan your PostgreSQL upgrade: + +1. Find out which node is the leader and which node is a replica: + + ```shell + gitlab-ctl patroni members + ``` + + NOTE: **Note:** + `gitlab-ctl pg-upgrade` tries to detect the role of the node. If for any reason the auto-detection + does not work or you believe it did not detect the role correctly, you can use the `--leader` or `--replica` + arguments to manually override it. + +1. Stop Patroni **only on replicas**. + + ```shell + sudo gitlab-ctl stop patroni + ``` + +1. Enable the maintenance mode on the **application node**: + + ```shell + sudo gitlab-ctl deploy-page up + ``` + +1. Upgrade PostgreSQL on **the leader node** and make sure that the upgrade is completed successfully: + + ```shell + sudo gitlab-ctl pg-upgrade -V 12 + ``` + +1. Check the status of the leader and cluster. You can only proceed if you have a healthy leader: + + ```shell + gitlab-ctl patroni check-leader + + # OR + + gitlab-ctl patroni members + ``` + +1. You can now disable the maintenance mode on the **application node**: + + ```shell + sudo gitlab-ctl deploy-page down + ``` + +1. Upgrade PostgreSQL **on replicas** (you can do this in parallel on all of them): + + ```shell + sudo gitlab-ctl pg-upgrade -V 12 + ``` + +NOTE: **Note:** +Reverting PostgreSQL upgrade with `gitlab-ctl revert-pg-upgrade` has the same considerations as +`gitlab-ctl pg-upgrade`. You should follow the same procedure by first stopping the replicas, +then reverting the leader, and finally reverting the replicas. diff --git a/doc/administration/postgresql/standalone.md b/doc/administration/postgresql/standalone.md index 2747749066e..2ac74e8a4a0 100644 --- a/doc/administration/postgresql/standalone.md +++ b/doc/administration/postgresql/standalone.md @@ -1,15 +1,21 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Standalone PostgreSQL using Omnibus GitLab **(CORE ONLY)** If you wish to have your database service hosted separately from your GitLab -application server(s), you can do this using the PostgreSQL binaries packaged +application servers, you can do this using the PostgreSQL binaries packaged together with Omnibus GitLab. This is recommended as part of our [reference architecture for up to 2,000 users](../reference_architectures/2k_users.md). ## Setting it up -1. SSH into the PostgreSQL server. -1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab - package you want using **steps 1 and 2** from the GitLab downloads page. +1. SSH in to the PostgreSQL server. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package you want using *steps 1 and 2* from the GitLab downloads page. - Do not complete any other steps on the download page. 1. Generate a password hash for PostgreSQL. This assumes you will use the default username of `gitlab` (recommended). The command will request a password diff --git a/doc/administration/raketasks/check.md b/doc/administration/raketasks/check.md index 15014fffd01..bdccd6d5fe9 100644 --- a/doc/administration/raketasks/check.md +++ b/doc/administration/raketasks/check.md @@ -148,9 +148,38 @@ above. for the affected project(s). If the issue persists, try triggering `gc` via the -[Rails Console](../troubleshooting/navigating_gitlab_via_rails_console.md#starting-a-rails-console-session): +[Rails Console](../operations/rails_console.md#starting-a-rails-console-session): ```ruby p = Project.find_by_path("project-name") Projects::HousekeepingService.new(p, :gc).execute ``` + +### Delete references to missing remote uploads + +`gitlab-rake gitlab:uploads:check VERBOSE=1` detects remote objects that do not exist because they were +deleted externally but their references still exist in the GitLab database. + +Example output with error message: + +```shell +$ sudo gitlab-rake gitlab:uploads:check VERBOSE=1 +Checking integrity of Uploads +- 100..434: Failures: 2 +- Upload: 100: Remote object does not exist +- Upload: 101: Remote object does not exist +Done! +``` + +To delete these references to remote uploads that were deleted externally, open the [GitLab Rails Console](../operations/rails_console.md#starting-a-rails-console-session) and run: + +```ruby +uploads_deleted=0 +Upload.find_each do |upload| + next if upload.retrieve_uploader.file.exists? + uploads_deleted=uploads_deleted + 1 + p upload ### allow verification before destroy + # p upload.destroy! ### uncomment to actually destroy +end +p "#{uploads_deleted} remote objects were destroyed." +``` diff --git a/doc/administration/raketasks/doctor.md b/doc/administration/raketasks/doctor.md index 62d0af70706..c97aa5a4de1 100644 --- a/doc/administration/raketasks/doctor.md +++ b/doc/administration/raketasks/doctor.md @@ -47,9 +47,8 @@ I, [2020-06-11T17:18:15.575711 #27148] INFO -- : Done! ### Verbose mode -In order to get more detailed information about which -rows and columns cannot be decrypted, you can pass a VERBOSE -environment variable: +To get more detailed information about which rows and columns can't be +decrypted, you can pass a `VERBOSE` environment variable: **Omnibus Installation** diff --git a/doc/administration/raketasks/github_import.md b/doc/administration/raketasks/github_import.md index a46a2b34687..7f673a2c850 100644 --- a/doc/administration/raketasks/github_import.md +++ b/doc/administration/raketasks/github_import.md @@ -2,9 +2,8 @@ > [Introduced]( https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10308) in GitLab 9.1. -In order to retrieve and import GitHub repositories, you will need a -[GitHub personal access token](https://github.com/settings/tokens). -A username should be passed as the second argument to the Rake task +To retrieve and import GitHub repositories, you need a [GitHub personal access token](https://github.com/settings/tokens). +A username should be passed as the second argument to the Rake task, which will become the owner of the project. You can resume an import with the same command. diff --git a/doc/administration/raketasks/uploads/sanitize.md b/doc/administration/raketasks/uploads/sanitize.md index 9586ab2c6f4..54aa62059cf 100644 --- a/doc/administration/raketasks/uploads/sanitize.md +++ b/doc/administration/raketasks/uploads/sanitize.md @@ -1,6 +1,6 @@ # Uploads sanitize Rake tasks **(CORE ONLY)** -Since GitLab 11.9, EXIF data is automatically stripped from JPG or TIFF image uploads. +In GitLab 11.9 and later, EXIF data is automatically stripped from JPG or TIFF image uploads. EXIF data may contain sensitive information (for example, GPS location), so you can remove EXIF data from existing images that were uploaded to an earlier version of GitLab. diff --git a/doc/administration/read_only_gitlab.md b/doc/administration/read_only_gitlab.md new file mode 100644 index 00000000000..681102a8c39 --- /dev/null +++ b/doc/administration/read_only_gitlab.md @@ -0,0 +1,125 @@ +--- +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + +# Place GitLab into a read-only state **(CORE ONLY)** + +CAUTION: **Warning:** +This document should be used as a temporary solution. +There's work in progress to make this +[possible with Geo](https://gitlab.com/groups/gitlab-org/-/epics/2149). + +In some cases, you might want to place GitLab under a read-only state. +The configuration for doing so depends on your desired outcome. + +## Make the repositories read-only + +The first thing you'll want to accomplish is to ensure that no changes can be +made to your repositories. There's two ways you can accomplish that: + +- Either stop Unicorn/Puma to make the internal API unreachable: + + ```shell + sudo gitlab-ctl stop puma # or unicorn + ``` + +- Or, open up a Rails console: + + ```shell + sudo gitlab-rails console + ``` + + And set the repositories for all projects read-only: + + ```ruby + Project.all.find_each { |project| project.update!(repository_read_only: true) } + ``` + + When you're ready to revert this, you can do so with the following command: + + ```ruby + Project.all.find_each { |project| project.update!(repository_read_only: false) } + ``` + +## Shut down the GitLab UI + +If you don't mind shutting down the GitLab UI, then the easiest approach is to +stop `sidekiq` and `puma`/`unicorn`, and you'll effectively ensure that no +changes can be made to GitLab: + +```shell +sudo gitlab-ctl stop sidekiq +sudo gitlab-ctl stop puma # or unicorn +``` + +When you're ready to revert this: + +```shell +sudo gitlab-ctl start sidekiq +sudo gitlab-ctl start puma # or unicorn +``` + +## Make the database read-only + +If you want to allow users to use the GitLab UI, then you'll need to ensure that +the database is read-only: + +1. Take a [GitLab backup](../raketasks/backup_restore.md#back-up-gitlab) + in case things don't go as expected. +1. Enter PostgreSQL on the console as an admin user: + + ```shell + sudo \ + -u gitlab-psql /opt/gitlab/embedded/bin/psql \ + -h /var/opt/gitlab/postgresql gitlabhq_production + ``` + +1. Create the `gitlab_read_only` user. Note that the password is set to `mypassword`, + change that to your liking: + + ```sql + -- NOTE: Use the password defined earlier + CREATE USER gitlab_read_only WITH password 'mypassword'; + GRANT CONNECT ON DATABASE gitlabhq_production to gitlab_read_only; + GRANT USAGE ON SCHEMA public TO gitlab_read_only; + GRANT SELECT ON ALL TABLES IN SCHEMA public TO gitlab_read_only; + GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO gitlab_read_only; + + -- Tables created by "gitlab" should be made read-only for "gitlab_read_only" + -- automatically. + ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON TABLES TO gitlab_read_only; + ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON SEQUENCES TO gitlab_read_only; + ``` + +1. Get the hashed password of the `gitlab_read_only` user and copy the result: + + ```shell + sudo gitlab-ctl pg-password-md5 gitlab_read_only + ``` + +1. Edit `/etc/gitlab/gitlab.rb` and add the password from the previous step: + + ```ruby + postgresql['sql_user_password'] = 'a2e20f823772650f039284619ab6f239' + postgresql['sql_user'] = "gitlab_read_only" + ``` + +1. Reconfigure GitLab and restart PostgreSQL: + + ```shell + sudo gitlab-ctl reconfigure + sudo gitlab-ctl restart postgresql + ``` + +When you're ready to revert the read-only state, you'll need to remove the added +lines in `/etc/gitlab/gitlab.rb`, and reconfigure GitLab and restart PostgreSQL: + +```shell +sudo gitlab-ctl reconfigure +sudo gitlab-ctl restart postgresql +``` + +Once you verify all works as expected, you can remove the `gitlab_read_only` +user from the database. diff --git a/doc/administration/redis/replication_and_failover.md b/doc/administration/redis/replication_and_failover.md index ca041adb1d8..72a52ba0a1f 100644 --- a/doc/administration/redis/replication_and_failover.md +++ b/doc/administration/redis/replication_and_failover.md @@ -124,9 +124,9 @@ each other over the network. ### Sentinel setup overview Sentinels watch both other Sentinels and Redis nodes. Whenever a Sentinel -detects that a Redis node is not responding, it will announce that to the -other Sentinels. They have to reach the **quorum**, that is the minimum amount -of Sentinels that agrees a node is down, in order to be able to start a failover. +detects that a Redis node isn't responding, it announces the node's status to +the other Sentinels. The Sentinels have to reach a _quorum_ (the minimum amount +of Sentinels agreeing a node is down) to be able to start a failover. Whenever the **quorum** is met, the **majority** of all known Sentinel nodes need to be available and reachable, so that they can elect the Sentinel **leader** diff --git a/doc/administration/redis/replication_and_failover_external.md b/doc/administration/redis/replication_and_failover_external.md index ce452d30fc2..7561f1c0fbf 100644 --- a/doc/administration/redis/replication_and_failover_external.md +++ b/doc/administration/redis/replication_and_failover_external.md @@ -71,7 +71,7 @@ requirements: - All Redis servers in this guide must be configured to use a TCP connection instead of a socket. To configure Redis to use TCP connections you need to - define both `bind` and `port` in the Redis config file. You can bind to all + define both `bind` and `port` in the Redis configuration file. You can bind to all interfaces (`0.0.0.0`) or specify the IP of the desired interface (e.g., one from an internal network). - Since Redis 3.2, you must define a password to receive external connections @@ -228,13 +228,13 @@ which ideally should not have Redis or Sentinels in the same machine: sentinels: - host: 10.0.0.1 - port: 26379 # point to sentinel, not to redis port + port: 26379 # point to sentinel, not to redis port - host: 10.0.0.2 - port: 26379 # point to sentinel, not to redis port + port: 26379 # point to sentinel, not to redis port - host: 10.0.0.3 - port: 26379 # point to sentinel, not to redis port + port: 26379 # point to sentinel, not to redis port ``` 1. [Restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. @@ -353,13 +353,13 @@ or a failover promotes a different **Primary** node. sentinels: - host: 10.0.0.1 - port: 26379 # point to sentinel, not to redis port + port: 26379 # point to sentinel, not to redis port - host: 10.0.0.2 - port: 26379 # point to sentinel, not to redis port + port: 26379 # point to sentinel, not to redis port - host: 10.0.0.3 - port: 26379 # point to sentinel, not to redis port + port: 26379 # point to sentinel, not to redis port ``` 1. [Restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. diff --git a/doc/administration/redis/troubleshooting.md b/doc/administration/redis/troubleshooting.md index 402b60e5b7b..c9976ac791d 100644 --- a/doc/administration/redis/troubleshooting.md +++ b/doc/administration/redis/troubleshooting.md @@ -146,13 +146,13 @@ production: sentinels: - host: 10.0.0.1 - port: 26379 # point to sentinel, not to redis port + port: 26379 # point to sentinel, not to redis port - host: 10.0.0.2 - port: 26379 # point to sentinel, not to redis port + port: 26379 # point to sentinel, not to redis port - host: 10.0.0.3 - port: 26379 # point to sentinel, not to redis port + port: 26379 # point to sentinel, not to redis port ``` When in doubt, read the [Redis Sentinel documentation](https://redis.io/topics/sentinel). diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index 5f8ab6683a9..7910905ce54 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -17,21 +17,21 @@ full list of reference architectures, see | Service | Nodes | Configuration | GCP | AWS | Azure | |--------------------------------------------|-------------|-------------------------|-----------------|-------------|----------| -| External load balancing node | 1 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Consul | 3 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| PostgreSQL | 3 | 4 vCPU, 15GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Internal load balancing node | 1 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Redis - Cache | 3 | 4 vCPU, 15GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis - Queues / Shared State | 3 | 4 vCPU, 15GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7GB memory | g1-small | t2.small | B1MS | -| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7GB memory | g1-small | t2.small | B1MS | -| Gitaly | 2 (minimum) | 16 vCPU, 60GB memory | n1-standard-16 | m5.4xlarge | D16s v3 | -| Sidekiq | 4 | 4 vCPU, 15GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| GitLab Rails | 3 | 32 vCPU, 28.8GB memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | -| Monitoring node | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | -| Object Storage | n/a | n/a | n/a | n/a | n/a | -| NFS Server | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| External load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Consul | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| PostgreSQL | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Redis - Cache | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| Redis - Queues / Shared State | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7 GB memory | g1-small | t2.small | B1MS | +| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7 GB memory | g1-small | t2.small | B1MS | +| Gitaly | 2 (minimum) | 16 vCPU, 60 GB memory | n1-standard-16 | m5.4xlarge | D16s v3 | +| Sidekiq | 4 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| GitLab Rails | 3 | 32 vCPU, 28.8 GB memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | +| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| Object storage | n/a | n/a | n/a | n/a | n/a | +| NFS server | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | The Google Cloud Platform (GCP) architectures were built and tested using the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) @@ -40,41 +40,43 @@ or higher, are required for your CPU or node counts. For more information, see our [Sysbench](https://github.com/akopytov/sysbench)-based [CPU benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). -For data objects (such as LFS, Uploads, or Artifacts), an -[object storage service](#configure-the-object-storage) is recommended instead -of NFS where possible, due to better performance and availability. Since this -doesn't require a node to be set up, *Object Storage* is noted as not -applicable (n/a) in the previous table. +Due to better performance and availability, for data objects (such as LFS, +uploads, or artifacts), using an [object storage service](#configure-the-object-storage) +is recommended instead of using NFS. Using an object storage service also +doesn't require you to provision and maintain a node. ## Setup components To set up GitLab and its components to accommodate up to 10,000 users: 1. [Configure the external load balancing node](#configure-the-external-load-balancer) - that will handle the load balancing of the three GitLab application services nodes. + to handle the load balancing of the GitLab application services nodes. 1. [Configure Consul](#configure-consul). 1. [Configure PostgreSQL](#configure-postgresql), the database for GitLab. 1. [Configure PgBouncer](#configure-pgbouncer). -1. [Configure the internal load balancing node](#configure-the-internal-load-balancer) +1. [Configure the internal load balancing node](#configure-the-internal-load-balancer). 1. [Configure Redis](#configure-redis). 1. [Configure Gitaly](#configure-gitaly), which provides access to the Git repositories. 1. [Configure Sidekiq](#configure-sidekiq). 1. [Configure the main GitLab Rails application](#configure-gitlab-rails) - to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend requests (UI, API, Git - over HTTP/SSH). -1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab environment. -1. [Configure the Object Storage](#configure-the-object-storage) + to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend + requests (which include UI, API, and Git over HTTP/SSH). +1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab + environment. +1. [Configure the object storage](#configure-the-object-storage) used for shared data objects. -1. [Configure NFS (Optional)](#configure-nfs-optional) - to have shared disk storage service as an alternative to Gitaly and/or Object Storage (although - not recommended). NFS is required for GitLab Pages, you can skip this step if you're not using - that feature. +1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, + more advanced code search across your entire GitLab instance. +1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended) + to have shared disk storage service as an alternative to Gitaly or object + storage. You can skip this step if you're not using GitLab Pages (which + requires NFS). -We start with all servers on the same 10.6.0.0/24 private network range, they -can connect to each other freely on those addresses. +The servers start on the same 10.6.0.0/24 private network range, and can +connect to each other freely on these addresses. -Here is a list and description of each machine and the assigned IP: +The following list includes descriptions of each server and its assigned IP: - `10.6.0.10`: External Load Balancer - `10.6.0.11`: Consul 1 @@ -112,19 +114,18 @@ Here is a list and description of each machine and the assigned IP: ## Configure the external load balancer -NOTE: **Note:** +In an active/active GitLab configuration, you'll need a load balancer to route +traffic to the application servers. The specifics on which load balancer to use +or its exact configuration is beyond the scope of GitLab documentation. We hope +that if you're managing multi-node systems like GitLab, you already have a load +balancer of choice. Some load balancer examples include HAProxy (open-source), +F5 Big-IP LTM, and Citrix Net Scaler. This documentation outline the ports and +protocols needed for use with GitLab. + This architecture has been tested and validated with [HAProxy](https://www.haproxy.org/) as the load balancer. Although other load balancers with similar feature sets could also be used, those load balancers have not been validated. -In an active/active GitLab configuration, you will need a load balancer to route -traffic to the application servers. The specifics on which load balancer to use -or the exact configuration is beyond the scope of GitLab documentation. We hope -that if you're managing multi-node systems like GitLab you have a load balancer of -choice already. Some examples including HAProxy (open-source), F5 Big-IP LTM, -and Citrix Net Scaler. This documentation will outline what ports and protocols -you need to use with GitLab. - The next question is how you will handle SSL in your environment. There are several different options: @@ -247,14 +248,11 @@ with the other servers. To configure Consul: -1. SSH into the server that will host Consul. -1. [Download/install](https://about.gitlab.com/install/) the - Omnibus GitLab Enterprise Edition package using **steps 1 and 2** from the - GitLab downloads page. - - Make sure you select the correct Omnibus package, with the same version - the GitLab application is running. - - Do not complete any other steps on the download page. - +1. SSH in to the server that will host Consul. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to both follow _only_ installation steps 1 and 2 + on the page, and to select the correct Omnibus GitLab package, with the same version + and type (Community or Enterprise editions) as your current install. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby @@ -282,10 +280,9 @@ To configure Consul: 1. Go through the steps again for all the other Consul nodes, and make sure you set up the correct IPs. -NOTE: **Note:** -A Consul leader will be elected when the provisioning of the third Consul server is completed. -Viewing the Consul logs `sudo gitlab-ctl tail consul` will display -`...[INFO] consul: New leader elected: ...` +A Consul leader is _elected_ when the provisioning of the third Consul server is +complete. Viewing the Consul logs `sudo gitlab-ctl tail consul` displays +`...[INFO] consul: New leader elected: ...`. You can list the current Consul members (server, client): @@ -352,7 +349,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. #### PostgreSQL primary node -1. SSH into the PostgreSQL primary node. +1. SSH in to the PostgreSQL primary node. 1. Generate a password hash for the PostgreSQL username/password pair. This assumes you will use the default username of `gitlab` (recommended). The command will request a password and confirmation. Use the value that is output by this command in the next @@ -513,7 +510,7 @@ are supported and can be added if needed. #### PostgreSQL post-configuration -SSH into the **primary node**: +SSH in to the **primary node**: 1. Open a database prompt: @@ -548,7 +545,7 @@ SSH into the **primary node**: is not an IP address, it will need to be a resolvable name (via DNS or `/etc/hosts`) -SSH into the **secondary node**: +SSH in to the **secondary node**: 1. Set up the repmgr standby: @@ -662,7 +659,6 @@ The following IPs will be used as an example: 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - NOTE: **Note:** If an error `execute[generate databases.ini]` occurs, this is due to an existing [known issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4713). It will be resolved when you run a second `reconfigure` after the next step. @@ -796,35 +792,31 @@ to be used with GitLab. The following IPs will be used as an example: - `10.6.0.82`: Sentinel - Queues 2 - `10.6.0.83`: Sentinel - Queues 3 -NOTE: **Providing your own Redis instance:** -Managed Redis from cloud providers such as AWS ElastiCache will work. If these -services support high availability, be sure it is **not** the Redis Cluster type. -Redis version 5.0 or higher is required, as this is what ships with -Omnibus GitLab packages starting with GitLab 13.0. Older Redis versions -do not support an optional count argument to SPOP which is now required for -[Merge Trains](../../ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md). +### Providing your own Redis instance + +Managed Redis from cloud providers (such as AWS ElastiCache) will work. If these +services support high availability, be sure it _isn't_ of the Redis Cluster type. +Redis version 5.0 or higher is required, which is included with Omnibus GitLab +packages starting with GitLab 13.0. Older Redis versions don't support an +optional count argument to SPOP, which is required for [Merge Trains](../../ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md). Note the Redis node's IP address or hostname, port, and password (if required). -These will be necessary when configuring the -[GitLab application servers](#configure-gitlab-rails) later. +These will be necessary later when configuring the [GitLab application servers](#configure-gitlab-rails). ### Configure the Redis and Sentinel Cache cluster This is the section where we install and set up the new Redis Cache instances. -NOTE: **Note:** -Redis nodes (both primary and replica) will need the same password defined in -`redis['password']`. At any time during a failover the Sentinels can -reconfigure a node and change its status from primary to replica and vice versa. +Both the primary and replica Redis nodes need the same password defined in +`redis['password']`. At any time during a failover, the Sentinels can reconfigure +a node and change its status from primary to replica (and vice versa). #### Configure the primary Redis Cache node -1. SSH into the **Primary** Redis server. -1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab - package you want using **steps 1 and 2** from the GitLab downloads page. - - Make sure you select the correct Omnibus package, with the same version - and type (Community, Enterprise editions) of your current install. - - Do not complete any other steps on the download page. - +1. SSH in to the **Primary** Redis server. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to both follow _only_ installation steps 1 and 2 + on the page, and to select the correct Omnibus GitLab package, with the same version + and type (Community or Enterprise editions) as your current install. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby @@ -868,20 +860,17 @@ reconfigure a node and change its status from primary to replica and vice versa. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -NOTE: **Note:** -You can specify multiple roles like sentinel and Redis as: -`roles ['redis_sentinel_role', 'redis_master_role']`. -Read more about [roles](https://docs.gitlab.com/omnibus/roles/). +You can specify multiple roles, like sentinel and Redis, as: +`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +[roles](https://docs.gitlab.com/omnibus/roles/). #### Configure the replica Redis Cache nodes -1. SSH into the **replica** Redis server. -1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab - package you want using **steps 1 and 2** from the GitLab downloads page. - - Make sure you select the correct Omnibus package, with the same version - and type (Community, Enterprise editions) of your current install. - - Do not complete any other steps on the download page. - +1. SSH in to the **replica** Redis server. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to both follow _only_ installation steps 1 and 2 + on the page, and to select the correct Omnibus GitLab package, with the same version + and type (Community or Enterprise editions) as your current install. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby @@ -934,10 +923,9 @@ Read more about [roles](https://docs.gitlab.com/omnibus/roles/). 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. -NOTE: **Note:** -You can specify multiple roles like sentinel and Redis as: -`roles ['redis_sentinel_role', 'redis_master_role']`. -Read more about [roles](https://docs.gitlab.com/omnibus/roles/). +You can specify multiple roles, like sentinel and Redis, as: +`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +[roles](https://docs.gitlab.com/omnibus/roles/). These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-cache-nodes), and even after a @@ -955,13 +943,6 @@ are supported and can be added if needed. #### Configure the Sentinel Cache nodes -NOTE: **Note:** -If you are using an external Redis Sentinel instance, be sure -to exclude the `requirepass` parameter from the Sentinel -configuration. This parameter will cause clients to report `NOAUTH -Authentication required.`. [Redis Sentinel 3.2.x does not support -password authentication](https://github.com/antirez/redis/issues/3279). - Now that the Redis servers are all set up, let's configure the Sentinel servers. The following IPs will be used as an example: @@ -969,16 +950,19 @@ servers. The following IPs will be used as an example: - `10.6.0.72`: Sentinel - Cache 2 - `10.6.0.73`: Sentinel - Cache 3 -To configure the Sentinel Cache server: +NOTE: **Note:** +If you're using an external Redis Sentinel instance, be sure to exclude the +`requirepass` parameter from the Sentinel configuration. This parameter causes +clients to report `NOAUTH Authentication required.`. +[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279). -1. SSH into the server that will host Consul/Sentinel. -1. [Download/install](https://about.gitlab.com/install/) the - Omnibus GitLab Enterprise Edition package using **steps 1 and 2** from the - GitLab downloads page. - - Make sure you select the correct Omnibus package, with the same version - the GitLab application is running. - - Do not complete any other steps on the download page. +To configure the Sentinel Cache server: +1. SSH in to the server that will host Consul/Sentinel. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to both follow _only_ installation steps 1 and 2 + on the page, and to select the correct Omnibus GitLab package, with the same version + and type (Community or Enterprise editions) as your current install. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby @@ -1081,20 +1065,17 @@ To configure the Sentinel Cache server: This is the section where we install and set up the new Redis Queues instances. -NOTE: **Note:** -Redis nodes (both primary and replica) will need the same password defined in -`redis['password']`. At any time during a failover the Sentinels can -reconfigure a node and change its status from primary to replica and vice versa. +Both the primary and replica Redis nodes need the same password defined in +`redis['password']`. At any time during a failover, the Sentinels can reconfigure +a node and change its status from primary to replica (and vice versa). #### Configure the primary Redis Queues node -1. SSH into the **Primary** Redis server. -1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab - package you want using **steps 1 and 2** from the GitLab downloads page. - - Make sure you select the correct Omnibus package, with the same version - and type (Community, Enterprise editions) of your current install. - - Do not complete any other steps on the download page. - +1. SSH in to the **Primary** Redis server. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to both follow _only_ installation steps 1 and 2 + on the page, and to select the correct Omnibus GitLab package, with the same version + and type (Community or Enterprise editions) as your current install. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby @@ -1143,20 +1124,17 @@ reconfigure a node and change its status from primary to replica and vice versa. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -NOTE: **Note:** -You can specify multiple roles like sentinel and Redis as: -`roles ['redis_sentinel_role', 'redis_master_role']`. -Read more about [roles](https://docs.gitlab.com/omnibus/roles/). +You can specify multiple roles, like sentinel and Redis, as: +`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +[roles](https://docs.gitlab.com/omnibus/roles/). #### Configure the replica Redis Queues nodes -1. SSH into the **replica** Redis Queue server. -1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab - package you want using **steps 1 and 2** from the GitLab downloads page. - - Make sure you select the correct Omnibus package, with the same version - and type (Community, Enterprise editions) of your current install. - - Do not complete any other steps on the download page. - +1. SSH in to the **replica** Redis Queue server. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to both follow _only_ installation steps 1 and 2 + on the page, and to select the correct Omnibus GitLab package, with the same version + and type (Community or Enterprise editions) as your current install. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby @@ -1209,10 +1187,9 @@ Read more about [roles](https://docs.gitlab.com/omnibus/roles/). 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. -NOTE: **Note:** -You can specify multiple roles like sentinel and Redis as: -`roles ['redis_sentinel_role', 'redis_master_role']`. -Read more about [roles](https://docs.gitlab.com/omnibus/roles/). +You can specify multiple roles, like sentinel and Redis, as: +`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +[roles](https://docs.gitlab.com/omnibus/roles/). These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-queues-nodes), and even after a @@ -1230,13 +1207,6 @@ are supported and can be added if needed. #### Configure the Sentinel Queues nodes -NOTE: **Note:** -If you are using an external Redis Sentinel instance, be sure -to exclude the `requirepass` parameter from the Sentinel -configuration. This parameter will cause clients to report `NOAUTH -Authentication required.`. [Redis Sentinel 3.2.x does not support -password authentication](https://github.com/antirez/redis/issues/3279). - Now that the Redis servers are all set up, let's configure the Sentinel servers. The following IPs will be used as an example: @@ -1244,16 +1214,19 @@ servers. The following IPs will be used as an example: - `10.6.0.82`: Sentinel - Queues 2 - `10.6.0.83`: Sentinel - Queues 3 -To configure the Sentinel Queues server: +NOTE: **Note:** +If you're using an external Redis Sentinel instance, be sure to exclude the +`requirepass` parameter from the Sentinel configuration. This parameter causes +clients to report `NOAUTH Authentication required.`. +[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279). -1. SSH into the server that will host Sentinel. -1. [Download/install](https://about.gitlab.com/install/) the - Omnibus GitLab Enterprise Edition package using **steps 1 and 2** from the - GitLab downloads page. - - Make sure you select the correct Omnibus package, with the same version - the GitLab application is running. - - Do not complete any other steps on the download page. +To configure the Sentinel Queues server: +1. SSH in to the server that will host Sentinel. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to both follow _only_ installation steps 1 and 2 + on the page, and to select the correct Omnibus GitLab package, with the same version + and type (Community or Enterprise editions) as your current install. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby @@ -1362,52 +1335,48 @@ To configure the Sentinel Queues server: ## Configure Gitaly -Deploying Gitaly in its own server can benefit GitLab installations that are -larger than a single machine. - -The Gitaly node requirements are dependent on customer data, specifically the number of -projects and their repository sizes. Two nodes are recommended as an absolute minimum. -Each Gitaly node should store no more than 5TB of data and have the number of -[`gitaly-ruby` workers](../gitaly/index.md#gitaly-ruby) set to 20% of available CPUs. -Additional nodes should be considered in conjunction with a review of expected -data size and spread based on the recommendations above. - -It is also strongly recommended that all Gitaly nodes be set up with SSD disks with -a throughput of at least 8,000 IOPS for read operations and 2,000 IOPS for write, -as Gitaly has heavy I/O. These IOPS values are recommended only as a starter as with -time they may be adjusted higher or lower depending on the scale of your environment's workload. -If you're running the environment on a Cloud provider, you may need to refer to -their documentation on how to configure IOPS correctly. - -Some things to note: - -- The GitLab Rails application shards repositories into [repository storages](../repository_storage_paths.md). -- A Gitaly server can host one or more storages. -- A GitLab server can use one or more Gitaly servers. -- Gitaly addresses must be specified in such a way that they resolve - correctly for ALL Gitaly clients. +[Gitaly](../gitaly/index.md) server node requirements are dependent on data, +specifically the number of projects and those projects' sizes. It's recommended +that a Gitaly server node stores no more than 5 TB of data. Although this +reference architecture includes a recommendation for the number of Gitaly server +nodes to use, depending on your storage requirements, you may require additional +Gitaly server nodes. + +Due to Gitaly having notable input and output requirements, we strongly +recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs should +have a throughput of at least 8,000 input/output operations per second (IOPS) +for read operations and 2,000 IOPS for write operations. These IOPS values are +initial recommendations, and may be adjusted to greater or lesser values +depending on the scale of your environment's workload. If you're running the +environment on a Cloud provider, refer to their documentation about how to +configure IOPS correctly. + +Be sure to note the following items: + +- The GitLab Rails application shards repositories into + [repository storage paths](../repository_storage_paths.md). +- A Gitaly server can host one or more storage paths. +- A GitLab server can use one or more Gitaly server nodes. +- Gitaly addresses must be specified to be correctly resolvable for _all_ + Gitaly clients. - Gitaly servers must not be exposed to the public internet, as Gitaly's network traffic is unencrypted by default. The use of a firewall is highly recommended to restrict access to the Gitaly server. Another option is to [use TLS](#gitaly-tls-support). -TIP: **Tip:** -For more information about Gitaly's history and network architecture see the -[standalone Gitaly documentation](../gitaly/index.md). - -Note: **Note:** -The token referred to throughout the Gitaly documentation is -just an arbitrary password selected by the administrator. It is unrelated to -tokens created for the GitLab API or other similar web API tokens. +NOTE: **Note:** +The token referred to throughout the Gitaly documentation is an arbitrary +password selected by the administrator. This token is unrelated to tokens +created for the GitLab API or other similar web API tokens. -Below we describe how to configure two Gitaly servers, with IPs and -domain names: +This section describes how to configure two Gitaly servers, with the following +IPs and domain names: - `10.6.0.91`: Gitaly 1 (`gitaly1.internal`) - `10.6.0.92`: Gitaly 2 (`gitaly2.internal`) -The secret token is assumed to be `gitalysecret` and that -your GitLab installation has three repository storages: +Assumptions about your servers include having the secret token be `gitalysecret`, +and that your GitLab installation has three repository storages: - `default` on Gitaly 1 - `storage1` on Gitaly 1 @@ -1415,11 +1384,11 @@ your GitLab installation has three repository storages: On each node: -1. [Download/Install](https://about.gitlab.com/install/) the Omnibus GitLab - package you want using **steps 1 and 2** from the GitLab downloads page but - **without** providing the `EXTERNAL_URL` value. -1. Edit `/etc/gitlab/gitlab.rb` to configure storage paths, enable - the network listener and configure the token: +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to follow _only_ installation steps 1 and 2 + on the page, and _do not_ provide the `EXTERNAL_URL` value. +1. Edit `/etc/gitlab/gitlab.rb` to configure the storage paths, enable + the network listener, and configure the token: <!-- updates to following example must also be made at @@ -1467,39 +1436,39 @@ On each node: ``` 1. Append the following to `/etc/gitlab/gitlab.rb` for each respective server: - 1. On `gitaly1.internal`: - - ```ruby - git_data_dirs({ - 'default' => { - 'path' => '/var/opt/gitlab/git-data' - }, - 'storage1' => { - 'path' => '/mnt/gitlab/git-data' - }, - }) - ``` - - 1. On `gitaly2.internal`: - - ```ruby - git_data_dirs({ - 'storage2' => { - 'path' => '/mnt/gitlab/git-data' - }, - }) - ``` + - On `gitaly1.internal`: + + ```ruby + git_data_dirs({ + 'default' => { + 'path' => '/var/opt/gitlab/git-data' + }, + 'storage1' => { + 'path' => '/mnt/gitlab/git-data' + }, + }) + ``` + + - On `gitaly2.internal`: + + ```ruby + git_data_dirs({ + 'storage2' => { + 'path' => '/mnt/gitlab/git-data' + }, + }) + ``` <!-- updates to following example must also be made at https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab --> -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and + then replace the file of the same name on this server. If that file isn't on + this server, add the file from your Consul server to this server. -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). ### Gitaly TLS support @@ -1521,11 +1490,10 @@ Name. If you are addressing the Gitaly server by its IP address, you must add it as a Subject Alternative Name to the certificate. [gRPC does not support using an IP address as Common Name in a certificate](https://github.com/grpc/grpc/issues/2691). -NOTE: **Note:** -It is possible to configure Gitaly servers with both an -unencrypted listening address `listen_addr` and an encrypted listening -address `tls_listen_addr` at the same time. This allows you to do a -gradual transition from unencrypted to encrypted traffic, if necessary. +It's possible to configure Gitaly servers with both an unencrypted listening +address (`listen_addr`) and an encrypted listening address (`tls_listen_addr`) +at the same time. This allows you to do a gradual transition from unencrypted to +encrypted traffic, if necessary. To configure Gitaly with TLS: @@ -1580,10 +1548,10 @@ The following IPs will be used as an example: To configure the Sidekiq nodes, on each one: -1. SSH into the Sidekiq server. -1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab package -you want using steps 1 and 2 from the GitLab downloads page. -**Do not complete any other steps on the download page.** +1. SSH in to the Sidekiq server. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to follow _only_ installation steps 1 and 2 + on the page. 1. Open `/etc/gitlab/gitlab.rb` with your editor: ```ruby @@ -1701,15 +1669,14 @@ You can also run [multiple Sidekiq processes](../operations/extra_sidekiq_proces ## Configure GitLab Rails -NOTE: **Note:** -In our architectures we run each GitLab Rails node using the Puma webserver -and have its number of workers set to 90% of available CPUs along with four threads. For -nodes that are running Rails with other components the worker value should be reduced -accordingly where we've found 50% achieves a good balance but this is dependent -on workload. - This section describes how to configure the GitLab application (Rails) component. +In our architecture, we run each GitLab Rails node using the Puma webserver, and +have its number of workers set to 90% of available CPUs, with four threads. For +nodes running Rails with other components, the worker value should be reduced +accordingly. We've determined that a worker value of 50% achieves a good balance, +but this is dependent on workload. + The following IPs will be used as an example: - `10.6.0.111`: GitLab application 1 @@ -1718,10 +1685,9 @@ The following IPs will be used as an example: On each node perform the following: -1. Download/install Omnibus GitLab using **steps 1 and 2** from - [GitLab downloads](https://about.gitlab.com/install/). Do not complete other - steps on the download page. - +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to follow _only_ installation steps 1 and 2 + on the page. 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace the file of the same name on this server. If that file is not on this server, add the file from your Consul server to this server. @@ -1752,6 +1718,7 @@ On each node perform the following: roles ['application_role'] gitaly['enable'] = false nginx['enable'] = true + sidekiq['enable'] = false ## PostgreSQL connection details # Disable PostgreSQL on the application node @@ -1795,7 +1762,6 @@ On each node perform the following: # Set the network addresses that the exporters used for monitoring will listen on node_exporter['listen_address'] = '0.0.0.0:9100' gitlab_workhorse['prometheus_listen_addr'] = '0.0.0.0:9229' - sidekiq['listen_address'] = "0.0.0.0" puma['listen'] = '0.0.0.0' # Add the monitoring node's IP address to the monitoring whitelist and allow it to @@ -1836,7 +1802,7 @@ On each node perform the following: 1. Specify the necessary NFS mounts in `/etc/fstab`. The exact contents of `/etc/fstab` will depend on how you chose - to configure your NFS server. See the [NFS documentation](../high_availability/nfs.md) + to configure your NFS server. See the [NFS documentation](../nfs.md) for examples and the various options. 1. Create the shared directories. These may be different depending on your NFS @@ -1877,30 +1843,31 @@ On each node perform the following: 1. Optionally, from the Gitaly servers, confirm that Gitaly can perform callbacks to the internal API: ```shell - sudo /opt/gitlab/embedded/service/gitlab-shell/bin/check -config /opt/gitlab/embedded/service/gitlab-shell/config.yml + sudo /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml ``` -NOTE: **Note:** -When you specify `https` in the `external_url`, as in the example -above, GitLab assumes you have SSL certificates in `/etc/gitlab/ssl/`. If -certificates are not present, NGINX will fail to start. See the -[NGINX documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) -for more information. +When you specify `https` in the `external_url`, as in the previous example, +GitLab expects that the SSL certificates are in `/etc/gitlab/ssl/`. If the +certificates aren't present, NGINX will fail to start. For more information, see +the [NGINX documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https). ### GitLab Rails post-configuration -Initialize the GitLab database, by running the following in one of the Rails nodes: +1. Designate one application node for running database migrations during + installation and updates. Initialize the GitLab database and ensure all + migrations ran: -```shell -sudo gitlab-rake gitlab:db:configure -``` + ```shell + sudo gitlab-rake gitlab:db:configure + ``` -NOTE: **Note:** -If you encounter a `rake aborted!` error stating that PgBouncer is failing to connect to -PostgreSQL it may be that your PgBouncer node's IP address is missing from -PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` on your database nodes. See -[PgBouncer error `ERROR: pgbouncer cannot connect to server`](troubleshooting.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) -in the Troubleshooting section before proceeding. + If you encounter a `rake aborted!` error message stating that PgBouncer is + failing to connect to PostgreSQL, it may be that your PgBouncer node's IP + address is missing from PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` + on your database nodes. Before proceeding, see + [PgBouncer error `ERROR: pgbouncer cannot connect to server`](troubleshooting.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server). + +1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -1920,11 +1887,10 @@ The following IP will be used as an example: To configure the Monitoring node: -1. SSH into the Monitoring node. -1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab - package you want using **steps 1 and 2** from the GitLab downloads page. - Do not complete any other steps on the download page. - +1. SSH in to the Monitoring node. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to follow _only_ installation steps 1 and 2 + on the page. 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace the file of the same name on this server. If that file is not on this server, add the file from your Consul server to this server. @@ -1988,28 +1954,44 @@ It's recommended over [NFS](#configure-nfs-optional) and in general it's better in larger setups as object storage is typically much more performant, reliable, and scalable. -Object storage options that GitLab has tested, or is aware of customers using include: +GitLab has been tested on a number of object storage providers: -- SaaS/Cloud solutions such as [Amazon S3](https://aws.amazon.com/s3/), [Google cloud storage](https://cloud.google.com/storage). +- [Amazon S3](https://aws.amazon.com/s3/) +- [Google Cloud Storage](https://cloud.google.com/storage) +- [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces/) +- [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) +- [Openstack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) +- [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) - On-premises hardware and appliances from various storage vendors. -- MinIO. There is [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. - -For configuring GitLab to use Object Storage refer to the following guides -based on what features you intend to use: - -1. Configure [object storage for backups](../../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage). -1. Configure [object storage for job artifacts](../job_artifacts.md#using-object-storage) - including [incremental logging](../job_logs.md#new-incremental-logging-architecture). -1. Configure [object storage for LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage). -1. Configure [object storage for uploads](../uploads.md#using-object-storage). -1. Configure [object storage for merge request diffs](../merge_request_diffs.md#using-object-storage). -1. Configure [object storage for Container Registry](../packages/container_registry.md#use-object-storage) (optional feature). -1. Configure [object storage for Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage) (optional feature). -1. Configure [object storage for packages](../packages/index.md#using-object-storage) (optional feature). **(PREMIUM ONLY)** -1. Configure [object storage for Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature). **(PREMIUM ONLY)** -1. Configure [object storage for Pseudonymizer](../pseudonymizer.md#configuration) (optional feature). **(ULTIMATE ONLY)** -1. Configure [object storage for autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional - for improved performance). -1. Configure [object storage for Terraform state files](../terraform_state.md#using-object-storage). +- MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. + +There are two ways of specifying object storage configuration in GitLab: + +- [Consolidated form](../object_storage.md#consolidated-object-storage-configuration): A single credential is + shared by all supported object types. +- [Storage-specific form](../object_storage.md#storage-specific-configuration): Every object defines its + own object storage [connection and configuration](../object_storage.md#connection-settings). + +Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. + +For configuring object storage in GitLab 13.1 and earlier, or for storage types not +supported by consolidated configuration form, refer to the following guides based +on what features you intend to use: + +|Object storage type|Supported by consolidated configuration?| +|-------------------|----------------------------------------| +| [Backups](../../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage)|No| +| [Job artifacts](../job_artifacts.md#using-object-storage) including archived job logs | Yes | +| [LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage) | Yes | +| [Uploads](../uploads.md#using-object-storage) | Yes | +| [Container Registry](../packages/container_registry.md#use-object-storage) (optional feature) | No | +| [Merge request diffs](../merge_request_diffs.md#using-object-storage) | Yes | +| [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | +| [Packages](../packages/index.md#using-object-storage) (optional feature) | Yes | +| [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature) **(PREMIUM ONLY)** | Yes | +| [Pseudonymizer](../pseudonymizer.md#configuration) (optional feature) **(ULTIMATE ONLY)** | No | +| [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | +| [Terraform state files](../terraform_state.md#using-object-storage) | Yes | Using separate buckets for each data type is the recommended approach for GitLab. @@ -2033,13 +2015,29 @@ work. </a> </div> +## Configure Advanced Search **(STARTER ONLY)** + +You can leverage Elasticsearch and [enable Advanced Search](../../integration/elasticsearch.md) +for faster, more advanced code search across your entire GitLab instance. + +Elasticsearch cluster design and requirements are dependent on your specific +data. For recommended best practices about how to set up your Elasticsearch +cluster alongside your instance, read how to +[choose the optimal cluster configuration](../../integration/elasticsearch.md#guidance-on-choosing-optimal-cluster-configuration). + +<div align="right"> + <a type="button" class="btn btn-default" href="#setup-components"> + Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> + </a> +</div> + ## Configure NFS (optional) [Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly) are recommended over NFS wherever possible for improved performance. If you intend to use GitLab Pages, this currently [requires NFS](troubleshooting.md#gitlab-pages-requires-nfs). -See how to [configure NFS](../high_availability/nfs.md). +See how to [configure NFS](../nfs.md). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> diff --git a/doc/administration/reference_architectures/1k_users.md b/doc/administration/reference_architectures/1k_users.md index d376c1b7575..0cb7b8868c3 100644 --- a/doc/administration/reference_architectures/1k_users.md +++ b/doc/administration/reference_architectures/1k_users.md @@ -20,8 +20,8 @@ many organizations . | Users | Configuration | GCP | AWS | Azure | |--------------|-------------------------|----------------|-----------------|----------------| -| Up to 500 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | -| Up to 1,000 | 8 vCPU, 7.2GB memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | +| Up to 500 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| Up to 1,000 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | The Google Cloud Platform (GCP) architectures were built and tested using the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) @@ -30,20 +30,28 @@ or higher, are required for your CPU or node counts. For more information, see our [Sysbench](https://github.com/akopytov/sysbench)-based [CPU benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). -In addition to the stated configurations, we recommend having at least 2GB of +In addition to the stated configurations, we recommend having at least 2 GB of swap on your server, even if you currently have enough available memory. Having -swap will help reduce the chance of errors occurring if your available memory +swap helps to reduce the chance of errors occurring if your available memory changes. We also recommend configuring the kernel's swappiness setting to a lower value (such as `10`) to make the most of your memory, while still having the swap available when needed. ## Setup instructions -For this default reference architecture, to install GitLab use the standard +To install GitLab for this default reference architecture, use the standard [installation instructions](../../install/README.md). -NOTE: **Note:** -You can also optionally configure GitLab to use an -[external PostgreSQL service](../postgresql/external.md) or an -[external object storage service](../high_availability/object_storage.md) for -added performance and reliability at a reduced complexity cost. +You can also optionally configure GitLab to use an [external PostgreSQL service](../postgresql/external.md) +or an [external object storage service](../object_storage.md) for added +performance and reliability at a reduced complexity cost. + +## Configure Advanced Search **(STARTER ONLY)** + +You can leverage Elasticsearch and [enable Advanced Search](../../integration/elasticsearch.md) +for faster, more advanced code search across your entire GitLab instance. + +Elasticsearch cluster design and requirements are dependent on your specific +data. For recommended best practices about how to set up your Elasticsearch +cluster alongside your instance, read how to +[choose the optimal cluster configuration](../../integration/elasticsearch.md#guidance-on-choosing-optimal-cluster-configuration). diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index 2ef555bff29..bb6e2eb4376 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -17,21 +17,21 @@ full list of reference architectures, see | Service | Nodes | Configuration | GCP | AWS | Azure | |-----------------------------------------|-------------|-------------------------|-----------------|-------------|----------| -| External load balancing node | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | -| Consul | 3 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| PostgreSQL | 3 | 8 vCPU, 30GB memory | n1-standard-8 | m5.2xlarge | D8s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Internal load balancing node | 1 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Redis - Cache | 3 | 4 vCPU, 15GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis - Queues / Shared State | 3 | 4 vCPU, 15GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7GB memory | g1-small | t2.small | B1MS | -| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7GB memory | g1-small | t2.small | B1MS | -| Gitaly | 2 (minimum) | 32 vCPU, 120GB memory | n1-standard-32 | m5.8xlarge | D32s v3 | -| Sidekiq | 4 | 4 vCPU, 15GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| GitLab Rails | 5 | 32 vCPU, 28.8GB memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | -| Monitoring node | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | -| Object Storage | n/a | n/a | n/a | n/a | n/a | -| NFS Server | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| External load balancing node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| Consul | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| PostgreSQL | 3 | 8 vCPU, 30 GB memory | n1-standard-8 | m5.2xlarge | D8s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Internal load balancing node | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | c5.large | F2s v2 | +| Redis - Cache | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| Redis - Queues / Shared State | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7 GB memory | g1-small | t2.small | B1MS | +| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7 GB memory | g1-small | t2.small | B1MS | +| Gitaly | 2 (minimum) | 32 vCPU, 120 GB memory | n1-standard-32 | m5.8xlarge | D32s v3 | +| Sidekiq | 4 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| GitLab Rails | 5 | 32 vCPU, 28.8 GB memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | +| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| Object storage | n/a | n/a | n/a | n/a | n/a | +| NFS server | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | The Google Cloud Platform (GCP) architectures were built and tested using the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) @@ -40,41 +40,43 @@ or higher, are required for your CPU or node counts. For more information, see our [Sysbench](https://github.com/akopytov/sysbench)-based [CPU benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). -For data objects (such as LFS, Uploads, or Artifacts), an -[object storage service](#configure-the-object-storage) is recommended instead -of NFS where possible, due to better performance and availability. Since this -doesn't require a node to be set up, *Object Storage* is noted as not -applicable (n/a) in the previous table. +Due to better performance and availability, for data objects (such as LFS, +uploads, or artifacts), using an [object storage service](#configure-the-object-storage) +is recommended instead of using NFS. Using an object storage service also +doesn't require you to provision and maintain a node. ## Setup components To set up GitLab and its components to accommodate up to 25,000 users: 1. [Configure the external load balancing node](#configure-the-external-load-balancer) - that will handle the load balancing of the three GitLab application services nodes. + to handle the load balancing of the GitLab application services nodes. 1. [Configure Consul](#configure-consul). 1. [Configure PostgreSQL](#configure-postgresql), the database for GitLab. 1. [Configure PgBouncer](#configure-pgbouncer). -1. [Configure the internal load balancing node](#configure-the-internal-load-balancer) +1. [Configure the internal load balancing node](#configure-the-internal-load-balancer). 1. [Configure Redis](#configure-redis). 1. [Configure Gitaly](#configure-gitaly), which provides access to the Git repositories. 1. [Configure Sidekiq](#configure-sidekiq). 1. [Configure the main GitLab Rails application](#configure-gitlab-rails) - to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend requests (UI, API, Git - over HTTP/SSH). -1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab environment. -1. [Configure the Object Storage](#configure-the-object-storage) + to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend + requests (which include UI, API, and Git over HTTP/SSH). +1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab + environment. +1. [Configure the object storage](#configure-the-object-storage) used for shared data objects. -1. [Configure NFS (Optional)](#configure-nfs-optional) - to have shared disk storage service as an alternative to Gitaly and/or Object Storage (although - not recommended). NFS is required for GitLab Pages, you can skip this step if you're not using - that feature. +1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, + more advanced code search across your entire GitLab instance. +1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended) + to have shared disk storage service as an alternative to Gitaly or object + storage. You can skip this step if you're not using GitLab Pages (which + requires NFS). -We start with all servers on the same 10.6.0.0/24 private network range, they -can connect to each other freely on those addresses. +The servers start on the same 10.6.0.0/24 private network range, and can +connect to each other freely on these addresses. -Here is a list and description of each machine and the assigned IP: +The following list includes descriptions of each server and its assigned IP: - `10.6.0.10`: External Load Balancer - `10.6.0.11`: Consul 1 @@ -112,19 +114,18 @@ Here is a list and description of each machine and the assigned IP: ## Configure the external load balancer -NOTE: **Note:** +In an active/active GitLab configuration, you'll need a load balancer to route +traffic to the application servers. The specifics on which load balancer to use +or its exact configuration is beyond the scope of GitLab documentation. We hope +that if you're managing multi-node systems like GitLab, you already have a load +balancer of choice. Some load balancer examples include HAProxy (open-source), +F5 Big-IP LTM, and Citrix Net Scaler. This documentation outline the ports and +protocols needed for use with GitLab. + This architecture has been tested and validated with [HAProxy](https://www.haproxy.org/) as the load balancer. Although other load balancers with similar feature sets could also be used, those load balancers have not been validated. -In an active/active GitLab configuration, you will need a load balancer to route -traffic to the application servers. The specifics on which load balancer to use -or the exact configuration is beyond the scope of GitLab documentation. We hope -that if you're managing multi-node systems like GitLab you have a load balancer of -choice already. Some examples including HAProxy (open-source), F5 Big-IP LTM, -and Citrix Net Scaler. This documentation will outline what ports and protocols -you need to use with GitLab. - The next question is how you will handle SSL in your environment. There are several different options: @@ -247,14 +248,11 @@ with the other servers. To configure Consul: -1. SSH into the server that will host Consul. -1. [Download/install](https://about.gitlab.com/install/) the - Omnibus GitLab Enterprise Edition package using **steps 1 and 2** from the - GitLab downloads page. - - Make sure you select the correct Omnibus package, with the same version - the GitLab application is running. - - Do not complete any other steps on the download page. - +1. SSH in to the server that will host Consul. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to both follow _only_ installation steps 1 and 2 + on the page, and to select the correct Omnibus GitLab package, with the same version + and type (Community or Enterprise editions) as your current install. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby @@ -282,10 +280,9 @@ To configure Consul: 1. Go through the steps again for all the other Consul nodes, and make sure you set up the correct IPs. -NOTE: **Note:** -A Consul leader will be elected when the provisioning of the third Consul server is completed. -Viewing the Consul logs `sudo gitlab-ctl tail consul` will display -`...[INFO] consul: New leader elected: ...` +A Consul leader is _elected_ when the provisioning of the third Consul server is +complete. Viewing the Consul logs `sudo gitlab-ctl tail consul` displays +`...[INFO] consul: New leader elected: ...`. You can list the current Consul members (server, client): @@ -352,7 +349,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. #### PostgreSQL primary node -1. SSH into the PostgreSQL primary node. +1. SSH in to the PostgreSQL primary node. 1. Generate a password hash for the PostgreSQL username/password pair. This assumes you will use the default username of `gitlab` (recommended). The command will request a password and confirmation. Use the value that is output by this command in the next @@ -513,7 +510,7 @@ are supported and can be added if needed. #### PostgreSQL post-configuration -SSH into the **primary node**: +SSH in to the **primary node**: 1. Open a database prompt: @@ -548,7 +545,7 @@ SSH into the **primary node**: is not an IP address, it will need to be a resolvable name (via DNS or `/etc/hosts`) -SSH into the **secondary node**: +SSH in to the **secondary node**: 1. Set up the repmgr standby: @@ -662,7 +659,6 @@ The following IPs will be used as an example: 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - NOTE: **Note:** If an error `execute[generate databases.ini]` occurs, this is due to an existing [known issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4713). It will be resolved when you run a second `reconfigure` after the next step. @@ -796,35 +792,31 @@ to be used with GitLab. The following IPs will be used as an example: - `10.6.0.82`: Sentinel - Queues 2 - `10.6.0.83`: Sentinel - Queues 3 -NOTE: **Providing your own Redis instance:** -Managed Redis from cloud providers such as AWS ElastiCache will work. If these -services support high availability, be sure it is **not** the Redis Cluster type. -Redis version 5.0 or higher is required, as this is what ships with -Omnibus GitLab packages starting with GitLab 13.0. Older Redis versions -do not support an optional count argument to SPOP which is now required for -[Merge Trains](../../ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md). +### Providing your own Redis instance + +Managed Redis from cloud providers (such as AWS ElastiCache) will work. If these +services support high availability, be sure it _isn't_ of the Redis Cluster type. +Redis version 5.0 or higher is required, which is included with Omnibus GitLab +packages starting with GitLab 13.0. Older Redis versions don't support an +optional count argument to SPOP, which is required for [Merge Trains](../../ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md). Note the Redis node's IP address or hostname, port, and password (if required). -These will be necessary when configuring the -[GitLab application servers](#configure-gitlab-rails) later. +These will be necessary later when configuring the [GitLab application servers](#configure-gitlab-rails). ### Configure the Redis and Sentinel Cache cluster This is the section where we install and set up the new Redis Cache instances. -NOTE: **Note:** -Redis nodes (both primary and replica) will need the same password defined in -`redis['password']`. At any time during a failover the Sentinels can -reconfigure a node and change its status from primary to replica and vice versa. +Both the primary and replica Redis nodes need the same password defined in +`redis['password']`. At any time during a failover, the Sentinels can reconfigure +a node and change its status from primary to replica (and vice versa). #### Configure the primary Redis Cache node -1. SSH into the **Primary** Redis server. -1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab - package you want using **steps 1 and 2** from the GitLab downloads page. - - Make sure you select the correct Omnibus package, with the same version - and type (Community, Enterprise editions) of your current install. - - Do not complete any other steps on the download page. - +1. SSH in to the **Primary** Redis server. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to both follow _only_ installation steps 1 and 2 + on the page, and to select the correct Omnibus GitLab package, with the same version + and type (Community or Enterprise editions) as your current install. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby @@ -868,20 +860,17 @@ reconfigure a node and change its status from primary to replica and vice versa. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -NOTE: **Note:** -You can specify multiple roles like sentinel and Redis as: -`roles ['redis_sentinel_role', 'redis_master_role']`. -Read more about [roles](https://docs.gitlab.com/omnibus/roles/). +You can specify multiple roles, like sentinel and Redis, as: +`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +[roles](https://docs.gitlab.com/omnibus/roles/). #### Configure the replica Redis Cache nodes -1. SSH into the **replica** Redis server. -1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab - package you want using **steps 1 and 2** from the GitLab downloads page. - - Make sure you select the correct Omnibus package, with the same version - and type (Community, Enterprise editions) of your current install. - - Do not complete any other steps on the download page. - +1. SSH in to the **replica** Redis server. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to both follow _only_ installation steps 1 and 2 + on the page, and to select the correct Omnibus GitLab package, with the same version + and type (Community or Enterprise editions) as your current install. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby @@ -934,10 +923,9 @@ Read more about [roles](https://docs.gitlab.com/omnibus/roles/). 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. -NOTE: **Note:** -You can specify multiple roles like sentinel and Redis as: -`roles ['redis_sentinel_role', 'redis_master_role']`. -Read more about [roles](https://docs.gitlab.com/omnibus/roles/). +You can specify multiple roles, like sentinel and Redis, as: +`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +[roles](https://docs.gitlab.com/omnibus/roles/). These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-cache-nodes), and even after a @@ -955,13 +943,6 @@ are supported and can be added if needed. #### Configure the Sentinel Cache nodes -NOTE: **Note:** -If you are using an external Redis Sentinel instance, be sure -to exclude the `requirepass` parameter from the Sentinel -configuration. This parameter will cause clients to report `NOAUTH -Authentication required.`. [Redis Sentinel 3.2.x does not support -password authentication](https://github.com/antirez/redis/issues/3279). - Now that the Redis servers are all set up, let's configure the Sentinel servers. The following IPs will be used as an example: @@ -969,16 +950,19 @@ servers. The following IPs will be used as an example: - `10.6.0.72`: Sentinel - Cache 2 - `10.6.0.73`: Sentinel - Cache 3 -To configure the Sentinel Cache server: +NOTE: **Note:** +If you're using an external Redis Sentinel instance, be sure to exclude the +`requirepass` parameter from the Sentinel configuration. This parameter causes +clients to report `NOAUTH Authentication required.`. +[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279). -1. SSH into the server that will host Consul/Sentinel. -1. [Download/install](https://about.gitlab.com/install/) the - Omnibus GitLab Enterprise Edition package using **steps 1 and 2** from the - GitLab downloads page. - - Make sure you select the correct Omnibus package, with the same version - the GitLab application is running. - - Do not complete any other steps on the download page. +To configure the Sentinel Cache server: +1. SSH in to the server that will host Consul/Sentinel. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to both follow _only_ installation steps 1 and 2 + on the page, and to select the correct Omnibus GitLab package, with the same version + and type (Community or Enterprise editions) as your current install. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby @@ -1081,20 +1065,17 @@ To configure the Sentinel Cache server: This is the section where we install and set up the new Redis Queues instances. -NOTE: **Note:** -Redis nodes (both primary and replica) will need the same password defined in -`redis['password']`. At any time during a failover the Sentinels can -reconfigure a node and change its status from primary to replica and vice versa. +Both the primary and replica Redis nodes need the same password defined in +`redis['password']`. At any time during a failover, the Sentinels can reconfigure +a node and change its status from primary to replica (and vice versa). #### Configure the primary Redis Queues node -1. SSH into the **Primary** Redis server. -1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab - package you want using **steps 1 and 2** from the GitLab downloads page. - - Make sure you select the correct Omnibus package, with the same version - and type (Community, Enterprise editions) of your current install. - - Do not complete any other steps on the download page. - +1. SSH in to the **Primary** Redis server. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to both follow _only_ installation steps 1 and 2 + on the page, and to select the correct Omnibus GitLab package, with the same version + and type (Community or Enterprise editions) as your current install. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby @@ -1143,20 +1124,17 @@ reconfigure a node and change its status from primary to replica and vice versa. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -NOTE: **Note:** -You can specify multiple roles like sentinel and Redis as: -`roles ['redis_sentinel_role', 'redis_master_role']`. -Read more about [roles](https://docs.gitlab.com/omnibus/roles/). +You can specify multiple roles, like sentinel and Redis, as: +`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +[roles](https://docs.gitlab.com/omnibus/roles/). #### Configure the replica Redis Queues nodes -1. SSH into the **replica** Redis Queue server. -1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab - package you want using **steps 1 and 2** from the GitLab downloads page. - - Make sure you select the correct Omnibus package, with the same version - and type (Community, Enterprise editions) of your current install. - - Do not complete any other steps on the download page. - +1. SSH in to the **replica** Redis Queue server. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to both follow _only_ installation steps 1 and 2 + on the page, and to select the correct Omnibus GitLab package, with the same version + and type (Community or Enterprise editions) as your current install. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby @@ -1209,10 +1187,9 @@ Read more about [roles](https://docs.gitlab.com/omnibus/roles/). 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. -NOTE: **Note:** -You can specify multiple roles like sentinel and Redis as: -`roles ['redis_sentinel_role', 'redis_master_role']`. -Read more about [roles](https://docs.gitlab.com/omnibus/roles/). +You can specify multiple roles, like sentinel and Redis, as: +`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +[roles](https://docs.gitlab.com/omnibus/roles/). These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-queues-nodes), and even after a @@ -1230,13 +1207,6 @@ are supported and can be added if needed. #### Configure the Sentinel Queues nodes -NOTE: **Note:** -If you are using an external Redis Sentinel instance, be sure -to exclude the `requirepass` parameter from the Sentinel -configuration. This parameter will cause clients to report `NOAUTH -Authentication required.`. [Redis Sentinel 3.2.x does not support -password authentication](https://github.com/antirez/redis/issues/3279). - Now that the Redis servers are all set up, let's configure the Sentinel servers. The following IPs will be used as an example: @@ -1244,16 +1214,19 @@ servers. The following IPs will be used as an example: - `10.6.0.82`: Sentinel - Queues 2 - `10.6.0.83`: Sentinel - Queues 3 -To configure the Sentinel Queues server: +NOTE: **Note:** +If you're using an external Redis Sentinel instance, be sure to exclude the +`requirepass` parameter from the Sentinel configuration. This parameter causes +clients to report `NOAUTH Authentication required.`. +[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279). -1. SSH into the server that will host Sentinel. -1. [Download/install](https://about.gitlab.com/install/) the - Omnibus GitLab Enterprise Edition package using **steps 1 and 2** from the - GitLab downloads page. - - Make sure you select the correct Omnibus package, with the same version - the GitLab application is running. - - Do not complete any other steps on the download page. +To configure the Sentinel Queues server: +1. SSH in to the server that will host Sentinel. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to both follow _only_ installation steps 1 and 2 + on the page, and to select the correct Omnibus package, with the same version + and type (Community or Enterprise editions) as your current install. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby @@ -1362,52 +1335,48 @@ To configure the Sentinel Queues server: ## Configure Gitaly -Deploying Gitaly in its own server can benefit GitLab installations that are -larger than a single machine. - -The Gitaly node requirements are dependent on customer data, specifically the number of -projects and their repository sizes. Two nodes are recommended as an absolute minimum. -Each Gitaly node should store no more than 5TB of data and have the number of -[`gitaly-ruby` workers](../gitaly/index.md#gitaly-ruby) set to 20% of available CPUs. -Additional nodes should be considered in conjunction with a review of expected -data size and spread based on the recommendations above. - -It is also strongly recommended that all Gitaly nodes be set up with SSD disks with -a throughput of at least 8,000 IOPS for read operations and 2,000 IOPS for write, -as Gitaly has heavy I/O. These IOPS values are recommended only as a starter as with -time they may be adjusted higher or lower depending on the scale of your environment's workload. -If you're running the environment on a Cloud provider, you may need to refer to -their documentation on how to configure IOPS correctly. - -Some things to note: - -- The GitLab Rails application shards repositories into [repository storages](../repository_storage_paths.md). -- A Gitaly server can host one or more storages. -- A GitLab server can use one or more Gitaly servers. -- Gitaly addresses must be specified in such a way that they resolve - correctly for ALL Gitaly clients. +[Gitaly](../gitaly/index.md) server node requirements are dependent on data, +specifically the number of projects and those projects' sizes. It's recommended +that a Gitaly server node stores no more than 5 TB of data. Although this +reference architecture includes a recommendation for the number of Gitaly server +nodes to use, depending on your storage requirements, you may require additional +Gitaly server nodes. + +Due to Gitaly having notable input and output requirements, we strongly +recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs should +have a throughput of at least 8,000 input/output operations per second (IOPS) +for read operations and 2,000 IOPS for write operations. These IOPS values are +initial recommendations, and may be adjusted to greater or lesser values +depending on the scale of your environment's workload. If you're running the +environment on a Cloud provider, refer to their documentation about how to +configure IOPS correctly. + +Be sure to note the following items: + +- The GitLab Rails application shards repositories into + [repository storage paths](../repository_storage_paths.md). +- A Gitaly server can host one or more storage paths. +- A GitLab server can use one or more Gitaly server nodes. +- Gitaly addresses must be specified to be correctly resolvable for _all_ + Gitaly clients. - Gitaly servers must not be exposed to the public internet, as Gitaly's network traffic is unencrypted by default. The use of a firewall is highly recommended to restrict access to the Gitaly server. Another option is to [use TLS](#gitaly-tls-support). -TIP: **Tip:** -For more information about Gitaly's history and network architecture see the -[standalone Gitaly documentation](../gitaly/index.md). - -Note: **Note:** -The token referred to throughout the Gitaly documentation is -just an arbitrary password selected by the administrator. It is unrelated to -tokens created for the GitLab API or other similar web API tokens. +NOTE: **Note:** +The token referred to throughout the Gitaly documentation is an arbitrary +password selected by the administrator. This token is unrelated to tokens +created for the GitLab API or other similar web API tokens. -Below we describe how to configure two Gitaly servers, with IPs and -domain names: +This section describes how to configure two Gitaly servers, with the following +IPs and domain names: - `10.6.0.91`: Gitaly 1 (`gitaly1.internal`) - `10.6.0.92`: Gitaly 2 (`gitaly2.internal`) -The secret token is assumed to be `gitalysecret` and that -your GitLab installation has three repository storages: +Assumptions about your servers include having the secret token be `gitalysecret`, +and that your GitLab installation has three repository storages: - `default` on Gitaly 1 - `storage1` on Gitaly 1 @@ -1415,11 +1384,11 @@ your GitLab installation has three repository storages: On each node: -1. [Download/Install](https://about.gitlab.com/install/) the Omnibus GitLab - package you want using **steps 1 and 2** from the GitLab downloads page but - **without** providing the `EXTERNAL_URL` value. -1. Edit `/etc/gitlab/gitlab.rb` to configure storage paths, enable - the network listener and configure the token: +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to follow _only_ installation steps 1 and 2 + on the page, and _do not_ provide the `EXTERNAL_URL` value. +1. Edit `/etc/gitlab/gitlab.rb` to configure the storage paths, enable + the network listener, and configure the token: <!-- updates to following example must also be made at @@ -1467,39 +1436,39 @@ On each node: ``` 1. Append the following to `/etc/gitlab/gitlab.rb` for each respective server: - 1. On `gitaly1.internal`: - - ```ruby - git_data_dirs({ - 'default' => { - 'path' => '/var/opt/gitlab/git-data' - }, - 'storage1' => { - 'path' => '/mnt/gitlab/git-data' - }, - }) - ``` - - 1. On `gitaly2.internal`: - - ```ruby - git_data_dirs({ - 'storage2' => { - 'path' => '/mnt/gitlab/git-data' - }, - }) - ``` + - On `gitaly1.internal`: + + ```ruby + git_data_dirs({ + 'default' => { + 'path' => '/var/opt/gitlab/git-data' + }, + 'storage1' => { + 'path' => '/mnt/gitlab/git-data' + }, + }) + ``` + + - On `gitaly2.internal`: + + ```ruby + git_data_dirs({ + 'storage2' => { + 'path' => '/mnt/gitlab/git-data' + }, + }) + ``` <!-- updates to following example must also be made at https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab --> -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and + then replace the file of the same name on this server. If that file isn't on + this server, add the file from your Consul server to this server. -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). ### Gitaly TLS support @@ -1521,11 +1490,10 @@ Name. If you are addressing the Gitaly server by its IP address, you must add it as a Subject Alternative Name to the certificate. [gRPC does not support using an IP address as Common Name in a certificate](https://github.com/grpc/grpc/issues/2691). -NOTE: **Note:** -It is possible to configure Gitaly servers with both an -unencrypted listening address `listen_addr` and an encrypted listening -address `tls_listen_addr` at the same time. This allows you to do a -gradual transition from unencrypted to encrypted traffic, if necessary. +It's possible to configure Gitaly servers with both an unencrypted listening +address (`listen_addr`) and an encrypted listening address (`tls_listen_addr`) +at the same time. This allows you to do a gradual transition from unencrypted to +encrypted traffic, if necessary. To configure Gitaly with TLS: @@ -1580,10 +1548,10 @@ The following IPs will be used as an example: To configure the Sidekiq nodes, on each one: -1. SSH into the Sidekiq server. -1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab package -you want using steps 1 and 2 from the GitLab downloads page. -**Do not complete any other steps on the download page.** +1. SSH in to the Sidekiq server. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to follow _only_ installation steps 1 and 2 + on the page. 1. Open `/etc/gitlab/gitlab.rb` with your editor: ```ruby @@ -1701,15 +1669,14 @@ You can also run [multiple Sidekiq processes](../operations/extra_sidekiq_proces ## Configure GitLab Rails -NOTE: **Note:** -In our architectures we run each GitLab Rails node using the Puma webserver -and have its number of workers set to 90% of available CPUs along with four threads. For -nodes that are running Rails with other components the worker value should be reduced -accordingly where we've found 50% achieves a good balance but this is dependent -on workload. - This section describes how to configure the GitLab application (Rails) component. +In our architecture, we run each GitLab Rails node using the Puma webserver, and +have its number of workers set to 90% of available CPUs, with four threads. For +nodes running Rails with other components, the worker value should be reduced +accordingly. We've determined that a worker value of 50% achieves a good balance, +but this is dependent on workload. + The following IPs will be used as an example: - `10.6.0.111`: GitLab application 1 @@ -1718,10 +1685,9 @@ The following IPs will be used as an example: On each node perform the following: -1. Download/install Omnibus GitLab using **steps 1 and 2** from - [GitLab downloads](https://about.gitlab.com/install/). Do not complete other - steps on the download page. - +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to follow _only_ installation steps 1 and 2 + on the page. 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace the file of the same name on this server. If that file is not on this server, add the file from your Consul server to this server. @@ -1752,6 +1718,7 @@ On each node perform the following: roles ['application_role'] gitaly['enable'] = false nginx['enable'] = true + sidekiq['enable'] = false ## PostgreSQL connection details # Disable PostgreSQL on the application node @@ -1795,7 +1762,6 @@ On each node perform the following: # Set the network addresses that the exporters used for monitoring will listen on node_exporter['listen_address'] = '0.0.0.0:9100' gitlab_workhorse['prometheus_listen_addr'] = '0.0.0.0:9229' - sidekiq['listen_address'] = "0.0.0.0" puma['listen'] = '0.0.0.0' # Add the monitoring node's IP address to the monitoring whitelist and allow it to @@ -1836,7 +1802,7 @@ On each node perform the following: 1. Specify the necessary NFS mounts in `/etc/fstab`. The exact contents of `/etc/fstab` will depend on how you chose - to configure your NFS server. See the [NFS documentation](../high_availability/nfs.md) + to configure your NFS server. See the [NFS documentation](../nfs.md) for examples and the various options. 1. Create the shared directories. These may be different depending on your NFS @@ -1877,30 +1843,31 @@ On each node perform the following: 1. Optionally, from the Gitaly servers, confirm that Gitaly can perform callbacks to the internal API: ```shell - sudo /opt/gitlab/embedded/service/gitlab-shell/bin/check -config /opt/gitlab/embedded/service/gitlab-shell/config.yml + sudo /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml ``` -NOTE: **Note:** -When you specify `https` in the `external_url`, as in the example -above, GitLab assumes you have SSL certificates in `/etc/gitlab/ssl/`. If -certificates are not present, NGINX will fail to start. See the -[NGINX documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) -for more information. +When you specify `https` in the `external_url`, as in the previous example, +GitLab expects that the SSL certificates are in `/etc/gitlab/ssl/`. If the +certificates aren't present, NGINX will fail to start. For more information, see +the [NGINX documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https). ### GitLab Rails post-configuration -Initialize the GitLab database, by running the following in one of the Rails nodes: +1. Designate one application node for running database migrations during + installation and updates. Initialize the GitLab database and ensure all + migrations ran: -```shell -sudo gitlab-rake gitlab:db:configure -``` + ```shell + sudo gitlab-rake gitlab:db:configure + ``` -NOTE: **Note:** -If you encounter a `rake aborted!` error stating that PgBouncer is failing to connect to -PostgreSQL it may be that your PgBouncer node's IP address is missing from -PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` on your database nodes. See -[PgBouncer error `ERROR: pgbouncer cannot connect to server`](troubleshooting.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) -in the Troubleshooting section before proceeding. + If you encounter a `rake aborted!` error message stating that PgBouncer is + failing to connect to PostgreSQL, it may be that your PgBouncer node's IP + address is missing from PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` + on your database nodes. Before proceeding, see + [PgBouncer error `ERROR: pgbouncer cannot connect to server`](troubleshooting.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server). + +1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -1920,11 +1887,10 @@ The following IP will be used as an example: To configure the Monitoring node: -1. SSH into the Monitoring node. -1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab - package you want using **steps 1 and 2** from the GitLab downloads page. - Do not complete any other steps on the download page. - +1. SSH in to the Monitoring node. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to follow _only_ installation steps 1 and 2 + on the page. 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace the file of the same name on this server. If that file is not on this server, add the file from your Consul server to this server. @@ -1988,28 +1954,44 @@ It's recommended over [NFS](#configure-nfs-optional) and in general it's better in larger setups as object storage is typically much more performant, reliable, and scalable. -Object storage options that GitLab has tested, or is aware of customers using include: +GitLab has been tested on a number of object storage providers: -- SaaS/Cloud solutions such as [Amazon S3](https://aws.amazon.com/s3/), [Google cloud storage](https://cloud.google.com/storage). +- [Amazon S3](https://aws.amazon.com/s3/) +- [Google Cloud Storage](https://cloud.google.com/storage) +- [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces/) +- [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) +- [Openstack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) +- [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) - On-premises hardware and appliances from various storage vendors. -- MinIO. There is [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. - -For configuring GitLab to use Object Storage refer to the following guides -based on what features you intend to use: - -1. Configure [object storage for backups](../../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage). -1. Configure [object storage for job artifacts](../job_artifacts.md#using-object-storage) - including [incremental logging](../job_logs.md#new-incremental-logging-architecture). -1. Configure [object storage for LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage). -1. Configure [object storage for uploads](../uploads.md#using-object-storage). -1. Configure [object storage for merge request diffs](../merge_request_diffs.md#using-object-storage). -1. Configure [object storage for Container Registry](../packages/container_registry.md#use-object-storage) (optional feature). -1. Configure [object storage for Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage) (optional feature). -1. Configure [object storage for packages](../packages/index.md#using-object-storage) (optional feature). **(PREMIUM ONLY)** -1. Configure [object storage for Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature). **(PREMIUM ONLY)** -1. Configure [object storage for Pseudonymizer](../pseudonymizer.md#configuration) (optional feature). **(ULTIMATE ONLY)** -1. Configure [object storage for autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional - for improved performance). -1. Configure [object storage for Terraform state files](../terraform_state.md#using-object-storage). +- MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. + +There are two ways of specifying object storage configuration in GitLab: + +- [Consolidated form](../object_storage.md#consolidated-object-storage-configuration): A single credential is + shared by all supported object types. +- [Storage-specific form](../object_storage.md#storage-specific-configuration): Every object defines its + own object storage [connection and configuration](../object_storage.md#connection-settings). + +Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. + +For configuring object storage in GitLab 13.1 and earlier, or for storage types not +supported by consolidated configuration form, refer to the following guides based +on what features you intend to use: + +|Object storage type|Supported by consolidated configuration?| +|-------------------|----------------------------------------| +| [Backups](../../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage)|No| +| [Job artifacts](../job_artifacts.md#using-object-storage) including archived job logs | Yes | +| [LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage) | Yes | +| [Uploads](../uploads.md#using-object-storage) | Yes | +| [Container Registry](../packages/container_registry.md#use-object-storage) (optional feature) | No | +| [Merge request diffs](../merge_request_diffs.md#using-object-storage) | Yes | +| [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | +| [Packages](../packages/index.md#using-object-storage) (optional feature) | Yes | +| [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature) **(PREMIUM ONLY)** | Yes | +| [Pseudonymizer](../pseudonymizer.md#configuration) (optional feature) **(ULTIMATE ONLY)** | No | +| [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | +| [Terraform state files](../terraform_state.md#using-object-storage) | Yes | Using separate buckets for each data type is the recommended approach for GitLab. @@ -2033,13 +2015,29 @@ work. </a> </div> +## Configure Advanced Search **(STARTER ONLY)** + +You can leverage Elasticsearch and [enable Advanced Search](../../integration/elasticsearch.md) +for faster, more advanced code search across your entire GitLab instance. + +Elasticsearch cluster design and requirements are dependent on your specific +data. For recommended best practices about how to set up your Elasticsearch +cluster alongside your instance, read how to +[choose the optimal cluster configuration](../../integration/elasticsearch.md#guidance-on-choosing-optimal-cluster-configuration). + +<div align="right"> + <a type="button" class="btn btn-default" href="#setup-components"> + Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> + </a> +</div> + ## Configure NFS (optional) [Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly) are recommended over NFS wherever possible for improved performance. If you intend to use GitLab Pages, this currently [requires NFS](troubleshooting.md#gitlab-pages-requires-nfs). -See how to [configure NFS](../high_availability/nfs.md). +See how to [configure NFS](../nfs.md). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index 34b90964fbf..4863329b695 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -17,14 +17,14 @@ For a full list of reference architectures, see | Service | Nodes | Configuration | GCP | AWS | Azure | |------------------------------------------|--------|-------------------------|----------------|--------------|---------| -| Load balancer | 1 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| PostgreSQL | 1 | 2 vCPU, 7.5GB memory | n1-standard-2 | m5.large | D2s v3 | -| Redis | 1 | 1 vCPU, 3.75GB memory | n1-standard-1 | m5.large | D2s v3 | -| Gitaly | 1 | 4 vCPU, 15GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| GitLab Rails | 2 | 8 vCPU, 7.2GB memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | -| Monitoring node | 1 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Load balancer | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| PostgreSQL | 1 | 2 vCPU, 7.5 GB memory | n1-standard-2 | m5.large | D2s v3 | +| Redis | 1 | 1 vCPU, 3.75 GB memory | n1-standard-1 | m5.large | D2s v3 | +| Gitaly | 1 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| GitLab Rails | 2 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | +| Monitoring node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | | Object storage | n/a | n/a | n/a | n/a | n/a | -| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | The Google Cloud Platform (GCP) architectures were built and tested using the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) @@ -43,7 +43,7 @@ doesn't require you to provision and maintain a node. To set up GitLab and its components to accommodate up to 2,000 users: 1. [Configure the external load balancing node](#configure-the-external-load-balancer) - to handle the load balancing of the two GitLab application services nodes. + to handle the load balancing of the GitLab application services nodes. 1. [Configure PostgreSQL](#configure-postgresql), the database for GitLab. 1. [Configure Redis](#configure-redis). 1. [Configure Gitaly](#configure-gitaly), which provides access to the Git @@ -55,6 +55,8 @@ To set up GitLab and its components to accommodate up to 2,000 users: environment. 1. [Configure the object storage](#configure-the-object-storage) used for shared data objects. +1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, + more advanced code search across your entire GitLab instance. 1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended) to have shared disk storage service as an alternative to Gitaly or object storage. You can skip this step if you're not using GitLab Pages (which @@ -62,18 +64,17 @@ To set up GitLab and its components to accommodate up to 2,000 users: ## Configure the external load balancer -NOTE: **Note:** -This architecture has been tested and validated with [HAProxy](https://www.haproxy.org/). -Although you can use a load balancer with a similar set of features, GitLab -hasn't validated other load balancers. - In an active/active GitLab configuration, you'll need a load balancer to route -traffic to the application servers. The specifics for which load balancer to -use or its exact configuration is out of scope for the GitLab documentation. -If you're managing multi-node systems (including GitLab) you'll probably -already have a load balancer of choice. Some examples including HAProxy -(open-source), F5 Big-IP LTM, and Citrix Net Scaler. This documentation -includes the ports and protocols for use with GitLab. +traffic to the application servers. The specifics on which load balancer to use +or its exact configuration is beyond the scope of GitLab documentation. We hope +that if you're managing multi-node systems like GitLab, you already have a load +balancer of choice. Some load balancer examples include HAProxy (open-source), +F5 Big-IP LTM, and Citrix Net Scaler. This documentation outline the ports and +protocols needed for use with GitLab. + +This architecture has been tested and validated with [HAProxy](https://www.haproxy.org/) +as the load balancer. Although other load balancers with similar feature sets +could also be used, those load balancers have not been validated. The next question is how you will handle SSL in your environment. There are several different options: @@ -206,10 +207,10 @@ further configuration steps. ### Standalone PostgreSQL using Omnibus GitLab -1. SSH into the PostgreSQL server. -1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab - package you want using **steps 1 and 2** from the GitLab downloads page. - - Do not complete any other steps on the download page. +1. SSH in to the PostgreSQL server. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to follow _only_ installation steps 1 and 2 + on the page. 1. Generate a password hash for PostgreSQL. This assumes you will use the default username of `gitlab` (recommended). The command will request a password and confirmation. Use the value that is output by this command in the next @@ -300,11 +301,10 @@ The Omnibus GitLab package can be used to configure a standalone Redis server. The steps below are the minimum necessary to configure a Redis server with Omnibus: -1. SSH into the Redis server. -1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab - package you want using **steps 1 and 2** from the GitLab downloads page. - - Do not complete any other steps on the download page. - +1. SSH in to the Redis server. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to follow _only_ installation steps 1 and 2 + on the page. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby @@ -393,12 +393,11 @@ The following procedure describes how to configure a single Gitaly server named `gitaly1.internal` with the secret token `gitalysecret`. We assume your GitLab installation has two repository storages: `default` and `storage1`. -To configure the Gitaly server: +To configure the Gitaly server, on the server node you want to use for Gitaly: -1. On the server node you want to use for Gitaly, - [download and install](https://about.gitlab.com/install/) your selected - Omnibus GitLab package using *steps 1 and 2* from the GitLab downloads page, - but *without* providing the `EXTERNAL_URL` value. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to follow _only_ installation steps 1 and 2 + on the page, and _do not_ provide the `EXTERNAL_URL` value. 1. Edit the Gitaly server node's `/etc/gitlab/gitlab.rb` file to configure storage paths, enable the network listener, and to configure the token: @@ -464,7 +463,7 @@ To configure the Gitaly server: 1. Confirm that Gitaly can perform callbacks to the internal API: ```shell - sudo /opt/gitlab/embedded/service/gitlab-shell/bin/check -config /opt/gitlab/embedded/service/gitlab-shell/config.yml + sudo /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml ``` ### Gitaly TLS support @@ -487,11 +486,10 @@ Name. If you are addressing the Gitaly server by its IP address, you must add it as a Subject Alternative Name to the certificate. [gRPC does not support using an IP address as Common Name in a certificate](https://github.com/grpc/grpc/issues/2691). -NOTE: **Note:** -It is possible to configure Gitaly servers with both an -unencrypted listening address `listen_addr` and an encrypted listening -address `tls_listen_addr` at the same time. This allows you to do a -gradual transition from unencrypted to encrypted traffic, if necessary. +It's possible to configure Gitaly servers with both an unencrypted listening +address (`listen_addr`) and an encrypted listening address (`tls_listen_addr`) +at the same time. This allows you to do a gradual transition from unencrypted to +encrypted traffic, if necessary. To configure Gitaly with TLS: @@ -535,14 +533,14 @@ To configure Gitaly with TLS: ## Configure GitLab Rails -NOTE: **Note:** -In our architectures we run each GitLab Rails node using the Puma webserver -and have its number of workers set to 90% of available CPUs along with four threads. For -nodes that are running Rails with other components the worker value should be reduced -accordingly where we've found 50% achieves a good balance but this is dependent -on workload. - This section describes how to configure the GitLab application (Rails) component. + +In our architecture, we run each GitLab Rails node using the Puma webserver, and +have its number of workers set to 90% of available CPUs, with four threads. For +nodes running Rails with other components, the worker value should be reduced +accordingly. We've determined that a worker value of 50% achieves a good balance, +but this is dependent on workload. + On each node perform the following: 1. If you're [using NFS](#configure-nfs-optional): @@ -570,10 +568,10 @@ On each node perform the following: mkdir -p /var/opt/gitlab/.ssh /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/git-data ``` -1. Download/install Omnibus GitLab using **steps 1 and 2** from - [GitLab downloads](https://about.gitlab.com/install/). Do not complete other - steps on the download page. -1. Create/edit `/etc/gitlab/gitlab.rb` and use the following configuration. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to follow _only_ installation steps 1 and 2 + on the page. +1. Create or edit `/etc/gitlab/gitlab.rb` and use the following configuration. To maintain uniformity of links across nodes, the `external_url` on the application server should point to the external URL that users will use to access GitLab. This would be the URL of the [load balancer](#configure-the-external-load-balancer) @@ -664,12 +662,33 @@ On each node perform the following: sudo gitlab-ctl tail gitaly ``` -NOTE: **Note:** -When you specify `https` in the `external_url`, as in the example -above, GitLab assumes you have SSL certificates in `/etc/gitlab/ssl/`. If -certificates are not present, NGINX will fail to start. See the -[NGINX documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) -for more information. +1. Save the `/etc/gitlab/gitlab-secrets.json` file from one of the two + application nodes and install it on the other application node and the + [Gitaly node](#configure-gitaly) and + [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +When you specify `https` in the `external_url`, as in the previous example, +GitLab expects that the SSL certificates are in `/etc/gitlab/ssl/`. If the +certificates aren't present, NGINX will fail to start. For more information, see +the [NGINX documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https). + +### GitLab Rails post-configuration + +1. Designate one application node for running database migrations during + installation and updates. Initialize the GitLab database and ensure all + migrations ran: + + ```shell + sudo gitlab-rake gitlab:db:configure + ``` + + If you encounter a `rake aborted!` error message stating that PgBouncer is + failing to connect to PostgreSQL, it may be that your PgBouncer node's IP + address is missing from PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` + on your database nodes. Before proceeding, see + [PgBouncer error `ERROR: pgbouncer cannot connect to server`](troubleshooting.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server). + +1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -683,10 +702,10 @@ The Omnibus GitLab package can be used to configure a standalone Monitoring node running [Prometheus](../monitoring/prometheus/index.md) and [Grafana](../monitoring/performance/grafana_configuration.md): -1. SSH into the Monitoring node. -1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab - package you want using **steps 1 and 2** from the GitLab downloads page. - Do not complete any other steps on the download page. +1. SSH in to the Monitoring node. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to follow _only_ installation steps 1 and 2 + on the page. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby @@ -806,30 +825,44 @@ data, and is recommended over [NFS](#configure-nfs-optional). In general, object storage services are better for larger environments, as object storage is typically much more performant, reliable, and scalable. -Object storage options that GitLab has either tested or is aware of customers -using, includes: - -- SaaS/Cloud solutions (such as [Amazon S3](https://aws.amazon.com/s3/) or - [Google Cloud Storage](https://cloud.google.com/storage)). -- On-premises hardware and appliances, from various storage vendors. -- MinIO ([Deployment guide](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html)). - -To configure GitLab to use object storage, refer to the following guides based -on the features you intend to use: - -1. [Object storage for backups](../../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage). -1. [Object storage for job artifacts](../job_artifacts.md#using-object-storage) - including [incremental logging](../job_logs.md#new-incremental-logging-architecture). -1. [Object storage for LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage). -1. [Object storage for uploads](../uploads.md#using-object-storage). -1. [Object storage for merge request diffs](../merge_request_diffs.md#using-object-storage). -1. [Object storage for Container Registry](../packages/container_registry.md#use-object-storage) (optional feature). -1. [Object storage for Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage) (optional feature). -1. [Object storage for packages](../packages/index.md#using-object-storage) (optional feature). **(PREMIUM ONLY)** -1. [Object storage for Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature). **(PREMIUM ONLY)** -1. [Object storage for Pseudonymizer](../pseudonymizer.md#configuration) (optional feature). **(ULTIMATE ONLY)** -1. [Object storage for autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional, for improved performance). -1. [Object storage for Terraform state files](../terraform_state.md#using-object-storage). +GitLab has been tested on a number of object storage providers: + +- [Amazon S3](https://aws.amazon.com/s3/) +- [Google Cloud Storage](https://cloud.google.com/storage) +- [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces/) +- [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) +- [Openstack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) +- [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) +- On-premises hardware and appliances from various storage vendors. +- MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. + +There are two ways of specifying object storage configuration in GitLab: + +- [Consolidated form](../object_storage.md#consolidated-object-storage-configuration): A single credential is + shared by all supported object types. +- [Storage-specific form](../object_storage.md#storage-specific-configuration): Every object defines its + own object storage [connection and configuration](../object_storage.md#connection-settings). + +Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. + +For configuring object storage in GitLab 13.1 and earlier, or for storage types not +supported by consolidated configuration form, refer to the following guides based +on what features you intend to use: + +|Object storage type|Supported by consolidated configuration?| +|-------------------|----------------------------------------| +| [Backups](../../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage)|No| +| [Job artifacts](../job_artifacts.md#using-object-storage) including archived job logs | Yes | +| [LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage) | Yes | +| [Uploads](../uploads.md#using-object-storage) | Yes | +| [Container Registry](../packages/container_registry.md#use-object-storage) (optional feature) | No | +| [Merge request diffs](../merge_request_diffs.md#using-object-storage) | Yes | +| [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | +| [Packages](../packages/index.md#using-object-storage) (optional feature) | Yes | +| [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature) **(PREMIUM ONLY)** | Yes | +| [Pseudonymizer](../pseudonymizer.md#configuration) (optional feature) **(ULTIMATE ONLY)** | No | +| [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | +| [Terraform state files](../terraform_state.md#using-object-storage) | Yes | Using separate buckets for each data type is the recommended approach for GitLab. @@ -851,6 +884,22 @@ functioning backups is encountered. </a> </div> +## Configure Advanced Search **(STARTER ONLY)** + +You can leverage Elasticsearch and [enable Advanced Search](../../integration/elasticsearch.md) +for faster, more advanced code search across your entire GitLab instance. + +Elasticsearch cluster design and requirements are dependent on your specific +data. For recommended best practices about how to set up your Elasticsearch +cluster alongside your instance, read how to +[choose the optimal cluster configuration](../../integration/elasticsearch.md#guidance-on-choosing-optimal-cluster-configuration). + +<div align="right"> + <a type="button" class="btn btn-default" href="#setup-components"> + Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> + </a> +</div> + ## Configure NFS (optional) For improved performance, [object storage](#configure-the-object-storage), diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index be944586e43..70d0cae6dfd 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -24,18 +24,18 @@ costly-to-operate environment by using the | Service | Nodes | Configuration | GCP | AWS | Azure | |--------------------------------------------|-------------|-----------------------|----------------|-------------|---------| -| External load balancing node | 1 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Redis | 3 | 2 vCPU, 7.5GB memory | n1-standard-2 | m5.large | D2s v3 | -| Consul + Sentinel | 3 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| PostgreSQL | 3 | 2 vCPU, 7.5GB memory | n1-standard-2 | m5.large | D2s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Internal load balancing node | 1 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Gitaly | 2 (minimum) | 4 vCPU, 15GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Sidekiq | 4 | 2 vCPU, 7.5GB memory | n1-standard-2 | m5.large | D2s v3 | -| GitLab Rails | 3 | 8 vCPU, 7.2GB memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | -| Monitoring node | 1 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Object Storage | n/a | n/a | n/a | n/a | n/a | -| NFS Server (optional, not recommended) | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| External load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Redis | 3 | 2 vCPU, 7.5 GB memory | n1-standard-2 | m5.large | D2s v3 | +| Consul + Sentinel | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| PostgreSQL | 3 | 2 vCPU, 7.5 GB memory | n1-standard-2 | m5.large | D2s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Gitaly | 2 (minimum) | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| Sidekiq | 4 | 2 vCPU, 7.5 GB memory | n1-standard-2 | m5.large | D2s v3 | +| GitLab Rails | 3 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | +| Monitoring node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Object storage | n/a | n/a | n/a | n/a | n/a | +| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | The Google Cloud Platform (GCP) architectures were built and tested using the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) @@ -44,41 +44,43 @@ or higher, are required for your CPU or node counts. For more information, see our [Sysbench](https://github.com/akopytov/sysbench)-based [CPU benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). -For data objects (such as LFS, Uploads, or Artifacts), an -[object storage service](#configure-the-object-storage) is recommended instead -of NFS where possible, due to better performance and availability. Since this -doesn't require a node to be set up, *Object Storage* is noted as not -applicable (n/a) in the previous table. +Due to better performance and availability, for data objects (such as LFS, +uploads, or artifacts), using an [object storage service](#configure-the-object-storage) +is recommended instead of using NFS. Using an object storage service also +doesn't require you to provision and maintain a node. ## Setup components To set up GitLab and its components to accommodate up to 3,000 users: 1. [Configure the external load balancing node](#configure-the-external-load-balancer) - that will handle the load balancing of the two GitLab application services nodes. + to handle the load balancing of the GitLab application services nodes. 1. [Configure Redis](#configure-redis). 1. [Configure Consul and Sentinel](#configure-consul-and-sentinel). 1. [Configure PostgreSQL](#configure-postgresql), the database for GitLab. 1. [Configure PgBouncer](#configure-pgbouncer). -1. [Configure the internal load balancing node](#configure-the-internal-load-balancer) +1. [Configure the internal load balancing node](#configure-the-internal-load-balancer). 1. [Configure Gitaly](#configure-gitaly), which provides access to the Git repositories. 1. [Configure Sidekiq](#configure-sidekiq). 1. [Configure the main GitLab Rails application](#configure-gitlab-rails) - to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend requests (UI, API, Git - over HTTP/SSH). -1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab environment. -1. [Configure the Object Storage](#configure-the-object-storage) + to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend + requests (which include UI, API, and Git over HTTP/SSH). +1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab + environment. +1. [Configure the object storage](#configure-the-object-storage) used for shared data objects. -1. [Configure NFS (Optional)](#configure-nfs-optional) - to have shared disk storage service as an alternative to Gitaly and/or Object Storage (although - not recommended). NFS is required for GitLab Pages, you can skip this step if you're not using - that feature. +1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, + more advanced code search across your entire GitLab instance. +1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended) + to have shared disk storage service as an alternative to Gitaly or object + storage. You can skip this step if you're not using GitLab Pages (which + requires NFS). -We start with all servers on the same 10.6.0.0/16 private network range, they -can connect to each other freely on those addresses. +The servers start on the same 10.6.0.0/24 private network range, and can +connect to each other freely on these addresses. -Here is a list and description of each machine and the assigned IP: +The following list includes descriptions of each server and its assigned IP: - `10.6.0.10`: External Load Balancer - `10.6.0.61`: Redis Primary @@ -107,19 +109,18 @@ Here is a list and description of each machine and the assigned IP: ## Configure the external load balancer -NOTE: **Note:** +In an active/active GitLab configuration, you'll need a load balancer to route +traffic to the application servers. The specifics on which load balancer to use +or its exact configuration is beyond the scope of GitLab documentation. We hope +that if you're managing multi-node systems like GitLab, you already have a load +balancer of choice. Some load balancer examples include HAProxy (open-source), +F5 Big-IP LTM, and Citrix Net Scaler. This documentation outline the ports and +protocols needed for use with GitLab. + This architecture has been tested and validated with [HAProxy](https://www.haproxy.org/) as the load balancer. Although other load balancers with similar feature sets could also be used, those load balancers have not been validated. -In an active/active GitLab configuration, you will need a load balancer to route -traffic to the application servers. The specifics on which load balancer to use -or the exact configuration is beyond the scope of GitLab documentation. We hope -that if you're managing multi-node systems like GitLab you have a load balancer of -choice already. Some examples including HAProxy (open-source), F5 Big-IP LTM, -and Citrix Net Scaler. This documentation will outline what ports and protocols -you need to use with GitLab. - The next question is how you will handle SSL in your environment. There are several different options: @@ -277,20 +278,17 @@ The requirements for a Redis setup are the following: ([Internet](https://gitlab.com/gitlab-org/gitlab-foss/uploads/c4cc8cd353604bd80315f9384035ff9e/The_Internet_IT_Crowd.png)), using a firewall. -NOTE: **Note:** -Redis nodes (both primary and replica) will need the same password defined in -`redis['password']`. At any time during a failover the Sentinels can -reconfigure a node and change its status from primary to replica and vice versa. +Both the primary and replica Redis nodes need the same password defined in +`redis['password']`. At any time during a failover, the Sentinels can reconfigure +a node and change its status from primary to replica (and vice versa). #### Configuring the primary Redis instance -1. SSH into the **Primary** Redis server. -1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab - package you want using **steps 1 and 2** from the GitLab downloads page. - - Make sure you select the correct Omnibus package, with the same version - and type (Community, Enterprise editions) of your current install. - - Do not complete any other steps on the download page. - +1. SSH in to the **Primary** Redis server. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to both follow _only_ installation steps 1 and 2 + on the page, and to select the correct Omnibus GitLab package, with the same version + and type (Community or Enterprise editions) as your current install. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby @@ -334,18 +332,17 @@ reconfigure a node and change its status from primary to replica and vice versa. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -NOTE: **Note:** -You can specify multiple roles like sentinel and Redis as: -`roles ['redis_sentinel_role', 'redis_master_role']`. -Read more about [roles](https://docs.gitlab.com/omnibus/roles/). +You can specify multiple roles, like sentinel and Redis, as: +`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +[roles](https://docs.gitlab.com/omnibus/roles/). -You can list the current Redis Primary, Replica status via: +You can list the current Redis Primary, Replica status by using: ```shell /opt/gitlab/embedded/bin/redis-cli -h <host> -a 'redis-password-goes-here' info replication ``` -Show running GitLab services via: +Show running GitLab services by using: ```shell gitlab-ctl status @@ -363,13 +360,11 @@ run: redis-exporter: (pid 30075) 76861s; run: log: (pid 29674) 76896s #### Configuring the replica Redis instances -1. SSH into the **replica** Redis server. -1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab - package you want using **steps 1 and 2** from the GitLab downloads page. - - Make sure you select the correct Omnibus package, with the same version - and type (Community, Enterprise editions) of your current install. - - Do not complete any other steps on the download page. - +1. SSH in to the **replica** Redis server. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to both follow _only_ installation steps 1 and 2 + on the page, and to select the correct Omnibus GitLab package, with the same version + and type (Community or Enterprise editions) as your current install. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby @@ -422,10 +417,9 @@ run: redis-exporter: (pid 30075) 76861s; run: log: (pid 29674) 76896s 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. -NOTE: **Note:** -You can specify multiple roles like sentinel and Redis as: -`roles ['redis_sentinel_role', 'redis_master_role']`. -Read more about [roles](https://docs.gitlab.com/omnibus/roles/). +You can specify multiple roles, like sentinel and Redis, as: +`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +[roles](https://docs.gitlab.com/omnibus/roles/). These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after a failover, as the nodes will be managed by the [Sentinels](#configure-consul-and-sentinel), and even after a @@ -443,13 +437,6 @@ are supported and can be added if needed. ## Configure Consul and Sentinel -NOTE: **Note:** -If you are using an external Redis Sentinel instance, be sure -to exclude the `requirepass` parameter from the Sentinel -configuration. This parameter will cause clients to report `NOAUTH -Authentication required.`. [Redis Sentinel 3.2.x does not support -password authentication](https://github.com/antirez/redis/issues/3279). - Now that the Redis servers are all set up, let's configure the Sentinel servers. The following IPs will be used as an example: @@ -457,16 +444,19 @@ servers. The following IPs will be used as an example: - `10.6.0.12`: Consul/Sentinel 2 - `10.6.0.13`: Consul/Sentinel 3 -To configure the Sentinel: +NOTE: **Note:** +If you're using an external Redis Sentinel instance, be sure to exclude the +`requirepass` parameter from the Sentinel configuration. This parameter causes +clients to report `NOAUTH Authentication required.`. +[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279). -1. SSH into the server that will host Consul/Sentinel. -1. [Download/install](https://about.gitlab.com/install/) the - Omnibus GitLab Enterprise Edition package using **steps 1 and 2** from the - GitLab downloads page. - - Make sure you select the correct Omnibus package, with the same version - the GitLab application is running. - - Do not complete any other steps on the download page. +To configure the Sentinel: +1. SSH in to the server that will host Consul/Sentinel. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to both follow _only_ installation steps 1 and 2 + on the page, and to select the correct Omnibus GitLab package, with the same version + and type (Community or Enterprise editions) as your current install. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby @@ -556,10 +546,9 @@ To configure the Sentinel: 1. Go through the steps again for all the other Consul/Sentinel nodes, and make sure you set up the correct IPs. -NOTE: **Note:** -A Consul leader will be elected when the provisioning of the third Consul server is completed. -Viewing the Consul logs `sudo gitlab-ctl tail consul` will display -`...[INFO] consul: New leader elected: ...` +A Consul leader is _elected_ when the provisioning of the third Consul server is +complete. Viewing the Consul logs `sudo gitlab-ctl tail consul` displays +`...[INFO] consul: New leader elected: ...`. You can list the current Consul members (server, client): @@ -627,7 +616,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. #### PostgreSQL primary node -1. SSH into the PostgreSQL primary node. +1. SSH in to the PostgreSQL primary node. 1. Generate a password hash for the PostgreSQL username/password pair. This assumes you will use the default username of `gitlab` (recommended). The command will request a password and confirmation. Use the value that is output by this command in the next @@ -812,7 +801,7 @@ are supported and can be added if needed. #### PostgreSQL post-configuration -SSH into the **primary node**: +SSH in to the **primary node**: 1. Open a database prompt: @@ -848,7 +837,7 @@ SSH into the **primary node**: is not an IP address, it will need to be a resolvable name (via DNS or `/etc/hosts`) -SSH into the **secondary node**: +SSH in to the **secondary node**: 1. Set up the repmgr standby: @@ -1069,51 +1058,48 @@ Refer to your preferred Load Balancer's documentation for further guidance. ## Configure Gitaly -Deploying Gitaly in its own server can benefit GitLab installations that are -larger than a single machine. - -The Gitaly node requirements are dependent on customer data, specifically the number of -projects and their repository sizes. Two nodes are recommended as an absolute minimum. -Each Gitaly node should store no more than 5TB of data and have the number of -[`gitaly-ruby` workers](../gitaly/index.md#gitaly-ruby) set to 20% of available CPUs. -Additional nodes should be considered in conjunction with a review of expected -data size and spread based on the recommendations above. - -It is also strongly recommended that all Gitaly nodes be set up with SSD disks with -a throughput of at least 8,000 IOPS for read operations and 2,000 IOPS for write, -as Gitaly has heavy I/O. These IOPS values are recommended only as a starter as with -time they may be adjusted higher or lower depending on the scale of your environment's workload. -If you're running the environment on a Cloud provider, you may need to refer to -their documentation on how to configure IOPS correctly. - -Some things to note: - -- The GitLab Rails application shards repositories into [repository storages](../repository_storage_paths.md). -- A Gitaly server can host one or more storages. -- A GitLab server can use one or more Gitaly servers. -- Gitaly addresses must be specified in such a way that they resolve - correctly for ALL Gitaly clients. +[Gitaly](../gitaly/index.md) server node requirements are dependent on data, +specifically the number of projects and those projects' sizes. It's recommended +that a Gitaly server node stores no more than 5 TB of data. Although this +reference architecture includes a recommendation for the number of Gitaly server +nodes to use, depending on your storage requirements, you may require additional +Gitaly server nodes. + +Due to Gitaly having notable input and output requirements, we strongly +recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs should +have a throughput of at least 8,000 input/output operations per second (IOPS) +for read operations and 2,000 IOPS for write operations. These IOPS values are +initial recommendations, and may be adjusted to greater or lesser values +depending on the scale of your environment's workload. If you're running the +environment on a Cloud provider, refer to their documentation about how to +configure IOPS correctly. + +Be sure to note the following items: + +- The GitLab Rails application shards repositories into + [repository storage paths](../repository_storage_paths.md). +- A Gitaly server can host one or more storage paths. +- A GitLab server can use one or more Gitaly server nodes. +- Gitaly addresses must be specified to be correctly resolvable for _all_ + Gitaly clients. - Gitaly servers must not be exposed to the public internet, as Gitaly's network traffic is unencrypted by default. The use of a firewall is highly recommended to restrict access to the Gitaly server. Another option is to [use TLS](#gitaly-tls-support). -TIP: **Tip:** -For more information about Gitaly's history and network architecture see the -[standalone Gitaly documentation](../gitaly/index.md). - -Note: **Note:** The token referred to throughout the Gitaly documentation is -just an arbitrary password selected by the administrator. It is unrelated to -tokens created for the GitLab API or other similar web API tokens. +NOTE: **Note:** +The token referred to throughout the Gitaly documentation is an arbitrary +password selected by the administrator. This token is unrelated to tokens +created for the GitLab API or other similar web API tokens. -Below we describe how to configure two Gitaly servers, with IPs and -domain names: +This section describes how to configure two Gitaly servers, with the following +IPs and domain names: - `10.6.0.51`: Gitaly 1 (`gitaly1.internal`) - `10.6.0.52`: Gitaly 2 (`gitaly2.internal`) -The secret token is assumed to be `gitalysecret` and that -your GitLab installation has three repository storages: +Assumptions about your servers include having the secret token be `gitalysecret`, +and that your GitLab installation has three repository storages: - `default` on Gitaly 1 - `storage1` on Gitaly 1 @@ -1121,11 +1107,11 @@ your GitLab installation has three repository storages: On each node: -1. [Download/Install](https://about.gitlab.com/install/) the Omnibus GitLab - package you want using **steps 1 and 2** from the GitLab downloads page but - **without** providing the `EXTERNAL_URL` value. -1. Edit `/etc/gitlab/gitlab.rb` to configure storage paths, enable - the network listener and configure the token: +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to follow _only_ installation steps 1 and 2 + on the page, and _do not_ provide the `EXTERNAL_URL` value. +1. Edit `/etc/gitlab/gitlab.rb` to configure the storage paths, enable + the network listener, and configure the token: <!-- updates to following example must also be made at @@ -1189,39 +1175,39 @@ On each node: ``` 1. Append the following to `/etc/gitlab/gitlab.rb` for each respective server: - 1. On `gitaly1.internal`: - - ```ruby - git_data_dirs({ - 'default' => { - 'path' => '/var/opt/gitlab/git-data' - }, - 'storage1' => { - 'path' => '/mnt/gitlab/git-data' - }, - }) - ``` - - 1. On `gitaly2.internal`: - - ```ruby - git_data_dirs({ - 'storage2' => { - 'path' => '/mnt/gitlab/git-data' - }, - }) - ``` + - On `gitaly1.internal`: + + ```ruby + git_data_dirs({ + 'default' => { + 'path' => '/var/opt/gitlab/git-data' + }, + 'storage1' => { + 'path' => '/mnt/gitlab/git-data' + }, + }) + ``` + + - On `gitaly2.internal`: + + ```ruby + git_data_dirs({ + 'storage2' => { + 'path' => '/mnt/gitlab/git-data' + }, + }) + ``` <!-- updates to following example must also be made at https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab --> -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). 1. Confirm that Gitaly can perform callbacks to the internal API: ```shell - sudo /opt/gitlab/embedded/service/gitlab-shell/bin/check -config /opt/gitlab/embedded/service/gitlab-shell/config.yml + sudo /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml ``` 1. Verify the GitLab services are running: @@ -1259,11 +1245,10 @@ Name. If you are addressing the Gitaly server by its IP address, you must add it as a Subject Alternative Name to the certificate. [gRPC does not support using an IP address as Common Name in a certificate](https://github.com/grpc/grpc/issues/2691). -NOTE: **Note:** -It is possible to configure Gitaly servers with both an -unencrypted listening address `listen_addr` and an encrypted listening -address `tls_listen_addr` at the same time. This allows you to do a -gradual transition from unencrypted to encrypted traffic, if necessary. +It's possible to configure Gitaly servers with both an unencrypted listening +address (`listen_addr`) and an encrypted listening address (`tls_listen_addr`) +at the same time. This allows you to do a gradual transition from unencrypted to +encrypted traffic, if necessary. To configure Gitaly with TLS: @@ -1317,10 +1302,10 @@ The following IPs will be used as an example: To configure the Sidekiq nodes, one each one: -1. SSH into the Sidekiq server. -1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab package -you want using steps 1 and 2 from the GitLab downloads page. -**Do not complete any other steps on the download page.** +1. SSH in to the Sidekiq server. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to follow _only_ installation steps 1 and 2 + on the page. 1. Open `/etc/gitlab/gitlab.rb` with your editor: ```ruby @@ -1431,14 +1416,14 @@ You can also run [multiple Sidekiq processes](../operations/extra_sidekiq_proces ## Configure GitLab Rails -NOTE: **Note:** -In our architectures we run each GitLab Rails node using the Puma webserver -and have its number of workers set to 90% of available CPUs along with four threads. For -nodes that are running Rails with other components the worker value should be reduced -accordingly where we've found 50% achieves a good balance but this is dependent -on workload. - This section describes how to configure the GitLab application (Rails) component. + +In our architecture, we run each GitLab Rails node using the Puma webserver, and +have its number of workers set to 90% of available CPUs, with four threads. For +nodes running Rails with other components, the worker value should be reduced +accordingly. We've determined that a worker value of 50% achieves a good balance, +but this is dependent on workload. + On each node perform the following: 1. If you're [using NFS](#configure-nfs-optional): @@ -1466,10 +1451,10 @@ On each node perform the following: mkdir -p /var/opt/gitlab/.ssh /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/git-data ``` -1. Download/install Omnibus GitLab using **steps 1 and 2** from - [GitLab downloads](https://about.gitlab.com/install/). Do not complete other - steps on the download page. -1. Create/edit `/etc/gitlab/gitlab.rb` and use the following configuration. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to follow _only_ installation steps 1 and 2 + on the page. +1. Create or edit `/etc/gitlab/gitlab.rb` and use the following configuration. To maintain uniformity of links across nodes, the `external_url` on the application server should point to the external URL that users will use to access GitLab. This would be the URL of the [external load balancer](#configure-the-external-load-balancer) @@ -1582,6 +1567,11 @@ On each node perform the following: sudo gitlab-ctl tail gitaly ``` +1. Save the `/etc/gitlab/gitlab-secrets.json` file from one of the two + application nodes and install it on the other application node, the + [Gitaly node](#configure-gitaly) and the [Sidekiq node](#configure-sidekiq) and + [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + 1. Verify the GitLab services are running: ```shell @@ -1600,12 +1590,10 @@ On each node perform the following: run: puma: (pid 4936) 8645s; run: log: (pid 29656) 79161s ``` -NOTE: **Note:** -When you specify `https` in the `external_url`, as in the example -above, GitLab assumes you have SSL certificates in `/etc/gitlab/ssl/`. If -certificates are not present, NGINX will fail to start. See the -[NGINX documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) -for more information. +When you specify `https` in the `external_url`, as in the previous example, +GitLab expects that the SSL certificates are in `/etc/gitlab/ssl/`. If the +certificates aren't present, NGINX will fail to start. For more information, see +the [NGINX documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https). ### GitLab Rails post-configuration @@ -1615,12 +1603,11 @@ for more information. gitlab-rake gitlab:db:configure ``` - NOTE: **Note:** - If you encounter a `rake aborted!` error stating that PgBouncer is failing to connect to - PostgreSQL it may be that your PgBouncer node's IP address is missing from - PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` on your database nodes. See - [PgBouncer error `ERROR: pgbouncer cannot connect to server`](troubleshooting.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) - in the Troubleshooting section before proceeding. + If you encounter a `rake aborted!` error message stating that PgBouncer is + failing to connect to PostgreSQL, it may be that your PgBouncer node's IP + address is missing from PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` + on your database nodes. Before proceeding, see + [PgBouncer error `ERROR: pgbouncer cannot connect to server`](troubleshooting.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server). 1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md). @@ -1636,10 +1623,10 @@ The Omnibus GitLab package can be used to configure a standalone Monitoring node running [Prometheus](../monitoring/prometheus/index.md) and [Grafana](../monitoring/performance/grafana_configuration.md): -1. SSH into the Monitoring node. -1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab - package you want using **steps 1 and 2** from the GitLab downloads page. - Do not complete any other steps on the download page. +1. SSH in to the Monitoring node. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to follow _only_ installation steps 1 and 2 + on the page. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby @@ -1714,28 +1701,44 @@ It's recommended over [NFS](#configure-nfs-optional) and in general it's better in larger setups as object storage is typically much more performant, reliable, and scalable. -Object storage options that GitLab has tested, or is aware of customers using include: +GitLab has been tested on a number of object storage providers: -- SaaS/Cloud solutions such as [Amazon S3](https://aws.amazon.com/s3/), [Google cloud storage](https://cloud.google.com/storage). +- [Amazon S3](https://aws.amazon.com/s3/) +- [Google Cloud Storage](https://cloud.google.com/storage) +- [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces/) +- [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) +- [Openstack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) +- [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) - On-premises hardware and appliances from various storage vendors. -- MinIO. There is [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. - -For configuring GitLab to use Object Storage refer to the following guides -based on what features you intend to use: - -1. Configure [object storage for backups](../../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage). -1. Configure [object storage for job artifacts](../job_artifacts.md#using-object-storage) - including [incremental logging](../job_logs.md#new-incremental-logging-architecture). -1. Configure [object storage for LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage). -1. Configure [object storage for uploads](../uploads.md#using-object-storage). -1. Configure [object storage for merge request diffs](../merge_request_diffs.md#using-object-storage). -1. Configure [object storage for Container Registry](../packages/container_registry.md#use-object-storage) (optional feature). -1. Configure [object storage for Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage) (optional feature). -1. Configure [object storage for packages](../packages/index.md#using-object-storage) (optional feature). **(PREMIUM ONLY)** -1. Configure [object storage for Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature). **(PREMIUM ONLY)** -1. Configure [object storage for Pseudonymizer](../pseudonymizer.md#configuration) (optional feature). **(ULTIMATE ONLY)** -1. Configure [object storage for autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional - for improved performance). -1. Configure [object storage for Terraform state files](../terraform_state.md#using-object-storage). +- MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. + +There are two ways of specifying object storage configuration in GitLab: + +- [Consolidated form](../object_storage.md#consolidated-object-storage-configuration): A single credential is + shared by all supported object types. +- [Storage-specific form](../object_storage.md#storage-specific-configuration): Every object defines its + own object storage [connection and configuration](../object_storage.md#connection-settings). + +Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. + +For configuring object storage in GitLab 13.1 and earlier, or for storage types not +supported by consolidated configuration form, refer to the following guides based +on what features you intend to use: + +|Object storage type|Supported by consolidated configuration?| +|-------------------|----------------------------------------| +| [Backups](../../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage)|No| +| [Job artifacts](../job_artifacts.md#using-object-storage) including archived job logs | Yes | +| [LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage) | Yes | +| [Uploads](../uploads.md#using-object-storage) | Yes | +| [Container Registry](../packages/container_registry.md#use-object-storage) (optional feature) | No | +| [Merge request diffs](../merge_request_diffs.md#using-object-storage) | Yes | +| [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | +| [Packages](../packages/index.md#using-object-storage) (optional feature) | Yes | +| [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature) **(PREMIUM ONLY)** | Yes | +| [Pseudonymizer](../pseudonymizer.md#configuration) (optional feature) **(ULTIMATE ONLY)** | No | +| [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | +| [Terraform state files](../terraform_state.md#using-object-storage) | Yes | Using separate buckets for each data type is the recommended approach for GitLab. @@ -1759,6 +1762,22 @@ work. </a> </div> +## Configure Advanced Search **(STARTER ONLY)** + +You can leverage Elasticsearch and [enable Advanced Search](../../integration/elasticsearch.md) +for faster, more advanced code search across your entire GitLab instance. + +Elasticsearch cluster design and requirements are dependent on your specific +data. For recommended best practices about how to set up your Elasticsearch +cluster alongside your instance, read how to +[choose the optimal cluster configuration](../../integration/elasticsearch.md#guidance-on-choosing-optimal-cluster-configuration). + +<div align="right"> + <a type="button" class="btn btn-default" href="#setup-components"> + Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> + </a> +</div> + ## Configure NFS (optional) [Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly) diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index e812eed0227..44fbe2db504 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -17,21 +17,21 @@ full list of reference architectures, see | Service | Nodes | Configuration | GCP | AWS | Azure | |-----------------------------------------|-------------|-------------------------|-----------------|--------------|----------| -| External load balancing node | 1 | 8 vCPU, 7.2GB memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | -| Consul | 3 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| PostgreSQL | 3 | 16 vCPU, 60GB memory | n1-standard-16 | m5.4xlarge | D16s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Internal load balancing node | 1 | 8 vCPU, 7.2GB memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | -| Redis - Cache | 3 | 4 vCPU, 15GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis - Queues / Shared State | 3 | 4 vCPU, 15GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7GB memory | g1-small | t2.small | B1MS | -| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7GB memory | g1-small | t2.small | B1MS | -| Gitaly | 2 (minimum) | 64 vCPU, 240GB memory | n1-standard-64 | m5.16xlarge | D64s v3 | -| Sidekiq | 4 | 4 vCPU, 15GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| GitLab Rails | 12 | 32 vCPU, 28.8GB memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | -| Monitoring node | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | -| Object Storage | n/a | n/a | n/a | n/a | n/a | -| NFS Server | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| External load balancing node | 1 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | +| Consul | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| PostgreSQL | 3 | 16 vCPU, 60 GB memory | n1-standard-16 | m5.4xlarge | D16s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Internal load balancing node | 1 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | +| Redis - Cache | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| Redis - Queues / Shared State | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7 GB memory | g1-small | t2.small | B1MS | +| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7 GB memory | g1-small | t2.small | B1MS | +| Gitaly | 2 (minimum) | 64 vCPU, 240 GB memory | n1-standard-64 | m5.16xlarge | D64s v3 | +| Sidekiq | 4 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| GitLab Rails | 12 | 32 vCPU, 28.8 GB memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | +| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| Object storage | n/a | n/a | n/a | n/a | n/a | +| NFS server | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | The Google Cloud Platform (GCP) architectures were built and tested using the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) @@ -40,41 +40,43 @@ or higher, are required for your CPU or node counts. For more information, see our [Sysbench](https://github.com/akopytov/sysbench)-based [CPU benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). -For data objects (such as LFS, Uploads, or Artifacts), an -[object storage service](#configure-the-object-storage) is recommended instead -of NFS where possible, due to better performance and availability. Since this -doesn't require a node to be set up, *Object Storage* is noted as not -applicable (n/a) in the previous table. +Due to better performance and availability, for data objects (such as LFS, +uploads, or artifacts), using an [object storage service](#configure-the-object-storage) +is recommended instead of using NFS. Using an object storage service also +doesn't require you to provision and maintain a node. ## Setup components To set up GitLab and its components to accommodate up to 50,000 users: 1. [Configure the external load balancing node](#configure-the-external-load-balancer) - that will handle the load balancing of the three GitLab application services nodes. + to handle the load balancing of the GitLab application services nodes. 1. [Configure Consul](#configure-consul). 1. [Configure PostgreSQL](#configure-postgresql), the database for GitLab. 1. [Configure PgBouncer](#configure-pgbouncer). -1. [Configure the internal load balancing node](#configure-the-internal-load-balancer) +1. [Configure the internal load balancing node](#configure-the-internal-load-balancer). 1. [Configure Redis](#configure-redis). 1. [Configure Gitaly](#configure-gitaly), which provides access to the Git repositories. 1. [Configure Sidekiq](#configure-sidekiq). 1. [Configure the main GitLab Rails application](#configure-gitlab-rails) - to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend requests (UI, API, Git - over HTTP/SSH). -1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab environment. -1. [Configure the Object Storage](#configure-the-object-storage) + to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend + requests (which include UI, API, and Git over HTTP/SSH). +1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab + environment. +1. [Configure the object storage](#configure-the-object-storage) used for shared data objects. -1. [Configure NFS (Optional)](#configure-nfs-optional) - to have shared disk storage service as an alternative to Gitaly and/or Object Storage (although - not recommended). NFS is required for GitLab Pages, you can skip this step if you're not using - that feature. +1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, + more advanced code search across your entire GitLab instance. +1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended) + to have shared disk storage service as an alternative to Gitaly or object + storage. You can skip this step if you're not using GitLab Pages (which + requires NFS). -We start with all servers on the same 10.6.0.0/24 private network range, they -can connect to each other freely on those addresses. +The servers start on the same 10.6.0.0/24 private network range, and can +connect to each other freely on these addresses. -Here is a list and description of each machine and the assigned IP: +The following list includes descriptions of each server and its assigned IP: - `10.6.0.10`: External Load Balancer - `10.6.0.11`: Consul 1 @@ -112,19 +114,18 @@ Here is a list and description of each machine and the assigned IP: ## Configure the external load balancer -NOTE: **Note:** +In an active/active GitLab configuration, you'll need a load balancer to route +traffic to the application servers. The specifics on which load balancer to use +or its exact configuration is beyond the scope of GitLab documentation. We hope +that if you're managing multi-node systems like GitLab, you already have a load +balancer of choice. Some load balancer examples include HAProxy (open-source), +F5 Big-IP LTM, and Citrix Net Scaler. This documentation outline the ports and +protocols needed for use with GitLab. + This architecture has been tested and validated with [HAProxy](https://www.haproxy.org/) as the load balancer. Although other load balancers with similar feature sets could also be used, those load balancers have not been validated. -In an active/active GitLab configuration, you will need a load balancer to route -traffic to the application servers. The specifics on which load balancer to use -or the exact configuration is beyond the scope of GitLab documentation. We hope -that if you're managing multi-node systems like GitLab you have a load balancer of -choice already. Some examples including HAProxy (open-source), F5 Big-IP LTM, -and Citrix Net Scaler. This documentation will outline what ports and protocols -you need to use with GitLab. - The next question is how you will handle SSL in your environment. There are several different options: @@ -247,14 +248,11 @@ with the other servers. To configure Consul: -1. SSH into the server that will host Consul. -1. [Download/install](https://about.gitlab.com/install/) the - Omnibus GitLab Enterprise Edition package using **steps 1 and 2** from the - GitLab downloads page. - - Make sure you select the correct Omnibus package, with the same version - the GitLab application is running. - - Do not complete any other steps on the download page. - +1. SSH in to the server that will host Consul. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to both follow _only_ installation steps 1 and 2 + on the page, and to select the correct Omnibus GitLab package, with the same version + and type (Community or Enterprise editions) as your current install. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby @@ -282,10 +280,9 @@ To configure Consul: 1. Go through the steps again for all the other Consul nodes, and make sure you set up the correct IPs. -NOTE: **Note:** -A Consul leader will be elected when the provisioning of the third Consul server is completed. -Viewing the Consul logs `sudo gitlab-ctl tail consul` will display -`...[INFO] consul: New leader elected: ...` +A Consul leader is _elected_ when the provisioning of the third Consul server is +complete. Viewing the Consul logs `sudo gitlab-ctl tail consul` displays +`...[INFO] consul: New leader elected: ...`. You can list the current Consul members (server, client): @@ -352,7 +349,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. #### PostgreSQL primary node -1. SSH into the PostgreSQL primary node. +1. SSH in to the PostgreSQL primary node. 1. Generate a password hash for the PostgreSQL username/password pair. This assumes you will use the default username of `gitlab` (recommended). The command will request a password and confirmation. Use the value that is output by this command in the next @@ -513,7 +510,7 @@ are supported and can be added if needed. #### PostgreSQL post-configuration -SSH into the **primary node**: +SSH in to the **primary node**: 1. Open a database prompt: @@ -548,7 +545,7 @@ SSH into the **primary node**: is not an IP address, it will need to be a resolvable name (via DNS or `/etc/hosts`) -SSH into the **secondary node**: +SSH in to the **secondary node**: 1. Set up the repmgr standby: @@ -662,7 +659,6 @@ The following IPs will be used as an example: 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - NOTE: **Note:** If an error `execute[generate databases.ini]` occurs, this is due to an existing [known issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4713). It will be resolved when you run a second `reconfigure` after the next step. @@ -796,35 +792,31 @@ to be used with GitLab. The following IPs will be used as an example: - `10.6.0.82`: Sentinel - Queues 2 - `10.6.0.83`: Sentinel - Queues 3 -NOTE: **Providing your own Redis instance:** -Managed Redis from cloud providers such as AWS ElastiCache will work. If these -services support high availability, be sure it is **not** the Redis Cluster type. -Redis version 5.0 or higher is required, as this is what ships with -Omnibus GitLab packages starting with GitLab 13.0. Older Redis versions -do not support an optional count argument to SPOP which is now required for -[Merge Trains](../../ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md). +### Providing your own Redis instance + +Managed Redis from cloud providers (such as AWS ElastiCache) will work. If these +services support high availability, be sure it _isn't_ of the Redis Cluster type. +Redis version 5.0 or higher is required, which is included with Omnibus GitLab +packages starting with GitLab 13.0. Older Redis versions don't support an +optional count argument to SPOP, which is required for [Merge Trains](../../ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md). Note the Redis node's IP address or hostname, port, and password (if required). -These will be necessary when configuring the -[GitLab application servers](#configure-gitlab-rails) later. +These will be necessary later when configuring the [GitLab application servers](#configure-gitlab-rails). ### Configure the Redis and Sentinel Cache cluster This is the section where we install and set up the new Redis Cache instances. -NOTE: **Note:** -Redis nodes (both primary and replica) will need the same password defined in -`redis['password']`. At any time during a failover the Sentinels can -reconfigure a node and change its status from primary to replica and vice versa. +Both the primary and replica Redis nodes need the same password defined in +`redis['password']`. At any time during a failover, the Sentinels can reconfigure +a node and change its status from primary to replica (and vice versa). #### Configure the primary Redis Cache node -1. SSH into the **Primary** Redis server. -1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab - package you want using **steps 1 and 2** from the GitLab downloads page. - - Make sure you select the correct Omnibus package, with the same version - and type (Community, Enterprise editions) of your current install. - - Do not complete any other steps on the download page. - +1. SSH in to the **Primary** Redis server. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to both follow _only_ installation steps 1 and 2 + on the page, and to select the correct Omnibus GitLab package, with the same version + and type (Community or Enterprise editions) as your current install. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby @@ -868,20 +860,17 @@ reconfigure a node and change its status from primary to replica and vice versa. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -NOTE: **Note:** -You can specify multiple roles like sentinel and Redis as: -`roles ['redis_sentinel_role', 'redis_master_role']`. -Read more about [roles](https://docs.gitlab.com/omnibus/roles/). +You can specify multiple roles, like sentinel and Redis, as: +`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +[roles](https://docs.gitlab.com/omnibus/roles/). #### Configure the replica Redis Cache nodes -1. SSH into the **replica** Redis server. -1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab - package you want using **steps 1 and 2** from the GitLab downloads page. - - Make sure you select the correct Omnibus package, with the same version - and type (Community, Enterprise editions) of your current install. - - Do not complete any other steps on the download page. - +1. SSH in to the **replica** Redis server. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to both follow _only_ installation steps 1 and 2 + on the page, and to select the correct Omnibus GitLab package, with the same version + and type (Community or Enterprise editions) as your current install. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby @@ -934,10 +923,9 @@ Read more about [roles](https://docs.gitlab.com/omnibus/roles/). 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. -NOTE: **Note:** -You can specify multiple roles like sentinel and Redis as: -`roles ['redis_sentinel_role', 'redis_master_role']`. -Read more about [roles](https://docs.gitlab.com/omnibus/roles/). +You can specify multiple roles, like sentinel and Redis, as: +`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +[roles](https://docs.gitlab.com/omnibus/roles/). These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-cache-nodes), and even after a @@ -955,13 +943,6 @@ are supported and can be added if needed. #### Configure the Sentinel Cache nodes -NOTE: **Note:** -If you are using an external Redis Sentinel instance, be sure -to exclude the `requirepass` parameter from the Sentinel -configuration. This parameter will cause clients to report `NOAUTH -Authentication required.`. [Redis Sentinel 3.2.x does not support -password authentication](https://github.com/antirez/redis/issues/3279). - Now that the Redis servers are all set up, let's configure the Sentinel servers. The following IPs will be used as an example: @@ -969,16 +950,19 @@ servers. The following IPs will be used as an example: - `10.6.0.72`: Sentinel - Cache 2 - `10.6.0.73`: Sentinel - Cache 3 -To configure the Sentinel Cache server: +NOTE: **Note:** +If you're using an external Redis Sentinel instance, be sure to exclude the +`requirepass` parameter from the Sentinel configuration. This parameter causes +clients to report `NOAUTH Authentication required.`. +[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279). -1. SSH into the server that will host Consul/Sentinel. -1. [Download/install](https://about.gitlab.com/install/) the - Omnibus GitLab Enterprise Edition package using **steps 1 and 2** from the - GitLab downloads page. - - Make sure you select the correct Omnibus package, with the same version - the GitLab application is running. - - Do not complete any other steps on the download page. +To configure the Sentinel Cache server: +1. SSH in to the server that will host Consul/Sentinel. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to both follow _only_ installation steps 1 and 2 + on the page, and to select the correct Omnibus GitLab package, with the same version + and type (Community or Enterprise editions) as your current install. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby @@ -1081,20 +1065,17 @@ To configure the Sentinel Cache server: This is the section where we install and set up the new Redis Queues instances. -NOTE: **Note:** -Redis nodes (both primary and replica) will need the same password defined in -`redis['password']`. At any time during a failover the Sentinels can -reconfigure a node and change its status from primary to replica and vice versa. +Both the primary and replica Redis nodes need the same password defined in +`redis['password']`. At any time during a failover, the Sentinels can reconfigure +a node and change its status from primary to replica (and vice versa). #### Configure the primary Redis Queues node -1. SSH into the **Primary** Redis server. -1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab - package you want using **steps 1 and 2** from the GitLab downloads page. - - Make sure you select the correct Omnibus package, with the same version - and type (Community, Enterprise editions) of your current install. - - Do not complete any other steps on the download page. - +1. SSH in to the **Primary** Redis server. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to both follow _only_ installation steps 1 and 2 + on the page, and to select the correct Omnibus GitLab package, with the same version + and type (Community or Enterprise editions) as your current install. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby @@ -1143,20 +1124,17 @@ reconfigure a node and change its status from primary to replica and vice versa. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -NOTE: **Note:** -You can specify multiple roles like sentinel and Redis as: -`roles ['redis_sentinel_role', 'redis_master_role']`. -Read more about [roles](https://docs.gitlab.com/omnibus/roles/). +You can specify multiple roles, like sentinel and Redis, as: +`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +[roles](https://docs.gitlab.com/omnibus/roles/). #### Configure the replica Redis Queues nodes -1. SSH into the **replica** Redis Queue server. -1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab - package you want using **steps 1 and 2** from the GitLab downloads page. - - Make sure you select the correct Omnibus package, with the same version - and type (Community, Enterprise editions) of your current install. - - Do not complete any other steps on the download page. - +1. SSH in to the **replica** Redis Queue server. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to both follow _only_ installation steps 1 and 2 + on the page, and to select the correct Omnibus GitLab package, with the same version + and type (Community or Enterprise editions) as your current install. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby @@ -1209,10 +1187,9 @@ Read more about [roles](https://docs.gitlab.com/omnibus/roles/). 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. -NOTE: **Note:** -You can specify multiple roles like sentinel and Redis as: -`roles ['redis_sentinel_role', 'redis_master_role']`. -Read more about [roles](https://docs.gitlab.com/omnibus/roles/). +You can specify multiple roles, like sentinel and Redis, as: +`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +[roles](https://docs.gitlab.com/omnibus/roles/). These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-queues-nodes), and even after a @@ -1230,13 +1207,6 @@ are supported and can be added if needed. #### Configure the Sentinel Queues nodes -NOTE: **Note:** -If you are using an external Redis Sentinel instance, be sure -to exclude the `requirepass` parameter from the Sentinel -configuration. This parameter will cause clients to report `NOAUTH -Authentication required.`. [Redis Sentinel 3.2.x does not support -password authentication](https://github.com/antirez/redis/issues/3279). - Now that the Redis servers are all set up, let's configure the Sentinel servers. The following IPs will be used as an example: @@ -1244,16 +1214,19 @@ servers. The following IPs will be used as an example: - `10.6.0.82`: Sentinel - Queues 2 - `10.6.0.83`: Sentinel - Queues 3 -To configure the Sentinel Queues server: +NOTE: **Note:** +If you're using an external Redis Sentinel instance, be sure to exclude the +`requirepass` parameter from the Sentinel configuration. This parameter causes +clients to report `NOAUTH Authentication required.`. +[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279). -1. SSH into the server that will host Sentinel. -1. [Download/install](https://about.gitlab.com/install/) the - Omnibus GitLab Enterprise Edition package using **steps 1 and 2** from the - GitLab downloads page. - - Make sure you select the correct Omnibus package, with the same version - the GitLab application is running. - - Do not complete any other steps on the download page. +To configure the Sentinel Queues server: +1. SSH in to the server that will host Sentinel. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to both follow _only_ installation steps 1 and 2 + on the page, and to select the correct Omnibus GitLab package, with the same version + and type (Community or Enterprise editions) as your current install. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby @@ -1362,52 +1335,48 @@ To configure the Sentinel Queues server: ## Configure Gitaly -Deploying Gitaly in its own server can benefit GitLab installations that are -larger than a single machine. - -The Gitaly node requirements are dependent on customer data, specifically the number of -projects and their repository sizes. Two nodes are recommended as an absolute minimum. -Each Gitaly node should store no more than 5TB of data and have the number of -[`gitaly-ruby` workers](../gitaly/index.md#gitaly-ruby) set to 20% of available CPUs. -Additional nodes should be considered in conjunction with a review of expected -data size and spread based on the recommendations above. - -It is also strongly recommended that all Gitaly nodes be set up with SSD disks with -a throughput of at least 8,000 IOPS for read operations and 2,000 IOPS for write, -as Gitaly has heavy I/O. These IOPS values are recommended only as a starter as with -time they may be adjusted higher or lower depending on the scale of your environment's workload. -If you're running the environment on a Cloud provider, you may need to refer to -their documentation on how to configure IOPS correctly. - -Some things to note: - -- The GitLab Rails application shards repositories into [repository storages](../repository_storage_paths.md). -- A Gitaly server can host one or more storages. -- A GitLab server can use one or more Gitaly servers. -- Gitaly addresses must be specified in such a way that they resolve - correctly for ALL Gitaly clients. +[Gitaly](../gitaly/index.md) server node requirements are dependent on data, +specifically the number of projects and those projects' sizes. It's recommended +that a Gitaly server node stores no more than 5 TB of data. Although this +reference architecture includes a recommendation for the number of Gitaly server +nodes to use, depending on your storage requirements, you may require additional +Gitaly server nodes. + +Due to Gitaly having notable input and output requirements, we strongly +recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs should +have a throughput of at least 8,000 input/output operations per second (IOPS) +for read operations and 2,000 IOPS for write operations. These IOPS values are +initial recommendations, and may be adjusted to greater or lesser values +depending on the scale of your environment's workload. If you're running the +environment on a Cloud provider, refer to their documentation about how to +configure IOPS correctly. + +Be sure to note the following items: + +- The GitLab Rails application shards repositories into + [repository storage paths](../repository_storage_paths.md). +- A Gitaly server can host one or more storage paths. +- A GitLab server can use one or more Gitaly server nodes. +- Gitaly addresses must be specified to be correctly resolvable for _all_ + Gitaly clients. - Gitaly servers must not be exposed to the public internet, as Gitaly's network traffic is unencrypted by default. The use of a firewall is highly recommended to restrict access to the Gitaly server. Another option is to [use TLS](#gitaly-tls-support). -TIP: **Tip:** -For more information about Gitaly's history and network architecture see the -[standalone Gitaly documentation](../gitaly/index.md). - -Note: **Note:** -The token referred to throughout the Gitaly documentation is -just an arbitrary password selected by the administrator. It is unrelated to -tokens created for the GitLab API or other similar web API tokens. +NOTE: **Note:** +The token referred to throughout the Gitaly documentation is an arbitrary +password selected by the administrator. This token is unrelated to tokens +created for the GitLab API or other similar web API tokens. -Below we describe how to configure two Gitaly servers, with IPs and -domain names: +This section describes how to configure two Gitaly servers, with the following +IPs and domain names: - `10.6.0.91`: Gitaly 1 (`gitaly1.internal`) - `10.6.0.92`: Gitaly 2 (`gitaly2.internal`) -The secret token is assumed to be `gitalysecret` and that -your GitLab installation has three repository storages: +Assumptions about your servers include having the secret token be `gitalysecret`, +and that your GitLab installation has three repository storages: - `default` on Gitaly 1 - `storage1` on Gitaly 1 @@ -1415,11 +1384,11 @@ your GitLab installation has three repository storages: On each node: -1. [Download/Install](https://about.gitlab.com/install/) the Omnibus GitLab - package you want using **steps 1 and 2** from the GitLab downloads page but - **without** providing the `EXTERNAL_URL` value. -1. Edit `/etc/gitlab/gitlab.rb` to configure storage paths, enable - the network listener and configure the token: +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to follow _only_ installation steps 1 and 2 + on the page, and _do not_ provide the `EXTERNAL_URL` value. +1. Edit `/etc/gitlab/gitlab.rb` to configure the storage paths, enable + the network listener, and configure the token: <!-- updates to following example must also be made at @@ -1467,39 +1436,39 @@ On each node: ``` 1. Append the following to `/etc/gitlab/gitlab.rb` for each respective server: - 1. On `gitaly1.internal`: - - ```ruby - git_data_dirs({ - 'default' => { - 'path' => '/var/opt/gitlab/git-data' - }, - 'storage1' => { - 'path' => '/mnt/gitlab/git-data' - }, - }) - ``` - - 1. On `gitaly2.internal`: - - ```ruby - git_data_dirs({ - 'storage2' => { - 'path' => '/mnt/gitlab/git-data' - }, - }) - ``` + - On `gitaly1.internal`: + + ```ruby + git_data_dirs({ + 'default' => { + 'path' => '/var/opt/gitlab/git-data' + }, + 'storage1' => { + 'path' => '/mnt/gitlab/git-data' + }, + }) + ``` + + - On `gitaly2.internal`: + + ```ruby + git_data_dirs({ + 'storage2' => { + 'path' => '/mnt/gitlab/git-data' + }, + }) + ``` <!-- updates to following example must also be made at https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab --> -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and + then replace the file of the same name on this server. If that file isn't on + this server, add the file from your Consul server to this server. -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). ### Gitaly TLS support @@ -1521,11 +1490,10 @@ Name. If you are addressing the Gitaly server by its IP address, you must add it as a Subject Alternative Name to the certificate. [gRPC does not support using an IP address as Common Name in a certificate](https://github.com/grpc/grpc/issues/2691). -NOTE: **Note:** -It is possible to configure Gitaly servers with both an -unencrypted listening address `listen_addr` and an encrypted listening -address `tls_listen_addr` at the same time. This allows you to do a -gradual transition from unencrypted to encrypted traffic, if necessary. +It's possible to configure Gitaly servers with both an unencrypted listening +address (`listen_addr`) and an encrypted listening address (`tls_listen_addr`) +at the same time. This allows you to do a gradual transition from unencrypted to +encrypted traffic, if necessary. To configure Gitaly with TLS: @@ -1580,10 +1548,10 @@ The following IPs will be used as an example: To configure the Sidekiq nodes, on each one: -1. SSH into the Sidekiq server. -1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab package -you want using steps 1 and 2 from the GitLab downloads page. -**Do not complete any other steps on the download page.** +1. SSH in to the Sidekiq server. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to follow _only_ installation steps 1 and 2 + on the page. 1. Open `/etc/gitlab/gitlab.rb` with your editor: ```ruby @@ -1701,15 +1669,14 @@ You can also run [multiple Sidekiq processes](../operations/extra_sidekiq_proces ## Configure GitLab Rails -NOTE: **Note:** -In our architectures we run each GitLab Rails node using the Puma webserver -and have its number of workers set to 90% of available CPUs along with four threads. For -nodes that are running Rails with other components the worker value should be reduced -accordingly where we've found 50% achieves a good balance but this is dependent -on workload. - This section describes how to configure the GitLab application (Rails) component. +In our architecture, we run each GitLab Rails node using the Puma webserver, and +have its number of workers set to 90% of available CPUs, with four threads. For +nodes running Rails with other components, the worker value should be reduced +accordingly. We've determined that a worker value of 50% achieves a good balance, +but this is dependent on workload. + The following IPs will be used as an example: - `10.6.0.111`: GitLab application 1 @@ -1718,10 +1685,9 @@ The following IPs will be used as an example: On each node perform the following: -1. Download/install Omnibus GitLab using **steps 1 and 2** from - [GitLab downloads](https://about.gitlab.com/install/). Do not complete other - steps on the download page. - +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to follow _only_ installation steps 1 and 2 + on the page. 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace the file of the same name on this server. If that file is not on this server, add the file from your Consul server to this server. @@ -1752,6 +1718,7 @@ On each node perform the following: roles ['application_role'] gitaly['enable'] = false nginx['enable'] = true + sidekiq['enable'] = false ## PostgreSQL connection details # Disable PostgreSQL on the application node @@ -1795,7 +1762,6 @@ On each node perform the following: # Set the network addresses that the exporters used for monitoring will listen on node_exporter['listen_address'] = '0.0.0.0:9100' gitlab_workhorse['prometheus_listen_addr'] = '0.0.0.0:9229' - sidekiq['listen_address'] = "0.0.0.0" puma['listen'] = '0.0.0.0' # Add the monitoring node's IP address to the monitoring whitelist and allow it to @@ -1836,7 +1802,7 @@ On each node perform the following: 1. Specify the necessary NFS mounts in `/etc/fstab`. The exact contents of `/etc/fstab` will depend on how you chose - to configure your NFS server. See the [NFS documentation](../high_availability/nfs.md) + to configure your NFS server. See the [NFS documentation](../nfs.md) for examples and the various options. 1. Create the shared directories. These may be different depending on your NFS @@ -1877,30 +1843,31 @@ On each node perform the following: 1. Optionally, from the Gitaly servers, confirm that Gitaly can perform callbacks to the internal API: ```shell - sudo /opt/gitlab/embedded/service/gitlab-shell/bin/check -config /opt/gitlab/embedded/service/gitlab-shell/config.yml + sudo /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml ``` -NOTE: **Note:** -When you specify `https` in the `external_url`, as in the example -above, GitLab assumes you have SSL certificates in `/etc/gitlab/ssl/`. If -certificates are not present, NGINX will fail to start. See the -[NGINX documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) -for more information. +When you specify `https` in the `external_url`, as in the previous example, +GitLab expects that the SSL certificates are in `/etc/gitlab/ssl/`. If the +certificates aren't present, NGINX will fail to start. For more information, see +the [NGINX documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https). ### GitLab Rails post-configuration -Initialize the GitLab database, by running the following in one of the Rails nodes: +1. Designate one application node for running database migrations during + installation and updates. Initialize the GitLab database and ensure all + migrations ran: -```shell -sudo gitlab-rake gitlab:db:configure -``` + ```shell + sudo gitlab-rake gitlab:db:configure + ``` -NOTE: **Note:** -If you encounter a `rake aborted!` error stating that PgBouncer is failing to connect to -PostgreSQL it may be that your PgBouncer node's IP address is missing from -PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` on your database nodes. See -[PgBouncer error `ERROR: pgbouncer cannot connect to server`](troubleshooting.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) -in the Troubleshooting section before proceeding. + If you encounter a `rake aborted!` error message stating that PgBouncer is + failing to connect to PostgreSQL, it may be that your PgBouncer node's IP + address is missing from PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` + on your database nodes. Before proceeding, see + [PgBouncer error `ERROR: pgbouncer cannot connect to server`](troubleshooting.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server). + +1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -1920,11 +1887,10 @@ The following IP will be used as an example: To configure the Monitoring node: -1. SSH into the Monitoring node. -1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab - package you want using **steps 1 and 2** from the GitLab downloads page. - Do not complete any other steps on the download page. - +1. SSH in to the Monitoring node. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to follow _only_ installation steps 1 and 2 + on the page. 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace the file of the same name on this server. If that file is not on this server, add the file from your Consul server to this server. @@ -1988,28 +1954,44 @@ It's recommended over [NFS](#configure-nfs-optional) and in general it's better in larger setups as object storage is typically much more performant, reliable, and scalable. -Object storage options that GitLab has tested, or is aware of customers using include: +GitLab has been tested on a number of object storage providers: -- SaaS/Cloud solutions such as [Amazon S3](https://aws.amazon.com/s3/), [Google cloud storage](https://cloud.google.com/storage). +- [Amazon S3](https://aws.amazon.com/s3/) +- [Google Cloud Storage](https://cloud.google.com/storage) +- [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces/) +- [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) +- [Openstack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) +- [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) - On-premises hardware and appliances from various storage vendors. -- MinIO. There is [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. - -For configuring GitLab to use Object Storage refer to the following guides -based on what features you intend to use: - -1. Configure [object storage for backups](../../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage). -1. Configure [object storage for job artifacts](../job_artifacts.md#using-object-storage) - including [incremental logging](../job_logs.md#new-incremental-logging-architecture). -1. Configure [object storage for LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage). -1. Configure [object storage for uploads](../uploads.md#using-object-storage). -1. Configure [object storage for merge request diffs](../merge_request_diffs.md#using-object-storage). -1. Configure [object storage for Container Registry](../packages/container_registry.md#use-object-storage) (optional feature). -1. Configure [object storage for Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage) (optional feature). -1. Configure [object storage for packages](../packages/index.md#using-object-storage) (optional feature). **(PREMIUM ONLY)** -1. Configure [object storage for Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature). **(PREMIUM ONLY)** -1. Configure [object storage for Pseudonymizer](../pseudonymizer.md#configuration) (optional feature). **(ULTIMATE ONLY)** -1. Configure [object storage for autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional - for improved performance). -1. Configure [object storage for Terraform state files](../terraform_state.md#using-object-storage). +- MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. + +There are two ways of specifying object storage configuration in GitLab: + +- [Consolidated form](../object_storage.md#consolidated-object-storage-configuration): A single credential is + shared by all supported object types. +- [Storage-specific form](../object_storage.md#storage-specific-configuration): Every object defines its + own object storage [connection and configuration](../object_storage.md#connection-settings). + +Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. + +For configuring object storage in GitLab 13.1 and earlier, or for storage types not +supported by consolidated configuration form, refer to the following guides based +on what features you intend to use: + +|Object storage type|Supported by consolidated configuration?| +|-------------------|----------------------------------------| +| [Backups](../../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage)|No| +| [Job artifacts](../job_artifacts.md#using-object-storage) including archived job logs | Yes | +| [LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage) | Yes | +| [Uploads](../uploads.md#using-object-storage) | Yes | +| [Container Registry](../packages/container_registry.md#use-object-storage) (optional feature) | No | +| [Merge request diffs](../merge_request_diffs.md#using-object-storage) | Yes | +| [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | +| [Packages](../packages/index.md#using-object-storage) (optional feature) | Yes | +| [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature) **(PREMIUM ONLY)** | Yes | +| [Pseudonymizer](../pseudonymizer.md#configuration) (optional feature) **(ULTIMATE ONLY)** | No | +| [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | +| [Terraform state files](../terraform_state.md#using-object-storage) | Yes | Using separate buckets for each data type is the recommended approach for GitLab. @@ -2033,13 +2015,29 @@ work. </a> </div> +## Configure Advanced Search **(STARTER ONLY)** + +You can leverage Elasticsearch and [enable Advanced Search](../../integration/elasticsearch.md) +for faster, more advanced code search across your entire GitLab instance. + +Elasticsearch cluster design and requirements are dependent on your specific +data. For recommended best practices about how to set up your Elasticsearch +cluster alongside your instance, read how to +[choose the optimal cluster configuration](../../integration/elasticsearch.md#guidance-on-choosing-optimal-cluster-configuration). + +<div align="right"> + <a type="button" class="btn btn-default" href="#setup-components"> + Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> + </a> +</div> + ## Configure NFS (optional) [Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly) are recommended over NFS wherever possible for improved performance. If you intend to use GitLab Pages, this currently [requires NFS](troubleshooting.md#gitlab-pages-requires-nfs). -See how to [configure NFS](../high_availability/nfs.md). +See how to [configure NFS](../nfs.md). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index 6dfa588b092..9f83950a927 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -24,18 +24,18 @@ costly-to-operate environment by using the | Service | Nodes | Configuration | GCP | AWS | Azure | |--------------------------------------------|-------------|-------------------------|----------------|-------------|----------| -| External load balancing node | 1 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Redis | 3 | 2 vCPU, 7.5GB memory | n1-standard-2 | m5.large | D2s v3 | -| Consul + Sentinel | 3 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| PostgreSQL | 3 | 2 vCPU, 7.5GB memory | n1-standard-2 | m5.large | D2s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Internal load balancing node | 1 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Gitaly | 2 (minimum) | 8 vCPU, 30GB memory | n1-standard-8 | m5.2xlarge | D8s v3 | -| Sidekiq | 4 | 2 vCPU, 7.5GB memory | n1-standard-2 | m5.large | D2s v3 | -| GitLab Rails | 3 | 16 vCPU, 14.4GB memory | n1-highcpu-16 | c5.4xlarge | F16s v2 | -| Monitoring node | 1 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Object Storage | n/a | n/a | n/a | n/a | n/a | -| NFS Server (optional, not recommended) | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| External load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Redis | 3 | 2 vCPU, 7.5 GB memory | n1-standard-2 | m5.large | D2s v3 | +| Consul + Sentinel | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| PostgreSQL | 3 | 2 vCPU, 7.5 GB memory | n1-standard-2 | m5.large | D2s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Gitaly | 2 (minimum) | 8 vCPU, 30 GB memory | n1-standard-8 | m5.2xlarge | D8s v3 | +| Sidekiq | 4 | 2 vCPU, 7.5 GB memory | n1-standard-2 | m5.large | D2s v3 | +| GitLab Rails | 3 | 16 vCPU, 14.4 GB memory | n1-highcpu-16 | c5.4xlarge | F16s v2 | +| Monitoring node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Object storage | n/a | n/a | n/a | n/a | n/a | +| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | The Google Cloud Platform (GCP) architectures were built and tested using the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) @@ -44,41 +44,43 @@ or higher, are required for your CPU or node counts. For more information, see our [Sysbench](https://github.com/akopytov/sysbench)-based [CPU benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). -For data objects (such as LFS, Uploads, or Artifacts), an -[object storage service](#configure-the-object-storage) is recommended instead -of NFS where possible, due to better performance and availability. Since this -doesn't require a node to be set up, *Object Storage* is noted as not -applicable (n/a) in the previous table. +Due to better performance and availability, for data objects (such as LFS, +uploads, or artifacts), using an [object storage service](#configure-the-object-storage) +is recommended instead of using NFS. Using an object storage service also +doesn't require you to provision and maintain a node. ## Setup components To set up GitLab and its components to accommodate up to 5,000 users: 1. [Configure the external load balancing node](#configure-the-external-load-balancer) - that will handle the load balancing of the two GitLab application services nodes. + to handle the load balancing of the GitLab application services nodes. 1. [Configure Redis](#configure-redis). 1. [Configure Consul and Sentinel](#configure-consul-and-sentinel). 1. [Configure PostgreSQL](#configure-postgresql), the database for GitLab. 1. [Configure PgBouncer](#configure-pgbouncer). -1. [Configure the internal load balancing node](#configure-the-internal-load-balancer) +1. [Configure the internal load balancing node](#configure-the-internal-load-balancer). 1. [Configure Gitaly](#configure-gitaly), which provides access to the Git repositories. 1. [Configure Sidekiq](#configure-sidekiq). 1. [Configure the main GitLab Rails application](#configure-gitlab-rails) - to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend requests (UI, API, Git - over HTTP/SSH). -1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab environment. -1. [Configure the Object Storage](#configure-the-object-storage) + to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend + requests (which include UI, API, and Git over HTTP/SSH). +1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab + environment. +1. [Configure the object storage](#configure-the-object-storage) used for shared data objects. -1. [Configure NFS (Optional)](#configure-nfs-optional) - to have shared disk storage service as an alternative to Gitaly and/or Object Storage (although - not recommended). NFS is required for GitLab Pages, you can skip this step if you're not using - that feature. +1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, + more advanced code search across your entire GitLab instance. +1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended) + to have shared disk storage service as an alternative to Gitaly or object + storage. You can skip this step if you're not using GitLab Pages (which + requires NFS). -We start with all servers on the same 10.6.0.0/16 private network range, they -can connect to each other freely on those addresses. +The servers start on the same 10.6.0.0/24 private network range, and can +connect to each other freely on these addresses. -Here is a list and description of each machine and the assigned IP: +The following list includes descriptions of each server and its assigned IP: - `10.6.0.10`: External Load Balancer - `10.6.0.61`: Redis Primary @@ -107,19 +109,18 @@ Here is a list and description of each machine and the assigned IP: ## Configure the external load balancer -NOTE: **Note:** +In an active/active GitLab configuration, you'll need a load balancer to route +traffic to the application servers. The specifics on which load balancer to use +or its exact configuration is beyond the scope of GitLab documentation. We hope +that if you're managing multi-node systems like GitLab, you already have a load +balancer of choice. Some load balancer examples include HAProxy (open-source), +F5 Big-IP LTM, and Citrix Net Scaler. This documentation outline the ports and +protocols needed for use with GitLab. + This architecture has been tested and validated with [HAProxy](https://www.haproxy.org/) as the load balancer. Although other load balancers with similar feature sets could also be used, those load balancers have not been validated. -In an active/active GitLab configuration, you will need a load balancer to route -traffic to the application servers. The specifics on which load balancer to use -or the exact configuration is beyond the scope of GitLab documentation. We hope -that if you're managing multi-node systems like GitLab you have a load balancer of -choice already. Some examples including HAProxy (open-source), F5 Big-IP LTM, -and Citrix Net Scaler. This documentation will outline what ports and protocols -you need to use with GitLab. - The next question is how you will handle SSL in your environment. There are several different options: @@ -277,20 +278,17 @@ The requirements for a Redis setup are the following: ([Internet](https://gitlab.com/gitlab-org/gitlab-foss/uploads/c4cc8cd353604bd80315f9384035ff9e/The_Internet_IT_Crowd.png)), using a firewall. -NOTE: **Note:** -Redis nodes (both primary and replica) will need the same password defined in -`redis['password']`. At any time during a failover the Sentinels can -reconfigure a node and change its status from primary to replica and vice versa. +Both the primary and replica Redis nodes need the same password defined in +`redis['password']`. At any time during a failover, the Sentinels can reconfigure +a node and change its status from primary to replica (and vice versa). #### Configuring the primary Redis instance -1. SSH into the **Primary** Redis server. -1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab - package you want using **steps 1 and 2** from the GitLab downloads page. - - Make sure you select the correct Omnibus package, with the same version - and type (Community, Enterprise editions) of your current install. - - Do not complete any other steps on the download page. - +1. SSH in to the **Primary** Redis server. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to both follow _only_ installation steps 1 and 2 + on the page, and to select the correct Omnibus GitLab package, with the same version + and type (Community or Enterprise editions) as your current install. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby @@ -334,10 +332,9 @@ reconfigure a node and change its status from primary to replica and vice versa. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -NOTE: **Note:** -You can specify multiple roles like sentinel and Redis as: -`roles ['redis_sentinel_role', 'redis_master_role']`. -Read more about [roles](https://docs.gitlab.com/omnibus/roles/). +You can specify multiple roles, like sentinel and Redis, as: +`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +[roles](https://docs.gitlab.com/omnibus/roles/). You can list the current Redis Primary, Replica status via: @@ -363,13 +360,11 @@ run: redis-exporter: (pid 30075) 76861s; run: log: (pid 29674) 76896s #### Configuring the replica Redis instances -1. SSH into the **replica** Redis server. -1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab - package you want using **steps 1 and 2** from the GitLab downloads page. - - Make sure you select the correct Omnibus package, with the same version - and type (Community, Enterprise editions) of your current install. - - Do not complete any other steps on the download page. - +1. SSH in to the **replica** Redis server. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to both follow _only_ installation steps 1 and 2 + on the page, and to select the correct Omnibus GitLab package, with the same version + and type (Community or Enterprise editions) as your current install. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby @@ -422,10 +417,9 @@ run: redis-exporter: (pid 30075) 76861s; run: log: (pid 29674) 76896s 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. -NOTE: **Note:** -You can specify multiple roles like sentinel and Redis as: -`roles ['redis_sentinel_role', 'redis_master_role']`. -Read more about [roles](https://docs.gitlab.com/omnibus/roles/). +You can specify multiple roles, like sentinel and Redis, as: +`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +[roles](https://docs.gitlab.com/omnibus/roles/). These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after a failover, as the nodes will be managed by the [Sentinels](#configure-consul-and-sentinel), and even after a @@ -443,13 +437,6 @@ are supported and can be added if needed. ## Configure Consul and Sentinel -NOTE: **Note:** -If you are using an external Redis Sentinel instance, be sure -to exclude the `requirepass` parameter from the Sentinel -configuration. This parameter will cause clients to report `NOAUTH -Authentication required.`. [Redis Sentinel 3.2.x does not support -password authentication](https://github.com/antirez/redis/issues/3279). - Now that the Redis servers are all set up, let's configure the Sentinel servers. The following IPs will be used as an example: @@ -457,16 +444,19 @@ servers. The following IPs will be used as an example: - `10.6.0.12`: Consul/Sentinel 2 - `10.6.0.13`: Consul/Sentinel 3 -To configure the Sentinel: +NOTE: **Note:** +If you're using an external Redis Sentinel instance, be sure to exclude the +`requirepass` parameter from the Sentinel configuration. This parameter causes +clients to report `NOAUTH Authentication required.`. +[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279). -1. SSH into the server that will host Consul/Sentinel. -1. [Download/install](https://about.gitlab.com/install/) the - Omnibus GitLab Enterprise Edition package using **steps 1 and 2** from the - GitLab downloads page. - - Make sure you select the correct Omnibus package, with the same version - the GitLab application is running. - - Do not complete any other steps on the download page. +To configure the Sentinel: +1. SSH in to the server that will host Consul/Sentinel. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to both follow _only_ installation steps 1 and 2 + on the page, and to select the correct Omnibus GitLab package, with the same version + and type (Community or Enterprise editions) as your current install. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby @@ -556,10 +546,9 @@ To configure the Sentinel: 1. Go through the steps again for all the other Consul/Sentinel nodes, and make sure you set up the correct IPs. -NOTE: **Note:** -A Consul leader will be elected when the provisioning of the third Consul server is completed. -Viewing the Consul logs `sudo gitlab-ctl tail consul` will display -`...[INFO] consul: New leader elected: ...` +A Consul leader is _elected_ when the provisioning of the third Consul server is +complete. Viewing the Consul logs `sudo gitlab-ctl tail consul` displays +`...[INFO] consul: New leader elected: ...`. You can list the current Consul members (server, client): @@ -627,7 +616,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. #### PostgreSQL primary node -1. SSH into the PostgreSQL primary node. +1. SSH in to the PostgreSQL primary node. 1. Generate a password hash for the PostgreSQL username/password pair. This assumes you will use the default username of `gitlab` (recommended). The command will request a password and confirmation. Use the value that is output by this command in the next @@ -812,7 +801,7 @@ are supported and can be added if needed. #### PostgreSQL post-configuration -SSH into the **primary node**: +SSH in to the **primary node**: 1. Open a database prompt: @@ -847,7 +836,7 @@ SSH into the **primary node**: is not an IP address, it will need to be a resolvable name (via DNS or `/etc/hosts`) -SSH into the **secondary node**: +SSH in to the **secondary node**: 1. Set up the repmgr standby: @@ -1068,51 +1057,48 @@ Refer to your preferred Load Balancer's documentation for further guidance. ## Configure Gitaly -Deploying Gitaly in its own server can benefit GitLab installations that are -larger than a single machine. - -The Gitaly node requirements are dependent on customer data, specifically the number of -projects and their repository sizes. Two nodes are recommended as an absolute minimum. -Each Gitaly node should store no more than 5TB of data and have the number of -[`gitaly-ruby` workers](../gitaly/index.md#gitaly-ruby) set to 20% of available CPUs. -Additional nodes should be considered in conjunction with a review of expected -data size and spread based on the recommendations above. - -It is also strongly recommended that all Gitaly nodes be set up with SSD disks with -a throughput of at least 8,000 IOPS for read operations and 2,000 IOPS for write, -as Gitaly has heavy I/O. These IOPS values are recommended only as a starter as with -time they may be adjusted higher or lower depending on the scale of your environment's workload. -If you're running the environment on a Cloud provider, you may need to refer to -their documentation on how to configure IOPS correctly. - -Some things to note: - -- The GitLab Rails application shards repositories into [repository storages](../repository_storage_paths.md). -- A Gitaly server can host one or more storages. -- A GitLab server can use one or more Gitaly servers. -- Gitaly addresses must be specified in such a way that they resolve - correctly for ALL Gitaly clients. +[Gitaly](../gitaly/index.md) server node requirements are dependent on data, +specifically the number of projects and those projects' sizes. It's recommended +that a Gitaly server node stores no more than 5 TB of data. Although this +reference architecture includes a recommendation for the number of Gitaly server +nodes to use, depending on your storage requirements, you may require additional +Gitaly server nodes. + +Due to Gitaly having notable input and output requirements, we strongly +recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs should +have a throughput of at least 8,000 input/output operations per second (IOPS) +for read operations and 2,000 IOPS for write operations. These IOPS values are +initial recommendations, and may be adjusted to greater or lesser values +depending on the scale of your environment's workload. If you're running the +environment on a Cloud provider, refer to their documentation about how to +configure IOPS correctly. + +Be sure to note the following items: + +- The GitLab Rails application shards repositories into + [repository storage paths](../repository_storage_paths.md). +- A Gitaly server can host one or more storage paths. +- A GitLab server can use one or more Gitaly server nodes. +- Gitaly addresses must be specified to be correctly resolvable for _all_ + Gitaly clients. - Gitaly servers must not be exposed to the public internet, as Gitaly's network traffic is unencrypted by default. The use of a firewall is highly recommended to restrict access to the Gitaly server. Another option is to [use TLS](#gitaly-tls-support). -TIP: **Tip:** -For more information about Gitaly's history and network architecture see the -[standalone Gitaly documentation](../gitaly/index.md). - -Note: **Note:** The token referred to throughout the Gitaly documentation is -just an arbitrary password selected by the administrator. It is unrelated to -tokens created for the GitLab API or other similar web API tokens. +NOTE: **Note:** +The token referred to throughout the Gitaly documentation is an arbitrary +password selected by the administrator. This token is unrelated to tokens +created for the GitLab API or other similar web API tokens. -Below we describe how to configure two Gitaly servers, with IPs and -domain names: +This section describes how to configure two Gitaly servers, with the following +IPs and domain names: - `10.6.0.51`: Gitaly 1 (`gitaly1.internal`) - `10.6.0.52`: Gitaly 2 (`gitaly2.internal`) -The secret token is assumed to be `gitalysecret` and that -your GitLab installation has three repository storages: +Assumptions about your servers include having the secret token be `gitalysecret`, +and that your GitLab installation has three repository storages: - `default` on Gitaly 1 - `storage1` on Gitaly 1 @@ -1120,11 +1106,11 @@ your GitLab installation has three repository storages: On each node: -1. [Download/Install](https://about.gitlab.com/install/) the Omnibus GitLab - package you want using **steps 1 and 2** from the GitLab downloads page but - **without** providing the `EXTERNAL_URL` value. -1. Edit `/etc/gitlab/gitlab.rb` to configure storage paths, enable - the network listener and configure the token: +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to follow _only_ installation steps 1 and 2 + on the page, and _do not_ provide the `EXTERNAL_URL` value. +1. Edit `/etc/gitlab/gitlab.rb` to configure the storage paths, enable + the network listener, and configure the token: <!-- updates to following example must also be made at @@ -1188,39 +1174,39 @@ On each node: ``` 1. Append the following to `/etc/gitlab/gitlab.rb` for each respective server: - 1. On `gitaly1.internal`: - - ```ruby - git_data_dirs({ - 'default' => { - 'path' => '/var/opt/gitlab/git-data' - }, - 'storage1' => { - 'path' => '/mnt/gitlab/git-data' - }, - }) - ``` - - 1. On `gitaly2.internal`: - - ```ruby - git_data_dirs({ - 'storage2' => { - 'path' => '/mnt/gitlab/git-data' - }, - }) - ``` + - On `gitaly1.internal`: + + ```ruby + git_data_dirs({ + 'default' => { + 'path' => '/var/opt/gitlab/git-data' + }, + 'storage1' => { + 'path' => '/mnt/gitlab/git-data' + }, + }) + ``` + + - On `gitaly2.internal`: + + ```ruby + git_data_dirs({ + 'storage2' => { + 'path' => '/mnt/gitlab/git-data' + }, + }) + ``` <!-- updates to following example must also be made at https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab --> -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). 1. Confirm that Gitaly can perform callbacks to the internal API: ```shell - sudo /opt/gitlab/embedded/service/gitlab-shell/bin/check -config /opt/gitlab/embedded/service/gitlab-shell/config.yml + sudo /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml ``` 1. Verify the GitLab services are running: @@ -1258,11 +1244,10 @@ Name. If you are addressing the Gitaly server by its IP address, you must add it as a Subject Alternative Name to the certificate. [gRPC does not support using an IP address as Common Name in a certificate](https://github.com/grpc/grpc/issues/2691). -NOTE: **Note:** -It is possible to configure Gitaly servers with both an -unencrypted listening address `listen_addr` and an encrypted listening -address `tls_listen_addr` at the same time. This allows you to do a -gradual transition from unencrypted to encrypted traffic, if necessary. +It's possible to configure Gitaly servers with both an unencrypted listening +address (`listen_addr`) and an encrypted listening address (`tls_listen_addr`) +at the same time. This allows you to do a gradual transition from unencrypted to +encrypted traffic, if necessary. To configure Gitaly with TLS: @@ -1316,10 +1301,10 @@ The following IPs will be used as an example: To configure the Sidekiq nodes, one each one: -1. SSH into the Sidekiq server. -1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab package -you want using steps 1 and 2 from the GitLab downloads page. -**Do not complete any other steps on the download page.** +1. SSH in to the Sidekiq server. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to follow _only_ installation steps 1 and 2 + on the page. 1. Open `/etc/gitlab/gitlab.rb` with your editor: ```ruby @@ -1430,14 +1415,14 @@ You can also run [multiple Sidekiq processes](../operations/extra_sidekiq_proces ## Configure GitLab Rails -NOTE: **Note:** -In our architectures we run each GitLab Rails node using the Puma webserver -and have its number of workers set to 90% of available CPUs along with four threads. For -nodes that are running Rails with other components the worker value should be reduced -accordingly where we've found 50% achieves a good balance but this is dependent -on workload. - This section describes how to configure the GitLab application (Rails) component. + +In our architecture, we run each GitLab Rails node using the Puma webserver, and +have its number of workers set to 90% of available CPUs, with four threads. For +nodes running Rails with other components, the worker value should be reduced +accordingly. We've determined that a worker value of 50% achieves a good balance, +but this is dependent on workload. + On each node perform the following: 1. If you're [using NFS](#configure-nfs-optional): @@ -1465,10 +1450,10 @@ On each node perform the following: mkdir -p /var/opt/gitlab/.ssh /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/git-data ``` -1. Download/install Omnibus GitLab using **steps 1 and 2** from - [GitLab downloads](https://about.gitlab.com/install/). Do not complete other - steps on the download page. -1. Create/edit `/etc/gitlab/gitlab.rb` and use the following configuration. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to follow _only_ installation steps 1 and 2 + on the page. +1. Create or edit `/etc/gitlab/gitlab.rb` and use the following configuration. To maintain uniformity of links across nodes, the `external_url` on the application server should point to the external URL that users will use to access GitLab. This would be the URL of the [external load balancer](#configure-the-external-load-balancer) @@ -1581,6 +1566,11 @@ On each node perform the following: sudo gitlab-ctl tail gitaly ``` +1. Save the `/etc/gitlab/gitlab-secrets.json` file from one of the two + application nodes and install it on the other application node, the + [Gitaly node](#configure-gitaly) and the [Sidekiq node](#configure-sidekiq) and + [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + 1. Verify the GitLab services are running: ```shell @@ -1599,12 +1589,10 @@ On each node perform the following: run: puma: (pid 4936) 8645s; run: log: (pid 29656) 79161s ``` -NOTE: **Note:** -When you specify `https` in the `external_url`, as in the example -above, GitLab assumes you have SSL certificates in `/etc/gitlab/ssl/`. If -certificates are not present, NGINX will fail to start. See the -[NGINX documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) -for more information. +When you specify `https` in the `external_url`, as in the previous example, +GitLab expects that the SSL certificates are in `/etc/gitlab/ssl/`. If the +certificates aren't present, NGINX will fail to start. For more information, see +the [NGINX documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https). ### GitLab Rails post-configuration @@ -1614,12 +1602,11 @@ for more information. gitlab-rake gitlab:db:configure ``` - NOTE: **Note:** - If you encounter a `rake aborted!` error stating that PgBouncer is failing to connect to - PostgreSQL it may be that your PgBouncer node's IP address is missing from - PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` on your database nodes. See - [PgBouncer error `ERROR: pgbouncer cannot connect to server`](troubleshooting.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) - in the Troubleshooting section before proceeding. + If you encounter a `rake aborted!` error message stating that PgBouncer is + failing to connect to PostgreSQL, it may be that your PgBouncer node's IP + address is missing from PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` + on your database nodes. Before proceeding, see + [PgBouncer error `ERROR: pgbouncer cannot connect to server`](troubleshooting.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server). 1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md). @@ -1635,10 +1622,10 @@ The Omnibus GitLab package can be used to configure a standalone Monitoring node running [Prometheus](../monitoring/prometheus/index.md) and [Grafana](../monitoring/performance/grafana_configuration.md): -1. SSH into the Monitoring node. -1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab - package you want using **steps 1 and 2** from the GitLab downloads page. - Do not complete any other steps on the download page. +1. SSH in to the Monitoring node. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to follow _only_ installation steps 1 and 2 + on the page. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby @@ -1713,28 +1700,44 @@ It's recommended over [NFS](#configure-nfs-optional) and in general it's better in larger setups as object storage is typically much more performant, reliable, and scalable. -Object storage options that GitLab has tested, or is aware of customers using include: +GitLab has been tested on a number of object storage providers: -- SaaS/Cloud solutions such as [Amazon S3](https://aws.amazon.com/s3/), [Google cloud storage](https://cloud.google.com/storage). +- [Amazon S3](https://aws.amazon.com/s3/) +- [Google Cloud Storage](https://cloud.google.com/storage) +- [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces/) +- [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) +- [Openstack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) +- [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) - On-premises hardware and appliances from various storage vendors. -- MinIO. There is [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. - -For configuring GitLab to use Object Storage refer to the following guides -based on what features you intend to use: - -1. Configure [object storage for backups](../../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage). -1. Configure [object storage for job artifacts](../job_artifacts.md#using-object-storage) - including [incremental logging](../job_logs.md#new-incremental-logging-architecture). -1. Configure [object storage for LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage). -1. Configure [object storage for uploads](../uploads.md#using-object-storage). -1. Configure [object storage for merge request diffs](../merge_request_diffs.md#using-object-storage). -1. Configure [object storage for Container Registry](../packages/container_registry.md#use-object-storage) (optional feature). -1. Configure [object storage for Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage) (optional feature). -1. Configure [object storage for packages](../packages/index.md#using-object-storage) (optional feature). **(PREMIUM ONLY)** -1. Configure [object storage for Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature). **(PREMIUM ONLY)** -1. Configure [object storage for Pseudonymizer](../pseudonymizer.md#configuration) (optional feature). **(ULTIMATE ONLY)** -1. Configure [object storage for autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional - for improved performance). -1. Configure [object storage for Terraform state files](../terraform_state.md#using-object-storage). +- MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. + +There are two ways of specifying object storage configuration in GitLab: + +- [Consolidated form](../object_storage.md#consolidated-object-storage-configuration): A single credential is + shared by all supported object types. +- [Storage-specific form](../object_storage.md#storage-specific-configuration): Every object defines its + own object storage [connection and configuration](../object_storage.md#connection-settings). + +Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. + +For configuring object storage in GitLab 13.1 and earlier, or for storage types not +supported by consolidated configuration form, refer to the following guides based +on what features you intend to use: + +|Object storage type|Supported by consolidated configuration?| +|-------------------|----------------------------------------| +| [Backups](../../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage)|No| +| [Job artifacts](../job_artifacts.md#using-object-storage) including archived job logs | Yes | +| [LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage) | Yes | +| [Uploads](../uploads.md#using-object-storage) | Yes | +| [Container Registry](../packages/container_registry.md#use-object-storage) (optional feature) | No | +| [Merge request diffs](../merge_request_diffs.md#using-object-storage) | Yes | +| [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | +| [Packages](../packages/index.md#using-object-storage) (optional feature) | Yes | +| [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature) **(PREMIUM ONLY)** | Yes | +| [Pseudonymizer](../pseudonymizer.md#configuration) (optional feature) **(ULTIMATE ONLY)** | No | +| [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | +| [Terraform state files](../terraform_state.md#using-object-storage) | Yes | Using separate buckets for each data type is the recommended approach for GitLab. @@ -1758,6 +1761,22 @@ work. </a> </div> +## Configure Advanced Search **(STARTER ONLY)** + +You can leverage Elasticsearch and [enable Advanced Search](../../integration/elasticsearch.md) +for faster, more advanced code search across your entire GitLab instance. + +Elasticsearch cluster design and requirements are dependent on your specific +data. For recommended best practices about how to set up your Elasticsearch +cluster alongside your instance, read how to +[choose the optimal cluster configuration](../../integration/elasticsearch.md#guidance-on-choosing-optimal-cluster-configuration). + +<div align="right"> + <a type="button" class="btn btn-default" href="#setup-components"> + Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> + </a> +</div> + ## Configure NFS (optional) [Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly) diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index 3964b8daeb7..8816d0eecf4 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -105,7 +105,7 @@ is the least complex to setup. This provides a point-in-time recovery of a prede > - Supported tiers: [GitLab Starter, Premium, and Ultimate](https://about.gitlab.com/pricing/) This requires separating out GitLab into multiple application nodes with an added -[load balancer](../high_availability/load_balancer.md). The load balancer will distribute traffic +[load balancer](../load_balancer.md). The load balancer will distribute traffic across GitLab application nodes. Meanwhile, each application node connects to a shared file server and database systems on the back end. This way, if one of the application servers fails, the workflow is not interrupted. diff --git a/doc/administration/reply_by_email_postfix_setup.md b/doc/administration/reply_by_email_postfix_setup.md index f950134889d..f4fcbefa403 100644 --- a/doc/administration/reply_by_email_postfix_setup.md +++ b/doc/administration/reply_by_email_postfix_setup.md @@ -71,7 +71,7 @@ The instructions make the assumption that you will be using the email address `i sudo postfix start ``` -1. Send the new `incoming` user a dummy email to test SMTP, by entering the following into the SMTP prompt: +1. Send the new `incoming` user an email to test SMTP, by entering the following into the SMTP prompt: ```plaintext ehlo localhost @@ -112,7 +112,7 @@ The instructions make the assumption that you will be using the email address `i q ``` -1. Log out of the `incoming` account and go back to being `root`: +1. Sign out of the `incoming` account, and go back to being `root`: ```shell logout @@ -164,7 +164,7 @@ Courier, which we will install later to add IMAP authentication, requires mailbo try the above steps again, substituting `heirloom-mailx` for the `mail` command._ -1. Log out of the `incoming` account and go back to being `root`: +1. Sign out of the `incoming` account, and go back to being `root`: ```shell logout @@ -251,7 +251,7 @@ Courier, which we will install later to add IMAP authentication, requires mailbo If you get a `Connection refused` error instead, make sure your firewall is set up to allow inbound traffic on port 25. - 1. Send the `incoming` user a dummy email to test SMTP, by entering the following into the SMTP prompt: + 1. Send the `incoming` user an email to test SMTP, by entering the following into the SMTP prompt: ```plaintext ehlo gitlab.example.com @@ -288,7 +288,7 @@ Courier, which we will install later to add IMAP authentication, requires mailbo q ``` - 1. Log out of the `incoming` account and go back to being `root`: + 1. Sign out of the `incoming` account, and go back to being `root`: ```shell logout diff --git a/doc/administration/repository_storage_types.md b/doc/administration/repository_storage_types.md index 1c036cfe2ac..b272c4b463e 100644 --- a/doc/administration/repository_storage_types.md +++ b/doc/administration/repository_storage_types.md @@ -83,7 +83,7 @@ The "Gitaly relative path" is shown there, for example: This is the path under `/var/opt/gitlab/git-data/repositories/` on a default Omnibus installation. -In a [Rails console](troubleshooting/debug.md#starting-a-rails-console-session), +In a [Rails console](operations/rails_console.md#starting-a-rails-console-session), get this information using either the numeric project ID or the full path: ```ruby @@ -95,7 +95,7 @@ Project.find_by_full_path('group/project').disk_path To translate from a hashed storage path to a project name: -1. Start a [Rails console](troubleshooting/debug.md#starting-a-rails-console-session). +1. Start a [Rails console](operations/rails_console.md#starting-a-rails-console-session). 1. Run the following: ```ruby diff --git a/doc/administration/server_hooks.md b/doc/administration/server_hooks.md index 54b2cd43265..b5e975d397f 100644 --- a/doc/administration/server_hooks.md +++ b/doc/administration/server_hooks.md @@ -156,22 +156,6 @@ NOTE: **Note:** While other environment variables can be passed to server hooks, your application should not rely on them as they can change. -## Transition to Go - -> - Introduced in GitLab 13.2 using feature flags. -> - In GitLab 13.4, `update` Ruby [implementation removed](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/2501). -> - In GitLab 13.4, `post-receive` Go implementation [made default](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/2502). - -The following server hooks have been re-implemented in Go: - -- `pre-receive`, with the Go implementation used by default. To use the Ruby implementation instead, - [disable](feature_flags.md#enable-or-disable-the-feature) the `:gitaly_go_preceive_hook` feature - flag. -- `update`, with Go implementation always used. No Ruby implementation is available. -- `post-receive`, with the Go implementation used by default. To use the Ruby implementation - instead, [disable](feature_flags.md#enable-or-disable-the-feature) the - `:gitaly_go_postreceive_hook` feature flag. - ## Custom error messages > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/5073) in GitLab 8.10. diff --git a/doc/administration/smime_signing_email.md b/doc/administration/smime_signing_email.md index b1e7e349978..6ded7fdd7d0 100644 --- a/doc/administration/smime_signing_email.md +++ b/doc/administration/smime_signing_email.md @@ -75,8 +75,8 @@ extensions), which contain the following in a single encrypted file: - Intermediate certificates (if any) - Private key -In order to export the required files in PEM encoding from the PKCS#12 file, -the `openssl` command can be used: +To export the required files in PEM encoding from the PKCS#12 file, the +`openssl` command can be used: ```shell #-- Extract private key in PEM encoding (no password, unencrypted) diff --git a/doc/administration/snippets/index.md b/doc/administration/snippets/index.md index 95de3b8c183..5e61d20c683 100644 --- a/doc/administration/snippets/index.md +++ b/doc/administration/snippets/index.md @@ -20,8 +20,8 @@ abuse of the feature. The default value is **52428800 Bytes** (50 MB). The content size limit will be applied when a snippet is created or updated. -In order not to break any existing snippets, the limit doesn't have any -effect on them until a snippet is edited again and the content changes. +This limit doesn't affect existing snippets until they're updated and their +content changes. ### Snippets size limit configuration diff --git a/doc/administration/terraform_state.md b/doc/administration/terraform_state.md index 76e54acce16..55e166d2bf7 100644 --- a/doc/administration/terraform_state.md +++ b/doc/administration/terraform_state.md @@ -1,3 +1,9 @@ +--- +stage: Configure +group: Configure +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Terraform state administration (alpha) > [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2673) in GitLab 12.10. diff --git a/doc/administration/troubleshooting/debug.md b/doc/administration/troubleshooting/debug.md index a1b4df9b94e..ef95bdc8602 100644 --- a/doc/administration/troubleshooting/debug.md +++ b/doc/administration/troubleshooting/debug.md @@ -5,27 +5,15 @@ in production. ## Starting a Rails console session -Troubleshooting and debugging your GitLab instance often requires a -[Rails console](https://guides.rubyonrails.org/command_line.html#rails-console). +Troubleshooting and debugging your GitLab instance often requires a Rails console. + +Your type of GitLab installation determines how +[to start a rails console](../operations/rails_console.md). See also: - [GitLab Rails Console Cheat Sheet](gitlab_rails_cheat_sheet.md). - [Navigating GitLab via Rails console](navigating_gitlab_via_rails_console.md). -**For Omnibus installations** - -```shell -sudo gitlab-rails console -``` - -**For installations from source** - -```shell -sudo -u git -H bundle exec rails console -e production -``` - -Kubernetes: the console is in the task-runner pod, refer to our [Kubernetes cheat sheet](kubernetes_cheat_sheet.md#gitlab-specific-kubernetes-information) for details. - ### Enabling Active Record logging You can enable output of Active Record debug logging in the Rails console @@ -98,7 +86,7 @@ sudo -u git -H bundle exec rails runner -e production /path/to/script.rb A common problem is that mails are not being sent for some reason. Suppose you configured an SMTP server, but you're not seeing mail delivered. Here's how to check the settings: -1. Run a [Rails console](#starting-a-rails-console-session). +1. Run a [Rails console](../operations/rails_console.md#starting-a-rails-console-session). 1. Look at the ActionMailer `delivery_method` to make sure it matches what you intended. If you configured SMTP, it should say `:smtp`. If you're using @@ -238,7 +226,7 @@ separate Rails process to debug the issue: 1. Log in to your GitLab account. 1. Copy the URL that is causing problems (e.g. `https://gitlab.com/ABC`). 1. Create a Personal Access Token for your user (Profile Settings -> Access Tokens). -1. Bring up the [GitLab Rails console.](#starting-a-rails-console-session) +1. Bring up the [GitLab Rails console.](../operations/rails_console.md#starting-a-rails-console-session) 1. At the Rails console, run: ```ruby diff --git a/doc/administration/troubleshooting/elasticsearch.md b/doc/administration/troubleshooting/elasticsearch.md index e13261e3074..6de5bb71d75 100644 --- a/doc/administration/troubleshooting/elasticsearch.md +++ b/doc/administration/troubleshooting/elasticsearch.md @@ -164,7 +164,7 @@ Troubleshooting search result issues is rather straight forward on Elasticsearch The first step is to confirm GitLab is using Elasticsearch for the search function. To do this: -1. Confirm the integration is enabled in **Admin Area > Settings > Integrations**. +1. Confirm the integration is enabled in **Admin Area > Settings > General**. 1. Confirm searches utilize Elasticsearch by accessing the rails console (`sudo gitlab-rails console`) and running the following commands: @@ -206,7 +206,7 @@ The best place to start is to determine if the issue is with creating an empty i If it is, check on the Elasticsearch side to determine if the `gitlab-production` (the name for the GitLab index) exists. If it exists, manually delete it on the Elasticsearch side and attempt to recreate it from the -[`recreate_index`](../../integration/elasticsearch.md#gitlab-elasticsearch-rake-tasks) +[`recreate_index`](../../integration/elasticsearch.md#gitlab-advanced-search-rake-tasks) Rake task. If you still encounter issues, try creating an index manually on the Elasticsearch @@ -225,8 +225,8 @@ during the indexing of projects. If errors do occur, they will either stem from If the indexing process does not present errors, you will want to check the status of the indexed projects. You can do this via the following Rake tasks: -- [`sudo gitlab-rake gitlab:elastic:index_projects_status`](../../integration/elasticsearch.md#gitlab-elasticsearch-rake-tasks) (shows the overall status) -- [`sudo gitlab-rake gitlab:elastic:projects_not_indexed`](../../integration/elasticsearch.md#gitlab-elasticsearch-rake-tasks) (shows specific projects that are not indexed) +- [`sudo gitlab-rake gitlab:elastic:index_projects_status`](../../integration/elasticsearch.md#gitlab-advanced-search-rake-tasks) (shows the overall status) +- [`sudo gitlab-rake gitlab:elastic:projects_not_indexed`](../../integration/elasticsearch.md#gitlab-advanced-search-rake-tasks) (shows specific projects that are not indexed) If: diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index 9a23a115765..dbda889d370 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -222,11 +222,12 @@ namespace = Namespace.find_by_full_path("") ```ruby # ID will be the webhook_id -WebHookLog.where(web_hook_id: ID).each_slice(ID) do |slice| - slice.each(&:destroy) -end +hook=WebHook.find(ID) + +WebHooks::DestroyService.new(current_user).execute(hook) -WebHook.find(ID).destroy +#In case the service gets timeout consider removing webhook_logs +hook.web_hook_logs.limit(BATCH_SIZE).delete_all ``` ### Bulk update service integration password for _all_ projects @@ -285,6 +286,16 @@ GitlabShellWorker.perform_in(0, :remove_repository, p.repository_storage, p.wiki p.create_wiki ### creates the wiki project on the filesystem ``` +## Issue boards + +### In case of issue boards not loading properly and it's getting time out. We need to call the Issue Rebalancing service to fix this + +```ruby +p=Project.find_by_full_path('PROJECT PATH') + +IssueRebalancingService.new(p.issues.take).execute +``` + ## Imports / Exports ```ruby @@ -411,6 +422,9 @@ user.skip_reconfirmation! # Active users on the instance, now User.active.count +# Users taking a seat on the instance +License.current.current_active_users_count + # The historical max on the instance as of the past year ::HistoricalData.max_historical_user_count ``` @@ -506,6 +520,54 @@ group = Group.find_by_path_or_name('group-name') group.project_creation_level=0 ``` +## SCIM + +### Fixing bad SCIM identities + +```ruby +def delete_bad_scim(email, group_path) + output = "" + u = User.find_by_email(email) + uid = u.id + g = Group.find_by_full_path(group_path) + saml_prov_id = SamlProvider.find_by(group_id: g.id).id + saml = Identity.where(user_id: uid, saml_provider_id: saml_prov_id) + scim = ScimIdentity.where(user_id: uid , group_id: g.id) + if saml[0] + saml_eid = saml[0].extern_uid + output += "%s," % [email] + output += "SAML: %s," % [saml_eid] + if scim[0] + scim_eid = scim[0].extern_uid + output += "SCIM: %s" % [scim_eid] + if saml_eid == scim_eid + output += " Identities matched, not deleted \n" + else + scim[0].destroy + output += " Deleted \n" + end + else + output = "ERROR No SCIM identify found for: [%s]\n" % [email] + puts output + return 1 + end + else + output = "ERROR No SAML identify found for: [%s]\n" % [email] + puts output + return 1 + end + puts output + return 0 +end + +# In case of multiple emails +emails = [email1, email2] + +emails.each do |e| + delete_bad_scim(e,'GROUPPATH') +end +``` + ## Routes ### Remove redirecting routes @@ -525,8 +587,8 @@ conflicting_permanent_redirects.destroy_all ### Close a merge request properly (if merged but still marked as open) ```ruby -p = Project.find_by_full_path('') -m = project.merge_requests.find_by(iid: ) +p = Project.find_by_full_path('<full/path/to/project>') +m = p.merge_requests.find_by(iid: <iid>) u = User.find_by_username('') MergeRequests::PostMergeService.new(p, u).execute(m) ``` diff --git a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md index 01532032b49..7c5a9e0d79f 100644 --- a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md +++ b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md @@ -84,8 +84,7 @@ and they will assist you with any issues you are having. ## GitLab-specific Kubernetes information -- Minimal config that can be used to test a Kubernetes Helm chart can be found - [here](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/620). +- Minimal configuration that can be used to [test a Kubernetes Helm chart](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/620). - Tailing logs of a separate pod. An example for a Webservice pod: @@ -190,7 +189,7 @@ and they will assist you with any issues you are having. be possible to use [Updating GitLab using the Helm Chart](https://docs.gitlab.com/charts/index.html#updating-gitlab-using-the-helm-chart) for upgrades. -- How to apply changes to GitLab config: +- How to apply changes to GitLab configuration: - Modify the `gitlab.yaml` file. - Run the following command to apply changes: @@ -255,7 +254,7 @@ to those documents for details. helm install gitlab -f <path-to-yaml-file> gitlab/gitlab ``` - If you want to modify some GitLab settings, you can use the above-mentioned config + If you want to modify some GitLab settings, you can use the above-mentioned configuration as a base and create your own YAML file. - Monitor the installation progress via `helm status gitlab` and `minikube dashboard`. diff --git a/doc/administration/troubleshooting/log_parsing.md b/doc/administration/troubleshooting/log_parsing.md index dcd1df2f423..7914628a756 100644 --- a/doc/administration/troubleshooting/log_parsing.md +++ b/doc/administration/troubleshooting/log_parsing.md @@ -3,11 +3,11 @@ We recommend using log aggregation and search tools like Kibana and Splunk whenever possible, but if they are not available you can still quickly parse [GitLab logs](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26311) in JSON format -(the default since GitLab 12.0) using [`jq`](https://stedolan.github.io/jq/). +(the default in GitLab 12.0 and later) using [`jq`](https://stedolan.github.io/jq/). ## What is JQ? -As noted in its [manual](https://stedolan.github.io/jq/manual/), jq is a command-line JSON processor. The following examples +As noted in its [manual](https://stedolan.github.io/jq/manual/), `jq` is a command-line JSON processor. The following examples include use cases targeted for parsing GitLab log files. ## Parsing Logs diff --git a/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md b/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md index 571973c12d9..a1485249b0e 100644 --- a/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md +++ b/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md @@ -3,7 +3,7 @@ At the heart of GitLab is a web application [built using the Ruby on Rails framework](https://about.gitlab.com/blog/2018/10/29/why-we-use-rails-to-build-gitlab/). Thanks to this, we also get access to the amazing tools built right into Rails. -In this guide, we'll introduce the [Rails console](debug.md#starting-a-rails-console-session) +In this guide, we'll introduce the [Rails console](../operations/rails_console.md#starting-a-rails-console-session) and the basics of interacting with your GitLab instance from the command line. CAUTION: **Caution:** @@ -20,20 +20,10 @@ Rails experience is helpful to have but not a must. ## Starting a Rails console session -Omnibus GitLab comes with a convenient wrapper command which automatically loads -the production GitLab environment: +Your type of GitLab installation determines how +[to start a rails console](../operations/rails_console.md). -```shell -sudo gitlab-rails console -``` - -For source installations, you'll have to instead run: - -```shell -sudo -u git -H bundle exec rails console -e production -``` - -Further code examples will all take place inside the Rails console and also +The following code examples will all take place inside the Rails console and also assume an Omnibus GitLab installation. ## Active Record objects diff --git a/doc/administration/troubleshooting/sidekiq.md b/doc/administration/troubleshooting/sidekiq.md index 404e806c5d9..b7762f8ac3e 100644 --- a/doc/administration/troubleshooting/sidekiq.md +++ b/doc/administration/troubleshooting/sidekiq.md @@ -7,7 +7,7 @@ may be filling up. Users will notice when this happens because new branches may not show up and merge requests may not be updated. The following are some troubleshooting steps that will help you diagnose the bottleneck. -NOTE **Note:** +NOTE: **Note:** GitLab administrators/users should consider working through these debug steps with GitLab Support so the backtraces can be analyzed by our team. It may reveal a bug or necessary improvement in GitLab. diff --git a/doc/administration/troubleshooting/ssl.md b/doc/administration/troubleshooting/ssl.md index e6c081e1eea..4d4b9755fa9 100644 --- a/doc/administration/troubleshooting/ssl.md +++ b/doc/administration/troubleshooting/ssl.md @@ -23,7 +23,7 @@ After configuring a GitLab instance with an internal CA certificate, you might n More details here: https://curl.haxx.se/docs/sslcerts.html ``` -- Testing via the [rails console](debug.md#starting-a-rails-console-session) also fails: +- Testing via the [rails console](../operations/rails_console.md#starting-a-rails-console-session) also fails: ```ruby uri = URI.parse("https://gitlab.domain.tld") @@ -205,6 +205,6 @@ Some of these errors come from the Excon Ruby gem, and could be generated in cir where GitLab is configured to initiate an HTTPS session to a remote server that is serving just HTTP. -One scenario is that you're using [object storage](../high_availability/object_storage.md) +One scenario is that you're using [object storage](../object_storage.md) which is not served under HTTPS. GitLab is misconfigured and attempts a TLS handshake, but the object storage will respond with plain HTTP. diff --git a/doc/administration/troubleshooting/tracing_correlation_id.md b/doc/administration/troubleshooting/tracing_correlation_id.md index 03c342595a3..ae9ebd90951 100644 --- a/doc/administration/troubleshooting/tracing_correlation_id.md +++ b/doc/administration/troubleshooting/tracing_correlation_id.md @@ -4,7 +4,7 @@ type: reference # Finding relevant log entries with a correlation ID -Since GitLab 11.6, a unique request tracking ID, known as the "correlation ID" has been +In GitLab 11.6 and later, a unique request tracking ID, known as the "correlation ID" has been logged by the GitLab instance for most requests. Each individual request to GitLab gets its own correlation ID, which then gets logged in each GitLab component's logs for that request. This makes it easier to trace behavior in a @@ -122,5 +122,5 @@ If you have done some horizontal scaling in your GitLab infrastructure, then you will need to search across _all_ of your GitLab nodes. You can do this with some sort of log aggregation software like Loki, ELK, Splunk, or others. -You can use a tool like Ansible or PSSH (parellel SSH) that can execute identical commands across your servers in +You can use a tool like Ansible or PSSH (parallel SSH) that can execute identical commands across your servers in parallel, or craft your own solution. diff --git a/doc/administration/uploads.md b/doc/administration/uploads.md index 71a41719003..91de089c45e 100644 --- a/doc/administration/uploads.md +++ b/doc/administration/uploads.md @@ -2,6 +2,35 @@ Uploads represent all user data that may be sent to GitLab as a single file. As an example, avatars and notes' attachments are uploads. Uploads are integral to GitLab functionality, and therefore cannot be disabled. +## Upload parameters + +> - [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/214785) in GitLab 13.5. +> - It's [deployed behind a feature flag](../user/feature_flags.md), enabled by default. +> - It's enabled on GitLab.com. +> - It's recommended for production use. +> - For GitLab self-managed instances, GitLab administrators can opt to disable it. **(CORE ONLY)** + +In 13.5 and later, upload parameters are passed [between Workhorse and GitLab Rails](../development/architecture.md#simplified-component-overview) differently than they +were before. + +This change is deployed behind a feature flag that is **enabled by default**. + +If you experience any issues with upload, +[GitLab administrators with access to the GitLab Rails console](./feature_flags.md) +can opt to disable it. + +To enable it: + +```ruby +Feature.enable(:upload_middleware_jwt_params_handler) +``` + +To disable it: + +```ruby +Feature.disable(:upload_middleware_jwt_params_handler) +``` + ## Using local storage NOTE: **Note:** @@ -58,7 +87,7 @@ This configuration relies on valid AWS credentials to be configured already. [Read more about using object storage with GitLab](object_storage.md). NOTE: **Note:** -We recommend using the [consolidated object storage settings](object_storage.md#consolidated-object-storage-configuration). The following instructions apply to the original config format. +We recommend using the [consolidated object storage settings](object_storage.md#consolidated-object-storage-configuration). The following instructions apply to the original configuration format. ## Object Storage Settings @@ -68,9 +97,9 @@ For source installations the following settings are nested under `uploads:` and |---------|-------------|---------| | `enabled` | Enable/disable object storage | `false` | | `remote_directory` | The bucket name where Uploads will be stored| | -| `direct_upload` | Set to true to remove Puma from the Upload path. Workhorse handles the actual Artifact Upload to Object Storage while Puma does minimal processing to keep track of the upload. There is no need for local shared storage. The option may be removed if support for a single storage type for all files is introduced. Read more on [direct upload](../development/uploads.md#direct-upload). | `false` | -| `background_upload` | Set to false to disable automatic upload. Option may be removed once upload is direct to S3 (if `direct_upload` is set to `true` it will override `background_upload`) | `true` | -| `proxy_download` | Set to true to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` | +| `direct_upload` | Set to `true` to remove Puma from the Upload path. Workhorse handles the actual Artifact Upload to Object Storage while Puma does minimal processing to keep track of the upload. There is no need for local shared storage. The option may be removed if support for a single storage type for all files is introduced. Read more on [direct upload](../development/uploads.md#direct-upload). | `false` | +| `background_upload` | Set to `false` to disable automatic upload. Option may be removed once upload is direct to S3 (if `direct_upload` is set to `true` it will override `background_upload`) | `true` | +| `proxy_download` | Set to `true` to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` | | `connection` | Various connection options described below | | ### Connection settings |
