diff options
Diffstat (limited to 'doc/administration')
77 files changed, 4275 insertions, 2800 deletions
diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md index d5755474c00..9aa533d54f6 100644 --- a/doc/administration/audit_events.md +++ b/doc/administration/audit_events.md @@ -167,7 +167,7 @@ It may make the user interface for your project or audit events very busy, and t to prevent performance degradations on GitLab instances with very high Git write traffic. In an upcoming release, Audit Events for Git push events will be enabled -by default. Follow [#7865](https://gitlab.com/gitlab-org/gitlab/-/issues/7865) for updates. +by default. Follow our [Partitioning strategy for Audit Events epic](https://gitlab.com/groups/gitlab-org/-/epics/3206) for updates. If you still wish to enable **Repository push** events in your instance, follow the steps bellow. diff --git a/doc/administration/auth/img/okta_admin_panel.png b/doc/administration/auth/img/okta_admin_panel.png Binary files differdeleted file mode 100644 index 12e21956715..00000000000 --- a/doc/administration/auth/img/okta_admin_panel.png +++ /dev/null diff --git a/doc/administration/auth/ldap/google_secure_ldap.md b/doc/administration/auth/ldap/google_secure_ldap.md index 2b75d864352..55ccf6653a3 100644 --- a/doc/administration/auth/ldap/google_secure_ldap.md +++ b/doc/administration/auth/ldap/google_secure_ldap.md @@ -20,7 +20,7 @@ The steps below cover: ## Configuring Google LDAP client -1. Navigate to <https://admin.google.com/Dashboard> and sign in as a Google Workspace domain administrator. +1. Go to <https://admin.google.com/Dashboard> and sign in as a Google Workspace domain administrator. 1. Go to **Apps > LDAP > Add Client**. diff --git a/doc/administration/auth/ldap/index.md b/doc/administration/auth/ldap/index.md index 466ae8e108c..0e55efba8ae 100644 --- a/doc/administration/auth/ldap/index.md +++ b/doc/administration/auth/ldap/index.md @@ -180,9 +180,16 @@ production: | `allow_username_or_email_login` | If enabled, GitLab ignores everything after the first `@` in the LDAP username submitted by the user on sign-in. If you are using `uid: 'userPrincipalName'` on ActiveDirectory you need to disable this setting, because the userPrincipalName contains an `@`. | no | boolean | | `block_auto_created_users` | To maintain tight control over the number of billable users on your GitLab installation, enable this setting to keep new users blocked until they have been cleared by an administrator (default: false). | no | boolean | | `base` | Base where we can search for users. | yes | `'ou=people,dc=gitlab,dc=example'` or `'DC=mydomain,DC=com'` | -| `user_filter` | Filter LDAP users. Format: [RFC 4515](https://tools.ietf.org/search/rfc4515) Note: GitLab does not support `omniauth-ldap`'s custom filter syntax. | no | `'(employeeType=developer)'` or `'(&(objectclass=user)(|(samaccountname=momo)(samaccountname=toto)))'` | +| `user_filter` | Filter LDAP users. Format: [RFC 4515](https://tools.ietf.org/search/rfc4515) Note: GitLab does not support `omniauth-ldap`'s custom filter syntax. | no | For examples, read [Examples of user filters](#examples-of-user-filters). | | `lowercase_usernames` | If lowercase_usernames is enabled, GitLab converts the name to lower case. | no | boolean | +#### Examples of user filters + +Some examples of the `user_filter` field syntax: + +- `'(employeeType=developer)'` +- `'(&(objectclass=user)(|(samaccountname=momo)(samaccountname=toto)))'` + ### SSL Configuration Settings **(FREE SELF)** | Setting | Description | Required | Examples | @@ -469,8 +476,8 @@ be mandatory and clients cannot be authenticated with the TLS protocol. ## Multiple LDAP servers **(PREMIUM SELF)** -With GitLab Enterprise Edition Starter, you can configure multiple LDAP servers -that your GitLab instance connects to. +With GitLab, you can configure multiple LDAP servers that your GitLab instance +connects to. To add another LDAP server: @@ -698,8 +705,8 @@ When enabled, the following applies: To enable it you need to: 1. [Enable LDAP](#configuration) -1. Navigate to **Admin Area > Settings -> Visibility and access controls**. -1. Make sure the "Lock memberships to LDAP synchronization" checkbox is enabled. +1. Go to **Admin Area > Settings > Visibility and access controls**. +1. Make sure the **Lock memberships to LDAP synchronization** checkbox is selected. ### Adjusting LDAP group sync schedule **(PREMIUM SELF)** diff --git a/doc/administration/auth/ldap/ldap-troubleshooting.md b/doc/administration/auth/ldap/ldap-troubleshooting.md index 438f591856b..f8360e331b6 100644 --- a/doc/administration/auth/ldap/ldap-troubleshooting.md +++ b/doc/administration/auth/ldap/ldap-troubleshooting.md @@ -191,7 +191,7 @@ have to be taken here: will associate this profile to the LDAP identity. The user can do either of these steps [in their -profile](../../../user/profile/index.md#user-profile) or an administrator can do it. +profile](../../../user/profile/index.md#access-your-user-profile) or an administrator can do it. #### Debug LDAP user filter @@ -330,10 +330,10 @@ things to check to debug the situation. group](index.md#adding-group-links). - Check that the user has an LDAP identity: 1. Sign in to GitLab as an administrator user. - 1. Navigate to **Admin area -> Users**. + 1. Go to **Admin area > Users**. 1. Search for the user - 1. Open the user, by clicking on their name. Do not click 'Edit'. - 1. Navigate to the **Identities** tab. There should be an LDAP identity with + 1. Open the user by clicking their name. Do not click **Edit**. + 1. Select the **Identities** tab. There should be an LDAP identity with an LDAP DN as the 'Identifier'. If not, this user hasn't signed in with LDAP yet and must do so first. - You've waited an hour or [the configured diff --git a/doc/administration/auth/smartcard.md b/doc/administration/auth/smartcard.md index dfeee5e7ac4..39c47c6c495 100644 --- a/doc/administration/auth/smartcard.md +++ b/doc/administration/auth/smartcard.md @@ -117,9 +117,15 @@ attribute. As a prerequisite, you must use an LDAP server that: ```ruby gitlab_rails['smartcard_enabled'] = true gitlab_rails['smartcard_ca_file'] = "/etc/ssl/certs/CA.pem" + gitlab_rails['smartcard_client_certificate_required_host'] = "smartcard.example.com" gitlab_rails['smartcard_client_certificate_required_port'] = 3444 ``` + NOTE: **Note** + Assign a value to at least one of the following variables: + gitlab_rails['smartcard_client_certificate_required_host'] or + gitlab_rails['smartcard_client_certificate_required_port']. + 1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab for the changes to take effect. diff --git a/doc/administration/compliance.md b/doc/administration/compliance.md index 721b0dbb957..04369a76fb3 100644 --- a/doc/administration/compliance.md +++ b/doc/administration/compliance.md @@ -20,7 +20,7 @@ relevant compliance standards. |**[Enforce TOS acceptance](../user/admin_area/settings/terms.md)**<br>Enforce your users accepting new terms of service by blocking GitLab traffic.|Free+||Instance| |**[Email all users of a project, group, or entire server](../tools/email.md)**<br>An administrator can email groups of users based on project or group membership, or email everyone using the GitLab instance. This is great for scheduled maintenance or upgrades.|Premium+||Instance| |**[Omnibus package supports log forwarding](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-forwarding)**<br>Forward your logs to a central system.|Premium+||Instance| -|**[Lock project membership to group](../user/group/index.md#member-lock)**<br>Group owners can prevent new members from being added to projects within a group.|Premium+|✓|Group| +|**[Lock project membership to group](../user/group/index.md#prevent-members-from-being-added-to-a-group)**<br>Group owners can prevent new members from being added to projects within a group.|Premium+|✓|Group| |**[LDAP group sync](auth/ldap/index.md#group-sync)**<br>GitLab Enterprise Edition gives administrators the ability to automatically sync groups and manage SSH keys, permissions, and authentication, so you can focus on building your product, not configuring your tools.|Premium+||Instance| |**[LDAP group sync filters](auth/ldap/index.md#group-sync)**<br>GitLab Enterprise Edition Premium gives more flexibility to synchronize with LDAP based on filters, meaning you can leverage LDAP attributes to map GitLab permissions.|Premium+||Instance| |**[Audit events](audit_events.md)**<br>To maintain the integrity of your code, GitLab Enterprise Edition Premium gives administrators the ability to view any modifications made within the GitLab server in an advanced audit events system, so you can control, analyze, and track every change.|Premium+|✓|Instance, Group, Project| diff --git a/doc/administration/consul.md b/doc/administration/consul.md index dfc859e30c2..926267a414a 100644 --- a/doc/administration/consul.md +++ b/doc/administration/consul.md @@ -146,7 +146,7 @@ sudo gitlab-ctl restart consul ### Consul nodes unable to communicate By default, Consul will attempt to -[bind](https://www.consul.io/docs/agent/options.html#_bind) to `0.0.0.0`, but +[bind](https://www.consul.io/docs/agent/options#_bind) to `0.0.0.0`, but it will advertise the first private IP address on the node for other Consul nodes to communicate with it. If the other nodes cannot communicate with a node on this address, then the cluster will have a failed status. diff --git a/doc/administration/feature_flags.md b/doc/administration/feature_flags.md index 55727822654..6882797064b 100644 --- a/doc/administration/feature_flags.md +++ b/doc/administration/feature_flags.md @@ -13,7 +13,7 @@ to deploy features in an early stage of development so that they can be incrementally rolled out. Before making them permanently available, features can be deployed behind -flags for a [number of reasons](../development/feature_flags/index.md#when-to-use-feature-flags), such as: +flags for a [number of reasons](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle#when-to-use-feature-flags), such as: - To test the feature. - To get feedback from users and customers while in an early stage of the development of the feature. diff --git a/doc/administration/file_hooks.md b/doc/administration/file_hooks.md index 1d028b80257..c60f0040496 100644 --- a/doc/administration/file_hooks.md +++ b/doc/administration/file_hooks.md @@ -5,7 +5,7 @@ info: "To determine the technical writer assigned to the Stage/Group associated type: reference --- -# File hooks **(FREE)** +# File hooks **(FREE SELF)** > - Introduced in GitLab 10.6. > - Until GitLab 12.8, the feature name was Plugins. diff --git a/doc/administration/geo/disaster_recovery/background_verification.md b/doc/administration/geo/disaster_recovery/background_verification.md index f573b64b5f1..8d3745130bd 100644 --- a/doc/administration/geo/disaster_recovery/background_verification.md +++ b/doc/administration/geo/disaster_recovery/background_verification.md @@ -58,14 +58,14 @@ Feature.enable('geo_repository_verification') ## Repository verification -Navigate to the **Admin Area > Geo** dashboard on the **primary** node and expand +Go to the **Admin Area > Geo** dashboard on the **primary** node and expand the **Verification information** tab for that node to view automatic checksumming status for repositories and wikis. Successes are shown in green, pending work in gray, and failures in red.  -Navigate to the **Admin Area > Geo** dashboard on the **secondary** node and expand +Go to the **Admin Area > Geo** dashboard on the **secondary** node and expand the **Verification information** tab for that node to view automatic verification status for repositories and wikis. As with checksumming, successes are shown in green, pending work in gray, and failures in red. @@ -92,7 +92,7 @@ data. The default and recommended re-verification interval is 7 days, though an interval as short as 1 day can be set. Shorter intervals reduce risk but increase load and vice versa. -Navigate to the **Admin Area > Geo** dashboard on the **primary** node, and +Go to the **Admin Area > Geo** dashboard on the **primary** node, and click the **Edit** button for the **primary** node to customize the minimum re-verification interval: @@ -141,7 +141,7 @@ sudo gitlab-rake geo:verification:wiki:reset If the **primary** and **secondary** nodes have a checksum verification mismatch, the cause may not be apparent. To find the cause of a checksum mismatch: -1. Navigate to the **Admin Area > Overview > Projects** dashboard on the **primary** node, find the +1. Go to the **Admin Area > Overview > Projects** dashboard on the **primary** node, find the project that you want to check the checksum differences and click on the **Edit** button:  @@ -149,7 +149,7 @@ If the **primary** and **secondary** nodes have a checksum verification mismatch 1. On the project administration page get the **Gitaly storage name**, and **Gitaly relative path**:  -1. Navigate to the project's repository directory on both **primary** and **secondary** nodes +1. Go to the project's repository directory on both **primary** and **secondary** nodes (the path is usually `/var/opt/gitlab/git-data/repositories`). Note that if `git_data_dirs` is customized, check the directory layout on your server to be sure. diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md index e64d0d4983e..96c6482e3db 100644 --- a/doc/administration/geo/disaster_recovery/planned_failover.md +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -110,7 +110,7 @@ The maintenance window won't end until Geo replication and verification is completely finished. To keep the window as short as possible, you should ensure these processes are close to 100% as possible during active use. -Navigate to the **Admin Area > Geo** dashboard on the **secondary** node to +Go to the **Admin Area > Geo** dashboard on the **secondary** node to review status. Replicated objects (shown in green) should be close to 100%, and there should be no failures (shown in red). If a large proportion of objects aren't yet replicated (shown in gray), consider giving the node more diff --git a/doc/administration/geo/glossary.md b/doc/administration/geo/glossary.md index e9d57284dd2..98a29c27a89 100644 --- a/doc/administration/geo/glossary.md +++ b/doc/administration/geo/glossary.md @@ -23,7 +23,7 @@ these definitions yet. |---------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------|-------------------------------------------------| | Node | An individual server that runs GitLab either with a specific role or as a whole (e.g. a Rails application node). In a cloud context this can be a specific machine type. | GitLab | instance, server | | Site | One or a collection of nodes running a single GitLab application. A site can be single-node or multi-node. | GitLab | deployment, installation instance | -| Single-node site | A specific configuration of GitLab that uses exactly one node. | GitLab | single-server, single-instance +| Single-node site | A specific configuration of GitLab that uses exactly one node. | GitLab | single-server, single-instance | Multi-node site | A specific configuration of GitLab that uses more than one node. | GitLab | multi-server, multi-instance, high availability | | Primary site | A GitLab site that is configured to be read and writable. There can only be a single primary site. | Geo-specific | Geo deployment, Primary node | | Secondary site(s) | GitLab site that is configured to be read-only. There can be one or more secondary sites. | Geo-specific | Geo deployment, Secondary node | diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md index 154815efa51..7dbb0c78166 100644 --- a/doc/administration/geo/replication/configuration.md +++ b/doc/administration/geo/replication/configuration.md @@ -252,7 +252,7 @@ on the **secondary** node. Geo synchronizes repositories over HTTP/HTTPS, and therefore requires this clone method to be enabled. This is enabled by default, but if converting an existing node to Geo it should be checked: -1. Navigate to **Admin Area > Settings** (`/admin/application_settings/general`) on the **primary** node. +1. Go to **Admin Area > Settings** (`/admin/application_settings/general`) on the **primary** node. 1. Expand "Visibility and access controls". 1. Ensure "Enabled Git access protocols" is set to either "Both SSH and HTTP(S)" or "Only HTTP(S)". diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index f2913dd55ce..dfb460c9f46 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -45,8 +45,8 @@ verification methods: | Blobs | Archived CI build traces _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | | Blobs | Container registry _(file system)_ | Geo with API/Docker API | _Not implemented_ | | Blobs | Container registry _(object storage)_ | Geo with API/Managed/Docker API (*2*) | _Not implemented_ | -| Blobs | Package registry _(file system)_ | Geo with API | _Not implemented_ | -| Blobs | Package registry _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | +| Blobs | Package registry _(file system)_ | Geo with API | SHA256 checksum | +| Blobs | Package registry _(object storage)_ | Geo with API/Managed (*2*) | SHA256 checksum | | Blobs | Versioned Terraform State _(file system)_ | Geo with API | _Not implemented_ | | Blobs | Versioned Terraform State _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | | Blobs | External Merge Request Diffs _(file system)_ | Geo with API | _Not implemented_ | @@ -118,7 +118,7 @@ We use PostgreSQL's own replication functionality to replicate data from the **p We use Redis both as a cache store and to hold persistent data for our background jobs system. Because both use-cases has data that are exclusive to the same Geo node, we don't replicate it between nodes. -Elasticsearch is an optional database, that can enable advanced searching capabilities, like improved Global Search +Elasticsearch is an optional database, that can enable advanced searching capabilities, like improved Advanced Search in both source-code level and user generated content in Issues / Merge-Requests and discussions. Currently it's not supported in Geo. @@ -175,24 +175,24 @@ successfully, you must replicate their data using some other means. | [Application data in PostgreSQL](../../postgresql/index.md) | **Yes** (10.2) | **Yes** (10.2) | No | | | [Project repository](../../../user/project/repository/) | **Yes** (10.2) | **Yes** (10.7) | No | | | [Project wiki repository](../../../user/project/wiki/) | **Yes** (10.2) | **Yes** (10.7) | No | -| [Group wiki repository](../../../user/group/index.md#group-wikis) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/208147) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/208147) | No | | +| [Group wiki repository](../../../user/group/index.md#group-wikis) | [**Yes** (13.10)](https://gitlab.com/gitlab-org/gitlab/-/issues/208147) | No | No | Behind feature flag `geo_group_wiki_repository_replication`, enabled by default | | [Uploads](../../uploads.md) | **Yes** (10.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | No | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing the output between them. | | [LFS objects](../../lfs/index.md) | **Yes** (10.2) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8922) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing the output between them. GitLab versions 11.11.x and 12.0.x are affected by [a bug that prevents any new LFS objects from replicating](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). | -| [Personal snippets](../../../user/snippets.md#personal-snippets) | **Yes** (10.2) | **Yes** (10.2) | No | | -| [Project snippets](../../../user/snippets.md#project-snippets) | **Yes** (10.2) | **Yes** (10.2) | No | | +| [Personal snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | No | | +| [Project snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | No | | | [CI job artifacts (other than Job Logs)](../../../ci/pipelines/job_artifacts.md) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Via Object Storage provider if supported. Native Geo support (Beta) . | Verified only manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing the output between them | | [Job logs](../../job_logs.md) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing the output between them | | [Object pools for forked project deduplication](../../../development/git_object_deduplication.md) | **Yes** | No | No | | | [Container Registry](../../packages/container_registry.md) | **Yes** (12.3) | No | No | Disabled by default. See [instructions](docker_registry.md) to enable. | | [Content in object storage (beta)](object_storage.md) | **Yes** (12.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/13845) | No | | | [Project designs repository](../../../user/project/issues/design_management.md) | **Yes** (12.7) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/32467) | Via Object Storage provider if supported. Native Geo support (Beta). | | -| [Package Registry for npm](../../../user/packages/npm_registry/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | -| [Package Registry for Maven](../../../user/packages/maven_repository/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | -| [Package Registry for Conan](../../../user/packages/conan_repository/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | -| [Package Registry for NuGet](../../../user/packages/nuget_repository/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | -| [Package Registry for PyPI](../../../user/packages/pypi_repository/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | -| [Package Registry for Composer](../../../user/packages/composer_repository/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | -| [Package Registry for generic packages](../../../user/packages/generic_packages/index.md) | **Yes** (13.5) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | +| [Package Registry for npm](../../../user/packages/npm_registry/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | +| [Package Registry for Maven](../../../user/packages/maven_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | +| [Package Registry for Conan](../../../user/packages/conan_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | +| [Package Registry for NuGet](../../../user/packages/nuget_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | +| [Package Registry for PyPI](../../../user/packages/pypi_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | +| [Package Registry for Composer](../../../user/packages/composer_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | +| [Package Registry for generic packages](../../../user/packages/generic_packages/index.md) | **Yes** (13.5) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | | [Versioned Terraform State](../../terraform_state.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_terraform_state_version_replication`, enabled by default | | [External merge request diffs](../../merge_request_diffs.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_merge_request_diff_replication`, enabled by default | | [Versioned snippets](../../../user/snippets.md#versioned-snippets) | [**Yes** (13.7)](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2810) | No | | @@ -201,4 +201,4 @@ successfully, you must replicate their data using some other means. | [GitLab Pages](../../pages/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/589) | No | No | | | [CI Pipeline Artifacts](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/pipeline_artifact.rb) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Persists additional artifacts after a pipeline completes | | [Dependency proxy images](../../../user/packages/dependency_proxy/index.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/259694) | No | No | Blocked on [Geo: Secondary Mimicry](https://gitlab.com/groups/gitlab-org/-/epics/1528). Note that replication of this cache is not needed for Disaster Recovery purposes because it can be recreated from external sources. | -| [Vulnerability Export](../../../user/application_security/vulnerability_report/#export-vulnerabilities) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Not planned because they are ephemeral and sensitive. They can be regenerated on demand. | +| [Vulnerability Export](../../../user/application_security/vulnerability_report/#export-vulnerability-details) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Not planned because they are ephemeral and sensitive. They can be regenerated on demand. | diff --git a/doc/administration/geo/replication/location_aware_git_url.md b/doc/administration/geo/replication/location_aware_git_url.md index 0eea792d374..272b746015b 100644 --- a/doc/administration/geo/replication/location_aware_git_url.md +++ b/doc/administration/geo/replication/location_aware_git_url.md @@ -51,8 +51,8 @@ If you haven't yet set up a Geo _primary_ node and _secondary_ node, see the In a Route53 Hosted Zone, traffic policies can be used to set up a variety of routing configurations. -1. Navigate to the -[Route53 dashboard](https://console.aws.amazon.com/route53/home) and click +1. Go to the +[Route53 dashboard](https://console.aws.amazon.com/route53/home) and select **Traffic policies**.  diff --git a/doc/administration/geo/replication/remove_geo_node.md b/doc/administration/geo/replication/remove_geo_node.md index 26150a6f536..06cf5375f0d 100644 --- a/doc/administration/geo/replication/remove_geo_node.md +++ b/doc/administration/geo/replication/remove_geo_node.md @@ -9,7 +9,7 @@ type: howto **Secondary** nodes can be removed from the Geo cluster using the Geo administration page of the **primary** node. To remove a **secondary** node: -1. Navigate to **Admin Area > Geo** (`/admin/geo/nodes`). +1. Go to **Admin Area > Geo** (`/admin/geo/nodes`). 1. Click the **Remove** button for the **secondary** node you want to remove. 1. Confirm by clicking **Remove** when the prompt appears. diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index 3b1f60a0f3f..079a3713c73 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -428,7 +428,7 @@ To solve this: 1. Log into the secondary Geo node. -1. Back up [the `.git` folder](../../repository_storage_types.md#translating-hashed-storage-paths). +1. Back up [the `.git` folder](../../repository_storage_types.md#translate-hashed-storage-paths). 1. Optional: [Spot-check](../../troubleshooting/log_parsing.md#find-all-projects-affected-by-a-fatal-git-problem)) a few of those IDs whether they indeed correspond diff --git a/doc/administration/geo/replication/version_specific_updates.md b/doc/administration/geo/replication/version_specific_updates.md index be2ce0ac2c0..883e5d44b69 100644 --- a/doc/administration/geo/replication/version_specific_updates.md +++ b/doc/administration/geo/replication/version_specific_updates.md @@ -11,6 +11,15 @@ Review this page for update instructions for your version. These steps accompany the [general steps](updating_the_geo_nodes.md#general-update-steps) for updating Geo nodes. +## Updating to GitLab 13.9 + +We've detected an issue [with a column rename](https://gitlab.com/gitlab-org/gitlab/-/issues/324160) +that may prevent upgrades to GitLab 13.9.0, 13.9.1, 13.9.2 and 13.9.3. +We are working on a patch and recommend delaying any upgrade attempt until a fixed version +is released. + +More details are available [in this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/324160). + ## Updating to GitLab 13.7 We've detected an issue with the `FetchRemove` call used by Geo secondaries. diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md index 1272e7d1419..7128d4283d1 100644 --- a/doc/administration/geo/setup/database.md +++ b/doc/administration/geo/setup/database.md @@ -496,7 +496,7 @@ A production-ready and secure setup requires at least three Consul nodes, three Patroni nodes, one internal load-balancing node on the primary site, and a similar configuration for the secondary site. The internal load balancer provides a single endpoint for connecting to the Patroni cluster's leader whenever a new leader is -elected. Be sure to use [password credentials](../..//postgresql/replication_and_failover.md#database-authorization-for-patroni) and other database best practices. +elected. Be sure to use [password credentials](../../postgresql/replication_and_failover.md#database-authorization-for-patroni) and other database best practices. Similar to `repmgr`, using Patroni on a secondary node is optional. diff --git a/doc/administration/geo/setup/external_database.md b/doc/administration/geo/setup/external_database.md index 8e7d8049467..1b0082687e6 100644 --- a/doc/administration/geo/setup/external_database.md +++ b/doc/administration/geo/setup/external_database.md @@ -49,7 +49,7 @@ developed and tested. We aim to be compatible with most external To set up an external database, you can either: -- Set up streaming replication yourself (for example, in AWS RDS). +- Set up [streaming replication](https://www.postgresql.org/docs/11/warm-standby.html#STREAMING-REPLICATION-SLOTS) yourself (for example AWS RDS, bare metal not managed by Omnibus, etc.). - Perform the Omnibus configuration manually as follows. #### Leverage your cloud provider's tools to replicate the primary database diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md new file mode 100644 index 00000000000..7e3647d1e34 --- /dev/null +++ b/doc/administration/gitaly/configure_gitaly.md @@ -0,0 +1,943 @@ +--- +stage: Create +group: Gitaly +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: reference +--- + +# Configure Gitaly + +The Gitaly service itself is configured by using a [TOML configuration file](reference.md). + +To change Gitaly settings: + +**For Omnibus GitLab** + +1. Edit `/etc/gitlab/gitlab.rb` and add or change the + [Gitaly settings](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/1dd07197c7e5ae23626aad5a4a070a800b670380/files/gitlab-config-template/gitlab.rb.template#L1622-1676). +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +**For installations from source** + +1. Edit `/home/git/gitaly/config.toml` and add or change the [Gitaly settings](https://gitlab.com/gitlab-org/gitaly/blob/master/config.toml.example). +1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). + +The following configuration options are also available: + +- Enabling [TLS support](#enable-tls-support). +- Configuring the [number of `gitaly-ruby` workers](#configure-number-of-gitaly-ruby-workers). +- Limiting [RPC concurrency](#limit-rpc-concurrency). + +## Run Gitaly on its own server + +By default, Gitaly is run on the same server as Gitaly clients and is +[configured as above](#configure-gitaly). Single-server installations are best served by +this default configuration used by: + +- [Omnibus GitLab](https://docs.gitlab.com/omnibus/). +- The GitLab [source installation guide](../../install/installation.md). + +However, Gitaly can be deployed to its own server, which can benefit GitLab installations that span +multiple machines. + +NOTE: +When configured to run on their own servers, Gitaly servers +[must be upgraded](https://docs.gitlab.com/omnibus/update/#upgrading-gitaly-servers) before Gitaly +clients in your cluster. + +The process for setting up Gitaly on its own server is: + +1. [Install Gitaly](#install-gitaly). +1. [Configure authentication](#configure-authentication). +1. [Configure Gitaly servers](#configure-gitaly-servers). +1. [Configure Gitaly clients](#configure-gitaly-clients). +1. [Disable Gitaly where not required](#disable-gitaly-where-not-required-optional) (optional). + +When running Gitaly on its own server, note the following regarding GitLab versions: + +- From GitLab 11.4, Gitaly was able to serve all Git requests without requiring a shared NFS mount + for Git repository data, except for the + [Elasticsearch indexer](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). +- From GitLab 11.8, the Elasticsearch indexer also uses Gitaly for data access. NFS can still be + leveraged for redundancy on block-level Git data, but should be mounted only on the Gitaly + servers. +- From GitLab 11.8 to 12.2, it is possible to use Elasticsearch in a Gitaly setup that doesn't use + NFS. To use Elasticsearch in these versions, the + [repository indexer](../../integration/elasticsearch.md#elasticsearch-repository-indexer) + must be enabled in your GitLab configuration. +- [In GitLab 12.3 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/6481), the new indexer is + the default and no configuration is required. + +### Network architecture + +The following list depicts the network architecture of Gitaly: + +- GitLab Rails shards repositories into [repository storages](../repository_storage_paths.md). +- `/config/gitlab.yml` contains a map from storage names to `(Gitaly address, Gitaly token)` pairs. +- The `storage name` -\> `(Gitaly address, Gitaly token)` map in `/config/gitlab.yml` is the single + source of truth for the Gitaly network topology. +- A `(Gitaly address, Gitaly token)` corresponds to a Gitaly server. +- A Gitaly server hosts one or more storages. +- A Gitaly client can use one or more Gitaly servers. +- Gitaly addresses must be specified in such a way that they resolve correctly for **all** Gitaly + clients. +- Gitaly clients are: + - Puma or Unicorn. + - Sidekiq. + - GitLab Workhorse. + - GitLab Shell. + - Elasticsearch indexer. + - Gitaly itself. +- A Gitaly server must be able to make RPC calls **to itself** by using its own + `(Gitaly address, Gitaly token)` pair as specified in `/config/gitlab.yml`. +- Authentication is done through a static token which is shared among the Gitaly and GitLab Rails + nodes. + +The following digraph illustrates communication between Gitaly servers and GitLab Rails showing +the default ports for HTTP and HTTPs communication. + + + +WARNING: +Gitaly servers must not be exposed to the public internet as Gitaly's network traffic is unencrypted +by default. The use of firewall is highly recommended to restrict access to the Gitaly server. +Another option is to [use TLS](#enable-tls-support). + +In the following sections, we describe how to configure two Gitaly servers with secret token +`abc123secret`: + +- `gitaly1.internal`. +- `gitaly2.internal`. + +We assume your GitLab installation has three repository storages: + +- `default`. +- `storage1`. +- `storage2`. + +You can use as few as one server with one repository storage if desired. + +NOTE: +The token referred to throughout the Gitaly documentation is just an arbitrary password selected by +the administrator. It is unrelated to tokens created for the GitLab API or other similar web API +tokens. + +### Install Gitaly + +Install Gitaly on each Gitaly server using either Omnibus GitLab or install it from source: + +- For Omnibus GitLab, [download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package you want but **do not** provide the `EXTERNAL_URL=` value. +- To install from source, follow the steps at + [Install Gitaly](../../install/installation.md#install-gitaly). + +### Configure authentication + +Gitaly and GitLab use two shared secrets for authentication: + +- One to authenticate gRPC requests to Gitaly. +- A second for authentication callbacks from GitLab Shell to the GitLab internal API. + +**For Omnibus GitLab** + +To configure the Gitaly token: + +1. On the Gitaly clients, edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['gitaly_token'] = 'abc123secret' + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. On the Gitaly server, edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitaly['auth_token'] = 'abc123secret' + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +There are two ways to configure the GitLab Shell token. + +Method 1: + +1. Copy `/etc/gitlab/gitlab-secrets.json` from the Gitaly client to same path on the Gitaly servers + (and any other Gitaly clients). +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) on Gitaly servers. + +Method 2: + +1. On the Gitaly clients, edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_shell['secret_token'] = 'shellsecret' + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. On the Gitaly servers, edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_shell['secret_token'] = 'shellsecret' + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +**For installations from source** + +1. Copy `/home/git/gitlab/.gitlab_shell_secret` from the Gitaly client to the same path on the + Gitaly servers (and any other Gitaly clients). +1. On the Gitaly clients, edit `/home/git/gitlab/config/gitlab.yml`: + + ```yaml + gitlab: + gitaly: + token: 'abc123secret' + ``` + +1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). +1. On the Gitaly servers, edit `/home/git/gitaly/config.toml`: + + ```toml + [auth] + token = 'abc123secret' + ``` + +1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). + +### Configure Gitaly servers + +On the Gitaly servers, you must configure storage paths and enable the network listener. +The Gitaly server must be able to read, write, and set permissions on the configured path. + +If you want to reduce the risk of downtime when you enable authentication, you can temporarily +disable enforcement. For more information, see the documentation on configuring +[Gitaly authentication](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/configuration/README.md#authentication). + +**For Omnibus GitLab** + +1. Edit `/etc/gitlab/gitlab.rb`: + + <!-- + updates to following example must also be made at + https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab + --> + + ```ruby + # Avoid running unnecessary services on the Gitaly server + postgresql['enable'] = false + redis['enable'] = false + nginx['enable'] = false + puma['enable'] = false + sidekiq['enable'] = false + gitlab_workhorse['enable'] = false + grafana['enable'] = false + gitlab_exporter['enable'] = false + + # If you run a separate monitoring node you can disable these services + alertmanager['enable'] = false + prometheus['enable'] = false + + # If you don't run a separate monitoring node you can + # enable Prometheus access & disable these extra services. + # This makes Prometheus listen on all interfaces. You must use firewalls to restrict access to this address/port. + # prometheus['listen_address'] = '0.0.0.0:9090' + # prometheus['monitor_kubernetes'] = false + + # If you don't want to run monitoring services uncomment the following (not recommended) + # node_exporter['enable'] = false + + # Prevent database connections during 'gitlab-ctl reconfigure' + gitlab_rails['rake_cache_clear'] = false + gitlab_rails['auto_migrate'] = false + + # Configure the gitlab-shell API callback URL. Without this, `git push` will + # fail. This can be your 'front door' GitLab URL or an internal load + # balancer. + # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from Gitaly client to Gitaly server. + gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' + + # Authentication token to ensure only authorized servers can communicate with + # Gitaly server + gitaly['auth_token'] = 'AUTH_TOKEN' + + # Make Gitaly accept connections on all network interfaces. You must use + # firewalls to restrict access to this address/port. + # Comment out following line if you only want to support TLS connections + gitaly['listen_addr'] = "0.0.0.0:8075" + ``` + +1. Append the following to `/etc/gitlab/gitlab.rb` for each respective Gitaly server: + + <!-- + updates to following example must also be made at + https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab + --> + + On `gitaly1.internal`: + + ```ruby + git_data_dirs({ + 'default' => { + 'path' => '/var/opt/gitlab/git-data' + }, + 'storage1' => { + 'path' => '/mnt/gitlab/git-data' + }, + }) + ``` + + On `gitaly2.internal`: + + ```ruby + git_data_dirs({ + 'storage2' => { + 'path' => '/srv/gitlab/git-data' + }, + }) + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Run `sudo /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml` + to confirm that Gitaly can perform callbacks to the GitLab internal API. + +**For installations from source** + +1. Edit `/home/git/gitaly/config.toml`: + + ```toml + listen_addr = '0.0.0.0:8075' + + internal_socket_dir = '/var/opt/gitlab/gitaly' + + [logging] + format = 'json' + level = 'info' + dir = '/var/log/gitaly' + ``` + +1. Append the following to `/home/git/gitaly/config.toml` for each respective Gitaly server: + + On `gitaly1.internal`: + + ```toml + [[storage]] + name = 'default' + path = '/var/opt/gitlab/git-data/repositories' + + [[storage]] + name = 'storage1' + path = '/mnt/gitlab/git-data/repositories' + ``` + + On `gitaly2.internal`: + + ```toml + [[storage]] + name = 'storage2' + path = '/srv/gitlab/git-data/repositories' + ``` + +1. Edit `/home/git/gitlab-shell/config.yml`: + + ```yaml + gitlab_url: https://gitlab.example.com + ``` + +1. Save the files and [restart GitLab](../restart_gitlab.md#installations-from-source). +1. Run `sudo -u git /home/git/gitaly/gitaly-hooks check /home/git/gitaly/config.toml` + to confirm that Gitaly can perform callbacks to the GitLab internal API. + +### Configure Gitaly clients + +As the final step, you must update Gitaly clients to switch from using local Gitaly service to use +the Gitaly servers you just configured. + +This can be risky because anything that prevents your Gitaly clients from reaching the Gitaly +servers causes all Gitaly requests to fail. For example, any sort of network, firewall, or name +resolution problems. + +Additionally, you must [disable Rugged](../nfs.md#improving-nfs-performance-with-gitlab) +if previously enabled manually. + +Gitaly makes the following assumptions: + +- Your `gitaly1.internal` Gitaly server can be reached at `gitaly1.internal:8075` from your Gitaly + clients, and that Gitaly server can read, write, and set permissions on `/var/opt/gitlab/git-data` and + `/mnt/gitlab/git-data`. +- Your `gitaly2.internal` Gitaly server can be reached at `gitaly2.internal:8075` from your Gitaly + clients, and that Gitaly server can read, write, and set permissions on `/srv/gitlab/git-data`. +- Your `gitaly1.internal` and `gitaly2.internal` Gitaly servers can reach each other. + +You can't define Gitaly servers with some as a local Gitaly server +(without `gitaly_address`) and some as remote +server (with `gitaly_address`) unless you use +[mixed configuration](#mixed-configuration). + +**For Omnibus GitLab** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + git_data_dirs({ + 'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, + 'storage1' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, + 'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' }, + }) + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Run `sudo gitlab-rake gitlab:gitaly:check` on the Gitaly client (for example, the + Rails application) to confirm it can connect to Gitaly servers. +1. Tail the logs to see the requests: + + ```shell + sudo gitlab-ctl tail gitaly + ``` + +**For installations from source** + +1. Edit `/home/git/gitlab/config/gitlab.yml`: + + ```yaml + gitlab: + repositories: + storages: + default: + gitaly_address: tcp://gitaly1.internal:8075 + path: /some/local/path + storage1: + gitaly_address: tcp://gitaly1.internal:8075 + path: /some/local/path + storage2: + gitaly_address: tcp://gitaly2.internal:8075 + path: /some/local/path + ``` + + NOTE: + `/some/local/path` should be set to a local folder that exists, however no data is stored in + this folder. This requirement is scheduled to be removed when + [this issue](https://gitlab.com/gitlab-org/gitaly/-/issues/1282) is resolved. + +1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). +1. Run `sudo -u git -H bundle exec rake gitlab:gitaly:check RAILS_ENV=production` to confirm the + Gitaly client can connect to Gitaly servers. +1. Tail the logs to see the requests: + + ```shell + tail -f /home/git/gitlab/log/gitaly.log + ``` + +When you tail the Gitaly logs on your Gitaly server, you should see requests coming in. One sure way +to trigger a Gitaly request is to clone a repository from GitLab over HTTP or HTTPS. + +WARNING: +If you have [server hooks](../server_hooks.md) configured, either per repository or globally, you +must move these to the Gitaly servers. If you have multiple Gitaly servers, copy your server hooks +to all Gitaly servers. + +#### Mixed configuration + +GitLab can reside on the same server as one of many Gitaly servers, but doesn't support +configuration that mixes local and remote configuration. The following setup is incorrect, because: + +- All addresses must be reachable from the other Gitaly servers. +- `storage1` is assigned a Unix socket for `gitaly_address` which is + invalid for some of the Gitaly servers. + +```ruby +git_data_dirs({ + 'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, + 'storage1' => { 'path' => '/mnt/gitlab/git-data' }, + 'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' }, +}) +``` + +To combine local and remote Gitaly servers, use an external address for the local Gitaly server. For +example: + +```ruby +git_data_dirs({ + 'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, + # Address of the GitLab server that has Gitaly running on it + 'storage1' => { 'gitaly_address' => 'tcp://gitlab.internal:8075', 'path' => '/mnt/gitlab/git-data' }, + 'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' }, +}) + +# Make Gitaly accept connections on all network interfaces +gitaly['listen_addr'] = "0.0.0.0:8075" + +# Or for TLS +gitaly['tls_listen_addr'] = "0.0.0.0:9999" +gitaly['certificate_path'] = "/etc/gitlab/ssl/cert.pem" +gitaly['key_path'] = "/etc/gitlab/ssl/key.pem" +``` + +`path` can be included only for storage shards on the local Gitaly server. +If it's excluded, default Git storage directory is used for that storage shard. + +### Disable Gitaly where not required (optional) + +If you run Gitaly [as a remote service](#run-gitaly-on-its-own-server), consider +disabling the local Gitaly service that runs on your GitLab server by default, and run it +only where required. + +Disabling Gitaly on the GitLab instance makes sense only when you run GitLab in a custom cluster configuration, where +Gitaly runs on a separate machine from the GitLab instance. Disabling Gitaly on all machines in the cluster is not +a valid configuration (some machines much act as Gitaly servers). + +To disable Gitaly on a GitLab server: + +**For Omnibus GitLab** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitaly['enable'] = false + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +**For installations from source** + +1. Edit `/etc/default/gitlab`: + + ```shell + gitaly_enabled=false + ``` + +1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). + +## Enable TLS support + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/22602) in GitLab 11.8. +> - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/3160) in GitLab 13.6, outgoing TLS connections to GitLab provide client certificates if configured. + +Gitaly supports TLS encryption. To communicate with a Gitaly instance that listens for secure +connections, use the `tls://` URL scheme in the `gitaly_address` of the corresponding +storage entry in the GitLab configuration. + +Gitaly provides the same server certificates as client certificates in TLS +connections to GitLab. This can be used as part of a mutual TLS authentication strategy +when combined with reverse proxies (for example, NGINX) that validate client certificate +to grant access to GitLab. + +You must supply your own certificates as this isn't provided automatically. The certificate +corresponding to each Gitaly server must be installed on that Gitaly server. + +Additionally, the certificate (or its certificate authority) must be installed on all: + +- Gitaly servers. +- Gitaly clients that communicate with it. + +Note the following: + +- The certificate must specify the address you use to access the Gitaly server. You must add the hostname or IP address as a Subject Alternative Name to the certificate. +- You can configure Gitaly servers with both an unencrypted listening address `listen_addr` and an + encrypted listening address `tls_listen_addr` at the same time. This allows you to gradually + transition from unencrypted to encrypted traffic if necessary. + +To configure Gitaly with TLS: + +**For Omnibus GitLab** + +1. Create certificates for Gitaly servers. +1. On the Gitaly clients, copy the certificates (or their certificate authority) into + `/etc/gitlab/trusted-certs`: + + ```shell + sudo cp cert.pem /etc/gitlab/trusted-certs/ + ``` + +1. On the Gitaly clients, edit `git_data_dirs` in `/etc/gitlab/gitlab.rb` as follows: + + ```ruby + git_data_dirs({ + 'default' => { 'gitaly_address' => 'tls://gitaly1.internal:9999' }, + 'storage1' => { 'gitaly_address' => 'tls://gitaly1.internal:9999' }, + 'storage2' => { 'gitaly_address' => 'tls://gitaly2.internal:9999' }, + }) + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. On the Gitaly servers, create the `/etc/gitlab/ssl` directory and copy your key and certificate + there: + + ```shell + sudo mkdir -p /etc/gitlab/ssl + sudo chmod 755 /etc/gitlab/ssl + sudo cp key.pem cert.pem /etc/gitlab/ssl/ + sudo chmod 644 key.pem cert.pem + ``` + +1. Copy all Gitaly server certificates (or their certificate authority) to + `/etc/gitlab/trusted-certs` on all Gitaly servers and clients + so that Gitaly servers and clients trust the certificate when calling into themselves + or other Gitaly servers: + + ```shell + sudo cp cert1.pem cert2.pem /etc/gitlab/trusted-certs/ + ``` + +1. Edit `/etc/gitlab/gitlab.rb` and add: + + <!-- + updates to following example must also be made at + https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab + --> + + ```ruby + gitaly['tls_listen_addr'] = "0.0.0.0:9999" + gitaly['certificate_path'] = "/etc/gitlab/ssl/cert.pem" + gitaly['key_path'] = "/etc/gitlab/ssl/key.pem" + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Verify Gitaly traffic is being served over TLS by + [observing the types of Gitaly connections](#observe-type-of-gitaly-connections). +1. (Optional) Improve security by: + 1. Disabling non-TLS connections by commenting out or deleting `gitaly['listen_addr']` in + `/etc/gitlab/gitlab.rb`. + 1. Saving the file. + 1. [Reconfiguring GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +**For installations from source** + +1. Create certificates for Gitaly servers. +1. On the Gitaly clients, copy the certificates into the system trusted certificates: + + ```shell + sudo cp cert.pem /usr/local/share/ca-certificates/gitaly.crt + sudo update-ca-certificates + ``` + +1. On the Gitaly clients, edit `storages` in `/home/git/gitlab/config/gitlab.yml` as follows: + + ```yaml + gitlab: + repositories: + storages: + default: + gitaly_address: tls://gitaly1.internal:9999 + path: /some/local/path + storage1: + gitaly_address: tls://gitaly1.internal:9999 + path: /some/local/path + storage2: + gitaly_address: tls://gitaly2.internal:9999 + path: /some/local/path + ``` + + NOTE: + `/some/local/path` should be set to a local folder that exists, however no data is stored + in this folder. This requirement is scheduled to be removed when + [Gitaly issue #1282](https://gitlab.com/gitlab-org/gitaly/-/issues/1282) is resolved. + +1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). +1. On the Gitaly servers, create or edit `/etc/default/gitlab` and add: + + ```shell + export SSL_CERT_DIR=/etc/gitlab/ssl + ``` + +1. On the Gitaly servers, create the `/etc/gitlab/ssl` directory and copy your key and certificate there: + + ```shell + sudo mkdir -p /etc/gitlab/ssl + sudo chmod 755 /etc/gitlab/ssl + sudo cp key.pem cert.pem /etc/gitlab/ssl/ + sudo chmod 644 key.pem cert.pem + ``` + +1. Copy all Gitaly server certificates (or their certificate authority) to the system trusted + certificates folder so Gitaly server trusts the certificate when calling into itself or other Gitaly + servers. + + ```shell + sudo cp cert.pem /usr/local/share/ca-certificates/gitaly.crt + sudo update-ca-certificates + ``` + +1. Edit `/home/git/gitaly/config.toml` and add: + + ```toml + tls_listen_addr = '0.0.0.0:9999' + + [tls] + certificate_path = '/etc/gitlab/ssl/cert.pem' + key_path = '/etc/gitlab/ssl/key.pem' + ``` + +1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). +1. Verify Gitaly traffic is being served over TLS by + [observing the types of Gitaly connections](#observe-type-of-gitaly-connections). +1. (Optional) Improve security by: + 1. Disabling non-TLS connections by commenting out or deleting `listen_addr` in + `/home/git/gitaly/config.toml`. + 1. Saving the file. + 1. [Restarting GitLab](../restart_gitlab.md#installations-from-source). + +### Observe type of Gitaly connections + +[Prometheus](../monitoring/prometheus/index.md) can be used observe what type of connections Gitaly +is serving a production environment. Use the following Prometheus query: + +```prometheus +sum(rate(gitaly_connections_total[5m])) by (type) +``` + +## `gitaly-ruby` + +Gitaly was developed to replace the Ruby application code in GitLab. + +To save time and avoid the risk of rewriting existing application logic, we chose to copy some +application code from GitLab into Gitaly. + +To be able to run that code, `gitaly-ruby` was created, which is a "sidecar" process for the main +Gitaly Go process. Some examples of things that are implemented in `gitaly-ruby` are: + +- RPCs that deal with wikis. +- RPCs that create commits on behalf of a user, such as merge commits. + +We recommend: + +- At least 300 MB memory per worker. +- No more than one worker per core. + +NOTE: +`gitaly-ruby` is planned to be eventually removed. To track progress, see the +[Remove the Gitaly-Ruby sidecar](https://gitlab.com/groups/gitlab-org/-/epics/2862) epic. + +### Configure number of `gitaly-ruby` workers + +`gitaly-ruby` has much less capacity than Gitaly implemented in Go. If your Gitaly server has to handle lots of +requests, the default setting of having just one active `gitaly-ruby` sidecar might not be enough. + +If you see `ResourceExhausted` errors from Gitaly, it's very likely that you have not enough +`gitaly-ruby` capacity. + +You can increase the number of `gitaly-ruby` processes on your Gitaly server with the following +settings: + +**For Omnibus GitLab** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + # Default is 2 workers. The minimum is 2; 1 worker is always reserved as + # a passive stand-by. + gitaly['ruby_num_workers'] = 4 + ``` + +1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +**For installations from source** + +1. Edit `/home/git/gitaly/config.toml`: + + ```toml + [gitaly-ruby] + num_workers = 4 + ``` + +1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). + +## Limit RPC concurrency + +Clone traffic can put a large strain on your Gitaly service. The bulk of the work gets done in the +either of the following RPCs: + +- `SSHUploadPack` (for Git SSH). +- `PostUploadPack` (for Git HTTP). + +To prevent such workloads from overwhelming your Gitaly server, you can set concurrency limits in +Gitaly's configuration file. For example: + +```ruby +# in /etc/gitlab/gitlab.rb + +gitaly['concurrency'] = [ + { + 'rpc' => "/gitaly.SmartHTTPService/PostUploadPack", + 'max_per_repo' => 20 + }, + { + 'rpc' => "/gitaly.SSHService/SSHUploadPack", + 'max_per_repo' => 20 + } +] +``` + +This limits the number of in-flight RPC calls for the given RPCs. The limit is applied per +repository. In the example above: + +- Each repository served by the Gitaly server can have at most 20 simultaneous `PostUploadPack` RPC + calls in flight, and the same for `SSHUploadPack`. +- If another request comes in for a repository that has used up its 20 slots, that request gets + queued. + +You can observe the behavior of this queue using the Gitaly logs and Prometheus: + +- In the Gitaly logs, look for the string (or structured log field) `acquire_ms`. Messages that have + this field are reporting about the concurrency limiter. +- In Prometheus, look for the following metrics: + + - `gitaly_rate_limiting_in_progress`. + - `gitaly_rate_limiting_queued`. + - `gitaly_rate_limiting_seconds`. + +NOTE: +Although the name of the Prometheus metric contains `rate_limiting`, it's a concurrency limiter, not +a rate limiter. If a Gitaly client makes 1,000 requests in a row very quickly, concurrency doesn't +exceed 1, and the concurrency limiter has no effect. + +## Background Repository Optimization + +Empty directories and unneeded configuration settings may accumulate in a repository and +slow down Git operations. Gitaly can schedule a daily background task with a maximum duration +to clean up these items and improve performance. + +WARNING: +This is an experimental feature and may place significant load on the host while running. +Make sure to schedule this during off-peak hours and keep the duration short (for example, 30-60 minutes). + +**For Omnibus GitLab** + +Edit `/etc/gitlab/gitlab.rb` and add: + +```ruby +gitaly['daily_maintenance_start_hour'] = 4 +gitaly['daily_maintenance_start_minute'] = 30 +gitaly['daily_maintenance_duration'] = '30m' +gitaly['daily_maintenance_storages'] = ["default"] +``` + +**For installations from source** + +Edit `/home/git/gitaly/config.toml` and add: + +```toml +[daily_maintenance] +start_hour = 4 +start_minute = 30 +duration = '30m' +storages = ["default"] +``` + +## Rotate Gitaly authentication token + +Rotating credentials in a production environment often requires downtime, causes outages, or both. + +However, you can rotate Gitaly credentials without a service interruption. Rotating a Gitaly +authentication token involves: + +- [Verifying authentication monitoring](#verify-authentication-monitoring). +- [Enabling "auth transitioning" mode](#enable-auth-transitioning-mode). +- [Updating Gitaly authentication tokens](#update-gitaly-authentication-token). +- [Ensuring there are no authentication failures](#ensure-there-are-no-authentication-failures). +- [Disabling "auth transitioning" mode](#disable-auth-transitioning-mode). +- [Verifying authentication is enforced](#verify-authentication-is-enforced). + +This procedure also works if you are running GitLab on a single server. In that case, "Gitaly +server" and "Gitaly client" refers to the same machine. + +### Verify authentication monitoring + +Before rotating a Gitaly authentication token, verify that you can monitor the authentication +behavior of your GitLab installation using Prometheus. Use the following Prometheus query: + +```prometheus +sum(rate(gitaly_authentications_total[5m])) by (enforced, status) +``` + +In a system where authentication is configured correctly and where you have live traffic, you +see something like this: + +```prometheus +{enforced="true",status="ok"} 4424.985419441742 +``` + +There may also be other numbers with rate 0. We care only about the non-zero numbers. + +The only non-zero number should have `enforced="true",status="ok"`. If you have other non-zero +numbers, something is wrong in your configuration. + +The `status="ok"` number reflects your current request rate. In the example above, Gitaly is +handling about 4000 requests per second. + +Now that you have established that you can monitor the Gitaly authentication behavior of your GitLab +installation, you can begin the rest of the procedure. + +### Enable "auth transitioning" mode + +Temporarily disable Gitaly authentication on the Gitaly servers by putting them into "auth +transitioning" mode as follows: + +```ruby +# in /etc/gitlab/gitlab.rb +gitaly['auth_transitioning'] = true +``` + +After you have made this change, your [Prometheus query](#verify-authentication-monitoring) +should return something like: + +```prometheus +{enforced="false",status="would be ok"} 4424.985419441742 +``` + +Because `enforced="false"`, it is safe to start rolling out the new token. + +### Update Gitaly authentication token + +To update to a new Gitaly authentication token, on each Gitaly client **and** Gitaly server: + +1. Update the configuration: + + ```ruby + # in /etc/gitlab/gitlab.rb + + gitaly['auth_token'] = '<new secret token>' + ``` + +1. Restart Gitaly: + + ```shell + gitlab-ctl restart gitaly + ``` + +If you run your [Prometheus query](#verify-authentication-monitoring) while this change is +being rolled out, you see non-zero values for the `enforced="false",status="denied"` counter. + +### Ensure there are no authentication failures + +After the new token is set, and all services involved have been restarted, you will +[temporarily see](#verify-authentication-monitoring) a mix of: + +- `status="would be ok"`. +- `status="denied"`. + +After the new token is picked up by all Gitaly clients and Gitaly servers, the +**only non-zero rate** should be `enforced="false",status="would be ok"`. + +### Disable "auth transitioning" mode + +To re-enable Gitaly authentication, disable "auth transitioning" mode. Update the configuration on +your Gitaly servers as follows: + +```ruby +# in /etc/gitlab/gitlab.rb +gitaly['auth_transitioning'] = false +``` + +WARNING: +Without completing this step, you have **no Gitaly authentication**. + +### Verify authentication is enforced + +Refresh your [Prometheus query](#verify-authentication-monitoring). You should now see a similar +result as you did at the start. For example: + +```prometheus +{enforced="true",status="ok"} 4424.985419441742 +``` + +Note that `enforced="true"` means that authentication is being enforced. diff --git a/doc/administration/gitaly/img/gitaly_network_13_9.png b/doc/administration/gitaly/img/gitaly_network_13_9.png Binary files differnew file mode 100644 index 00000000000..e678f6d7abb --- /dev/null +++ b/doc/administration/gitaly/img/gitaly_network_13_9.png diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index f02b9b8fc1a..b314fa85af7 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -40,931 +40,8 @@ The following is a high-level architecture overview of how Gitaly is used. ## Configure Gitaly -The Gitaly service itself is configured by using a [TOML configuration file](reference.md). - -To change Gitaly settings: - -**For Omnibus GitLab** - -1. Edit `/etc/gitlab/gitlab.rb` and add or change the - [Gitaly settings](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/1dd07197c7e5ae23626aad5a4a070a800b670380/files/gitlab-config-template/gitlab.rb.template#L1622-1676). -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). - -**For installations from source** - -1. Edit `/home/git/gitaly/config.toml` and add or change the [Gitaly settings](https://gitlab.com/gitlab-org/gitaly/blob/master/config.toml.example). -1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). - -The following configuration options are also available: - -- Enabling [TLS support](#enable-tls-support). -- Configuring the [number of `gitaly-ruby` workers](#configure-number-of-gitaly-ruby-workers). -- Limiting [RPC concurrency](#limit-rpc-concurrency). - -## Run Gitaly on its own server - -By default, Gitaly is run on the same server as Gitaly clients and is -[configured as above](#configure-gitaly). Single-server installations are best served by -this default configuration used by: - -- [Omnibus GitLab](https://docs.gitlab.com/omnibus/). -- The GitLab [source installation guide](../../install/installation.md). - -However, Gitaly can be deployed to its own server, which can benefit GitLab installations that span -multiple machines. - -NOTE: -When configured to run on their own servers, Gitaly servers -[must be upgraded](https://docs.gitlab.com/omnibus/update/#upgrading-gitaly-servers) before Gitaly -clients in your cluster. - -The process for setting up Gitaly on its own server is: - -1. [Install Gitaly](#install-gitaly). -1. [Configure authentication](#configure-authentication). -1. [Configure Gitaly servers](#configure-gitaly-servers). -1. [Configure Gitaly clients](#configure-gitaly-clients). -1. [Disable Gitaly where not required](#disable-gitaly-where-not-required-optional) (optional). - -When running Gitaly on its own server, note the following regarding GitLab versions: - -- From GitLab 11.4, Gitaly was able to serve all Git requests without requiring a shared NFS mount - for Git repository data, except for the - [Elasticsearch indexer](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). -- From GitLab 11.8, the Elasticsearch indexer also uses Gitaly for data access. NFS can still be - leveraged for redundancy on block-level Git data, but should be mounted only on the Gitaly - servers. -- From GitLab 11.8 to 12.2, it is possible to use Elasticsearch in a Gitaly setup that doesn't use - NFS. To use Elasticsearch in these versions, the - [repository indexer](../../integration/elasticsearch.md#elasticsearch-repository-indexer) - must be enabled in your GitLab configuration. -- [In GitLab 12.3 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/6481), the new indexer is - the default and no configuration is required. - -### Network architecture - -The following list depicts the network architecture of Gitaly: - -- GitLab Rails shards repositories into [repository storages](../repository_storage_paths.md). -- `/config/gitlab.yml` contains a map from storage names to `(Gitaly address, Gitaly token)` pairs. -- The `storage name` -\> `(Gitaly address, Gitaly token)` map in `/config/gitlab.yml` is the single - source of truth for the Gitaly network topology. -- A `(Gitaly address, Gitaly token)` corresponds to a Gitaly server. -- A Gitaly server hosts one or more storages. -- A Gitaly client can use one or more Gitaly servers. -- Gitaly addresses must be specified in such a way that they resolve correctly for **all** Gitaly - clients. -- Gitaly clients are: - - Puma or Unicorn. - - Sidekiq. - - GitLab Workhorse. - - GitLab Shell. - - Elasticsearch indexer. - - Gitaly itself. -- A Gitaly server must be able to make RPC calls **to itself** by using its own - `(Gitaly address, Gitaly token)` pair as specified in `/config/gitlab.yml`. -- Authentication is done through a static token which is shared among the Gitaly and GitLab Rails - nodes. - -WARNING: -Gitaly servers must not be exposed to the public internet as Gitaly's network traffic is unencrypted -by default. The use of firewall is highly recommended to restrict access to the Gitaly server. -Another option is to [use TLS](#enable-tls-support). - -In the following sections, we describe how to configure two Gitaly servers with secret token -`abc123secret`: - -- `gitaly1.internal`. -- `gitaly2.internal`. - -We assume your GitLab installation has three repository storages: - -- `default`. -- `storage1`. -- `storage2`. - -You can use as few as one server with one repository storage if desired. - -NOTE: -The token referred to throughout the Gitaly documentation is just an arbitrary password selected by -the administrator. It is unrelated to tokens created for the GitLab API or other similar web API -tokens. - -### Install Gitaly - -Install Gitaly on each Gitaly server using either Omnibus GitLab or install it from source: - -- For Omnibus GitLab, [download and install](https://about.gitlab.com/install/) the Omnibus GitLab - package you want but **do not** provide the `EXTERNAL_URL=` value. -- To install from source, follow the steps at - [Install Gitaly](../../install/installation.md#install-gitaly). - -### Configure authentication - -Gitaly and GitLab use two shared secrets for authentication: - -- One to authenticate gRPC requests to Gitaly. -- A second for authentication callbacks from GitLab Shell to the GitLab internal API. - -**For Omnibus GitLab** - -To configure the Gitaly token: - -1. On the Gitaly clients, edit `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab_rails['gitaly_token'] = 'abc123secret' - ``` - -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -1. On the Gitaly server, edit `/etc/gitlab/gitlab.rb`: - - ```ruby - gitaly['auth_token'] = 'abc123secret' - ``` - -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). - -There are two ways to configure the GitLab Shell token. - -Method 1: - -1. Copy `/etc/gitlab/gitlab-secrets.json` from the Gitaly client to same path on the Gitaly servers - (and any other Gitaly clients). -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) on Gitaly servers. - -Method 2: - -1. On the Gitaly clients, edit `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab_shell['secret_token'] = 'shellsecret' - ``` - -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -1. On the Gitaly servers, edit `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab_shell['secret_token'] = 'shellsecret' - ``` - -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). - -**For installations from source** - -1. Copy `/home/git/gitlab/.gitlab_shell_secret` from the Gitaly client to the same path on the - Gitaly servers (and any other Gitaly clients). -1. On the Gitaly clients, edit `/home/git/gitlab/config/gitlab.yml`: - - ```yaml - gitlab: - gitaly: - token: 'abc123secret' - ``` - -1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). -1. On the Gitaly servers, edit `/home/git/gitaly/config.toml`: - - ```toml - [auth] - token = 'abc123secret' - ``` - -1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). - -### Configure Gitaly servers - -On the Gitaly servers, you must configure storage paths and enable the network listener. - -If you want to reduce the risk of downtime when you enable authentication, you can temporarily -disable enforcement. For more information, see the documentation on configuring -[Gitaly authentication](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/configuration/README.md#authentication). - -**For Omnibus GitLab** - -1. Edit `/etc/gitlab/gitlab.rb`: - - <!-- - updates to following example must also be made at - https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab - --> - - ```ruby - # /etc/gitlab/gitlab.rb - - # Avoid running unnecessary services on the Gitaly server - postgresql['enable'] = false - redis['enable'] = false - nginx['enable'] = false - puma['enable'] = false - sidekiq['enable'] = false - gitlab_workhorse['enable'] = false - grafana['enable'] = false - gitlab_exporter['enable'] = false - - # If you run a separate monitoring node you can disable these services - alertmanager['enable'] = false - prometheus['enable'] = false - - # If you don't run a separate monitoring node you can - # enable Prometheus access & disable these extra services. - # This makes Prometheus listen on all interfaces. You must use firewalls to restrict access to this address/port. - # prometheus['listen_address'] = '0.0.0.0:9090' - # prometheus['monitor_kubernetes'] = false - - # If you don't want to run monitoring services uncomment the following (not recommended) - # node_exporter['enable'] = false - - # Prevent database connections during 'gitlab-ctl reconfigure' - gitlab_rails['rake_cache_clear'] = false - gitlab_rails['auto_migrate'] = false - - # Configure the gitlab-shell API callback URL. Without this, `git push` will - # fail. This can be your 'front door' GitLab URL or an internal load - # balancer. - # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from Gitaly client to Gitaly server. - gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' - - # Make Gitaly accept connections on all network interfaces. You must use - # firewalls to restrict access to this address/port. - # Comment out following line if you only want to support TLS connections - gitaly['listen_addr'] = "0.0.0.0:8075" - ``` - -1. Append the following to `/etc/gitlab/gitlab.rb` for each respective Gitaly server: - - <!-- - updates to following example must also be made at - https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab - --> - - On `gitaly1.internal`: - - ```ruby - git_data_dirs({ - 'default' => { - 'path' => '/var/opt/gitlab/git-data' - }, - 'storage1' => { - 'path' => '/mnt/gitlab/git-data' - }, - }) - ``` - - On `gitaly2.internal`: - - ```ruby - git_data_dirs({ - 'storage2' => { - 'path' => '/srv/gitlab/git-data' - }, - }) - ``` - -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -1. Run `sudo /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml` - to confirm that Gitaly can perform callbacks to the GitLab internal API. - -**For installations from source** - -1. Edit `/home/git/gitaly/config.toml`: - - ```toml - listen_addr = '0.0.0.0:8075' - - internal_socket_dir = '/var/opt/gitlab/gitaly' - - [logging] - format = 'json' - level = 'info' - dir = '/var/log/gitaly' - ``` - -1. Append the following to `/home/git/gitaly/config.toml` for each respective Gitaly server: - - On `gitaly1.internal`: - - ```toml - [[storage]] - name = 'default' - path = '/var/opt/gitlab/git-data/repositories' - - [[storage]] - name = 'storage1' - path = '/mnt/gitlab/git-data/repositories' - ``` - - On `gitaly2.internal`: - - ```toml - [[storage]] - name = 'storage2' - path = '/srv/gitlab/git-data/repositories' - ``` - -1. Edit `/home/git/gitlab-shell/config.yml`: - - ```yaml - gitlab_url: https://gitlab.example.com - ``` - -1. Save the files and [restart GitLab](../restart_gitlab.md#installations-from-source). -1. Run `sudo -u git /home/git/gitaly/gitaly-hooks check /home/git/gitaly/config.toml` - to confirm that Gitaly can perform callbacks to the GitLab internal API. - -### Configure Gitaly clients - -As the final step, you must update Gitaly clients to switch from using local Gitaly service to use -the Gitaly servers you just configured. - -This can be risky because anything that prevents your Gitaly clients from reaching the Gitaly -servers causes all Gitaly requests to fail. For example, any sort of network, firewall, or name -resolution problems. - -Additionally, you must [disable Rugged](../nfs.md#improving-nfs-performance-with-gitlab) -if previously enabled manually. - -Gitaly makes the following assumptions: - -- Your `gitaly1.internal` Gitaly server can be reached at `gitaly1.internal:8075` from your Gitaly - clients, and that Gitaly server can read, write, and set permissions on `/mnt/gitlab/default` and - `/mnt/gitlab/storage1`. -- Your `gitaly2.internal` Gitaly server can be reached at `gitaly2.internal:8075` from your Gitaly - clients, and that Gitaly server can read, write, and set permissions on `/mnt/gitlab/storage2`. -- Your `gitaly1.internal` and `gitaly2.internal` Gitaly servers can reach each other. - -You can't define Gitaly servers with some as a local Gitaly server -(without `gitaly_address`) and some as remote -server (with `gitaly_address`) unless you use -[mixed configuration](#mixed-configuration). - -**For Omnibus GitLab** - -1. Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - git_data_dirs({ - 'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, - 'storage1' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, - 'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' }, - }) - ``` - -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -1. Run `sudo gitlab-rake gitlab:gitaly:check` on the Gitaly client (for example, the - Rails application) to confirm it can connect to Gitaly servers. -1. Tail the logs to see the requests: - - ```shell - sudo gitlab-ctl tail gitaly - ``` - -**For installations from source** - -1. Edit `/home/git/gitlab/config/gitlab.yml`: - - ```yaml - gitlab: - repositories: - storages: - default: - gitaly_address: tcp://gitaly1.internal:8075 - path: /some/local/path - storage1: - gitaly_address: tcp://gitaly1.internal:8075 - path: /some/local/path - storage2: - gitaly_address: tcp://gitaly2.internal:8075 - path: /some/local/path - ``` - - NOTE: - `/some/local/path` should be set to a local folder that exists, however no data is stored in - this folder. This requirement is scheduled to be removed when - [this issue](https://gitlab.com/gitlab-org/gitaly/-/issues/1282) is resolved. - -1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). -1. Run `sudo -u git -H bundle exec rake gitlab:gitaly:check RAILS_ENV=production` to confirm the - Gitaly client can connect to Gitaly servers. -1. Tail the logs to see the requests: - - ```shell - tail -f /home/git/gitlab/log/gitaly.log - ``` - -When you tail the Gitaly logs on your Gitaly server, you should see requests coming in. One sure way -to trigger a Gitaly request is to clone a repository from GitLab over HTTP or HTTPS. - -WARNING: -If you have [server hooks](../server_hooks.md) configured, either per repository or globally, you -must move these to the Gitaly servers. If you have multiple Gitaly servers, copy your server hooks -to all Gitaly servers. - -#### Mixed configuration - -GitLab can reside on the same server as one of many Gitaly servers, but doesn't support -configuration that mixes local and remote configuration. The following setup is incorrect, because: - -- All addresses must be reachable from the other Gitaly servers. -- `storage1` is assigned a Unix socket for `gitaly_address` which is - invalid for some of the Gitaly servers. - -```ruby -git_data_dirs({ - 'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, - 'storage1' => { 'path' => '/mnt/gitlab/git-data' }, - 'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' }, -}) -``` - -To combine local and remote Gitaly servers, use an external address for the local Gitaly server. For -example: - -```ruby -git_data_dirs({ - 'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, - # Address of the GitLab server that has Gitaly running on it - 'storage1' => { 'gitaly_address' => 'tcp://gitlab.internal:8075', 'path' => '/mnt/gitlab/git-data' }, - 'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' }, -}) - -# Make Gitaly accept connections on all network interfaces -gitaly['listen_addr'] = "0.0.0.0:8075" - -# Or for TLS -gitaly['tls_listen_addr'] = "0.0.0.0:9999" -gitaly['certificate_path'] = "/etc/gitlab/ssl/cert.pem" -gitaly['key_path'] = "/etc/gitlab/ssl/key.pem" -``` - -`path` can be included only for storage shards on the local Gitaly server. -If it's excluded, default Git storage directory is used for that storage shard. - -### Disable Gitaly where not required (optional) - -If you run Gitaly [as a remote service](#run-gitaly-on-its-own-server), consider -disabling the local Gitaly service that runs on your GitLab server by default, and run it -only where required. - -Disabling Gitaly on the GitLab instance makes sense only when you run GitLab in a custom cluster configuration, where -Gitaly runs on a separate machine from the GitLab instance. Disabling Gitaly on all machines in the cluster is not -a valid configuration (some machines much act as Gitaly servers). - -To disable Gitaly on a GitLab server: - -**For Omnibus GitLab** - -1. Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - gitaly['enable'] = false - ``` - -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). - -**For installations from source** - -1. Edit `/etc/default/gitlab`: - - ```shell - gitaly_enabled=false - ``` - -1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). - -## Enable TLS support - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/22602) in GitLab 11.8. -> - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/3160) in GitLab 13.6, outgoing TLS connections to GitLab provide client certificates if configured. - -Gitaly supports TLS encryption. To communicate with a Gitaly instance that listens for secure -connections, use the `tls://` URL scheme in the `gitaly_address` of the corresponding -storage entry in the GitLab configuration. - -Gitaly provides the same server certificates as client certificates in TLS -connections to GitLab. This can be used as part of a mutual TLS authentication strategy -when combined with reverse proxies (for example, NGINX) that validate client certificate -to grant access to GitLab. - -You must supply your own certificates as this isn't provided automatically. The certificate -corresponding to each Gitaly server must be installed on that Gitaly server. - -Additionally, the certificate (or its certificate authority) must be installed on all: - -- Gitaly servers. -- Gitaly clients that communicate with it. - -Note the following: - -- The certificate must specify the address you use to access the Gitaly server. You must add the hostname or IP address as a Subject Alternative Name to the certificate. -- You can configure Gitaly servers with both an unencrypted listening address `listen_addr` and an - encrypted listening address `tls_listen_addr` at the same time. This allows you to gradually - transition from unencrypted to encrypted traffic if necessary. - -To configure Gitaly with TLS: - -**For Omnibus GitLab** - -1. Create certificates for Gitaly servers. -1. On the Gitaly clients, copy the certificates (or their certificate authority) into - `/etc/gitlab/trusted-certs`: - - ```shell - sudo cp cert.pem /etc/gitlab/trusted-certs/ - ``` - -1. On the Gitaly clients, edit `git_data_dirs` in `/etc/gitlab/gitlab.rb` as follows: - - ```ruby - git_data_dirs({ - 'default' => { 'gitaly_address' => 'tls://gitaly1.internal:9999' }, - 'storage1' => { 'gitaly_address' => 'tls://gitaly1.internal:9999' }, - 'storage2' => { 'gitaly_address' => 'tls://gitaly2.internal:9999' }, - }) - ``` - -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -1. On the Gitaly servers, create the `/etc/gitlab/ssl` directory and copy your key and certificate - there: - - ```shell - sudo mkdir -p /etc/gitlab/ssl - sudo chmod 755 /etc/gitlab/ssl - sudo cp key.pem cert.pem /etc/gitlab/ssl/ - sudo chmod 644 key.pem cert.pem - ``` - -1. Copy all Gitaly server certificates (or their certificate authority) to - `/etc/gitlab/trusted-certs` so that Gitaly servers trust the certificate when calling into themselves - or other Gitaly servers: - - ```shell - sudo cp cert1.pem cert2.pem /etc/gitlab/trusted-certs/ - ``` - -1. Edit `/etc/gitlab/gitlab.rb` and add: - - <!-- - updates to following example must also be made at - https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab - --> - - ```ruby - gitaly['tls_listen_addr'] = "0.0.0.0:9999" - gitaly['certificate_path'] = "/etc/gitlab/ssl/cert.pem" - gitaly['key_path'] = "/etc/gitlab/ssl/key.pem" - ``` - -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -1. Verify Gitaly traffic is being served over TLS by - [observing the types of Gitaly connections](#observe-type-of-gitaly-connections). -1. (Optional) Improve security by: - 1. Disabling non-TLS connections by commenting out or deleting `gitaly['listen_addr']` in - `/etc/gitlab/gitlab.rb`. - 1. Saving the file. - 1. [Reconfiguring GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). - -**For installations from source** - -1. Create certificates for Gitaly servers. -1. On the Gitaly clients, copy the certificates into the system trusted certificates: - - ```shell - sudo cp cert.pem /usr/local/share/ca-certificates/gitaly.crt - sudo update-ca-certificates - ``` - -1. On the Gitaly clients, edit `storages` in `/home/git/gitlab/config/gitlab.yml` as follows: - - ```yaml - gitlab: - repositories: - storages: - default: - gitaly_address: tls://gitaly1.internal:9999 - path: /some/local/path - storage1: - gitaly_address: tls://gitaly1.internal:9999 - path: /some/local/path - storage2: - gitaly_address: tls://gitaly2.internal:9999 - path: /some/local/path - ``` - - NOTE: - `/some/local/path` should be set to a local folder that exists, however no data is stored - in this folder. This requirement is scheduled to be removed when - [Gitaly issue #1282](https://gitlab.com/gitlab-org/gitaly/-/issues/1282) is resolved. - -1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). -1. On the Gitaly servers, create or edit `/etc/default/gitlab` and add: - - ```shell - export SSL_CERT_DIR=/etc/gitlab/ssl - ``` - -1. On the Gitaly servers, create the `/etc/gitlab/ssl` directory and copy your key and certificate there: - - ```shell - sudo mkdir -p /etc/gitlab/ssl - sudo chmod 755 /etc/gitlab/ssl - sudo cp key.pem cert.pem /etc/gitlab/ssl/ - sudo chmod 644 key.pem cert.pem - ``` - -1. Copy all Gitaly server certificates (or their certificate authority) to the system trusted - certificates folder so Gitaly server trusts the certificate when calling into itself or other Gitaly - servers. - - ```shell - sudo cp cert.pem /usr/local/share/ca-certificates/gitaly.crt - sudo update-ca-certificates - ``` - -1. Edit `/home/git/gitaly/config.toml` and add: - - ```toml - tls_listen_addr = '0.0.0.0:9999' - - [tls] - certificate_path = '/etc/gitlab/ssl/cert.pem' - key_path = '/etc/gitlab/ssl/key.pem' - ``` - -1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). -1. Verify Gitaly traffic is being served over TLS by - [observing the types of Gitaly connections](#observe-type-of-gitaly-connections). -1. (Optional) Improve security by: - 1. Disabling non-TLS connections by commenting out or deleting `listen_addr` in - `/home/git/gitaly/config.toml`. - 1. Saving the file. - 1. [Restarting GitLab](../restart_gitlab.md#installations-from-source). - -### Observe type of Gitaly connections - -[Prometheus](../monitoring/prometheus/index.md) can be used observe what type of connections Gitaly -is serving a production environment. Use the following Prometheus query: - -```prometheus -sum(rate(gitaly_connections_total[5m])) by (type) -``` - -## `gitaly-ruby` - -Gitaly was developed to replace the Ruby application code in GitLab. - -To save time and avoid the risk of rewriting existing application logic, we chose to copy some -application code from GitLab into Gitaly. - -To be able to run that code, `gitaly-ruby` was created, which is a "sidecar" process for the main -Gitaly Go process. Some examples of things that are implemented in `gitaly-ruby` are: - -- RPCs that deal with wikis. -- RPCs that create commits on behalf of a user, such as merge commits. - -We recommend: - -- At least 300 MB memory per worker. -- No more than one worker per core. - -NOTE: -`gitaly-ruby` is planned to be eventually removed. To track progress, see the -[Remove the Gitaly-Ruby sidecar](https://gitlab.com/groups/gitlab-org/-/epics/2862) epic. - -### Configure number of `gitaly-ruby` workers - -`gitaly-ruby` has much less capacity than Gitaly implemented in Go. If your Gitaly server has to handle lots of -requests, the default setting of having just one active `gitaly-ruby` sidecar might not be enough. - -If you see `ResourceExhausted` errors from Gitaly, it's very likely that you have not enough -`gitaly-ruby` capacity. - -You can increase the number of `gitaly-ruby` processes on your Gitaly server with the following -settings: - -**For Omnibus GitLab** - -1. Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - # Default is 2 workers. The minimum is 2; 1 worker is always reserved as - # a passive stand-by. - gitaly['ruby_num_workers'] = 4 - ``` - -1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). - -**For installations from source** - -1. Edit `/home/git/gitaly/config.toml`: - - ```toml - [gitaly-ruby] - num_workers = 4 - ``` - -1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). - -## Limit RPC concurrency - -Clone traffic can put a large strain on your Gitaly service. The bulk of the work gets done in the -either of the following RPCs: - -- `SSHUploadPack` (for Git SSH). -- `PostUploadPack` (for Git HTTP). - -To prevent such workloads from overwhelming your Gitaly server, you can set concurrency limits in -Gitaly's configuration file. For example: - -```ruby -# in /etc/gitlab/gitlab.rb - -gitaly['concurrency'] = [ - { - 'rpc' => "/gitaly.SmartHTTPService/PostUploadPack", - 'max_per_repo' => 20 - }, - { - 'rpc' => "/gitaly.SSHService/SSHUploadPack", - 'max_per_repo' => 20 - } -] -``` - -This limits the number of in-flight RPC calls for the given RPCs. The limit is applied per -repository. In the example above: - -- Each repository served by the Gitaly server can have at most 20 simultaneous `PostUploadPack` RPC - calls in flight, and the same for `SSHUploadPack`. -- If another request comes in for a repository that has used up its 20 slots, that request gets - queued. - -You can observe the behavior of this queue using the Gitaly logs and Prometheus: - -- In the Gitaly logs, look for the string (or structured log field) `acquire_ms`. Messages that have - this field are reporting about the concurrency limiter. -- In Prometheus, look for the following metrics: - - - `gitaly_rate_limiting_in_progress`. - - `gitaly_rate_limiting_queued`. - - `gitaly_rate_limiting_seconds`. - -NOTE: -Although the name of the Prometheus metric contains `rate_limiting`, it's a concurrency limiter, not -a rate limiter. If a Gitaly client makes 1,000 requests in a row very quickly, concurrency doesn't -exceed 1, and the concurrency limiter has no effect. - -## Background Repository Optimization - -Empty directories and unneeded config settings may accumulate in a repository and -slow down Git operations. Gitaly can schedule a daily background task with a maximum duration -to clean up these items and improve performance. - -WARNING: -This is an experimental feature and may place significant load on the host while running. -Make sure to schedule this during off-peak hours and keep the duration short (for example, 30-60 minutes). - -**For Omnibus GitLab** - -Edit `/etc/gitlab/gitlab.rb` and add: - -```ruby -gitaly['daily_maintenance_start_hour'] = 4 -gitaly['daily_maintenance_start_minute'] = 30 -gitaly['daily_maintenance_duration'] = '30m' -gitaly['daily_maintenance_storages'] = ["default"] -``` - -**For installations from source** - -Edit `/home/git/gitaly/config.toml` and add: - -```toml -[daily_maintenance] -start_hour = 4 -start_minute = 30 -duration = '30m' -storages = ["default"] -``` - -## Rotate Gitaly authentication token - -Rotating credentials in a production environment often requires downtime, causes outages, or both. - -However, you can rotate Gitaly credentials without a service interruption. Rotating a Gitaly -authentication token involves: - -- [Verifying authentication monitoring](#verify-authentication-monitoring). -- [Enabling "auth transitioning" mode](#enable-auth-transitioning-mode). -- [Updating Gitaly authentication tokens](#update-gitaly-authentication-token). -- [Ensuring there are no authentication failures](#ensure-there-are-no-authentication-failures). -- [Disabling "auth transitioning" mode](#disable-auth-transitioning-mode). -- [Verifying authentication is enforced](#verify-authentication-is-enforced). - -This procedure also works if you are running GitLab on a single server. In that case, "Gitaly -server" and "Gitaly client" refers to the same machine. - -### Verify authentication monitoring - -Before rotating a Gitaly authentication token, verify that you can monitor the authentication -behavior of your GitLab installation using Prometheus. Use the following Prometheus query: - -```prometheus -sum(rate(gitaly_authentications_total[5m])) by (enforced, status) -``` - -In a system where authentication is configured correctly and where you have live traffic, you -see something like this: - -```prometheus -{enforced="true",status="ok"} 4424.985419441742 -``` - -There may also be other numbers with rate 0. We care only about the non-zero numbers. - -The only non-zero number should have `enforced="true",status="ok"`. If you have other non-zero -numbers, something is wrong in your configuration. - -The `status="ok"` number reflects your current request rate. In the example above, Gitaly is -handling about 4000 requests per second. - -Now that you have established that you can monitor the Gitaly authentication behavior of your GitLab -installation, you can begin the rest of the procedure. - -### Enable "auth transitioning" mode - -Temporarily disable Gitaly authentication on the Gitaly servers by putting them into "auth -transitioning" mode as follows: - -```ruby -# in /etc/gitlab/gitlab.rb -gitaly['auth_transitioning'] = true -``` - -After you have made this change, your [Prometheus query](#verify-authentication-monitoring) -should return something like: - -```prometheus -{enforced="false",status="would be ok"} 4424.985419441742 -``` - -Because `enforced="false"`, it is safe to start rolling out the new token. - -### Update Gitaly authentication token - -To update to a new Gitaly authentication token, on each Gitaly client **and** Gitaly server: - -1. Update the configuration: - - ```ruby - # in /etc/gitlab/gitlab.rb - - gitaly['auth_token'] = '<new secret token>' - ``` - -1. Restart Gitaly: - - ```shell - gitlab-ctl restart gitaly - ``` - -If you run your [Prometheus query](#verify-authentication-monitoring) while this change is -being rolled out, you see non-zero values for the `enforced="false",status="denied"` counter. - -### Ensure there are no authentication failures - -After the new token is set, and all services involved have been restarted, you will -[temporarily see](#verify-authentication-monitoring) a mix of: - -- `status="would be ok"`. -- `status="denied"`. - -After the new token is picked up by all Gitaly clients and Gitaly servers, the -**only non-zero rate** should be `enforced="false",status="would be ok"`. - -### Disable "auth transitioning" mode - -To re-enable Gitaly authentication, disable "auth transitioning" mode. Update the configuration on -your Gitaly servers as follows: - -```ruby -# in /etc/gitlab/gitlab.rb -gitaly['auth_transitioning'] = false -``` - -WARNING: -Without completing this step, you have **no Gitaly authentication**. - -### Verify authentication is enforced - -Refresh your [Prometheus query](#verify-authentication-monitoring). You should now see a similar -result as you did at the start. For example: - -```prometheus -{enforced="true",status="ok"} 4424.985419441742 -``` - -Note that `enforced="true"` means that authentication is being enforced. +Gitaly comes pre-configured with Omnibus GitLab. For more information on customizing your +Gitaly installation, see [Configure Gitaly](configure_gitaly.md). ## Direct Git access bypassing Gitaly @@ -1059,6 +136,12 @@ To see if GitLab can access the repository file system directly, we use the foll Direct Git access is enable by default in Omnibus GitLab because it fills in the correct repository paths in the GitLab configuration file `config/gitlab.yml`. This satisfies the UUID check. +WARNING: +If directly copying repository data from a GitLab server to Gitaly, ensure that the metadata file, +default path `/var/opt/gitlab/git-data/repositories/.gitaly-metadata`, is not included in the transfer. +Copying this file causes GitLab to use the Rugged patches for repositories hosted on the Gitaly server, +leading to `Error creating pipeline` and `Commit not found` errors, or stale data. + ### Transition to Gitaly Cluster For the sake of removing complexity, we must remove direct Git access in GitLab. However, we can't @@ -1150,7 +233,7 @@ internally pools and re-uses those across RPCs. ### Observing `gitaly-ruby` traffic -[`gitaly-ruby`](#gitaly-ruby) is an internal implementation detail of Gitaly, +[`gitaly-ruby`](configure_gitaly.md#gitaly-ruby) is an internal implementation detail of Gitaly, so, there's not that much visibility into what goes on inside `gitaly-ruby` processes. @@ -1178,7 +261,7 @@ If you run Gitaly on its own server and notice these conditions: make changes to them in the web UI. Gitaly may be failing to authenticate with the Gitaly client because it has the -[wrong secrets file](#configure-gitaly-servers). +[wrong secrets file](configure_gitaly.md#configure-gitaly-servers). Confirm the following are all true: @@ -1198,7 +281,7 @@ Confirm the following are all true: successfully creates the project, but doesn't create the README. - When [tailing the logs](https://docs.gitlab.com/omnibus/settings/logs.html#tail-logs-in-a-console-on-the-server) on a Gitaly client and reproducing the error, you get `401` errors - when reaching the `/api/v4/internal/allowed` endpoint: + when reaching the [`/api/v4/internal/allowed`](../../development/internal_api.md) endpoint: ```shell # api_json.log @@ -1259,7 +342,7 @@ Confirm the following are all true: [IP] - - [18/Jul/2019:00:30:14 +0000] "POST /api/v4/internal/allowed HTTP/1.1" 401 30 "" "Ruby" ``` -To fix this problem, confirm that your [`gitlab-secrets.json` file](#configure-gitaly-servers) +To fix this problem, confirm that your [`gitlab-secrets.json` file](configure_gitaly.md#configure-gitaly-servers) on the Gitaly server matches the one on Gitaly client. If it doesn't match, update the secrets file on the Gitaly server to match the Gitaly client, then [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure). diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index 45f478b8d16..80b5c1bb799 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -701,7 +701,7 @@ Particular attention should be shown to: `gitaly-2`, and `gitaly-3` as Gitaly storage names. For more information on Gitaly server configuration, see our [Gitaly -documentation](index.md#configure-gitaly-servers). +documentation](configure_gitaly.md#configure-gitaly-servers). 1. SSH into the **Gitaly** node and login as root: @@ -834,7 +834,7 @@ scope of the GitLab documentation. NOTE: The load balancer must be configured to accept traffic from the Gitaly nodes in addition to the GitLab nodes. Some requests handled by -[`gitaly-ruby`](index.md#gitaly-ruby) sidecar processes call into the main Gitaly +[`gitaly-ruby`](configure_gitaly.md#gitaly-ruby) sidecar processes call into the main Gitaly process. `gitaly-ruby` uses the Gitaly address set in the GitLab server's `git_data_dirs` setting to make this connection. diff --git a/doc/administration/gitaly/reference.md b/doc/administration/gitaly/reference.md index 5105b9ed0d4..f08b03017e4 100644 --- a/doc/administration/gitaly/reference.md +++ b/doc/administration/gitaly/reference.md @@ -91,7 +91,7 @@ certificate_path = '/home/git/cert.cert' key_path = '/home/git/key.pem' ``` -[Read more](index.md#enable-tls-support) about TLS in Gitaly. +[Read more](configure_gitaly.md#enable-tls-support) about TLS in Gitaly. ### Storage diff --git a/doc/administration/index.md b/doc/administration/index.md index e5f20e3ba05..15a3bee4004 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -130,7 +130,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Kerberos authentication](../integration/kerberos.md). - See also other [authentication](../topics/authentication/index.md#gitlab-administrators) topics (for example, enforcing 2FA). - [Email users](../tools/email.md): Email GitLab users from within GitLab. -- [User Cohorts](../user/admin_area/analytics/user_cohorts.md): Display the monthly cohorts of new users and their activities over time. +- [User Cohorts](../user/admin_area/user_cohorts.md): Display the monthly cohorts of new users and their activities over time. - [Audit events](audit_events.md): View the changes made within the GitLab server for: - Groups and projects. - Instances. @@ -176,7 +176,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Job logs](job_logs.md): Information about the job logs. - [Register runners](../ci/runners/README.md#types-of-runners): Learn how to register and configure runners. - [Shared runners pipelines quota](../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota): Limit the usage of pipeline minutes for shared runners. -- [Enable/disable Auto DevOps](../topics/autodevops/index.md#enablingdisabling-auto-devops): Enable or disable Auto DevOps for your instance. +- [Enable/disable Auto DevOps](../topics/autodevops/index.md#enable-or-disable-auto-devops): Enable or disable Auto DevOps for your instance. ## Snippet settings diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md index 27e8b13812f..ac37ee00210 100644 --- a/doc/administration/instance_limits.md +++ b/doc/administration/instance_limits.md @@ -107,7 +107,7 @@ Limit the maximum daily member invitations allowed per group hierarchy. Clone traffic can put a large strain on your Gitaly service. To prevent such workloads from overwhelming your Gitaly server, you can set concurrency limits in Gitaly’s configuration file. -Read more on [Gitaly concurrency limits](gitaly/index.md#limit-rpc-concurrency). +Read more on [Gitaly concurrency limits](gitaly/configure_gitaly.md#limit-rpc-concurrency). - **Default rate limit** - Disabled @@ -526,7 +526,7 @@ amount of memory during indexing. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/201826) in GitLab 12.8. -You can set a limit on the content of text fields indexed for Global Search. +You can set a limit on the content of text fields indexed for Advanced Search. Setting a maximum helps to reduce the load of the indexing processes. If any text field exceeds this limit then the text will be truncated to this number of characters and the rest will not be indexed and hence will not be searchable. diff --git a/doc/administration/instance_review.md b/doc/administration/instance_review.md index 56a305fa4da..f2ba3a5a562 100644 --- a/doc/administration/instance_review.md +++ b/doc/administration/instance_review.md @@ -4,7 +4,7 @@ group: Conversion info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Instance Review +# Instance Review **(FREE SELF)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/6995) in [GitLab Free](https://about.gitlab.com/pricing/) 11.3. diff --git a/doc/administration/integration/kroki.md b/doc/administration/integration/kroki.md index 5e458399773..9e9ea62c44e 100644 --- a/doc/administration/integration/kroki.md +++ b/doc/administration/integration/kroki.md @@ -56,7 +56,7 @@ You need to enable Kroki integration from Settings under Admin Area. To do that, log in with an administrator account and follow these steps: 1. Select the Admin Area (**{admin}**) icon. -1. Navigate to **Settings > General**. +1. Go to **Settings > General**. 1. Expand the **Kroki** section. 1. Select **Enable Kroki** checkbox. 1. Enter the **Kroki URL**. diff --git a/doc/administration/integration/plantuml.md b/doc/administration/integration/plantuml.md index dbbe17cccc8..834f4047fdd 100644 --- a/doc/administration/integration/plantuml.md +++ b/doc/administration/integration/plantuml.md @@ -5,22 +5,94 @@ info: "To determine the technical writer assigned to the Stage/Group associated type: reference, howto --- -# PlantUML & GitLab **(FREE)** - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/8537) in GitLab 8.16. +# PlantUML and GitLab **(FREE)** When [PlantUML](https://plantuml.com) integration is enabled and configured in -GitLab you can create diagrams in AsciiDoc and Markdown documents -created in snippets, wikis, and repositories. +GitLab, you can create diagrams in snippets, wikis, and repositories. To set up +the integration, you must: + +1. [Configure your PlantUML server](#configure-your-plantuml-server). +1. [Configure local PlantUML access](#configure-local-plantuml-access). +1. [Configure PlantUML security](#configure-plantuml-security). +1. [Enable the integration](#enable-plantuml-integration). + +After completing the integration, PlantUML converts `plantuml` +blocks to an HTML image tag, with the source pointing to the PlantUML instance. The PlantUML +diagram delimiters `@startuml`/`@enduml` aren't required, as these are replaced +by the `plantuml` block: + +- **Markdown** + + ````markdown + ```plantuml + Bob -> Alice : hello + Alice -> Bob : hi + ``` + ```` + +- **AsciiDoc** + + ```plaintext + [plantuml, format="png", id="myDiagram", width="200px"] + ---- + Bob->Alice : hello + Alice -> Bob : hi + ---- + ``` + +- **reStructuredText** -## PlantUML Server + ```plaintext + .. plantuml:: + :caption: Caption with **bold** and *italic* + + Bob -> Alice: hello + Alice -> Bob: hi + ``` -Before you can enable PlantUML in GitLab; set up your own PlantUML -server to generate the diagrams. + Although you can use the `uml::` directive for compatibility with + [`sphinxcontrib-plantuml`](https://pypi.org/project/sphinxcontrib-plantuml/), + GitLab supports only the `caption` option. + +If the PlantUML server is correctly configured, these examples should render a +diagram instead of the code block: + +```plantuml +Bob -> Alice : hello +Alice -> Bob : hi +``` + +Inside the block you can add any of the diagrams PlantUML supports, such as: + +- [Activity](https://plantuml.com/activity-diagram-legacy) +- [Class](https://plantuml.com/class-diagram) +- [Component](https://plantuml.com/component-diagram) +- [Object](https://plantuml.com/object-diagram) +- [Sequence](https://plantuml.com/sequence-diagram) +- [State](https://plantuml.com/state-diagram) +- [Use Case](https://plantuml.com/use-case-diagram) + +You can add parameters to block definitions: + +- `format`: Can be either `png` (default) or `svg`. Use `svg` with care, as it's + not supported by all browsers, and isn't supported by Markdown. +- `id`: A CSS ID added to the diagram HTML tag. +- `width`: Width attribute added to the image tag. +- `height`: Height attribute added to the image tag. + +Markdown does not support any parameters, and always uses PNG format. + +## Configure your PlantUML server + +Before you can enable PlantUML in GitLab, set up your own PlantUML +server to generate the diagrams: + +- [In Docker](#docker). +- [In Debian/Ubuntu](#debianubuntu). ### Docker -With Docker, you can just run a container like this: +To run a PlantUML container in Docker, run this command: ```shell docker run -d --name plantuml -p 8080:8080 plantuml/plantuml-server:tomcat @@ -29,8 +101,9 @@ docker run -d --name plantuml -p 8080:8080 plantuml/plantuml-server:tomcat The **PlantUML URL** is the hostname of the server running the container. When running GitLab in Docker, it must have access to the PlantUML container. -You can achieve that by using [Docker Compose](https://docs.docker.com/compose/). -A basic `docker-compose.yml` file could contain: +To achieve that, use [Docker Compose](https://docs.docker.com/compose/). +In this basic `docker-compose.yml` file, PlantUML is accessible to GitLab at the URL +`http://plantuml:8080/`: ```yaml version: "3" @@ -46,37 +119,32 @@ services: container_name: plantuml ``` -In this scenario, PlantUML is accessible to GitLab at the URL -`http://plantuml:8080/`. - ### Debian/Ubuntu -You can also install and configure a PlantUML server in Debian/Ubuntu distributions using Tomcat. +You can install and configure a PlantUML server in Debian/Ubuntu distributions +using Tomcat: -First you need to create a `plantuml.war` file from the source code: +1. Run these commands to create a `plantuml.war` file from the source code: -```shell -sudo apt-get install graphviz openjdk-8-jdk git-core maven -git clone https://github.com/plantuml/plantuml-server.git -cd plantuml-server -mvn package -``` + ```shell + sudo apt-get install graphviz openjdk-8-jdk git-core maven + git clone https://github.com/plantuml/plantuml-server.git + cd plantuml-server + mvn package + ``` -The above sequence of commands generates a `.war` file you can deploy with Tomcat: +1. Deploy the `.war` file from the previous step with these commands: -```shell -sudo apt-get install tomcat8 -sudo cp target/plantuml.war /var/lib/tomcat8/webapps/plantuml.war -sudo chown tomcat8:tomcat8 /var/lib/tomcat8/webapps/plantuml.war -sudo service tomcat8 restart -``` + ```shell + sudo apt-get install tomcat8 + sudo cp target/plantuml.war /var/lib/tomcat8/webapps/plantuml.war + sudo chown tomcat8:tomcat8 /var/lib/tomcat8/webapps/plantuml.war + sudo service tomcat8 restart + ``` -After the Tomcat service restarts, the PlantUML service is ready and -listening for requests on port 8080: - -```plaintext -http://localhost:8080/plantuml -``` +The Tomcat service should restart. After the restart is complete, the +PlantUML service is ready and listening for requests on port 8080: +`http://localhost:8080/plantuml` To change these defaults, edit the `/etc/tomcat8/server.xml` file. @@ -85,40 +153,44 @@ The default URL is different when using this approach. The Docker-based image makes the service available at the root URL, with no relative path. Adjust the configuration below accordingly. -### Making local PlantUML accessible using custom GitLab setup +## Configure local PlantUML access -The PlantUML server runs locally on your server, so it is not accessible -externally by default. As such, it is necessary to catch external PlantUML -calls and redirect them to the local server. +The PlantUML server runs locally on your server, so it can't be accessed +externally by default. Your server must catch external PlantUML +calls to `https://gitlab.example.com/-/plantuml/` and redirect them to the +local PlantUML server. Depending on your setup, the URL is either of the +following: -The idea is to redirect each call to `https://gitlab.example.com/-/plantuml/` -to the local PlantUML server `http://plantuml:8080/` or `http://localhost:8080/plantuml/`, depending on your setup. +- `http://plantuml:8080/` +- `http://localhost:8080/plantuml/` -To enable the redirection, add the following line in `/etc/gitlab/gitlab.rb`: +If you're running [GitLab with TLS](https://docs.gitlab.com/omnibus/settings/ssl.html) +you must configure this redirection, because PlantUML uses the insecure HTTP protocol. +Newer browsers such as [Google Chrome 86+](https://www.chromestatus.com/feature/4926989725073408) +don't load insecure HTTP resources on pages served over HTTPS. -```ruby -# Docker deployment -nginx['custom_gitlab_server_config'] = "location /-/plantuml/ { \n proxy_cache off; \n proxy_pass http://plantuml:8080/; \n}\n" +To enable this redirection: -# Built from source -nginx['custom_gitlab_server_config'] = "location /-/plantuml { \n rewrite ^/-/(plantuml.*) /$1 break;\n proxy_cache off; \n proxy_pass http://localhost:8080/plantuml; \n}\n" -``` +1. Add the following line in `/etc/gitlab/gitlab.rb`, depending on your setup method: -To activate the changes, run the following command: + ```ruby + # Docker deployment + nginx['custom_gitlab_server_config'] = "location /-/plantuml/ { \n proxy_cache off; \n proxy_pass http://plantuml:8080/; \n}\n" -```shell -sudo gitlab-ctl reconfigure -``` + # Built from source + nginx['custom_gitlab_server_config'] = "location /-/plantuml { \n rewrite ^/-/(plantuml.*) /$1 break;\n proxy_cache off; \n proxy_pass http://localhost:8080/plantuml; \n}\n" + ``` -Note that the redirection through GitLab must be configured -when running [GitLab with TLS](https://docs.gitlab.com/omnibus/settings/ssl.html) -due to PlantUML's use of the insecure HTTP protocol. Newer browsers such -as [Google Chrome 86+](https://www.chromestatus.com/feature/4926989725073408) -do not load insecure HTTP resources on a page served over HTTPS. +1. To activate the changes, run the following command: -### Security + ```shell + sudo gitlab-ctl reconfigure + ``` -PlantUML has features that allow fetching network resources. +### Configure PlantUML security + +PlantUML has features that allow fetching network resources. If you self-host the +PlantUML server, put network controls in place to isolate it. ```plaintext @startuml @@ -129,98 +201,29 @@ stop; @enduml ``` -**If you self-host the PlantUML server, network controls should be put in place to isolate it.** - -## GitLab +## Enable PlantUML integration -You need to enable PlantUML integration from Settings under Admin Area. To do -that, sign in with an Administrator account, and then do following: +After configuring your local PlantUML server, you're ready to enable the PlantUML integration: -1. In GitLab, go to **Admin Area > Settings > General**. -1. Expand the **PlantUML** section. +1. Sign in to GitLab as an [Administrator](../../user/permissions.md) user. +1. In the top menu, click **{admin}** **Admin Area**. +1. In the left sidebar, go to **Settings > General** and expand the **PlantUML** section. 1. Select the **Enable PlantUML** check box. -1. Set the PlantUML instance as `https://gitlab.example.com/-/plantuml/`. - -NOTE: -If you are using a PlantUML server running v1.2020.9 and -above (for example, [plantuml.com](https://plantuml.com)), set the `PLANTUML_ENCODING` -environment variable to enable the `deflate` compression. On Omnibus GitLab, -this can be set in `/etc/gitlab.rb`: - -```ruby -gitlab_rails['env'] = { 'PLANTUML_ENCODING' => 'deflate' } -``` - -From GitLab 13.1 and later, PlantUML integration now -[requires a header prefix in the URL](https://github.com/plantuml/plantuml/issues/117#issuecomment-6235450160) -to distinguish different encoding types. - -## Creating Diagrams - -With PlantUML integration enabled and configured, we can start adding diagrams to -our AsciiDoc snippets, wikis, and repositories using delimited blocks: - -- **Markdown** - - ````markdown - ```plantuml - Bob -> Alice : hello - Alice -> Bob : hi - ``` - ```` - -- **AsciiDoc** - - ```plaintext - [plantuml, format="png", id="myDiagram", width="200px"] - ---- - Bob->Alice : hello - Alice -> Bob : hi - ---- - ``` - -- **reStructuredText** - - ```plaintext - .. plantuml:: - :caption: Caption with **bold** and *italic* - - Bob -> Alice: hello - Alice -> Bob: hi - ``` +1. Set the PlantUML instance as `https://gitlab.example.com/-/plantuml/`, + and click **Save changes**. - You can also use the `uml::` directive for compatibility with - [`sphinxcontrib-plantuml`](https://pypi.org/project/sphinxcontrib-plantuml/), - but GitLab only supports the `caption` option. +Depending on your PlantUML and GitLab version numbers, you may also need to take +these steps: -The above blocks are converted to an HTML image tag with source pointing to the -PlantUML instance. If the PlantUML server is correctly configured, this should -render a nice diagram instead of the block: +- For PlantUML servers running v1.2020.9 and above, such as [plantuml.com](https://plantuml.com), + you must set the `PLANTUML_ENCODING` environment variable to enable the `deflate` + compression. In Omnibus GitLab, you can set this value in `/etc/gitlab.rb` with + this command: -```plantuml -Bob -> Alice : hello -Alice -> Bob : hi -``` - -Inside the block you can add any of the diagrams PlantUML supports, such as: - -- [Sequence](https://plantuml.com/sequence-diagram) -- [Use Case](https://plantuml.com/use-case-diagram) -- [Class](https://plantuml.com/class-diagram) -- [Activity](https://plantuml.com/activity-diagram-legacy) -- [Component](https://plantuml.com/component-diagram) -- [State](https://plantuml.com/state-diagram), -- [Object](https://plantuml.com/object-diagram) - -You do not need to use the PlantUML -diagram delimiters `@startuml`/`@enduml`, as these are replaced by the AsciiDoc `plantuml` block. - -Some parameters can be added to the AsciiDoc block definition: - -- `format`: Can be either `png` or `svg`. Note that `svg` is not supported by - all browsers so use with care. The default is `png`. -- `id`: A CSS ID added to the diagram HTML tag. -- `width`: Width attribute added to the image tag. -- `height`: Height attribute added to the image tag. + ```ruby + gitlab_rails['env'] = { 'PLANTUML_ENCODING' => 'deflate' } + ``` -Markdown does not support any parameters and always uses PNG format. +- For GitLab versions 13.1 and later, PlantUML integration now + [requires a header prefix in the URL](https://github.com/plantuml/plantuml/issues/117#issuecomment-6235450160) + to distinguish different encoding types. diff --git a/doc/administration/integration/terminal.md b/doc/administration/integration/terminal.md index 0f26eb83d17..644e2d905ae 100644 --- a/doc/administration/integration/terminal.md +++ b/doc/administration/integration/terminal.md @@ -4,7 +4,7 @@ group: Ecosystem info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Web terminals +# Web terminals **(FREE)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/7690) in GitLab 8.15. diff --git a/doc/administration/invalidate_markdown_cache.md b/doc/administration/invalidate_markdown_cache.md index 211316534ee..7a880c81843 100644 --- a/doc/administration/invalidate_markdown_cache.md +++ b/doc/administration/invalidate_markdown_cache.md @@ -1,6 +1,6 @@ --- -stage: Create -group: Source Code +stage: Plan +group: Project Management info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" type: reference --- diff --git a/doc/administration/issue_closing_pattern.md b/doc/administration/issue_closing_pattern.md index 32d367bb15e..3b9ec74e6ee 100644 --- a/doc/administration/issue_closing_pattern.md +++ b/doc/administration/issue_closing_pattern.md @@ -1,6 +1,6 @@ --- stage: Create -group: Source Code +group: Code Review info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" type: reference --- @@ -12,7 +12,7 @@ This is the administration documentation. There is a separate [user documentatio on issue closing pattern. When a commit or merge request resolves one or more issues, it is possible to -automatically have these issues closed when the commit or merge request lands +automatically close these issues when the commit or merge request lands in the project's default branch. ## Change the issue closing pattern @@ -35,7 +35,7 @@ Because Rubular doesn't understand `%{issue_ref}`, you can replace this by expression of your liking: ```ruby - gitlab_rails['gitlab_issue_closing_pattern'] = "\b((?:[Cc]los(?:e[sd]?|ing)|\b[Ff]ix(?:e[sd]|ing)?|\b[Rr]esolv(?:e[sd]?|ing)|\b[Ii]mplement(?:s|ed|ing)?)(:?) +(?:(?:issues? +)?%{issue_ref}(?:(?: *,? +and +| *,? *)?)|([A-Z][A-Z0-9_]+-\d+))+)" + gitlab_rails['gitlab_issue_closing_pattern'] = /\b((?:[Cc]los(?:e[sd]?|ing)|\b[Ff]ix(?:e[sd]|ing)?|\b[Rr]esolv(?:e[sd]?|ing)|\b[Ii]mplement(?:s|ed|ing)?)(:?) +(?:(?:issues? +)?%{issue_ref}(?:(?: *,? +and +| *,? *)?)|([A-Z][A-Z0-9_]+-\d+))+)/.source ``` 1. [Reconfigure](restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab for the changes to take effect. diff --git a/doc/administration/logs.md b/doc/administration/logs.md index 17ecb324417..1950403ce35 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -14,10 +14,12 @@ Find more about them [in Audit Events documentation](audit_events.md). System log files are typically plain text in a standard log file format. This guide talks about how to read and use these system log files. -Read more about how to -[customize logging on Omnibus GitLab installations](https://docs.gitlab.com/omnibus/settings/logs.html) +Read more about the log system and using the logs: + +- [Customize logging on Omnibus GitLab installations](https://docs.gitlab.com/omnibus/settings/logs.html) including adjusting log retention, log forwarding, switching logs from JSON to plain text logging, and more. +- [How to parse and analyze JSON logs](troubleshooting/log_parsing.md). ## `production_json.log` @@ -87,7 +89,7 @@ In addition, the log contains the originating IP address, (`remote_ip`), the user's ID (`user_id`), and username (`username`). Some endpoints such as `/search` may make requests to Elasticsearch if using -[Advanced Search](../user/search/advanced_global_search.md). These +[Advanced Search](../user/search/advanced_search.md). These additionally log `elasticsearch_calls` and `elasticsearch_call_duration_s`, which correspond to: diff --git a/doc/administration/maintenance_mode/index.md b/doc/administration/maintenance_mode/index.md index 61d321f2176..1cdbda05749 100644 --- a/doc/administration/maintenance_mode/index.md +++ b/doc/administration/maintenance_mode/index.md @@ -172,7 +172,7 @@ You can monitor queues and disable jobs in **Admin Area > Monitoring > Backgroun [Incident management](../../operations/incident_management/index.md) functions will be limited. The creation of [alerts](../../operations/incident_management/alerts.md) and [incidents](../../operations/incident_management/incidents.md#incident-creation) will be paused entirely. Notifications and paging on alerts and incidents will therefore be disabled. -### Feature flags +### Feature flags - [Development feature flags](../../development/feature_flags/index.md) cannot be turned on or off through the API, but can be toggled through the Rails console. - [The feature flag service](../../operations/feature_flags.md) will respond to feature flag checks but feature flags cannot be toggled diff --git a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md index 1262c4192f6..f389a2d2176 100644 --- a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md +++ b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md @@ -32,7 +32,7 @@ metrics exposed by the [GitLab exporter](../prometheus/gitlab_metrics.md#metrics ## Creating the self monitoring project -1. Navigate to **Admin Area > Settings > Metrics and profiling**, and expand the **Self monitoring** section. +1. Go to **Admin Area > Settings > Metrics and profiling** and expand the **Self monitoring** section. 1. Toggle the **Create Project** button on. 1. Once your GitLab instance creates the project, GitLab displays a link to the project in the text above the **Create Project** toggle. You can also find it under **Projects > Your projects**. @@ -42,7 +42,7 @@ WARNING: Deleting the self monitoring project removes any changes made to the project. If you create the project again, it's created in its default state. -1. Navigate to **Admin Area > Settings > Metrics and profiling**, and expand the **Self monitoring** section. +1. Go to **Admin Area > Settings > Metrics and profiling** and expand the **Self monitoring** section. 1. Toggle the **Create Project** button off. 1. In the confirmation dialog that opens, click **Delete project**. It can take a few seconds for it to be deleted. diff --git a/doc/administration/monitoring/index.md b/doc/administration/monitoring/index.md index ac09acf2d2e..d2e5f40b171 100644 --- a/doc/administration/monitoring/index.md +++ b/doc/administration/monitoring/index.md @@ -11,9 +11,19 @@ Explore our features to monitor your GitLab instance: - [GitLab self-monitoring](gitlab_self_monitoring_project/index.md): The GitLab instance administration project helps to monitor the GitLab instance and take action on alerts. -- [Performance monitoring](performance/index.md): GitLab Performance Monitoring makes it possible to measure a wide variety of statistics of your instance. -- [Prometheus](prometheus/index.md): Prometheus is a powerful time-series monitoring service, providing a flexible platform for monitoring GitLab and other software products. -- [GitHub imports](github_imports.md): Monitor the health and progress of the GitHub importer with various Prometheus metrics. -- [Monitoring uptime](../../user/admin_area/monitoring/health_check.md): Check the server status using the health check endpoint. - - [IP whitelists](ip_whitelist.md): Configure GitLab for monitoring endpoints that provide health check information when probed. -- [`nginx_status`](https://docs.gitlab.com/omnibus/settings/nginx.html#enablingdisabling-nginx_status): Monitor your NGINX server status +- [Performance monitoring](performance/index.md): GitLab Performance Monitoring + makes it possible to measure a wide variety of statistics of your instance. +- [Prometheus](prometheus/index.md): Prometheus is a powerful time-series monitoring + service, providing a flexible platform for monitoring GitLab and other software + products. +- [GitHub imports](github_imports.md): Monitor the health and progress of the GitHub + importer with various Prometheus metrics. +- [Monitoring uptime](../../user/admin_area/monitoring/health_check.md): Check the + server status using the health check endpoint. + - [IP whitelists](ip_whitelist.md): Configure GitLab for monitoring endpoints that + provide health check information when probed. +- [`nginx_status`](https://docs.gitlab.com/omnibus/settings/nginx.html#enablingdisabling-nginx_status): + Monitor your NGINX server status. +- [Auto Monitoring](../../topics/autodevops/stages.md#auto-monitoring): Automated + monitoring for your application’s server and response metrics, provided by + [Auto DevOps](../../topics/autodevops/index.md). diff --git a/doc/administration/monitoring/performance/gitlab_configuration.md b/doc/administration/monitoring/performance/gitlab_configuration.md index 03eba67292d..a0f170e93d9 100644 --- a/doc/administration/monitoring/performance/gitlab_configuration.md +++ b/doc/administration/monitoring/performance/gitlab_configuration.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w GitLab Performance Monitoring is disabled by default. To enable it and change any of its settings: -1. Navigate to **Admin Area > Settings > Metrics and profiling** +1. Go to **Admin Area > Settings > Metrics and profiling** (`/admin/application_settings/metrics_and_profiling`). 1. Add the necessary configuration changes. 1. Restart all GitLab for the changes to take effect: diff --git a/doc/administration/monitoring/performance/grafana_configuration.md b/doc/administration/monitoring/performance/grafana_configuration.md index e6dda2ac87a..62d4f3c2625 100644 --- a/doc/administration/monitoring/performance/grafana_configuration.md +++ b/doc/administration/monitoring/performance/grafana_configuration.md @@ -62,7 +62,7 @@ repository. After setting up Grafana, you can enable a link to access it easily from the GitLab sidebar: -1. Navigate to the **Admin Area > Settings > Metrics and profiling**. +1. Go to the **Admin Area > Settings > Metrics and profiling**. 1. Expand **Metrics - Grafana**. 1. Check the **Enable access to Grafana** checkbox. 1. Configure the **Grafana URL**: diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md index 0516cbc2f8b..39e5e4f6c00 100644 --- a/doc/administration/monitoring/performance/performance_bar.md +++ b/doc/administration/monitoring/performance/performance_bar.md @@ -6,6 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Performance Bar **(FREE SELF)** +> The **Stats** field [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/271551) in GitLab SaaS 13.9. + You can display the GitLab Performance Bar to see statistics for the performance of a page. When activated, it looks as follows: @@ -53,6 +55,8 @@ From left to right, it displays: - **Request Selector**: a select box displayed on the right-hand side of the Performance Bar which enables you to view these metrics for any requests made while the current page was open. Only the first two requests per unique URL are captured. +- **Stats** (optional): if the `GITLAB_PERFORMANCE_BAR_STATS_URL` environment variable is set, + this URL is displayed in the bar. In GitLab 13.9 and later, used only in GitLab SaaS. ## Request warnings @@ -77,7 +81,7 @@ The GitLab Performance Bar is disabled by default. To enable it for a given grou 1. Sign in as a user with Administrator [permissions](../../../user/permissions.md). 1. In the menu bar, click **Admin Area**. -1. Navigate to **Settings > Metrics and profiling** +1. Go to **Settings > Metrics and profiling** (`admin/application_settings/metrics_and_profiling`), and expand the section **Profiling - Performance bar**. 1. Click **Enable access to the Performance Bar**. diff --git a/doc/administration/monitoring/performance/request_profiling.md b/doc/administration/monitoring/performance/request_profiling.md index f0ac8f991d3..ee035c6ad7a 100644 --- a/doc/administration/monitoring/performance/request_profiling.md +++ b/doc/administration/monitoring/performance/request_profiling.md @@ -10,7 +10,7 @@ To profile a request: 1. Sign in to GitLab as a user with Administrator or Maintainer [permissions](../../../user/permissions.md). 1. In the navigation bar, click **Admin area**. -1. Navigate to **Monitoring > Requests Profiles**. +1. Go to **Monitoring > Requests Profiles**. 1. In the **Requests Profiles** section, copy the token. 1. Pass the headers `X-Profile-Token: <token>` and `X-Profile-Mode: <mode>`(where `<mode>` can be `execution` or `memory`) to the request you want to profile. When diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index 38b26041901..3c6c984cbe3 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w To enable the GitLab Prometheus metrics: 1. Log into GitLab as a user with [administrator permissions](../../../user/permissions.md). -1. Navigate to **Admin Area > Settings > Metrics and profiling**. +1. Go to **Admin Area > Settings > Metrics and profiling**. 1. Find the **Metrics - Prometheus** section, and click **Enable Prometheus Metrics**. 1. [Restart GitLab](../../restart_gitlab.md#omnibus-gitlab-restart) for the changes to take effect. @@ -33,95 +33,102 @@ For enabling and viewing metrics from Sidekiq nodes, see [Sidekiq metrics](#side The following metrics are available: -| Metric | Type | Since | Description | Labels | -|:---------------------------------------------------------------|:----------|------:|:------------------------------------------------------------------------------------------------------|:--------------------------------------------------------| -| `gitlab_banzai_cached_render_real_duration_seconds` | Histogram | 9.4 | Duration of rendering Markdown into HTML when cached output exists | `controller`, `action` | -| `gitlab_banzai_cacheless_render_real_duration_seconds` | Histogram | 9.4 | Duration of rendering Markdown into HTML when cached output does not exist | `controller`, `action` | -| `gitlab_cache_misses_total` | Counter | 10.2 | Cache read miss | `controller`, `action` | -| `gitlab_cache_operation_duration_seconds` | Histogram | 10.2 | Cache access time | | -| `gitlab_cache_operations_total` | Counter | 12.2 | Cache operations by controller or action | `controller`, `action`, `operation` | -| `gitlab_ci_pipeline_creation_duration_seconds` | Histogram | 13.0 | Time in seconds it takes to create a CI/CD pipeline | | -| `gitlab_ci_pipeline_size_builds` | Histogram | 13.1 | Total number of builds within a pipeline grouped by a pipeline source | `source` | -| `job_waiter_started_total` | Counter | 12.9 | Number of batches of jobs started where a web request is waiting for the jobs to complete | `worker` | -| `job_waiter_timeouts_total` | Counter | 12.9 | Number of batches of jobs that timed out where a web request is waiting for the jobs to complete | `worker` | -| `gitlab_database_transaction_seconds` | Histogram | 12.1 | Time spent in database transactions, in seconds | | -| `gitlab_method_call_duration_seconds` | Histogram | 10.2 | Method calls real duration | `controller`, `action`, `module`, `method` | -| `gitlab_page_out_of_bounds` | Counter | 12.8 | Counter for the PageLimiter pagination limit being hit | `controller`, `action`, `bot` | -| `gitlab_rails_queue_duration_seconds` | Histogram | 9.4 | Measures latency between GitLab Workhorse forwarding a request to Rails | | -| `gitlab_sql_duration_seconds` | Histogram | 10.2 | SQL execution time, excluding `SCHEMA` operations and `BEGIN` / `COMMIT` | | -| `gitlab_ruby_threads_max_expected_threads` | Gauge | 13.3 | Maximum number of threads expected to be running and performing application work | | -| `gitlab_ruby_threads_running_threads` | Gauge | 13.3 | Number of running Ruby threads by name | | -| `gitlab_transaction_cache_<key>_count_total` | Counter | 10.2 | Counter for total Rails cache calls (per key) | | -| `gitlab_transaction_cache_<key>_duration_total` | Counter | 10.2 | Counter for total time (seconds) spent in Rails cache calls (per key) | | -| `gitlab_transaction_cache_count_total` | Counter | 10.2 | Counter for total Rails cache calls (aggregate) | | -| `gitlab_transaction_cache_duration_total` | Counter | 10.2 | Counter for total time (seconds) spent in Rails cache calls (aggregate) | | -| `gitlab_transaction_cache_read_hit_count_total` | Counter | 10.2 | Counter for cache hits for Rails cache calls | `controller`, `action` | -| `gitlab_transaction_cache_read_miss_count_total` | Counter | 10.2 | Counter for cache misses for Rails cache calls | `controller`, `action` | -| `gitlab_transaction_duration_seconds` | Histogram | 10.2 | Duration for all transactions (`gitlab_transaction_*` metrics) | `controller`, `action` | -| `gitlab_transaction_event_build_found_total` | Counter | 9.4 | Counter for build found for API /jobs/request | | -| `gitlab_transaction_event_build_invalid_total` | Counter | 9.4 | Counter for build invalid due to concurrency conflict for API /jobs/request | | -| `gitlab_transaction_event_build_not_found_cached_total` | Counter | 9.4 | Counter for cached response of build not found for API /jobs/request | | -| `gitlab_transaction_event_build_not_found_total` | Counter | 9.4 | Counter for build not found for API /jobs/request | | -| `gitlab_transaction_event_change_default_branch_total` | Counter | 9.4 | Counter when default branch is changed for any repository | | -| `gitlab_transaction_event_create_repository_total` | Counter | 9.4 | Counter when any repository is created | | -| `gitlab_transaction_event_etag_caching_cache_hit_total` | Counter | 9.4 | Counter for ETag cache hit. | `endpoint` | -| `gitlab_transaction_event_etag_caching_header_missing_total` | Counter | 9.4 | Counter for ETag cache miss - header missing | `endpoint` | -| `gitlab_transaction_event_etag_caching_key_not_found_total` | Counter | 9.4 | Counter for ETag cache miss - key not found | `endpoint` | -| `gitlab_transaction_event_etag_caching_middleware_used_total` | Counter | 9.4 | Counter for ETag middleware accessed | `endpoint` | -| `gitlab_transaction_event_etag_caching_resource_changed_total` | Counter | 9.4 | Counter for ETag cache miss - resource changed | `endpoint` | -| `gitlab_transaction_event_fork_repository_total` | Counter | 9.4 | Counter for repository forks (RepositoryForkWorker). Only incremented when source repository exists | | -| `gitlab_transaction_event_import_repository_total` | Counter | 9.4 | Counter for repository imports (RepositoryImportWorker) | | -| `gitlab_transaction_event_patch_hard_limit_bytes_hit_total` | Counter | 13.9 | Counter for diff patch size limit hits | | -| `gitlab_transaction_event_push_branch_total` | Counter | 9.4 | Counter for all branch pushes | | -| `gitlab_transaction_event_push_commit_total` | Counter | 9.4 | Counter for commits | `branch` | -| `gitlab_transaction_event_push_tag_total` | Counter | 9.4 | Counter for tag pushes | | -| `gitlab_transaction_event_rails_exception_total` | Counter | 9.4 | Counter for number of rails exceptions | | -| `gitlab_transaction_event_receive_email_total` | Counter | 9.4 | Counter for received emails | `handler` | -| `gitlab_transaction_event_remote_mirrors_failed_total` | Counter | 10.8 | Counter for failed remote mirrors | | -| `gitlab_transaction_event_remote_mirrors_finished_total` | Counter | 10.8 | Counter for finished remote mirrors | | -| `gitlab_transaction_event_remote_mirrors_running_total` | Counter | 10.8 | Counter for running remote mirrors | | -| `gitlab_transaction_event_remove_branch_total` | Counter | 9.4 | Counter when a branch is removed for any repository | | -| `gitlab_transaction_event_remove_repository_total` | Counter | 9.4 | Counter when a repository is removed | | -| `gitlab_transaction_event_remove_tag_total` | Counter | 9.4 | Counter when a tag is remove for any repository | | -| `gitlab_transaction_event_sidekiq_exception_total` | Counter | 9.4 | Counter of Sidekiq exceptions | | -| `gitlab_transaction_event_stuck_import_jobs_total` | Counter | 9.4 | Count of stuck import jobs | `projects_without_jid_count`, `projects_with_jid_count` | -| `gitlab_transaction_event_update_build_total` | Counter | 9.4 | Counter for update build for API `/jobs/request/:id` | | -| `gitlab_transaction_new_redis_connections_total` | Counter | 9.4 | Counter for new Redis connections | | -| `gitlab_transaction_queue_duration_total` | Counter | 9.4 | Duration jobs were enqueued before processing | | -| `gitlab_transaction_rails_queue_duration_total` | Counter | 9.4 | Measures latency between GitLab Workhorse forwarding a request to Rails | `controller`, `action` | -| `gitlab_transaction_view_duration_total` | Counter | 9.4 | Duration for views | `controller`, `action`, `view` | -| `gitlab_view_rendering_duration_seconds` | Histogram | 10.2 | Duration for views (histogram) | `controller`, `action`, `view` | -| `http_requests_total` | Counter | 9.4 | Rack request count | `method`, `status` | -| `http_request_duration_seconds` | Histogram | 9.4 | HTTP response time from rack middleware | `method` | -| `gitlab_transaction_db_count_total` | Counter | 13.1 | Counter for total number of SQL calls | `controller`, `action` | -| `gitlab_transaction_db_write_count_total` | Counter | 13.1 | Counter for total number of write SQL calls | `controller`, `action` | -| `gitlab_transaction_db_cached_count_total` | Counter | 13.1 | Counter for total number of cached SQL calls | `controller`, `action` | -| `http_elasticsearch_requests_duration_seconds` **(PREMIUM)** | Histogram | 13.1 | Elasticsearch requests duration during web transactions | `controller`, `action` | -| `http_elasticsearch_requests_total` **(PREMIUM)** | Counter | 13.1 | Elasticsearch requests count during web transactions | `controller`, `action` | -| `pipelines_created_total` | Counter | 9.4 | Counter of pipelines created | | -| `rack_uncaught_errors_total` | Counter | 9.4 | Rack connections handling uncaught errors count | | -| `user_session_logins_total` | Counter | 9.4 | Counter of how many users have logged in since GitLab was started or restarted | | -| `upload_file_does_not_exist` | Counter | 10.7 | Number of times an upload record could not find its file. Made available in all tiers in GitLab 11.5. | | -| `failed_login_captcha_total` | Gauge | 11.0 | Counter of failed CAPTCHA attempts during login | | -| `successful_login_captcha_total` | Gauge | 11.0 | Counter of successful CAPTCHA attempts during login | | -| `auto_devops_pipelines_completed_total` | Counter | 12.7 | Counter of completed Auto DevOps pipelines, labeled by status | | -| `gitlab_metrics_dashboard_processing_time_ms` | Summary | 12.10 | Metrics dashboard processing time in milliseconds | service, stages | -| `action_cable_active_connections` | Gauge | 13.4 | Number of ActionCable WS clients currently connected | `server_mode` | -| `action_cable_pool_min_size` | Gauge | 13.4 | Minimum number of worker threads in ActionCable thread pool | `server_mode` | -| `action_cable_pool_max_size` | Gauge | 13.4 | Maximum number of worker threads in ActionCable thread pool | `server_mode` | -| `action_cable_pool_current_size` | Gauge | 13.4 | Current number of worker threads in ActionCable thread pool | `server_mode` | -| `action_cable_pool_largest_size` | Gauge | 13.4 | Largest number of worker threads observed so far in ActionCable thread pool | `server_mode` | -| `action_cable_pool_pending_tasks` | Gauge | 13.4 | Number of tasks waiting to be executed in ActionCable thread pool | `server_mode` | -| `action_cable_pool_tasks_total` | Gauge | 13.4 | Total number of tasks executed in ActionCable thread pool | `server_mode` | -| `gitlab_issuable_fast_count_by_state_total` | Counter | 13.5 | Total number of row count operations on issue/merge request list pages | | -| `gitlab_issuable_fast_count_by_state_failures_total` | Counter | 13.5 | Number of soft-failed row count operations on issue/merge request list pages | | -| `gitlab_external_http_total` | Counter | 13.8 | Total number of HTTP calls to external systems | `controller`, `action` | -| `gitlab_external_http_duration_seconds` | Counter | 13.8 | Duration in seconds spent on each HTTP call to external systems | | -| `gitlab_external_http_exception_total` | Counter | 13.8 | Total number of exceptions raised when making external HTTP calls | | -| `ci_report_parser_duration_seconds` | Histogram | 13.9 | Time to parse CI/CD report artifacts | `parser` | -| `pipeline_graph_link_calculation_duration_seconds` | Histogram | 13.9 | Total time spent calculating links, in seconds | | -| `pipeline_graph_links_total` | Histogram | 13.9 | Number of links per graph | | -| `pipeline_graph_links_per_job_ratio` | Histogram | 13.9 | Ratio of links to job per graph | | +| Metric | Type | Since | Description | Labels | +| :--------------------------------------------------------------- | :---------- | ------: | :-------------------------------------------------------------------------------------------------------------------- | :-------------------------------------------------------- | +| `gitlab_banzai_cached_render_real_duration_seconds` | Histogram | 9.4 | Duration of rendering Markdown into HTML when cached output exists | `controller`, `action` | +| `gitlab_banzai_cacheless_render_real_duration_seconds` | Histogram | 9.4 | Duration of rendering Markdown into HTML when cached output does not exist | `controller`, `action` | +| `gitlab_cache_misses_total` | Counter | 10.2 | Cache read miss | `controller`, `action` | +| `gitlab_cache_operation_duration_seconds` | Histogram | 10.2 | Cache access time | | +| `gitlab_cache_operations_total` | Counter | 12.2 | Cache operations by controller or action | `controller`, `action`, `operation` | +| `gitlab_ci_pipeline_creation_duration_seconds` | Histogram | 13.0 | Time in seconds it takes to create a CI/CD pipeline | | +| `gitlab_ci_pipeline_size_builds` | Histogram | 13.1 | Total number of builds within a pipeline grouped by a pipeline source | `source` | +| `job_waiter_started_total` | Counter | 12.9 | Number of batches of jobs started where a web request is waiting for the jobs to complete | `worker` | +| `job_waiter_timeouts_total` | Counter | 12.9 | Number of batches of jobs that timed out where a web request is waiting for the jobs to complete | `worker` | +| `gitlab_database_transaction_seconds` | Histogram | 12.1 | Time spent in database transactions, in seconds | | +| `gitlab_method_call_duration_seconds` | Histogram | 10.2 | Method calls real duration | `controller`, `action`, `module`, `method` | +| `gitlab_page_out_of_bounds` | Counter | 12.8 | Counter for the PageLimiter pagination limit being hit | `controller`, `action`, `bot` | +| `gitlab_rails_queue_duration_seconds` | Histogram | 9.4 | Measures latency between GitLab Workhorse forwarding a request to Rails | | +| `gitlab_sql_duration_seconds` | Histogram | 10.2 | SQL execution time, excluding `SCHEMA` operations and `BEGIN` / `COMMIT` | | +| `gitlab_sql_<role>_duration_seconds` | Histogram | 13.10 | SQL execution time, excluding `SCHEMA` operations and `BEGIN` / `COMMIT`, grouped by database roles (primary/replica) | | +| `gitlab_ruby_threads_max_expected_threads` | Gauge | 13.3 | Maximum number of threads expected to be running and performing application work | | +| `gitlab_ruby_threads_running_threads` | Gauge | 13.3 | Number of running Ruby threads by name | | +| `gitlab_transaction_cache_<key>_count_total` | Counter | 10.2 | Counter for total Rails cache calls (per key) | | +| `gitlab_transaction_cache_<key>_duration_total` | Counter | 10.2 | Counter for total time (seconds) spent in Rails cache calls (per key) | | +| `gitlab_transaction_cache_count_total` | Counter | 10.2 | Counter for total Rails cache calls (aggregate) | | +| `gitlab_transaction_cache_duration_total` | Counter | 10.2 | Counter for total time (seconds) spent in Rails cache calls (aggregate) | | +| `gitlab_transaction_cache_read_hit_count_total` | Counter | 10.2 | Counter for cache hits for Rails cache calls | `controller`, `action` | +| `gitlab_transaction_cache_read_miss_count_total` | Counter | 10.2 | Counter for cache misses for Rails cache calls | `controller`, `action` | +| `gitlab_transaction_duration_seconds` | Histogram | 10.2 | Duration for all transactions (`gitlab_transaction_*` metrics) | `controller`, `action` | +| `gitlab_transaction_event_build_found_total` | Counter | 9.4 | Counter for build found for API /jobs/request | | +| `gitlab_transaction_event_build_invalid_total` | Counter | 9.4 | Counter for build invalid due to concurrency conflict for API /jobs/request | | +| `gitlab_transaction_event_build_not_found_cached_total` | Counter | 9.4 | Counter for cached response of build not found for API /jobs/request | | +| `gitlab_transaction_event_build_not_found_total` | Counter | 9.4 | Counter for build not found for API /jobs/request | | +| `gitlab_transaction_event_change_default_branch_total` | Counter | 9.4 | Counter when default branch is changed for any repository | | +| `gitlab_transaction_event_create_repository_total` | Counter | 9.4 | Counter when any repository is created | | +| `gitlab_transaction_event_etag_caching_cache_hit_total` | Counter | 9.4 | Counter for ETag cache hit. | `endpoint` | +| `gitlab_transaction_event_etag_caching_header_missing_total` | Counter | 9.4 | Counter for ETag cache miss - header missing | `endpoint` | +| `gitlab_transaction_event_etag_caching_key_not_found_total` | Counter | 9.4 | Counter for ETag cache miss - key not found | `endpoint` | +| `gitlab_transaction_event_etag_caching_middleware_used_total` | Counter | 9.4 | Counter for ETag middleware accessed | `endpoint` | +| `gitlab_transaction_event_etag_caching_resource_changed_total` | Counter | 9.4 | Counter for ETag cache miss - resource changed | `endpoint` | +| `gitlab_transaction_event_fork_repository_total` | Counter | 9.4 | Counter for repository forks (RepositoryForkWorker). Only incremented when source repository exists | | +| `gitlab_transaction_event_import_repository_total` | Counter | 9.4 | Counter for repository imports (RepositoryImportWorker) | | +| `gitlab_transaction_event_patch_hard_limit_bytes_hit_total` | Counter | 13.9 | Counter for diff patch size limit hits | | +| `gitlab_transaction_event_push_branch_total` | Counter | 9.4 | Counter for all branch pushes | | +| `gitlab_transaction_event_push_commit_total` | Counter | 9.4 | Counter for commits | `branch` | +| `gitlab_transaction_event_push_tag_total` | Counter | 9.4 | Counter for tag pushes | | +| `gitlab_transaction_event_rails_exception_total` | Counter | 9.4 | Counter for number of rails exceptions | | +| `gitlab_transaction_event_receive_email_total` | Counter | 9.4 | Counter for received emails | `handler` | +| `gitlab_transaction_event_remote_mirrors_failed_total` | Counter | 10.8 | Counter for failed remote mirrors | | +| `gitlab_transaction_event_remote_mirrors_finished_total` | Counter | 10.8 | Counter for finished remote mirrors | | +| `gitlab_transaction_event_remote_mirrors_running_total` | Counter | 10.8 | Counter for running remote mirrors | | +| `gitlab_transaction_event_remove_branch_total` | Counter | 9.4 | Counter when a branch is removed for any repository | | +| `gitlab_transaction_event_remove_repository_total` | Counter | 9.4 | Counter when a repository is removed | | +| `gitlab_transaction_event_remove_tag_total` | Counter | 9.4 | Counter when a tag is remove for any repository | | +| `gitlab_transaction_event_sidekiq_exception_total` | Counter | 9.4 | Counter of Sidekiq exceptions | | +| `gitlab_transaction_event_stuck_import_jobs_total` | Counter | 9.4 | Count of stuck import jobs | `projects_without_jid_count`, `projects_with_jid_count` | +| `gitlab_transaction_event_update_build_total` | Counter | 9.4 | Counter for update build for API `/jobs/request/:id` | | +| `gitlab_transaction_new_redis_connections_total` | Counter | 9.4 | Counter for new Redis connections | | +| `gitlab_transaction_queue_duration_total` | Counter | 9.4 | Duration jobs were enqueued before processing | | +| `gitlab_transaction_rails_queue_duration_total` | Counter | 9.4 | Measures latency between GitLab Workhorse forwarding a request to Rails | `controller`, `action` | +| `gitlab_transaction_view_duration_total` | Counter | 9.4 | Duration for views | `controller`, `action`, `view` | +| `gitlab_view_rendering_duration_seconds` | Histogram | 10.2 | Duration for views (histogram) | `controller`, `action`, `view` | +| `http_requests_total` | Counter | 9.4 | Rack request count | `method`, `status` | +| `http_request_duration_seconds` | Histogram | 9.4 | HTTP response time from rack middleware | `method` | +| `gitlab_transaction_db_count_total` | Counter | 13.1 | Counter for total number of SQL calls | `controller`, `action` | +| `gitlab_transaction_db_<role>_count_total` | Counter | 13.10 | Counter for total number of SQL calls, grouped by database roles (primary/replica) | `controller`, `action` | +| `gitlab_transaction_db_write_count_total` | Counter | 13.1 | Counter for total number of write SQL calls | `controller`, `action` | +| `gitlab_transaction_db_cached_count_total` | Counter | 13.1 | Counter for total number of cached SQL calls | `controller`, `action` | +| `gitlab_transaction_db_<role>_cached_count_total` | Counter | 13.1 | Counter for total number of cached SQL calls, grouped by database roles (primary/replica) | `controller`, `action` | +| `http_elasticsearch_requests_duration_seconds` **(PREMIUM)** | Histogram | 13.1 | Elasticsearch requests duration during web transactions | `controller`, `action` | +| `http_elasticsearch_requests_total` **(PREMIUM)** | Counter | 13.1 | Elasticsearch requests count during web transactions | `controller`, `action` | +| `pipelines_created_total` | Counter | 9.4 | Counter of pipelines created | | +| `rack_uncaught_errors_total` | Counter | 9.4 | Rack connections handling uncaught errors count | | +| `user_session_logins_total` | Counter | 9.4 | Counter of how many users have logged in since GitLab was started or restarted | | +| `upload_file_does_not_exist` | Counter | 10.7 | Number of times an upload record could not find its file. Made available in all tiers in GitLab 11.5. | | +| `failed_login_captcha_total` | Gauge | 11.0 | Counter of failed CAPTCHA attempts during login | | +| `successful_login_captcha_total` | Gauge | 11.0 | Counter of successful CAPTCHA attempts during login | | +| `auto_devops_pipelines_completed_total` | Counter | 12.7 | Counter of completed Auto DevOps pipelines, labeled by status | | +| `gitlab_metrics_dashboard_processing_time_ms` | Summary | 12.10 | Metrics dashboard processing time in milliseconds | service, stages | +| `action_cable_active_connections` | Gauge | 13.4 | Number of ActionCable WS clients currently connected | `server_mode` | +| `action_cable_broadcasts_total` | Counter | 13.10 | The number of ActionCable broadcasts emitted | `server_mode` | +| `action_cable_pool_min_size` | Gauge | 13.4 | Minimum number of worker threads in ActionCable thread pool | `server_mode` | +| `action_cable_pool_max_size` | Gauge | 13.4 | Maximum number of worker threads in ActionCable thread pool | `server_mode` | +| `action_cable_pool_current_size` | Gauge | 13.4 | Current number of worker threads in ActionCable thread pool | `server_mode` | +| `action_cable_pool_largest_size` | Gauge | 13.4 | Largest number of worker threads observed so far in ActionCable thread pool | `server_mode` | +| `action_cable_pool_pending_tasks` | Gauge | 13.4 | Number of tasks waiting to be executed in ActionCable thread pool | `server_mode` | +| `action_cable_pool_tasks_total` | Gauge | 13.4 | Total number of tasks executed in ActionCable thread pool | `server_mode` | +| `action_cable_single_client_transmissions_total` | Counter | 13.10 | The number of ActionCable messages transmitted to any client in any channel | `server_mode` | +| `action_cable_subscription_confirmations_total` | Counter | 13.10 | The number of ActionCable subscriptions from clients confirmed | `server_mode` | +| `action_cable_subscription_rejections_total` | Counter | 13.10 | The number of ActionCable subscriptions from clients rejected | `server_mode` | +| `gitlab_issuable_fast_count_by_state_total` | Counter | 13.5 | Total number of row count operations on issue/merge request list pages | | +| `gitlab_issuable_fast_count_by_state_failures_total` | Counter | 13.5 | Number of soft-failed row count operations on issue/merge request list pages | | +| `gitlab_external_http_total` | Counter | 13.8 | Total number of HTTP calls to external systems | `controller`, `action` | +| `gitlab_external_http_duration_seconds` | Counter | 13.8 | Duration in seconds spent on each HTTP call to external systems | | +| `gitlab_external_http_exception_total` | Counter | 13.8 | Total number of exceptions raised when making external HTTP calls | | +| `ci_report_parser_duration_seconds` | Histogram | 13.9 | Time to parse CI/CD report artifacts | `parser` | +| `pipeline_graph_link_calculation_duration_seconds` | Histogram | 13.9 | Total time spent calculating links, in seconds | | +| `pipeline_graph_links_total` | Histogram | 13.9 | Number of links per graph | | +| `pipeline_graph_links_per_job_ratio` | Histogram | 13.9 | Ratio of links to job per graph | | ## Metrics controlled by a feature flag @@ -214,11 +221,18 @@ configuration option in `gitlab.yml`. These metrics are served from the | `geo_snippet_repositories_synced` | Gauge | 13.4 | Number of syncable snippets synced on secondary | `url` | | `geo_snippet_repositories_failed` | Gauge | 13.4 | Number of syncable snippets failed on secondary | `url` | | `geo_snippet_repositories_registry` | Gauge | 13.4 | Number of syncable snippets in the registry | `url` | +| `geo_group_wiki_repositories` | Gauge | 13.10 | Number of group wikis on primary | `url` | +| `geo_group_wiki_repositories_checksummed` | Gauge | 13.10 | Number of group wikis checksummed on primary | `url` | +| `geo_group_wiki_repositories_checksum_failed` | Gauge | 13.10 | Number of group wikis failed to calculate the checksum on primary | `url` | +| `geo_group_wiki_repositories_synced` | Gauge | 13.10 | Number of syncable group wikis synced on secondary | `url` | +| `geo_group_wiki_repositories_failed` | Gauge | 13.10 | Number of syncable group wikis failed on secondary | `url` | +| `geo_group_wiki_repositories_registry` | Gauge | 13.10 | Number of syncable group wikis in the registry | `url` | | `limited_capacity_worker_running_jobs` | Gauge | 13.5 | Number of running jobs | `worker` | | `limited_capacity_worker_max_running_jobs` | Gauge | 13.5 | Maximum number of running jobs | `worker` | | `limited_capacity_worker_remaining_work_count` | Gauge | 13.5 | Number of jobs waiting to be enqueued | `worker` | | `destroyed_job_artifacts_count_total` | Counter | 13.6 | Number of destroyed expired job artifacts | | | `destroyed_pipeline_artifacts_count_total` | Counter | 13.8 | Number of destroyed expired pipeline artifacts | | +| `gitlab_optimistic_locking_retries` | Histogram | 13.10 | Number of retry attempts to execute optimistic retry lock | | ## Database load balancing metrics **(PREMIUM SELF)** diff --git a/doc/administration/nfs.md b/doc/administration/nfs.md index a7009b702ed..59b18f7a74a 100644 --- a/doc/administration/nfs.md +++ b/doc/administration/nfs.md @@ -365,21 +365,26 @@ You may be able to avoid timeouts and data loss using `actimeo=0` and `lookupcac we expect the performance reduction will still be significant. As noted above, we strongly recommend upgrading to [Gitaly Cluster](gitaly/praefect.md) as soon as possible. -### Avoid using AWS's Elastic File System (EFS) +### Avoid using cloud-based file systems -GitLab strongly recommends against using AWS Elastic File System (EFS). -Our support team will not be able to assist on performance issues related to -file system access. +GitLab strongly recommends against using cloud-based file systems such as: -Customers and users have reported that AWS EFS does not perform well for the GitLab -use-case. Workloads where many small files are written in a serialized manner, like `git`, -are not well-suited for EFS. EBS with an NFS server on top will perform much better. +- AWS Elastic File System (EFS). +- Google Cloud Filestore. +- Azure Files. -If you do choose to use EFS, avoid storing GitLab log files (e.g. those in `/var/log/gitlab`) +Our support team cannot assist with performance issues related to cloud-based file system access. + +Customers and users have reported that these file systems don't perform well for +the file system access GitLab requires. Workloads where many small files are written in +a serialized manner, like `git`, are not well suited to cloud-based file systems. + +If you do choose to use these, avoid storing GitLab log files (for example, those in `/var/log/gitlab`) there because this will also affect performance. We recommend that the log files be stored on a local volume. -For more details on another person's experience with EFS, see this [Commit Brooklyn 2019 video](https://youtu.be/K6OS8WodRBQ?t=313). +For more details on the experience of using a cloud-based file systems with GitLab, +see this [Commit Brooklyn 2019 video](https://youtu.be/K6OS8WodRBQ?t=313). ### Avoid using CephFS and GlusterFS diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md index 3b43f53dc82..cbef752773b 100644 --- a/doc/administration/object_storage.md +++ b/doc/administration/object_storage.md @@ -405,8 +405,8 @@ Rackspace Cloud is supported only with the storage-specific form. | `provider` | The provider name | `Rackspace` | | `rackspace_username` | The username of the Rackspace account with access to the container | `joe.smith` | | `rackspace_api_key` | The API key of the Rackspace account with access to the container | `ABC123DEF456ABC123DEF456ABC123DE` | -| `rackspace_region` | The Rackspace storage region to use, a three letter code from the [list of service access endpoints](https://developer.rackspace.com/docs/cloud-files/v1/general-api-info/service-access/) | `iad` | -| `rackspace_temp_url_key` | The private key you have set in the Rackspace API for [temporary URLs](https://developer.rackspace.com/docs/cloud-files/v1/use-cases/public-access-to-your-cloud-files-account/#tempurl). | `ABC123DEF456ABC123DEF456ABC123DE` | +| `rackspace_region` | The Rackspace storage region to use, a three letter code from the [list of service access endpoints](https://docs.rackspace.com/docs/cloud-files/v1/general-api-info/service-access/) | `iad` | +| `rackspace_temp_url_key` | The private key you have set in the Rackspace API for [temporary URLs](https://docs.rackspace.com/docs/cloud-files/v1/use-cases/public-access-to-your-cloud-files-account/#tempurl). | `ABC123DEF456ABC123DEF456ABC123DE` | Regardless of whether the container has public access enabled or disabled, Fog uses the TempURL method to grant access to LFS objects. If you see error @@ -572,13 +572,15 @@ See the following additional guides: ## Warnings, limitations, and known issues -### Separate buckets required when using Helm +### Use separate buckets -Generally, using the same bucket for your Object Storage is fine to do -for convenience. +Using separate buckets for each data type is the recommended approach for GitLab. +This ensures there are no collisions across the various types of data GitLab stores. +There are plans to [enable the use of a single bucket](https://gitlab.com/gitlab-org/gitlab/-/issues/292958) +in the future. -However, if you're using or planning to use Helm, separate buckets will -be required as there is a [known limitation with restorations of Helm chart backups](https://docs.gitlab.com/charts/advanced/external-object-storage/#lfs-artifacts-uploads-packages-external-diffs-pseudonymizer). +Helm-based installs require separate buckets to +[handle backup restorations](https://docs.gitlab.com/charts/advanced/external-object-storage/#lfs-artifacts-uploads-packages-external-diffs-pseudonymizer) ### S3 API compatibility issues diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md index 330bd9a43da..ec5bf9d6379 100644 --- a/doc/administration/operations/fast_ssh_key_lookup.md +++ b/doc/administration/operations/fast_ssh_key_lookup.md @@ -6,9 +6,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Fast lookup of authorized SSH keys in the database -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/1631) in [GitLab Starter](https://about.gitlab.com/pricing/) 9.3. -> - [Available in](https://gitlab.com/gitlab-org/gitlab/-/issues/3953) GitLab Community Edition 10.4. - NOTE: This document describes a drop-in replacement for the `authorized_keys` file. For normal (non-deploy key) users, consider using @@ -219,8 +216,6 @@ the database. The following instructions can be used to build OpenSSH 7.5: ## SELinux support and limitations -> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/2855) in GitLab 10.5. - GitLab supports `authorized_keys` database lookups with [SELinux](https://en.wikipedia.org/wiki/Security-Enhanced_Linux). Because the SELinux policy is static, GitLab doesn't support the ability to change diff --git a/doc/administration/operations/moving_repositories.md b/doc/administration/operations/moving_repositories.md index da3b3dbbc15..fc39d8e2074 100644 --- a/doc/administration/operations/moving_repositories.md +++ b/doc/administration/operations/moving_repositories.md @@ -20,7 +20,7 @@ The GitLab API is the recommended way to move Git repositories: For more information, see: -- [Configuring additional storage for Gitaly](../gitaly/index.md#network-architecture). This +- [Configuring additional storage for Gitaly](../gitaly/configure_gitaly.md#network-architecture). This example configures additional storage called `storage1` and `storage2`. - [The API documentation](../../api/project_repository_storage_moves.md) details the endpoints for querying and scheduling project repository moves. diff --git a/doc/administration/operations/rails_console.md b/doc/administration/operations/rails_console.md index b40560bf6e5..1fe1ea96bff 100644 --- a/doc/administration/operations/rails_console.md +++ b/doc/administration/operations/rails_console.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# The Rails Console +# Rails console **(FREE SELF)** The [Rails console](https://guides.rubyonrails.org/command_line.html#rails-console). provides a way to interact with your GitLab instance from the command line. @@ -148,3 +148,23 @@ Traceback (most recent call last): [traceback removed] /opt/gitlab/..../runner_command.rb:42:in `load': cannot load such file -- /tmp/helloworld.rb (LoadError) ``` + +In case you encouter a similar error to this: + +```plaintext +[root ~]# sudo gitlab-rails runner helloworld.rb +Please specify a valid ruby command or the path of a script to run. +Run 'rails runner -h' for help. + +undefined local variable or method `helloworld' for main:Object +``` + +You can either move the file to the `/tmp` directory or create a new directory onwed by the user `git` and save the script in that directory as illustrated below: + +```shell +sudo mkdir /scripts +sudo mv /script_path/helloworld.rb /scripts +sudo chown -R git:git /scripts +sudo chmod 700 /scripts +sudo gitlab-rails runner /scripts/helloworld.rb +``` diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md index 63bb969afce..2f6b6a2f629 100644 --- a/doc/administration/packages/container_registry.md +++ b/doc/administration/packages/container_registry.md @@ -71,7 +71,7 @@ Where: | `enabled` | `true` or `false`. Enables the Registry in GitLab. By default this is `false`. | | `host` | The host URL under which the Registry runs and users can use. | | `port` | The port the external Registry domain listens on. | -| `api_url` | The internal API URL under which the Registry is exposed. It defaults to `http://localhost:5000`. | +| `api_url` | The internal API URL under which the Registry is exposed. It defaults to `http://localhost:5000`. Do not change this unless you are setting up an [external Docker registry](#use-an-external-container-registry-with-gitlab-as-an-auth-endpoint). | | `key` | The private key location that is a pair of Registry's `rootcertbundle`. Read the [token auth configuration documentation](https://docs.docker.com/registry/configuration/#token). | | `path` | This should be the same directory like specified in Registry's `rootdirectory`. Read the [storage configuration documentation](https://docs.docker.com/registry/configuration/#storage). This path needs to be readable by the GitLab user, the web-server user and the Registry user. Read more in [#configure-storage-for-the-container-registry](#configure-storage-for-the-container-registry). | | `issuer` | This should be the same value as configured in Registry's `issuer`. Read the [token auth configuration documentation](https://docs.docker.com/registry/configuration/#token). | @@ -630,18 +630,18 @@ You can use GitLab as an auth endpoint with an external container registry. ```ruby gitlab_rails['registry_enabled'] = true - gitlab_rails['registry_api_url'] = "http://localhost:5000" + gitlab_rails['registry_api_url'] = "https://<external_registry_host>:5000" gitlab_rails['registry_issuer'] = "gitlab-issuer" ``` - `gitlab_rails['registry_enabled'] = true` is needed to enable GitLab - Container Registry features and authentication endpoint. The GitLab bundled - Container Registry service does not start, even with this enabled. - - `gitlab_rails['registry_api_url'] = "http://localhost:5000"` can - carry a different hostname and port depending on where the external registry - is hosted. It must also specify `https` if the external registry is - configured to use TLS. + - `gitlab_rails['registry_enabled'] = true` is needed to enable GitLab + Container Registry features and authentication endpoint. The GitLab bundled + Container Registry service does not start, even with this enabled. + - `gitlab_rails['registry_api_url'] = "http://<external_registry_host>:5000"` + must be changed to match the host where Registry is installed. + It must also specify `https` if the external registry is + configured to use TLS. Read more on the + [Docker registry documentation](https://docs.docker.com/registry/deploying/). 1. A certificate-key pair is required for GitLab and the external container registry to communicate securely. You need to create a certificate-key @@ -688,12 +688,14 @@ You can use GitLab as an auth endpoint with an external container registry. enabled: true host: "registry.gitlab.example.com" port: "5005" - api_url: "http://localhost:5000" - path: /var/opt/gitlab/gitlab-rails/shared/registry - key: /var/opt/gitlab/gitlab-rails/certificate.key + api_url: "https://<external_registry_host>:5000" + path: /var/lib/registry + key: /path/to/keyfile issuer: gitlab-issuer ``` + [Read more](#enable-the-container-registry) about what these parameters mean. + 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. ## Configure Container Registry notifications @@ -900,12 +902,12 @@ not directly accessible via `tag`: sudo gitlab-ctl registry-garbage-collect -m ``` -Without the `-m` flag, the Container Registry only removes layers that are not referenced by any manifest, tagged or not. - Since this is a way more destructive operation, this behavior is disabled by default. You are likely expecting this way of operation, but before doing that, ensure that you have backed up all registry data. +When the command is used without the `-m` flag, the Container Registry only removes layers that are not referenced by any manifest, tagged or not. + ### Performing garbage collection without downtime > [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/764) in GitLab 8.8. @@ -1063,6 +1065,70 @@ encounter this error. Administrators can increase the token duration in **Admin area > Settings > CI/CD > Container Registry > Authorization token duration (minutes)**. +### Docker login attempt fails with: 'token signed by untrusted key' + +[Registry relies on GitLab to validate credentials](https://docs.gitlab.com/omnibus/architecture/registry/). +If the registry fails to authenticate valid login attempts, you get the following error message: + +```shell +# docker login gitlab.company.com:4567 +Username: user +Password: +Error response from daemon: login attempt to https://gitlab.company.com:4567/v2/ failed with status: 401 Unauthorized +``` + +And more specifically, this appears in the `/var/log/gitlab/registry/current` log file: + +```plaintext +level=info msg="token signed by untrusted key with ID: "TOKE:NL6Q:7PW6:EXAM:PLET:OKEN:BG27:RCIB:D2S3:EXAM:PLET:OKEN"" +level=warning msg="error authorizing context: invalid token" go.version=go1.12.7 http.request.host="gitlab.company.com:4567" http.request.id=74613829-2655-4f96-8991-1c9fe33869b8 http.request.method=GET http.request.remoteaddr=10.72.11.20 http.request.uri="/v2/" http.request.useragent="docker/19.03.2 go/go1.12.8 git-commit/6a30dfc kernel/3.10.0-693.2.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.2 \(linux\))" +``` + +GitLab uses the contents of the certificate key pair's two sides to encrypt the authentication token +for the Registry. This message means that those contents do not align. + +Check which files are in use: + +- `grep -A6 'auth:' /var/opt/gitlab/registry/config.yml` + + ```yaml + ## Container Registry Certificate + auth: + token: + realm: https://gitlab.my.net/jwt/auth + service: container_registry + issuer: omnibus-gitlab-issuer + --> rootcertbundle: /var/opt/gitlab/registry/gitlab-registry.crt + autoredirect: false + ``` + +- `grep -A9 'Container Registry' /var/opt/gitlab/gitlab-rails/etc/gitlab.yml` + + ```yaml + ## Container Registry Key + registry: + enabled: true + host: gitlab.company.com + port: 4567 + api_url: http://127.0.0.1:5000 # internal address to the registry, will be used by GitLab to directly communicate with API + path: /var/opt/gitlab/gitlab-rails/shared/registry + --> key: /var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key + issuer: omnibus-gitlab-issuer + notification_secret: + ``` + +The output of these `openssl` commands should match, proving that the cert-key pair is a match: + +```shell +openssl x509 -noout -modulus -in /var/opt/gitlab/registry/gitlab-registry.crt | openssl sha256 +openssl rsa -noout -modulus -in /var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key | openssl sha256 +``` + +If the two pieces of the certificate do not align, remove the files and run `gitlab-ctl reconfigure` +to regenerate the pair. If you have overridden the automatically generated self-signed pair with +your own certificates and have made sure that their contents align, you can delete the 'registry' +section in your `/etc/gitlab/gitlab-secrets.json` and run `gitlab-ctl reconfigure`. + ### AWS S3 with the GitLab registry error when pushing large images When using AWS S3 with the GitLab registry, an error may occur when pushing @@ -1144,7 +1210,7 @@ project or branch name. Special characters can include: - Trailing hyphen/dash - Double hyphen/dash -To get around this, you can [change the group path](../../user/group/index.md#changing-a-groups-path), +To get around this, you can [change the group path](../../user/group/index.md#change-a-groups-path), [change the project path](../../user/project/settings/index.md#renaming-a-repository) or change the branch name. Another option is to create a [push rule](../../push_rules/push_rules.md) to prevent this at the instance level. diff --git a/doc/administration/packages/dependency_proxy.md b/doc/administration/packages/dependency_proxy.md index 16cfdc8c784..720734bf344 100644 --- a/doc/administration/packages/dependency_proxy.md +++ b/doc/administration/packages/dependency_proxy.md @@ -47,6 +47,10 @@ To enable the dependency proxy feature: Since Puma is already the default web server for installations from source as of GitLab 12.9, no further changes are needed. +**Multi-node GitLab installations** + +Follow the steps for **Omnibus GitLab installation** for each Web and Sidekiq nodes. + ## Changing the storage path By default, the dependency proxy files are stored locally, but you can change the default diff --git a/doc/administration/packages/index.md b/doc/administration/packages/index.md index b1d2a82e31a..012afd6a419 100644 --- a/doc/administration/packages/index.md +++ b/doc/administration/packages/index.md @@ -43,14 +43,14 @@ guides you through the process. | CocoaPods | [#36890](https://gitlab.com/gitlab-org/gitlab/-/issues/36890) | | Conda | [#36891](https://gitlab.com/gitlab-org/gitlab/-/issues/36891) | | CRAN | [#36892](https://gitlab.com/gitlab-org/gitlab/-/issues/36892) | -| Debian | [WIP: Merge Request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/44746) | +| Debian | [Draft: Merge Request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/50438) | | Opkg | [#36894](https://gitlab.com/gitlab-org/gitlab/-/issues/36894) | | P2 | [#36895](https://gitlab.com/gitlab-org/gitlab/-/issues/36895) | | Puppet | [#36897](https://gitlab.com/gitlab-org/gitlab/-/issues/36897) | | RPM | [#5932](https://gitlab.com/gitlab-org/gitlab/-/issues/5932) | | RubyGems | [#803](https://gitlab.com/gitlab-org/gitlab/-/issues/803) | | SBT | [#36898](https://gitlab.com/gitlab-org/gitlab/-/issues/36898) | -| Terraform | [WIP: Merge Request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/18834) | +| Terraform | [Draft: Merge Request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/18834) | | Vagrant | [#36899](https://gitlab.com/gitlab-org/gitlab/-/issues/36899) | <!-- vale gitlab.Spelling = YES --> diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index 655e35c3e60..058445f00be 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -178,15 +178,21 @@ outside world. pages_external_url 'https://example.io' pages_nginx['redirect_http_to_https'] = true + ``` + +1. If you haven’t named your certificate and key `example.io.crt` and `example.io.key` +then you'll need to also add the full paths as shown below: + + ```ruby pages_nginx['ssl_certificate'] = "/etc/gitlab/ssl/pages-nginx.crt" pages_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/pages-nginx.key" ``` - where `pages-nginx.crt` and `pages-nginx.key` are the SSL cert and key, - respectively. - 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +WARNING: +Multiple wildcards for one instance is not supported. Only one wildcard per instance can be assigned. + ### Additional configuration for Docker container The GitLab Pages daemon doesn't have permissions to bind mounts when it runs @@ -214,56 +220,63 @@ and take effect after you [reconfigure GitLab](../restart_gitlab.md#omnibus-gitl Most of these settings don't need to be configured manually unless you need more granular control over how the Pages daemon runs and serves content in your environment. -| Setting | Description | -| ------- | ----------- | -| `pages_external_url` | The URL where GitLab Pages is accessible, including protocol (HTTP / HTTPS). If `https://` is used, additional configuration is required. See [Wildcard domains with TLS support](#wildcard-domains-with-tls-support) and [Custom domains with TLS support](#custom-domains-with-tls-support) for details. -| `gitlab_pages[]` | | -| `access_control` | Whether to enable [access control](index.md#access-control). -| `api_secret_key` | Full path to file with secret key used to authenticate with the GitLab API. Auto-generated when left unset. -| `artifacts_server` | Enable viewing [artifacts](../job_artifacts.md) in GitLab Pages. -| `artifacts_server_timeout` | Timeout (in seconds) for a proxied request to the artifacts server. -| `artifacts_server_url` | API URL to proxy artifact requests to. Defaults to GitLab `external URL` + `/api/v4`, for example `https://gitlab.com/api/v4`. When running a [separate Pages server](#running-gitlab-pages-on-a-separate-server), this URL must point to the main GitLab server's API. -| `auth_redirect_uri` | Callback URL for authenticating with GitLab. Defaults to project's subdomain of `pages_external_url` + `/auth`. -| `auth_secret` | Secret key for signing authentication requests. Leave blank to pull automatically from GitLab during OAuth registration. -| `dir` | Working directory for configuration and secrets files. -| `enable` | Enable or disable GitLab Pages on the current system. -| `external_http` | Configure Pages to bind to one or more secondary IP addresses, serving HTTP requests. Multiple addresses can be given as an array, along with exact ports, for example `['1.2.3.4', '1.2.3.5:8063']`. Sets value for `listen_http`. -| `external_https` | Configure Pages to bind to one or more secondary IP addresses, serving HTTPS requests. Multiple addresses can be given as an array, along with exact ports, for example `['1.2.3.4', '1.2.3.5:8063']`. Sets value for `listen_https`. -| `gitlab_client_http_timeout` | GitLab API HTTP client connection timeout in seconds (default: 10s). -| `gitlab_client_jwt_expiry` | JWT Token expiry time in seconds (default: 30s). -| `domain_config_source` | Domain configuration source (default: `auto`) -| `gitlab_id` | The OAuth application public ID. Leave blank to automatically fill when Pages authenticates with GitLab. -| `gitlab_secret` | The OAuth application secret. Leave blank to automatically fill when Pages authenticates with GitLab. -| `auth_scope` | The OAuth application scope to use for authentication. Must match GitLab Pages OAuth application settings. Leave blank to use `api` scope by default. -| `gitlab_server` | Server to use for authentication when access control is enabled; defaults to GitLab `external_url`. -| `headers` | Specify any additional http headers that should be sent to the client with each response. -| `inplace_chroot` | On [systems that don't support bind-mounts](index.md#additional-configuration-for-docker-container), this instructs GitLab Pages to `chroot` into its `pages_path` directory. Some caveats exist when using in-place `chroot`; refer to the GitLab Pages [README](https://gitlab.com/gitlab-org/gitlab-pages/blob/master/README.md#caveats) for more information. -| `insecure_ciphers` | Use default list of cipher suites, may contain insecure ones like 3DES and RC4. -| `internal_gitlab_server` | Internal GitLab server address used exclusively for API requests. Useful if you want to send that traffic over an internal load balancer. Defaults to GitLab `external_url`. -| `listen_proxy` | The addresses to listen on for reverse-proxy requests. Pages binds to these addresses' network sockets and receives incoming requests from them. Sets the value of `proxy_pass` in `$nginx-dir/conf/gitlab-pages.conf`. -| `log_directory` | Absolute path to a log directory. -| `log_format` | The log output format: `text` or `json`. -| `log_verbose` | Verbose logging, true/false. -| `max_connections` | Limit on the number of concurrent connections to the HTTP, HTTPS or proxy listeners. -| `metrics_address` | The address to listen on for metrics requests. -| `redirect_http` | Redirect pages from HTTP to HTTPS, true/false. -| `sentry_dsn` | The address for sending Sentry crash reporting to. -| `sentry_enabled` | Enable reporting and logging with Sentry, true/false. -| `sentry_environment` | The environment for Sentry crash reporting. -| `status_uri` | The URL path for a status page, for example, `/@status`. -| `tls_max_version` | Specifies the maximum SSL/TLS version ("ssl3", "tls1.0", "tls1.1" or "tls1.2"). -| `tls_min_version` | Specifies the minimum SSL/TLS version ("ssl3", "tls1.0", "tls1.1" or "tls1.2"). -| `use_http2` | Enable HTTP2 support. -| `gitlab_pages['env'][]` | | -| `http_proxy` | Configure GitLab Pages to use an HTTP Proxy to mediate traffic between Pages and GitLab. Sets an environment variable `http_proxy` when starting Pages daemon. -| `gitlab_rails[]` | | -| `pages_domain_verification_cron_worker` | Schedule for verifying custom GitLab Pages domains. -| `pages_domain_ssl_renewal_cron_worker` | Schedule for obtaining and renewing SSL certificates through Let's Encrypt for GitLab Pages domains. -| `pages_domain_removal_cron_worker` | Schedule for removing unverified custom GitLab Pages domains. -| `pages_path` | The directory on disk where pages are stored, defaults to `GITLAB-RAILS/shared/pages`. -| `pages_nginx[]` | | -| `enable` | Include a virtual host `server{}` block for Pages inside NGINX. Needed for NGINX to proxy traffic back to the Pages daemon. Set to `false` if the Pages daemon should directly receive all requests, for example, when using [custom domains](index.md#custom-domains). -| `FF_ENABLE_REDIRECTS` | Feature flag to disable redirects (enabled by default). Read the [redirects documentation](../../user/project/pages/redirects.md#disable-redirects) for more information. | +| Setting | Description | +|-----------------------------------------|-------------| +| `pages_external_url` | The URL where GitLab Pages is accessible, including protocol (HTTP / HTTPS). If `https://` is used, additional configuration is required. See [Wildcard domains with TLS support](#wildcard-domains-with-tls-support) and [Custom domains with TLS support](#custom-domains-with-tls-support) for details. | +| **`gitlab_pages[]`** | | +| `access_control` | Whether to enable [access control](index.md#access-control). | +| `api_secret_key` | Full path to file with secret key used to authenticate with the GitLab API. Auto-generated when left unset. | +| `artifacts_server` | Enable viewing [artifacts](../job_artifacts.md) in GitLab Pages. | +| `artifacts_server_timeout` | Timeout (in seconds) for a proxied request to the artifacts server. | +| `artifacts_server_url` | API URL to proxy artifact requests to. Defaults to GitLab `external URL` + `/api/v4`, for example `https://gitlab.com/api/v4`. When running a [separate Pages server](#running-gitlab-pages-on-a-separate-server), this URL must point to the main GitLab server's API. | +| `auth_redirect_uri` | Callback URL for authenticating with GitLab. Defaults to project's subdomain of `pages_external_url` + `/auth`. | +| `auth_secret` | Secret key for signing authentication requests. Leave blank to pull automatically from GitLab during OAuth registration. | +| `dir` | Working directory for configuration and secrets files. | +| `enable` | Enable or disable GitLab Pages on the current system. | +| `external_http` | Configure Pages to bind to one or more secondary IP addresses, serving HTTP requests. Multiple addresses can be given as an array, along with exact ports, for example `['1.2.3.4', '1.2.3.5:8063']`. Sets value for `listen_http`. | +| `external_https` | Configure Pages to bind to one or more secondary IP addresses, serving HTTPS requests. Multiple addresses can be given as an array, along with exact ports, for example `['1.2.3.4', '1.2.3.5:8063']`. Sets value for `listen_https`. | +| `gitlab_client_http_timeout` | GitLab API HTTP client connection timeout in seconds (default: 10s). | +| `gitlab_client_jwt_expiry` | JWT Token expiry time in seconds (default: 30s). | +| `gitlab_cache_expiry` | The maximum time a domain's configuration is stored in the cache (default: 600s). | +| `gitlab_cache_refresh` | The interval at which a domain's configuration is set to be due to refresh (default: 60s). | +| `gitlab_cache_cleanup` | The interval at which expired items are removed from the cache (default: 60s). | +| `gitlab_retrieval_timeout` | The maximum time to wait for a response from the GitLab API per request (default: 30s). | +| `gitlab_retrieval_interval` | The interval to wait before retrying to resolve a domain's configuration via the GitLab API (default: 1s). | +| `gitlab_retrieval_retries` | The maximum number of times to retry to resolve a domain's configuration via the API (default: 3). | +| `domain_config_source` | Domain configuration source (default: `auto`) | +| `gitlab_id` | The OAuth application public ID. Leave blank to automatically fill when Pages authenticates with GitLab. | +| `gitlab_secret` | The OAuth application secret. Leave blank to automatically fill when Pages authenticates with GitLab. | +| `auth_scope` | The OAuth application scope to use for authentication. Must match GitLab Pages OAuth application settings. Leave blank to use `api` scope by default. | +| `gitlab_server` | Server to use for authentication when access control is enabled; defaults to GitLab `external_url`. | +| `headers` | Specify any additional http headers that should be sent to the client with each response. Multiple headers can be given as an array, header and value as one string, for example `['my-header: myvalue', 'my-other-header: my-other-value']` | +| `inplace_chroot` | On [systems that don't support bind-mounts](index.md#additional-configuration-for-docker-container), this instructs GitLab Pages to `chroot` into its `pages_path` directory. Some caveats exist when using in-place `chroot`; refer to the GitLab Pages [README](https://gitlab.com/gitlab-org/gitlab-pages/blob/master/README.md#caveats) for more information. | +| `insecure_ciphers` | Use default list of cipher suites, may contain insecure ones like 3DES and RC4. | +| `internal_gitlab_server` | Internal GitLab server address used exclusively for API requests. Useful if you want to send that traffic over an internal load balancer. Defaults to GitLab `external_url`. | +| `listen_proxy` | The addresses to listen on for reverse-proxy requests. Pages binds to these addresses' network sockets and receives incoming requests from them. Sets the value of `proxy_pass` in `$nginx-dir/conf/gitlab-pages.conf`. | +| `log_directory` | Absolute path to a log directory. | +| `log_format` | The log output format: `text` or `json`. | +| `log_verbose` | Verbose logging, true/false. | +| `propagate_correlation_id` | Set to true (false by default) to re-use existing Correlation ID from the incoming request header `X-Request-ID` if present. If a reverse proxy sets this header, the value will be propagated in the request chain. | +| `max_connections` | Limit on the number of concurrent connections to the HTTP, HTTPS or proxy listeners. | +| `metrics_address` | The address to listen on for metrics requests. | +| `redirect_http` | Redirect pages from HTTP to HTTPS, true/false. | +| `sentry_dsn` | The address for sending Sentry crash reporting to. | +| `sentry_enabled` | Enable reporting and logging with Sentry, true/false. | +| `sentry_environment` | The environment for Sentry crash reporting. | +| `status_uri` | The URL path for a status page, for example, `/@status`. | +| `tls_max_version` | Specifies the maximum SSL/TLS version ("ssl3", "tls1.0", "tls1.1" or "tls1.2"). | +| `tls_min_version` | Specifies the minimum SSL/TLS version ("ssl3", "tls1.0", "tls1.1" or "tls1.2"). | +| `use_http2` | Enable HTTP2 support. | +| **`gitlab_pages['env'][]`** | | +| `http_proxy` | Configure GitLab Pages to use an HTTP Proxy to mediate traffic between Pages and GitLab. Sets an environment variable `http_proxy` when starting Pages daemon. | +| **`gitlab_rails[]`** | | +| `pages_domain_verification_cron_worker` | Schedule for verifying custom GitLab Pages domains. | +| `pages_domain_ssl_renewal_cron_worker` | Schedule for obtaining and renewing SSL certificates through Let's Encrypt for GitLab Pages domains. | +| `pages_domain_removal_cron_worker` | Schedule for removing unverified custom GitLab Pages domains. | +| `pages_path` | The directory on disk where pages are stored, defaults to `GITLAB-RAILS/shared/pages`. | +| **`pages_nginx[]`** | | +| `enable` | Include a virtual host `server{}` block for Pages inside NGINX. Needed for NGINX to proxy traffic back to the Pages daemon. Set to `false` if the Pages daemon should directly receive all requests, for example, when using [custom domains](index.md#custom-domains). | +| `FF_ENABLE_REDIRECTS` | Feature flag to disable redirects (enabled by default). Read the [redirects documentation](../../user/project/pages/redirects.md#disable-redirects) for more information. | --- @@ -347,7 +360,7 @@ When adding a custom domain, users are required to prove they own it by adding a GitLab-controlled verification code to the DNS records for that domain. If your user base is private or otherwise trusted, you can disable the -verification requirement. Navigate to **Admin Area > Settings > Preferences** and +verification requirement. Go to **Admin Area > Settings > Preferences** and uncheck **Require users to prove ownership of custom domains** in the **Pages** section. This setting is enabled by default. @@ -362,7 +375,7 @@ sites served under a custom domain. To enable it, you must: 1. Choose an email address on which you want to receive notifications about expiring domains. -1. Navigate to your instance's **Admin Area > Settings > Preferences** and expand **Pages** settings. +1. Go to your instance's **Admin Area > Settings > Preferences** and expand **Pages** settings. 1. Enter the email address for receiving notifications and accept Let's Encrypt's Terms of Service as shown below. 1. Click **Save changes**. @@ -403,6 +416,8 @@ all the App nodes and Sidekiq nodes. #### Using Pages with reduced authentication scope +> [Introduced](https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/423) in GitLab 13.10. + By default, the Pages daemon uses the `api` scope to authenticate. You can configure this. For example, this reduces the scope to `read_api` in `/etc/gitlab/gitlab.rb`: @@ -414,7 +429,7 @@ The scope to use for authentication must match the GitLab Pages OAuth applicatio pre-existing applications must modify the GitLab Pages OAuth application. Follow these steps to do this: -1. Navigate to your instance's **Admin Area > Settings > Applications** and expand **GitLab Pages** +1. Go to your instance's **Admin Area > Settings > Applications** and expand **GitLab Pages** settings. 1. Clear the `api` scope's checkbox and select the desired scope's checkbox (for example, `read_api`). @@ -432,7 +447,7 @@ This can be useful to preserve information published with Pages websites to the of your instance only. To do that: -1. Navigate to your instance's **Admin Area > Settings > Preferences** and expand **Pages** settings. +1. Go to your instance's **Admin Area > Settings > Preferences** and expand **Pages** settings. 1. Check the **Disable public access to Pages sites** checkbox. 1. Click **Save changes**. @@ -521,6 +536,25 @@ Follow the steps below to configure verbose logging of GitLab Pages daemon. 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +## Propagating the correlation ID + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/438) in GitLab 13.10. + +Setting the `propagate_correlation_id` to true will allow installations behind a reverse proxy generate +and set a correlation ID to requests sent to GitLab Pages. When a reverse proxy sets the header value `X-Request-ID`, +the value will be propagated in the request chain. +Users [can find the correlation ID in the logs](../troubleshooting/tracing_correlation_id.md#identify-the-correlation-id-for-a-request). + +To enable the propagation of the correlation ID: + +1. Set the parameter to true in `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_pages['propagate_correlation_id'] = true + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + ## Change storage path Follow the steps below to change the default path where GitLab Pages' contents @@ -599,13 +633,13 @@ The default is 100MB. To override the global maximum pages size for a specific project: -1. Navigate to your project's **Settings > Pages** page. +1. Go to your project's **Settings > Pages** page. 1. Edit the **Maximum size of pages**. 1. Click **Save changes**. To override the global maximum pages size for a specific group: -1. Navigate to your group's **Settings > General** page and expand **Pages**. +1. Go to your group's **Settings > General** page and expand **Pages**. 1. Edit the **Maximum size of pages**. 1. Click **Save changes**. @@ -718,9 +752,11 @@ The default value for Omnibus installations is `nil`. If left unchanged, GitLab Pages tries to use any available source (either `gitlab` or `disk`). The preferred source is `gitlab`, which uses [API-based configuration](#gitlab-api-based-configuration). +On large GitLab instances, using the API-based configuration will significantly improve the pages daemon startup time, as there is no need to load all custom domains configuration into memory. + For more details see this [blog post](https://about.gitlab.com/blog/2020/08/03/how-gitlab-pages-uses-the-gitlab-api-to-serve-content/). -### Deprecated domain_config_source +### Deprecated `domain_config_source` WARNING: The flag `gitlab_pages['domain_config_source']` is deprecated for use in [GitLab 13.9](https://gitlab.com/gitlab-org/gitlab/-/issues/217913), @@ -767,6 +803,44 @@ gitlab_pages['domain_config_source'] = "disk" For other common issues, see the [troubleshooting section](#failed-to-connect-to-the-internal-gitlab-api) or report an issue. +### GitLab API cache configuration + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/520) in GitLab 13.10. + +API-based configuration uses a caching mechanism to improve performance and reliability of serving Pages. +The cache behavior can be modified by changing the cache settings, however, the recommended values are set for you and should only be modified if needed. +Incorrect configuration of these values may result in intermittent +or persistent errors, or the Pages Daemon serving old content. + +NOTE: +Expiry, interval and timeout flags use [Golang's duration formatting](https://golang.org/pkg/time/#ParseDuration). +A duration string is a possibly signed sequence of decimal numbers, +each with optional fraction and a unit suffix, such as "300ms", "1.5h" or "2h45m". +Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". + +Examples: + +- Increasing `gitlab_cache_expiry` will allow items to exist in the cache longer. +This setting might be useful if the communication between GitLab Pages and GitLab Rails +is not stable. + +- Increasing `gitlab_cache_refresh` will reduce the frequency at which GitLab Pages +requests a domain's configuration from GitLab Rails. This setting might be useful +GitLab Pages generates too many requests to GitLab API and content does not change frequently. + +- Decreasing `gitlab_cache_cleanup` will remove expired items from the cache more frequently, +reducing the memory usage of your Pages node. + +- Decreasing `gitlab_retrieval_timeout` allows you to stop the request to GitLab Rails +more quickly. Increasing it will allow more time to receive a response from the API, +useful in slow networking environments. + +- Decreasing `gitlab_retrieval_interval` will make requests to the API more frequently, +only when there is an error response from the API, for example a connection timeout. + +- Decreasing `gitlab_retrieval_retries` will reduce the number of times a domain's +configuration is tried to be resolved automatically before reporting an error. + ## Using object storage > [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5577) in GitLab 13.6. @@ -837,7 +911,7 @@ In installations from source: remote_directory: "pages" # The bucket name connection: provider: AWS # Only AWS supported at the moment - aws_access_key_id: AWS_ACESS_KEY_ID + aws_access_key_id: AWS_ACCESS_KEY_ID aws_secret_access_key: AWS_SECRET_ACCESS_KEY region: eu-central-1 ``` @@ -944,14 +1018,15 @@ to define the explicit address that the GitLab Pages daemon should listen on: gitlab_pages['listen_proxy'] = '127.0.0.1:8090' ``` -### 404 error after transferring project to a different group or user +### 404 error after transferring the project to a different group or user, or changing project path If you encounter a `404 Not Found` error a Pages site after transferring a project to -another group or user, you must trigger a domain configuration update for Pages. To do -so, write something in the `.update` file. The Pages daemon monitors for changes to this -file, and reloads the configuration when changes occur. +another group or user, or changing project path, you must trigger a domain configuration +update for Pages. To do so, write something in the `.update` file. The Pages daemon +monitors for changes to this file, and reloads the configuration when changes occur. -Use this example to fix a `404 Not Found` error after transferring a project with Pages: +Use this example to fix a `404 Not Found` error after transferring a project or changing +a project path with Pages: ```shell date > /var/opt/gitlab/gitlab-rails/shared/pages/.update diff --git a/doc/administration/postgresql/replication_and_failover.md b/doc/administration/postgresql/replication_and_failover.md index e80e38fe5d1..a9a38ce1beb 100644 --- a/doc/administration/postgresql/replication_and_failover.md +++ b/doc/administration/postgresql/replication_and_failover.md @@ -900,7 +900,12 @@ You can switch an exiting database cluster to use Patroni instead of repmgr with ``` 1. Repeat the last two steps for all replica nodes. `gitlab.rb` should look the same on all nodes. -1. Optional: You can remove `gitlab_repmgr` database and role on the primary. +1. If present, remove the `gitlab_repmgr` database and role on the primary. If you don't delete the `gitlab_repmgr` + database, upgrading PostgreSQL 11 to 12 will fail with: + + ```plaintext + could not load library "$libdir/repmgr_funcs": ERROR: could not access file "$libdir/repmgr_funcs": No such file or directory + ``` ### Upgrading PostgreSQL major version in a Patroni cluster diff --git a/doc/administration/pseudonymizer.md b/doc/administration/pseudonymizer.md index 5f1272b1f4a..4aa73212e43 100644 --- a/doc/administration/pseudonymizer.md +++ b/doc/administration/pseudonymizer.md @@ -24,7 +24,7 @@ be textually exported. This ensures that: ## Configuration -To configure the pseudonymizer, you need to: +To configure the Pseudonymizer, you need to: - Provide a manifest file that describes which fields should be included or pseudonymized ([example `manifest.yml` file](https://gitlab.com/gitlab-org/gitlab/tree/master/config/pseudonymizer.yml)). @@ -87,7 +87,7 @@ To configure the pseudonymizer, you need to: ## Usage -You can optionally run the pseudonymizer using the following environment variables: +You can optionally run the Pseudonymizer using the following environment variables: - `PSEUDONYMIZER_OUTPUT_DIR` - where to store the output CSV files (defaults to `/tmp`) - `PSEUDONYMIZER_BATCH` - the batch size when querying the DB (defaults to `100000`) diff --git a/doc/administration/raketasks/check.md b/doc/administration/raketasks/check.md index 5ec268ac769..8d2ca103c82 100644 --- a/doc/administration/raketasks/check.md +++ b/doc/administration/raketasks/check.md @@ -245,3 +245,9 @@ Upload.find_each do |upload| end p "#{uploads_deleted} remote objects were destroyed." ``` + +### Delete references to missing LFS objects + +If `gitlab-rake gitlab:lfs:check VERBOSE=1` detects LFS objects that exist in the database +but not on disk, [follow the procedure in the LFS documentation](../../topics/git/lfs/index.md#missing-lfs-objects) +to remove the database entries. diff --git a/doc/administration/raketasks/github_import.md b/doc/administration/raketasks/github_import.md index c29865be56c..0338732e886 100644 --- a/doc/administration/raketasks/github_import.md +++ b/doc/administration/raketasks/github_import.md @@ -19,8 +19,8 @@ before/after the brackets. Also, some shells (for example, `zsh`) can interpret ## Caveats -If the GitHub [rate limit](https://docs.github.com/v3/#rate-limiting) is reached while importing, -the importing process waits (`sleep()`) until it can continue importing. +If the GitHub [rate limit](https://docs.github.com/en/rest/reference/rate-limit) is reached while +importing, the importing process waits (`sleep()`) until it can continue importing. ## Importing multiple projects diff --git a/doc/administration/raketasks/maintenance.md b/doc/administration/raketasks/maintenance.md index 3494ceb701e..fa95f38f37c 100644 --- a/doc/administration/raketasks/maintenance.md +++ b/doc/administration/raketasks/maintenance.md @@ -110,7 +110,7 @@ The `gitlab:check` Rake task runs the following Rake tasks: It checks that each component was set up according to the installation guide and suggest fixes for issues found. This command must be run from your application server and doesn't work correctly on -component servers like [Gitaly](../gitaly/index.md#run-gitaly-on-its-own-server). +component servers like [Gitaly](../gitaly/configure_gitaly.md#run-gitaly-on-its-own-server). You may also have a look at our troubleshooting guides for: @@ -246,8 +246,9 @@ have been corrupted, you should reinstall the omnibus package. ## Check TCP connectivity to a remote site Sometimes you need to know if your GitLab installation can connect to a TCP -service on another machine - perhaps a PostgreSQL or HTTPS server. A Rake task -is included to help you with this: +service on another machine (for example a PostgreSQL or web server) +in order to troubleshoot proxy issues. +A Rake task is included to help you with this. **Omnibus Installation** diff --git a/doc/administration/read_only_gitlab.md b/doc/administration/read_only_gitlab.md index cb7b621bf16..698da80a07c 100644 --- a/doc/administration/read_only_gitlab.md +++ b/doc/administration/read_only_gitlab.md @@ -6,10 +6,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Place GitLab into a read-only state **(FREE SELF)** -WARNING: -This document should be used as a temporary solution. -There's work in progress to make this -[possible with Geo](https://gitlab.com/groups/gitlab-org/-/epics/2149). +NOTE: +In GitLab 13.9 and later, the recommended method to +place GitLab in a read-only state is to enable +[maintenance mode](../administration/maintenance_mode/index.md). In some cases, you might want to place GitLab under a read-only state. The configuration for doing so depends on your desired outcome. diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index d4d522ab1b8..a51641f661f 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -26,14 +26,14 @@ full list of reference architectures, see | Redis - Queues / Shared State | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | | Redis Sentinel - Cache | 3 | 1 vCPU, 1.7 GB memory | g1-small | t3.small | B1MS | | Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7 GB memory | g1-small | t3.small | B1MS | -| Gitaly Cluster | 3 | 16 vCPU, 60 GB memory | n1-standard-16 | m5.4xlarge | D16s v3 | +| Gitaly | 3 | 16 vCPU, 60 GB memory | n1-standard-16 | m5.4xlarge | D16s v3 | | Praefect | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | | Praefect PostgreSQL | 1+* | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | | Sidekiq | 4 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | | GitLab Rails | 3 | 32 vCPU, 28.8 GB memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | | Monitoring node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | | Object storage | n/a | n/a | n/a | n/a | n/a | -| NFS server | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| NFS server | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | ```plantuml @startuml 10k @@ -206,7 +206,7 @@ The following list includes descriptions of each server and its assigned IP: - `10.6.0.111`: GitLab application 1 - `10.6.0.112`: GitLab application 2 - `10.6.0.113`: GitLab application 3 -- `10.6.0.121`: Prometheus +- `10.6.0.151`: Prometheus ## Configure the external load balancer @@ -1508,7 +1508,7 @@ Praefect requires several secret tokens to secure communications across the Clus Gitaly Cluster nodes are configured in Praefect via a `virtual storage`. Each storage contains the details of each Gitaly node that makes up the cluster. Each storage is also given a name -and this name is used in several areas of the config. In this guide, the name of the storage will be +and this name is used in several areas of the configuration. In this guide, the name of the storage will be `default`. Also, this guide is geared towards new installs, if upgrading an existing environment to use Gitaly Cluster, you may need to use a different name. Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more info. @@ -1756,7 +1756,7 @@ Note the following: necessary. - The Internal Load Balancer will also access to the certificates and need to be configured - to allow for TLS passthrough. + to allow for TLS pass-through. Refer to the load balancers documentation on how to configure this. To configure Praefect with TLS: @@ -1927,7 +1927,7 @@ To configure the Sidekiq nodes, on each one: node_exporter['listen_address'] = '0.0.0.0:9100' # Rails Status for prometheus - gitlab_rails['monitoring_whitelist'] = ['10.6.0.121/32', '127.0.0.0/8'] + gitlab_rails['monitoring_whitelist'] = ['10.6.0.151/32', '127.0.0.0/8'] ############################# ### Object storage ### @@ -1940,13 +1940,13 @@ To configure the Sidekiq nodes, on each one: 'google_project' => '<gcp-project-name>', 'google_json_key_location' => '<path-to-gcp-service-account-key>' } - gitlab_rails['object_store']['objects']['artifacts']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['lfs']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['uploads']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-bucket-name>" + gitlab_rails['object_store']['objects']['artifacts']['bucket'] = "<gcp-artifacts-bucket-name>" + gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = "<gcp-external-diffs-bucket-name>" + gitlab_rails['object_store']['objects']['lfs']['bucket'] = "<gcp-lfs-bucket-name>" + gitlab_rails['object_store']['objects']['uploads']['bucket'] = "<gcp-uploads-bucket-name>" + gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-packages-bucket-name>" + gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>" + gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>" ``` 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace @@ -2055,8 +2055,8 @@ On each node perform the following: # Add the monitoring node's IP address to the monitoring whitelist and allow it to # scrape the NGINX metrics - gitlab_rails['monitoring_whitelist'] = ['10.6.0.121/32', '127.0.0.0/8'] - nginx['status']['options']['allow'] = ['10.6.0.121/32', '127.0.0.0/8'] + gitlab_rails['monitoring_whitelist'] = ['10.6.0.151/32', '127.0.0.0/8'] + nginx['status']['options']['allow'] = ['10.6.0.151/32', '127.0.0.0/8'] ############################# ### Object storage ### @@ -2069,13 +2069,13 @@ On each node perform the following: 'google_project' => '<gcp-project-name>', 'google_json_key_location' => '<path-to-gcp-service-account-key>' } - gitlab_rails['object_store']['objects']['artifacts']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['lfs']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['uploads']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-bucket-name>" + gitlab_rails['object_store']['objects']['artifacts']['bucket'] = "<gcp-artifacts-bucket-name>" + gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = "<gcp-external-diffs-bucket-name>" + gitlab_rails['object_store']['objects']['lfs']['bucket'] = "<gcp-lfs-bucket-name>" + gitlab_rails['object_store']['objects']['uploads']['bucket'] = "<gcp-uploads-bucket-name>" + gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-packages-bucket-name>" + gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>" + gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>" ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -2192,7 +2192,7 @@ running [Prometheus](../monitoring/prometheus/index.md) and The following IP will be used as an example: -- `10.6.0.121`: Prometheus +- `10.6.0.151`: Prometheus To configure the Monitoring node: @@ -2303,20 +2303,9 @@ on what features you intend to use: | [Terraform state files](../terraform_state.md#using-object-storage) | Yes | Using separate buckets for each data type is the recommended approach for GitLab. - -A limitation of our configuration is that each use of object storage is separately configured. -[We have an issue for improving this](https://gitlab.com/gitlab-org/gitlab/-/issues/23345) -and easily using one bucket with separate folders is one improvement that this might bring. - -There is at least one specific issue with using the same bucket: -when GitLab is deployed with the Helm chart restore from backup -[will not properly function](https://docs.gitlab.com/charts/advanced/external-object-storage/#lfs-artifacts-uploads-packages-external-diffs-pseudonymizer) -unless separate buckets are used. - -One risk of using a single bucket would be if your organization decided to -migrate GitLab to the Helm deployment in the future. GitLab would run, but the situation with -backups might not be realized until the organization had a critical requirement for the backups to -work. +This ensures there are no collisions across the various types of data GitLab stores. +There are plans to [enable the use of a single bucket](https://gitlab.com/gitlab-org/gitlab/-/issues/292958) +in the future. <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -2324,7 +2313,7 @@ work. </a> </div> -## Configure Advanced Search **(PREMIUM SELF)** +## Configure Advanced Search You can leverage Elasticsearch and [enable Advanced Search](../../integration/elasticsearch.md) for faster, more advanced code search across your entire GitLab instance. @@ -2360,16 +2349,6 @@ as soon as possible. </a> </div> -## Troubleshooting - -See the [troubleshooting documentation](troubleshooting.md). - -<div align="right"> - <a type="button" class="btn btn-default" href="#setup-components"> - Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> - </a> -</div> - ## Cloud Native Deployment (optional) Hybrid installations leverage the benefits of both cloud native and traditional @@ -2422,3 +2401,9 @@ Webservice pods. Expand available resources using the ratio of 1 vCPU to 1.25 GB _per each worker process_ for each additional Webservice pod. For further information on resource usage, see the [Webservice resources](https://docs.gitlab.com/charts/charts/gitlab/webservice/#resources). + +<div align="right"> + <a type="button" class="btn btn-default" href="#setup-components"> + Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> + </a> +</div> diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index 48c72bb930d..8cf9efe1d2c 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -12,100 +12,110 @@ full list of reference architectures, see [Available reference architectures](index.md#available-reference-architectures). > - **Supported users (approximate):** 25,000 -> - **High Availability:** Yes +> - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) > - **Test requests per second (RPS) rates:** API: 500 RPS, Web: 50 RPS, Git (Pull): 50 RPS, Git (Push): 10 RPS | Service | Nodes | Configuration | GCP | AWS | Azure | |-----------------------------------------|-------------|-------------------------|-----------------|-------------|----------| -| External load balancing node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | -| Consul | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | -| PostgreSQL | 3 | 8 vCPU, 30 GB memory | n1-standard-8 | `m5.2xlarge` | D8s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | -| Internal load balancing node | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | `c5.large` | F2s v2 | -| Redis - Cache | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | -| Redis - Queues / Shared State | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | -| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7 GB memory | g1-small | `t3.small` | B1MS | -| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7 GB memory | g1-small | `t3.small` | B1MS | -| Gitaly | 2 (minimum) | 32 vCPU, 120 GB memory | n1-standard-32 | `m5.8xlarge` | D32s v3 | -| Sidekiq | 4 | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | -| GitLab Rails | 5 | 32 vCPU, 28.8 GB memory | n1-highcpu-32 | `c5.9xlarge` | F32s v2 | -| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | +| External load balancing node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| Consul | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| PostgreSQL | 3 | 16 vCPU, 60 GB memory | n1-standard-16 | m5.4xlarge | D16s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Internal load balancing node | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | c5.large | F2s v2 | +| Redis - Cache | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| Redis - Queues / Shared State | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7 GB memory | g1-small | t3.small | B1MS | +| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7 GB memory | g1-small | t3.small | B1MS | +| Gitaly | 3 | 32 vCPU, 120 GB memory | n1-standard-32 | m5.8xlarge | D32s v3 | +| Praefect | 3 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| Praefect PostgreSQL | 1+* | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Sidekiq | 4 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| GitLab Rails | 5 | 32 vCPU, 28.8 GB memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | +| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | | Object storage | n/a | n/a | n/a | n/a | n/a | -| NFS server | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | - -```mermaid -stateDiagram-v2 - [*] --> LoadBalancer - LoadBalancer --> ApplicationServer - - ApplicationServer --> BackgroundJobs - ApplicationServer --> Gitaly - ApplicationServer --> Redis_Cache - ApplicationServer --> Redis_Queues - ApplicationServer --> PgBouncer - PgBouncer --> Database - ApplicationServer --> ObjectStorage - BackgroundJobs --> ObjectStorage - - ApplicationMonitoring -->ApplicationServer - ApplicationMonitoring -->PgBouncer - ApplicationMonitoring -->Database - ApplicationMonitoring -->BackgroundJobs - - ApplicationServer --> Consul - - Consul --> Database - Consul --> PgBouncer - Redis_Cache --> Consul - Redis_Queues --> Consul - BackgroundJobs --> Consul - - state Consul { - "Consul_1..3" - } - - state Database { - "PG_Primary_Node" - "PG_Secondary_Node_1..2" - } - - state Redis_Cache { - "R_Cache_Primary_Node" - "R_Cache_Replica_Node_1..2" - "R_Cache_Sentinel_1..3" - } - - state Redis_Queues { - "R_Queues_Primary_Node" - "R_Queues_Replica_Node_1..2" - "R_Queues_Sentinel_1..3" - } - - state Gitaly { - "Gitaly_1..2" - } - - state BackgroundJobs { - "Sidekiq_1..4" - } - - state ApplicationServer { - "GitLab_Rails_1..5" - } - - state LoadBalancer { - "LoadBalancer_1" - } - - state ApplicationMonitoring { - "Prometheus" - "Grafana" - } - - state PgBouncer { - "Internal_Load_Balancer" - "PgBouncer_1..3" - } +| NFS server | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | + +```plantuml +@startuml 25k +card "**External Load Balancer**" as elb #6a9be7 +card "**Internal Load Balancer**" as ilb #9370DB + +together { + collections "**GitLab Rails** x5" as gitlab #32CD32 + collections "**Sidekiq** x4" as sidekiq #ff8dd1 +} + +together { + card "**Prometheus + Grafana**" as monitor #7FFFD4 + collections "**Consul** x3" as consul #e76a9b +} + +card "Gitaly Cluster" as gitaly_cluster { + collections "**Praefect** x3" as praefect #FF8C00 + collections "**Gitaly** x3" as gitaly #FF8C00 + card "**Praefect PostgreSQL***\n//Non fault-tolerant//" as praefect_postgres #FF8C00 + + praefect -[#FF8C00]-> gitaly + praefect -[#FF8C00]> praefect_postgres +} + +card "Database" as database { + collections "**PGBouncer** x3" as pgbouncer #4EA7FF + card "**PostgreSQL** (Primary)" as postgres_primary #4EA7FF + collections "**PostgreSQL** (Secondary) x2" as postgres_secondary #4EA7FF + + pgbouncer -[#4EA7FF]-> postgres_primary + postgres_primary .[#4EA7FF]> postgres_secondary +} + +card "redis" as redis { + collections "**Redis Persistent** x3" as redis_persistent #FF6347 + collections "**Redis Cache** x3" as redis_cache #FF6347 + collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347 + collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347 + + redis_persistent <.[#FF6347]- redis_persistent_sentinel + redis_cache <.[#FF6347]- redis_cache_sentinel +} + +cloud "**Object Storage**" as object_storage #white + +elb -[#6a9be7]-> gitlab +elb -[#6a9be7]--> monitor + +gitlab -[#32CD32]> sidekiq +gitlab -[#32CD32]--> ilb +gitlab -[#32CD32]-> object_storage +gitlab -[#32CD32]---> redis +gitlab -[hidden]-> monitor +gitlab -[hidden]-> consul + +sidekiq -[#ff8dd1]--> ilb +sidekiq -[#ff8dd1]-> object_storage +sidekiq -[#ff8dd1]---> redis +sidekiq -[hidden]-> monitor +sidekiq -[hidden]-> consul + +ilb -[#9370DB]-> gitaly_cluster +ilb -[#9370DB]-> database + +consul .[#e76a9b]u-> gitlab +consul .[#e76a9b]u-> sidekiq +consul .[#e76a9b]> monitor +consul .[#e76a9b]-> database +consul .[#e76a9b]-> gitaly_cluster +consul .[#e76a9b,norank]--> redis + +monitor .[#7FFFD4]u-> gitlab +monitor .[#7FFFD4]u-> sidekiq +monitor .[#7FFFD4]> consul +monitor .[#7FFFD4]-> database +monitor .[#7FFFD4]-> gitaly_cluster +monitor .[#7FFFD4,norank]--> redis +monitor .[#7FFFD4]> ilb +monitor .[#7FFFD4,norank]u--> elb + +@enduml ``` The Google Cloud Platform (GCP) architectures were built and tested using the @@ -120,19 +130,25 @@ uploads, or artifacts), using an [object storage service](#configure-the-object- is recommended instead of using NFS. Using an object storage service also doesn't require you to provision and maintain a node. +It's also worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and +that to achieve full High Availability a third party PostgreSQL database solution will be required. +We hope to offer a built in solutions for these restrictions in the future but in the meantime a non HA PostgreSQL server +can be set up via Omnibus GitLab, which the above specs reflect. Refer to the following issues for more information: [`omnibus-gitlab#5919`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919) & [`gitaly#3398`](https://gitlab.com/gitlab-org/gitaly/-/issues/3398) + ## Setup components To set up GitLab and its components to accommodate up to 25,000 users: -1. [Configure the external load balancing node](#configure-the-external-load-balancer) +1. [Configure the external load balancer](#configure-the-external-load-balancer) to handle the load balancing of the GitLab application services nodes. +1. [Configure the internal load balancer](#configure-the-internal-load-balancer). + to handle the load balancing of GitLab application internal connections. 1. [Configure Consul](#configure-consul). 1. [Configure PostgreSQL](#configure-postgresql), the database for GitLab. 1. [Configure PgBouncer](#configure-pgbouncer). -1. [Configure the internal load balancing node](#configure-the-internal-load-balancer). 1. [Configure Redis](#configure-redis). -1. [Configure Gitaly](#configure-gitaly), - which provides access to the Git repositories. +1. [Configure Gitaly Cluster](#configure-gitaly-cluster), + provides access to the Git repositories. 1. [Configure Sidekiq](#configure-sidekiq). 1. [Configure the main GitLab Rails application](#configure-gitlab-rails) to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend @@ -178,6 +194,11 @@ The following list includes descriptions of each server and its assigned IP: - `10.6.0.83`: Sentinel - Queues 3 - `10.6.0.91`: Gitaly 1 - `10.6.0.92`: Gitaly 2 +- `10.6.0.93`: Gitaly 3 +- `10.6.0.131`: Praefect 1 +- `10.6.0.132`: Praefect 2 +- `10.6.0.133`: Praefect 3 +- `10.6.0.141`: Praefect PostgreSQL 1 (non HA) - `10.6.0.101`: Sidekiq 1 - `10.6.0.102`: Sidekiq 2 - `10.6.0.103`: Sidekiq 3 @@ -185,7 +206,9 @@ The following list includes descriptions of each server and its assigned IP: - `10.6.0.111`: GitLab application 1 - `10.6.0.112`: GitLab application 2 - `10.6.0.113`: GitLab application 3 -- `10.6.0.121`: Prometheus +- `10.6.0.114`: GitLab application 4 +- `10.6.0.115`: GitLab application 5 +- `10.6.0.151`: Prometheus ## Configure the external load balancer @@ -308,6 +331,71 @@ Configure DNS for an alternate SSH hostname such as `altssh.gitlab.example.com`. </a> </div> +## Configure the internal load balancer + +The Internal Load Balancer is used to balance any internal connections the GitLab environment requires +such as connections to [PgBouncer](#configure-pgbouncer) and [Praefect](#configure-praefect) (Gitaly Cluster). + +Note that it's a separate node from the External Load Balancer and shouldn't have any access externally. + +The following IP will be used as an example: + +- `10.6.0.40`: Internal Load Balancer + +Here's how you could do it with [HAProxy](https://www.haproxy.org/): + +```plaintext +global + log /dev/log local0 + log localhost local1 notice + log stdout format raw local0 + +defaults + log global + default-server inter 10s fall 3 rise 2 + balance leastconn + +frontend internal-pgbouncer-tcp-in + bind *:6432 + mode tcp + option tcplog + + default_backend pgbouncer + +frontend internal-praefect-tcp-in + bind *:2305 + mode tcp + option tcplog + option clitcpka + + default_backend praefect + +backend pgbouncer + mode tcp + option tcp-check + + server pgbouncer1 10.6.0.21:6432 check + server pgbouncer2 10.6.0.22:6432 check + server pgbouncer3 10.6.0.23:6432 check + +backend praefect + mode tcp + option tcp-check + option srvtcpka + + server praefect1 10.6.0.131:2305 check + server praefect2 10.6.0.132:2305 check + server praefect3 10.6.0.133:2305 check +``` + +Refer to your preferred Load Balancer's documentation for further guidance. + +<div align="right"> + <a type="button" class="btn btn-default" href="#setup-components"> + Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> + </a> +</div> + ## Configure Consul The following IPs will be used as an example: @@ -662,52 +750,6 @@ The following IPs will be used as an example: </a> </div> -### Configure the internal load balancer - -If you're running more than one PgBouncer node as recommended, then at this time you'll need to set -up a TCP internal load balancer to serve each correctly. - -The following IP will be used as an example: - -- `10.6.0.40`: Internal Load Balancer - -Here's how you could do it with [HAProxy](https://www.haproxy.org/): - -```plaintext -global - log /dev/log local0 - log localhost local1 notice - log stdout format raw local0 - -defaults - log global - default-server inter 10s fall 3 rise 2 - balance leastconn - -frontend internal-pgbouncer-tcp-in - bind *:6432 - mode tcp - option tcplog - - default_backend pgbouncer - -backend pgbouncer - mode tcp - option tcp-check - - server pgbouncer1 10.6.0.21:6432 check - server pgbouncer2 10.6.0.22:6432 check - server pgbouncer3 10.6.0.23:6432 check -``` - -Refer to your preferred Load Balancer's documentation for further guidance. - -<div align="right"> - <a type="button" class="btn btn-default" href="#setup-components"> - Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> - </a> -</div> - ## Configure Redis Using [Redis](https://redis.io/) in scalable environment is possible using a **Primary** x **Replica** @@ -1302,19 +1344,283 @@ To configure the Sentinel Queues server: </a> </div> -## Configure Gitaly +## Configure Gitaly Cluster -NOTE: -[Gitaly Cluster](../gitaly/praefect.md) support -for the Reference Architectures is being -worked on as a [collaborative effort](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/1) between the Quality Engineering and Gitaly teams. When this component has been verified -some Architecture specs will likely change as a result to support the new -and improved designed. +[Gitaly Cluster](../gitaly/praefect.md) is a GitLab provided and recommended fault tolerant solution for storing Git repositories. +In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down. + +The recommended cluster setup includes the following components: + +- 3 Gitaly nodes: Replicated storage of Git repositories. +- 3 Praefect nodes: Router and transaction manager for Gitaly Cluster. +- 1 Praefect PostgreSQL node: Database server for Praefect. A third-party solution + is required for Praefect database connections to be made highly available. +- 1 load balancer: A load balancer is required for Praefect. The + [internal load balancer](#configure-the-internal-load-balancer) will be used. + +This section will detail how to configure the recommended standard setup in order. +For more advanced setups refer to the [standalone Gitaly Cluster documentation](../gitaly/praefect.md). + +### Configure Praefect PostgreSQL + +Praefect, the routing and transaction manager for Gitaly Cluster, requires its own database server to store data on Gitaly Cluster status. + +If you want to have a highly available setup, Praefect requires a third-party PostgreSQL database. +A built-in solution is being [worked on](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919). + +#### Praefect non-HA PostgreSQL standalone using Omnibus GitLab + +The following IPs will be used as an example: + +- `10.6.0.141`: Praefect PostgreSQL + +First, make sure to [install](https://about.gitlab.com/install/) +the Linux GitLab package in the Praefect PostgreSQL node. Following the steps, +install the necessary dependencies from step 1, and add the +GitLab package repository from step 2. When installing GitLab +in the second step, do not supply the `EXTERNAL_URL` value. + +1. SSH in to the Praefect PostgreSQL node. +1. Create a strong password to be used for the Praefect PostgreSQL user. Take note of this password as `<praefect_postgresql_password>`. +1. Generate the password hash for the Praefect PostgreSQL username/password pair. This assumes you will use the default + username of `praefect` (recommended). The command will request the password `<praefect_postgresql_password>` + and confirmation. Use the value that is output by this command in the next + step as the value of `<praefect_postgresql_password_hash>`: + + ```shell + sudo gitlab-ctl pg-password-md5 praefect + ``` + +1. Edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: + + ```ruby + # Disable all components except PostgreSQL and Consul + roles ['postgres_role'] + repmgr['enable'] = false + patroni['enable'] = false + + # PostgreSQL configuration + postgresql['listen_address'] = '0.0.0.0' + postgresql['max_connections'] = 200 + + gitlab_rails['auto_migrate'] = false + + # Configure the Consul agent + consul['enable'] = true + ## Enable service discovery for Prometheus + consul['monitoring_service_discovery'] = true + + # START user configuration + # Please set the real values as explained in Required Information section + # + # Replace PRAEFECT_POSTGRESQL_PASSWORD_HASH with a generated md5 value + postgresql['sql_user_password'] = "<praefect_postgresql_password_hash>" + + # Replace XXX.XXX.XXX.XXX/YY with Network Address + postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/24) + + # Set the network addresses that the exporters will listen on for monitoring + node_exporter['listen_address'] = '0.0.0.0:9100' + postgres_exporter['listen_address'] = '0.0.0.0:9187' + + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + # + # END user configuration + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Follow the [post configuration](#praefect-postgresql-post-configuration). + +<div align="right"> + <a type="button" class="btn btn-default" href="#setup-components"> + Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> + </a> +</div> + +#### Praefect HA PostgreSQL third-party solution + +[As noted](#configure-praefect-postgresql), a third-party PostgreSQL solution for +Praefect's database is recommended if aiming for full High Availability. + +There are many third-party solutions for PostgreSQL HA. The solution selected must have the following to work with Praefect: + +- A static IP for all connections that doesn't change on failover. +- [`LISTEN`](https://www.postgresql.org/docs/12/sql-listen.html) SQL functionality must be supported. + +Examples of the above could include [Google's Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) or [Amazon RDS](https://aws.amazon.com/rds/). + +Once the database is set up, follow the [post configuration](#praefect-postgresql-post-configuration). + +#### Praefect PostgreSQL post-configuration + +After the Praefect PostgreSQL server has been set up, you'll then need to configure the user and database for Praefect to use. + +We recommend the user be named `praefect` and the database `praefect_production`, and these can be configured as standard in PostgreSQL. +The password for the user is the same as the one you configured earlier as `<praefect_postgresql_password>`. -[Gitaly](../gitaly/index.md) server node requirements are dependent on data, -specifically the number of projects and those projects' sizes. It's recommended -that a Gitaly server node stores no more than 5 TB of data. Depending on your -repository storage requirements, you may require additional Gitaly server nodes. +This is how this would work with a Omnibus GitLab PostgreSQL setup: + +1. SSH in to the Praefect PostgreSQL node. +1. Connect to the PostgreSQL server with administrative access. + The `gitlab-psql` user should be used here for this as it's added by default in Omnibus. + The database `template1` is used because it is created by default on all PostgreSQL servers. + + ```shell + /opt/gitlab/embedded/bin/psql -U gitlab-psql -d template1 -h POSTGRESQL_SERVER_ADDRESS + ``` + +1. Create the new user `praefect`, replacing `<praefect_postgresql_password>`: + + ```shell + CREATE ROLE praefect WITH LOGIN CREATEDB PASSWORD <praefect_postgresql_password>; + ``` + +1. Reconnect to the PostgreSQL server, this time as the `praefect` user: + + ```shell + /opt/gitlab/embedded/bin/psql -U praefect -d template1 -h POSTGRESQL_SERVER_ADDRESS + ``` + +1. Create a new database `praefect_production`: + + ```shell + CREATE DATABASE praefect_production WITH ENCODING=UTF8; + ``` + +<div align="right"> + <a type="button" class="btn btn-default" href="#setup-components"> + Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> + </a> +</div> + +### Configure Praefect + +Praefect is the router and transaction manager for Gitaly Cluster and all connections to Gitaly go through +it. This section details how to configure it. + +Praefect requires several secret tokens to secure communications across the Cluster: + +- `<praefect_external_token>`: Used for repositories hosted on your Gitaly cluster and can only be accessed by Gitaly clients that carry this token. +- `<praefect_internal_token>`: Used for replication traffic inside your Gitaly cluster. This is distinct from `praefect_external_token` because Gitaly clients must not be able to access internal nodes of the Praefect cluster directly; that could lead to data loss. +- `<praefect_postgresql_password>`: The Praefect PostgreSQL password defined in the previous section is also required as part of this setup. + +Gitaly Cluster nodes are configured in Praefect via a `virtual storage`. Each storage contains +the details of each Gitaly node that makes up the cluster. Each storage is also given a name +and this name is used in several areas of the config. In this guide, the name of the storage will be +`default`. Also, this guide is geared towards new installs, if upgrading an existing environment +to use Gitaly Cluster, you may need to use a different name. +Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more info. + +The following IPs will be used as an example: + +- `10.6.0.131`: Praefect 1 +- `10.6.0.132`: Praefect 2 +- `10.6.0.133`: Praefect 3 + +To configure the Praefect nodes, on each one: + +1. SSH in to the Praefect server. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to follow _only_ installation steps 1 and 2 + on the page. +1. Edit the `/etc/gitlab/gitlab.rb` file to configure Praefect: + + ```ruby + # Avoid running unnecessary services on the Gitaly server + postgresql['enable'] = false + redis['enable'] = false + nginx['enable'] = false + puma['enable'] = false + unicorn['enable'] = false + sidekiq['enable'] = false + gitlab_workhorse['enable'] = false + grafana['enable'] = false + + # If you run a separate monitoring node you can disable these services + alertmanager['enable'] = false + prometheus['enable'] = false + + # Praefect Configuration + praefect['enable'] = true + praefect['listen_addr'] = '0.0.0.0:2305' + + gitlab_rails['rake_cache_clear'] = false + gitlab_rails['auto_migrate'] = false + + # Configure the Consul agent + consul['enable'] = true + ## Enable service discovery for Prometheus + consul['monitoring_service_discovery'] = true + + # START user configuration + # Please set the real values as explained in Required Information section + # + + # Praefect External Token + # This is needed by clients outside the cluster (like GitLab Shell) to communicate with the Praefect cluster + praefect['auth_token'] = '<praefect_external_token>' + + # Praefect Database Settings + praefect['database_host'] = '10.6.0.141' + praefect['database_port'] = 5432 + # `no_proxy` settings must always be a direct connection for caching + praefect['database_host_no_proxy'] = '10.6.0.141' + praefect['database_port_no_proxy'] = 5432 + praefect['database_dbname'] = 'praefect_production' + praefect['database_user'] = 'praefect' + praefect['database_password'] = '<praefect_postgresql_password>' + + # Praefect Virtual Storage config + # Name of storage hash must match storage name in git_data_dirs on GitLab + # server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1') + praefect['virtual_storages'] = { + 'default' => { + 'gitaly-1' => { + 'address' => 'tcp://10.6.0.91:8075', + 'token' => '<praefect_internal_token>', + 'primary' => true + }, + 'gitaly-2' => { + 'address' => 'tcp://10.6.0.92:8075', + 'token' => '<praefect_internal_token>' + }, + 'gitaly-3' => { + 'address' => 'tcp://10.6.0.93:8075', + 'token' => '<praefect_internal_token>' + }, + } + } + + # Set the network addresses that the exporters will listen on for monitoring + node_exporter['listen_address'] = '0.0.0.0:9100' + praefect['prometheus_listen_addr'] = '0.0.0.0:9652' + + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + # + # END user configuration + ``` + + 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and + then replace the file of the same name on this server. If that file isn't on + this server, add the file from your Consul server to this server. + + 1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +### Configure Gitaly + +The [Gitaly](../gitaly/index.md) server nodes that make up the cluster have +requirements that are dependent on data, specifically the number of projects +and those projects' sizes. It's recommended that a Gitaly Cluster stores +no more than 5 TB of data on each node. Depending on your +repository storage requirements, you may require additional Gitaly Clusters. Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs @@ -1325,36 +1631,21 @@ adjusted to greater or lesser values depending on the scale of your environment's workload. If you're running the environment on a Cloud provider, refer to their documentation about how to configure IOPS correctly. -Be sure to note the following items: +Gitaly servers must not be exposed to the public internet, as Gitaly's network +traffic is unencrypted by default. The use of a firewall is highly recommended +to restrict access to the Gitaly server. Another option is to +[use TLS](#gitaly-cluster-tls-support). -- The GitLab Rails application shards repositories into - [repository storage paths](../repository_storage_paths.md). -- A Gitaly server can host one or more storage paths. -- A GitLab server can use one or more Gitaly server nodes. -- Gitaly addresses must be specified to be correctly resolvable for all Gitaly - clients. -- Gitaly servers must not be exposed to the public internet, as Gitaly's network - traffic is unencrypted by default. The use of a firewall is highly recommended - to restrict access to the Gitaly server. Another option is to - [use TLS](#gitaly-tls-support). +For configuring Gitaly you should note the following: -NOTE: -The token referred to throughout the Gitaly documentation is an arbitrary -password selected by the administrator. This token is unrelated to tokens -created for the GitLab API or other similar web API tokens. - -This section describes how to configure two Gitaly servers, with the following -IPs and domain names: - -- `10.6.0.91`: Gitaly 1 (`gitaly1.internal`) -- `10.6.0.92`: Gitaly 2 (`gitaly2.internal`) +- `git_data_dirs` should be configured to reflect the storage path for the specific Gitaly node +- `auth_token` should be the same as `praefect_internal_token` -Assumptions about your servers include having the secret token be `gitalysecret`, -and that your GitLab installation has three repository storages: +The following IPs will be used as an example: -- `default` on Gitaly 1 -- `storage1` on Gitaly 1 -- `storage2` on Gitaly 2 +- `10.6.0.91`: Gitaly 1 +- `10.6.0.92`: Gitaly 2 +- `10.6.0.93`: Gitaly 3 On each node: @@ -1364,21 +1655,9 @@ On each node: 1. Edit the Gitaly server node's `/etc/gitlab/gitlab.rb` file to configure storage paths, enable the network listener, and to configure the token: - <!-- - updates to following example must also be made at - https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab - --> - ```ruby # /etc/gitlab/gitlab.rb - # Gitaly and GitLab use two shared secrets for authentication, one to authenticate gRPC requests - # to Gitaly, and a second for authentication callbacks from GitLab-Shell to the GitLab internal API. - # The following two values must be the same as their respective values - # of the GitLab Rails application setup - gitaly['auth_token'] = 'gitalysecret' - gitlab_shell['secret_token'] = 'shellsecret' - # Avoid running unnecessary services on the Gitaly server postgresql['enable'] = false redis['enable'] = false @@ -1407,36 +1686,42 @@ On each node: # firewalls to restrict access to this address/port. # Comment out following line if you only want to support TLS connections gitaly['listen_addr'] = "0.0.0.0:8075" + + # Gitaly Auth Token + # Should be the same as praefect_internal_token + gitaly['auth_token'] = '<praefect_internal_token>' ``` 1. Append the following to `/etc/gitlab/gitlab.rb` for each respective server: - - On `gitaly1.internal`: + - On Gitaly node 1: ```ruby git_data_dirs({ - 'default' => { - 'path' => '/var/opt/gitlab/git-data' - }, - 'storage1' => { - 'path' => '/mnt/gitlab/git-data' - }, + "gitaly-1" => { + "path" => "/var/opt/gitlab/git-data" + } }) ``` - - On `gitaly2.internal`: + - On Gitaly node 2: ```ruby git_data_dirs({ - 'storage2' => { - 'path' => '/mnt/gitlab/git-data' - }, + "gitaly-2" => { + "path" => "/var/opt/gitlab/git-data" + } }) ``` - <!-- - updates to following example must also be made at - https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab - --> + - On Gitaly node 3: + + ```ruby + git_data_dirs({ + "gitaly-3" => { + "path" => "/var/opt/gitlab/git-data" + } + }) + ``` 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and then replace the file of the same name on this server. If that file isn't on @@ -1444,34 +1729,44 @@ On each node: 1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -### Gitaly TLS support +### Gitaly Cluster TLS support -Gitaly supports TLS encryption. To be able to communicate -with a Gitaly instance that listens for secure connections you will need to use `tls://` URL -scheme in the `gitaly_address` of the corresponding storage entry in the GitLab configuration. +Praefect supports TLS encryption. To communicate with a Praefect instance that listens +for secure connections, you must: -You will need to bring your own certificates as this isn't provided automatically. -The certificate, or its certificate authority, must be installed on all Gitaly -nodes (including the Gitaly node using the certificate) and on all client nodes -that communicate with it following the procedure described in -[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). +- Use a `tls://` URL scheme in the `gitaly_address` of the corresponding storage entry + in the GitLab configuration. +- Bring your own certificates because this isn't provided automatically. The certificate + corresponding to each Praefect server must be installed on that Praefect server. -NOTE: -The self-signed certificate must specify the address you use to access the -Gitaly server. If you are addressing the Gitaly server by a hostname, you can -either use the Common Name field for this, or add it as a Subject Alternative -Name. If you are addressing the Gitaly server by its IP address, you must add it -as a Subject Alternative Name to the certificate. -[gRPC does not support using an IP address as Common Name in a certificate](https://github.com/grpc/grpc/issues/2691). +Additionally the certificate, or its certificate authority, must be installed on all Gitaly servers +and on all Praefect clients that communicate with it following the procedure described in +[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates) (and repeated below). + +Note the following: -It's possible to configure Gitaly servers with both an unencrypted listening -address (`listen_addr`) and an encrypted listening address (`tls_listen_addr`) -at the same time. This allows you to do a gradual transition from unencrypted to -encrypted traffic, if necessary. +- The certificate must specify the address you use to access the Praefect server. If + addressing the Praefect server by: -To configure Gitaly with TLS: + - Hostname, you can either use the Common Name field for this, or add it as a Subject + Alternative Name. + - IP address, you must add it as a Subject Alternative Name to the certificate. -1. Create the `/etc/gitlab/ssl` directory and copy your key and certificate there: +- You can configure Praefect servers with both an unencrypted listening address + `listen_addr` and an encrypted listening address `tls_listen_addr` at the same time. + This allows you to do a gradual transition from unencrypted to encrypted traffic, if + necessary. + +- The Internal Load Balancer will also access to the certificates and need to be configured + to allow for TLS passthrough. + Refer to the load balancers documentation on how to configure this. + +To configure Praefect with TLS: + +1. Create certificates for Praefect servers. + +1. On the Praefect servers, create the `/etc/gitlab/ssl` directory and copy your key + and certificate there: ```shell sudo mkdir -p /etc/gitlab/ssl @@ -1480,27 +1775,34 @@ To configure Gitaly with TLS: sudo chmod 644 key.pem cert.pem ``` -1. Copy the cert to `/etc/gitlab/trusted-certs` so Gitaly will trust the cert when - calling into itself: +1. Edit `/etc/gitlab/gitlab.rb` and add: - ```shell - sudo cp /etc/gitlab/ssl/cert.pem /etc/gitlab/trusted-certs/ + ```ruby + praefect['tls_listen_addr'] = "0.0.0.0:3305" + praefect['certificate_path'] = "/etc/gitlab/ssl/cert.pem" + praefect['key_path'] = "/etc/gitlab/ssl/key.pem" ``` -1. Edit `/etc/gitlab/gitlab.rb` and add: +1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure). - <!-- - updates to following example must also be made at - https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab - --> +1. On the Praefect clients (including each Gitaly server), copy the certificates, + or their certificate authority, into `/etc/gitlab/trusted-certs`: - ```ruby - gitaly['tls_listen_addr'] = "0.0.0.0:9999" - gitaly['certificate_path'] = "/etc/gitlab/ssl/cert.pem" - gitaly['key_path'] = "/etc/gitlab/ssl/key.pem" + ```shell + sudo cp cert.pem /etc/gitlab/trusted-certs/ ``` -1. Delete `gitaly['listen_addr']` to allow only encrypted connections. +1. On the Praefect clients (except Gitaly servers), edit `git_data_dirs` in + `/etc/gitlab/gitlab.rb` as follows: + + ```ruby + git_data_dirs({ + "default" => { + "gitaly_address" => 'tls://LOAD_BALANCER_SERVER_ADDRESS:2305', + "gitaly_token" => 'PRAEFECT_EXTERNAL_TOKEN' + } + }) + ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -1587,12 +1889,15 @@ To configure the Sidekiq nodes, on each one: ### Gitaly ### ####################################### + # git_data_dirs get configured for the Praefect virtual storage + # Address is Internal Load Balancer for Praefect + # Token is praefect_external_token git_data_dirs({ - 'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, - 'storage1' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, - 'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' }, + "default" => { + "gitaly_address" => "tcp://10.6.0.40:2305", # internal load balancer IP + "gitaly_token" => '<praefect_external_token>' + } }) - gitlab_rails['gitaly_token'] = 'YOUR_TOKEN' ####################################### ### Postgres ### @@ -1624,7 +1929,7 @@ To configure the Sidekiq nodes, on each one: node_exporter['listen_address'] = '0.0.0.0:9100' # Rails Status for prometheus - gitlab_rails['monitoring_whitelist'] = ['10.6.0.121/32', '127.0.0.0/8'] + gitlab_rails['monitoring_whitelist'] = ['10.6.0.151/32', '127.0.0.0/8'] ############################# ### Object storage ### @@ -1637,13 +1942,13 @@ To configure the Sidekiq nodes, on each one: 'google_project' => '<gcp-project-name>', 'google_json_key_location' => '<path-to-gcp-service-account-key>' } - gitlab_rails['object_store']['objects']['artifacts']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['lfs']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['uploads']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-bucket-name>" + gitlab_rails['object_store']['objects']['artifacts']['bucket'] = "<gcp-artifacts-bucket-name>" + gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = "<gcp-external-diffs-bucket-name>" + gitlab_rails['object_store']['objects']['lfs']['bucket'] = "<gcp-lfs-bucket-name>" + gitlab_rails['object_store']['objects']['uploads']['bucket'] = "<gcp-uploads-bucket-name>" + gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-packages-bucket-name>" + gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>" + gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>" ``` 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace @@ -1671,6 +1976,8 @@ The following IPs will be used as an example: - `10.6.0.111`: GitLab application 1 - `10.6.0.112`: GitLab application 2 - `10.6.0.113`: GitLab application 3 +- `10.6.0.114`: GitLab application 4 +- `10.6.0.115`: GitLab application 5 On each node perform the following: @@ -1690,17 +1997,14 @@ On each node perform the following: ```ruby external_url 'https://gitlab.example.com' - # Gitaly and GitLab use two shared secrets for authentication, one to authenticate gRPC requests - # to Gitaly, and a second for authentication callbacks from GitLab-Shell to the GitLab internal API. - # The following two values must be the same as their respective values - # of the Gitaly setup - gitlab_rails['gitaly_token'] = 'gitalysecret' - gitlab_shell['secret_token'] = 'shellsecret' - + # git_data_dirs get configured for the Praefect virtual storage + # Address is Internal Load Balancer for Praefect + # Token is praefect_external_token git_data_dirs({ - 'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, - 'storage1' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, - 'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' }, + "default" => { + "gitaly_address" => "tcp://10.6.0.40:2305", # internal load balancer IP + "gitaly_token" => '<praefect_external_token>' + } }) ## Disable components that will not be on the GitLab application server @@ -1755,8 +2059,8 @@ On each node perform the following: # Add the monitoring node's IP address to the monitoring whitelist and allow it to # scrape the NGINX metrics - gitlab_rails['monitoring_whitelist'] = ['10.6.0.121/32', '127.0.0.0/8'] - nginx['status']['options']['allow'] = ['10.6.0.121/32', '127.0.0.0/8'] + gitlab_rails['monitoring_whitelist'] = ['10.6.0.151/32', '127.0.0.0/8'] + nginx['status']['options']['allow'] = ['10.6.0.151/32', '127.0.0.0/8'] ############################# ### Object storage ### @@ -1769,24 +2073,25 @@ On each node perform the following: 'google_project' => '<gcp-project-name>', 'google_json_key_location' => '<path-to-gcp-service-account-key>' } - gitlab_rails['object_store']['objects']['artifacts']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['lfs']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['uploads']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-bucket-name>" + gitlab_rails['object_store']['objects']['artifacts']['bucket'] = "<gcp-artifacts-bucket-name>" + gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = "<gcp-external-diffs-bucket-name>" + gitlab_rails['object_store']['objects']['lfs']['bucket'] = "<gcp-lfs-bucket-name>" + gitlab_rails['object_store']['objects']['uploads']['bucket'] = "<gcp-uploads-bucket-name>" + gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-packages-bucket-name>" + gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>" + gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>" ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -1. If you're using [Gitaly with TLS support](#gitaly-tls-support), make sure the +1. If you're using [Gitaly with TLS support](#gitaly-cluster-tls-support), make sure the `git_data_dirs` entry is configured with `tls` instead of `tcp`: ```ruby git_data_dirs({ - 'default' => { 'gitaly_address' => 'tls://gitaly1.internal:9999' }, - 'storage1' => { 'gitaly_address' => 'tls://gitaly1.internal:9999' }, - 'storage2' => { 'gitaly_address' => 'tls://gitaly2.internal:9999' }, + "default" => { + "gitaly_address" => "tls://10.6.0.40:2305", # internal load balancer IP + "gitaly_token" => '<praefect_external_token>' + } }) ``` @@ -1891,7 +2196,7 @@ running [Prometheus](../monitoring/prometheus/index.md) and The following IP will be used as an example: -- `10.6.0.121`: Prometheus +- `10.6.0.151`: Prometheus To configure the Monitoring node: @@ -2002,20 +2307,9 @@ on what features you intend to use: | [Terraform state files](../terraform_state.md#using-object-storage) | Yes | Using separate buckets for each data type is the recommended approach for GitLab. - -A limitation of our configuration is that each use of object storage is separately configured. -[We have an issue for improving this](https://gitlab.com/gitlab-org/gitlab/-/issues/23345) -and easily using one bucket with separate folders is one improvement that this might bring. - -There is at least one specific issue with using the same bucket: -when GitLab is deployed with the Helm chart restore from backup -[will not properly function](https://docs.gitlab.com/charts/advanced/external-object-storage/#lfs-artifacts-uploads-packages-external-diffs-pseudonymizer) -unless separate buckets are used. - -One risk of using a single bucket would be if your organization decided to -migrate GitLab to the Helm deployment in the future. GitLab would run, but the situation with -backups might not be realized until the organization had a critical requirement for the backups to -work. +This ensures there are no collisions across the various types of data GitLab stores. +There are plans to [enable the use of a single bucket](https://gitlab.com/gitlab-org/gitlab/-/issues/292958) +in the future. <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -2023,7 +2317,7 @@ work. </a> </div> -## Configure Advanced Search **(PREMIUM SELF)** +## Configure Advanced Search You can leverage Elasticsearch and [enable Advanced Search](../../integration/elasticsearch.md) for faster, more advanced code search across your entire GitLab instance. @@ -2058,13 +2352,3 @@ as soon as possible. Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> </a> </div> - -## Troubleshooting - -See the [troubleshooting documentation](troubleshooting.md). - -<div align="right"> - <a type="button" class="btn btn-default" href="#setup-components"> - Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> - </a> -</div> diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index 9ad6054104a..fc0afbff194 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -668,13 +668,13 @@ On each node perform the following: 'google_project' => '<gcp-project-name>', 'google_json_key_location' => '<path-to-gcp-service-account-key>' } - gitlab_rails['object_store']['objects']['artifacts']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['lfs']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['uploads']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-bucket-name>" + gitlab_rails['object_store']['objects']['artifacts']['bucket'] = "<gcp-artifacts-bucket-name>" + gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = "<gcp-external-diffs-bucket-name>" + gitlab_rails['object_store']['objects']['lfs']['bucket'] = "<gcp-lfs-bucket-name>" + gitlab_rails['object_store']['objects']['uploads']['bucket'] = "<gcp-uploads-bucket-name>" + gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-packages-bucket-name>" + gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>" + gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>" ## Uncomment and edit the following options if you have set up NFS ## @@ -920,18 +920,9 @@ on what features you intend to use: | [Terraform state files](../terraform_state.md#using-object-storage) | Yes | Using separate buckets for each data type is the recommended approach for GitLab. - -A limitation of our configuration is that each use of object storage is -separately configured. We have an [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/23345) -for improving this, which would allow for one bucket with separate folders. - -Using a single bucket when GitLab is deployed with the Helm chart causes -restoring from a backup to -[not function properly](https://docs.gitlab.com/charts/advanced/external-object-storage/#lfs-artifacts-uploads-packages-external-diffs-pseudonymizer). -Although you may not be using a Helm deployment right now, if you migrate -GitLab to a Helm deployment later, GitLab would still work, but you may not -realize backups aren't working correctly until a critical requirement for -functioning backups is encountered. +This ensures there are no collisions across the various types of data GitLab stores. +There are plans to [enable the use of a single bucket](https://gitlab.com/gitlab-org/gitlab/-/issues/292958) +in the future. <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -975,13 +966,3 @@ as soon as possible. Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> </a> </div> - -## Troubleshooting - -See the [troubleshooting documentation](troubleshooting.md). - -<div align="right"> - <a type="button" class="btn btn-default" href="#setup-components"> - Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> - </a> -</div> diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index 175c4318d78..7f7777455c2 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -7,91 +7,121 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Reference architecture: up to 3,000 users **(PREMIUM SELF)** -This page describes GitLab reference architecture for up to 3,000 users. -It is designed to help your organization achieve a -highly-available GitLab deployment. If you do not have the expertise or need to -maintain a highly-available environment, you can have a simpler and less -costly-to-operate environment by using the -[2,000-user reference architecture](2k_users.md). -If you have fewer than 3,000 users and still want a highly-available GitLab deployment, -you should still use this reference architecture but scale down the node sizes. +This GitLab reference architecture can help you deploy GitLab to up to 3,000 +users, and then maintain uptime and access for those users. You can also use +this architecture to provide improved GitLab uptime and availability for fewer +than 3,000 users. For fewer users, reduce the stated node sizes as needed. + +If maintining a high level of uptime for your GitLab environment isn't a +requirement, or if you don't have the expertise to maintain this sort of +environment, we recommend using the [2,000-user reference architecture](2k_users.md) +for your GitLab installation. For a full list of reference architectures, see [Available reference architectures](index.md#available-reference-architectures). > - **Supported users (approximate):** 3,000 -> - **High Availability:** Yes +> - **High Availability:** Yes, although [Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution > - **Test requests per second (RPS) rates:** API: 60 RPS, Web: 6 RPS, Git (Pull): 6 RPS, Git (Push): 1 RPS | Service | Nodes | Configuration | GCP | AWS | Azure | |--------------------------------------------|-------------|-----------------------|----------------|-------------|---------| -| External load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | -| Redis | 3 | 2 vCPU, 7.5 GB memory | n1-standard-2 | `m5.large` | D2s v3 | -| Consul + Sentinel | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | -| PostgreSQL | 3 | 2 vCPU, 7.5 GB memory | n1-standard-2 | `m5.large` | D2s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | -| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | -| Gitaly | 2 (minimum) | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | -| Sidekiq | 4 | 2 vCPU, 7.5 GB memory | n1-standard-2 | `m5.large` | D2s v3 | -| GitLab Rails | 3 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | `c5.2xlarge` | F8s v2 | -| Monitoring node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| External load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Redis | 3 | 2 vCPU, 7.5 GB memory | n1-standard-2 | m5.large | D2s v3 | +| Consul + Sentinel | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| PostgreSQL | 3 | 2 vCPU, 7.5 GB memory | n1-standard-2 | m5.large | D2s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Gitaly | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| Praefect | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Praefect PostgreSQL | 1+* | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Sidekiq | 4 | 2 vCPU, 7.5 GB memory | n1-standard-2 | m5.large | D2s v3 | +| GitLab Rails | 3 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | +| Monitoring node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | | Object storage | n/a | n/a | n/a | n/a | n/a | -| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | - -```mermaid -stateDiagram-v2 - [*] --> LoadBalancer - LoadBalancer --> ApplicationServer - - ApplicationServer --> BackgroundJobs - ApplicationServer --> Gitaly - ApplicationServer --> Redis - ApplicationServer --> PgBouncer - PgBouncer --> Database - ApplicationServer --> ObjectStorage - BackgroundJobs --> ObjectStorage - - ApplicationMonitoring -->ApplicationServer - ApplicationMonitoring -->Redis - ApplicationMonitoring -->PgBouncer - ApplicationMonitoring -->Database - ApplicationMonitoring -->BackgroundJobs - - state Database { - "PG_Primary_Node" - "PG_Secondary_Node_1..2" - } - state Redis { - "R_Primary_Node" - "R_Replica_Node_1..2" - "R_Consul/Sentinel_1..3" - } - - state Gitaly { - "Gitaly_1..2" - } - - state BackgroundJobs { - "Sidekiq_1..4" - } - - state ApplicationServer { - "GitLab_Rails_1..3" - } - - state LoadBalancer { - "LoadBalancer_1" - } - - state ApplicationMonitoring { - "Prometheus" - "Grafana" - } - - state PgBouncer { - "Internal_Load_Balancer" - "PgBouncer_1..3" - } +| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | + +```plantuml +@startuml 3k +card "**External Load Balancer**" as elb #6a9be7 +card "**Internal Load Balancer**" as ilb #9370DB + +together { + collections "**GitLab Rails** x3" as gitlab #32CD32 + collections "**Sidekiq** x4" as sidekiq #ff8dd1 +} + +together { + card "**Prometheus + Grafana**" as monitor #7FFFD4 + collections "**Consul** x3" as consul #e76a9b +} + +card "Gitaly Cluster" as gitaly_cluster { + collections "**Praefect** x3" as praefect #FF8C00 + collections "**Gitaly** x3" as gitaly #FF8C00 + card "**Praefect PostgreSQL***\n//Non fault-tolerant//" as praefect_postgres #FF8C00 + + praefect -[#FF8C00]-> gitaly + praefect -[#FF8C00]> praefect_postgres +} + +card "Database" as database { + collections "**PGBouncer** x3" as pgbouncer #4EA7FF + card "**PostgreSQL** (Primary)" as postgres_primary #4EA7FF + collections "**PostgreSQL** (Secondary) x2" as postgres_secondary #4EA7FF + + pgbouncer -[#4EA7FF]-> postgres_primary + postgres_primary .[#4EA7FF]> postgres_secondary +} + +card "redis" as redis { + collections "**Redis Persistent** x3" as redis_persistent #FF6347 + collections "**Redis Cache** x3" as redis_cache #FF6347 + collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347 + collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347 + + redis_persistent <.[#FF6347]- redis_persistent_sentinel + redis_cache <.[#FF6347]- redis_cache_sentinel +} + +cloud "**Object Storage**" as object_storage #white + +elb -[#6a9be7]-> gitlab +elb -[#6a9be7]--> monitor + +gitlab -[#32CD32]> sidekiq +gitlab -[#32CD32]--> ilb +gitlab -[#32CD32]-> object_storage +gitlab -[#32CD32]---> redis +gitlab -[hidden]-> monitor +gitlab -[hidden]-> consul + +sidekiq -[#ff8dd1]--> ilb +sidekiq -[#ff8dd1]-> object_storage +sidekiq -[#ff8dd1]---> redis +sidekiq -[hidden]-> monitor +sidekiq -[hidden]-> consul + +ilb -[#9370DB]-> gitaly_cluster +ilb -[#9370DB]-> database + +consul .[#e76a9b]u-> gitlab +consul .[#e76a9b]u-> sidekiq +consul .[#e76a9b]> monitor +consul .[#e76a9b]-> database +consul .[#e76a9b]-> gitaly_cluster +consul .[#e76a9b,norank]--> redis + +monitor .[#7FFFD4]u-> gitlab +monitor .[#7FFFD4]u-> sidekiq +monitor .[#7FFFD4]> consul +monitor .[#7FFFD4]-> database +monitor .[#7FFFD4]-> gitaly_cluster +monitor .[#7FFFD4,norank]--> redis +monitor .[#7FFFD4]> ilb +monitor .[#7FFFD4,norank]u--> elb + +@enduml ``` The Google Cloud Platform (GCP) architectures were built and tested using the @@ -106,19 +136,30 @@ uploads, or artifacts), using an [object storage service](#configure-the-object- is recommended instead of using NFS. Using an object storage service also doesn't require you to provision and maintain a node. +[Praefect requires its own database server](../gitaly/praefect.md#postgresql), +and a third-party PostgreSQL database solution is required to achieve full +high availability. Although we hope to offer a built-in solution for these +restrictions in the future, you can set up a non-HA PostgreSQL server by using +Omnibus GitLab (which the previous specifications reflect). Refer to the +following issues for more information: + +- [`omnibus-gitlab#5919`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919) +- [`gitaly#3398`](https://gitlab.com/gitlab-org/gitaly/-/issues/3398) + ## Setup components To set up GitLab and its components to accommodate up to 3,000 users: -1. [Configure the external load balancing node](#configure-the-external-load-balancer) +1. [Configure the external load balancer](#configure-the-external-load-balancer) to handle the load balancing of the GitLab application services nodes. +1. [Configure the internal load balancer](#configure-the-internal-load-balancer). + to handle the load balancing of GitLab application internal connections. 1. [Configure Redis](#configure-redis). 1. [Configure Consul and Sentinel](#configure-consul-and-sentinel). 1. [Configure PostgreSQL](#configure-postgresql), the database for GitLab. 1. [Configure PgBouncer](#configure-pgbouncer). -1. [Configure the internal load balancing node](#configure-the-internal-load-balancer). -1. [Configure Gitaly](#configure-gitaly), - which provides access to the Git repositories. +1. [Configure Gitaly Cluster](#configure-gitaly-cluster), + provides access to the Git repositories. 1. [Configure Sidekiq](#configure-sidekiq). 1. [Configure the main GitLab Rails application](#configure-gitlab-rails) to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend @@ -155,6 +196,11 @@ The following list includes descriptions of each server and its assigned IP: - `10.6.0.20`: Internal Load Balancer - `10.6.0.51`: Gitaly 1 - `10.6.0.52`: Gitaly 2 +- `10.6.0.93`: Gitaly 3 +- `10.6.0.131`: Praefect 1 +- `10.6.0.132`: Praefect 2 +- `10.6.0.133`: Praefect 3 +- `10.6.0.141`: Praefect PostgreSQL 1 (non HA) - `10.6.0.71`: Sidekiq 1 - `10.6.0.72`: Sidekiq 2 - `10.6.0.73`: Sidekiq 3 @@ -285,6 +331,71 @@ Configure DNS for an alternate SSH hostname such as `altssh.gitlab.example.com`. </a> </div> +## Configure the internal load balancer + +The Internal Load Balancer is used to balance any internal connections the GitLab environment requires +such as connections to [PgBouncer](#configure-pgbouncer) and [Praefect](#configure-praefect) (Gitaly Cluster). + +Note that it's a separate node from the External Load Balancer and shouldn't have any access externally. + +The following IP will be used as an example: + +- `10.6.0.40`: Internal Load Balancer + +Here's how you could do it with [HAProxy](https://www.haproxy.org/): + +```plaintext +global + log /dev/log local0 + log localhost local1 notice + log stdout format raw local0 + +defaults + log global + default-server inter 10s fall 3 rise 2 + balance leastconn + +frontend internal-pgbouncer-tcp-in + bind *:6432 + mode tcp + option tcplog + + default_backend pgbouncer + +frontend internal-praefect-tcp-in + bind *:2305 + mode tcp + option tcplog + option clitcpka + + default_backend praefect + +backend pgbouncer + mode tcp + option tcp-check + + server pgbouncer1 10.6.0.21:6432 check + server pgbouncer2 10.6.0.22:6432 check + server pgbouncer3 10.6.0.23:6432 check + +backend praefect + mode tcp + option tcp-check + option srvtcpka + + server praefect1 10.6.0.131:2305 check + server praefect2 10.6.0.132:2305 check + server praefect3 10.6.0.133:2305 check +``` + +Refer to your preferred Load Balancer's documentation for further guidance. + +<div align="right"> + <a type="button" class="btn btn-default" href="#setup-components"> + Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> + </a> +</div> + ## Configure Redis Using [Redis](https://redis.io/) in scalable environment is possible using a **Primary** x **Replica** @@ -925,45 +1036,96 @@ The following IPs will be used as an example: </a> </div> -### Configure the internal load balancer +## Configure Gitaly Cluster -If you're running more than one PgBouncer node as recommended, then at this time you'll need to set -up a TCP internal load balancer to serve each correctly. +[Gitaly Cluster](../gitaly/praefect.md) is a GitLab provided and recommended fault tolerant solution for storing Git repositories. +In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down. -The following IP will be used as an example: +The recommended cluster setup includes the following components: -- `10.6.0.20`: Internal Load Balancer +- 3 Gitaly nodes: Replicated storage of Git repositories. +- 3 Praefect nodes: Router and transaction manager for Gitaly Cluster. +- 1 Praefect PostgreSQL node: Database server for Praefect. A third-party solution + is required for Praefect database connections to be made highly available. +- 1 load balancer: A load balancer is required for Praefect. The + [internal load balancer](#configure-the-internal-load-balancer) will be used. -Here's how you could do it with [HAProxy](https://www.haproxy.org/): +This section will detail how to configure the recommended standard setup in order. +For more advanced setups refer to the [standalone Gitaly Cluster documentation](../gitaly/praefect.md). -```plaintext -global - log /dev/log local0 - log localhost local1 notice - log stdout format raw local0 +### Configure Praefect PostgreSQL -defaults - log global - default-server inter 10s fall 3 rise 2 - balance leastconn +Praefect, the routing and transaction manager for Gitaly Cluster, requires its own database server to store data on Gitaly Cluster status. -frontend internal-pgbouncer-tcp-in - bind *:6432 - mode tcp - option tcplog +If you want to have a highly available setup, Praefect requires a third-party PostgreSQL database. +A built-in solution is being [worked on](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919). - default_backend pgbouncer +#### Praefect non-HA PostgreSQL standalone using Omnibus GitLab -backend pgbouncer - mode tcp - option tcp-check +The following IPs will be used as an example: - server pgbouncer1 10.6.0.21:6432 check - server pgbouncer2 10.6.0.22:6432 check - server pgbouncer3 10.6.0.23:6432 check -``` +- `10.6.0.141`: Praefect PostgreSQL -Refer to your preferred Load Balancer's documentation for further guidance. +First, make sure to [install](https://about.gitlab.com/install/) +the Linux GitLab package in the Praefect PostgreSQL node. Following the steps, +install the necessary dependencies from step 1, and add the +GitLab package repository from step 2. When installing GitLab +in the second step, do not supply the `EXTERNAL_URL` value. + +1. SSH in to the Praefect PostgreSQL node. +1. Create a strong password to be used for the Praefect PostgreSQL user. Take note of this password as `<praefect_postgresql_password>`. +1. Generate the password hash for the Praefect PostgreSQL username/password pair. This assumes you will use the default + username of `praefect` (recommended). The command will request the password `<praefect_postgresql_password>` + and confirmation. Use the value that is output by this command in the next + step as the value of `<praefect_postgresql_password_hash>`: + + ```shell + sudo gitlab-ctl pg-password-md5 praefect + ``` + +1. Edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: + + ```ruby + # Disable all components except PostgreSQL and Consul + roles ['postgres_role'] + repmgr['enable'] = false + patroni['enable'] = false + + # PostgreSQL configuration + postgresql['listen_address'] = '0.0.0.0' + postgresql['max_connections'] = 200 + + gitlab_rails['auto_migrate'] = false + + # Configure the Consul agent + consul['enable'] = true + ## Enable service discovery for Prometheus + consul['monitoring_service_discovery'] = true + + # START user configuration + # Please set the real values as explained in Required Information section + # + # Replace PRAEFECT_POSTGRESQL_PASSWORD_HASH with a generated md5 value + postgresql['sql_user_password'] = "<praefect_postgresql_password_hash>" + + # Replace XXX.XXX.XXX.XXX/YY with Network Address + postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/24) + + # Set the network addresses that the exporters will listen on for monitoring + node_exporter['listen_address'] = '0.0.0.0:9100' + postgres_exporter['listen_address'] = '0.0.0.0:9187' + + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + # + # END user configuration + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Follow the [post configuration](#praefect-postgresql-post-configuration). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -971,19 +1133,186 @@ Refer to your preferred Load Balancer's documentation for further guidance. </a> </div> -## Configure Gitaly +#### Praefect HA PostgreSQL third-party solution -NOTE: -[Gitaly Cluster](../gitaly/praefect.md) support -for the Reference Architectures is being -worked on as a [collaborative effort](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/1) between the Quality Engineering and Gitaly teams. When this component has been verified -some Architecture specs will likely change as a result to support the new -and improved designed. +[As noted](#configure-praefect-postgresql), a third-party PostgreSQL solution for +Praefect's database is recommended if aiming for full High Availability. + +There are many third-party solutions for PostgreSQL HA. The solution selected must have the following to work with Praefect: + +- A static IP for all connections that doesn't change on failover. +- [`LISTEN`](https://www.postgresql.org/docs/12/sql-listen.html) SQL functionality must be supported. + +Examples of the above could include [Google's Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) or [Amazon RDS](https://aws.amazon.com/rds/). + +Once the database is set up, follow the [post configuration](#praefect-postgresql-post-configuration). + +#### Praefect PostgreSQL post-configuration + +After the Praefect PostgreSQL server has been set up, you'll then need to configure the user and database for Praefect to use. + +We recommend the user be named `praefect` and the database `praefect_production`, and these can be configured as standard in PostgreSQL. +The password for the user is the same as the one you configured earlier as `<praefect_postgresql_password>`. + +This is how this would work with a Omnibus GitLab PostgreSQL setup: + +1. SSH in to the Praefect PostgreSQL node. +1. Connect to the PostgreSQL server with administrative access. + The `gitlab-psql` user should be used here for this as it's added by default in Omnibus. + The database `template1` is used because it is created by default on all PostgreSQL servers. + + ```shell + /opt/gitlab/embedded/bin/psql -U gitlab-psql -d template1 -h POSTGRESQL_SERVER_ADDRESS + ``` + +1. Create the new user `praefect`, replacing `<praefect_postgresql_password>`: + + ```shell + CREATE ROLE praefect WITH LOGIN CREATEDB PASSWORD <praefect_postgresql_password>; + ``` + +1. Reconnect to the PostgreSQL server, this time as the `praefect` user: + + ```shell + /opt/gitlab/embedded/bin/psql -U praefect -d template1 -h POSTGRESQL_SERVER_ADDRESS + ``` + +1. Create a new database `praefect_production`: + + ```shell + CREATE DATABASE praefect_production WITH ENCODING=UTF8; + ``` + +<div align="right"> + <a type="button" class="btn btn-default" href="#setup-components"> + Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> + </a> +</div> + +### Configure Praefect + +Praefect is the router and transaction manager for Gitaly Cluster and all connections to Gitaly go through +it. This section details how to configure it. + +Praefect requires several secret tokens to secure communications across the Cluster: + +- `<praefect_external_token>`: Used for repositories hosted on your Gitaly cluster and can only be accessed by Gitaly clients that carry this token. +- `<praefect_internal_token>`: Used for replication traffic inside your Gitaly cluster. This is distinct from `praefect_external_token` because Gitaly clients must not be able to access internal nodes of the Praefect cluster directly; that could lead to data loss. +- `<praefect_postgresql_password>`: The Praefect PostgreSQL password defined in the previous section is also required as part of this setup. + +Gitaly Cluster nodes are configured in Praefect via a `virtual storage`. Each storage contains +the details of each Gitaly node that makes up the cluster. Each storage is also given a name +and this name is used in several areas of the config. In this guide, the name of the storage will be +`default`. Also, this guide is geared towards new installs, if upgrading an existing environment +to use Gitaly Cluster, you may need to use a different name. +Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more info. + +The following IPs will be used as an example: + +- `10.6.0.131`: Praefect 1 +- `10.6.0.132`: Praefect 2 +- `10.6.0.133`: Praefect 3 + +To configure the Praefect nodes, on each one: + +1. SSH in to the Praefect server. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to follow _only_ installation steps 1 and 2 + on the page. +1. Edit the `/etc/gitlab/gitlab.rb` file to configure Praefect: + + ```ruby + # Avoid running unnecessary services on the Gitaly server + postgresql['enable'] = false + redis['enable'] = false + nginx['enable'] = false + puma['enable'] = false + unicorn['enable'] = false + sidekiq['enable'] = false + gitlab_workhorse['enable'] = false + grafana['enable'] = false + + # If you run a separate monitoring node you can disable these services + alertmanager['enable'] = false + prometheus['enable'] = false + + # Praefect Configuration + praefect['enable'] = true + praefect['listen_addr'] = '0.0.0.0:2305' + + gitlab_rails['rake_cache_clear'] = false + gitlab_rails['auto_migrate'] = false + + # Configure the Consul agent + consul['enable'] = true + ## Enable service discovery for Prometheus + consul['monitoring_service_discovery'] = true + + # START user configuration + # Please set the real values as explained in Required Information section + # + + # Praefect External Token + # This is needed by clients outside the cluster (like GitLab Shell) to communicate with the Praefect cluster + praefect['auth_token'] = '<praefect_external_token>' + + # Praefect Database Settings + praefect['database_host'] = '10.6.0.141' + praefect['database_port'] = 5432 + # `no_proxy` settings must always be a direct connection for caching + praefect['database_host_no_proxy'] = '10.6.0.141' + praefect['database_port_no_proxy'] = 5432 + praefect['database_dbname'] = 'praefect_production' + praefect['database_user'] = 'praefect' + praefect['database_password'] = '<praefect_postgresql_password>' + + # Praefect Virtual Storage config + # Name of storage hash must match storage name in git_data_dirs on GitLab + # server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1') + praefect['virtual_storages'] = { + 'default' => { + 'gitaly-1' => { + 'address' => 'tcp://10.6.0.91:8075', + 'token' => '<praefect_internal_token>', + 'primary' => true + }, + 'gitaly-2' => { + 'address' => 'tcp://10.6.0.92:8075', + 'token' => '<praefect_internal_token>' + }, + 'gitaly-3' => { + 'address' => 'tcp://10.6.0.93:8075', + 'token' => '<praefect_internal_token>' + }, + } + } + + # Set the network addresses that the exporters will listen on for monitoring + node_exporter['listen_address'] = '0.0.0.0:9100' + praefect['prometheus_listen_addr'] = '0.0.0.0:9652' + + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + # + # END user configuration + ``` + + 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and + then replace the file of the same name on this server. If that file isn't on + this server, add the file from your Consul server to this server. -[Gitaly](../gitaly/index.md) server node requirements are dependent on data, -specifically the number of projects and those projects' sizes. It's recommended -that a Gitaly server node stores no more than 5 TB of data. Depending on your -repository storage requirements, you may require additional Gitaly server nodes. + 1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +### Configure Gitaly + +The [Gitaly](../gitaly/index.md) server nodes that make up the cluster have +requirements that are dependent on data, specifically the number of projects +and those projects' sizes. It's recommended that a Gitaly Cluster stores +no more than 5 TB of data on each node. Depending on your +repository storage requirements, you may require additional Gitaly Clusters. Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs @@ -994,36 +1323,21 @@ adjusted to greater or lesser values depending on the scale of your environment's workload. If you're running the environment on a Cloud provider, refer to their documentation about how to configure IOPS correctly. -Be sure to note the following items: - -- The GitLab Rails application shards repositories into - [repository storage paths](../repository_storage_paths.md). -- A Gitaly server can host one or more storage paths. -- A GitLab server can use one or more Gitaly server nodes. -- Gitaly addresses must be specified to be correctly resolvable for all Gitaly - clients. -- Gitaly servers must not be exposed to the public internet, as Gitaly's network - traffic is unencrypted by default. The use of a firewall is highly recommended - to restrict access to the Gitaly server. Another option is to - [use TLS](#gitaly-tls-support). - -NOTE: -The token referred to throughout the Gitaly documentation is an arbitrary -password selected by the administrator. This token is unrelated to tokens -created for the GitLab API or other similar web API tokens. +Gitaly servers must not be exposed to the public internet, as Gitaly's network +traffic is unencrypted by default. The use of a firewall is highly recommended +to restrict access to the Gitaly server. Another option is to +[use TLS](#gitaly-cluster-tls-support). -This section describes how to configure two Gitaly servers, with the following -IPs and domain names: +For configuring Gitaly you should note the following: -- `10.6.0.51`: Gitaly 1 (`gitaly1.internal`) -- `10.6.0.52`: Gitaly 2 (`gitaly2.internal`) +- `git_data_dirs` should be configured to reflect the storage path for the specific Gitaly node +- `auth_token` should be the same as `praefect_internal_token` -Assumptions about your servers include having the secret token be `gitalysecret`, -and that your GitLab installation has three repository storages: +The following IPs will be used as an example: -- `default` on Gitaly 1 -- `storage1` on Gitaly 1 -- `storage2` on Gitaly 2 +- `10.6.0.91`: Gitaly 1 +- `10.6.0.92`: Gitaly 2 +- `10.6.0.93`: Gitaly 3 On each node: @@ -1033,21 +1347,9 @@ On each node: 1. Edit the Gitaly server node's `/etc/gitlab/gitlab.rb` file to configure storage paths, enable the network listener, and to configure the token: - <!-- - updates to following example must also be made at - https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab - --> - ```ruby # /etc/gitlab/gitlab.rb - # Gitaly and GitLab use two shared secrets for authentication, one to authenticate gRPC requests - # to Gitaly, and a second for authentication callbacks from GitLab-Shell to the GitLab internal API. - # The following two values must be the same as their respective values - # of the GitLab Rails application setup - gitaly['auth_token'] = 'gitalysecret' - gitlab_shell['secret_token'] = 'shellsecret' - # Avoid running unnecessary services on the Gitaly server postgresql['enable'] = false redis['enable'] = false @@ -1057,7 +1359,6 @@ On each node: sidekiq['enable'] = false gitlab_workhorse['enable'] = false grafana['enable'] = false - gitlab_exporter['enable'] = false # If you run a separate monitoring node you can disable these services alertmanager['enable'] = false @@ -1078,101 +1379,86 @@ On each node: # Comment out following line if you only want to support TLS connections gitaly['listen_addr'] = "0.0.0.0:8075" - ## Enable service discovery for Prometheus - consul['enable'] = true - consul['monitoring_service_discovery'] = true - - # Set the network addresses that the exporters will listen on for monitoring - gitaly['prometheus_listen_addr'] = "0.0.0.0:9236" - node_exporter['listen_address'] = '0.0.0.0:9100' - gitlab_rails['prometheus_address'] = '10.6.0.81:9090' - - ## The IPs of the Consul server nodes - ## You can also use FQDNs and intermix them with IPs - consul['configuration'] = { - retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), - } + # Gitaly Auth Token + # Should be the same as praefect_internal_token + gitaly['auth_token'] = '<praefect_internal_token>' ``` 1. Append the following to `/etc/gitlab/gitlab.rb` for each respective server: - - On `gitaly1.internal`: + - On Gitaly node 1: ```ruby git_data_dirs({ - 'default' => { - 'path' => '/var/opt/gitlab/git-data' - }, - 'storage1' => { - 'path' => '/mnt/gitlab/git-data' - }, + "gitaly-1" => { + "path" => "/var/opt/gitlab/git-data" + } }) ``` - - On `gitaly2.internal`: + - On Gitaly node 2: ```ruby git_data_dirs({ - 'storage2' => { - 'path' => '/mnt/gitlab/git-data' - }, + "gitaly-2" => { + "path" => "/var/opt/gitlab/git-data" + } + }) + ``` + + - On Gitaly node 3: + + ```ruby + git_data_dirs({ + "gitaly-3" => { + "path" => "/var/opt/gitlab/git-data" + } }) ``` - <!-- - updates to following example must also be made at - https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab - --> +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and + then replace the file of the same name on this server. If that file isn't on + this server, add the file from your Consul server to this server. 1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -1. Confirm that Gitaly can perform callbacks to the internal API: - ```shell - sudo /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml - ``` +### Gitaly Cluster TLS support -1. Verify the GitLab services are running: +Praefect supports TLS encryption. To communicate with a Praefect instance that listens +for secure connections, you must: - ```shell - sudo gitlab-ctl status - ``` +- Use a `tls://` URL scheme in the `gitaly_address` of the corresponding storage entry + in the GitLab configuration. +- Bring your own certificates because this isn't provided automatically. The certificate + corresponding to each Praefect server must be installed on that Praefect server. - The output should be similar to the following: +Additionally the certificate, or its certificate authority, must be installed on all Gitaly servers +and on all Praefect clients that communicate with it following the procedure described in +[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates) (and repeated below). - ```plaintext - run: consul: (pid 30339) 77006s; run: log: (pid 29878) 77020s - run: gitaly: (pid 30351) 77005s; run: log: (pid 29660) 77040s - run: logrotate: (pid 7760) 3213s; run: log: (pid 29782) 77032s - run: node-exporter: (pid 30378) 77004s; run: log: (pid 29812) 77026s - ``` +Note the following: -### Gitaly TLS support +- The certificate must specify the address you use to access the Praefect server. If + addressing the Praefect server by: -Gitaly supports TLS encryption. To be able to communicate -with a Gitaly instance that listens for secure connections you will need to use `tls://` URL -scheme in the `gitaly_address` of the corresponding storage entry in the GitLab configuration. + - Hostname, you can either use the Common Name field for this, or add it as a Subject + Alternative Name. + - IP address, you must add it as a Subject Alternative Name to the certificate. -You will need to bring your own certificates as this isn't provided automatically. -The certificate, or its certificate authority, must be installed on all Gitaly -nodes (including the Gitaly node using the certificate) and on all client nodes -that communicate with it following the procedure described in -[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). +- You can configure Praefect servers with both an unencrypted listening address + `listen_addr` and an encrypted listening address `tls_listen_addr` at the same time. + This allows you to do a gradual transition from unencrypted to encrypted traffic, if + necessary. -NOTE: -The self-signed certificate must specify the address you use to access the -Gitaly server. If you are addressing the Gitaly server by a hostname, you can -either use the Common Name field for this, or add it as a Subject Alternative -Name. If you are addressing the Gitaly server by its IP address, you must add it -as a Subject Alternative Name to the certificate. -[gRPC does not support using an IP address as Common Name in a certificate](https://github.com/grpc/grpc/issues/2691). +- The Internal Load Balancer will also access to the certificates and need to be configured + to allow for TLS passthrough. + Refer to the load balancers documentation on how to configure this. -It's possible to configure Gitaly servers with both an unencrypted listening -address (`listen_addr`) and an encrypted listening address (`tls_listen_addr`) -at the same time. This allows you to do a gradual transition from unencrypted to -encrypted traffic, if necessary. +To configure Praefect with TLS: -To configure Gitaly with TLS: +1. Create certificates for Praefect servers. -1. Create the `/etc/gitlab/ssl` directory and copy your key and certificate there: +1. On the Praefect servers, create the `/etc/gitlab/ssl` directory and copy your key + and certificate there: ```shell sudo mkdir -p /etc/gitlab/ssl @@ -1181,27 +1467,35 @@ To configure Gitaly with TLS: sudo chmod 644 key.pem cert.pem ``` -1. Copy the cert to `/etc/gitlab/trusted-certs` so Gitaly will trust the cert when - calling into itself: +1. Edit `/etc/gitlab/gitlab.rb` and add: - ```shell - sudo cp /etc/gitlab/ssl/cert.pem /etc/gitlab/trusted-certs/ + ```ruby + praefect['tls_listen_addr'] = "0.0.0.0:3305" + praefect['certificate_path'] = "/etc/gitlab/ssl/cert.pem" + praefect['key_path'] = "/etc/gitlab/ssl/key.pem" ``` -1. Edit `/etc/gitlab/gitlab.rb` and add: +1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +1. On the Praefect clients (including each Gitaly server), copy the certificates, + or their certificate authority, into `/etc/gitlab/trusted-certs`: - <!-- - updates to following example must also be made at - https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab - --> + ```shell + sudo cp cert.pem /etc/gitlab/trusted-certs/ + ``` + +1. On the Praefect clients (except Gitaly servers), edit `git_data_dirs` in + `/etc/gitlab/gitlab.rb` as follows: ```ruby - gitaly['tls_listen_addr'] = "0.0.0.0:9999" - gitaly['certificate_path'] = "/etc/gitlab/ssl/cert.pem" - gitaly['key_path'] = "/etc/gitlab/ssl/key.pem" + git_data_dirs({ + "default" => { + "gitaly_address" => 'tls://LOAD_BALANCER_SERVER_ADDRESS:2305', + "gitaly_token" => 'PRAEFECT_EXTERNAL_TOKEN' + } + }) ``` -1. Delete `gitaly['listen_addr']` to allow only encrypted connections. 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). <div align="right"> @@ -1272,17 +1566,20 @@ To configure the Sidekiq nodes, one each one: ### Gitaly ### ####################################### + # git_data_dirs get configured for the Praefect virtual storage + # Address is Internal Load Balancer for Praefect + # Token is praefect_external_token git_data_dirs({ - 'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, - 'storage1' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, - 'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' }, + "default" => { + "gitaly_address" => "tcp://10.6.0.40:2305", # internal load balancer IP + "gitaly_token" => '<praefect_external_token>' + } }) - gitlab_rails['gitaly_token'] = 'YOUR_TOKEN' ####################################### ### Postgres ### ####################################### - gitlab_rails['db_host'] = '10.6.0.20' # internal load balancer IP + gitlab_rails['db_host'] = '10.6.0.40' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '<postgresql_user_password>' gitlab_rails['db_adapter'] = 'postgresql' @@ -1322,13 +1619,14 @@ To configure the Sidekiq nodes, one each one: 'google_project' => '<gcp-project-name>', 'google_json_key_location' => '<path-to-gcp-service-account-key>' } - gitlab_rails['object_store']['objects']['artifacts']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['lfs']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['uploads']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-bucket-name>" + gitlab_rails['object_store']['objects']['artifacts']['bucket'] = "<gcp-artifacts-bucket-name>" + gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = "<gcp-external-diffs-bucket-name>" + gitlab_rails['object_store']['objects']['lfs']['bucket'] = "<gcp-lfs-bucket-name>" + gitlab_rails['object_store']['objects']['uploads']['bucket'] = "<gcp-uploads-bucket-name>" + gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-packages-bucket-name>" + gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>" + gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>" + ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -1400,17 +1698,14 @@ On each node perform the following: ```ruby external_url 'https://gitlab.example.com' - # Gitaly and GitLab use two shared secrets for authentication, one to authenticate gRPC requests - # to Gitaly, and a second for authentication callbacks from GitLab-Shell to the GitLab internal API. - # The following two values must be the same as their respective values - # of the Gitaly setup - gitlab_rails['gitaly_token'] = 'gitalysecret' - gitlab_shell['secret_token'] = 'shellsecret' - + # git_data_dirs get configured for the Praefect virtual storage + # Address is Interal Load Balancer for Praefect + # Token is praefect_external_token git_data_dirs({ - 'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, - 'storage1' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, - 'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' }, + "default" => { + "gitaly_address" => "tcp://10.6.0.40:2305", # internal load balancer IP + "gitaly_token" => '<praefect_external_token>' + } }) ## Disable components that will not be on the GitLab application server @@ -1489,23 +1784,24 @@ On each node perform the following: 'google_project' => '<gcp-project-name>', 'google_json_key_location' => '<path-to-gcp-service-account-key>' } - gitlab_rails['object_store']['objects']['artifacts']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['lfs']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['uploads']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-bucket-name>" + gitlab_rails['object_store']['objects']['artifacts']['bucket'] = "<gcp-artifacts-bucket-name>" + gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = "<gcp-external-diffs-bucket-name>" + gitlab_rails['object_store']['objects']['lfs']['bucket'] = "<gcp-lfs-bucket-name>" + gitlab_rails['object_store']['objects']['uploads']['bucket'] = "<gcp-uploads-bucket-name>" + gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-packages-bucket-name>" + gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>" + gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>" ``` -1. If you're using [Gitaly with TLS support](#gitaly-tls-support), make sure the +1. If you're using [Gitaly with TLS support](#gitaly-cluster-tls-support), make sure the `git_data_dirs` entry is configured with `tls` instead of `tcp`: ```ruby git_data_dirs({ - 'default' => { 'gitaly_address' => 'tls://gitaly1.internal:9999' }, - 'storage1' => { 'gitaly_address' => 'tls://gitaly1.internal:9999' }, - 'storage2' => { 'gitaly_address' => 'tls://gitaly2.internal:9999' }, + "default" => { + "gitaly_address" => "tls://10.6.0.40:2305", # internal load balancer IP + "gitaly_token" => '<praefect_external_token>' + } }) ``` @@ -1697,20 +1993,9 @@ on what features you intend to use: | [Terraform state files](../terraform_state.md#using-object-storage) | Yes | Using separate buckets for each data type is the recommended approach for GitLab. - -A limitation of our configuration is that each use of object storage is separately configured. -[We have an issue for improving this](https://gitlab.com/gitlab-org/gitlab/-/issues/23345) -and easily using one bucket with separate folders is one improvement that this might bring. - -There is at least one specific issue with using the same bucket: -when GitLab is deployed with the Helm chart restore from backup -[will not properly function](https://docs.gitlab.com/charts/advanced/external-object-storage/#lfs-artifacts-uploads-packages-external-diffs-pseudonymizer) -unless separate buckets are used. - -One risk of using a single bucket would be if your organization decided to -migrate GitLab to the Helm deployment in the future. GitLab would run, but the situation with -backups might not be realized until the organization had a critical requirement for the backups to -work. +This ensures there are no collisions across the various types of data GitLab stores. +There are plans to [enable the use of a single bucket](https://gitlab.com/gitlab-org/gitlab/-/issues/292958) +in the future. <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -1718,7 +2003,7 @@ work. </a> </div> -## Configure Advanced Search **(PREMIUM SELF)** +## Configure Advanced Search You can leverage Elasticsearch and [enable Advanced Search](../../integration/elasticsearch.md) for faster, more advanced code search across your entire GitLab instance. @@ -1753,13 +2038,3 @@ as soon as possible. Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> </a> </div> - -## Troubleshooting - -See the [troubleshooting documentation](troubleshooting.md). - -<div align="right"> - <a type="button" class="btn btn-default" href="#setup-components"> - Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> - </a> -</div> diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index 0b22a2d3602..606701a4d83 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -12,100 +12,110 @@ full list of reference architectures, see [Available reference architectures](index.md#available-reference-architectures). > - **Supported users (approximate):** 50,000 -> - **High Availability:** Yes +> - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) > - **Test requests per second (RPS) rates:** API: 1000 RPS, Web: 100 RPS, Git (Pull): 100 RPS, Git (Push): 20 RPS | Service | Nodes | Configuration | GCP | AWS | Azure | |-----------------------------------------|-------------|-------------------------|-----------------|--------------|----------| -| External load balancing node | 1 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | `c5.2xlarge` | F8s v2 | -| Consul | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | -| PostgreSQL | 3 | 16 vCPU, 60 GB memory | n1-standard-16 | `m5.4xlarge` | D16s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | -| Internal load balancing node | 1 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | `c5.2xlarge` | F8s v2 | -| Redis - Cache | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | -| Redis - Queues / Shared State | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | -| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7 GB memory | g1-small | `t3.small` | B1MS | -| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7 GB memory | g1-small | `t3.small` | B1MS | -| Gitaly | 2 (minimum) | 64 vCPU, 240 GB memory | n1-standard-64 | `m5.16xlarge` | D64s v3 | -| Sidekiq | 4 | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | -| GitLab Rails | 12 | 32 vCPU, 28.8 GB memory | n1-highcpu-32 | `c5.9xlarge` | F32s v2 | -| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | +| External load balancing node | 1 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | +| Consul | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| PostgreSQL | 3 | 32 vCPU, 120 GB memory | n1-standard-32 | m5.8xlarge | D32s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Internal load balancing node | 1 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | +| Redis - Cache | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| Redis - Queues / Shared State | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7 GB memory | g1-small | t3.small | B1MS | +| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7 GB memory | g1-small | t3.small | B1MS | +| Gitaly | 3 | 64 vCPU, 240 GB memory | n1-standard-64 | m5.16xlarge | D64s v3 | +| Praefect | 3 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| Praefect PostgreSQL | 1+* | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Sidekiq | 4 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| GitLab Rails | 12 | 32 vCPU, 28.8 GB memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | +| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | | Object storage | n/a | n/a | n/a | n/a | n/a | -| NFS server | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | - -```mermaid -stateDiagram-v2 - [*] --> LoadBalancer - LoadBalancer --> ApplicationServer - - ApplicationServer --> BackgroundJobs - ApplicationServer --> Gitaly - ApplicationServer --> Redis_Cache - ApplicationServer --> Redis_Queues - ApplicationServer --> PgBouncer - PgBouncer --> Database - ApplicationServer --> ObjectStorage - BackgroundJobs --> ObjectStorage - - ApplicationMonitoring -->ApplicationServer - ApplicationMonitoring -->PgBouncer - ApplicationMonitoring -->Database - ApplicationMonitoring -->BackgroundJobs - - ApplicationServer --> Consul - - Consul --> Database - Consul --> PgBouncer - Redis_Cache --> Consul - Redis_Queues --> Consul - BackgroundJobs --> Consul - - state Consul { - "Consul_1..3" - } - - state Database { - "PG_Primary_Node" - "PG_Secondary_Node_1..2" - } - - state Redis_Cache { - "R_Cache_Primary_Node" - "R_Cache_Replica_Node_1..2" - "R_Cache_Sentinel_1..3" - } - - state Redis_Queues { - "R_Queues_Primary_Node" - "R_Queues_Replica_Node_1..2" - "R_Queues_Sentinel_1..3" - } - - state Gitaly { - "Gitaly_1..2" - } - - state BackgroundJobs { - "Sidekiq_1..4" - } - - state ApplicationServer { - "GitLab_Rails_1..12" - } - - state LoadBalancer { - "LoadBalancer_1" - } - - state ApplicationMonitoring { - "Prometheus" - "Grafana" - } - - state PgBouncer { - "Internal_Load_Balancer" - "PgBouncer_1..3" - } +| NFS server | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | + +```plantuml +@startuml 50k +card "**External Load Balancer**" as elb #6a9be7 +card "**Internal Load Balancer**" as ilb #9370DB + +together { + collections "**GitLab Rails** x12" as gitlab #32CD32 + collections "**Sidekiq** x4" as sidekiq #ff8dd1 +} + +together { + card "**Prometheus + Grafana**" as monitor #7FFFD4 + collections "**Consul** x3" as consul #e76a9b +} + +card "Gitaly Cluster" as gitaly_cluster { + collections "**Praefect** x3" as praefect #FF8C00 + collections "**Gitaly** x3" as gitaly #FF8C00 + card "**Praefect PostgreSQL***\n//Non fault-tolerant//" as praefect_postgres #FF8C00 + + praefect -[#FF8C00]-> gitaly + praefect -[#FF8C00]> praefect_postgres +} + +card "Database" as database { + collections "**PGBouncer** x3" as pgbouncer #4EA7FF + card "**PostgreSQL** (Primary)" as postgres_primary #4EA7FF + collections "**PostgreSQL** (Secondary) x2" as postgres_secondary #4EA7FF + + pgbouncer -[#4EA7FF]-> postgres_primary + postgres_primary .[#4EA7FF]> postgres_secondary +} + +card "redis" as redis { + collections "**Redis Persistent** x3" as redis_persistent #FF6347 + collections "**Redis Cache** x3" as redis_cache #FF6347 + collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347 + collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347 + + redis_persistent <.[#FF6347]- redis_persistent_sentinel + redis_cache <.[#FF6347]- redis_cache_sentinel +} + +cloud "**Object Storage**" as object_storage #white + +elb -[#6a9be7]-> gitlab +elb -[#6a9be7]--> monitor + +gitlab -[#32CD32]> sidekiq +gitlab -[#32CD32]--> ilb +gitlab -[#32CD32]-> object_storage +gitlab -[#32CD32]---> redis +gitlab -[hidden]-> monitor +gitlab -[hidden]-> consul + +sidekiq -[#ff8dd1]--> ilb +sidekiq -[#ff8dd1]-> object_storage +sidekiq -[#ff8dd1]---> redis +sidekiq -[hidden]-> monitor +sidekiq -[hidden]-> consul + +ilb -[#9370DB]-> gitaly_cluster +ilb -[#9370DB]-> database + +consul .[#e76a9b]u-> gitlab +consul .[#e76a9b]u-> sidekiq +consul .[#e76a9b]> monitor +consul .[#e76a9b]-> database +consul .[#e76a9b]-> gitaly_cluster +consul .[#e76a9b,norank]--> redis + +monitor .[#7FFFD4]u-> gitlab +monitor .[#7FFFD4]u-> sidekiq +monitor .[#7FFFD4]> consul +monitor .[#7FFFD4]-> database +monitor .[#7FFFD4]-> gitaly_cluster +monitor .[#7FFFD4,norank]--> redis +monitor .[#7FFFD4]> ilb +monitor .[#7FFFD4,norank]u--> elb + +@enduml ``` The Google Cloud Platform (GCP) architectures were built and tested using the @@ -120,19 +130,25 @@ uploads, or artifacts), using an [object storage service](#configure-the-object- is recommended instead of using NFS. Using an object storage service also doesn't require you to provision and maintain a node. +It's also worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and +that to achieve full High Availability a third party PostgreSQL database solution will be required. +We hope to offer a built in solutions for these restrictions in the future but in the meantime a non HA PostgreSQL server +can be set up via Omnibus GitLab, which the above specs reflect. Refer to the following issues for more information: [`omnibus-gitlab#5919`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919) & [`gitaly#3398`](https://gitlab.com/gitlab-org/gitaly/-/issues/3398) + ## Setup components To set up GitLab and its components to accommodate up to 50,000 users: -1. [Configure the external load balancing node](#configure-the-external-load-balancer) +1. [Configure the external load balancer](#configure-the-external-load-balancer) to handle the load balancing of the GitLab application services nodes. +1. [Configure the internal load balancer](#configure-the-internal-load-balancer). + to handle the loa 1. [Configure Consul](#configure-consul). 1. [Configure PostgreSQL](#configure-postgresql), the database for GitLab. 1. [Configure PgBouncer](#configure-pgbouncer). -1. [Configure the internal load balancing node](#configure-the-internal-load-balancer). 1. [Configure Redis](#configure-redis). -1. [Configure Gitaly](#configure-gitaly), - which provides access to the Git repositories. +1. [Configure Gitaly Cluster](#configure-gitaly-cluster), + provides access to the Git repositories. 1. [Configure Sidekiq](#configure-sidekiq). 1. [Configure the main GitLab Rails application](#configure-gitlab-rails) to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend @@ -178,6 +194,11 @@ The following list includes descriptions of each server and its assigned IP: - `10.6.0.83`: Sentinel - Queues 3 - `10.6.0.91`: Gitaly 1 - `10.6.0.92`: Gitaly 2 +- `10.6.0.93`: Gitaly 3 +- `10.6.0.131`: Praefect 1 +- `10.6.0.132`: Praefect 2 +- `10.6.0.133`: Praefect 3 +- `10.6.0.141`: Praefect PostgreSQL 1 (non HA) - `10.6.0.101`: Sidekiq 1 - `10.6.0.102`: Sidekiq 2 - `10.6.0.103`: Sidekiq 3 @@ -185,7 +206,16 @@ The following list includes descriptions of each server and its assigned IP: - `10.6.0.111`: GitLab application 1 - `10.6.0.112`: GitLab application 2 - `10.6.0.113`: GitLab application 3 -- `10.6.0.121`: Prometheus +- `10.6.0.114`: GitLab application 4 +- `10.6.0.115`: GitLab application 5 +- `10.6.0.116`: GitLab application 6 +- `10.6.0.117`: GitLab application 7 +- `10.6.0.118`: GitLab application 8 +- `10.6.0.119`: GitLab application 9 +- `10.6.0.120`: GitLab application 10 +- `10.6.0.121`: GitLab application 11 +- `10.6.0.122`: GitLab application 12 +- `10.6.0.151`: Prometheus ## Configure the external load balancer @@ -308,6 +338,71 @@ Configure DNS for an alternate SSH hostname such as `altssh.gitlab.example.com`. </a> </div> +## Configure the internal load balancer + +The Internal Load Balancer is used to balance any internal connections the GitLab environment requires +such as connections to [PgBouncer](#configure-pgbouncer) and [Praefect](#configure-praefect) (Gitaly Cluster). + +Note that it's a separate node from the External Load Balancer and shouldn't have any access externally. + +The following IP will be used as an example: + +- `10.6.0.40`: Internal Load Balancer + +Here's how you could do it with [HAProxy](https://www.haproxy.org/): + +```plaintext +global + log /dev/log local0 + log localhost local1 notice + log stdout format raw local0 + +defaults + log global + default-server inter 10s fall 3 rise 2 + balance leastconn + +frontend internal-pgbouncer-tcp-in + bind *:6432 + mode tcp + option tcplog + + default_backend pgbouncer + +frontend internal-praefect-tcp-in + bind *:2305 + mode tcp + option tcplog + option clitcpka + + default_backend praefect + +backend pgbouncer + mode tcp + option tcp-check + + server pgbouncer1 10.6.0.21:6432 check + server pgbouncer2 10.6.0.22:6432 check + server pgbouncer3 10.6.0.23:6432 check + +backend praefect + mode tcp + option tcp-check + option srvtcpka + + server praefect1 10.6.0.131:2305 check + server praefect2 10.6.0.132:2305 check + server praefect3 10.6.0.133:2305 check +``` + +Refer to your preferred Load Balancer's documentation for further guidance. + +<div align="right"> + <a type="button" class="btn btn-default" href="#setup-components"> + Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> + </a> +</div> + ## Configure Consul The following IPs will be used as an example: @@ -662,52 +757,6 @@ The following IPs will be used as an example: </a> </div> -### Configure the internal load balancer - -If you're running more than one PgBouncer node as recommended, then at this time you'll need to set -up a TCP internal load balancer to serve each correctly. - -The following IP will be used as an example: - -- `10.6.0.40`: Internal Load Balancer - -Here's how you could do it with [HAProxy](https://www.haproxy.org/): - -```plaintext -global - log /dev/log local0 - log localhost local1 notice - log stdout format raw local0 - -defaults - log global - default-server inter 10s fall 3 rise 2 - balance leastconn - -frontend internal-pgbouncer-tcp-in - bind *:6432 - mode tcp - option tcplog - - default_backend pgbouncer - -backend pgbouncer - mode tcp - option tcp-check - - server pgbouncer1 10.6.0.21:6432 check - server pgbouncer2 10.6.0.22:6432 check - server pgbouncer3 10.6.0.23:6432 check -``` - -Refer to your preferred Load Balancer's documentation for further guidance. - -<div align="right"> - <a type="button" class="btn btn-default" href="#setup-components"> - Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> - </a> -</div> - ## Configure Redis Using [Redis](https://redis.io/) in scalable environment is possible using a **Primary** x **Replica** @@ -1302,19 +1351,283 @@ To configure the Sentinel Queues server: </a> </div> -## Configure Gitaly +## Configure Gitaly Cluster -NOTE: -[Gitaly Cluster](../gitaly/praefect.md) support -for the Reference Architectures is being -worked on as a [collaborative effort](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/1) between the Quality Engineering and Gitaly teams. When this component has been verified -some Architecture specs will likely change as a result to support the new -and improved designed. +[Gitaly Cluster](../gitaly/praefect.md) is a GitLab provided and recommended fault tolerant solution for storing Git repositories. +In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down. + +The recommended cluster setup includes the following components: + +- 3 Gitaly nodes: Replicated storage of Git repositories. +- 3 Praefect nodes: Router and transaction manager for Gitaly Cluster. +- 1 Praefect PostgreSQL node: Database server for Praefect. A third-party solution + is required for Praefect database connections to be made highly available. +- 1 load balancer: A load balancer is required for Praefect. The + [internal load balancer](#configure-the-internal-load-balancer) will be used. + +This section will detail how to configure the recommended standard setup in order. +For more advanced setups refer to the [standalone Gitaly Cluster documentation](../gitaly/praefect.md). + +### Configure Praefect PostgreSQL + +Praefect, the routing and transaction manager for Gitaly Cluster, requires its own database server to store data on Gitaly Cluster status. + +If you want to have a highly available setup, Praefect requires a third-party PostgreSQL database. +A built-in solution is being [worked on](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919). + +#### Praefect non-HA PostgreSQL standalone using Omnibus GitLab + +The following IPs will be used as an example: + +- `10.6.0.141`: Praefect PostgreSQL + +First, make sure to [install](https://about.gitlab.com/install/) +the Linux GitLab package in the Praefect PostgreSQL node. Following the steps, +install the necessary dependencies from step 1, and add the +GitLab package repository from step 2. When installing GitLab +in the second step, do not supply the `EXTERNAL_URL` value. + +1. SSH in to the Praefect PostgreSQL node. +1. Create a strong password to be used for the Praefect PostgreSQL user. Take note of this password as `<praefect_postgresql_password>`. +1. Generate the password hash for the Praefect PostgreSQL username/password pair. This assumes you will use the default + username of `praefect` (recommended). The command will request the password `<praefect_postgresql_password>` + and confirmation. Use the value that is output by this command in the next + step as the value of `<praefect_postgresql_password_hash>`: + + ```shell + sudo gitlab-ctl pg-password-md5 praefect + ``` + +1. Edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: + + ```ruby + # Disable all components except PostgreSQL and Consul + roles ['postgres_role'] + repmgr['enable'] = false + patroni['enable'] = false + + # PostgreSQL configuration + postgresql['listen_address'] = '0.0.0.0' + postgresql['max_connections'] = 200 + + gitlab_rails['auto_migrate'] = false + + # Configure the Consul agent + consul['enable'] = true + ## Enable service discovery for Prometheus + consul['monitoring_service_discovery'] = true + + # START user configuration + # Please set the real values as explained in Required Information section + # + # Replace PRAEFECT_POSTGRESQL_PASSWORD_HASH with a generated md5 value + postgresql['sql_user_password'] = "<praefect_postgresql_password_hash>" + + # Replace XXX.XXX.XXX.XXX/YY with Network Address + postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/24) + + # Set the network addresses that the exporters will listen on for monitoring + node_exporter['listen_address'] = '0.0.0.0:9100' + postgres_exporter['listen_address'] = '0.0.0.0:9187' + + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + # + # END user configuration + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Follow the [post configuration](#praefect-postgresql-post-configuration). + +<div align="right"> + <a type="button" class="btn btn-default" href="#setup-components"> + Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> + </a> +</div> + +#### Praefect HA PostgreSQL third-party solution + +[As noted](#configure-praefect-postgresql), a third-party PostgreSQL solution for +Praefect's database is recommended if aiming for full High Availability. + +There are many third-party solutions for PostgreSQL HA. The solution selected must have the following to work with Praefect: + +- A static IP for all connections that doesn't change on failover. +- [`LISTEN`](https://www.postgresql.org/docs/12/sql-listen.html) SQL functionality must be supported. + +Examples of the above could include [Google's Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) or [Amazon RDS](https://aws.amazon.com/rds/). + +Once the database is set up, follow the [post configuration](#praefect-postgresql-post-configuration). + +#### Praefect PostgreSQL post-configuration + +After the Praefect PostgreSQL server has been set up, you'll then need to configure the user and database for Praefect to use. + +We recommend the user be named `praefect` and the database `praefect_production`, and these can be configured as standard in PostgreSQL. +The password for the user is the same as the one you configured earlier as `<praefect_postgresql_password>`. -[Gitaly](../gitaly/index.md) server node requirements are dependent on data, -specifically the number of projects and those projects' sizes. It's recommended -that a Gitaly server node stores no more than 5 TB of data. Depending on your -repository storage requirements, you may require additional Gitaly server nodes. +This is how this would work with a Omnibus GitLab PostgreSQL setup: + +1. SSH in to the Praefect PostgreSQL node. +1. Connect to the PostgreSQL server with administrative access. + The `gitlab-psql` user should be used here for this as it's added by default in Omnibus. + The database `template1` is used because it is created by default on all PostgreSQL servers. + + ```shell + /opt/gitlab/embedded/bin/psql -U gitlab-psql -d template1 -h POSTGRESQL_SERVER_ADDRESS + ``` + +1. Create the new user `praefect`, replacing `<praefect_postgresql_password>`: + + ```shell + CREATE ROLE praefect WITH LOGIN CREATEDB PASSWORD <praefect_postgresql_password>; + ``` + +1. Reconnect to the PostgreSQL server, this time as the `praefect` user: + + ```shell + /opt/gitlab/embedded/bin/psql -U praefect -d template1 -h POSTGRESQL_SERVER_ADDRESS + ``` + +1. Create a new database `praefect_production`: + + ```shell + CREATE DATABASE praefect_production WITH ENCODING=UTF8; + ``` + +<div align="right"> + <a type="button" class="btn btn-default" href="#setup-components"> + Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> + </a> +</div> + +### Configure Praefect + +Praefect is the router and transaction manager for Gitaly Cluster and all connections to Gitaly go through +it. This section details how to configure it. + +Praefect requires several secret tokens to secure communications across the Cluster: + +- `<praefect_external_token>`: Used for repositories hosted on your Gitaly cluster and can only be accessed by Gitaly clients that carry this token. +- `<praefect_internal_token>`: Used for replication traffic inside your Gitaly cluster. This is distinct from `praefect_external_token` because Gitaly clients must not be able to access internal nodes of the Praefect cluster directly; that could lead to data loss. +- `<praefect_postgresql_password>`: The Praefect PostgreSQL password defined in the previous section is also required as part of this setup. + +Gitaly Cluster nodes are configured in Praefect via a `virtual storage`. Each storage contains +the details of each Gitaly node that makes up the cluster. Each storage is also given a name +and this name is used in several areas of the config. In this guide, the name of the storage will be +`default`. Also, this guide is geared towards new installs, if upgrading an existing environment +to use Gitaly Cluster, you may need to use a different name. +Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more info. + +The following IPs will be used as an example: + +- `10.6.0.131`: Praefect 1 +- `10.6.0.132`: Praefect 2 +- `10.6.0.133`: Praefect 3 + +To configure the Praefect nodes, on each one: + +1. SSH in to the Praefect server. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to follow _only_ installation steps 1 and 2 + on the page. +1. Edit the `/etc/gitlab/gitlab.rb` file to configure Praefect: + + ```ruby + # Avoid running unnecessary services on the Gitaly server + postgresql['enable'] = false + redis['enable'] = false + nginx['enable'] = false + puma['enable'] = false + unicorn['enable'] = false + sidekiq['enable'] = false + gitlab_workhorse['enable'] = false + grafana['enable'] = false + + # If you run a separate monitoring node you can disable these services + alertmanager['enable'] = false + prometheus['enable'] = false + + # Praefect Configuration + praefect['enable'] = true + praefect['listen_addr'] = '0.0.0.0:2305' + + gitlab_rails['rake_cache_clear'] = false + gitlab_rails['auto_migrate'] = false + + # Configure the Consul agent + consul['enable'] = true + ## Enable service discovery for Prometheus + consul['monitoring_service_discovery'] = true + + # START user configuration + # Please set the real values as explained in Required Information section + # + + # Praefect External Token + # This is needed by clients outside the cluster (like GitLab Shell) to communicate with the Praefect cluster + praefect['auth_token'] = '<praefect_external_token>' + + # Praefect Database Settings + praefect['database_host'] = '10.6.0.141' + praefect['database_port'] = 5432 + # `no_proxy` settings must always be a direct connection for caching + praefect['database_host_no_proxy'] = '10.6.0.141' + praefect['database_port_no_proxy'] = 5432 + praefect['database_dbname'] = 'praefect_production' + praefect['database_user'] = 'praefect' + praefect['database_password'] = '<praefect_postgresql_password>' + + # Praefect Virtual Storage config + # Name of storage hash must match storage name in git_data_dirs on GitLab + # server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1') + praefect['virtual_storages'] = { + 'default' => { + 'gitaly-1' => { + 'address' => 'tcp://10.6.0.91:8075', + 'token' => '<praefect_internal_token>', + 'primary' => true + }, + 'gitaly-2' => { + 'address' => 'tcp://10.6.0.92:8075', + 'token' => '<praefect_internal_token>' + }, + 'gitaly-3' => { + 'address' => 'tcp://10.6.0.93:8075', + 'token' => '<praefect_internal_token>' + }, + } + } + + # Set the network addresses that the exporters will listen on for monitoring + node_exporter['listen_address'] = '0.0.0.0:9100' + praefect['prometheus_listen_addr'] = '0.0.0.0:9652' + + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + # + # END user configuration + ``` + + 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and + then replace the file of the same name on this server. If that file isn't on + this server, add the file from your Consul server to this server. + + 1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +### Configure Gitaly + +The [Gitaly](../gitaly/index.md) server nodes that make up the cluster have +requirements that are dependent on data, specifically the number of projects +and those projects' sizes. It's recommended that a Gitaly Cluster stores +no more than 5 TB of data on each node. Depending on your +repository storage requirements, you may require additional Gitaly Clusters. Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs @@ -1325,36 +1638,21 @@ adjusted to greater or lesser values depending on the scale of your environment's workload. If you're running the environment on a Cloud provider, refer to their documentation about how to configure IOPS correctly. -Be sure to note the following items: +Gitaly servers must not be exposed to the public internet, as Gitaly's network +traffic is unencrypted by default. The use of a firewall is highly recommended +to restrict access to the Gitaly server. Another option is to +[use TLS](#gitaly-cluster-tls-support). -- The GitLab Rails application shards repositories into - [repository storage paths](../repository_storage_paths.md). -- A Gitaly server can host one or more storage paths. -- A GitLab server can use one or more Gitaly server nodes. -- Gitaly addresses must be specified to be correctly resolvable for all Gitaly - clients. -- Gitaly servers must not be exposed to the public internet, as Gitaly's network - traffic is unencrypted by default. The use of a firewall is highly recommended - to restrict access to the Gitaly server. Another option is to - [use TLS](#gitaly-tls-support). +For configuring Gitaly you should note the following: -NOTE: -The token referred to throughout the Gitaly documentation is an arbitrary -password selected by the administrator. This token is unrelated to tokens -created for the GitLab API or other similar web API tokens. - -This section describes how to configure two Gitaly servers, with the following -IPs and domain names: - -- `10.6.0.91`: Gitaly 1 (`gitaly1.internal`) -- `10.6.0.92`: Gitaly 2 (`gitaly2.internal`) +- `git_data_dirs` should be configured to reflect the storage path for the specific Gitaly node +- `auth_token` should be the same as `praefect_internal_token` -Assumptions about your servers include having the secret token be `gitalysecret`, -and that your GitLab installation has three repository storages: +The following IPs will be used as an example: -- `default` on Gitaly 1 -- `storage1` on Gitaly 1 -- `storage2` on Gitaly 2 +- `10.6.0.91`: Gitaly 1 +- `10.6.0.92`: Gitaly 2 +- `10.6.0.93`: Gitaly 3 On each node: @@ -1364,21 +1662,9 @@ On each node: 1. Edit the Gitaly server node's `/etc/gitlab/gitlab.rb` file to configure storage paths, enable the network listener, and to configure the token: - <!-- - updates to following example must also be made at - https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab - --> - ```ruby # /etc/gitlab/gitlab.rb - # Gitaly and GitLab use two shared secrets for authentication, one to authenticate gRPC requests - # to Gitaly, and a second for authentication callbacks from GitLab-Shell to the GitLab internal API. - # The following two values must be the same as their respective values - # of the GitLab Rails application setup - gitaly['auth_token'] = 'gitalysecret' - gitlab_shell['secret_token'] = 'shellsecret' - # Avoid running unnecessary services on the Gitaly server postgresql['enable'] = false redis['enable'] = false @@ -1407,36 +1693,42 @@ On each node: # firewalls to restrict access to this address/port. # Comment out following line if you only want to support TLS connections gitaly['listen_addr'] = "0.0.0.0:8075" + + # Gitaly Auth Token + # Should be the same as praefect_internal_token + gitaly['auth_token'] = '<praefect_internal_token>' ``` 1. Append the following to `/etc/gitlab/gitlab.rb` for each respective server: - - On `gitaly1.internal`: + - On Gitaly node 1: ```ruby git_data_dirs({ - 'default' => { - 'path' => '/var/opt/gitlab/git-data' - }, - 'storage1' => { - 'path' => '/mnt/gitlab/git-data' - }, + "gitaly-1" => { + "path" => "/var/opt/gitlab/git-data" + } }) ``` - - On `gitaly2.internal`: + - On Gitaly node 2: ```ruby git_data_dirs({ - 'storage2' => { - 'path' => '/mnt/gitlab/git-data' - }, + "gitaly-2" => { + "path" => "/var/opt/gitlab/git-data" + } }) ``` - <!-- - updates to following example must also be made at - https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab - --> + - On Gitaly node 3: + + ```ruby + git_data_dirs({ + "gitaly-3" => { + "path" => "/var/opt/gitlab/git-data" + } + }) + ``` 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and then replace the file of the same name on this server. If that file isn't on @@ -1444,34 +1736,44 @@ On each node: 1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -### Gitaly TLS support +### Gitaly Cluster TLS support -Gitaly supports TLS encryption. To be able to communicate -with a Gitaly instance that listens for secure connections you will need to use `tls://` URL -scheme in the `gitaly_address` of the corresponding storage entry in the GitLab configuration. +Praefect supports TLS encryption. To communicate with a Praefect instance that listens +for secure connections, you must: -You will need to bring your own certificates as this isn't provided automatically. -The certificate, or its certificate authority, must be installed on all Gitaly -nodes (including the Gitaly node using the certificate) and on all client nodes -that communicate with it following the procedure described in -[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). +- Use a `tls://` URL scheme in the `gitaly_address` of the corresponding storage entry + in the GitLab configuration. +- Bring your own certificates because this isn't provided automatically. The certificate + corresponding to each Praefect server must be installed on that Praefect server. -NOTE: -The self-signed certificate must specify the address you use to access the -Gitaly server. If you are addressing the Gitaly server by a hostname, you can -either use the Common Name field for this, or add it as a Subject Alternative -Name. If you are addressing the Gitaly server by its IP address, you must add it -as a Subject Alternative Name to the certificate. -[gRPC does not support using an IP address as Common Name in a certificate](https://github.com/grpc/grpc/issues/2691). +Additionally the certificate, or its certificate authority, must be installed on all Gitaly servers +and on all Praefect clients that communicate with it following the procedure described in +[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates) (and repeated below). + +Note the following: -It's possible to configure Gitaly servers with both an unencrypted listening -address (`listen_addr`) and an encrypted listening address (`tls_listen_addr`) -at the same time. This allows you to do a gradual transition from unencrypted to -encrypted traffic, if necessary. +- The certificate must specify the address you use to access the Praefect server. If + addressing the Praefect server by: -To configure Gitaly with TLS: + - Hostname, you can either use the Common Name field for this, or add it as a Subject + Alternative Name. + - IP address, you must add it as a Subject Alternative Name to the certificate. -1. Create the `/etc/gitlab/ssl` directory and copy your key and certificate there: +- You can configure Praefect servers with both an unencrypted listening address + `listen_addr` and an encrypted listening address `tls_listen_addr` at the same time. + This allows you to do a gradual transition from unencrypted to encrypted traffic, if + necessary. + +- The Internal Load Balancer will also access to the certificates and need to be configured + to allow for TLS passthrough. + Refer to the load balancers documentation on how to configure this. + +To configure Praefect with TLS: + +1. Create certificates for Praefect servers. + +1. On the Praefect servers, create the `/etc/gitlab/ssl` directory and copy your key + and certificate there: ```shell sudo mkdir -p /etc/gitlab/ssl @@ -1480,27 +1782,34 @@ To configure Gitaly with TLS: sudo chmod 644 key.pem cert.pem ``` -1. Copy the cert to `/etc/gitlab/trusted-certs` so Gitaly will trust the cert when - calling into itself: +1. Edit `/etc/gitlab/gitlab.rb` and add: - ```shell - sudo cp /etc/gitlab/ssl/cert.pem /etc/gitlab/trusted-certs/ + ```ruby + praefect['tls_listen_addr'] = "0.0.0.0:3305" + praefect['certificate_path'] = "/etc/gitlab/ssl/cert.pem" + praefect['key_path'] = "/etc/gitlab/ssl/key.pem" ``` -1. Edit `/etc/gitlab/gitlab.rb` and add: +1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure). - <!-- - updates to following example must also be made at - https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab - --> +1. On the Praefect clients (including each Gitaly server), copy the certificates, + or their certificate authority, into `/etc/gitlab/trusted-certs`: - ```ruby - gitaly['tls_listen_addr'] = "0.0.0.0:9999" - gitaly['certificate_path'] = "/etc/gitlab/ssl/cert.pem" - gitaly['key_path'] = "/etc/gitlab/ssl/key.pem" + ```shell + sudo cp cert.pem /etc/gitlab/trusted-certs/ ``` -1. Delete `gitaly['listen_addr']` to allow only encrypted connections. +1. On the Praefect clients (except Gitaly servers), edit `git_data_dirs` in + `/etc/gitlab/gitlab.rb` as follows: + + ```ruby + git_data_dirs({ + "default" => { + "gitaly_address" => 'tls://LOAD_BALANCER_SERVER_ADDRESS:2305', + "gitaly_token" => 'PRAEFECT_EXTERNAL_TOKEN' + } + }) + ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -1587,12 +1896,15 @@ To configure the Sidekiq nodes, on each one: ### Gitaly ### ####################################### + # git_data_dirs get configured for the Praefect virtual storage + # Address is Internal Load Balancer for Praefect + # Token is praefect_external_token git_data_dirs({ - 'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, - 'storage1' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, - 'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' }, + "default" => { + "gitaly_address" => "tcp://10.6.0.40:2305", # internal load balancer IP + "gitaly_token" => '<praefect_external_token>' + } }) - gitlab_rails['gitaly_token'] = 'YOUR_TOKEN' ####################################### ### Postgres ### @@ -1624,7 +1936,7 @@ To configure the Sidekiq nodes, on each one: node_exporter['listen_address'] = '0.0.0.0:9100' # Rails Status for prometheus - gitlab_rails['monitoring_whitelist'] = ['10.6.0.121/32', '127.0.0.0/8'] + gitlab_rails['monitoring_whitelist'] = ['10.6.0.151/32', '127.0.0.0/8'] ############################# ### Object storage ### @@ -1637,13 +1949,13 @@ To configure the Sidekiq nodes, on each one: 'google_project' => '<gcp-project-name>', 'google_json_key_location' => '<path-to-gcp-service-account-key>' } - gitlab_rails['object_store']['objects']['artifacts']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['lfs']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['uploads']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-bucket-name>" + gitlab_rails['object_store']['objects']['artifacts']['bucket'] = "<gcp-artifacts-bucket-name>" + gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = "<gcp-external-diffs-bucket-name>" + gitlab_rails['object_store']['objects']['lfs']['bucket'] = "<gcp-lfs-bucket-name>" + gitlab_rails['object_store']['objects']['uploads']['bucket'] = "<gcp-uploads-bucket-name>" + gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-packages-bucket-name>" + gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>" + gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>" ``` 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace @@ -1671,6 +1983,15 @@ The following IPs will be used as an example: - `10.6.0.111`: GitLab application 1 - `10.6.0.112`: GitLab application 2 - `10.6.0.113`: GitLab application 3 +- `10.6.0.114`: GitLab application 4 +- `10.6.0.115`: GitLab application 5 +- `10.6.0.116`: GitLab application 6 +- `10.6.0.117`: GitLab application 7 +- `10.6.0.118`: GitLab application 8 +- `10.6.0.119`: GitLab application 9 +- `10.6.0.120`: GitLab application 10 +- `10.6.0.121`: GitLab application 11 +- `10.6.0.122`: GitLab application 12 On each node perform the following: @@ -1690,17 +2011,14 @@ On each node perform the following: ```ruby external_url 'https://gitlab.example.com' - # Gitaly and GitLab use two shared secrets for authentication, one to authenticate gRPC requests - # to Gitaly, and a second for authentication callbacks from GitLab-Shell to the GitLab internal API. - # The following two values must be the same as their respective values - # of the Gitaly setup - gitlab_rails['gitaly_token'] = 'gitalysecret' - gitlab_shell['secret_token'] = 'shellsecret' - + # git_data_dirs get configured for the Praefect virtual storage + # Address is Interal Load Balancer for Praefect + # Token is praefect_external_token git_data_dirs({ - 'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, - 'storage1' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, - 'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' }, + "default" => { + "gitaly_address" => "tcp://10.6.0.40:2305", # internal load balancer IP + "gitaly_token" => '<praefect_external_token>' + } }) ## Disable components that will not be on the GitLab application server @@ -1755,8 +2073,8 @@ On each node perform the following: # Add the monitoring node's IP address to the monitoring whitelist and allow it to # scrape the NGINX metrics - gitlab_rails['monitoring_whitelist'] = ['10.6.0.121/32', '127.0.0.0/8'] - nginx['status']['options']['allow'] = ['10.6.0.121/32', '127.0.0.0/8'] + gitlab_rails['monitoring_whitelist'] = ['10.6.0.151/32', '127.0.0.0/8'] + nginx['status']['options']['allow'] = ['10.6.0.151/32', '127.0.0.0/8'] ############################# ### Object storage ### @@ -1769,24 +2087,25 @@ On each node perform the following: 'google_project' => '<gcp-project-name>', 'google_json_key_location' => '<path-to-gcp-service-account-key>' } - gitlab_rails['object_store']['objects']['artifacts']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['lfs']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['uploads']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-bucket-name>" + gitlab_rails['object_store']['objects']['artifacts']['bucket'] = "<gcp-artifacts-bucket-name>" + gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = "<gcp-external-diffs-bucket-name>" + gitlab_rails['object_store']['objects']['lfs']['bucket'] = "<gcp-lfs-bucket-name>" + gitlab_rails['object_store']['objects']['uploads']['bucket'] = "<gcp-uploads-bucket-name>" + gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-packages-bucket-name>" + gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>" + gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>" ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -1. If you're using [Gitaly with TLS support](#gitaly-tls-support), make sure the +1. If you're using [Gitaly with TLS support](#gitaly-cluster-tls-support), make sure the `git_data_dirs` entry is configured with `tls` instead of `tcp`: ```ruby git_data_dirs({ - 'default' => { 'gitaly_address' => 'tls://gitaly1.internal:9999' }, - 'storage1' => { 'gitaly_address' => 'tls://gitaly1.internal:9999' }, - 'storage2' => { 'gitaly_address' => 'tls://gitaly2.internal:9999' }, + "default" => { + "gitaly_address" => "tls://10.6.0.40:2305", # internal load balancer IP + "gitaly_token" => '<praefect_external_token>' + } }) ``` @@ -1891,7 +2210,7 @@ running [Prometheus](../monitoring/prometheus/index.md) and The following IP will be used as an example: -- `10.6.0.121`: Prometheus +- `10.6.0.151`: Prometheus To configure the Monitoring node: @@ -2002,20 +2321,9 @@ on what features you intend to use: | [Terraform state files](../terraform_state.md#using-object-storage) | Yes | Using separate buckets for each data type is the recommended approach for GitLab. - -A limitation of our configuration is that each use of object storage is separately configured. -[We have an issue for improving this](https://gitlab.com/gitlab-org/gitlab/-/issues/23345) -and easily using one bucket with separate folders is one improvement that this might bring. - -There is at least one specific issue with using the same bucket: -when GitLab is deployed with the Helm chart restore from backup -[will not properly function](https://docs.gitlab.com/charts/advanced/external-object-storage/#lfs-artifacts-uploads-packages-external-diffs-pseudonymizer) -unless separate buckets are used. - -One risk of using a single bucket would be if your organization decided to -migrate GitLab to the Helm deployment in the future. GitLab would run, but the situation with -backups might not be realized until the organization had a critical requirement for the backups to -work. +This ensures there are no collisions across the various types of data GitLab stores. +There are plans to [enable the use of a single bucket](https://gitlab.com/gitlab-org/gitlab/-/issues/292958) +in the future. <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -2023,7 +2331,7 @@ work. </a> </div> -## Configure Advanced Search **(PREMIUM SELF)** +## Configure Advanced Search You can leverage Elasticsearch and [enable Advanced Search](../../integration/elasticsearch.md) for faster, more advanced code search across your entire GitLab instance. @@ -2058,13 +2366,3 @@ as soon as possible. Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> </a> </div> - -## Troubleshooting - -See the [troubleshooting documentation](troubleshooting.md). - -<div align="right"> - <a type="button" class="btn btn-default" href="#setup-components"> - Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> - </a> -</div> diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index 37d35c299fa..0a236eac243 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -19,79 +19,107 @@ costly-to-operate environment by using the [2,000-user reference architecture](2k_users.md). > - **Supported users (approximate):** 5,000 -> - **High Availability:** Yes +> - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) > - **Test requests per second (RPS) rates:** API: 100 RPS, Web: 10 RPS, Git (Pull): 10 RPS, Git (Push): 2 RPS | Service | Nodes | Configuration | GCP | AWS | Azure | |--------------------------------------------|-------------|-------------------------|----------------|-------------|----------| -| External load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | -| Redis | 3 | 2 vCPU, 7.5 GB memory | n1-standard-2 | `m5.large` | D2s v3 | -| Consul + Sentinel | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | -| PostgreSQL | 3 | 2 vCPU, 7.5 GB memory | n1-standard-2 | `m5.large` | D2s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | -| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | -| Gitaly | 2 (minimum) | 8 vCPU, 30 GB memory | n1-standard-8 | `m5.2xlarge` | D8s v3 | -| Sidekiq | 4 | 2 vCPU, 7.5 GB memory | n1-standard-2 | `m5.large` | D2s v3 | -| GitLab Rails | 3 | 16 vCPU, 14.4 GB memory | n1-highcpu-16 | `c5.4xlarge` | F16s v2 | -| Monitoring node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| External load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Redis | 3 | 2 vCPU, 7.5 GB memory | n1-standard-2 | m5.large | D2s v3 | +| Consul + Sentinel | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| PostgreSQL | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Gitaly | 3 | 8 vCPU, 30 GB memory | n1-standard-8 | m5.2xlarge | D8s v3 | +| Praefect | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Praefect PostgreSQL | 1+* | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Sidekiq | 4 | 2 vCPU, 7.5 GB memory | n1-standard-2 | m5.large | D2s v3 | +| GitLab Rails | 3 | 16 vCPU, 14.4 GB memory | n1-highcpu-16 | c5.4xlarge | F16s v2 | +| Monitoring node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | | Object storage | n/a | n/a | n/a | n/a | n/a | -| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | - -```mermaid -stateDiagram-v2 - [*] --> LoadBalancer - LoadBalancer --> ApplicationServer - - ApplicationServer --> BackgroundJobs - ApplicationServer --> Gitaly - ApplicationServer --> Redis - ApplicationServer --> PgBouncer - PgBouncer --> Database - ApplicationServer --> ObjectStorage - BackgroundJobs --> ObjectStorage - - ApplicationMonitoring -->ApplicationServer - ApplicationMonitoring -->Redis - ApplicationMonitoring -->PgBouncer - ApplicationMonitoring -->Database - ApplicationMonitoring -->BackgroundJobs - - state Database { - "PG_Primary_Node" - "PG_Secondary_Node_1..2" - } - - state Redis { - "R_Primary_Node" - "R_Replica_Node_1..2" - "R_Consul/Sentinel_1..3" - } - - state Gitaly { - "Gitaly_1..2" - } - - state BackgroundJobs { - "Sidekiq_1..4" - } - - state ApplicationServer { - "GitLab_Rails_1..3" - } - - state LoadBalancer { - "LoadBalancer_1" - } - - state ApplicationMonitoring { - "Prometheus" - "Grafana" - } - - state PgBouncer { - "Internal_Load_Balancer" - "PgBouncer_1..3" - } +| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | + +```plantuml +@startuml 5k +card "**External Load Balancer**" as elb #6a9be7 +card "**Internal Load Balancer**" as ilb #9370DB + +together { + collections "**GitLab Rails** x3" as gitlab #32CD32 + collections "**Sidekiq** x4" as sidekiq #ff8dd1 +} + +together { + card "**Prometheus + Grafana**" as monitor #7FFFD4 + collections "**Consul** x3" as consul #e76a9b +} + +card "Gitaly Cluster" as gitaly_cluster { + collections "**Praefect** x3" as praefect #FF8C00 + collections "**Gitaly** x3" as gitaly #FF8C00 + card "**Praefect PostgreSQL***\n//Non fault-tolerant//" as praefect_postgres #FF8C00 + + praefect -[#FF8C00]-> gitaly + praefect -[#FF8C00]> praefect_postgres +} + +card "Database" as database { + collections "**PGBouncer** x3" as pgbouncer #4EA7FF + card "**PostgreSQL** (Primary)" as postgres_primary #4EA7FF + collections "**PostgreSQL** (Secondary) x2" as postgres_secondary #4EA7FF + + pgbouncer -[#4EA7FF]-> postgres_primary + postgres_primary .[#4EA7FF]> postgres_secondary +} + +card "redis" as redis { + collections "**Redis Persistent** x3" as redis_persistent #FF6347 + collections "**Redis Cache** x3" as redis_cache #FF6347 + collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347 + collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347 + + redis_persistent <.[#FF6347]- redis_persistent_sentinel + redis_cache <.[#FF6347]- redis_cache_sentinel +} + +cloud "**Object Storage**" as object_storage #white + +elb -[#6a9be7]-> gitlab +elb -[#6a9be7]--> monitor + +gitlab -[#32CD32]> sidekiq +gitlab -[#32CD32]--> ilb +gitlab -[#32CD32]-> object_storage +gitlab -[#32CD32]---> redis +gitlab -[hidden]-> monitor +gitlab -[hidden]-> consul + +sidekiq -[#ff8dd1]--> ilb +sidekiq -[#ff8dd1]-> object_storage +sidekiq -[#ff8dd1]---> redis +sidekiq -[hidden]-> monitor +sidekiq -[hidden]-> consul + +ilb -[#9370DB]-> gitaly_cluster +ilb -[#9370DB]-> database + +consul .[#e76a9b]u-> gitlab +consul .[#e76a9b]u-> sidekiq +consul .[#e76a9b]> monitor +consul .[#e76a9b]-> database +consul .[#e76a9b]-> gitaly_cluster +consul .[#e76a9b,norank]--> redis + +monitor .[#7FFFD4]u-> gitlab +monitor .[#7FFFD4]u-> sidekiq +monitor .[#7FFFD4]> consul +monitor .[#7FFFD4]-> database +monitor .[#7FFFD4]-> gitaly_cluster +monitor .[#7FFFD4,norank]--> redis +monitor .[#7FFFD4]> ilb +monitor .[#7FFFD4,norank]u--> elb + +@enduml ``` The Google Cloud Platform (GCP) architectures were built and tested using the @@ -106,19 +134,25 @@ uploads, or artifacts), using an [object storage service](#configure-the-object- is recommended instead of using NFS. Using an object storage service also doesn't require you to provision and maintain a node. +It's also worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and +that to achieve full High Availability a third party PostgreSQL database solution will be required. +We hope to offer a built in solutions for these restrictions in the future but in the meantime a non HA PostgreSQL server +can be set up via Omnibus GitLab, which the above specs reflect. Refer to the following issues for more information: [`omnibus-gitlab#5919`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919) & [`gitaly#3398`](https://gitlab.com/gitlab-org/gitaly/-/issues/3398) + ## Setup components To set up GitLab and its components to accommodate up to 5,000 users: -1. [Configure the external load balancing node](#configure-the-external-load-balancer) +1. [Configure the external load balancer](#configure-the-external-load-balancer) to handle the load balancing of the GitLab application services nodes. +1. [Configure the internal load balancer](#configure-the-internal-load-balancer). + to handle the load balancing of GitLab application internal connections. 1. [Configure Redis](#configure-redis). 1. [Configure Consul and Sentinel](#configure-consul-and-sentinel). 1. [Configure PostgreSQL](#configure-postgresql), the database for GitLab. 1. [Configure PgBouncer](#configure-pgbouncer). -1. [Configure the internal load balancing node](#configure-the-internal-load-balancer). -1. [Configure Gitaly](#configure-gitaly), - which provides access to the Git repositories. +1. [Configure Gitaly Cluster](#configure-gitaly-cluster), + provides access to the Git repositories. 1. [Configure Sidekiq](#configure-sidekiq). 1. [Configure the main GitLab Rails application](#configure-gitlab-rails) to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend @@ -155,6 +189,11 @@ The following list includes descriptions of each server and its assigned IP: - `10.6.0.20`: Internal Load Balancer - `10.6.0.51`: Gitaly 1 - `10.6.0.52`: Gitaly 2 +- `10.6.0.93`: Gitaly 3 +- `10.6.0.131`: Praefect 1 +- `10.6.0.132`: Praefect 2 +- `10.6.0.133`: Praefect 3 +- `10.6.0.141`: Praefect PostgreSQL 1 (non HA) - `10.6.0.71`: Sidekiq 1 - `10.6.0.72`: Sidekiq 2 - `10.6.0.73`: Sidekiq 3 @@ -285,6 +324,71 @@ Configure DNS for an alternate SSH hostname such as `altssh.gitlab.example.com`. </a> </div> +## Configure the internal load balancer + +The Internal Load Balancer is used to balance any internal connections the GitLab environment requires +such as connections to [PgBouncer](#configure-pgbouncer) and [Praefect](#configure-praefect) (Gitaly Cluster). + +Note that it's a separate node from the External Load Balancer and shouldn't have any access externally. + +The following IP will be used as an example: + +- `10.6.0.40`: Internal Load Balancer + +Here's how you could do it with [HAProxy](https://www.haproxy.org/): + +```plaintext +global + log /dev/log local0 + log localhost local1 notice + log stdout format raw local0 + +defaults + log global + default-server inter 10s fall 3 rise 2 + balance leastconn + +frontend internal-pgbouncer-tcp-in + bind *:6432 + mode tcp + option tcplog + + default_backend pgbouncer + +frontend internal-praefect-tcp-in + bind *:2305 + mode tcp + option tcplog + option clitcpka + + default_backend praefect + +backend pgbouncer + mode tcp + option tcp-check + + server pgbouncer1 10.6.0.21:6432 check + server pgbouncer2 10.6.0.22:6432 check + server pgbouncer3 10.6.0.23:6432 check + +backend praefect + mode tcp + option tcp-check + option srvtcpka + + server praefect1 10.6.0.131:2305 check + server praefect2 10.6.0.132:2305 check + server praefect3 10.6.0.133:2305 check +``` + +Refer to your preferred Load Balancer's documentation for further guidance. + +<div align="right"> + <a type="button" class="btn btn-default" href="#setup-components"> + Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> + </a> +</div> + ## Configure Redis Using [Redis](https://redis.io/) in scalable environment is possible using a **Primary** x **Replica** @@ -924,45 +1028,96 @@ The following IPs will be used as an example: </a> </div> -### Configure the internal load balancer +## Configure Gitaly Cluster -If you're running more than one PgBouncer node as recommended, then at this time you'll need to set -up a TCP internal load balancer to serve each correctly. +[Gitaly Cluster](../gitaly/praefect.md) is a GitLab provided and recommended fault tolerant solution for storing Git repositories. +In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down. -The following IP will be used as an example: +The recommended cluster setup includes the following components: -- `10.6.0.20`: Internal Load Balancer +- 3 Gitaly nodes: Replicated storage of Git repositories. +- 3 Praefect nodes: Router and transaction manager for Gitaly Cluster. +- 1 Praefect PostgreSQL node: Database server for Praefect. A third-party solution + is required for Praefect database connections to be made highly available. +- 1 load balancer: A load balancer is required for Praefect. The + [internal load balancer](#configure-the-internal-load-balancer) will be used. -Here's how you could do it with [HAProxy](https://www.haproxy.org/): +This section will detail how to configure the recommended standard setup in order. +For more advanced setups refer to the [standalone Gitaly Cluster documentation](../gitaly/praefect.md). -```plaintext -global - log /dev/log local0 - log localhost local1 notice - log stdout format raw local0 +### Configure Praefect PostgreSQL -defaults - log global - default-server inter 10s fall 3 rise 2 - balance leastconn +Praefect, the routing and transaction manager for Gitaly Cluster, requires its own database server to store data on Gitaly Cluster status. -frontend internal-pgbouncer-tcp-in - bind *:6432 - mode tcp - option tcplog +If you want to have a highly available setup, Praefect requires a third-party PostgreSQL database. +A built-in solution is being [worked on](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919). - default_backend pgbouncer +#### Praefect non-HA PostgreSQL standalone using Omnibus GitLab -backend pgbouncer - mode tcp - option tcp-check +The following IPs will be used as an example: - server pgbouncer1 10.6.0.21:6432 check - server pgbouncer2 10.6.0.22:6432 check - server pgbouncer3 10.6.0.23:6432 check -``` +- `10.6.0.141`: Praefect PostgreSQL -Refer to your preferred Load Balancer's documentation for further guidance. +First, make sure to [install](https://about.gitlab.com/install/) +the Linux GitLab package in the Praefect PostgreSQL node. Following the steps, +install the necessary dependencies from step 1, and add the +GitLab package repository from step 2. When installing GitLab +in the second step, do not supply the `EXTERNAL_URL` value. + +1. SSH in to the Praefect PostgreSQL node. +1. Create a strong password to be used for the Praefect PostgreSQL user. Take note of this password as `<praefect_postgresql_password>`. +1. Generate the password hash for the Praefect PostgreSQL username/password pair. This assumes you will use the default + username of `praefect` (recommended). The command will request the password `<praefect_postgresql_password>` + and confirmation. Use the value that is output by this command in the next + step as the value of `<praefect_postgresql_password_hash>`: + + ```shell + sudo gitlab-ctl pg-password-md5 praefect + ``` + +1. Edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: + + ```ruby + # Disable all components except PostgreSQL and Consul + roles ['postgres_role'] + repmgr['enable'] = false + patroni['enable'] = false + + # PostgreSQL configuration + postgresql['listen_address'] = '0.0.0.0' + postgresql['max_connections'] = 200 + + gitlab_rails['auto_migrate'] = false + + # Configure the Consul agent + consul['enable'] = true + ## Enable service discovery for Prometheus + consul['monitoring_service_discovery'] = true + + # START user configuration + # Please set the real values as explained in Required Information section + # + # Replace PRAEFECT_POSTGRESQL_PASSWORD_HASH with a generated md5 value + postgresql['sql_user_password'] = "<praefect_postgresql_password_hash>" + + # Replace XXX.XXX.XXX.XXX/YY with Network Address + postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/24) + + # Set the network addresses that the exporters will listen on for monitoring + node_exporter['listen_address'] = '0.0.0.0:9100' + postgres_exporter['listen_address'] = '0.0.0.0:9187' + + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + # + # END user configuration + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Follow the [post configuration](#praefect-postgresql-post-configuration). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -970,19 +1125,186 @@ Refer to your preferred Load Balancer's documentation for further guidance. </a> </div> -## Configure Gitaly +#### Praefect HA PostgreSQL third-party solution -NOTE: -[Gitaly Cluster](../gitaly/praefect.md) support -for the Reference Architectures is being -worked on as a [collaborative effort](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/1) between the Quality Engineering and Gitaly teams. When this component has been verified -some Architecture specs will likely change as a result to support the new -and improved designed. +[As noted](#configure-praefect-postgresql), a third-party PostgreSQL solution for +Praefect's database is recommended if aiming for full High Availability. + +There are many third-party solutions for PostgreSQL HA. The solution selected must have the following to work with Praefect: + +- A static IP for all connections that doesn't change on failover. +- [`LISTEN`](https://www.postgresql.org/docs/12/sql-listen.html) SQL functionality must be supported. + +Examples of the above could include [Google's Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) or [Amazon RDS](https://aws.amazon.com/rds/). + +Once the database is set up, follow the [post configuration](#praefect-postgresql-post-configuration). + +#### Praefect PostgreSQL post-configuration + +After the Praefect PostgreSQL server has been set up, you'll then need to configure the user and database for Praefect to use. + +We recommend the user be named `praefect` and the database `praefect_production`, and these can be configured as standard in PostgreSQL. +The password for the user is the same as the one you configured earlier as `<praefect_postgresql_password>`. -[Gitaly](../gitaly/index.md) server node requirements are dependent on data, -specifically the number of projects and those projects' sizes. It's recommended -that a Gitaly server node stores no more than 5 TB of data. Depending on your -repository storage requirements, you may require additional Gitaly server nodes. +This is how this would work with a Omnibus GitLab PostgreSQL setup: + +1. SSH in to the Praefect PostgreSQL node. +1. Connect to the PostgreSQL server with administrative access. + The `gitlab-psql` user should be used here for this as it's added by default in Omnibus. + The database `template1` is used because it is created by default on all PostgreSQL servers. + + ```shell + /opt/gitlab/embedded/bin/psql -U gitlab-psql -d template1 -h POSTGRESQL_SERVER_ADDRESS + ``` + +1. Create the new user `praefect`, replacing `<praefect_postgresql_password>`: + + ```shell + CREATE ROLE praefect WITH LOGIN CREATEDB PASSWORD <praefect_postgresql_password>; + ``` + +1. Reconnect to the PostgreSQL server, this time as the `praefect` user: + + ```shell + /opt/gitlab/embedded/bin/psql -U praefect -d template1 -h POSTGRESQL_SERVER_ADDRESS + ``` + +1. Create a new database `praefect_production`: + + ```shell + CREATE DATABASE praefect_production WITH ENCODING=UTF8; + ``` + +<div align="right"> + <a type="button" class="btn btn-default" href="#setup-components"> + Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> + </a> +</div> + +### Configure Praefect + +Praefect is the router and transaction manager for Gitaly Cluster and all connections to Gitaly go through +it. This section details how to configure it. + +Praefect requires several secret tokens to secure communications across the Cluster: + +- `<praefect_external_token>`: Used for repositories hosted on your Gitaly cluster and can only be accessed by Gitaly clients that carry this token. +- `<praefect_internal_token>`: Used for replication traffic inside your Gitaly cluster. This is distinct from `praefect_external_token` because Gitaly clients must not be able to access internal nodes of the Praefect cluster directly; that could lead to data loss. +- `<praefect_postgresql_password>`: The Praefect PostgreSQL password defined in the previous section is also required as part of this setup. + +Gitaly Cluster nodes are configured in Praefect via a `virtual storage`. Each storage contains +the details of each Gitaly node that makes up the cluster. Each storage is also given a name +and this name is used in several areas of the config. In this guide, the name of the storage will be +`default`. Also, this guide is geared towards new installs, if upgrading an existing environment +to use Gitaly Cluster, you may need to use a different name. +Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more info. + +The following IPs will be used as an example: + +- `10.6.0.131`: Praefect 1 +- `10.6.0.132`: Praefect 2 +- `10.6.0.133`: Praefect 3 + +To configure the Praefect nodes, on each one: + +1. SSH in to the Praefect server. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to follow _only_ installation steps 1 and 2 + on the page. +1. Edit the `/etc/gitlab/gitlab.rb` file to configure Praefect: + + ```ruby + # Avoid running unnecessary services on the Gitaly server + postgresql['enable'] = false + redis['enable'] = false + nginx['enable'] = false + puma['enable'] = false + unicorn['enable'] = false + sidekiq['enable'] = false + gitlab_workhorse['enable'] = false + grafana['enable'] = false + + # If you run a separate monitoring node you can disable these services + alertmanager['enable'] = false + prometheus['enable'] = false + + # Praefect Configuration + praefect['enable'] = true + praefect['listen_addr'] = '0.0.0.0:2305' + + gitlab_rails['rake_cache_clear'] = false + gitlab_rails['auto_migrate'] = false + + # Configure the Consul agent + consul['enable'] = true + ## Enable service discovery for Prometheus + consul['monitoring_service_discovery'] = true + + # START user configuration + # Please set the real values as explained in Required Information section + # + + # Praefect External Token + # This is needed by clients outside the cluster (like GitLab Shell) to communicate with the Praefect cluster + praefect['auth_token'] = '<praefect_external_token>' + + # Praefect Database Settings + praefect['database_host'] = '10.6.0.141' + praefect['database_port'] = 5432 + # `no_proxy` settings must always be a direct connection for caching + praefect['database_host_no_proxy'] = '10.6.0.141' + praefect['database_port_no_proxy'] = 5432 + praefect['database_dbname'] = 'praefect_production' + praefect['database_user'] = 'praefect' + praefect['database_password'] = '<praefect_postgresql_password>' + + # Praefect Virtual Storage config + # Name of storage hash must match storage name in git_data_dirs on GitLab + # server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1') + praefect['virtual_storages'] = { + 'default' => { + 'gitaly-1' => { + 'address' => 'tcp://10.6.0.91:8075', + 'token' => '<praefect_internal_token>', + 'primary' => true + }, + 'gitaly-2' => { + 'address' => 'tcp://10.6.0.92:8075', + 'token' => '<praefect_internal_token>' + }, + 'gitaly-3' => { + 'address' => 'tcp://10.6.0.93:8075', + 'token' => '<praefect_internal_token>' + }, + } + } + + # Set the network addresses that the exporters will listen on for monitoring + node_exporter['listen_address'] = '0.0.0.0:9100' + praefect['prometheus_listen_addr'] = '0.0.0.0:9652' + + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + # + # END user configuration + ``` + + 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and + then replace the file of the same name on this server. If that file isn't on + this server, add the file from your Consul server to this server. + + 1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +### Configure Gitaly + +The [Gitaly](../gitaly/index.md) server nodes that make up the cluster have +requirements that are dependent on data, specifically the number of projects +and those projects' sizes. It's recommended that a Gitaly Cluster stores +no more than 5 TB of data on each node. Depending on your +repository storage requirements, you may require additional Gitaly Clusters. Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs @@ -993,36 +1315,21 @@ adjusted to greater or lesser values depending on the scale of your environment's workload. If you're running the environment on a Cloud provider, refer to their documentation about how to configure IOPS correctly. -Be sure to note the following items: - -- The GitLab Rails application shards repositories into - [repository storage paths](../repository_storage_paths.md). -- A Gitaly server can host one or more storage paths. -- A GitLab server can use one or more Gitaly server nodes. -- Gitaly addresses must be specified to be correctly resolvable for all Gitaly - clients. -- Gitaly servers must not be exposed to the public internet, as Gitaly's network - traffic is unencrypted by default. The use of a firewall is highly recommended - to restrict access to the Gitaly server. Another option is to - [use TLS](#gitaly-tls-support). - -NOTE: -The token referred to throughout the Gitaly documentation is an arbitrary -password selected by the administrator. This token is unrelated to tokens -created for the GitLab API or other similar web API tokens. +Gitaly servers must not be exposed to the public internet, as Gitaly's network +traffic is unencrypted by default. The use of a firewall is highly recommended +to restrict access to the Gitaly server. Another option is to +[use TLS](#gitaly-cluster-tls-support). -This section describes how to configure two Gitaly servers, with the following -IPs and domain names: +For configuring Gitaly you should note the following: -- `10.6.0.51`: Gitaly 1 (`gitaly1.internal`) -- `10.6.0.52`: Gitaly 2 (`gitaly2.internal`) +- `git_data_dirs` should be configured to reflect the storage path for the specific Gitaly node +- `auth_token` should be the same as `praefect_internal_token` -Assumptions about your servers include having the secret token be `gitalysecret`, -and that your GitLab installation has three repository storages: +The following IPs will be used as an example: -- `default` on Gitaly 1 -- `storage1` on Gitaly 1 -- `storage2` on Gitaly 2 +- `10.6.0.91`: Gitaly 1 +- `10.6.0.92`: Gitaly 2 +- `10.6.0.93`: Gitaly 3 On each node: @@ -1032,21 +1339,9 @@ On each node: 1. Edit the Gitaly server node's `/etc/gitlab/gitlab.rb` file to configure storage paths, enable the network listener, and to configure the token: - <!-- - updates to following example must also be made at - https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab - --> - ```ruby # /etc/gitlab/gitlab.rb - # Gitaly and GitLab use two shared secrets for authentication, one to authenticate gRPC requests - # to Gitaly, and a second for authentication callbacks from GitLab-Shell to the GitLab internal API. - # The following two values must be the same as their respective values - # of the GitLab Rails application setup - gitaly['auth_token'] = 'gitalysecret' - gitlab_shell['secret_token'] = 'shellsecret' - # Avoid running unnecessary services on the Gitaly server postgresql['enable'] = false redis['enable'] = false @@ -1056,7 +1351,6 @@ On each node: sidekiq['enable'] = false gitlab_workhorse['enable'] = false grafana['enable'] = false - gitlab_exporter['enable'] = false # If you run a separate monitoring node you can disable these services alertmanager['enable'] = false @@ -1077,101 +1371,86 @@ On each node: # Comment out following line if you only want to support TLS connections gitaly['listen_addr'] = "0.0.0.0:8075" - ## Enable service discovery for Prometheus - consul['enable'] = true - consul['monitoring_service_discovery'] = true - - # Set the network addresses that the exporters will listen on for monitoring - gitaly['prometheus_listen_addr'] = "0.0.0.0:9236" - node_exporter['listen_address'] = '0.0.0.0:9100' - gitlab_rails['prometheus_address'] = '10.6.0.81:9090' - - ## The IPs of the Consul server nodes - ## You can also use FQDNs and intermix them with IPs - consul['configuration'] = { - retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), - } + # Gitaly Auth Token + # Should be the same as praefect_internal_token + gitaly['auth_token'] = '<praefect_internal_token>' ``` 1. Append the following to `/etc/gitlab/gitlab.rb` for each respective server: - - On `gitaly1.internal`: + - On Gitaly node 1: ```ruby git_data_dirs({ - 'default' => { - 'path' => '/var/opt/gitlab/git-data' - }, - 'storage1' => { - 'path' => '/mnt/gitlab/git-data' - }, + "gitaly-1" => { + "path" => "/var/opt/gitlab/git-data" + } }) ``` - - On `gitaly2.internal`: + - On Gitaly node 2: ```ruby git_data_dirs({ - 'storage2' => { - 'path' => '/mnt/gitlab/git-data' - }, + "gitaly-2" => { + "path" => "/var/opt/gitlab/git-data" + } }) ``` - <!-- - updates to following example must also be made at - https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab - --> + - On Gitaly node 3: + + ```ruby + git_data_dirs({ + "gitaly-3" => { + "path" => "/var/opt/gitlab/git-data" + } + }) + ``` + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and + then replace the file of the same name on this server. If that file isn't on + this server, add the file from your Consul server to this server. 1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -1. Confirm that Gitaly can perform callbacks to the internal API: - ```shell - sudo /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml - ``` +### Gitaly Cluster TLS support -1. Verify the GitLab services are running: +Praefect supports TLS encryption. To communicate with a Praefect instance that listens +for secure connections, you must: - ```shell - sudo gitlab-ctl status - ``` +- Use a `tls://` URL scheme in the `gitaly_address` of the corresponding storage entry + in the GitLab configuration. +- Bring your own certificates because this isn't provided automatically. The certificate + corresponding to each Praefect server must be installed on that Praefect server. - The output should be similar to the following: +Additionally the certificate, or its certificate authority, must be installed on all Gitaly servers +and on all Praefect clients that communicate with it following the procedure described in +[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates) (and repeated below). - ```plaintext - run: consul: (pid 30339) 77006s; run: log: (pid 29878) 77020s - run: gitaly: (pid 30351) 77005s; run: log: (pid 29660) 77040s - run: logrotate: (pid 7760) 3213s; run: log: (pid 29782) 77032s - run: node-exporter: (pid 30378) 77004s; run: log: (pid 29812) 77026s - ``` +Note the following: -### Gitaly TLS support +- The certificate must specify the address you use to access the Praefect server. If + addressing the Praefect server by: -Gitaly supports TLS encryption. To be able to communicate -with a Gitaly instance that listens for secure connections you will need to use `tls://` URL -scheme in the `gitaly_address` of the corresponding storage entry in the GitLab configuration. + - Hostname, you can either use the Common Name field for this, or add it as a Subject + Alternative Name. + - IP address, you must add it as a Subject Alternative Name to the certificate. -You will need to bring your own certificates as this isn't provided automatically. -The certificate, or its certificate authority, must be installed on all Gitaly -nodes (including the Gitaly node using the certificate) and on all client nodes -that communicate with it following the procedure described in -[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). +- You can configure Praefect servers with both an unencrypted listening address + `listen_addr` and an encrypted listening address `tls_listen_addr` at the same time. + This allows you to do a gradual transition from unencrypted to encrypted traffic, if + necessary. -NOTE: -The self-signed certificate must specify the address you use to access the -Gitaly server. If you are addressing the Gitaly server by a hostname, you can -either use the Common Name field for this, or add it as a Subject Alternative -Name. If you are addressing the Gitaly server by its IP address, you must add it -as a Subject Alternative Name to the certificate. -[gRPC does not support using an IP address as Common Name in a certificate](https://github.com/grpc/grpc/issues/2691). +- The Internal Load Balancer will also access to the certificates and need to be configured + to allow for TLS passthrough. + Refer to the load balancers documentation on how to configure this. -It's possible to configure Gitaly servers with both an unencrypted listening -address (`listen_addr`) and an encrypted listening address (`tls_listen_addr`) -at the same time. This allows you to do a gradual transition from unencrypted to -encrypted traffic, if necessary. +To configure Praefect with TLS: -To configure Gitaly with TLS: +1. Create certificates for Praefect servers. -1. Create the `/etc/gitlab/ssl` directory and copy your key and certificate there: +1. On the Praefect servers, create the `/etc/gitlab/ssl` directory and copy your key + and certificate there: ```shell sudo mkdir -p /etc/gitlab/ssl @@ -1180,27 +1459,35 @@ To configure Gitaly with TLS: sudo chmod 644 key.pem cert.pem ``` -1. Copy the cert to `/etc/gitlab/trusted-certs` so Gitaly will trust the cert when - calling into itself: +1. Edit `/etc/gitlab/gitlab.rb` and add: - ```shell - sudo cp /etc/gitlab/ssl/cert.pem /etc/gitlab/trusted-certs/ + ```ruby + praefect['tls_listen_addr'] = "0.0.0.0:3305" + praefect['certificate_path'] = "/etc/gitlab/ssl/cert.pem" + praefect['key_path'] = "/etc/gitlab/ssl/key.pem" ``` -1. Edit `/etc/gitlab/gitlab.rb` and add: +1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +1. On the Praefect clients (including each Gitaly server), copy the certificates, + or their certificate authority, into `/etc/gitlab/trusted-certs`: + + ```shell + sudo cp cert.pem /etc/gitlab/trusted-certs/ + ``` - <!-- - updates to following example must also be made at - https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab - --> +1. On the Praefect clients (except Gitaly servers), edit `git_data_dirs` in + `/etc/gitlab/gitlab.rb` as follows: ```ruby - gitaly['tls_listen_addr'] = "0.0.0.0:9999" - gitaly['certificate_path'] = "/etc/gitlab/ssl/cert.pem" - gitaly['key_path'] = "/etc/gitlab/ssl/key.pem" + git_data_dirs({ + "default" => { + "gitaly_address" => 'tls://LOAD_BALANCER_SERVER_ADDRESS:2305', + "gitaly_token" => 'PRAEFECT_EXTERNAL_TOKEN' + } + }) ``` -1. Delete `gitaly['listen_addr']` to allow only encrypted connections. 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). <div align="right"> @@ -1269,17 +1556,20 @@ To configure the Sidekiq nodes, one each one: ### Gitaly ### ####################################### + # git_data_dirs get configured for the Praefect virtual storage + # Address is Internal Load Balancer for Praefect + # Token is praefect_external_token git_data_dirs({ - 'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, - 'storage1' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, - 'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' }, + "default" => { + "gitaly_address" => "tcp://10.6.0.40:2305", # internal load balancer IP + "gitaly_token" => '<praefect_external_token>' + } }) - gitlab_rails['gitaly_token'] = 'YOUR_TOKEN' ####################################### ### Postgres ### ####################################### - gitlab_rails['db_host'] = '10.6.0.20' # internal load balancer IP + gitlab_rails['db_host'] = '10.6.0.40' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '<postgresql_user_password>' gitlab_rails['db_adapter'] = 'postgresql' @@ -1319,13 +1609,13 @@ To configure the Sidekiq nodes, one each one: 'google_project' => '<gcp-project-name>', 'google_json_key_location' => '<path-to-gcp-service-account-key>' } - gitlab_rails['object_store']['objects']['artifacts']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['lfs']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['uploads']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-bucket-name>" + gitlab_rails['object_store']['objects']['artifacts']['bucket'] = "<gcp-artifacts-bucket-name>" + gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = "<gcp-external-diffs-bucket-name>" + gitlab_rails['object_store']['objects']['lfs']['bucket'] = "<gcp-lfs-bucket-name>" + gitlab_rails['object_store']['objects']['uploads']['bucket'] = "<gcp-uploads-bucket-name>" + gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-packages-bucket-name>" + gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>" + gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>" ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -1397,17 +1687,14 @@ On each node perform the following: ```ruby external_url 'https://gitlab.example.com' - # Gitaly and GitLab use two shared secrets for authentication, one to authenticate gRPC requests - # to Gitaly, and a second for authentication callbacks from GitLab-Shell to the GitLab internal API. - # The following two values must be the same as their respective values - # of the Gitaly setup - gitlab_rails['gitaly_token'] = 'gitalysecret' - gitlab_shell['secret_token'] = 'shellsecret' - + # git_data_dirs get configured for the Praefect virtual storage + # Address is Interal Load Balancer for Praefect + # Token is praefect_external_token git_data_dirs({ - 'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, - 'storage1' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, - 'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' }, + "default" => { + "gitaly_address" => "tcp://10.6.0.40:2305", # internal load balancer IP + "gitaly_token" => '<praefect_external_token>' + } }) ## Disable components that will not be on the GitLab application server @@ -1471,13 +1758,13 @@ On each node perform the following: 'google_project' => '<gcp-project-name>', 'google_json_key_location' => '<path-to-gcp-service-account-key>' } - gitlab_rails['object_store']['objects']['artifacts']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['lfs']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['uploads']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-bucket-name>" - gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-bucket-name>" + gitlab_rails['object_store']['objects']['artifacts']['bucket'] = "<gcp-artifacts-bucket-name>" + gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = "<gcp-external-diffs-bucket-name>" + gitlab_rails['object_store']['objects']['lfs']['bucket'] = "<gcp-lfs-bucket-name>" + gitlab_rails['object_store']['objects']['uploads']['bucket'] = "<gcp-uploads-bucket-name>" + gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-packages-bucket-name>" + gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>" + gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>" ## Uncomment and edit the following options if you have set up NFS ## @@ -1495,14 +1782,15 @@ On each node perform the following: #registry['gid'] = 9002 ``` -1. If you're using [Gitaly with TLS support](#gitaly-tls-support), make sure the +1. If you're using [Gitaly with TLS support](#gitaly-cluster-tls-support), make sure the `git_data_dirs` entry is configured with `tls` instead of `tcp`: ```ruby git_data_dirs({ - 'default' => { 'gitaly_address' => 'tls://gitaly1.internal:9999' }, - 'storage1' => { 'gitaly_address' => 'tls://gitaly1.internal:9999' }, - 'storage2' => { 'gitaly_address' => 'tls://gitaly2.internal:9999' }, + "default" => { + "gitaly_address" => "tls://10.6.0.40:2305", # internal load balancer IP + "gitaly_token" => '<praefect_external_token>' + } }) ``` @@ -1694,20 +1982,9 @@ on what features you intend to use: | [Terraform state files](../terraform_state.md#using-object-storage) | Yes | Using separate buckets for each data type is the recommended approach for GitLab. - -A limitation of our configuration is that each use of object storage is separately configured. -[We have an issue for improving this](https://gitlab.com/gitlab-org/gitlab/-/issues/23345) -and easily using one bucket with separate folders is one improvement that this might bring. - -There is at least one specific issue with using the same bucket: -when GitLab is deployed with the Helm chart restore from backup -[will not properly function](https://docs.gitlab.com/charts/advanced/external-object-storage/#lfs-artifacts-uploads-packages-external-diffs-pseudonymizer) -unless separate buckets are used. - -One risk of using a single bucket would be if your organization decided to -migrate GitLab to the Helm deployment in the future. GitLab would run, but the situation with -backups might not be realized until the organization had a critical requirement for the backups to -work. +This ensures there are no collisions across the various types of data GitLab stores. +There are plans to [enable the use of a single bucket](https://gitlab.com/gitlab-org/gitlab/-/issues/292958) +in the future. <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -1715,7 +1992,7 @@ work. </a> </div> -## Configure Advanced Search **(PREMIUM SELF)** +## Configure Advanced Search You can leverage Elasticsearch and [enable Advanced Search](../../integration/elasticsearch.md) for faster, more advanced code search across your entire GitLab instance. @@ -1750,13 +2027,3 @@ as soon as possible. Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> </a> </div> - -## Troubleshooting - -See the [troubleshooting documentation](troubleshooting.md). - -<div align="right"> - <a type="button" class="btn btn-default" href="#setup-components"> - Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> - </a> -</div> diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index b149cbc6e9d..d5d4f8ac0cb 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -5,7 +5,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Reference architectures +# Reference architectures **(FREE SELF)** You can set up GitLab on a single server or scale it up to serve many users. This page details the recommended Reference Architectures that were built and @@ -92,7 +92,7 @@ Also, not implementing extra servers for GitLab doesn't necessarily mean that yo more downtime. Depending on your needs and experience level, single servers can have more actual perceived uptime for your users. -### Automated backups **(FREE SELF)** +### Automated backups > - Level of complexity: **Low** > - Required domain knowledge: PostgreSQL, GitLab configurations, Git diff --git a/doc/administration/reference_architectures/troubleshooting.md b/doc/administration/reference_architectures/troubleshooting.md index cab45a99ee4..60a6ee0132f 100644 --- a/doc/administration/reference_architectures/troubleshooting.md +++ b/doc/administration/reference_architectures/troubleshooting.md @@ -257,7 +257,7 @@ sudo GRPC_TRACE=all GRPC_VERBOSITY=DEBUG gitlab-rake gitlab:gitaly:check ### Observing `gitaly-ruby` traffic -[`gitaly-ruby`](../gitaly/index.md#gitaly-ruby) is an internal implementation detail of Gitaly, +[`gitaly-ruby`](../gitaly/configure_gitaly.md#gitaly-ruby) is an internal implementation detail of Gitaly, so, there's not that much visibility into what goes on inside `gitaly-ruby` processes. @@ -301,7 +301,7 @@ Confirm the following are all true: - Creating a new project and [initializing it with a README](../../user/project/working_with_projects.md#blank-projects) successfully creates the project but doesn't create the README. - When [tailing the logs](https://docs.gitlab.com/omnibus/settings/logs.html#tail-logs-in-a-console-on-the-server) on an app node and reproducing the error, you get `401` errors - when reaching the `/api/v4/internal/allowed` endpoint: + when reaching the [`/api/v4/internal/allowed`](../../development/internal_api.md) endpoint: ```shell # api_json.log diff --git a/doc/administration/repository_checks.md b/doc/administration/repository_checks.md index 404a7bd77c1..d603e5d8c92 100644 --- a/doc/administration/repository_checks.md +++ b/doc/administration/repository_checks.md @@ -1,11 +1,11 @@ --- stage: Create -group: Editor +group: Gitaly info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" type: reference --- -# Repository checks +# Repository checks **(FREE)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/3232) in GitLab 8.7. @@ -44,3 +44,23 @@ in the [`repocheck.log` file](logs.md#repochecklog) on disk: If the periodic repository check causes false alarms, you can clear all repository check states by going to **Admin Area > Settings > Repository** (`/admin/application_settings/repository`) and clicking **Clear all repository checks**. + +## Run a check manually + +[`git fsck`](https://git-scm.com/docs/git-fsck) is a read-only check that you can run +manually against the repository on the [Gitaly server](gitaly/index.md). + +- For Omnibus GitLab installations, repositories are stored by default in + `/var/opt/gitlab/git-data/repositories`. +- [Identify the subdirectory that contains the repository](repository_storage_types.md#from-project-name-to-hashed-path) + that you need to check. + +To run a check (for example): + +```shell +sudo /opt/gitlab/embedded/bin/git -C /var/opt/gitlab/git-data/repositories/@hashed/0b/91/0b91...f9.git fsck +``` + +You can also run [Rake tasks](raketasks/check.md#repository-integrity) for checking Git +repositories, which can be used to run `git fsck` against all repositories and generate +repository checksums, as a way to compare repositories on different servers. diff --git a/doc/administration/repository_storage_paths.md b/doc/administration/repository_storage_paths.md index 5884a8b3283..e342ef8c04b 100644 --- a/doc/administration/repository_storage_paths.md +++ b/doc/administration/repository_storage_paths.md @@ -13,7 +13,9 @@ GitLab stores [repositories](../user/project/repository/index.md) on repository storage is either: - A `gitaly_address`, which points to a [Gitaly node](gitaly/index.md). -- A `path`, which points directly a directory where the repository is stored. +- A `path`, which points directly a directory where the repositories are stored. This method is + deprecated and [scheduled to be removed](https://gitlab.com/gitlab-org/gitaly/-/issues/1690) in + GitLab 14.0. GitLab allows you to define multiple repository storages to distribute the storage load between several mount points. For example: @@ -23,7 +25,7 @@ several mount points. For example: ```ruby git_data_dirs({ 'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, - 'storage1' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' }, + 'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' }, }) ``` @@ -31,18 +33,23 @@ several mount points. For example: ```plaintext default: - path: /mnt/git-storage-1 + gitaly_address: tcp://gitaly1.example:8075 storage2: - path: /mnt/git-storage-2 + gitaly_address: tcp://gitaly2.example:8075 ``` -For more information on +For more information on: - Configuring Gitaly, see [Configure Gitaly](gitaly/index.md#configure-gitaly). - Configuring direct repository access, see the following section below. ## Configure repository storage paths +WARNING: +The following information is for configuring GitLab to directly access repositories. This +configuration option is deprecated in favor of using [Gitaly](gitaly/index.md) and is scheduled to +[be removed in GitLab 14.0](https://gitlab.com/gitlab-org/gitaly/-/issues/1690). + To configure repository storage paths: 1. Edit the necessary configuration files: @@ -76,7 +83,7 @@ For [backups](../raketasks/backup_restore.md) to work correctly: Omnibus GitLab takes care of these issues for you, but for source installations you should be extra careful. -While restoring a backup, the current contents of `/home/git/repositories` are moved to +While restoring a backup, the current contents of `/home/git/repositories` are moved to `/home/git/repositories.old`. If `/home/git/repositories` is a mount point, then `mv` would be moving things between mount points, and problems can occur. @@ -96,10 +103,6 @@ For compatibility reasons `gitlab.yml` has a different structure than Omnibus Gi example. Then Omnibus GitLab creates a `repositories` directory under that path to use with `gitlab.yml`. -NOTE: -This example uses NFS. We do not recommend using EFS for storage as it may impact GitLab performance. -Read the [relevant documentation](nfs.md#avoid-using-awss-elastic-file-system-efs) for more details. - **For installations from source** 1. Edit `gitlab.yml` and add the storage paths: @@ -111,10 +114,10 @@ Read the [relevant documentation](nfs.md#avoid-using-awss-elastic-file-system-ef storages: # You must have at least a 'default' repository storage path. default: path: /home/git/repositories - nfs_1: - path: /mnt/nfs1/repositories - nfs_2: - path: /mnt/nfs2/repositories + storage1: + path: /mnt/storage1/repositories + storage2: + path: /mnt/storage2/repositories ``` 1. [Restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. @@ -126,8 +129,8 @@ Edit `/etc/gitlab/gitlab.rb` by appending the rest of the paths to the default o ```ruby git_data_dirs({ "default" => { "path" => "/var/opt/gitlab/git-data" }, - "nfs_1" => { "path" => "/mnt/nfs1/git-data" }, - "nfs_2" => { "path" => "/mnt/nfs2/git-data" } + "storage1" => { "path" => "/mnt/storage1/git-data" }, + "storage2" => { "path" => "/mnt/storage2/git-data" } }) ``` @@ -153,6 +156,5 @@ often it is chosen. That is, `(storage weight) / (sum of all weights) * 100 = ch ## Move repositories -To move a repository to a different repository path, use the -[Project repository storage moves](../api/project_repository_storage_moves.md) API. Use +To move a repository to a different repository path, use the same process as [migrating existing repositories to Gitaly Cluster](gitaly/praefect.md#migrate-existing-repositories-to-gitaly-cluster). diff --git a/doc/administration/repository_storage_types.md b/doc/administration/repository_storage_types.md index a5c323be4ce..29e31fcb6ef 100644 --- a/doc/administration/repository_storage_types.md +++ b/doc/administration/repository_storage_types.md @@ -7,51 +7,53 @@ type: reference, howto # Repository storage types **(FREE SELF)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/28283) in GitLab 10.0. -> - Hashed storage became the default for new installations in GitLab 12.0 -> - Hashed storage is enabled by default for new and renamed projects in GitLab 13.0. +GitLab can be configured to use one or multiple repository storages. These storages can be: -GitLab can be configured to use one or multiple repository storage paths/shard -locations that can be: +- Accessed via [Gitaly](gitaly/index.md), optionally on + [its own server](gitaly/configure_gitaly.md#run-gitaly-on-its-own-server). +- Mounted to the local disk. This [method](repository_storage_paths.md#configure-repository-storage-paths) + is deprecated and [scheduled to be removed](https://gitlab.com/groups/gitlab-org/-/epics/2320) in + GitLab 14.0. +- Exposed as an NFS shared volume. This method is deprecated and + [scheduled to be removed](https://gitlab.com/groups/gitlab-org/-/epics/3371) in GitLab 14.0. -- Mounted to the local disk -- Exposed as an NFS shared volume -- Accessed via [Gitaly](gitaly/index.md) on its own machine. +In GitLab: -In GitLab, this is configured in `/etc/gitlab/gitlab.rb` by the `git_data_dirs({})` -configuration hash. The storage layouts discussed here apply to any shard -defined in it. +- Repository storages are configured in: + - `/etc/gitlab/gitlab.rb` by the `git_data_dirs({})` configuration hash for Omnibus GitLab + installations. + - `gitlab.yml` by the `repositories.storages` key for installations from source. +- The `default` repository storage is available in any installations that haven't customized it. By + default, it points to a Gitaly node. -The `default` repository shard that is available in any installations -that haven't customized it, points to the local folder: `/var/opt/gitlab/git-data`. -Anything discussed below is expected to be part of that folder. +The repository storage types documented here apply to any repository storage defined in +`git_data_dirs({})` or `repositories.storages`. ## Hashed storage -NOTE: -In GitLab 13.0, hashed storage is enabled by default and the legacy storage is -deprecated. Support for legacy storage is scheduled to be removed in GitLab 14.0. -If you haven't migrated yet, check the -[migration instructions](raketasks/storage.md#migrate-to-hashed-storage). -The option to choose between hashed and legacy storage in the admin area has -been disabled. - -Hashed storage is the storage behavior we rolled out with 10.0. Instead -of coupling project URL and the folder structure where the repository is -stored on disk, we couple a hash based on the project's ID. This makes -the folder structure immutable, and therefore eliminates any requirement to -synchronize state from URLs to disk structure. This means that renaming a group, -user, or project costs only the database transaction, and takes effect -immediately. - -The hash also helps spread the repositories more evenly on the disk. The -top-level directory contains fewer folders than the total number of top-level -namespaces. - -The hash format is based on the hexadecimal representation of SHA256: -`SHA256(project.id)`. The top-level folder uses the first 2 characters, followed -by another folder with the next 2 characters. They are both stored in a special -`@hashed` folder, to be able to co-exist with existing Legacy Storage projects: +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/28283) in GitLab 10.0. +> - Made the default for new installations in GitLab 12.0. +> - Enabled by default for new and renamed projects in GitLab 13.0. + +Hashed storage stores projects on disk in a location based on a hash of the project's ID. Hashed +storage is different to [legacy storage](#legacy-storage) where a project is stored based on: + +- The project's URL. +- The folder structure where the repository is stored on disk. + +This makes the folder structure immutable and eliminates the need to synchronize state from URLs to +disk structure. This means that renaming a group, user, or project: + +- Costs only the database transaction. +- Takes effect immediately. + +The hash also helps spread the repositories more evenly on the disk. The top-level directory +contains fewer folders than the total number of top-level namespaces. + +The hash format is based on the hexadecimal representation of a SHA256, calculated with +`SHA256(project.id)`. The top-level folder uses the first two characters, followed by another folder +with the next two characters. They are both stored in a special `@hashed` folder so they can +co-exist with existing legacy storage projects. For example: ```ruby # Project's repository: @@ -61,53 +63,59 @@ by another folder with the next 2 characters. They are both stored in a special "@hashed/#{hash[0..1]}/#{hash[2..3]}/#{hash}.wiki.git" ``` -### Translating hashed storage paths +### Translate hashed storage paths + +Troubleshooting problems with the Git repositories, adding hooks, and other tasks requires you +translate between the human-readable project name and the hashed storage path. You can translate: -Troubleshooting problems with the Git repositories, adding hooks, and other -tasks requires you translate between the human readable project name -and the hashed storage path. +- From a [project's name to its hashed path](#from-project-name-to-hashed-path). +- From a [hashed path to a project's name](#from-hashed-path-to-project-name). #### From project name to hashed path -The hashed path is shown on the project's page in the [admin area](../user/admin_area/index.md#administering-projects). +Administrators can look up a project's hashed path from its name or ID using: + +- The [Admin area](../user/admin_area/index.md#administering-projects). +- A Rails console. + +To look up a project's hash path in the Admin Area: -To access the Projects page, go to **Admin Area > Overview > Projects** and then -open up the page for the project. +1. Go to the **Admin Area** (**{admin}**). +1. Go to **Overview > Projects** and select the project. -The "Gitaly relative path" is shown there, for example: +The **Gitaly relative path** is displayed there and looks similar to: ```plaintext "@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git" ``` -This is the path under `/var/opt/gitlab/git-data/repositories/` on a -default Omnibus installation. +To look up a project's hash path using a Rails console: -In a [Rails console](operations/rails_console.md#starting-a-rails-console-session), -get this information using either the numeric project ID or the full path: +1. Start a [Rails console](operations/rails_console.md#starting-a-rails-console-session). +1. Run a command similar to this example (use either the project's ID or its name): -```ruby -Project.find(16).disk_path -Project.find_by_full_path('group/project').disk_path -``` + ```ruby + Project.find(16).disk_path + Project.find_by_full_path('group/project').disk_path + ``` #### From hashed path to project name -To translate from a hashed storage path to a project name: +Administrators can look up a project's name from its hashed storage path using a Rails console. To +look up a project's name from its hashed storage path: 1. Start a [Rails console](operations/rails_console.md#starting-a-rails-console-session). -1. Run the following: +1. Run a command similar to this example: -```ruby -ProjectRepository.find_by(disk_path: '@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9').project -``` + ```ruby + ProjectRepository.find_by(disk_path: '@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9').project + ``` -The quoted string in that command is the directory tree you can find on your -GitLab server. For example, on a default Omnibus installation this would be -`/var/opt/gitlab/git-data/repositories/@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git` +The quoted string in that command is the directory tree you can find on your GitLab server. For +example, on a default Omnibus installation this would be `/var/opt/gitlab/git-data/repositories/@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git` with `.git` from the end of the directory name removed. -The output includes the project ID and the project name: +The output includes the project ID and the project name. For example: ```plaintext => #<Project id:16 it/supportteam/ticketsystem> @@ -117,57 +125,61 @@ The output includes the project ID and the project name: > [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/1606) in GitLab 12.1. -WARNING: -Do not run `git prune` or `git gc` in pool repositories! This can -cause data loss in "real" repositories that depend on the pool in -question. +Object pools are repositories used to deduplicate forks of public and internal projects and +contain the objects from the source project. Using `objects/info/alternates`, the source project and +forks use the object pool for shared objects. For more information, see +[How Git object deduplication works in GitLab](../development/git_object_deduplication.md). -Forks of public and internal projects are deduplicated by creating a third repository, the -object pool, containing the objects from the source project. Using -`objects/info/alternates`, the source project and forks use the object pool for -shared objects. Objects are moved from the source project to the object pool -when housekeeping is run on the source project. +Objects are moved from the source project to the object pool when housekeeping is run on the source +project. Object pool repositories are stored similarly to regular repositories: ```ruby # object pool paths "@pools/#{hash[0..1]}/#{hash[2..3]}/#{hash}.git" ``` -### Hashed storage coverage migration - -Files stored in an S3-compatible endpoint do not have the downsides -mentioned earlier, if they are not prefixed with `#{namespace}/#{project_name}`. -This is true for CI Cache and LFS Objects. - -In the table below, you can find the coverage of the migration to the hashed storage. - -| Storable Object | Legacy storage | Hashed storage | S3 Compatible | GitLab Version | -| --------------- | -------------- | -------------- | ------------- | -------------- | -| Repository | Yes | Yes | - | 10.0 | -| Attachments | Yes | Yes | - | 10.2 | -| Avatars | Yes | No | - | - | -| Pages | Yes | No | - | - | -| Docker Registry | Yes | No | - | - | -| CI Build Logs | No | No | - | - | -| CI Artifacts | No | No | Yes | 9.4 / 10.6 | -| CI Cache | No | No | Yes | - | -| LFS Objects | Yes | Similar | Yes | 10.0 / 10.7 | -| Repository pools| No | Yes | - | 11.6 | +WARNING: +Do not run `git prune` or `git gc` in object pool repositories. This can cause data loss in the +regular repositories that depend on the object pool. + +### Object storage support + +This table shows which storable objects are storable in each storage type: + +| Storable object | Legacy storage | Hashed storage | S3 compatible | GitLab version | +|:-----------------|:---------------|:---------------|:--------------|:---------------| +| Repository | Yes | Yes | - | 10.0 | +| Attachments | Yes | Yes | - | 10.2 | +| Avatars | Yes | No | - | - | +| Pages | Yes | No | - | - | +| Docker Registry | Yes | No | - | - | +| CI/CD job logs | No | No | - | - | +| CI/CD artifacts | No | No | Yes | 9.4 / 10.6 | +| CI/CD cache | No | No | Yes | - | +| LFS objects | Yes | Similar | Yes | 10.0 / 10.7 | +| Repository pools | No | Yes | - | 11.6 | + +Files stored in an S3-compatible endpoint can have the same advantages as +[hashed storage](#hashed-storage), as long as they are not prefixed with +`#{namespace}/#{project_name}`. This is true for CI/CD cache and LFS objects. #### Avatars -Each file is stored in a folder with its `id` from the database. The filename is always `avatar.png` for user avatars. -When avatar is replaced, `Upload` model is destroyed and a new one takes place with different `id`. +Each file is stored in a directory that matches the `id` assigned to it in the database. The +filename is always `avatar.png` for user avatars. When an avatar is replaced, the `Upload` model is +destroyed and a new one takes place with a different `id`. + +#### CI/CD artifacts -#### CI artifacts +CI/CD artifacts are: -CI Artifacts are S3 compatible since **9.4** (GitLab Premium), and available in GitLab Free since -**10.6**. +- S3-compatible since GitLab 9.4, initially available in [GitLab Premium](https://about.gitlab.com/pricing/). +- Available in [GitLab Free](https://about.gitlab.com/pricing/) since GitLab 10.6. #### LFS objects [LFS Objects in GitLab](../topics/git/lfs/index.md) implement a similar -storage pattern using 2 chars, 2 level folders, following Git's own implementation: +storage pattern using two characters and two-level folders, following Git's own implementation: ```ruby "shared/lfs-objects/#{oid[0..1}/#{oid[2..3]}/#{oid[4..-1]}" @@ -176,40 +188,32 @@ storage pattern using 2 chars, 2 level folders, following Git's own implementati "shared/lfs-objects/89/09/029eb962194cfb326259411b22ae3f4a814b5be4f80651735aeef9f3229c" ``` -LFS objects are also [S3 compatible](lfs/index.md#storing-lfs-objects-in-remote-object-storage). +LFS objects are also [S3-compatible](lfs/index.md#storing-lfs-objects-in-remote-object-storage). ## Legacy storage WARNING: -In GitLab 13.0, hashed storage is enabled by default and the legacy storage is -deprecated. If you haven't migrated yet, check the -[migration instructions](raketasks/storage.md#migrate-to-hashed-storage). -Support for legacy storage is scheduled to be removed in GitLab 14.0. If you're on GitLab -13.0 and later, switching new projects to legacy storage is not possible. -The option to choose between hashed and legacy storage in the admin area has -been disabled. - -Legacy storage is the storage behavior prior to version 10.0. For historical -reasons, GitLab replicated the same mapping structure from the projects URLs: - -- Project's repository: `#{namespace}/#{project_name}.git` -- Project's wiki: `#{namespace}/#{project_name}.wiki.git` - -This structure enables you to migrate from existing solutions to GitLab, and -for Administrators to find where the repository is stored. - -This approach also has some drawbacks: - -Storage location concentrates a huge number of top-level namespaces. The -impact can be reduced by the introduction of -[multiple storage paths](repository_storage_paths.md). - -Because backups are a snapshot of the same URL mapping, if you try to recover a -very old backup, you need to verify whether any project has taken the place of -an old removed or renamed project sharing the same URL. This means that -`mygroup/myproject` from your backup may not be the same original project that -is at that same URL today. - -Any change in the URL needs to be reflected on disk (when groups / users or -projects are renamed). This can add a lot of load in big installations, -especially if using any type of network based file system. +In GitLab 13.0, legacy storage is deprecated. If you haven't migrated to hashed storage yet, check +the [migration instructions](raketasks/storage.md#migrate-to-hashed-storage). Support for legacy +storage is [scheduled to be removed](https://gitlab.com/gitlab-org/gitaly/-/issues/1690) in GitLab +14.0. In GitLab 13.0 and later, switching new projects to legacy storage is not possible. The +option to choose between hashed and legacy storage in the Admin Area is disabled. + +Legacy storage was the storage behavior prior to version GitLab 10.0. For historical reasons, +GitLab replicated the same mapping structure from the projects URLs: + +- Project's repository: `#{namespace}/#{project_name}.git`. +- Project's wiki: `#{namespace}/#{project_name}.wiki.git`. + +This structure enabled you to migrate from existing solutions to GitLab, and for Administrators to +find where the repository was stored. This approach also had some drawbacks: + +- Storage location concentrated a large number of top-level namespaces. The impact could be + reduced by [multiple repository storage paths](repository_storage_paths.md). +- Because backups were a snapshot of the same URL mapping, if you tried to recover a very old + backup, you needed to verify whether any project had taken the place of an old removed or renamed + project sharing the same URL. This meant that `mygroup/myproject` from your backup may not have + been the same original project that was at that same URL today. +- Any change in the URL needed to be reflected on disk (when groups, users, or projects were + renamed. This could add a lot of load in big installations, especially if using any type of + network-based file system. diff --git a/doc/administration/server_hooks.md b/doc/administration/server_hooks.md index 93b899d5146..671da505e56 100644 --- a/doc/administration/server_hooks.md +++ b/doc/administration/server_hooks.md @@ -36,7 +36,7 @@ Note the following about server hooks: - [Webhooks](../user/project/integrations/webhooks.md). - [GitLab CI/CD](../ci/README.md). - [Push Rules](../push_rules/push_rules.md), for a user-configurable Git hook - interface. **(STARTER)** + interface. - Server hooks aren't replicated to [Geo](geo/index.md) secondary nodes. ## Create a server hook for a repository @@ -51,10 +51,10 @@ repository directory might not exactly match the instructions below. In that cas Follow the steps below to set up a server-side hook for a repository: -1. Navigate to **Admin area > Projects** and click on the project you want to add a server hook to. +1. Go to **Admin area > Projects** and select the project you want to add a server hook to. 1. Locate the **Gitaly relative path** on the page that appears. This is where the server hook must be implemented. For information on interpreting the relative path, see - [Translating hashed storage paths](repository_storage_types.md#translating-hashed-storage-paths). + [Translate hashed storage paths](repository_storage_types.md#translate-hashed-storage-paths). 1. On the file system, create a new directory in this location called `custom_hooks`. 1. Inside the new `custom_hooks` directory, create a file with a name matching the hook type. For example, for a pre-receive hook the filename should be `pre-receive` with no extension. @@ -104,8 +104,6 @@ Now test the hook to check whether it is functioning properly. ## Chained hooks -> [Introduced](https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/93) in GitLab Shell 4.1.0 and GitLab 8.15. - Server hooks set [per project](#create-a-server-hook-for-a-repository) or [globally](#create-a-global-server-hook-for-all-repositories) can be executed in a chain. @@ -128,7 +126,7 @@ Any other names are ignored. Files in `.d` directories must be executable and not match the backup file pattern (`*~`). -For `<project>.git` you need to [translate](repository_storage_types.md#translating-hashed-storage-paths) +For `<project>.git` you need to [translate](repository_storage_types.md#translate-hashed-storage-paths) your project name into the hashed storage format that GitLab uses. ## Environment Variables @@ -158,8 +156,6 @@ them as they can change. ## Custom error messages -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/5073) in GitLab 8.10. - To have custom error messages appear in the GitLab UI when a commit is declined or an error occurs during the Git hook, your script should: diff --git a/doc/administration/static_objects_external_storage.md b/doc/administration/static_objects_external_storage.md index 0f3dab743e9..fcd2bbc035f 100644 --- a/doc/administration/static_objects_external_storage.md +++ b/doc/administration/static_objects_external_storage.md @@ -5,7 +5,7 @@ info: "To determine the technical writer assigned to the Stage/Group associated type: reference --- -# Static objects external storage +# Static objects external storage **(FREE)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/31025) in GitLab 12.3. @@ -16,7 +16,7 @@ from an external storage, such as a Content Delivery Network (CDN). To configure external storage for static objects: -1. Navigate to **Admin Area > Settings > Repository**. +1. Go to **Admin Area > Settings > Repository**. 1. Expand the **Repository static objects** section. 1. Enter the base URL and an arbitrary token. When you [set up external storage](#set-up-external-storage), use a script that sets these values as `ORIGIN_HOSTNAME` and `STORAGE_TOKEN`. diff --git a/doc/administration/terraform_state.md b/doc/administration/terraform_state.md index 6e5d6b274b6..0f3fdf4bb93 100644 --- a/doc/administration/terraform_state.md +++ b/doc/administration/terraform_state.md @@ -20,6 +20,33 @@ These locations can be configured using the options described below. Use [external object storage](https://docs.gitlab.com/charts/advanced/external-object-storage/#lfs-artifacts-uploads-packages-external-diffs-pseudonymizer-terraform-state-dependency-proxy) configuration for [GitLab Helm chart](https://docs.gitlab.com/charts/) installations. +## Disabling Terraform state + +To disable terraform state site-wide, follow the steps below. +A GitLab administrator may want to disable Terraform state to reduce diskspace or if Terraform is not used in your instance. +To do so, follow the steps below according to your installation's type. + +**In Omnibus installations:** + +1. Edit `/etc/gitlab/gitlab.rb` and add the following line: + + ```ruby + gitlab_rails['terraform_state_enabled'] = false + ``` + +1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +**In installations from source:** + +1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: + + ```yaml + terraform_state: + enabled: false + ``` + +1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. + ## Using local storage The default configuration uses local storage. To change the location where @@ -121,7 +148,7 @@ See [the available connection settings for different providers](object_storage.m remote_directory: "terraform" # The bucket name connection: provider: AWS # Only AWS supported at the moment - aws_access_key_id: AWS_ACESS_KEY_ID + aws_access_key_id: AWS_ACCESS_KEY_ID aws_secret_access_key: AWS_SECRET_ACCESS_KEY region: eu-central-1 ``` diff --git a/doc/administration/troubleshooting/debug.md b/doc/administration/troubleshooting/debug.md index a6073e34d58..5a8ee1c5c94 100644 --- a/doc/administration/troubleshooting/debug.md +++ b/doc/administration/troubleshooting/debug.md @@ -255,7 +255,7 @@ separate Rails process to debug the issue: ### GitLab: API is not accessible This often occurs when GitLab Shell attempts to request authorization via the -internal API (e.g., `http://localhost:8080/api/v4/internal/allowed`), and +[internal API](../../development/internal_api.md) (e.g., `http://localhost:8080/api/v4/internal/allowed`), and something in the check fails. There are many reasons why this may happen: 1. Timeout connecting to a database (e.g., PostgreSQL or Redis) @@ -271,8 +271,8 @@ strace -ttTfyyy -s 1024 -p <PID of unicorn worker> -o /tmp/unicorn.txt ``` If you cannot isolate which Unicorn worker is the issue, try to run `strace` -on all the Unicorn workers to see where the `/internal/allowed` endpoint gets -stuck: +on all the Unicorn workers to see where the +[`/internal/allowed`](../../development/internal_api.md) endpoint gets stuck: ```shell ps auwx | grep unicorn | awk '{ print " -p " $2}' | xargs strace -ttTfyyy -s 1024 -o /tmp/unicorn.txt diff --git a/doc/administration/troubleshooting/elasticsearch.md b/doc/administration/troubleshooting/elasticsearch.md index 0f60c43ef9e..606697b5247 100644 --- a/doc/administration/troubleshooting/elasticsearch.md +++ b/doc/administration/troubleshooting/elasticsearch.md @@ -36,7 +36,7 @@ The type of problem will determine what steps to take. The possible troubleshoot - Indexing. - Integration. - Performance. -- Background Migrations. +- Advanced Search Migrations. ### Search Results workflow @@ -148,7 +148,7 @@ graph TD; F7(Escalate to<br>GitLab support.) ``` -### Background Migrations workflow +### Advanced Search Migrations workflow ```mermaid graph TD; @@ -180,7 +180,7 @@ Most Elasticsearch troubleshooting can be broken down into 4 categories: - [Troubleshooting indexing](#troubleshooting-indexing) - [Troubleshooting integration](#troubleshooting-integration) - [Troubleshooting performance](#troubleshooting-performance) -- [Troubleshooting background migrations](#troubleshooting-background-migrations) +- [Troubleshooting Advanced Search migrations](#troubleshooting-advanced-search-migrations) Generally speaking, if it does not fall into those four categories, it is either: @@ -294,7 +294,7 @@ Beyond that, you will want to review the error. If it is: GitLab support. - An OS issue, you will want to reach out to your systems administrator. - A `Faraday::TimeoutError (execution expired)` error **and** you're using a proxy, - [set a custom `gitlab_rails['env']` environment variable, called `no_proxy`](https://docs.gitlab.com/omnibus/settings/environment-variables.html) + [set a custom `gitlab_rails['env']` environment variable, called `no_proxy`](https://docs.gitlab.com/omnibus/settings/environment-variables.html) with the IP address of your Elasticsearch host. ### Troubleshooting performance @@ -356,15 +356,16 @@ dig further into these. Feel free to reach out to GitLab support, but this is likely to be something a skilled Elasticsearch administrator has more experience with. -### Troubleshooting background migrations +### Troubleshooting Advanced Search migrations -Troubleshooting background migration failures can be difficult and may require contacting -an Elasticsearch administrator or GitLab Support. +Troubleshooting Advanced Search migration failures can be difficult and may +require contacting an Elasticsearch administrator or GitLab Support. -The best place to start while debugging issues with a background migration is the -[`elasticsearch.log` file](../logs.md#elasticsearchlog). Migrations will -print information while a migration is in progress and any errors encountered. -Apply fixes for any errors found in the log and retry the migration. +The best place to start while debugging issues with an Advanced Search +migration is the [`elasticsearch.log` file](../logs.md#elasticsearchlog). +Migrations will log information while a migration is in progress and any +errors encountered. Apply fixes for any errors found in the log and retry +the migration. If you still encounter issues after retrying the migration, reach out to GitLab support. diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index 55e707042ba..69eb639a8db 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -1045,6 +1045,67 @@ project = Project.find_by_full_path('<group/project>') Geo::RepositorySyncService.new(project).execute ``` +### Blob types newer than uploads/artifacts/LFS + +- `Packages::PackageFile` +- `Terraform::StateVersion` +- `MergeRequestDiff` + +`Packages::PackageFile` is used in the following examples, but things generally work the same for the other Blob types. + +#### The Replicator + +The main kinds of classes are Registry, Model, and Replicator. If you have an instance of one of these classes, you can get the others. The Registry and Model mostly manage PostgreSQL DB state. The Replicator knows how to replicate/verify (or it can call a service to do it): + +```ruby +model_record = Packages::PackageFile.last +model_record.replicator.registry.replicator.model_record # just showing that these methods exist +``` + +#### Replicate a package file, synchronously, given an ID + +```ruby +model_record = Packages::PackageFile.find(id) +model_record.replicator.send(:download) +``` + +#### Replicate a package file, synchronously, given a registry ID + +```ruby +registry = Geo::PackageFileRegistry.find(registry_id) +registry.replicator.send(:download) +``` + +### Repository types newer than project/wiki repositories + +- `SnippetRepository` +- `GroupWikiRepository` + +`SnippetRepository` is used in the examples below, but things generally work the same for the other Repository types. + +#### The Replicator + +The main kinds of classes are Registry, Model, and Replicator. If you have an instance of one of these classes, you can get the others. The Registry and Model mostly manage PostgreSQL DB state. The Replicator knows how to replicate/verify (or it can call a service to do it). + +```ruby +model_record = SnippetRepository.last +model_record.replicator.registry.replicator.model_record # just showing that these methods exist +``` + +#### Replicate a snippet repository, synchronously, given an ID + +```ruby +model_record = SnippetRepository.find(id) +model_record.replicator.send(:sync_repository) +``` + +#### Replicate a snippet repository, synchronously, given a registry ID + +```ruby +registry = Geo::SnippetRepositoryRegistry.find(registry_id) +registry.replicator.send(:sync_repository) +``` + ### Generate usage ping #### Generate or get the cached usage ping diff --git a/doc/administration/troubleshooting/log_parsing.md b/doc/administration/troubleshooting/log_parsing.md index 25300d036ed..2900ce58940 100644 --- a/doc/administration/troubleshooting/log_parsing.md +++ b/doc/administration/troubleshooting/log_parsing.md @@ -41,6 +41,20 @@ jq -cR 'fromjson?' file.json | jq <COMMAND> By default `jq` will error out when it encounters a line that is not valid JSON. This skips over all invalid lines and parses the rest. +#### Print a JSON log's time range + +```shell +cat log.json | (head -1; tail -1) | jq '.time' +``` + +Use `zcat` if the file has been rotated and compressed: + +```shell +zcat @400000006026b71d1a7af804.s | (head -1; tail -1) | jq '.time' + +zcat some_json.log.25.gz | (head -1; tail -1) | jq '.time' +``` + ### Parsing `production_json.log` and `api_json.log` #### Find all requests with a 5XX status code diff --git a/doc/administration/uploads.md b/doc/administration/uploads.md index d3a9777775f..949687cfa0a 100644 --- a/doc/administration/uploads.md +++ b/doc/administration/uploads.md @@ -146,7 +146,7 @@ _The uploads are stored by default in remote_directory: "uploads" # The bucket name connection: provider: AWS # Only AWS supported at the moment - aws_access_key_id: AWS_ACESS_KEY_ID + aws_access_key_id: AWS_ACCESS_KEY_ID aws_secret_access_key: AWS_SECRET_ACCESS_KEY region: eu-central-1 ``` |
