diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2022-11-17 11:33:21 +0000 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2022-11-17 11:33:21 +0000 |
| commit | 7021455bd1ed7b125c55eb1b33c5a01f2bc55ee0 (patch) | |
| tree | 5bdc2229f5198d516781f8d24eace62fc7e589e9 /doc/administration | |
| parent | 185b095e93520f96e9cfc31d9c3e69b498cdab7c (diff) | |
| download | gitlab-ce-7021455bd1ed7b125c55eb1b33c5a01f2bc55ee0.tar.gz | |
Add latest changes from gitlab-org/gitlab@15-6-stable-eev15.6.0-rc42
Diffstat (limited to 'doc/administration')
109 files changed, 1898 insertions, 1995 deletions
diff --git a/doc/administration/audit_event_streaming.md b/doc/administration/audit_event_streaming.md index ca60b6142fe..0af1af12a60 100644 --- a/doc/administration/audit_event_streaming.md +++ b/doc/administration/audit_event_streaming.md @@ -312,10 +312,7 @@ Streamed audit events have a predictable schema in the body of the response. > - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/357211) in GitLab 15.0. > - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/357211) in GitLab 15.1 by default. > - [Added `details.author_class` field](https://gitlab.com/gitlab-org/gitlab/-/issues/363876) in GitLab 15.3. - -FLAG: -On self-managed GitLab, by default this feature is available. To hide the -feature, ask an administrator to [disable the feature flag](feature_flags.md) named `audit_event_streaming_git_operations`. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/101583) in GitLab 15.6. Feature flag `audit_event_streaming_git_operations` removed. Streaming audit events can be sent when signed-in users push, pull, or clone a project's remote Git repositories: diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md index 11600bc2da6..0aa0d163972 100644 --- a/doc/administration/audit_events.md +++ b/doc/administration/audit_events.md @@ -4,37 +4,22 @@ group: Compliance info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Audit Events **(PREMIUM)** +# Audit events **(PREMIUM)** -GitLab offers a way to view the changes made within the GitLab server for owners and administrators -on a [paid plan](https://about.gitlab.com/pricing/). +Use audit events to track important events, including who performed the related action and when. +You can use audit events to track, for example: -GitLab system administrators can also view all audit events by accessing the [`audit_json.log` file](logs/index.md#audit_jsonlog). -The JSON audit log does not include events that are [only streamed](../development/audit_event_guide/index.md#event-streaming). +- Who changed the permission level of a particular user for a GitLab project, and when. +- Who added a new user or removed a user, and when. -You can: +The GitLab API, database, and `audit_json.log` record many audit events. Some audit events are only available through +[streaming audit events](audit_event_streaming.md). -- Generate an [audit report](audit_reports.md) of audit events. -- [Stream audit events](audit_event_streaming.md) to an external endpoint. +You can also generate an [audit report](audit_reports.md) of audit events. -## Overview - -**Audit Events** is a tool for GitLab owners and administrators -to track important events such as who performed certain actions and the -time they happened. For example, these actions could be a change to a user -permission level, who added a new user, or who removed a user. - -## Use cases - -- Check who changed the permission level of a particular - user for a GitLab project. -- Track which users have access to a certain group of projects - in GitLab, and who gave them that permission level. - -## Retention policy - -There is no retention policy in place for audit events. -See the [Specify a retention period for audit events](https://gitlab.com/groups/gitlab-org/-/epics/7917) for more information. +NOTE: +You can't configure a retention policy for audit events, but epic +[7917](https://gitlab.com/groups/gitlab-org/-/epics/7917) proposes to change this. ## List of events @@ -115,7 +100,7 @@ From there, you can see the following actions: - Instance administrator started or stopped impersonation of a group member. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/300961) in GitLab 14.8. - Group deploy token was successfully created, revoked, or deleted. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/353452) in GitLab 14.9. - Failed attempt to create a group deploy token. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/353452) in GitLab 14.9. -- [IP restrictions](../user/group/access_and_permissions.md#restrict-group-access-by-ip-address) changed. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/358986) in GitLab 15.0. +- [IP restrictions](../user/group/access_and_permissions.md#restrict-access-to-groups-by-ip-address) changed. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/358986) in GitLab 15.0. - Changes to push rules. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/227629) in GitLab 15.0. - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/356152) in GitLab 15.1, changes to the following merge request approvals settings: - Prevent approval by author. @@ -124,6 +109,7 @@ From there, you can see the following actions: - Require user password to approve. - Remove all approvals when commits are added to the source branch. - Changes to streaming audit destination custom HTTP headers. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/366350) in GitLab 15.3. +- Group had a security policy project linked, changed, or unlinked. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/377877) in GitLab 15.6) Group events can also be accessed via the [Group Audit Events API](../api/audit_events.md#group-audit-events) @@ -194,6 +180,7 @@ From there, you can see the following actions: - Squash commit message template is updated ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/355805) in GitLab 15.0) - Default description template for merge requests is updated ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/355805) in GitLab 15.0) - Project was scheduled for deletion due to inactivity ([introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/85689) in GitLab 15.0) +- Project had a security policy project linked, changed, or unlinked ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/377877) in GitLab 15.6) Project events can also be accessed via the [Project Audit Events API](../api/audit_events.md#project-audit-events). diff --git a/doc/administration/auth/authentiq.md b/doc/administration/auth/authentiq.md index 1ac62b06fe7..d51601439f9 100644 --- a/doc/administration/auth/authentiq.md +++ b/doc/administration/auth/authentiq.md @@ -95,6 +95,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/auth/jwt.md b/doc/administration/auth/jwt.md index c7e7253ef72..c1e76d1c2ed 100644 --- a/doc/administration/auth/jwt.md +++ b/doc/administration/auth/jwt.md @@ -87,6 +87,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/auth/ldap/google_secure_ldap.md b/doc/administration/auth/ldap/google_secure_ldap.md index 2077c6f7baf..01197fdacdf 100644 --- a/doc/administration/auth/ldap/google_secure_ldap.md +++ b/doc/administration/auth/ldap/google_secure_ldap.md @@ -225,6 +225,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/auth/ldap/index.md b/doc/administration/auth/ldap/index.md index 3bb9350e960..6243f3da2d2 100644 --- a/doc/administration/auth/ldap/index.md +++ b/doc/administration/auth/ldap/index.md @@ -179,7 +179,7 @@ These configuration settings are available: | `allow_username_or_email_login` | If enabled, GitLab ignores everything after the first `@` in the LDAP username submitted by the user on sign-in. If you are using `uid: 'userPrincipalName'` on ActiveDirectory you must disable this setting, because the userPrincipalName contains an `@`. | **{dotted-circle}** No | boolean | | `block_auto_created_users` | To maintain tight control over the number of billable users on your GitLab installation, enable this setting to keep new users blocked until they have been cleared by an administrator (default: false). | **{dotted-circle}** No | boolean | | `base` | Base where we can search for users. | **{check-circle}** Yes | `'ou=people,dc=gitlab,dc=example'` or `'DC=mydomain,DC=com'` | -| `user_filter` | Filter LDAP users. Format: [RFC 4515](https://tools.ietf.org/search/rfc4515) Note: GitLab does not support `omniauth-ldap`'s custom filter syntax. | **{dotted-circle}** No | For examples, read [Examples of user filters](#examples-of-user-filters). | +| `user_filter` | Filter LDAP users. Format: [RFC 4515](https://www.rfc-editor.org/rfc/rfc4515.html) Note: GitLab does not support `omniauth-ldap`'s custom filter syntax. | **{dotted-circle}** No | For examples, read [Examples of user filters](#examples-of-user-filters). | | `lowercase_usernames` | If enabled, GitLab converts the name to lower case. | **{dotted-circle}** No | boolean | | `retry_empty_result_with_codes` | An array of LDAP query response code that attempt to retry the operation if the result/content is empty. For Google Secure LDAP, set this value to `[80]`. | **{dotted-circle}** No | `[80]` | @@ -281,7 +281,7 @@ This example results in a sign-in page with the following tabs: To limit all GitLab access to a subset of the LDAP users on your LDAP server, first narrow the configured `base`. However, to further filter users if -necessary, you can set up an LDAP user filter. The filter must comply with [RFC 4515](https://tools.ietf.org/search/rfc4515). +necessary, you can set up an LDAP user filter. The filter must comply with [RFC 4515](https://www.rfc-editor.org/rfc/rfc4515.html). - Example user filter for Omnibus GitLab instances: @@ -336,7 +336,7 @@ The `user_filter` DN can contain special characters. For example: ``` These characters must be escaped as documented in - [RFC 4515](https://tools.ietf.org/search/rfc4515). + [RFC 4515](https://www.rfc-editor.org/rfc/rfc4515.html). - Escape commas with `\2C`. For example: diff --git a/doc/administration/auth/ldap/ldap-troubleshooting.md b/doc/administration/auth/ldap/ldap-troubleshooting.md index 0ec482648a9..499c3c64af7 100644 --- a/doc/administration/auth/ldap/ldap-troubleshooting.md +++ b/doc/administration/auth/ldap/ldap-troubleshooting.md @@ -196,7 +196,7 @@ same user) has the email `email@example.com` set as a secondary email, which is throwing this error. We can check where this conflicting email address is coming from using the -[rails console](#rails-console). Once in the console, run the following: +[rails console](#rails-console). In the console, run the following: ```ruby # This searches for an email among the primary AND secondary emails @@ -546,7 +546,7 @@ this entry, it could be due to a mismatched DN stored in GitLab. See ```shell User with DN `uid=john0,ou=people,dc=example,dc=com` should have access to 'my_group' group but there is no user in GitLab with that -identity. Membership will be updated once the user signs in for +identity. Membership will be updated when the user signs in for the first time. ``` @@ -556,7 +556,7 @@ Finally, the following entry says syncing has finished for this group: Finished syncing all providers for 'my_group' group ``` -Once all the configured group links have been synchronized, GitLab looks +When all the configured group links have been synchronized, GitLab looks for any Administrators or External users to sync: ```shell @@ -614,6 +614,16 @@ ldap_group.member_dns ldap_group.member_uids ``` +#### LDAP synchronization does not remove group creator from group + +[LDAP synchronization](ldap_synchronization.md) should remove an LDAP group's creator +from that group, if that user does not exist in the group. If running LDAP synchronization +does not do this: + +1. Add the user to the LDAP group. +1. Wait until LDAP group synchronization has finished running. +1. Remove the user from the LDAP group. + ### User DN or/and email have changed When an LDAP user is created in GitLab, their LDAP DN is stored for later reference. @@ -735,7 +745,7 @@ To resolve this error, you must apply a new license to the GitLab instance witho 1. Remove or comment out the GitLab configuration lines for all non-primary LDAP servers. 1. [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so that it temporarily uses only one LDAP server. -1. Enter the [Rails console and add the license key](../../troubleshooting/gitlab_rails_cheat_sheet.md#add-a-license-through-the-console). +1. Enter the [Rails console and add the license key](../../../user/admin_area/license_file.md#add-a-license-through-the-console). 1. Re-enable the additional LDAP servers in the GitLab configuration and reconfigure GitLab again. ## Debugging Tools diff --git a/doc/administration/auth/ldap/ldap_synchronization.md b/doc/administration/auth/ldap/ldap_synchronization.md index af2b1400670..02b04861844 100644 --- a/doc/administration/auth/ldap/ldap_synchronization.md +++ b/doc/administration/auth/ldap/ldap_synchronization.md @@ -182,16 +182,18 @@ group, GitLab revokes their `admin` role when syncing. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/1793) in GitLab 12.0. -"Lock memberships to LDAP synchronization" setting allows instance administrators -to lock down user abilities to invite new members to a group. +GitLab administrators can prevent group members from inviting new members to subgroups that have their membership synchronized with LDAP. -When enabled, the following applies: +Global group membership lock only applies to subgroups of the top-level group where LDAP synchronization is configured. No user can modify the +membership of a top-level group configured for LDAP synchronization. + +When global group memberships lock is enabled: - Only an administrator can manage memberships of any group including access levels. - Users are not allowed to share a project with other groups or invite members to a project created in a group. -To enable it, you must: +To enable global group memberships lock: 1. [Configure LDAP](index.md#configure-ldap). 1. On the top bar, select **Main menu > Admin**. diff --git a/doc/administration/auth/oidc.md b/doc/administration/auth/oidc.md index bb263512e06..1f73d8bff38 100644 --- a/doc/administration/auth/oidc.md +++ b/doc/administration/auth/oidc.md @@ -154,7 +154,7 @@ different providers with Omnibus GitLab. ### Configure Google -See the [Google documentation](https://developers.google.com/identity/protocols/oauth2/openid-connect) +See the [Google documentation](https://developers.google.com/identity/openid-connect/openid-connect) for more details: ```ruby @@ -502,7 +502,7 @@ For your app, complete the following steps on Casdoor: ensure the Casdoor app has the following `Redirect URI`: `https://gitlab.example.com/users/auth/openid_connect/callback`. -See the [Casdoor documentation](https://casdoor.org/docs/integration/gitlab) for more details. +See the [Casdoor documentation](https://casdoor.org/docs/integration/ruby/gitlab) for more details. Example Omnibus GitLab configuration (file path: `/etc/gitlab/gitlab.rb`): diff --git a/doc/administration/auth/smartcard.md b/doc/administration/auth/smartcard.md index 11117e8a74c..5b6d299f171 100644 --- a/doc/administration/auth/smartcard.md +++ b/doc/administration/auth/smartcard.md @@ -342,6 +342,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/cicd.md b/doc/administration/cicd.md index 6899b572e8f..ad0671d4c13 100644 --- a/doc/administration/cicd.md +++ b/doc/administration/cicd.md @@ -101,6 +101,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/compliance.md b/doc/administration/compliance.md index ad345461776..ee7476ed6d4 100644 --- a/doc/administration/compliance.md +++ b/doc/administration/compliance.md @@ -48,9 +48,9 @@ settings and automation to ensure that whatever a compliance team has configured stays configured and working correctly. These features can help you automate compliance: -- [**Compliance frameworks**](../user/group/manage.md#compliance-frameworks) (for groups): Create a custom +- [**Compliance frameworks**](../user/group/compliance_frameworks.md) (for groups): Create a custom compliance framework at the group level to describe the type of compliance requirements any child project needs to follow. -- [**Compliance pipelines**](../user/group/manage.md#configure-a-compliance-pipeline) (for groups): Define a +- [**Compliance pipelines**](../user/group/compliance_frameworks.md#configure-a-compliance-pipeline) (for groups): Define a pipeline configuration to run for any projects with a given compliance framework. ## Audit management diff --git a/doc/administration/configure.md b/doc/administration/configure.md index 40f03d25e8d..91fec753f7e 100644 --- a/doc/administration/configure.md +++ b/doc/administration/configure.md @@ -30,7 +30,7 @@ The tables lists features on the left and provides their capabilities to the rig ## Geo Capabilities -If your availability needs to span multiple zones or multiple locations, please read about [Geo](geo/index.md). +If your availability needs to span multiple zones or multiple locations, read about [Geo](geo/index.md). | | Availability | Recoverability | Data Resiliency | Performance | Risks/Trade-offs| |-|--------------|----------------|-----------------|-------------|-----------------| @@ -46,5 +46,5 @@ The following table outlines failure modes and mitigation paths for the product | Single Gitaly Node + Geo Secondary | Downtime - Must restore from backup, can perform a manual failover to secondary | Downtime - Must restore from Backup, errors could have propagated to secondary | Manual intervention - failover to Geo secondary | | | Sharded Gitaly Install | Partial Downtime - Only repositories on impacted node affected, must restore from backup | Partial Downtime - Only repositories on impacted node affected, must restore from backup | Downtime - Must wait for outage to end | | | Sharded Gitaly Install + Geo Secondary | Partial Downtime - Only repositories on impacted node affected, must restore from backup, could perform manual failover to secondary for impacted repositories | Partial Downtime - Only repositories on impacted node affected, must restore from backup, errors could have propagated to secondary | Manual intervention - failover to Geo secondary | | -| Gitaly Cluster Install* | No Downtime - will swap repository primary to another node after 10 seconds | Not applicable; All writes are voted on by multiple Gitaly Cluster nodes | Downtime - Must wait for outage to end | Snapshot backups for Gitaly Cluster nodes not supported at this time | -| Gitaly Cluster Install* + Geo Secondary | No Downtime - will swap repository primary to another node after 10 seconds | Not applicable; All writes are voted on by multiple Gitaly Cluster nodes | Manual intervention - failover to Geo secondary | Snapshot backups for Gitaly Cluster nodes not supported at this time | +| Gitaly Cluster Install* | No Downtime - swaps repository primary to another node after 10 seconds | Not applicable; All writes are voted on by multiple Gitaly Cluster nodes | Downtime - Must wait for outage to end | Snapshot backups for Gitaly Cluster nodes not supported at this time | +| Gitaly Cluster Install* + Geo Secondary | No Downtime - swaps repository primary to another node after 10 seconds | Not applicable; All writes are voted on by multiple Gitaly Cluster nodes | Manual intervention - failover to Geo secondary | Snapshot backups for Gitaly Cluster nodes not supported at this time | diff --git a/doc/administration/consul.md b/doc/administration/consul.md index a6f76882c4d..965231db440 100644 --- a/doc/administration/consul.md +++ b/doc/administration/consul.md @@ -8,7 +8,7 @@ type: reference # How to set up Consul **(PREMIUM SELF)** A Consul cluster consists of both -[server and client agents](https://www.consul.io/docs/agent). +[server and client agents](https://developer.hashicorp.com/consul/docs/agent). The servers run on their own nodes and the clients run on other nodes that in turn communicate with the servers. @@ -99,7 +99,7 @@ Consul nodes communicate using the raft protocol. If the current leader goes offline, there must be a leader election. A leader node must exist to facilitate synchronization across the cluster. If too many nodes go offline at the same time, the cluster loses quorum and doesn't elect a leader due to -[broken consensus](https://www.consul.io/docs/architecture/consensus). +[broken consensus](https://developer.hashicorp.com/consul/docs/architecture/consensus). Consult the [troubleshooting section](#troubleshooting-consul) if the cluster is not able to recover after the upgrade. The [outage recovery](#outage-recovery) may @@ -148,7 +148,7 @@ you follow the Consul [outage recovery](#outage-recovery) process. To be safe, it's recommended that you only restart Consul in one node at a time to ensure the cluster remains intact. For larger clusters, it is possible to restart multiple nodes at a time. See the -[Consul consensus document](https://www.consul.io/docs/architecture/consensus#deployment-table) +[Consul consensus document](https://developer.hashicorp.com/consul/docs/architecture/consensus#deployment-table) for the number of failures it can tolerate. This is the number of simultaneous restarts it can sustain. @@ -161,7 +161,7 @@ sudo gitlab-ctl restart consul ### Consul nodes unable to communicate By default, Consul attempts to -[bind](https://www.consul.io/docs/agent/config/config-files#bind_addr) to `0.0.0.0`, but +[bind](https://developer.hashicorp.com/consul/docs/agent/config/config-files#bind_addr) to `0.0.0.0`, but it advertises the first private IP address on the node for other Consul nodes to communicate with it. If the other nodes cannot communicate with a node on this address, then the cluster has a failed status. @@ -249,5 +249,5 @@ Shortly after that, the client agents should rejoin as well. If you have taken advantage of Consul to store other data and want to restore the failed node, follow the -[Consul guide](https://learn.hashicorp.com/tutorials/consul/recovery-outage) +[Consul guide](https://developer.hashicorp.com/consul/tutorials/datacenter-operations/recovery-outage) to recover a failed cluster. diff --git a/doc/administration/environment_variables.md b/doc/administration/environment_variables.md index beaef7afe57..d2edc3085f1 100644 --- a/doc/administration/environment_variables.md +++ b/doc/administration/environment_variables.md @@ -50,5 +50,5 @@ To set environment variables, follow [these instructions](https://docs.gitlab.co It's possible to preconfigure the GitLab Docker image by adding the environment variable `GITLAB_OMNIBUS_CONFIG` to the `docker run` command. -For more information, see the [Pre-configure Docker container](https://docs.gitlab.com/omnibus/docker/#pre-configure-docker-container) +For more information, see the [Pre-configure Docker container](../install/docker.md#pre-configure-docker-container) section of the Omnibus GitLab documentation. diff --git a/doc/administration/feature_flags.md b/doc/administration/feature_flags.md index 4f8f7c22a55..f2a40b60536 100644 --- a/doc/administration/feature_flags.md +++ b/doc/administration/feature_flags.md @@ -118,10 +118,10 @@ Some feature flags can be enabled or disabled on a per project basis: Feature.enable(:<feature flag>, Project.find(<project id>)) ``` -For example, to enable the [`:product_analytics`](../operations/product_analytics.md) feature flag for project `1234`: +For example, to enable the `:my_awesome_feature` feature flag for project `1234`: ```ruby -Feature.enable(:product_analytics, Project.find(1234)) +Feature.enable(:my_awesome_feature, Project.find(1234)) ``` `Feature.enable` and `Feature.disable` always return `true`, even if the application doesn't use the flag: diff --git a/doc/administration/file_hooks.md b/doc/administration/file_hooks.md index 8fafb258593..f8b55a7c680 100644 --- a/doc/administration/file_hooks.md +++ b/doc/administration/file_hooks.md @@ -7,10 +7,8 @@ type: reference # File hooks **(FREE SELF)** -> Renamed feature from Plugins to File hooks in GitLab 12.8. - -With custom file hooks, GitLab administrators can introduce custom integrations -without modifying the GitLab source code. +Use custom file hooks (not to be confused with [server hooks](server_hooks.md) or [system hooks](system_hooks.md)), +to introduce custom integrations without modifying the GitLab source code. A file hook runs on each event. You can filter events or projects in a file hook's code, and create many file hooks as you need. Each file hook is diff --git a/doc/administration/geo/disaster_recovery/background_verification.md b/doc/administration/geo/disaster_recovery/background_verification.md index 699fe6851eb..99ac95cd536 100644 --- a/doc/administration/geo/disaster_recovery/background_verification.md +++ b/doc/administration/geo/disaster_recovery/background_verification.md @@ -122,7 +122,7 @@ be resynced with a back-off period. If you want to reset them manually, this Rake task marks projects where verification has failed or the checksum mismatch to be resynced without the back-off period: -Run the appropriate commands on a **Rails node on the primary** site. +Run the appropriate commands on a **Rails node on the secondary** site. For repositories: diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index 156a87614e6..dfa8d09e6ef 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -758,7 +758,7 @@ If you are running GitLab 14.4 and earlier: To promote the **secondary** cluster to a **primary** cluster, update `role: secondary` to `role: primary`. - If the cluster remains as a primary site, you can remove the entire `psql` section; it refers to the tracking database and is ignored whilst the cluster is acting as a primary site. + If the cluster remains as a primary site, you can remove the entire `psql` section; it refers to the tracking database and is ignored while the cluster is acting as a primary site. Update the cluster with the new configuration: diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md index 60101f5af8e..80707afacca 100644 --- a/doc/administration/geo/disaster_recovery/planned_failover.md +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -14,12 +14,12 @@ downtime. As replication between Geo sites is asynchronous, a planned failover requires a maintenance window in which updates to the **primary** site are blocked. The -length of this window is determined by your replication capacity - once the +length of this window is determined by your replication capacity - when the **secondary** site is completely synchronized with the **primary** site, the failover can occur without data loss. This document assumes you already have a fully configured, working Geo setup. -Please read it and the [Disaster Recovery](index.md) failover +Read this document and the [Disaster Recovery](index.md) failover documentation in full before proceeding. Planned failover is a major operation, and if performed incorrectly, there is a high risk of data loss. Consider rehearsing the procedure until you are comfortable with the necessary steps and diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md index a336f5ff8bc..68fd0c63e37 100644 --- a/doc/administration/geo/index.md +++ b/doc/administration/geo/index.md @@ -120,18 +120,18 @@ The following are required to run Geo: - [CentOS](https://www.centos.org) 7.4 or later - [Ubuntu](https://ubuntu.com) 16.04 or later - PostgreSQL 12 or 13 with [Streaming Replication](https://wiki.postgresql.org/wiki/Streaming_Replication) - - Note,[PostgreSQL 12 is deprecated](../../update/deprecations.md#postgresql-12-deprecated) and will be removed in GitLab 16.0. + - Note,[PostgreSQL 12 is deprecated](../../update/deprecations.md#postgresql-12-deprecated) and is removed in GitLab 16.0. - Git 2.9 or later - Git-lfs 2.4.2 or later on the user side when using LFS - All sites must run [the same GitLab and PostgreSQL versions](setup/database.md#postgresql-replication). -- If using different operating system versions between Geo sites, [check OS locale data compatibility](replication/troubleshooting.md#check-os-locale-data-compatibility) across Geo sites. +- If using different operating system versions between Geo sites, [check OS locale data compatibility](replication/troubleshooting.md#check-os-locale-data-compatibility) across Geo sites. Additionally, check the GitLab [minimum requirements](../../install/requirements.md), -and we recommend you use the latest version of GitLab for a better experience. +and use the latest version of GitLab for a better experience. ### Firewall rules -The following table lists basic ports that must be open between the **primary** and **secondary** sites for Geo. To simplify failovers, we recommend opening ports in both directions. +The following table lists basic ports that must be open between the **primary** and **secondary** sites for Geo. To simplify failovers, you should open ports in both directions. | Source site | Source port | Destination site | Destination port | Protocol | |-------------|-------------|------------------|------------------|-------------| @@ -306,7 +306,7 @@ For an example of how to set up a location-aware Git remote URL with AWS Route53 ### Backfill -Once a **secondary** site is set up, it starts replicating missing data from +When a **secondary** site is set up, it starts replicating missing data from the **primary** site in a process known as **backfill**. You can monitor the synchronization process on each Geo site from the **primary** site's **Geo Nodes** dashboard in your browser. diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md index fa74f16cdc8..55c5d3784c2 100644 --- a/doc/administration/geo/replication/configuration.md +++ b/doc/administration/geo/replication/configuration.md @@ -12,7 +12,7 @@ type: howto NOTE: This is the final step in setting up a **secondary** Geo site. Stages of the setup process must be completed in the documented order. -If not, [complete all prior stages](../setup/index.md#using-omnibus-gitlab) before procceed. +If not, [complete all prior stages](../setup/index.md#using-omnibus-gitlab) before proceeding. Make sure you [set up the database replication](../setup/database.md), and [configured fast lookup of authorized SSH keys](../../operations/fast_ssh_key_lookup.md) in **both primary and secondary sites**. @@ -239,8 +239,9 @@ keys must be manually replicated to the **secondary** site. If any of the checks fail, check the [troubleshooting documentation](troubleshooting.md). -Once added to the Geo administration page and restarted, the **secondary** site automatically starts -replicating missing data from the **primary** site in a process known as **backfill**. +After the **secondary** site is added to the Geo administration page and restarted, +the site automatically starts replicating missing data from the **primary** site +in a process known as **backfill**. Meanwhile, the **primary** site starts to notify each **secondary** site of any changes, so that the **secondary** site can act on those notifications immediately. diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index 566df2ee509..0198d2a63e8 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -201,8 +201,7 @@ successfully, you must replicate their data using some other means. |[CI job artifacts](../../../ci/pipelines/job_artifacts.md) | **Yes** (10.4) | **Yes** (14.10) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Verification is behind the feature flag `geo_job_artifact_replication`, enabled by default in 14.10. | |[CI Pipeline Artifacts](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/pipeline_artifact.rb) | [**Yes** (13.11)](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | [**Yes** (13.11)](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Persists additional artifacts after a pipeline completes. | |[CI Secure Files](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/secure_file.rb) | [**Yes** (15.3)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91430) | [**Yes** (15.3)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91430) | [**Yes** (15.3)](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91430) | [No](object_storage.md#verification-of-files-in-object-storage) | Verification is behind the feature flag `geo_ci_secure_file_replication`, enabled by default in 15.3. | -|[Container Registry](../../packages/container_registry.md) | **Yes** (12.3)* | No | No | No | Replication is behind feature flag `geo_container_repository_replication`, enabled by default. -Requires additional configuration. See [instructions](container_registry.md) to set up the Container Registry replication. | +|[Container Registry](../../packages/container_registry.md) | **Yes** (12.3)* | No | No | No | Replication is behind feature flag `geo_container_repository_replication`, enabled by default. Requires additional configuration. See [instructions](container_registry.md) to set up the Container Registry replication. | |[Infrastructure Registry](../../../user/packages/infrastructure_registry/index.md) | **Yes** (14.0) | **Yes** (14.0) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Behind feature flag `geo_package_file_replication`, enabled by default. | |[Project designs repository](../../../user/project/issues/design_management.md) | **Yes** (12.7) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/32467) | N/A | N/A | Designs also require replication of LFS objects and Uploads. | |[Package Registry](../../../user/packages/package_registry/index.md) | **Yes** (13.2) | **Yes** (13.10) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Behind feature flag `geo_package_file_replication`, enabled by default. | diff --git a/doc/administration/geo/replication/disable_geo.md b/doc/administration/geo/replication/disable_geo.md index 3230a92136f..c42130a62a7 100644 --- a/doc/administration/geo/replication/disable_geo.md +++ b/doc/administration/geo/replication/disable_geo.md @@ -24,8 +24,8 @@ To disable Geo, follow these steps: ## Remove all secondary Geo sites -To disable Geo, you need to first remove all your secondary Geo sites, which means replication will not happen -anymore on these sites. You can follow our docs to [remove your secondary Geo sites](remove_geo_site.md). +To disable Geo, you need to first remove all your secondary Geo sites, which means replication does not happen +anymore on these sites. You can follow our documentation to [remove your secondary Geo sites](remove_geo_site.md). If the current site that you want to keep using is a secondary site, you need to first promote it to primary. You can use our steps on [how to promote a secondary site](../disaster_recovery/index.md#step-3-promoting-a-secondary-site) diff --git a/doc/administration/geo/replication/docker_registry.md b/doc/administration/geo/replication/docker_registry.md deleted file mode 100644 index d0af6f2a66f..00000000000 --- a/doc/administration/geo/replication/docker_registry.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'container_registry.md' -remove_date: '2022-10-29' ---- - -This document was moved to [another location](container_registry.md). - -<!-- This redirect file can be deleted after <2022-10-29>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
\ No newline at end of file diff --git a/doc/administration/geo/replication/faq.md b/doc/administration/geo/replication/faq.md index 81afcc19bb1..311cdeee5b9 100644 --- a/doc/administration/geo/replication/faq.md +++ b/doc/administration/geo/replication/faq.md @@ -22,7 +22,7 @@ For each project to sync: 1. Geo issues a `git fetch geo --mirror` to get the latest information from the **primary** site. If there are no changes, the sync is fast. Otherwise, it has to pull the latest commits. -1. The **secondary** site updates the tracking database to store the fact that it has synced projects A, B, C, and so on. +1. The **secondary** site updates the tracking database to store the fact that it has synced projects by name. 1. Repeat until all projects are synced. When someone pushes a commit to the **primary** site, it generates an event in the GitLab database that the repository has changed. @@ -45,8 +45,8 @@ Read the documentation for [Disaster Recovery](../disaster_recovery/index.md). ## What data is replicated to a **secondary** site? We currently replicate project repositories, LFS objects, generated -attachments and avatars, and the whole database. This means user accounts, -issues, merge requests, groups, project data, and so on, are available for +attachments and avatars, and the whole database. This means information such as user accounts, +issues, merge requests, groups, and project data are available for query. For more details, see the [supported Geo data types](datatypes.md). @@ -58,8 +58,8 @@ Pushing directly to a **secondary** site (for both HTTP and SSH, including Git L ## How long does it take to have a commit replicated to a **secondary** site? All replication operations are asynchronous and are queued to be dispatched. Therefore, it depends on a lot of -factors including the amount of traffic, how big your commit is, the -connectivity between your sites, your hardware, and so on. +factors such as the amount of traffic, how big your commit is, the +connectivity between your sites, and your hardware. ## What if the SSH server runs at a different port? diff --git a/doc/administration/geo/replication/geo_validation_tests.md b/doc/administration/geo/replication/geo_validation_tests.md index 8fa5a45b579..f09422d1e26 100644 --- a/doc/administration/geo/replication/geo_validation_tests.md +++ b/doc/administration/geo/replication/geo_validation_tests.md @@ -29,7 +29,7 @@ The following are GitLab upgrade validation tests we performed. [Switch from repmgr to Patroni on a Geo primary site](https://gitlab.com/gitlab-org/gitlab/-/issues/224652): - Description: Tested switching from repmgr to Patroni on a multi-node Geo primary site. Used [the orchestrator tool](https://gitlab.com/gitlab-org/gitlab-orchestrator) to deploy a Geo installation with 3 database nodes managed by repmgr. With this approach, we were also able to address a related issue for [verifying a Geo installation with Patroni and PostgreSQL 11](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5113). -- Outcome: Partial success. We enabled Patroni on the primary site and set up database replication on the secondary site. However, we found that Patroni would delete the secondary site's replication slot whenever Patroni was restarted. Another issue is that when Patroni elects a new leader in the cluster, the secondary site will fail to automatically follow the new leader. Until these issues are resolved, we cannot officially support and recommend Patroni for Geo installations. +- Outcome: Partial success. We enabled Patroni on the primary site and set up database replication on the secondary site. However, we found that Patroni would delete the secondary site's replication slot whenever Patroni was restarted. Another issue is that when Patroni elects a new leader in the cluster, the secondary site fails to automatically follow the new leader. Until these issues are resolved, we cannot officially support and recommend Patroni for Geo installations. - Follow up issues/actions: - [Investigate permanent replication slot for Patroni with Geo single node secondary](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5528) @@ -213,7 +213,7 @@ The following are additional validation tests we performed. [Validate Object storage replication using GCP based object storage](https://gitlab.com/gitlab-org/gitlab/-/issues/351464): - Description: Tested the average time it takes for a single image to replicate from the primary object storage location to the secondary when using GCP based object storage replication and [GitLab based object storage replication](object_storage.md#enabling-gitlab-managed-object-storage-replication). This was tested by uploading a 1mb image to a project on the primary site every second for 60 seconds. The time was then measured until a image was available on the secondary site. This was achieved using a [Ruby Script](https://gitlab.com/gitlab-org/quality/geo-replication-tester). -- Outcome: GCP handles replication differently than other Cloud Providers. In GCP, the process is to a create single bucket that is either multi, dual or single region based. This means that the bucket will automatically store replicas in a region based on the option chosen. Even when using multi region, this will still only replicate within a single continent, the options being America, Europe, or Asia. At current there doesn't seem to be any way to replicate objects between continents using GCP based replication. For Geo managed replication the average time when replicating within the same region was 6 seconds, and when replicating cross region this rose to just 9 seconds. +- Outcome: GCP handles replication differently than other Cloud Providers. In GCP, the process is to a create single bucket that is either multi, dual, or single region based. This means that the bucket automatically stores replicas in a region based on the option chosen. Even when using multi region, this only replicates in a single continent, the options being America, Europe, or Asia. At current there doesn't seem to be any way to replicate objects between continents using GCP based replication. For Geo managed replication the average time when replicating in the same region was 6 seconds, and when replicating cross region this rose to just 9 seconds. ## Other tests diff --git a/doc/administration/geo/replication/location_aware_git_url.md b/doc/administration/geo/replication/location_aware_git_url.md index e0e113eebbd..dbe543f5a62 100644 --- a/doc/administration/geo/replication/location_aware_git_url.md +++ b/doc/administration/geo/replication/location_aware_git_url.md @@ -31,7 +31,7 @@ In this example, we have already set up: - `primary.example.com` as a Geo **primary** site. - `secondary.example.com` as a Geo **secondary** site. -We will create a `git.example.com` subdomain that will automatically direct +We create a `git.example.com` subdomain that automatically directs requests: - From Europe to the **secondary** site. diff --git a/doc/administration/geo/replication/remove_geo_site.md b/doc/administration/geo/replication/remove_geo_site.md index 62b1d9fdf7b..4b9f31dc08c 100644 --- a/doc/administration/geo/replication/remove_geo_site.md +++ b/doc/administration/geo/replication/remove_geo_site.md @@ -14,7 +14,8 @@ type: howto 1. Select the **Remove** button for the **secondary** site you want to remove. 1. Confirm by selecting **Remove** when the prompt appears. -Once removed from the Geo administration page, you must stop and uninstall the **secondary** site. For each node on your secondary Geo site: +After the **secondary** site is removed from the Geo administration page, you must +stop and uninstall this site. For each node on your secondary Geo site: 1. Stop GitLab: @@ -35,7 +36,7 @@ Once removed from the Geo administration page, you must stop and uninstall the * sudo rpm --erase gitlab-ee ``` -Once GitLab has been uninstalled from each node on the **secondary** site, the replication slot must be dropped from the **primary** site's database as follows: +When GitLab has been uninstalled from each node on the **secondary** site, the replication slot must be dropped from the **primary** site's database as follows: 1. On the **primary** site's database node, start a PostgreSQL console session: diff --git a/doc/administration/geo/replication/security_review.md b/doc/administration/geo/replication/security_review.md index 0231da53b9c..afe831dcb9c 100644 --- a/doc/administration/geo/replication/security_review.md +++ b/doc/administration/geo/replication/security_review.md @@ -25,8 +25,8 @@ from [owasp.org](https://owasp.org/). ### What data does the application receive, produce, and process? - Geo streams almost all data held by a GitLab instance between sites. This - includes full database replication, most files (user-uploaded attachments, - and so on) and repository + wiki data. In a typical configuration, this will + includes full database replication, most files such as user-uploaded attachments, + and repository + wiki data. In a typical configuration, this will happen across the public Internet, and be TLS-encrypted. - PostgreSQL replication is TLS-encrypted. - See also: [only TLSv1.2 should be supported](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/2948) @@ -37,7 +37,7 @@ from [owasp.org](https://owasp.org/). private projects. Geo replicates them all indiscriminately. "Selective sync" exists for files and repositories (but not database content), which would permit only less-sensitive projects to be replicated to a **secondary** site if desired. -- See also: [GitLab data classification policy](https://about.gitlab.com/handbook/engineering/security/data-classification-standard.html). +- See also: [GitLab data classification policy](https://about.gitlab.com/handbook/security/data-classification-standard.html). ### What data backup and retention requirements have been defined for the application? @@ -59,8 +59,8 @@ from [owasp.org](https://owasp.org/). (notably a HTTP/HTTPS web application, and HTTP/HTTPS or SSH Git repository access), but is constrained to read-only activities. The principal use case is envisioned to be cloning Git repositories from the **secondary** site in favor of the - **primary** site, but end-users may use the GitLab web interface to view projects, - issues, merge requests, snippets, and so on. + **primary** site, but end-users may use the GitLab web interface to view information like projects, + issues, merge requests, and snippets. ### What security expectations do the end‐users have? diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index 3f16c1552ad..fa668091c90 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -12,8 +12,9 @@ miss a step. Here is a list of steps you should take to attempt to fix problem: 1. Perform [basic troubleshooting](#basic-troubleshooting). -1. Fix any [replication errors](#fixing-replication-errors). +1. Fix any [PostgreSQL database replication errors](#fixing-postgresql-database-replication-errors). 1. Fix any [common](#fixing-common-errors) errors. +1. Fix any [non-PostgreSQL replication failures](#fixing-non-postgresql-replication-failures). ## Basic troubleshooting @@ -131,6 +132,8 @@ http://secondary.example.com/ To find more details about failed items, check [the `gitlab-rails/geo.log` file](../../logs/log_parsing.md#find-most-common-geo-sync-errors) +If you notice replication or verification failures, you can try to [resolve them](#fixing-non-postgresql-replication-failures). + ### Check if PostgreSQL replication is working To check if PostgreSQL replication is working, check if: @@ -185,6 +188,41 @@ This machine's Geo node name matches a database record ... no Learn more about recommended site names in the description of the Name field in [Geo Admin Area Common Settings](../../../user/admin_area/geo_sites.md#common-settings). +### Reverify all uploads (or any SSF data type which is verified) + +1. SSH into a GitLab Rails node in the primary Geo site. +1. Open [Rails console](../../operations/rails_console.md). +1. Mark all uploads as "pending verification": + +WARNING: +Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. + + ```ruby + Upload.verification_state_table_class.each_batch do |relation| + relation.update_all(verification_state: 0) + end + ``` + +1. This will cause the primary to start checksumming all Uploads. +1. When a primary successfully checksums a record, then all secondaries rechecksum as well, and they compare the values. + +A similar thing can be done for all Models handled by the [Geo Self-Service Framework](../../../development/geo/framework.md) which have implemented verification: + +- `LfsObject` +- `MergeRequestDiff` +- `Packages::PackageFile` +- `Terraform::StateVersion` +- `SnippetRepository` +- `Ci::PipelineArtifact` +- `PagesDeployment` +- `Upload` +- `Ci::JobArtifact` +- `Ci::SecureFile` + +NOTE: +`GroupWikiRepository` is not in the previous list since verification is not implemented. +There is an [issue to implement this functionality in the Admin Area UI](https://gitlab.com/gitlab-org/gitlab/-/issues/364729). + ### Message: `WARNING: oldest xmin is far in the past` and `pg_wal` size growing If a replication slot is inactive, @@ -311,52 +349,41 @@ sudo gitlab-rake gitlab:geo:check When performing a PostgreSQL major version (9 > 10) update this is expected. Follow the [initiate-the-replication-process](../setup/database.md#step-3-initiate-the-replication-process). -### Repository verification failures +### Message: Machine clock is synchronized ... Exception -[Start a Rails console session](../../../administration/operations/rails_console.md#starting-a-rails-console-session) -to gather the following, basic troubleshooting information. +The Rake task attempts to verify that the server clock is synchronized with NTP. Synchronized clocks +are required for Geo to function correctly. As an example, for security, when the server time on the +primary site and secondary site differ by about a minute or more, requests between Geo sites +will fail. If this check task fails to complete due to a reason other than mismatching times, it +does not necessarily mean that Geo will not work. -WARNING: -Any command that changes data directly could be damaging if not run correctly, or under the right conditions. We highly recommend running them in a test environment with a backup of the instance ready to be restored, just in case. +The Ruby gem which performs the check is hard coded with `pool.ntp.org` as its reference time source. -#### Get the number of verification failed repositories +- Exception message `Machine clock is synchronized ... Exception: Timeout::Error` -```ruby -Geo::ProjectRegistry.verification_failed('repository').count -``` + This issue occurs when your server cannot access the host `pool.ntp.org`. -#### Find the verification failed repositories +- Exception message `Machine clock is synchronized ... Exception: No route to host - recvfrom(2)` -```ruby -Geo::ProjectRegistry.verification_failed('repository') -``` + This issue occurs when the hostname `pool.ntp.org` resolves to a server which does not provide a time service. -#### Find repositories that failed to sync +There is [an issue open](https://gitlab.com/gitlab-org/gitlab/-/issues/381422) for this dependency on `pool.ntp.org`. -```ruby -Geo::ProjectRegistry.sync_failed('repository') -``` - -### Resync repositories - -[Start a Rails console session](../../../administration/operations/rails_console.md#starting-a-rails-console-session) -to enact the following, basic troubleshooting steps. - -#### Queue up all repositories for resync. Sidekiq handles each sync +To workaround this, do one of the following: -```ruby -Geo::ProjectRegistry.update_all(resync_repository: true, resync_wiki: true) -``` +- Add entries in `/etc/hosts` for `pool.ntp.org` to direct the request to valid local time servers. + This fixes the long timeout and the timeout error. +- Direct the check to any valid IP address. This resolves the timeout issue, but the check will fail + with the `No route to host` error, as noted above. -#### Sync individual repository now +[Cloud native GitLab deployments](https://docs.gitlab.com/charts/advanced/geo/#set-the-geo-primary-site) +generate an error because containers in Kubernetes do not have access to the host clock: -```ruby -project = Project.find_by_full_path('<group/project>') - -Geo::RepositorySyncService.new(project).execute +```plaintext +Machine clock is synchronized ... Exception: getaddrinfo: Servname not supported for ai_socktype ``` -## Fixing replication errors +## Fixing PostgreSQL database replication errors The following sections outline troubleshooting steps for fixing replication error messages (indicated by `Database replication working? ... no` in the @@ -469,7 +496,7 @@ This happens because the PostgreSQL certificate that the Omnibus GitLab package the Common Name `PostgreSQL`, but the replication is connecting to a different host and GitLab attempts to use the `verify-full` SSL mode by default. -In order to fix this, you can either: +To fix this issue, you can either: - Use the `--sslmode=verify-ca` argument with the `replicate-geo-database` command. - For an already replicated database, change `sslmode=verify-full` to `sslmode=verify-ca` @@ -837,120 +864,6 @@ This behavior affects only the following data types through GitLab 14.6: to make Geo visibly surface data loss risks. The sync/verification loop is therefore short-circuited. `last_sync_failure` is now set to `The file is missing on the Geo primary site`. -### Blob types - -- `Ci::JobArtifact` -- `Ci::PipelineArtifact` -- `Ci::SecureFile` -- `LfsObject` -- `MergeRequestDiff` -- `Packages::PackageFile` -- `PagesDeployment` -- `Terraform::StateVersion` -- `Upload` - -`Packages::PackageFile` is used in the following -[Rails console](../../../administration/operations/rails_console.md#starting-a-rails-console-session) -examples, but things generally work the same for the other types. - -WARNING: -Any command that changes data directly could be damaging if not run correctly, or under the right conditions. We highly recommend running them in a test environment with a backup of the instance ready to be restored, just in case. - -#### The Replicator - -The main kinds of classes are Registry, Model, and Replicator. If you have an instance of one of these classes, you can get the others. The Registry and Model mostly manage PostgreSQL DB state. The Replicator knows how to replicate/verify (or it can call a service to do it): - -```ruby -model_record = Packages::PackageFile.last -model_record.replicator.registry.replicator.model_record # just showing that these methods exist -``` - -#### Replicate a package file, synchronously, given an ID - -```ruby -model_record = Packages::PackageFile.find(id) -model_record.replicator.send(:download) -``` - -#### Replicate a package file, synchronously, given a registry ID - -```ruby -registry = Geo::PackageFileRegistry.find(registry_id) -registry.replicator.send(:download) -``` - -#### Verify package files on the secondary manually - -This iterates over all package files on the secondary, looking at the -`verification_checksum` stored in the database (which came from the primary) -and then calculate this value on the secondary to check if they match. This -does not change anything in the UI: - -```ruby -# Run on secondary -status = {} - -Packages::PackageFile.find_each do |package_file| - primary_checksum = package_file.verification_checksum - secondary_checksum = Packages::PackageFile.hexdigest(package_file.file.path) - verification_status = (primary_checksum == secondary_checksum) - - status[verification_status.to_s] ||= [] - status[verification_status.to_s] << package_file.id -end - -# Count how many of each value we get -status.keys.each {|key| puts "#{key} count: #{status[key].count}"} - -# See the output in its entirety -status -``` - -### Repository types newer than project/wiki repositories - -- `SnippetRepository` -- `GroupWikiRepository` - -`SnippetRepository` is used in the examples below, but things generally work the same for the other Repository types. - -#### The Replicator - -The main kinds of classes are Registry, Model, and Replicator. If you have an instance of one of these classes, you can get the others. The Registry and Model mostly manage PostgreSQL DB state. The Replicator knows how to replicate/verify (or it can call a service to do it). - -```ruby -model_record = SnippetRepository.last -model_record.replicator.registry.replicator.model_record # just showing that these methods exist -``` - -#### Replicate a snippet repository, synchronously, given an ID - -```ruby -model_record = SnippetRepository.find(id) -model_record.replicator.send(:sync_repository) -``` - -#### Replicate a snippet repository, synchronously, given a registry ID - -```ruby -registry = Geo::SnippetRepositoryRegistry.find(registry_id) -registry.replicator.send(:sync_repository) -``` - -### Find failed artifacts - -[Start a Rails console session](../../../administration/operations/rails_console.md#starting-a-rails-console-session) -to run the following commands: - -```ruby -Geo::JobArtifactRegistry.failed -``` - -#### Find `ID` of synced artifacts that are missing on primary - -```ruby -Geo::JobArtifactRegistry.synced.missing_on_primary.pluck(:artifact_id) -``` - #### Failed syncs with GitLab-managed object storage replication There is [an issue in GitLab 14.2 through 14.7](https://gitlab.com/gitlab-org/gitlab/-/issues/299819#note_822629467) @@ -1218,7 +1131,8 @@ If you set up a new secondary from scratch, you must also [remove the old site f The most common problems that prevent the database from replicating correctly are: -- **Secondary** sites cannot reach the **primary** site. Check credentials, [firewall rules](../index.md#firewall-rules), and so on. +- **Secondary** sites cannot reach the **primary** site. Check credentials and + [firewall rules](../index.md#firewall-rules). - SSL certificate problems. Make sure you copied `/etc/gitlab/gitlab-secrets.json` from the **primary** site. - Database storage disk is full. - Database replication slot is misconfigured. @@ -1320,6 +1234,217 @@ To fix this issue, set the primary site's internal URL to a URL that is: GeoNode.where(primary: true).first.update!(internal_url: "https://unique.url.for.primary.site") ``` +## Fixing non-PostgreSQL replication failures + +If you notice replication failures in `Admin > Geo > Sites` or the [Sync status Rake task](#sync-status-rake-task), you can try to resolve the failures with the following general steps: + +1. Geo will automatically retry failures. If the failures are new and few in number, or if you suspect the root cause is already resolved, then you can wait to see if the failures go away. +1. If failures were present for a long time, then many retries have already occurred, and the interval between automatic retries has increased to up to 4 hours depending on the type of failure. If you suspect the root cause is already resolved, you can [manually retry replication or verification](#manually-retry-replication-or-verification). +1. If the failures persist, use the following sections to try to resolve them. + +### Manually retry replication or verification + +Project Git repositories and Project Wiki Git repositories have the ability in `Admin > Geo > Replication` to `Resync all`, `Reverify all`, or for a single resource, `Resync` or `Reverify`. + +Adding this ability to other data types is proposed in issue [364725](https://gitlab.com/gitlab-org/gitlab/-/issues/364725). + +The following sections describe how to use internal application commands in the [Rails console](../../../administration/operations/rails_console.md#starting-a-rails-console-session) to cause replication or verification immediately. + +WARNING: +Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. + +### Blob types + +- `Ci::JobArtifact` +- `Ci::PipelineArtifact` +- `Ci::SecureFile` +- `LfsObject` +- `MergeRequestDiff` +- `Packages::PackageFile` +- `PagesDeployment` +- `Terraform::StateVersion` +- `Upload` + +`Packages::PackageFile` is used in the following +[Rails console](../../../administration/operations/rails_console.md#starting-a-rails-console-session) +examples, but things generally work the same for the other types. + +WARNING: +Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. + +#### The Replicator + +The main kinds of classes are Registry, Model, and Replicator. If you have an instance of one of these classes, you can get the others. The Registry and Model mostly manage PostgreSQL DB state. The Replicator knows how to replicate/verify (or it can call a service to do it): + +```ruby +model_record = Packages::PackageFile.last +model_record.replicator.registry.replicator.model_record # just showing that these methods exist +``` + +#### Replicate a package file, synchronously, given an ID + +```ruby +model_record = Packages::PackageFile.find(id) +model_record.replicator.send(:download) +``` + +#### Replicate a package file, synchronously, given a registry ID + +```ruby +registry = Geo::PackageFileRegistry.find(registry_id) +registry.replicator.send(:download) +``` + +#### Verify package files on the secondary manually + +This iterates over all package files on the secondary, looking at the +`verification_checksum` stored in the database (which came from the primary) +and then calculate this value on the secondary to check if they match. This +does not change anything in the UI: + +```ruby +# Run on secondary +status = {} + +Packages::PackageFile.find_each do |package_file| + primary_checksum = package_file.verification_checksum + secondary_checksum = Packages::PackageFile.hexdigest(package_file.file.path) + verification_status = (primary_checksum == secondary_checksum) + + status[verification_status.to_s] ||= [] + status[verification_status.to_s] << package_file.id +end + +# Count how many of each value we get +status.keys.each {|key| puts "#{key} count: #{status[key].count}"} + +# See the output in its entirety +status +``` + +### Reverify all uploads (or any SSF data type which is verified) + +1. SSH into a GitLab Rails node in the primary Geo site. +1. Open [Rails console](../../../administration/operations/rails_console.md#starting-a-rails-console-session). +1. Mark all uploads as "pending verification": + + ```ruby + Upload.verification_state_table_class.each_batch do |relation| + relation.update_all(verification_state: 0) + end + ``` + +1. This will cause the primary to start checksumming all Uploads. +1. When a primary successfully checksums a record, then all secondaries rechecksum as well, and they compare the values. + +For other SSF data types replace `Upload` in the command above with the desired model class. + +NOTE: +There is an [issue to implement this functionality in the Admin Area UI](https://gitlab.com/gitlab-org/gitlab/-/issues/364729). + +### Repository types, except for project or project wiki repositories + +- `SnippetRepository` +- `GroupWikiRepository` + +`SnippetRepository` is used in the examples below, but things generally work the same for the other Repository types. + +[Start a Rails console session](../../../administration/operations/rails_console.md#starting-a-rails-console-session) +to enact the following, basic troubleshooting steps. + +WARNING: +Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. + +#### The Replicator + +The main kinds of classes are Registry, Model, and Replicator. If you have an instance of one of these classes, you can get the others. The Registry and Model mostly manage PostgreSQL DB state. The Replicator knows how to replicate/verify (or it can call a service to do it). + +```ruby +model_record = SnippetRepository.last +model_record.replicator.registry.replicator.model_record # just showing that these methods exist +``` + +#### Replicate a snippet repository, synchronously, given an ID + +```ruby +model_record = SnippetRepository.find(id) +model_record.replicator.send(:sync_repository) +``` + +#### Replicate a snippet repository, synchronously, given a registry ID + +```ruby +registry = Geo::SnippetRepositoryRegistry.find(registry_id) +registry.replicator.send(:sync_repository) +``` + +### Find failed artifacts + +[Start a Rails console session](../../../administration/operations/rails_console.md#starting-a-rails-console-session) +to run the following commands: + +```ruby +Geo::JobArtifactRegistry.failed +``` + +#### Find `ID` of synced artifacts that are missing on primary + +```ruby +Geo::JobArtifactRegistry.synced.missing_on_primary.pluck(:artifact_id) +``` + +### Project or project wiki repositories + +#### Find repository verification failures + +[Start a Rails console session](../../../administration/operations/rails_console.md#starting-a-rails-console-session) +to gather the following, basic troubleshooting information. + +WARNING: +Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. + +##### Get the number of verification failed repositories + +```ruby +Geo::ProjectRegistry.verification_failed('repository').count +``` + +##### Find the verification failed repositories + +```ruby +Geo::ProjectRegistry.verification_failed('repository') +``` + +##### Find repositories that failed to sync + +```ruby +Geo::ProjectRegistry.sync_failed('repository') +``` + +#### Resync project and project wiki repositories + +[Start a Rails console session](../../../administration/operations/rails_console.md#starting-a-rails-console-session) +to enact the following, basic troubleshooting steps. + +WARNING: +Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. + +##### Queue up all repositories for resync + +When you run this, Sidekiq handles each sync. + +```ruby +Geo::ProjectRegistry.update_all(resync_repository: true, resync_wiki: true) +``` + +##### Sync individual repository now + +```ruby +project = Project.find_by_full_path('<group/project>') + +Geo::RepositorySyncService.new(project).execute +``` + ## Fixing client errors ### Authorization errors from LFS HTTP(S) client requests @@ -1390,10 +1515,6 @@ If the above steps are **not successful**, proceed through the next steps: 1. Verify you can connect to the newly-promoted **primary** site using the URL used previously for the **secondary** site. 1. If successful, the **secondary** site is now promoted to the **primary** site. -## Additional tools - -There are useful snippets for manipulating Geo internals in the [GitLab Rails Cheat Sheet](../../troubleshooting/gitlab_rails_cheat_sheet.md#geo). For example, you can find how to manually sync or verify a replicable in Rails console. - ## Check OS locale data compatibility If different operating systems or different operating system versions are deployed across Geo sites, we recommend that you perform a locale data compatibility check setting up Geo. diff --git a/doc/administration/geo/secondary_proxy/location_aware_external_url.md b/doc/administration/geo/secondary_proxy/location_aware_external_url.md index b983230a314..b71b813ee9f 100644 --- a/doc/administration/geo/secondary_proxy/location_aware_external_url.md +++ b/doc/administration/geo/secondary_proxy/location_aware_external_url.md @@ -15,10 +15,6 @@ advantage of closer Geo sites as they move. With [Geo proxying for secondary sites](index.md) web and Git requests are proxied from **secondary** sites to the **primary**. -Though these instructions use [AWS Route53](https://aws.amazon.com/route53/), -other services such as [Cloudflare](https://www.cloudflare.com/) can be used -as well. - ## Prerequisites This example creates a `gitlab.example.com` subdomain that automatically directs @@ -36,17 +32,20 @@ For this example, you need: - A working GitLab **primary** site that is accessible at `gitlab.example.com` _and_ `primary.example.com`. - A working GitLab **secondary** site. -- A Route53 Hosted Zone managing your domain for the Route53 setup. +- A DNS zone managing your domain. Although the following instructions use + [AWS Route53](https://aws.amazon.com/route53/) + and [GCP cloud DNS](https://cloud.google.com/dns/), other services such as + [Cloudflare](https://www.cloudflare.com/) can be used as well. If you haven't yet set up a Geo _primary_ site and _secondary_ site, see the [Geo setup instructions](../index.md#setup-instructions). ## AWS Route53 -### Create a traffic policy +In this example, you use a Route53 Hosted Zone managing your domain for the Route53 setup. In a Route53 Hosted Zone, traffic policies can be used to set up a variety of -routing configurations. +routing configurations. To create a traffic policy: 1. Go to the [Route53 dashboard](https://console.aws.amazon.com/route53/home) and select @@ -78,6 +77,30 @@ routing configurations. You have successfully set up a single host, like `gitlab.example.com`, which distributes traffic to your Geo sites by geolocation. +## GCP + +In this example, you create a GCP Cloud DNS zone managing your domain. + +When creating Geo-Based record sets, GCP applies a nearest match for the source region when the source of the traffic doesn't match any policy items exactly. To create a Geo-Based record set: + +1. Select **Network Services** > **Cloud DNS**. +1. Select the Zone configured for your domain. +1. Select **Add Record Set**. +1. Enter the DNS Name for your Location-aware public URL e.g. `gitlab.example.com`. +1. Select the **Routing Policy**: **Geo-Based**. +1. Select **Add Managed RRData**. + 1. Select **Source Region**: **us-central1**. + 1. Enter your `<**primary** IP address>`. + 1. Select **Done**. +1. Select **Add Managed RRData**. + 1. Select **Source Region**: **europe-west1**. + 1. Enter your `<**secondary** IP address>`. + 1. Select **Done**. +1. Select **Create**. + +You have successfully set up a single host, like `gitlab.example.com`, which +distributes traffic to your Geo sites using a location-aware URL. + ## Enable Geo proxying for secondary sites After setting up a single URL to use for all Geo sites, continue with the [steps to enable Geo proxying for secondary sites](index.md). diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md index 8ea8d6c4d8e..86caf5306b5 100644 --- a/doc/administration/geo/setup/database.md +++ b/doc/administration/geo/setup/database.md @@ -478,6 +478,12 @@ data before running `pg_basebackup`. Each Geo **secondary** site must have its own unique replication slot name. Using the same slot name between two secondaries breaks PostgreSQL replication. + NOTE: + Replication slot names must only contain lowercase letters, numbers, and the underscore character. + + When prompted, enter the _plaintext_ password you set up for the `gitlab_replicator` + user in the first step. + ```shell gitlab-ctl replicate-geo-database \ --slot-name=<secondary_site_name> \ @@ -486,12 +492,6 @@ data before running `pg_basebackup`. ``` NOTE: - Replication slot names must only contain lowercase letters, numbers, and the underscore character. - - When prompted, enter the _plaintext_ password you set up for the `gitlab_replicator` - user in the first step. - - NOTE: If you have generated custom PostgreSQL certificates, you will want to use `--sslmode=verify-full` (or omit the `sslmode` line entirely), to benefit from the extra validation of the full host name in the certificate CN / SAN for additional security. @@ -619,7 +619,7 @@ If you still haven't [migrated from repmgr to Patroni](#migrating-from-repmgr-to 1. Before migrating, we recommend that there is no replication lag between the **primary** and **secondary** sites and that replication is paused. In GitLab 13.2 and later, you can pause and resume replication with `gitlab-ctl geo-replication-pause` and `gitlab-ctl geo-replication-resume` on a Geo secondary database node. 1. Follow the [instructions to migrate repmgr to Patroni](../../postgresql/replication_and_failover.md#switching-from-repmgr-to-patroni). When configuring Patroni on each **primary** site database node, add `patroni['replication_slots'] = { '<slot_name>' => 'physical' }` to `gitlab.rb` where `<slot_name>` is the name of the replication slot for your **secondary** site. This ensures that Patroni recognizes the replication slot as permanent and not drop it upon restarting. -1. If database replication to the **secondary** site was paused before migration, resume replication once Patroni is confirmed working on the **primary** site. +1. If database replication to the **secondary** site was paused before migration, resume replication after Patroni is confirmed working on the **primary** site. ### Migrating a single PostgreSQL node to Patroni diff --git a/doc/administration/geo/setup/external_database.md b/doc/administration/geo/setup/external_database.md index eabed7c10f3..0fefc11f078 100644 --- a/doc/administration/geo/setup/external_database.md +++ b/doc/administration/geo/setup/external_database.md @@ -62,7 +62,7 @@ developed and tested. We aim to be compatible with most external To set up an external database, you can either: -- Set up [streaming replication](https://www.postgresql.org/docs/12/warm-standby.html#STREAMING-REPLICATION-SLOTS) yourself (for example Amazon RDS, bare metal not managed by Omnibus, and so on). +- Set up [streaming replication](https://www.postgresql.org/docs/12/warm-standby.html#STREAMING-REPLICATION-SLOTS) yourself (for example Amazon RDS, or bare metal not managed by Omnibus). - Perform the Omnibus configuration manually as follows. #### Leverage your cloud provider's tools to replicate the primary database @@ -78,7 +78,7 @@ cloud providers: - Azure Database for PostgreSQL - [Create and manage read replicas in Azure Database for PostgreSQL](https://learn.microsoft.com/en-us/azure/postgresql/single-server/how-to-read-replicas-portal) - Google Cloud SQL - [Creating read replicas](https://cloud.google.com/sql/docs/postgres/replication/create-replica) -Once your read-only replica is set up, you can skip to [configure your secondary site](#configure-secondary-site-to-use-the-external-read-replica) +When your read-only replica is set up, you can skip to [configure your secondary site](#configure-secondary-site-to-use-the-external-read-replica) #### Manually configure the primary database for replication diff --git a/doc/administration/git_protocol.md b/doc/administration/git_protocol.md index f900c5a6867..349a92de51e 100644 --- a/doc/administration/git_protocol.md +++ b/doc/administration/git_protocol.md @@ -25,7 +25,7 @@ From the server side, if we want to configure SSH we need to set the `sshd` server to accept the `GIT_PROTOCOL` environment. In installations using [GitLab Helm Charts](https://docs.gitlab.com/charts/) -and [All-in-one Docker image](https://docs.gitlab.com/omnibus/docker/), the SSH +and [All-in-one Docker image](../install/docker.md), the SSH service is already configured to accept the `GIT_PROTOCOL` environment. Users need not do anything more. @@ -36,7 +36,7 @@ the SSH configuration of your server manually by adding this line to the `/etc/s AcceptEnv GIT_PROTOCOL ``` -Once configured, restart the SSH daemon for the change to take effect: +When you have configured the SSH daemon, restart it for the change to take effect: ```shell # CentOS 6 / RHEL 6 diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md index e4aef2db9a8..e7a6fc35ced 100644 --- a/doc/administration/gitaly/configure_gitaly.md +++ b/doc/administration/gitaly/configure_gitaly.md @@ -57,21 +57,6 @@ The process for setting up Gitaly on its own server is: 1. [Configure Gitaly clients](#configure-gitaly-clients). 1. [Disable Gitaly where not required](#disable-gitaly-where-not-required-optional) (optional). -When running Gitaly on its own server, note the following regarding GitLab versions: - -- From GitLab 11.4, Gitaly was able to serve all Git requests without requiring a shared NFS mount - for Git repository data, except for the - [Elasticsearch indexer](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). -- From GitLab 11.8, the Elasticsearch indexer also uses Gitaly for data access. NFS can still be - leveraged for redundancy on block-level Git data, but should be mounted only on the Gitaly - servers. -- From GitLab 11.8 to 12.2, it is possible to use Elasticsearch in a Gitaly setup that doesn't use - NFS. To use Elasticsearch in these versions, the - [repository indexer](../../integration/advanced_search/elasticsearch.md#elasticsearch-repository-indexer) - must be enabled in your GitLab configuration. -- [In GitLab 12.3 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/6481), the new indexer is - the default and no configuration is required. - ### Network architecture The following list depicts the network architecture of Gitaly: @@ -1100,8 +1085,8 @@ benefit from it. It is orthogonal to: - The transport (HTTP or SSH). - Git protocol version (v0 or v2). -- The type of fetch (full clones, incremental fetches, shallow clones, - partial clones, and so on). +- The type of fetch, such as full clones, incremental fetches, shallow clones, + or partial clones. The strength of this cache is its ability to deduplicate concurrent identical fetches. It: @@ -1309,18 +1294,38 @@ following keys (in this example, to disable the `hasDotgit` consistency check): - In [GitLab 15.3](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6800) and later: ```ruby + ignored_blobs = "/etc/gitlab/instance_wide_ignored_git_blobs.txt" + gitaly['gitconfig'] = [ + + # Populate a file with one unabbreviated SHA-1 per line. + # See https://git-scm.com/docs/git-config#Documentation/git-config.txt-fsckskipList + { key: "fsck.skipList", value: ignored_blobs }, + { key: "fetch.fsck.skipList", value: ignored_blobs }, + { key: "receive.fsck.skipList", value: ignored_blobs }, + { key: "fsck.hasDotgit", value: "ignore" }, { key: "fetch.fsck.hasDotgit", value: "ignore" }, - { key: "receive.fsck.hasDotgit", value: "ignore "}, + { key: "receive.fsck.hasDotgit", value: "ignore" }, + { key: "fsck.missingSpaceBeforeEmail", value: "ignore" }, ] ``` - In GitLab 15.2 and earlier (legacy method): ```ruby - ignored_git_errors = ["hasDotgit = ignore"] + ignored_git_errors = [ + "hasDotgit = ignore", + "missingSpaceBeforeEmail = ignore", + ] omnibus_gitconfig['system'] = { + + # Populate a file with one unabbreviated SHA-1 per line. + # See https://git-scm.com/docs/git-config#Documentation/git-config.txt-fsckskipList + "fsck.skipList" => ignored_blobs + "fetch.fsck.skipList" => ignored_blobs, + "receive.fsck.skipList" => ignored_blobs, + "fsck" => ignored_git_errors, "fetch.fsck" => ignored_git_errors, "receive.fsck" => ignored_git_errors, @@ -1342,6 +1347,30 @@ value = "ignore" [[git.config]] key = "receive.fsck.hasDotgit" value = "ignore" + +[[git.config]] +key = "fsck.missingSpaceBeforeEmail" +value = "ignore" + +[[git.config]] +key = "fetch.fsck.missingSpaceBeforeEmail" +value = "ignore" + +[[git.config]] +key = "receive.fsck.missingSpaceBeforeEmail" +value = "ignore" + +[[git.config]] +key = "fsck.skipList" +value = "/etc/gitlab/instance_wide_ignored_git_blobs.txt" + +[[git.config]] +key = "fetch.fsck.skipList" +value = "/etc/gitlab/instance_wide_ignored_git_blobs.txt" + +[[git.config]] +key = "receive.fsck.skipList" +value = "/etc/gitlab/instance_wide_ignored_git_blobs.txt" ``` ## Configure commit signing for GitLab UI commits diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index 96447239116..1b7e1f0f7c6 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -45,7 +45,7 @@ repository storage is either: ## Before deploying Gitaly Cluster Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. -Before deploying Gitaly Cluster, please review: +Before deploying Gitaly Cluster, review: - Existing [known issues](#known-issues). - [Snapshot limitations](#snapshot-backup-and-recovery-limitations). @@ -66,13 +66,14 @@ Contact your Technical Account Manager or customer support if you have any quest ### Known issues The following table outlines current known issues impacting the use of Gitaly Cluster. For -the current status of these issues, please refer to the referenced issues and epics. +the current status of these issues, refer to the referenced issues and epics. | Issue | Summary | How to avoid | |:--------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------| | Gitaly Cluster + Geo - Issues retrying failed syncs | If Gitaly Cluster is used on a Geo secondary site, repositories that have failed to sync could continue to fail when Geo tries to resync them. Recovering from this state requires assistance from support to run manual steps. | No known solution prior to GitLab 15.0. In GitLab 15.0 to 15.2, enable the [`gitaly_praefect_generated_replica_paths` feature flag](#praefect-generated-replica-paths-gitlab-150-and-later). In GitLab 15.3, the feature flag is enabled by default. | | Praefect unable to insert data into the database due to migrations not being applied after an upgrade | If the database is not kept up to date with completed migrations, then the Praefect node is unable to perform normal operation. | Make sure the Praefect database is up and running with all migrations completed (For example: `/opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-migrate-status` should show a list of all applied migrations). Consider [requesting upgrade assistance](https://about.gitlab.com/support/scheduling-upgrade-assistance/) so your upgrade plan can be reviewed by support. | -| Restoring a Gitaly Cluster node from a snapshot in a running cluster | Because the Gitaly Cluster runs with consistent state, introducing a single node that is behind will result in the cluster not being able to reconcile the nodes data and other nodes data | Don't restore a single Gitaly Cluster node from a backup snapshot. If you must restore from backup, it's best to [shut down GitLab](../read_only_gitlab.md#shut-down-the-gitlab-ui), snapshot all Gitaly Cluster nodes at the same time and take a database dump of the Praefect database. | +| Restoring a Gitaly Cluster node from a snapshot in a running cluster | Because the Gitaly Cluster runs with consistent state, introducing a single node that is behind results in the cluster not being able to reconcile the nodes data and other nodes data | Don't restore a single Gitaly Cluster node from a backup snapshot. If you must restore from backup:<br/><br/>1. [Shut down GitLab](../read_only_gitlab.md#shut-down-the-gitlab-ui).<br/>2. Snapshot all Gitaly Cluster nodes at the same time.<br/>3. Take a database dump of the Praefect database. | +| Rebuilding or replacing an existing Gitaly Cluster node | There is no way to replace existing nodes in place because the Praefect database is relied on to determine the current state of each Gitaly node. Changing how Gitaly Cluster stores repositories is proposed in issue [4218](https://gitlab.com/gitlab-org/gitaly/-/issues/4218). Turning on [repository verification](praefect.md#repository-verification) is proposed in issue [4429](https://gitlab.com/gitlab-org/gitaly/-/issues/4429) so Praefect can detect that data is missing and needs to replicated to a new Gitaly node. | No known solution at this time. | ### Snapshot backup and recovery limitations @@ -83,11 +84,11 @@ during a restore, we recommend using the [official backup and restore Rake tasks The [incremental backup method](../../raketasks/backup_gitlab.md#incremental-repository-backups) can be used to speed up Gitaly Cluster backups. -If you are unable to use either method, please contact customer support for restoration help. +If you are unable to use either method, contact customer support for restoration help. ### What to do if you are on Gitaly Cluster experiencing an issue or limitation -Please contact customer support for immediate help in restoration or recovery. +Contact customer support for immediate help in restoration or recovery. ## Directly accessing repositories @@ -219,8 +220,7 @@ The availability objectives for Gitaly clusters assuming a single node failure a second. Failover requires ten consecutive failed health checks on each Praefect node. - Faster outage detection, to improve this speed to less than 1 second, - is tracked [in this issue](https://gitlab.com/gitlab-org/gitaly/-/issues/2608). +Improvements to RPO and RTO are proposed in epic [8903](https://gitlab.com/groups/gitlab-org/-/epics/8903). WARNING: If complete cluster failure occurs, disaster recovery plans should be executed. These can affect the @@ -320,9 +320,7 @@ follow the [hashed storage](../repository_storage_types.md#hashed-storage) schem > - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/4218) in GitLab 15.0 [with a flag](../feature_flags.md) named `gitaly_praefect_generated_replica_paths`. Disabled by default. > - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitaly/-/issues/4218) in GitLab 15.2. > - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4809) in GitLab 15.3. - -FLAG: -On self-managed GitLab, by default this feature is available. To hide the feature, ask an administrator to [disable the feature flag](../feature_flags.md) named `gitaly_praefect_generated_replica_paths`. On GitLab.com, this feature is available but can be configured by GitLab.com administrators only. +> - [Generally available](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4941) in GitLab 15.6. Feature flag `gitaly_praefect_generated_replica_paths` removed. When Gitaly Cluster creates a repository, it assigns the repository a unique and permanent ID called the _repository ID_. The repository ID is internal to Gitaly Cluster and doesn't relate to any IDs elsewhere in GitLab. If a repository is removed from Gitaly Cluster and later moved @@ -410,7 +408,7 @@ relative path of the repository in the metadata store. ### Moving beyond NFS Engineering support for NFS for Git repositories is deprecated. Technical support is planned to be unavailable starting -November 22, 2022. Please see our [statement of support](https://about.gitlab.com/support/statement-of-support/#gitaly-and-nfs) +November 22, 2022. See our [statement of support](https://about.gitlab.com/support/statement-of-support/#gitaly-and-nfs) for more details. [Network File System (NFS)](https://en.wikipedia.org/wiki/Network_File_System) @@ -470,12 +468,6 @@ including [horizontally distributing reads](https://gitlab.com/groups/gitlab-org #### Distributed reads -> - Introduced in GitLab 13.1 in [beta](../../policy/alpha-beta-support.md#beta-features) with feature flag `gitaly_distributed_reads` set to disabled. -> - [Made generally available and enabled by default](https://gitlab.com/gitlab-org/gitaly/-/issues/2951) in GitLab 13.3. -> - [Disabled by default](https://gitlab.com/gitlab-org/gitaly/-/issues/3178) in GitLab 13.5. -> - [Enabled by default](https://gitlab.com/gitlab-org/gitaly/-/issues/3334) in GitLab 13.8. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitaly/-/issues/3383) in GitLab 13.11. - Gitaly Cluster supports distribution of read operations across Gitaly nodes that are configured for the [virtual storage](#virtual-storage). @@ -496,31 +488,21 @@ You can [monitor distribution of reads](monitoring.md#monitor-gitaly-cluster) us #### Strong consistency -> - Introduced in GitLab 13.1 in [alpha](../../policy/alpha-beta-support.md#alpha-features), disabled by default. -> - Entered [beta](../../policy/alpha-beta-support.md#beta-features) in GitLab 13.2, disabled by default. -> - In GitLab 13.3, disabled unless primary-wins voting strategy is disabled. -> - From GitLab 13.4, enabled by default. -> - From GitLab 13.5, you must use Git v2.28.0 or higher on Gitaly nodes to enable strong consistency. -> - From GitLab 13.6, primary-wins voting strategy and the `gitaly_reference_transactions_primary_wins` feature flag was removed. -> - From GitLab 14.0, [Gitaly Cluster only supports strong consistency](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3575), and the `gitaly_reference_transactions` feature flag was removed. +> - In GitLab 13.6 to 13.12, strong consistency must be manually configured. Refer to [the 13.12 documentation](https://docs.gitlab.com/13.12/ee/administration/gitaly/praefect.html#strong-consistency). +> - In GitLab 14.0, strong consistency is the primary replication method. Gitaly Cluster provides strong consistency by writing changes synchronously to all healthy, up-to-date replicas. If a replica is outdated or unhealthy at the time of the transaction, the write is asynchronously replicated to it. +Strong consistency is the primary replication method. A subset of operations still use replication jobs +(eventual consistency) instead of strong consistency. Refer to the +[strong consistency epic](https://gitlab.com/groups/gitlab-org/-/epics/1189) for more information. + If strong consistency is unavailable, Gitaly Cluster guarantees eventual consistency. In this case. Gitaly Cluster replicates all writes to secondary Gitaly nodes after the write to the primary Gitaly node has occurred. -Strong consistency: - -- Is the primary replication method in GitLab 14.0 and later. A subset of operations still use replication jobs - (eventual consistency) instead of strong consistency. Refer to the - [strong consistency epic](https://gitlab.com/groups/gitlab-org/-/epics/1189) for more information. -- Must be configured in GitLab versions 13.1 to 13.12. For configuration information, refer to either: - - Documentation on your GitLab instance at `/help`. - - The [13.12 documentation](https://docs.gitlab.com/13.12/ee/administration/gitaly/praefect.html#strong-consistency). -- Is unavailable in GitLab 13.0 and earlier. - -For more information on monitoring strong consistency, see the Gitaly Cluster [Prometheus metrics documentation](monitoring.md#monitor-gitaly-cluster). +For more information on monitoring strong consistency, see the Gitaly Cluster +[Prometheus metrics documentation](monitoring.md#monitor-gitaly-cluster). #### Replication factor @@ -598,7 +580,7 @@ To migrate to Gitaly Cluster: 1. Create the required storage. Refer to [repository storage recommendations](praefect.md#repository-storage-recommendations). 1. Create and configure [Gitaly Cluster](praefect.md). -1. Configure the existing Gitaly instance [to use TPC](praefect.md#use-tcp-for-existing-gitlab-instances), if not already configured that way. +1. Configure the existing Gitaly instance [to use TCP](praefect.md#use-tcp-for-existing-gitlab-instances), if not already configured that way. 1. [Move the repositories](../operations/moving_repositories.md#move-repositories). To migrate to Gitaly Cluster, existing repositories stored outside Gitaly Cluster must be moved. There is no automatic migration but the moves can be scheduled with the GitLab API. @@ -718,4 +700,4 @@ The second facet presents the only real solution. For this, we developed ## NFS deprecation notice Engineering support for NFS for Git repositories is deprecated. Technical support is planned to be -unavailable beginning November 22, 2022. For further information, please see our [NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation) documentation. +unavailable beginning November 22, 2022. For further information, see our [NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation) documentation. diff --git a/doc/administration/gitaly/monitoring.md b/doc/administration/gitaly/monitoring.md index 9b7acd536b3..8d4f30c7c20 100644 --- a/doc/administration/gitaly/monitoring.md +++ b/doc/administration/gitaly/monitoring.md @@ -152,9 +152,6 @@ The following metrics are available from the `/metrics` endpoint: for replication to complete after the replication job starts. Available in GitLab 12.10 and later. - `gitaly_praefect_replication_delay_bucket`, a histogram measuring how much time passes between when the replication job is created and when it starts. Available in GitLab 12.10 and later. -- `gitaly_praefect_node_latency_bucket`, a histogram measuring the latency in Gitaly returning - health check information to Praefect. This indicates Praefect connection saturation. Available in - GitLab 12.10 and later. - `gitaly_praefect_connections_total`, the total number of connections to Praefect. [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4220) in GitLab 14.7. To monitor [strong consistency](index.md#strong-consistency), you can use the following Prometheus metrics: diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index e3b198d1012..8cf32a6aaa3 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -28,6 +28,9 @@ The minimum recommended configuration for a Gitaly Cluster requires: - 3 Praefect nodes - 3 Gitaly nodes (1 primary, 2 secondary) +You should configure an odd number of Gitaly nodes so that transactions have a tie-breaker in case one of the +Gitaly nodes fails in a mutating RPC call. + See the [design document](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/design_ha.md) for implementation details. @@ -155,6 +158,14 @@ We note in the instructions below where these secrets are required. NOTE: Omnibus GitLab installations can use `gitlab-secrets.json` for `GITLAB_SHELL_SECRET_TOKEN`. +### Customize time server setting + +By default, Gitaly and Praefect nodes use the time server at `pool.ntp.org` for time synchronization checks. You can customize this setting by adding the +following to `gitlab.rb` on each node: + +- `gitaly['env'] = { "NTP_HOST" => "ntp.example.com" }`, for Gitaly nodes. +- `praefect['env'] = { "NTP_HOST" => "ntp.example.com" }`, for Praefect nodes. + ### PostgreSQL NOTE: @@ -285,7 +296,7 @@ praefect['database_direct_dbname'] = 'praefect_production' #praefect['database_direct_sslrootcert'] = '...' ``` -Once configured, this connection is automatically used for the +When configured, this connection is automatically used for the [SQL LISTEN](https://www.postgresql.org/docs/11/sql-listen.html) feature and allows Praefect to receive notifications from PostgreSQL for cache invalidation. @@ -584,11 +595,7 @@ Updates to example must be made at: } ``` - NOTE: - In [GitLab 13.8 and earlier](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4988), - Gitaly nodes were configured directly under the virtual storage, and not under the `nodes` key. - -1. [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2013) in GitLab 13.1 and later, enable [distribution of reads](index.md#distributed-reads). +1. Enable [distribution of reads](index.md#distributed-reads). 1. Save the changes to `/etc/gitlab/gitlab.rb` and [reconfigure Praefect](../restart_gitlab.md#omnibus-gitlab-reconfigure): @@ -641,8 +648,6 @@ Updates to example must be made at: #### Enable TLS support -> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/1698) in GitLab 13.2. - Praefect supports TLS encryption. To communicate with a Praefect instance that listens for secure connections, you must: @@ -1369,7 +1374,8 @@ We recommend using [repository-specific primary nodes](#repository-specific-prim ### Repository-specific primary nodes -> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/3492) in GitLab 13.12. +> - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/3492) in GitLab 13.12, with primary elections run when Praefect starts or the cluster's consensus of a Gitaly node's health changes. +> - [Changed](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3543) in GitLab 14.1, primary elections are run lazily. Gitaly Cluster supports electing repository-specific primary Gitaly nodes. Repository-specific Gitaly primary nodes are enabled in `/etc/gitlab/gitlab.rb` by setting @@ -1386,14 +1392,8 @@ The `per_repository` election strategy solves this problem by electing a primary repository. Combined with [configurable replication factors](#configure-replication-factor), you can horizontally scale storage capacity and distribute write load across Gitaly nodes. -Primary elections are run: - -- In GitLab 14.1 and later, lazily. This means that Praefect doesn't immediately elect - a new primary node if the current one is unhealthy. A new primary is elected if it is - necessary to serve a request while the current primary is unavailable. -- In GitLab 13.12 to GitLab 14.0 when: - - Praefect starts up. - - The cluster's consensus of a Gitaly node's health changes. +Primary elections are run lazily. Praefect doesn't immediately elect a new primary node if the current +one is unhealthy. A new primary is elected if a request must be served while the current primary is unavailable. A valid primary node candidate is a Gitaly node that: @@ -1422,9 +1422,9 @@ To migrate existing clusters: 1. Praefect nodes didn't historically keep database records of every repository stored on the cluster. When the `per_repository` election strategy is configured, Praefect expects to have database records of - each repository. A [background database migration](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/2749) is - included in GitLab 13.6 and later to create any missing database records for repositories. Before migrating, - check Praefect's logs to verify that the database migration ran. + each repository. A [background database migration](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/2749) + creates any missing database records for repositories. Before migrating, check Praefect's logs to verify + that the database migration ran. Check Praefect's logs for `repository importer finished` message. The `virtual_storages` field contains the names of virtual storages and whether they've had any missing database records created. diff --git a/doc/administration/gitaly/recovery.md b/doc/administration/gitaly/recovery.md index b51454aa44e..56894f3e963 100644 --- a/doc/administration/gitaly/recovery.md +++ b/doc/administration/gitaly/recovery.md @@ -11,40 +11,15 @@ recovery and has Praefect tracking database tools. ## Primary node failure -Gitaly Cluster recovers from a failing primary Gitaly node by promoting a healthy secondary as the -new primary. +> - Introduced in GitLab 13.0, Gitaly Cluster, elects the secondary with the least unreplicated writes from the primary to be the new primary. There can still be some unreplicated writes, so [data loss can occur](#check-for-data-loss). +> - Primary node failure recovery support added in GitLab 14.1. -In GitLab 14.1 and later, Gitaly Cluster: +Gitaly Cluster recovers from a failing primary Gitaly node by promoting a healthy secondary as the new primary. Gitaly +Cluster: - Elects a healthy secondary with a fully up to date copy of the repository as the new primary. - Repository becomes unavailable if there are no fully up to date copies of it on healthy secondaries. -To minimize data loss in GitLab 13.0 to 14.0, Gitaly Cluster: - -- Switches repositories that are outdated on the new primary to [read-only mode](#read-only-mode). -- Elects the secondary with the least unreplicated writes from the primary to be the new - primary. Because there can still be some unreplicated writes, - [data loss can occur](#check-for-data-loss). - -### Read-only mode - -> - Introduced in GitLab 13.0 as [generally available](../../policy/alpha-beta-support.md#generally-available-ga). -> - Between GitLab 13.0 and GitLab 13.2, read-only mode applied to the whole virtual storage and occurred whenever failover occurred. -> - [In GitLab 13.3 and later](https://gitlab.com/gitlab-org/gitaly/-/issues/2862), read-only mode applies on a per-repository basis and only occurs if a new primary is out of date. If the failed primary contained unreplicated writes, [data loss can occur](#check-for-data-loss). -> - Removed in GitLab 14.1. Instead, repositories [become unavailable](#unavailable-repositories). - -When Gitaly Cluster switches to a new primary in GitLab 13.0 to 14.0, repositories enter read-only mode if they are -out-of-date. This can happen after failing over to an outdated secondary. Read-only mode eases data recovery efforts by -preventing writes that may conflict with the unreplicated writes on other nodes. - -To enable writes again in GitLab 13.0 to 14.0, an administrator can: - -1. [Check](#check-for-data-loss) for data loss. -1. Attempt to [recover](#data-recovery) missing data. -1. Either [enable writes](#enable-writes-or-accept-data-loss) in the virtual storage or - [accept data loss](#enable-writes-or-accept-data-loss) if necessary, depending on the version of - GitLab. - ### Unavailable repositories > - From GitLab 13.0 through 14.0, repositories became read-only if they were outdated on the primary but fully up to date on a healthy secondary. `dataloss` sub-command displays read-only repositories by default through these versions. @@ -144,9 +119,7 @@ Virtual storage: default #### Unavailable replicas of available repositories -NOTE: -In GitLab 14.0 and earlier, the flag is `-partially-replicated` and the output shows any repositories with assigned nodes with outdated -copies. +> Introduced in GitLab 14.0, flag renamed from `-partially-replicated` and behavior changed. To also list information of repositories which are available but are unavailable from some of the assigned nodes, use the `-partially-unavailable` flag. @@ -209,26 +182,17 @@ WARNING: `accept-dataloss` causes permanent data loss by overwriting other versions of the repository. Data [recovery efforts](#data-recovery) must be performed before using it. -Praefect provides the following subcommands to re-enable writes or accept data loss: - -- In GitLab 13.2 and earlier, `enable-writes` to re-enable virtual storage for writes after - data recovery attempts: +Praefect provides the following subcommands to re-enable writes or accept data loss. If it is not possible to bring one +of the up-to-date nodes back online, you might have to accept data loss: - ```shell - sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml enable-writes -virtual-storage <virtual-storage> - ``` - -- In GitLab 13.3 and later, if it is not possible to bring one of the up to date nodes back - online, you may have to accept data loss: - - ```shell - sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml accept-dataloss -virtual-storage <virtual-storage> -repository <relative-path> -authoritative-storage <storage-name> - ``` +```shell +sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml accept-dataloss -virtual-storage <virtual-storage> -repository <relative-path> -authoritative-storage <storage-name> +``` - When accepting data loss, Praefect: +When accepting data loss, Praefect: - 1. Marks the chosen copy of the repository as the latest version. - 1. Replicates the copy to the other assigned Gitaly nodes. +1. Marks the chosen copy of the repository as the latest version. +1. Replicates the copy to the other assigned Gitaly nodes. This process overwrites any other copy of the repository so care must be taken. diff --git a/doc/administration/gitaly/reference.md b/doc/administration/gitaly/reference.md index 3bf1e3136c0..8f7dc688e56 100644 --- a/doc/administration/gitaly/reference.md +++ b/doc/administration/gitaly/reference.md @@ -192,8 +192,7 @@ For historical reasons [GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell) contains the Git hooks that allow GitLab to validate and react to Git pushes. Because Gitaly "owns" Git pushes, GitLab Shell must therefore be -installed alongside Gitaly. We plan to -[simplify this](https://gitlab.com/gitlab-org/gitaly/-/issues/1226). +installed alongside Gitaly. | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md index 1c5f0d43864..7edce840396 100644 --- a/doc/administration/gitaly/troubleshooting.md +++ b/doc/administration/gitaly/troubleshooting.md @@ -41,17 +41,6 @@ The `gitaly-debug` command provides "production debugging" tools for Gitaly and performance. It is intended to help production engineers and support engineers investigate Gitaly performance problems. -If you're using GitLab 11.6 or newer, this tool should be installed on -your GitLab or Gitaly server already at `/opt/gitlab/embedded/bin/gitaly-debug`. -If you're investigating an older GitLab version you can compile this -tool offline and copy the executable to your server: - -```shell -git clone https://gitlab.com/gitlab-org/gitaly.git -cd cmd/gitaly-debug -GOOS=linux GOARCH=amd64 go build -o gitaly-debug -``` - To see the help page of `gitaly-debug` for a list of supported sub-commands, run: ```shell @@ -144,9 +133,8 @@ If you have Prometheus set up to scrape your Gitaly process, you can see request rates and error codes for individual RPCs in `gitaly-ruby` by querying `grpc_client_handled_total`. -- In theory, this metric does not differentiate between `gitaly-ruby` and other RPCs. -- In practice from GitLab 11.9, all gRPC calls made by Gitaly itself are internal calls from the - main Gitaly process to one of its `gitaly-ruby` sidecars. +All gRPC calls made by `gitaly-ruby` itself are internal calls from the main Gitaly process to one of its `gitaly-ruby` +sidecars. Assuming your `grpc_client_handled_total` counter only observes Gitaly, the following query shows you RPCs are (most likely) internally @@ -335,7 +323,7 @@ You might see the following in Gitaly and Praefect logs: } ``` -This is a GRPC call +This is a gRPC call [error response code](https://grpc.github.io/grpc/core/md_doc_statuscodes.html). If this error occurs, even though @@ -359,7 +347,7 @@ necessary because [this issue](https://gitlab.com/gitlab-org/gitaly/-/issues/252 If this error occurs even though file permissions are correct, it's likely that the Gitaly node is experiencing [clock drift](https://en.wikipedia.org/wiki/Clock_drift). -Please ensure that the GitLab and Gitaly nodes are synchronized and use an NTP time +Ensure that the GitLab and Gitaly nodes are synchronized and use an NTP time server to keep them synchronized if possible. ### Health check warnings @@ -604,6 +592,16 @@ For each replica, the following metadata is available: | `Valid Primary` | Indicates whether the replica is fit to serve as the primary node. If the repository's primary is not a valid primary, a failover occurs on the next write to the repository if there is another replica that is a valid primary. A replica is a valid primary if:<br><br>- It is stored on a healthy Gitaly node.<br>- It is fully up to date.<br>- It is not targeted by a pending deletion job from decreasing replication factor.<br>- It is assigned. | | `Verified At` | Indicates last successful verification of the replica by the [verification worker](praefect.md#repository-verification). If the replica has not yet been verified, `unverified` is displayed in place of the last successful verification time. Introduced in GitLab 15.0. | +#### Command fails with 'repository not found' + +If the supplied value for `-virtual-storage` is incorrect, the command returns the following error: + +```plaintext +get metadata: rpc error: code = NotFound desc = repository not found +``` + +The documented examples specify `-virtual-storage default`. Check the Praefect server setting `praefect['virtual_storages']` in `/etc/gitlab/gitlab.rb`. + ### Check that repositories are in sync Is [some cases](index.md#known-issues) the Praefect database can get out of sync with the underlying Gitaly nodes. To check that diff --git a/doc/administration/housekeeping.md b/doc/administration/housekeeping.md index 15287b917e7..2d3e937e047 100644 --- a/doc/administration/housekeeping.md +++ b/doc/administration/housekeeping.md @@ -1,43 +1,144 @@ --- stage: Systems -group: Distribution +group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Housekeeping **(FREE SELF)** -GitLab supports and automates housekeeping tasks in your current repository such as: +GitLab supports and automates housekeeping tasks in Git repositories to ensure +that they can be served as efficiently as possible. Housekeeping tasks include: -- Compressing Git objects. +- Compressing Git objects and revisions. - Removing unreachable objects. +- Removing stale data like lock files. +- Maintaining data structures that improve performance. +- Updating object pools to improve object deduplication across forks. -## Configure housekeeping +WARNING: +Do not manually execute Git commands to perform housekeeping in Git +repositories that are controlled by GitLab. Doing so may lead to corrupt +repositories and data loss. + +## Housekeeping strategy + +Gitaly can perform housekeeping tasks in a Git repository in two ways: + +- [Eager housekeeping](#eager-housekeeping) executes specific housekeeping tasks + independent of the state a repository is in. +- [Heuristical housekeeping](#heuristical-housekeeping) executes housekeeping + tasks based on a set of heuristics that determine what housekeeping tasks need + to be executed based on the repository state. + +### Eager housekeeping + +The "eager" housekeeping strategy executes housekeeping tasks in a repository +independent of the repository state. This is the default strategy as used by the +[manual trigger](#manual-trigger) and the [push-based trigger](#push-based-trigger). + +The eager housekeeping strategy is controlled by the GitLab application. +Depending on the trigger that caused the housekeeping job to run, GitLab asks +Gitaly to perform specific housekeeping tasks. Gitaly performs these tasks even +if the repository is in an optimized state. As a result, this strategy can be +inefficient in large repositories where performing the housekeeping tasks may +be slow. + +### Heuristical housekeeping + +> - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/2634) in GitLab 14.9 for the [manual trigger](#manual-trigger) and the [push-based trigger](#push-based-trigger) [with a flag](feature_flags.md) named `optimized_housekeeping`. Disabled by default. +> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/353607) in GitLab 14.10. + +FLAG: +On self-managed GitLab, by default this feature is not available for the [manual trigger](#manual-trigger) and the [push-based trigger](#push-based-trigger). +To make it available, ask an administrator to [enable the feature flag](feature_flags.md) named `optimized_housekeeping`. + +The heuristical (or "opportunistic") housekeeping strategy analyzes the +repository's state and executes housekeeping tasks only when it finds one or +more data structures are insufficiently optimized. This is the strategy used by +[scheduled housekeeping](#scheduled-housekeeping). It can optionally be enabled +for the [manual trigger](#manual-trigger) and the [push-based trigger](#push-based-trigger) +by enabling the `optimized_housekeeping` feature flag. + +Heuristical housekeeping uses the following information to decide on the tasks +it needs to run: + +- The number of loose and stale objects. +- The number of packfiles that contain already-compressed objects. +- The number of loose references. +- The presence of a commit-graph. + +The decision whether any of the analyzed data structures need to be optimized is +based on the size of the repository: + +- Objects are repacked frequently the bigger the total size of all objects. +- References are repacked less frequently the more references there are in + total. + +Gitaly does this to offset the fact that optimizing those data structures takes +more time the bigger they get. It is especially important in large +monorepositories (which receive a lot of traffic) to avoid optimizing them too +frequently. + +## Running housekeeping tasks + +There are different ways in which GitLab runs housekeeping tasks: -GitLab automatically runs `git gc` and `git repack` on repositories after Git pushes: +- A project's administrator can [manually trigger](#manual-trigger) repository + housekeeping tasks. +- GitLab can automatically schedule housekeeping tasks [after a number of Git pushes](#push-based-trigger). +- GitLab can [schedule a job](#scheduled-housekeeping) that runs housekeeping + tasks for all repositories in a configurable time frame. + +### Manual trigger + +Administrators of repositories can manually trigger housekeeping tasks in a +repository. In general this is not required as GitLab knows to automatically run +housekeeping tasks. The manual trigger can be useful when either: + +- A repository is known to require housekeeping. +- Automated push-based scheduling of housekeeping tasks has been disabled. + +To trigger housekeeping tasks manually: + +1. On the top bar, select **Main menu > Projects** and find your project. +1. On the left sidebar, select **Settings > General**. +1. Expand **Advanced**. +1. Select **Run housekeeping**. + +This starts an asynchronous background worker for the project's repository. The +background worker executes `git gc`, which performs a number of optimizations. + +### Push-based trigger + +GitLab automatically runs repository housekeeping tasks after a configured +number of pushes: - [`git gc`](https://git-scm.com/docs/git-gc) runs a number of housekeeping tasks such as: - Compressing Git objects to reduce disk space and increase performance. - Removing unreachable objects that may have been created from changes to the repository, like force-overwriting branches. - [`git repack`](https://git-scm.com/docs/git-repack) either: - - Runs an incremental repack, according to a [configured period](#housekeeping-options). This + - Runs an incremental repack, according to a [configured period](#configure-push-based-maintenance). This packs all loose objects into a new packfile and prunes the now-redundant loose objects. - - Runs a full repack, according to a [configured period](#housekeeping-options). This repacks all + - Runs a full repack, according to a [configured period](#configure-push-based-maintenance). This repacks all packfiles and loose objects into a single new packfile, and deletes the old now-redundant loose objects and packfiles. It also optionally creates bitmaps for the new packfile. +- [`git pack-refs`](https://git-scm.com/docs/git-pack-refs) compresses references + stored as loose files into a single file. + +#### Configure push-based maintenance -You can change how often this happens or turn it off: +You can change how often these tasks run when pushes occur, or you can turn +them off entirely: 1. On the top bar, select **Main menu > Admin**. 1. On the left sidebar, select **Settings > Repository**. 1. Expand **Repository maintenance**. -1. In the **Housekeeping** section, configure the [housekeeping options](#housekeeping-options). +1. In the **Housekeeping** section, configure the housekeeping options. 1. Select **Save changes**. -### Housekeeping options - The following housekeeping options are available: -- **Enable automatic repository housekeeping**: Regularly run `git repack` and `git gc`. If you +- **Enable automatic repository housekeeping**: Regularly run housekeeping tasks. If you keep this setting disabled for a long time, Git repository access on your GitLab server becomes slower and your repositories use more disk space. - **Incremental repack period**: Number of Git pushes after which an incremental `git repack` is @@ -60,30 +161,80 @@ Housekeeping also [removes unreferenced LFS files](../raketasks/cleanup.md#remov from your project on the same schedule as the `git gc` operation, freeing up storage space for your project. -WARNING: -Running `git gc` or `git repack` commands manually in the -[repository folder](repository_storage_types.md#from-project-name-to-hashed-path) -is discouraged. If the created pack files get incorrect access rights (that is, owned by the wrong user) -browsing to the project page might result in `404` and `503` errors. - -## How housekeeping handles pool repositories - -Housekeeping for pool repositories is handled differently from standard repositories. It is -ultimately performed by the Gitaly RPC `FetchIntoObjectPool`. - -This is the current call stack by which it is invoked: - -1. `Repositories::HousekeepingService#execute_gitlab_shell_gc` -1. `Projects::GitGarbageCollectWorker#perform` -1. `Projects::GitDeduplicationService#fetch_from_source` -1. `ObjectPool#fetch` -1. `ObjectPoolService#fetch` -1. `Gitaly::FetchIntoObjectPoolRequest` - -To manually invoke it from a [Rails console](operations/rails_console.md) if needed, you can call -`project.pool_repository.object_pool.fetch`. This is a potentially long-running task, though Gitaly -times out in about 8 hours. - -WARNING: -Do not run `git prune` or `git gc` in pool repositories! This can cause data loss in "real" -repositories that depend on the pool in question. +### Scheduled housekeeping + +While GitLab automatically performs housekeeping tasks based on the number of +pushes, it does not maintain repositories that don't receive any pushes at all. +As a result, inactive repositories or repositories that are only getting read +requests may not benefit from improvements in the repository housekeeping +strategy. + +Administrators can enable a background job that performs housekeeping in all +repositories at a customizable interval to remedy this situation. This +background job processes all repositories hosted by a Gitaly node in a random +order and eagerly performs housekeeping tasks on them. The Gitaly node will stop +processing repositories if it takes longer than the configured interval. + +#### Configure scheduled housekeeping + +Background maintenance of Git repositories is configured in Gitaly. By default, +Gitaly performs background repository maintenance every day at 12:00 noon for a +duration of 10 minutes. + +You can change this default in Gitaly configuration. The following snippet +enables daily background repository maintenance starting at 23:00 for 1 hour +for the `default` storage: + +```toml +[daily_maintenance] +start_hour = 23 +start_minute = 00 +duration = 1h +storages = ["default"] +``` + +Use the following snippet to completely disable background repository +maintenance: + +```toml +[daily_maintenance] +disabled = true +``` + +## Object pool repositories + +Object pool repositories are used by GitLab to deduplicate objects across forks +of a repository. When creating the first fork, we: + +1. Create an object pool repository that contains all objects of the repository + that is about to be forked. +1. Link the repository to this new object pool via Git's altenates mechanism. +1. Repack the repository so that it uses objects from the object pool. It thus + can drop its own copy of the objects. + +Any forks of this repository can now link against the object pool and thus only +have to keep objects that diverge from the primary repository. + +GitLab needs to perform special housekeeping operations in object pools: + +- Gitaly cannot ever delete unreachable objects from object pools because they + might be used by any of the forks that are connected to it. +- Gitaly must keep all objects reachable due to the same reason. Object pools + thus maintain references to unreachable "dangling" objects so that they don't + ever get deleted. +- GitLab must update object pools regularly to pull in new objects that have + been added in the primary repository. Otherwise, an object pool will become + increasingly inefficient at deduplicating objects. + +These housekeeping operations are performed by the specialized +`FetchIntoObjectPool` RPC that handles all of these special tasks while also +executing the regular housekeeping tasks we execute for normal Git +repositories. + +Object pools are getting optimized automatically whenever the primary member is +getting garbage collected. Therefore, the cadence can be configured using the +same Git GC period in that project. + +If you need to manually invoke the RPC from a [Rails console](operations/rails_console.md), +you can call `project.pool_repository.object_pool.fetch`. This is a potentially +long-running task, though Gitaly times out after about 8 hours. diff --git a/doc/administration/incoming_email.md b/doc/administration/incoming_email.md index 4959bacaaa4..433956bb066 100644 --- a/doc/administration/incoming_email.md +++ b/doc/administration/incoming_email.md @@ -41,7 +41,7 @@ in the mailbox for `user@example.com` . It is supported by providers such as Gmail, Google Apps, Yahoo! Mail, Outlook.com, and iCloud, as well as the [Postfix mail server](reply_by_email_postfix_setup.md), which you can run on-premises. Microsoft Exchange Server [does not support sub-addressing](#microsoft-exchange-server), -and Microsoft Office 365 [does not support sub-addressing by default](#microsoft-office-365) +and Microsoft Office 365 [does not support sub-addressing by default](#microsoft-office-365). NOTE: If your provider or server supports email sub-addressing, we recommend using it. diff --git a/doc/administration/index.md b/doc/administration/index.md index 7d684daf5a6..95db2b7a2e0 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -185,7 +185,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. ## Git configuration options -- [Server hooks](server_hooks.md): Server hooks (on the file system) for when webhooks aren't enough. +- [Git server hooks](server_hooks.md): Git server hooks (on the file system) for when webhooks aren't enough. Previously called server hooks. - [Git LFS configuration](lfs/index.md): Learn how to configure LFS for GitLab. - [Housekeeping](housekeeping.md): Keep your Git repositories tidy and fast. - [Configuring Git Protocol v2](git_protocol.md): Git protocol version 2 support. @@ -232,12 +232,9 @@ who are aware of the risks. - [Troubleshooting Kubernetes](https://docs.gitlab.com/charts/troubleshooting/kubernetes_cheat_sheet.html) - [Troubleshooting PostgreSQL](troubleshooting/postgresql.md) - [Guide to test environments](troubleshooting/test_environments.md) (for Support Engineers) -- [GitLab Rails console commands](troubleshooting/gitlab_rails_cheat_sheet.md) (for Support Engineers) - [Troubleshooting SSL](troubleshooting/ssl.md) - Related links: - [GitLab Developer Documentation](../development/index.md) - [Repairing and recovering broken Git repositories](https://git.seveas.net/repairing-and-recovering-broken-git-repositories.html) - [Testing with OpenSSL](https://www.feistyduck.com/library/openssl-cookbook/online/testing-with-openssl/index.html) - [`strace` zine](https://wizardzines.com/zines/strace/) -- GitLab.com-specific resources: - - [Example group SAML and SCIM configurations](../user/group/saml_sso/example_saml_config.md) diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md index 3f44b53a6d5..546c4667220 100644 --- a/doc/administration/instance_limits.md +++ b/doc/administration/instance_limits.md @@ -212,12 +212,20 @@ It's possible that this limit changes to a lower number in the future. ## Size of commit titles and descriptions -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/292039) in GitLab 13.9 +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/292039) in GitLab 13.9. -Commits with arbitrarily large messages may be pushed to GitLab, but when -displaying commits, titles (the first line of the commit message) -limits to 1KiB, and descriptions (the rest of the message) limits to -1MiB. +Commits with arbitrarily large messages may be pushed to GitLab, but the following +display limits apply: + +- **Title** - The first line of the commit message. Limited to 1 KiB. +- **Description** - The rest of the commit message. Limited to 1 MiB. + +When a commit is pushed, GitLab processes the title and description to replace +references to issues (`#123`) and merge requests (`!123`) with links to the +issues and merge requests. + +When a branch with a large number of commits is pushed, only the last 100 commits +are processed. ## Number of issues in the milestone overview @@ -359,7 +367,7 @@ header. Such emails don't create comments on issues or merge requests. ## Amount of data sent from Sentry through Error Tracking -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14926) in GitLab 12.6. +> [Limiting all Sentry responses](https://gitlab.com/gitlab-org/gitlab/-/issues/356448) introduced in GitLab 15.6. Sentry payloads sent to GitLab have a 1 MB maximum limit, both for security reasons and to limit memory consumption. diff --git a/doc/administration/integration/kroki.md b/doc/administration/integration/kroki.md index aa029be90e7..491eeb002e4 100644 --- a/doc/administration/integration/kroki.md +++ b/doc/administration/integration/kroki.md @@ -34,7 +34,7 @@ The [`yuzutech/kroki`](https://hub.docker.com/r/yuzutech/kroki) image contains t <!-- vale gitlab.Spelling = NO --> - [Bytefield](https://bytefield-svg.deepsymmetry.org/) -- [Ditaa](http://ditaa.sourceforge.net) +- [Ditaa](https://ditaa.sourceforge.net) - [Erd](https://github.com/BurntSushi/erd) - [GraphViz](https://www.graphviz.org/) - [Nomnoml](https://github.com/skanaar/nomnoml) diff --git a/doc/administration/integration/mailgun.md b/doc/administration/integration/mailgun.md index baf9e8c8a3b..d78cd9e1796 100644 --- a/doc/administration/integration/mailgun.md +++ b/doc/administration/integration/mailgun.md @@ -48,5 +48,5 @@ you're ready to enable the Mailgun integration: 1. Select the **Enable Mailgun** checkbox. 1. Enter the Mailgun HTTP webhook signing key as described in [the Mailgun documentation](https://documentation.mailgun.com/en/latest/user_manual.html#webhooks-1) and - shown in the [API security](https://app.mailgun.com/app/account/security/api_keys) section for your Mailgun account. + shown in the API security (`https://app.mailgun.com/app/account/security/api_keys`) section for your Mailgun account. 1. Select **Save changes**. diff --git a/doc/administration/issue_closing_pattern.md b/doc/administration/issue_closing_pattern.md index d10f5320109..e9150ae0650 100644 --- a/doc/administration/issue_closing_pattern.md +++ b/doc/administration/issue_closing_pattern.md @@ -17,7 +17,7 @@ in the project's default branch. ## Change the issue closing pattern -In order to change the pattern you need to have access to the server that GitLab +To change the pattern, you must have access to the server that GitLab is installed on. The default pattern can be located in [`gitlab.yml.example`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/gitlab.yml.example) diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index 1e9f20ded55..d79f322c3f5 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -200,7 +200,8 @@ sudo -u git -H bundle exec rake gitlab:artifacts:migrate RAILS_ENV=production You can optionally track progress and verify that all job artifacts migrated successfully using the [PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database): -- `sudo gitlab-rails dbconsole` for Omnibus GitLab instances. +- `sudo gitlab-rails dbconsole` for Omnibus GitLab 14.1 and earlier. +- `sudo gitlab-rails dbconsole --database main` for Omnibus GitLab 14.2 and later. - `sudo -u git -H psql -d gitlabhq_production` for source-installed instances. Verify `objectstg` below (where `store=2`) has count of all job artifacts: @@ -301,7 +302,7 @@ I/O. It instead inspects the metadata file which contains all the relevant information. This is especially important when there is a lot of artifacts, or an archive is a very large file. -When clicking on a specific file, [GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) extracts it +When selecting a specific file, [GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) extracts it from the archive and the download begins. This implementation saves space, memory and disk I/O. @@ -315,12 +316,132 @@ reasons are: - Users have configured job artifacts expiration to be longer than necessary. - The number of jobs run, and hence artifacts generated, is higher than expected. - Job logs are larger than expected, and have accumulated over time. +- The file system might run out of inodes because + [empty directories are left behind by artifact housekeeping](https://gitlab.com/gitlab-org/gitlab/-/issues/17465). + [The Rake task for _orphaned_ artifact files](../raketasks/cleanup.md#remove-orphan-artifact-files) + removes these. +- Artifact files might be left on disk and not deleted by housekeeping. Run the + [Rake task for _orphaned_ artifact files](../raketasks/cleanup.md#remove-orphan-artifact-files) + to remove these. This script should always find work to do, as it also removes empty directories (see above). +- [Artifact housekeeping was changed significantly](#artifacts-housekeeping-disabled-in-gitlab-146-to-152), + and you might need to enable a feature flag to used the updated system. In these and other cases, identify the projects most responsible for disk space usage, figure out what types of artifacts are using the most space, and in some cases, manually delete job artifacts to reclaim disk space. -One possible first step is to [clean up _orphaned_ artifact files](../raketasks/cleanup.md#remove-orphan-artifact-files). +#### Artifacts housekeeping disabled in GitLab 14.6 to 15.2 + +Artifact housekeeping was significantly changed in GitLab 14.10, and the changes +were back ported to GitLab 14.6 and later. The updated housekeeping must be +enabled with feature flags [until GitLab 15.3](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/92931). + +To check if the feature flags are enabled: + +1. Start a [Rails console](operations/rails_console.md#starting-a-rails-console-session). + +1. Check if the feature flags are enabled. + + - GitLab 14.10 and earlier: + + ```ruby + Feature.enabled?(:ci_detect_wrongly_expired_artifacts, default_enabled: :yaml) + Feature.enabled?(:ci_update_unlocked_job_artifacts, default_enabled: :yaml) + Feature.enabled?(:ci_destroy_unlocked_job_artifacts, default_enabled: :yaml) + ``` + + - GitLab 15.00 and later: + + ```ruby + Feature.enabled?(:ci_detect_wrongly_expired_artifacts) + Feature.enabled?(:ci_update_unlocked_job_artifacts) + Feature.enabled?(:ci_destroy_unlocked_job_artifacts) + ``` + +1. If any of the feature flags are disabled, enable them: + + ```ruby + Feature.enable(:ci_detect_wrongly_expired_artifacts) + Feature.enable(:ci_update_unlocked_job_artifacts) + Feature.enable(:ci_destroy_unlocked_job_artifacts) + ``` + +These changes include switching artifacts from `unlocked` to `locked` if +they [should be retained](../ci/pipelines/job_artifacts.md#keep-artifacts-from-most-recent-successful-jobs). + +Artifacts created before this feature was introduced have a status of `unknown`. After they expire, +these artifacts are not processed by the new housekeeping jobs. + +You can check the database to confirm if your instance has artifacts with the `unknown` status: + +1. Start a database console, on Omnibus: + + ```shell + sudo gitlab-psql + ``` + +1. Run this query: + + ```sql + select expire_at, file_type, locked, count(*) from ci_job_artifacts + where expire_at is not null and + file_type != 3 + group by expire_at, file_type, locked having count(*) > 1; + ``` + +If records are returned, then there are artifacts which the housekeeping job +is unable to process. For example: + +```plaintext + expire_at | file_type | locked | count +-------------------------------+-----------+--------+-------- + 2021-06-21 22:00:00+00 | 1 | 2 | 73614 + 2021-06-21 22:00:00+00 | 2 | 2 | 73614 + 2021-06-21 22:00:00+00 | 4 | 2 | 3522 + 2021-06-21 22:00:00+00 | 9 | 2 | 32 + 2021-06-21 22:00:00+00 | 12 | 2 | 163 +``` + +Artifacts with locked status `2` are `unknown`. Check +[issue #346261](https://gitlab.com/gitlab-org/gitlab/-/issues/346261#note_1028871458) +for more details. + +The Sidekiq worker that processes all `unknown` artifacts is enabled by default in +GitLab 15.3 and later. It analyzes the artifacts returned by the above database query and +determines which should be `locked` or `unlocked`. Artifacts are then deleted +by that worker if needed. + +The worker can be enabled on self-managed instances running GitLab 14.10 and later: + +1. Start a [Rails console](operations/rails_console.md#starting-a-rails-console-session). + +1. Check if the feature is enabled. + + - GitLab 14.10: + + ```ruby + Feature.enabled?(:ci_job_artifacts_backlog_work, default_enabled: :yaml) + ``` + + - GitLab 15.0 and later: + + ```ruby + Feature.enabled?(:ci_job_artifacts_backlog_work) + ``` + +1. Enable the feature, if needed: + + ```ruby + Feature.enable(:ci_job_artifacts_backlog_work) + ``` + +The worker processes 10,000 `unknown` artifacts every seven minutes, or roughly two million +in 24 hours. + +There is a related `ci_job_artifacts_backlog_large_loop_limit` feature flag +which causes the worker to process `unknown` artifacts +[in batches that are five times larger](https://gitlab.com/gitlab-org/gitlab/-/issues/356319). +This flag is not recommended for use on self-managed instances. #### List projects and builds with artifacts with a specific expiration (or no expiration) diff --git a/doc/administration/lfs/index.md b/doc/administration/lfs/index.md index 1b01c665ab8..8fdc98bd12a 100644 --- a/doc/administration/lfs/index.md +++ b/doc/administration/lfs/index.md @@ -174,10 +174,11 @@ For installations from source: RAILS_ENV=production sudo -u git -H bundle exec rake gitlab:lfs:migrate ``` -You can optionally track progress and verify that all packages migrated successfully using the +You can optionally track progress and verify that all LFS objects migrated successfully using the [PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database): -- `sudo gitlab-rails dbconsole` for Omnibus GitLab instances. +- `sudo gitlab-rails dbconsole` for Omnibus GitLab 14.1 and earlier. +- `sudo gitlab-rails dbconsole --database main` for Omnibus GitLab 14.2 and later. - `sudo -u git -H psql -d gitlabhq_production` for source-installed instances. Verify `objectstg` below (where `store=2`) has count of all LFS objects: diff --git a/doc/administration/libravatar.md b/doc/administration/libravatar.md index 5b2334bff8a..802a3be46fa 100644 --- a/doc/administration/libravatar.md +++ b/doc/administration/libravatar.md @@ -134,6 +134,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/logs.md b/doc/administration/logs.md deleted file mode 100644 index 7264cf6db98..00000000000 --- a/doc/administration/logs.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'logs/index.md' -remove_date: '2022-10-23' ---- - -This document was moved to [another location](logs/index.md). - -<!-- This redirect file can be deleted after <2022-10-23>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/logs/index.md b/doc/administration/logs/index.md index 2bcda759442..b0631c52a47 100644 --- a/doc/administration/logs/index.md +++ b/doc/administration/logs/index.md @@ -340,6 +340,12 @@ associated SSH key can download the project in question by using a `git fetch` o - `params`: Key-value pairs passed in a query string or HTTP body (sensitive parameters, such as passwords and tokens, are filtered out) - `ua`: The User-Agent of the requester +NOTE: +As of [`Grape Logging`](https://github.com/aserafin/grape_logging) v1.8.4, +the `view_duration_s` is calculated by [`duration_s - db_duration_s`](https://github.com/aserafin/grape_logging/blob/v1.8.4/lib/grape_logging/middleware/request_logger.rb#L117-L119). +Therefore, `view_duration_s` can be affected by multiple different factors, like read-write +process on Redis or external HTTP, not only the serialization process. + ## `application.log` Depending on your installation method, this file is located at: @@ -417,44 +423,16 @@ like this example: } ``` -## `kubernetes.log` +## `kubernetes.log` (DEPRECATED) + +> [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. Depending on your installation method, this file is located at: - Omnibus GitLab: `/var/log/gitlab/gitlab-rails/kubernetes.log` - Installations from source: `/home/git/gitlab/log/kubernetes.log` -It logs information related to the Kubernetes Integration, including errors -during installing cluster applications on your managed Kubernetes -clusters. - -Each line contains JSON that can be ingested by services like Elasticsearch and Splunk. -Line breaks have been added to the following example for clarity: - -```json -{ - "severity":"ERROR", - "time":"2018-11-23T15:14:54.652Z", - "exception":"Kubeclient::HttpError", - "error_code":401, - "service":"Clusters::Applications::CheckInstallationProgressService", - "app_id":14, - "project_ids":[1], - "group_ids":[], - "message":"Unauthorized" -} -{ - "severity":"ERROR", - "time":"2018-11-23T15:42:11.647Z", - "exception":"Kubeclient::HttpError", - "error_code":null, - "service":"Clusters::Applications::InstallService", - "app_id":2, - "project_ids":[19], - "group_ids":[], - "message":"SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)" -} -``` +It logs information related to [certificate-based clusters](../../user/project/clusters/index.md), such as connectivity errors. Each line contains JSON that can be ingested by services like Elasticsearch and Splunk. ## `git_json.log` diff --git a/doc/administration/logs/tracing_correlation_id.md b/doc/administration/logs/tracing_correlation_id.md index f651455a088..906dcd3cea9 100644 --- a/doc/administration/logs/tracing_correlation_id.md +++ b/doc/administration/logs/tracing_correlation_id.md @@ -103,7 +103,7 @@ sudo gitlab-ctl tail gitlab-rails/production_json.log | grep '"username":"bob"' ## Searching your logs for the correlation ID -Once you have the correlation ID you can start searching for relevant log +When you have the correlation ID you can start searching for relevant log entries. You can filter the lines by the correlation ID itself. Combining a `find` and `grep` should be sufficient to find the entries you are looking for. diff --git a/doc/administration/maintenance_mode/index.md b/doc/administration/maintenance_mode/index.md index 12f3c4c1cc3..9adb8ce2cd9 100644 --- a/doc/administration/maintenance_mode/index.md +++ b/doc/administration/maintenance_mode/index.md @@ -8,9 +8,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2149) in GitLab 13.9. -Maintenance Mode allows administrators to reduce write operations to a minimum while maintenance tasks are performed. The main goal is to block all external actions that change the internal state, including the PostgreSQL database, but especially files, Git repositories, Container repositories, and so on. +Maintenance Mode allows administrators to reduce write operations to a minimum while maintenance tasks are performed. The main goal is to block all external actions that change the internal state, including the PostgreSQL database, but especially files, Git repositories, and Container repositories. -Once Maintenance Mode is enabled, in-progress actions finish relatively quickly since no new actions are coming in, and internal state changes are minimal. +When Maintenance Mode is enabled, in-progress actions finish relatively quickly since no new actions are coming in, and internal state changes are minimal. In that state, various maintenance tasks are easier, and services can be stopped completely or be further degraded for a much shorter period of time than might otherwise be needed. For example, stopping cron jobs and draining queues should be fairly quick. @@ -150,7 +150,7 @@ is turned off. Deployments don't go through because pipelines are unfinished. -It is recommended to disable auto deploys during Maintenance Mode, and enable them once it is disabled. +It is recommended to disable auto deploys during Maintenance Mode, and enable them when it is disabled. #### Terraform integration diff --git a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md index ad864255f02..35dc64a0594 100644 --- a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md +++ b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md @@ -30,31 +30,31 @@ As an administrator, you can add new members to the group to give them the Maint This project can be used to: -- Self monitor your GitLab instance. The metrics dashboard of the project shows some basic resource +- Self-monitor your GitLab instance. The metrics dashboard of the project shows some basic resource usage charts, such as CPU and memory usage of each server in [Omnibus GitLab](https://docs.gitlab.com/omnibus/) installations. - Also configure your own [custom metrics](../../../operations/metrics/index.md#adding-custom-metrics) using metrics exposed by the [GitLab exporter](../prometheus/gitlab_metrics.md#metrics-available). -## Create the self monitoring project +## Create the self-monitoring project 1. On the top bar, select **Main menu > Admin**. -1. On the left sidebar, select **Settings > Metrics and profiling** and expand **Self monitoring**. -1. Toggle **Self monitoring** on. +1. On the left sidebar, select **Settings > Metrics and profiling** and expand **Self-monitoring**. +1. Toggle **Self-monitoring** on. 1. After your GitLab instance creates the project, GitLab displays a link to the - project in the text above the **Self monitoring** toggle. You can also find it + project in the text above the **Self-monitoring** toggle. You can also find it from the top bar by selecting **Main menu > Projects**. -## Delete the self monitoring project +## Delete the self-monitoring project WARNING: -Deleting the self monitoring project removes any changes made to the project. If +Deleting the self-monitoring project removes any changes made to the project. If you create the project again, it's created in its default state. 1. On the top bar, select **Main menu > Admin**. -1. On the left sidebar, go to **Settings > Metrics and profiling** and expand **Self monitoring**. -1. Toggle **Self monitoring** off. -1. In the confirmation dialog that opens, select **Delete self monitoring project**. +1. On the left sidebar, go to **Settings > Metrics and profiling** and expand **Self-monitoring**. +1. Toggle **Self-monitoring** off. +1. In the confirmation dialog that opens, select **Delete self-monitoring project**. It can take a few seconds for it to be deleted. 1. After the project is deleted, GitLab displays a message confirming your action. @@ -66,7 +66,7 @@ panels, provide a regular expression in the **Instance label regex** field. The dashboard uses metrics available in [Omnibus GitLab](https://docs.gitlab.com/omnibus/) installations. - + You can also [create your own dashboards](../../../operations/metrics/dashboards/index.md). @@ -85,12 +85,12 @@ you [configure it manually](../../../user/project/integrations/prometheus.md#man You can [add a Prometheus integration](../../../operations/incident_management/integrations.md) to GitLab to receive notifications of any alerts. -Once the integration is setup, you can +When the integration is set up, you can [take action on incoming alerts](../../../operations/metrics/alerts.md#trigger-actions-from-alerts). -## Add custom metrics to the self monitoring project +## Add custom metrics to the self-monitoring project -You can add custom metrics in the self monitoring project by: +You can add custom metrics in the self-monitoring project by: 1. [Duplicating](../../../operations/metrics/dashboards/index.md#duplicate-a-gitlab-defined-dashboard) the overview dashboard. 1. [Editing](../../../operations/metrics/index.md) the newly created dashboard file and configuring it with [dashboard YAML properties](../../../operations/metrics/dashboards/yaml.md). @@ -118,4 +118,4 @@ If this returns true, the first administrator user is an external user. If you face this issue, you can temporarily [make the administrator user a non-external user](../../../user/permissions.md#external-users) and then try to create the project. -Once the project is created, the administrator user can be changed back to an external user. +After the project is created, the administrator user can be changed back to an external user. diff --git a/doc/administration/monitoring/ip_allowlist.md b/doc/administration/monitoring/ip_allowlist.md index 400c70d0fde..94d6876b16f 100644 --- a/doc/administration/monitoring/ip_allowlist.md +++ b/doc/administration/monitoring/ip_allowlist.md @@ -6,9 +6,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w # IP whitelist **(FREE SELF)** -NOTE: -We intend to [rename IP whitelist as `IP allowlist`](https://gitlab.com/groups/gitlab-org/-/epics/3478). - GitLab provides some [monitoring endpoints](../../user/admin_area/monitoring/health_check.md) that provide health check information when probed. diff --git a/doc/administration/monitoring/performance/grafana_configuration.md b/doc/administration/monitoring/performance/grafana_configuration.md index a003a3f25bc..92e9672cdb6 100644 --- a/doc/administration/monitoring/performance/grafana_configuration.md +++ b/doc/administration/monitoring/performance/grafana_configuration.md @@ -133,15 +133,15 @@ However, you should **not** reinstate your old data _except_ under one of the fo If you require access to your old Grafana data but don't meet one of these criteria, you may consider: 1. Reinstating it temporarily. -1. [Exporting the dashboards](https://grafana.com/docs/grafana/latest/dashboards/export-import/#exporting-a-dashboard) you need. -1. Refreshing the data and [re-importing your dashboards](https://grafana.com/docs/grafana/latest/dashboards/export-import/#import-dashboard). +1. [Exporting the dashboards](https://grafana.com/docs/grafana/latest/dashboards/manage-dashboards/#export-and-import-dashboards) you need. +1. Refreshing the data and [re-importing your dashboards](https://grafana.com/docs/grafana/latest/dashboards/manage-dashboards/#export-and-import-dashboards). WARNING: These actions pose a temporary vulnerability while your old Grafana data is in use. Deciding to take any of these actions should be weighed carefully with your need to access existing data and dashboards. -For more information and further mitigation details, please refer to our +For more information and further mitigation details, refer to our [blog post on the security release](https://about.gitlab.com/releases/2019/08/12/critical-security-release-gitlab-12-dot-1-dot-6-released/). Read more on: diff --git a/doc/administration/monitoring/performance/index.md b/doc/administration/monitoring/performance/index.md index 0bea0836191..89f1270532a 100644 --- a/doc/administration/monitoring/performance/index.md +++ b/doc/administration/monitoring/performance/index.md @@ -41,7 +41,7 @@ Two types of metrics are collected: Transaction metrics are metrics that can be associated with a single transaction. This includes statistics such as the transaction duration, timings -of any executed SQL queries, time spent rendering HAML views, and so on. These metrics +of any executed SQL queries, and time spent rendering HAML views. These metrics are collected for every Rack request and Sidekiq job processed. ### Sampled Metrics diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index e8bdacb0e14..d3b28fd15bc 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -44,6 +44,8 @@ The following metrics are available: | `gitlab_ci_pipeline_size_builds` | Histogram | 13.1 | Total number of builds within a pipeline grouped by a pipeline source | `source` | | `gitlab_ci_runner_authentication_success_total` | Counter | 15.2 | Total number of times that runner authentication has succeeded | `type` | | `gitlab_ci_runner_authentication_failure_total` | Counter | 15.2 | Total number of times that runner authentication has failed +| `gitlab_ghost_user_migration_lag_seconds` | Gauge | 15.6 | The waiting time in seconds of the oldest scheduled record for ghost user migration | | +| `gitlab_ghost_user_migration_scheduled_records_total` | Gauge | 15.6 | The total number of scheduled ghost user migrations | | | `job_waiter_started_total` | Counter | 12.9 | Number of batches of jobs started where a web request is waiting for the jobs to complete | `worker` | | `job_waiter_timeouts_total` | Counter | 12.9 | Number of batches of jobs that timed out where a web request is waiting for the jobs to complete | `worker` | | `gitlab_ci_active_jobs` | Histogram | 14.2 | Count of active jobs when pipeline is created | | @@ -148,6 +150,8 @@ The following metrics are available: | `gitlab_ci_build_trace_errors_total` | Counter | 14.4 | Total amount of different error types on a build trace | `error_reason` | | `gitlab_presentable_object_cacheless_render_real_duration_seconds` | Histogram | 15.3 | Duration of real time spent caching and representing specific web request objects | `controller`, `action` | | `cached_object_operations_total` | Counter | 15.3 | Total number of objects cached for specific web requests | `controller`, `action` | +| `redis_hit_miss_operations_total` | Counter | 15.6 | Total number of Redis cache hits and misses | `cache_hit`, `caller_id`, `cache_identifier`, `feature_category`, `backing_resource` | +| `redis_cache_generation_duration_seconds` | Histogram | 15.6 | Time to generate Redis cache | `cache_hit`, `caller_id`, `cache_identifier`, `feature_category`, `backing_resource` | ## Metrics controlled by a feature flag @@ -155,7 +159,6 @@ The following metrics can be controlled by feature flags: | Metric | Feature Flag | |:---------------------------------------------------------------|:-------------------------------------------------------------------| -| `gitlab_method_call_duration_seconds` | `prometheus_metrics_method_instrumentation` | | `gitlab_view_rendering_duration_seconds` | `prometheus_metrics_view_instrumentation` | ## Praefect metrics @@ -318,6 +321,16 @@ configuration option in `gitlab.yml`. These metrics are served from the | `geo_ci_secure_files_verification_total` | Gauge | 15.3 | Number of secure files verifications tried on secondary | `url` | | `geo_ci_secure_files_verified` | Gauge | 15.3 | Number of secure files verified on secondary | `url` | | `geo_ci_secure_files_verification_failed` | Gauge | 15.3 | Number of secure files verifications failed on secondary | `url` | +| `geo_dependency_proxy_blob` | Gauge | 15.6 | Number of dependency proxy blobs on primary | | +| `geo_dependency_proxy_blob_checksum_total` | Gauge | 15.6 | Number of dependency proxy blobs tried to checksum on primary | | +| `geo_dependency_proxy_blob_checksummed` | Gauge | 15.6 | Number of dependency proxy blobs successfully checksummed on primary | | +| `geo_dependency_proxy_blob_checksum_failed` | Gauge | 15.6 | Number of dependency proxy blobs failed to calculate the checksum on primary | | +| `geo_dependency_proxy_blob_synced` | Gauge | 15.6 | Number of dependency proxy blobs synced on secondary | | +| `geo_dependency_proxy_blob_failed` | Gauge | 15.6 | Number of dependency proxy blobs failed to sync on secondary | | +| `geo_dependency_proxy_blob_registry` | Gauge | 15.6 | Number of dependency proxy blobs in the registry | | +| `geo_dependency_proxy_blob_verification_total` | Gauge | 15.6 | Number of dependency proxy blobs verifications tried on secondary | | +| `geo_dependency_proxy_blob_verified` | Gauge | 15.6 | Number of dependency proxy blobs verified on secondary | | +| `geo_dependency_proxy_blob_verification_failed` | Gauge | 15.6 | Number of dependency proxy blobs verifications failed on secondary | | ## Database load balancing metrics **(PREMIUM SELF)** @@ -374,6 +387,8 @@ Some basic Ruby runtime metrics are available: | `ruby_process_cpu_seconds_total` | Gauge | 12.0 | Total amount of CPU time per process | | `ruby_process_max_fds` | Gauge | 12.0 | Maximum number of open file descriptors per process | | `ruby_process_resident_memory_bytes` | Gauge | 12.0 | Memory usage by process (RSS/Resident Set Size) | +| `ruby_process_resident_anon_memory_bytes`| Gauge | 15.6 | Anonymous memory usage by process (RSS/Resident Set Size) | +| `ruby_process_resident_file_memory_bytes`| Gauge | 15.6 | File-backed memory usage by process (RSS/Resident Set Size) | | `ruby_process_unique_memory_bytes` | Gauge | 13.0 | Memory usage by process (USS/Unique Set Size) | | `ruby_process_proportional_memory_bytes` | Gauge | 13.0 | Memory usage by process (PSS/Proportional Set Size) | | `ruby_process_start_time_seconds` | Gauge | 12.0 | UNIX timestamp of process start time | @@ -399,7 +414,7 @@ These client metrics are meant to complement Redis server metrics. These metrics are broken down per [Redis instance](https://docs.gitlab.com/omnibus/settings/redis.html#running-with-multiple-redis-instances). These metrics all have a `storage` label which indicates the Redis -instance (`cache`, `shared_state`, and so on). +instance. For example, `cache` or `shared_state`. | Metric | Type | Since | Description | |:--------------------------------- |:------- |:----- |:----------- | diff --git a/doc/administration/nfs.md b/doc/administration/nfs.md index 9072bd1f344..85f35a1b188 100644 --- a/doc/administration/nfs.md +++ b/doc/administration/nfs.md @@ -10,7 +10,7 @@ type: reference NFS can be used as an alternative for object storage but this isn't typically recommended for performance reasons. -For data objects such as LFS, Uploads, Artifacts, and so on, an [Object Storage service](object_storage.md) +For data objects such as LFS, Uploads, and Artifacts, an [Object Storage service](object_storage.md) is recommended over NFS where possible, due to better performance. When eliminating the usage of NFS, there are [additional steps you need to take](object_storage.md#other-alternatives-to-file-system-storage) in addition to moving to Object Storage. @@ -44,7 +44,7 @@ GitLab support is unable to continue with the investigation if both: - The date of the request is on or after the release of GitLab version 15.6. - Support Engineers and Management determine that all reasonable non-NFS root causes have been exhausted. -If the issue is reproducible, or if it happens intermittently but regularly, GitLab Support can investigate providing the issue reproduces without the use of NFS. In order to reproduce without NFS, the affected repositories should be migrated to a different Gitaly shard, such as Gitaly cluster or a standalone Gitaly VM, backed with block storage. +If the issue is reproducible, or if it happens intermittently but regularly, GitLab Support can investigate providing the issue reproduces without the use of NFS. To reproduce without NFS, the affected repositories should be migrated to a different Gitaly shard, such as Gitaly cluster or a standalone Gitaly VM, backed with block storage. ### Why remove NFS for Git repository data @@ -52,7 +52,7 @@ If the issue is reproducible, or if it happens intermittently but regularly, Git NFS is not well-suited to a workload consisting of many small files, like Git repositories. NFS does provide a number of configuration options designed to improve performance. However, over time, a number of these mount options have proven to result in inconsistencies across multiple nodes mounting the NFS volume, up to and including data loss. Addressing these inconsistencies consume extraordinary development and support engineer time that hamper our ability to develop [Gitaly Cluster](gitaly/praefect.md), our purpose-built solution to addressing the deficiencies of NFS in this environment. -Please note that Gitaly Cluster provides highly-available Git repository storage. If this is not a requirement, single-node Gitaly backed by block storage is a suitable substitute. +Gitaly Cluster provides highly-available Git repository storage. If this is not a requirement, single-node Gitaly backed by block storage is a suitable substitute. Engineering support for NFS for Git repositories is deprecated. Technical support is planned to be unavailable from GitLab 15.0. No further enhancements are planned for this feature. @@ -338,7 +338,7 @@ following are the 4 locations need to be shared: | -------- | ----------- | --------------------- | | `/var/opt/gitlab/git-data` | Git repository data. This accounts for a large portion of your data | `git_data_dirs({"default" => { "path" => "/var/opt/gitlab/git-data"} })` | `/var/opt/gitlab/gitlab-rails/uploads` | User uploaded attachments | `gitlab_rails['uploads_directory'] = '/var/opt/gitlab/gitlab-rails/uploads'` -| `/var/opt/gitlab/gitlab-rails/shared` | Build artifacts, GitLab Pages, LFS objects, temp files, and so on. If you're using LFS this may also account for a large portion of your data | `gitlab_rails['shared_path'] = '/var/opt/gitlab/gitlab-rails/shared'` +| `/var/opt/gitlab/gitlab-rails/shared` | Objects such as build artifacts, GitLab Pages, LFS objects, and temp files. If you're using LFS this may also account for a large portion of your data | `gitlab_rails['shared_path'] = '/var/opt/gitlab/gitlab-rails/shared'` | `/var/opt/gitlab/gitlab-ci/builds` | GitLab CI/CD build traces | `gitlab_ci['builds_directory'] = '/var/opt/gitlab/gitlab-ci/builds'` Other GitLab directories should not be shared between nodes. They contain @@ -352,7 +352,7 @@ are empty before attempting a restore. Read more about the ## Testing NFS -Once you've set up the NFS server and client, you can verify NFS is configured correctly +When you've set up the NFS server and client, you can verify NFS is configured correctly by testing the following commands: ```shell diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md index 0e85635b3d2..d2c9c35148c 100644 --- a/doc/administration/object_storage.md +++ b/doc/administration/object_storage.md @@ -18,7 +18,7 @@ GitLab has been tested by vendors and customers on a number of object storage pr - [Amazon S3](https://aws.amazon.com/s3/) - [Google Cloud Storage](https://cloud.google.com/storage) - [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces) -- [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) +- [Oracle Cloud Infrastructure](https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) - [OpenStack Swift (S3 compatible mode)](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) - On-premises hardware and appliances from various storage vendors, whose list is not officially established. @@ -88,8 +88,8 @@ Object storage for <object type> must have a bucket specified If you want to use local storage for specific object types, you can [selectively disable object storages](#selectively-disabling-object-storage). -Most types of objects, such as CI artifacts, LFS files, upload -attachments, and so on can be saved in object storage by specifying a single +Most types of objects, such as CI artifacts, LFS files, and upload +attachments can be saved in object storage by specifying a single credential for object storage with multiple buckets. When the consolidated form is: @@ -279,7 +279,7 @@ Here are the valid connection parameters for GCS: | `google_project` | GCP project name. | `gcp-project-12345` | | `google_json_key_location` | JSON key path. | `/path/to/gcp-project-12345-abcde.json` | | `google_json_key_string` | JSON key string. | `{ "type": "service_account", "project_id": "example-project-382839", ... }` | -| `google_application_default` | Set to `true` to use [Google Cloud Application Default Credentials](https://cloud.google.com/docs/authentication/production#automatically) to locate service account credentials. | | +| `google_application_default` | Set to `true` to use [Google Cloud Application Default Credentials](https://cloud.google.com/docs/authentication#adc) to locate service account credentials. | | GitLab reads the value of `google_json_key_location`, then `google_json_key_string`, and finally, `google_application_default`. It uses the first of these settings that has a value. @@ -492,7 +492,7 @@ To migrate existing local data to object storage see the following guides: Prior to GitLab 13.2: - Object storage configuration for all types of objects such as CI/CD artifacts, LFS - files, upload attachments, and so on had to be configured independently. + files, and upload attachments had to be configured independently. - Object store connection parameters such as passwords and endpoint URLs had to be duplicated for each type. diff --git a/doc/administration/operations/moving_repositories.md b/doc/administration/operations/moving_repositories.md index 75078568c44..96c1fcc422d 100644 --- a/doc/administration/operations/moving_repositories.md +++ b/doc/administration/operations/moving_repositories.md @@ -376,14 +376,3 @@ sudo -u git -H bundle exec rake gitlab:list_repos SINCE='2015-10-1 12:00 UTC' |\ /home/git/repositories \ /mnt/gitlab/repositories ``` - -## Troubleshooting - -See the following for information on troubleshooting repository moves. - -### Repository move fails for archived projects - -Because of a [known issue](https://gitlab.com/gitlab-org/gitlab/-/issues/363670), -[archived projects](../../user/project/settings/index.md#advanced-project-settings) fail to move even though the data is cloned -by Gitaly. Make sure archived projects are -[unarchived](../../user/project/settings/index.md#unarchive-a-project) before initiating a move. diff --git a/doc/administration/operations/rails_console.md b/doc/administration/operations/rails_console.md index 1ef985b8938..efaf480c6df 100644 --- a/doc/administration/operations/rails_console.md +++ b/doc/administration/operations/rails_console.md @@ -59,6 +59,40 @@ you may run in the console. To turn off logging again, run: ActiveRecord::Base.logger = nil ``` +## Attributes + +View available attributes, formatted using pretty print (`pp`). + +For example, determine what attributes contain users' names and email addresses: + +```ruby +u = User.find_by_username('someuser') +pp u.attributes +``` + +Partial output: + +```plaintext +{"id"=>1234, + "email"=>"someuser@example.com", + "sign_in_count"=>99, + "name"=>"S User", + "username"=>"someuser", + "first_name"=>nil, + "last_name"=>nil, + "bot_type"=>nil} +``` + +Then make use of the attributes, [testing SMTP, for example](https://docs.gitlab.com/omnibus/settings/smtp.html#testing-the-smtp-configuration): + +```ruby +e = u.email +n = u.name +Notify.test_email(e, "Test email for #{n}", 'Test email').deliver_now +# +Notify.test_email(u.email, "Test email for #{u.name}", 'Test email').deliver_now +``` + ## Disable database statement timeout You can disable the PostgreSQL statement timeout for the current Rails console @@ -702,7 +736,7 @@ ApplicationSetting.current ### Open object in `irb` WARNING: -Any command that changes data directly could be damaging if not run correctly, or under the right conditions. We highly recommend running them in a test environment with a backup of the instance ready to be restored, just in case. +Commands that change data can cause damage if not run correctly or under the right conditions. Always run commands in a test environment first and have a backup instance ready to restore. Sometimes it is easier to go through a method if you are in the context of the object. You can shim into the namespace of `Object` to let you open `irb` in the context of any object: diff --git a/doc/administration/package_information/defaults.md b/doc/administration/package_information/defaults.md index c6a33ed7ba9..717e5ca31c9 100644 --- a/doc/administration/package_information/defaults.md +++ b/doc/administration/package_information/defaults.md @@ -68,8 +68,8 @@ over a network which will require, based on implementation, ports `111` and `2049` to be open. NOTE: -In some cases, the GitLab Registry will be automatically enabled by default. Please see [our documentation](../packages/container_registry.md) for more details +In some cases, the GitLab Registry will be automatically enabled by default. See [our documentation](../packages/container_registry.md) for more details. - [^Consul-notes]: If using additional Consul functionality, more ports may need to be opened. See the [official documentation](https://www.consul.io/docs/install/ports#ports-table) for the list. + [^Consul-notes]: If using additional Consul functionality, more ports may need to be opened. See the [official documentation](https://developer.hashicorp.com/consul/docs/install/ports#ports-table) for the list. [^Sidekiq-health]: If Sidekiq health check settings are not set, they will default to the Sidekiq metrics exporter settings. This default is deprecated and is set to be removed in [GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/347509). diff --git a/doc/administration/package_information/omnibus_packages.md b/doc/administration/package_information/omnibus_packages.md index ec406f8b458..7b3892ace70 100644 --- a/doc/administration/package_information/omnibus_packages.md +++ b/doc/administration/package_information/omnibus_packages.md @@ -72,7 +72,7 @@ Some drawbacks of a package with bundled dependencies: 1. Duplication with possibly existing software. 1. Less flexibility in configuration. -## Why would I install an omnibus package when I can use a system package? +## Why would you install an omnibus package when you can use a system package? The answer can be simplified to: less maintenance required. Instead of handling multiple packages that *can* break existing functionality if the versions are diff --git a/doc/administration/package_information/postgresql_versions.md b/doc/administration/package_information/postgresql_versions.md index 6409c5fdbc9..c1e9f7320ea 100644 --- a/doc/administration/package_information/postgresql_versions.md +++ b/doc/administration/package_information/postgresql_versions.md @@ -30,6 +30,7 @@ Read more about update policies and warnings in the PostgreSQL | GitLab version | PostgreSQL versions | Default version for fresh installs | Default version for upgrades | Notes | | -------------- | --------------------- | ---------------------------------- | ---------------------------- | ----- | +| 15.6 | 12.12, 13.8 | 13.8 | 12.12 | For upgrades, users can manually upgrade to 13.8 following the [upgrade documentation](https://docs.gitlab.com/omnibus/settings/database.html#gitlab-150-and-later). | | 15.0 | 12.10, 13.6 | 13.6 | 12.10 | For upgrades, users can manually upgrade to 13.6 following the [upgrade documentation](https://docs.gitlab.com/omnibus/settings/database.html#gitlab-150-and-later). | | 14.1 | 12.7, 13.3 | 12.7 | 12.7 | PostgreSQL 13 available for fresh installations if not using [Geo](../geo/index.md#requirements-for-running-geo) or [Patroni](../postgresql/index.md#postgresql-replication-and-failover-with-omnibus-gitlab). | 14.0 | 12.7 | 12.7 | 12.7 | HA installations with repmgr are no longer supported and are prevented from upgrading to Omnibus GitLab 14.0 | diff --git a/doc/administration/package_information/signed_packages.md b/doc/administration/package_information/signed_packages.md index 34c8148e807..e4566be7534 100644 --- a/doc/administration/package_information/signed_packages.md +++ b/doc/administration/package_information/signed_packages.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Package Signatures **(FREE SELF)** -As of the release of GitLab 9.5 on August 22, 2017, GitLab provides signed Omnibus GitLab packages for RPM and DEB based distributions. This means that all packages provided on <https://packages.gitlab.com> are signed, starting with `9.5.0`, and all future versions of supported branches (for example `9.3.x` and `9.4.x` after August 22, 2017). Any package version prior to August 22, 2017, will not be signed. Please pass the appropriate argument to your package manager. (Example: `yum --nogpgcheck`) +As of the release of GitLab 9.5 on August 22, 2017, GitLab provides signed Omnibus GitLab packages for RPM and DEB based distributions. This means that all packages provided on <https://packages.gitlab.com> are signed, starting with `9.5.0`, and all future versions of supported branches (for example `9.3.x` and `9.4.x` after August 22, 2017). Any package version prior to August 22, 2017, will not be signed. Pass the appropriate argument to your package manager. (Example: `yum --nogpgcheck`) Omnibus GitLab packages produced by GitLab are created via the [Omnibus](https://github.com/chef/omnibus) tool, for which GitLab has added DEB signing via `debsigs` in [our own fork](https://gitlab.com/gitlab-org/omnibus). This addition, combined with the existing functionality of RPM signing, allows GitLab to provide signed packages for all supported distributions using DEB or RPM. diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md index d04e3217f57..2623f2afd8d 100644 --- a/doc/administration/packages/container_registry.md +++ b/doc/administration/packages/container_registry.md @@ -1,6 +1,6 @@ --- stage: Package -group: Package +group: Container Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -198,7 +198,8 @@ docker login gitlab.example.com:5050 When the Registry is configured to use its own domain, you need a TLS certificate for that specific domain (for example, `registry.example.com`). You might need a wildcard certificate if hosted under a subdomain of your existing GitLab -domain, for example, `registry.gitlab.example.com`. +domain. For example, `*.gitlab.example.com`, is a wildcard that matches `registry.gitlab.example.com`, +and is distinct from `*.example.com`. As well as manually generated SSL certificates (explained here), certificates automatically generated by Let's Encrypt are also [supported in Omnibus installs](https://docs.gitlab.com/omnibus/settings/ssl.html). @@ -1156,7 +1157,7 @@ blobs start being deleted is anything permanent done. ## Configuring GitLab and Registry to run on separate nodes (Omnibus GitLab) By default, package assumes that both services are running on the same node. -In order to get GitLab and Registry to run on a separate nodes, separate configuration +To get GitLab and Registry to run on a separate nodes, separate configuration is necessary for Registry and GitLab. ### Configuring Registry diff --git a/doc/administration/packages/dependency_proxy.md b/doc/administration/packages/dependency_proxy.md index 789863e8ed0..f6eb6f85274 100644 --- a/doc/administration/packages/dependency_proxy.md +++ b/doc/administration/packages/dependency_proxy.md @@ -1,6 +1,6 @@ --- stage: Package -group: Package +group: Container Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -215,11 +215,12 @@ For installations from source: RAILS_ENV=production sudo -u git -H bundle exec rake gitlab:dependency_proxy:migrate ``` -You can optionally track progress and verify that all packages migrated successfully using the +You can optionally track progress and verify that all Dependency Proxy blobs and manifests migrated successfully using the [PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database): -- For Omnibus GitLab instances: `sudo gitlab-rails dbconsole` -- For installations from source: `sudo -u git -H psql -d gitlabhq_production` +- `sudo gitlab-rails dbconsole` for Omnibus GitLab 14.1 and earlier. +- `sudo gitlab-rails dbconsole --database main` for Omnibus GitLab 14.2 and later. +- `sudo -u git -H psql -d gitlabhq_production` for source-installed instances. Verify that `objectstg` (where `file_store = '2'`) has the count of all Dependency Proxy blobs and manifests for each respective query: diff --git a/doc/administration/packages/index.md b/doc/administration/packages/index.md index a7ab0fb3246..74d835eb744 100644 --- a/doc/administration/packages/index.md +++ b/doc/administration/packages/index.md @@ -1,6 +1,6 @@ --- stage: Package -group: Package +group: Package Registry info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -231,7 +231,8 @@ RAILS_ENV=production sudo -u git -H bundle exec rake gitlab:packages:migrate You can optionally track progress and verify that all packages migrated successfully using the [PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database): -- `sudo gitlab-rails dbconsole` for Omnibus GitLab instances. +- `sudo gitlab-rails dbconsole` for Omnibus GitLab 14.1 and earlier. +- `sudo gitlab-rails dbconsole --database main` for Omnibus GitLab 14.2 and later. - `sudo -u git -H psql -d gitlabhq_production` for source-installed instances. Verify `objectstg` below (where `file_store = '2'`) has count of all packages: diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index 922f9a27aad..3d31491a9d2 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -260,6 +260,7 @@ control over how the Pages daemon runs and serves content in your environment. | `gitlab_id` | The OAuth application public ID. Leave blank to automatically fill when Pages authenticates with GitLab. | | `gitlab_secret` | The OAuth application secret. Leave blank to automatically fill when Pages authenticates with GitLab. | | `auth_scope` | The OAuth application scope to use for authentication. Must match GitLab Pages OAuth application settings. Leave blank to use `api` scope by default. | +| `auth_cookie_session_timeout` | Authentication cookie session timeout in seconds (default: 600s). A value of `0` means the cookie is deleted after the browser session ends. | | `gitlab_server` | Server to use for authentication when access control is enabled; defaults to GitLab `external_url`. | | `headers` | Specify any additional http headers that should be sent to the client with each response. Multiple headers can be given as an array, header and value as one string, for example `['my-header: myvalue', 'my-other-header: my-other-value']` | | `enable_disk` | Allows the GitLab Pages daemon to serve content from disk. Shall be disabled if shared disk storage isn't available. | @@ -512,7 +513,7 @@ internet connectivity is gated by a proxy. To use a proxy for GitLab Pages: ### Using a custom Certificate Authority (CA) -When using certificates issued by a custom CA, [Access Control](../../user/project/pages/pages_access_control.md#gitlab-pages-access-control) and +When using certificates issued by a custom CA, [Access Control](../../user/project/pages/pages_access_control.md) and the [online view of HTML job artifacts](../../ci/pipelines/job_artifacts.md#download-job-artifacts) fails to work if the custom CA is not recognized. @@ -820,8 +821,8 @@ database encryption. Proceed with caution. It's possible to run GitLab Pages on multiple servers if you wish to distribute the load. You can do this through standard load balancing practices such as -configuring your DNS server to return multiple IPs for your Pages server, -configuring a load balancer to work at the IP level, and so on. If you wish to +configuring your DNS server to return multiple IPs for your Pages server, or +configuring a load balancer to work at the IP level. If you wish to set up GitLab Pages on multiple servers, perform the above procedure for each Pages server. @@ -1073,7 +1074,8 @@ sudo gitlab-rake gitlab:pages:deployments:migrate_to_object_storage You can track progress and verify that all Pages deployments migrated successfully using the [PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database): -- `sudo gitlab-rails dbconsole` for Omnibus GitLab instances. +- `sudo gitlab-rails dbconsole` for Omnibus GitLab 14.1 and earlier. +- `sudo gitlab-rails dbconsole --database main` for Omnibus GitLab 14.2 and later. - `sudo -u git -H psql -d gitlabhq_production` for source-installed instances. Verify `objectstg` below (where `store=2`) has count of all Pages deployments: @@ -1213,7 +1215,7 @@ the section below. This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/pages/source.md b/doc/administration/pages/source.md index 52556809845..e122d49a963 100644 --- a/doc/administration/pages/source.md +++ b/doc/administration/pages/source.md @@ -459,7 +459,7 @@ Pages access control is disabled by default. To enable it: auth-server=<URL of the GitLab instance> ``` -1. Users can now configure it in their [projects' settings](../../user/project/pages/introduction.md#gitlab-pages-access-control). +1. Users can now configure it in their [projects' settings](../../user/project/pages/pages_access_control.md). ## Change storage path diff --git a/doc/administration/polling.md b/doc/administration/polling.md index 11f26f081cb..deb6e89183d 100644 --- a/doc/administration/polling.md +++ b/doc/administration/polling.md @@ -6,8 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Polling interval multiplier **(FREE SELF)** -The GitLab UI polls for updates for different resources (issue notes, issue titles, pipeline -statuses, and so on) on a schedule appropriate to the resource. +The GitLab UI polls for updates for different resources (such as issue notes, issue titles, and pipeline +statuses) on a schedule appropriate to the resource. Adjust the multiplier on these schedules to adjust how frequently the GitLab UI polls for updates. If you set the multiplier to: diff --git a/doc/administration/postgresql/database_load_balancing.md b/doc/administration/postgresql/database_load_balancing.md index 6bf36ef4794..805c4b2023f 100644 --- a/doc/administration/postgresql/database_load_balancing.md +++ b/doc/administration/postgresql/database_load_balancing.md @@ -130,6 +130,7 @@ record. For example: | `interval` | The minimum time in seconds between checking the DNS record. | 60 | | `disconnect_timeout` | The time in seconds after which an old connection is closed, after the list of hosts was updated. | 120 | | `use_tcp` | Lookup DNS resources using TCP instead of UDP | false | +| `max_replica_pools` | The maximum number of replicas each Rails process connects to. This is useful if you run a lot of Postgres replicas and a lot of Rails processes because without this limit every Rails process connects to every replica by default. The default behavior is unlimited if not set. | nil | If `record_type` is set to `SRV`, then GitLab continues to use round-robin algorithm and ignores the `weight` and `priority` in the record. Since `SRV` records usually diff --git a/doc/administration/postgresql/replication_and_failover.md b/doc/administration/postgresql/replication_and_failover.md index 0ee48047944..ee90b120d05 100644 --- a/doc/administration/postgresql/replication_and_failover.md +++ b/doc/administration/postgresql/replication_and_failover.md @@ -606,7 +606,7 @@ Here is a list and description of each machine and the assigned IP: - `10.6.0.33`: PostgreSQL 3 - `10.6.0.41`: GitLab application -All passwords are set to `toomanysecrets`. Please do not use this password or derived hashes and the `external_url` for GitLab is `http://gitlab.example.com`. +All passwords are set to `toomanysecrets`. Do not use this password or derived hashes and the `external_url` for GitLab is `http://gitlab.example.com`. After the initial configuration, if a failover occurs, the PostgresSQL leader node changes to one of the available secondaries until it is failed back. @@ -957,7 +957,7 @@ For further details, see [Patroni documentation on this subject](https://patroni ### Switching from repmgr to Patroni WARNING: -Switching from repmgr to Patroni is straightforward, the other way around is *not*. Rolling back from Patroni to repmgr can be complicated and may involve deletion of data directory. If you need to do that, please contact GitLab support. +Switching from repmgr to Patroni is straightforward, the other way around is *not*. Rolling back from Patroni to repmgr can be complicated and may involve deletion of data directory. If you need to do that, contact GitLab support. You can switch an exiting database cluster to use Patroni instead of repmgr with the following steps: diff --git a/doc/administration/raketasks/uploads/migrate.md b/doc/administration/raketasks/uploads/migrate.md index b6f14bc6fa4..1f6e7fda082 100644 --- a/doc/administration/raketasks/uploads/migrate.md +++ b/doc/administration/raketasks/uploads/migrate.md @@ -42,10 +42,11 @@ gitlab-rake "gitlab:uploads:migrate:all" sudo RAILS_ENV=production -u git -H bundle exec rake gitlab:uploads:migrate:all ``` -You can optionally track progress and verify that all packages migrated successfully using the +You can optionally track progress and verify that all uploads migrated successfully using the [PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database): -- `sudo gitlab-rails dbconsole` for Omnibus GitLab instances. +- `sudo gitlab-rails dbconsole` for Omnibus GitLab 14.1 and earlier. +- `sudo gitlab-rails dbconsole --database main` for Omnibus GitLab 14.2 and later. - `sudo -u git -H psql -d gitlabhq_production` for source-installed instances. Verify `objectstg` below (where `store=2`) has count of all artifacts: diff --git a/doc/administration/redis/replication_and_failover.md b/doc/administration/redis/replication_and_failover.md index 1c2515099fe..2ba19aa6f0a 100644 --- a/doc/administration/redis/replication_and_failover.md +++ b/doc/administration/redis/replication_and_failover.md @@ -66,7 +66,7 @@ When a **Primary** fails to respond, it's the application's responsibility (in our case GitLab) to handle timeout and reconnect (querying a **Sentinel** for a new **Primary**). -To get a better understanding on how to correctly set up Sentinel, please read +To get a better understanding on how to correctly set up Sentinel, read the [Redis Sentinel](https://redis.io/docs/manual/sentinel/) documentation first, as failing to configure it correctly can lead to data loss or can bring your whole cluster down, invalidating the failover effort. @@ -350,7 +350,7 @@ Now that the Redis servers are all set up, let's configure the Sentinel servers. If you are not sure if your Redis servers are working and replicating -correctly, please read the [Troubleshooting Replication](troubleshooting.md#troubleshooting-redis-replication) +correctly, read the [Troubleshooting Replication](troubleshooting.md#troubleshooting-redis-replication) and fix it before proceeding with Sentinel setup. You must have at least `3` Redis Sentinel servers, and they need to diff --git a/doc/administration/redis/replication_and_failover_external.md b/doc/administration/redis/replication_and_failover_external.md index 7904fb1ded8..23c9ce33c2d 100644 --- a/doc/administration/redis/replication_and_failover_external.md +++ b/doc/administration/redis/replication_and_failover_external.md @@ -64,7 +64,7 @@ settings outlined in We cannot stress enough the importance of reading the [replication and failover](replication_and_failover.md) documentation of the Omnibus Redis HA as it provides some invaluable information to the configuration -of Redis. Please proceed to read it before going forward with this guide. +of Redis. Read it before going forward with this guide. Before proceeding on setting up the new Redis instances, here are some requirements: diff --git a/doc/administration/redis/troubleshooting.md b/doc/administration/redis/troubleshooting.md index e568daed961..29407f65efd 100644 --- a/doc/administration/redis/troubleshooting.md +++ b/doc/administration/redis/troubleshooting.md @@ -26,8 +26,8 @@ Start Redis troubleshooting with a basic Redis activity check: 1. Open a terminal on your GitLab server. 1. Run `gitlab-redis-cli --stat` and observe the output while it runs. -1. Go to your GitLab UI and browse to a handful of pages. Any page works, like - group or project overviews, issues, files in repositories, and so on. +1. Go to your GitLab UI and browse to a handful of pages. Any page works, such as + group or project overviews, issues, or files in repositories. 1. Check the `stat` output again and verify that the values for `keys`, `clients`, `requests`, and `connections` increases as you browse. If the numbers go up, basic Redis functionality is working and GitLab can connect to it. diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index 45939b48f78..a2463c6ff88 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -28,7 +28,7 @@ full list of reference architectures, see | Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | | Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | | Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | -| Gitaly<sup>5</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | +| Gitaly<sup>5 6</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | | Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | | Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | @@ -50,6 +50,7 @@ full list of reference architectures, see - [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work. 4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. +6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to the [Large Repositories](#large-repositories) for more info. <!-- markdownlint-enable MD029 --> NOTE: @@ -158,10 +159,45 @@ Any "burstable" instance types are not recommended due to inconsistent performan ### Supported infrastructure -As a general guidance, GitLab should run on most infrastructure such as reputable Cloud Providers (AWS, GCP) and their services, or self managed (ESXi) that meet both the specs detailed above, as well as any requirements in this section. However, this does not constitute a guarantee for every potential permutation. +As a general guidance, GitLab should run on most infrastructure such as reputable Cloud Providers (AWS, GCP) and their services, +or self managed (ESXi) that meet both the specs detailed above, as well as any requirements in this section. +However, this does not constitute a guarantee for every potential permutation. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. +### Additional workloads + +The Reference Architectures have been [designed and tested](index.md#validation-and-test-results) for standard GitLab setups with +good headroom in mind to cover most scenarios. However, if any additional workloads are being added on the nodes, +such as security software, you may still need to adjust the specs accordingly to compensate. + +This also applies for some GitLab features where it's possible to run custom scripts, for example [server hooks](../server_hooks.md). + +As a general rule, it's recommended to have robust monitoring in place to measure the impact of +any additional workloads to inform any changes needed to be made. + +### Large repositories + +The Reference Architectures were tested with repositories of varying sizes that follow best practices. + +However, large repositories or monorepos (several gigabytes or more) can **significantly** impact the performance +of Git and in turn the environment itself if best practices aren't being followed such as not storing +binary or blob files in LFS. Repositories are at the core of any environment the consequences can be wide-ranging +when they are not optimized. Some examples of this impact include [Git packing operations](https://git-scm.com/book/en/v2/Git-Internals-Packfiles) +taking longer and consuming high CPU / Memory resources or Git checkouts taking longer that affect both users and +CI pipelines alike. + +As such, large repositories come with notable cost and typically will require more resources to handle, +significantly so in some cases. It's therefore **strongly** recommended then to review large repositories +to ensure they maintain good repo health and reduce their size wherever possible. + +NOTE: +If best practices aren't followed and large repositories are present on the environment, +increased Gitaly specs may be required to ensure stable performance. + +Refer to the [Managing large repositories documentation](../../user/project/repository/managing_large_repositories.md) +for more information and guidance. + ### Praefect PostgreSQL It's worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and @@ -241,8 +277,7 @@ In a multi-node GitLab configuration, you'll need a load balancer to route traffic to the application servers. The specifics on which load balancer to use or its exact configuration is beyond the scope of GitLab documentation. We assume that if you're managing multi-node systems like GitLab, you already have a load -balancer of choice and that the routing methods used are distributing calls evenly -between all nodes. Some load balancer examples include HAProxy (open-source), +balancer of choice. Some load balancer examples include HAProxy (open-source), F5 Big-IP LTM, and Citrix Net Scaler. This documentation outline the ports and protocols needed for use with GitLab. @@ -250,47 +285,13 @@ This architecture has been tested and validated with [HAProxy](https://www.hapro as the load balancer. Although other load balancers with similar feature sets could also be used, those load balancers have not been validated. -The next question is how you will handle SSL in your environment. -There are several different options: +### Balancing algorithm -- [The application node terminates SSL](#application-node-terminates-ssl). -- [The load balancer terminates SSL without backend SSL](#load-balancer-terminates-ssl-without-backend-ssl) - and communication is not secure between the load balancer and the application node. -- [The load balancer terminates SSL with backend SSL](#load-balancer-terminates-ssl-with-backend-ssl) - and communication is *secure* between the load balancer and the application node. - -### Application node terminates SSL +We recommend that a least-connection load balancing algorithm or equivalent +is used wherever possible to ensure equal spread of calls to the nodes and good performance. -Configure your load balancer to pass connections on port 443 as `TCP` rather -than `HTTP(S)` protocol. This will pass the connection to the application node's -NGINX service untouched. NGINX will have the SSL certificate and listen on port 443. - -See the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html) -for details on managing SSL certificates and configuring NGINX. - -### Load balancer terminates SSL without backend SSL - -Configure your load balancer to use the `HTTP(S)` protocol rather than `TCP`. -The load balancer will then be responsible for managing SSL certificates and -terminating SSL. - -Since communication between the load balancer and GitLab will not be secure, -there is some additional configuration needed. See the -[proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/ssl.html#configure-a-reverse-proxy-or-load-balancer-ssl-termination) -for details. - -### Load balancer terminates SSL with backend SSL - -Configure your load balancers to use the 'HTTP(S)' protocol rather than 'TCP'. -The load balancers will be responsible for managing SSL certificates that -end users will see. - -Traffic will also be secure between the load balancers and NGINX in this -scenario. There is no need to add configuration for proxied SSL since the -connection will be secure all the way. However, configuration will need to be -added to GitLab to configure SSL certificates. See -the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html) -for details on managing SSL certificates and configuring NGINX. +We don't recommend the use of round-robin algorithms as they are known to not +spread connections equally in practice. ### Readiness checks @@ -351,6 +352,50 @@ Configure DNS for an alternate SSH hostname such as `altssh.gitlab.example.com`. | ------- | ------------ | -------- | | 443 | 22 | TCP | +### SSL + +The next question is how you will handle SSL in your environment. +There are several different options: + +- [The application node terminates SSL](#application-node-terminates-ssl). +- [The load balancer terminates SSL without backend SSL](#load-balancer-terminates-ssl-without-backend-ssl) + and communication is not secure between the load balancer and the application node. +- [The load balancer terminates SSL with backend SSL](#load-balancer-terminates-ssl-with-backend-ssl) + and communication is *secure* between the load balancer and the application node. + +#### Application node terminates SSL + +Configure your load balancer to pass connections on port 443 as `TCP` rather +than `HTTP(S)` protocol. This will pass the connection to the application node's +NGINX service untouched. NGINX will have the SSL certificate and listen on port 443. + +See the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html) +for details on managing SSL certificates and configuring NGINX. + +#### Load balancer terminates SSL without backend SSL + +Configure your load balancer to use the `HTTP(S)` protocol rather than `TCP`. +The load balancer will then be responsible for managing SSL certificates and +terminating SSL. + +Since communication between the load balancer and GitLab will not be secure, +there is some additional configuration needed. See the +[proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/ssl.html#configure-a-reverse-proxy-or-load-balancer-ssl-termination) +for details. + +#### Load balancer terminates SSL with backend SSL + +Configure your load balancers to use the 'HTTP(S)' protocol rather than 'TCP'. +The load balancers will be responsible for managing SSL certificates that +end users will see. + +Traffic will also be secure between the load balancers and NGINX in this +scenario. There is no need to add configuration for proxied SSL since the +connection will be secure all the way. However, configuration will need to be +added to GitLab to configure SSL certificates. See +the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html) +for details on managing SSL certificates and configuring NGINX. + <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> @@ -415,8 +460,14 @@ backend praefect ``` Refer to your preferred Load Balancer's documentation for further guidance. -Also ensure that the routing methods used are distributing calls evenly across -all nodes. + +### Balancing algorithm + +We recommend that a least-connection-based load balancing algorithm or equivalent +is used wherever possible to ensure equal spread of calls to the nodes and good performance. + +We don't recommend the use of round-robin algorithms as they are known to not +spread connections equally in practice. <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -1168,6 +1219,12 @@ NOTE: Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). For implementations with sharded Gitaly, use the same Gitaly specs. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section. +NOTE: +Gitaly has been designed and tested with repositories of varying sizes that follow best practices. +However, large repositories or monorepos not following these practices can significantly +impact Gitaly performance and requirements. +Refer to the [Large Repositories](#large-repositories) for more info. + The recommended cluster setup includes the following components: - 3 Gitaly nodes: Replicated storage of Git repositories. @@ -1475,9 +1532,15 @@ The [Gitaly](../gitaly/index.md) server nodes that make up the cluster have requirements that are dependent on data and load. NOTE: -The Reference Architecture specs have been designed with good headroom in mind -but for Gitaly, increased specs or additional -Gitaly Cluster arrays may be required for notably large data sets or load. +Increased specs for Gitaly nodes may be required in some circumstances such as +significantly large repositories or if any [additional workloads](#additional-workloads), +such as [server hooks](../server_hooks.md), have been added. + +NOTE: +Gitaly has been designed and tested with repositories of varying sizes that follow best practices. +However, large repositories or monorepos not following these practices can significantly +impact Gitaly performance and requirements. +Refer to the [Large Repositories](#large-repositories) for more info. Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs @@ -1790,7 +1853,7 @@ Updates to example must be made at: gitlab_rails['auto_migrate'] = false # Sidekiq - sidekiqp['enable'] = true + sidekiq['enable'] = true sidekiq['listen_address'] = "0.0.0.0" # Set number of Sidekiq queue processes to the same number as available CPUs @@ -2155,7 +2218,7 @@ GitLab has been tested on a number of object storage providers: - [Amazon S3](https://aws.amazon.com/s3/) - [Google Cloud Storage](https://cloud.google.com/storage) - [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces) -- [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) +- [Oracle Cloud Infrastructure](https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) - [OpenStack Swift (S3 compatibility mode)](https://docs.openstack.org/swift/latest/s3_compat.html) - MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. @@ -2237,6 +2300,10 @@ compute deployments. With this, _stateless_ components can benefit from cloud na workload management benefits while _stateful_ components are deployed in compute VMs with Omnibus to benefit from increased permanence. +Refer to the Helm charts [Advanced configuration](https://docs.gitlab.com/charts/advanced/) +documentation for setup instructions including guidance on what GitLab secrets to sync +between Kubernetes and the backend components. + NOTE: This is an **advanced** setup. Running services in Kubernetes is well known to be complex. **This setup is only recommended** if you have strong working @@ -2279,7 +2346,7 @@ services where applicable): | Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | | Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | | Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | -| Gitaly<sup>5</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | +| Gitaly<sup>5 6</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | | Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | | Object storage<sup>4</sup> | - | - | - | - | @@ -2297,6 +2364,7 @@ services where applicable): - [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work. 4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. +6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to the [Large Repositories](#large-repositories) for more info. <!-- markdownlint-enable MD029 --> NOTE: @@ -2403,7 +2471,7 @@ ratio for each additional pod. For further information on resource usage, see the [Sidekiq resources](https://docs.gitlab.com/charts/charts/gitlab/sidekiq/#resources). -### Supporting +#### Supporting The Supporting Node Pool is designed to house all supporting deployments that don't need to be on the Webservice and Sidekiq pools. @@ -2416,6 +2484,12 @@ to deploy these in this pool where possible and not in the Webservice or Sidekiq specifically to accommodate several additional deployments. However, if your deployments don't fit into the pool as given, you can increase the node pool accordingly. +## Secrets + +When setting up a Cloud Native Hybrid environment, it's worth noting that several secrets should be synced from backend VMs from the `/etc/gitlab/gitlab-secrets.json` file into Kubernetes. + +For this setup specifically, the [GitLab Rails](https://docs.gitlab.com/charts/installation/secrets.html#gitlab-rails-secret) and [GitLab Shell](https://docs.gitlab.com/charts/installation/secrets.html#gitlab-rails-secret) secrets should be synced. + <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> diff --git a/doc/administration/reference_architectures/1k_users.md b/doc/administration/reference_architectures/1k_users.md index a8e0e23512f..2a9636b6e05 100644 --- a/doc/administration/reference_architectures/1k_users.md +++ b/doc/administration/reference_architectures/1k_users.md @@ -82,10 +82,23 @@ Any "burstable" instance types are not recommended due to inconsistent performan ### Supported infrastructure -As a general guidance, GitLab should run on most infrastructure such as reputable Cloud Providers (AWS, GCP, Azure) and their services, or self managed (ESXi) that meet both the specs detailed above, as well as any requirements in this section. However, this does not constitute a guarantee for every potential permutation. +As a general guidance, GitLab should run on most infrastructure such as reputable Cloud Providers (AWS, GCP) and their services, +or self managed (ESXi) that meet both the specs detailed above, as well as any requirements in this section. +However, this does not constitute a guarantee for every potential permutation. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. +### Additional workloads + +The Reference Architectures have been [designed and tested](index.md#validation-and-test-results) for standard GitLab setups with +good headroom in mind to cover most scenarios. However, if any additional workloads are being added on the nodes, +such as security software, you may still need to adjust the specs accordingly to compensate. + +This also applies for some GitLab features where it's possible to run custom scripts, for example [server hooks](../server_hooks.md). + +As a general rule, it's recommended to have robust monitoring in place to measure the impact of +any additional workloads to inform any changes needed to be made. + ### Swap In addition to the stated configurations, we recommend having at least 2 GB of diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index 7d67ac48b73..84eba01fe11 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -28,7 +28,7 @@ full list of reference architectures, see | Internal load balancing node<sup>3</sup> | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | | Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | | Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | -| Gitaly<sup>5</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | +| Gitaly<sup>5 6</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | | Praefect<sup>5</sup> | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | | Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | @@ -50,6 +50,7 @@ full list of reference architectures, see - [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work. 4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. +6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to the [Large Repositories](#large-repositories) for more info. <!-- markdownlint-enable MD029 --> NOTE: @@ -158,10 +159,45 @@ Any "burstable" instance types are not recommended due to inconsistent performan ### Supported infrastructure -As a general guidance, GitLab should run on most infrastructure such as reputable Cloud Providers (AWS, GCP) and their services, or self managed (ESXi) that meet both the specs detailed above, as well as any requirements in this section. However, this does not constitute a guarantee for every potential permutation. +As a general guidance, GitLab should run on most infrastructure such as reputable Cloud Providers (AWS, GCP, Azure) and their services, +or self managed (ESXi) that meet both the specs detailed above, as well as any requirements in this section. +However, this does not constitute a guarantee for every potential permutation. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. +### Additional workloads + +The Reference Architectures have been [designed and tested](index.md#validation-and-test-results) for standard GitLab setups with +good headroom in mind to cover most scenarios. However, if any additional workloads are being added on the nodes, +such as security software, you may still need to adjust the specs accordingly to compensate. + +This also applies for some GitLab features where it's possible to run custom scripts, for example [server hooks](../server_hooks.md). + +As a general rule, it's recommended to have robust monitoring in place to measure the impact of +any additional workloads to inform any changes needed to be made. + +### Large repositories + +The Reference Architectures were tested with repositories of varying sizes that follow best practices. + +However, large repositories or monorepos (several gigabytes or more) can **significantly** impact the performance +of Git and in turn the environment itself if best practices aren't being followed such as not storing +binary or blob files in LFS. Repositories are at the core of any environment the consequences can be wide-ranging +when they are not optimized. Some examples of this impact include [Git packing operations](https://git-scm.com/book/en/v2/Git-Internals-Packfiles) +taking longer and consuming high CPU / Memory resources or Git checkouts taking longer that affect both users and +CI pipelines alike. + +As such, large repositories come with notable cost and typically will require more resources to handle, +significantly so in some cases. It's therefore **strongly** recommended then to review large repositories +to ensure they maintain good repo health and reduce their size wherever possible. + +NOTE: +If best practices aren't followed and large repositories are present on the environment, +increased Gitaly specs may be required to ensure stable performance. + +Refer to the [Managing large repositories documentation](../../user/project/repository/managing_large_repositories.md) +for more information and guidance. + ### Praefect PostgreSQL It's worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and @@ -243,8 +279,7 @@ In a multi-node GitLab configuration, you'll need a load balancer to route traffic to the application servers. The specifics on which load balancer to use or its exact configuration is beyond the scope of GitLab documentation. We assume that if you're managing multi-node systems like GitLab, you already have a load -balancer of choice and that the routing methods used are distributing calls evenly -between all nodes. Some load balancer examples include HAProxy (open-source), +balancer of choice. Some load balancer examples include HAProxy (open-source), F5 Big-IP LTM, and Citrix Net Scaler. This documentation outline the ports and protocols needed for use with GitLab. @@ -261,38 +296,13 @@ There are several different options: - [The load balancer terminates SSL with backend SSL](#load-balancer-terminates-ssl-with-backend-ssl) and communication is *secure* between the load balancer and the application node. -### Application node terminates SSL - -Configure your load balancer to pass connections on port 443 as `TCP` rather -than `HTTP(S)` protocol. This will pass the connection to the application node's -NGINX service untouched. NGINX will have the SSL certificate and listen on port 443. - -See the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html) -for details on managing SSL certificates and configuring NGINX. - -### Load balancer terminates SSL without backend SSL - -Configure your load balancer to use the `HTTP(S)` protocol rather than `TCP`. -The load balancer will then be responsible for managing SSL certificates and -terminating SSL. - -Since communication between the load balancer and GitLab will not be secure, -there is some additional configuration needed. See the -[proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/ssl.html#configure-a-reverse-proxy-or-load-balancer-ssl-termination) -for details. +### Balancing algorithm -### Load balancer terminates SSL with backend SSL +We recommend that a least-connection load balancing algorithm or equivalent +is used wherever possible to ensure equal spread of calls to the nodes and good performance. -Configure your load balancers to use the 'HTTP(S)' protocol rather than 'TCP'. -The load balancers will be responsible for managing SSL certificates that -end users will see. - -Traffic will also be secure between the load balancers and NGINX in this -scenario. There is no need to add configuration for proxied SSL since the -connection will be secure all the way. However, configuration will need to be -added to GitLab to configure SSL certificates. See the -[HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html) -for details on managing SSL certificates and configuring NGINX. +We don't recommend the use of round-robin algorithms as they are known to not +spread connections equally in practice. ### Readiness checks @@ -353,6 +363,50 @@ Configure DNS for an alternate SSH hostname such as `altssh.gitlab.example.com`. | ------- | ------------ | -------- | | 443 | 22 | TCP | +### SSL + +The next question is how you will handle SSL in your environment. +There are several different options: + +- [The application node terminates SSL](#application-node-terminates-ssl). +- [The load balancer terminates SSL without backend SSL](#load-balancer-terminates-ssl-without-backend-ssl) + and communication is not secure between the load balancer and the application node. +- [The load balancer terminates SSL with backend SSL](#load-balancer-terminates-ssl-with-backend-ssl) + and communication is *secure* between the load balancer and the application node. + +#### Application node terminates SSL + +Configure your load balancer to pass connections on port 443 as `TCP` rather +than `HTTP(S)` protocol. This will pass the connection to the application node's +NGINX service untouched. NGINX will have the SSL certificate and listen on port 443. + +See the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html) +for details on managing SSL certificates and configuring NGINX. + +#### Load balancer terminates SSL without backend SSL + +Configure your load balancer to use the `HTTP(S)` protocol rather than `TCP`. +The load balancer will then be responsible for managing SSL certificates and +terminating SSL. + +Since communication between the load balancer and GitLab will not be secure, +there is some additional configuration needed. See the +[proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/ssl.html#configure-a-reverse-proxy-or-load-balancer-ssl-termination) +for details. + +#### Load balancer terminates SSL with backend SSL + +Configure your load balancers to use the 'HTTP(S)' protocol rather than 'TCP'. +The load balancers will be responsible for managing SSL certificates that +end users will see. + +Traffic will also be secure between the load balancers and NGINX in this +scenario. There is no need to add configuration for proxied SSL since the +connection will be secure all the way. However, configuration will need to be +added to GitLab to configure SSL certificates. See +the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html) +for details on managing SSL certificates and configuring NGINX. + <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> @@ -417,8 +471,20 @@ backend praefect ``` Refer to your preferred Load Balancer's documentation for further guidance. -Also ensure that the routing methods used are distributing calls evenly across -all nodes. + +### Balancing algorithm + +We recommend that a least-connection load balancing algorithm or equivalent +is used wherever possible to ensure equal spread of calls to the nodes and good performance. + +We don't recommend the use of round-robin algorithms as they are known to not +spread connections equally in practice. + +<div align="right"> + <a type="button" class="btn btn-default" href="#setup-components"> + Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> + </a> +</div> <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -1173,6 +1239,12 @@ NOTE: Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). For implementations with sharded Gitaly, use the same Gitaly specs. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section. +NOTE: +Gitaly has been designed and tested with repositories of varying sizes that follow best practices. +However, large repositories or monorepos not following these practices can significantly +impact Gitaly performance and requirements. +Refer to the [Large Repositories](#large-repositories) for more info. + The recommended cluster setup includes the following components: - 3 Gitaly nodes: Replicated storage of Git repositories. @@ -1478,9 +1550,15 @@ The [Gitaly](../gitaly/index.md) server nodes that make up the cluster have requirements that are dependent on data and load. NOTE: -The Reference Architecture specs have been designed with good headroom in mind -but for Gitaly, increased specs or additional -Gitaly Cluster arrays may be required for notably large data sets or load. +Increased specs for Gitaly nodes may be required in some circumstances such as +significantly large repositories or if any [additional workloads](#additional-workloads), +such as [server hooks](../server_hooks.md), have been added. + +NOTE: +Gitaly has been designed and tested with repositories of varying sizes that follow best practices. +However, large repositories or monorepos not following these practices can significantly +impact Gitaly performance and requirements. +Refer to the [Large Repositories](#large-repositories) for more info. Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs @@ -2159,7 +2237,7 @@ GitLab has been tested on a number of object storage providers: - [Amazon S3](https://aws.amazon.com/s3/) - [Google Cloud Storage](https://cloud.google.com/storage) - [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces) -- [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) +- [Oracle Cloud Infrastructure](https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) - [OpenStack Swift (S3 compatibility mode)](https://docs.openstack.org/swift/latest/s3_compat.html) - MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. @@ -2241,6 +2319,10 @@ compute deployments. With this, _stateless_ components can benefit from cloud na workload management benefits while _stateful_ components are deployed in compute VMs with Omnibus to benefit from increased permanence. +Refer to the Helm charts [Advanced configuration](https://docs.gitlab.com/charts/advanced/) +documentation for setup instructions including guidance on what GitLab secrets to sync +between Kubernetes and the backend components. + NOTE: This is an **advanced** setup. Running services in Kubernetes is well known to be complex. **This setup is only recommended** if you have strong working @@ -2283,7 +2365,7 @@ services where applicable): | Internal load balancing node<sup>3</sup> | 1 | 4 vCPU, 3.6GB memory | `n1-highcpu-4` | `c5.xlarge` | | Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | | Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | -| Gitaly<sup>5</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | +| Gitaly<sup>5 6</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | | Praefect<sup>5</sup> | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | | Object storage<sup>4</sup> | - | - | - | - | @@ -2301,6 +2383,7 @@ services where applicable): - [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work. 4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. +6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to the [Large Repositories](#large-repositories) for more info. <!-- markdownlint-enable MD029 --> NOTE: diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index 61ea435f63f..1acae93f764 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -25,7 +25,7 @@ For a full list of reference architectures, see | Load balancer<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | PostgreSQL<sup>1</sup> | 1 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | | Redis<sup>2</sup> | 1 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `m5.large` | `D2s v3` | -| Gitaly | 1 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Gitaly<sup>5</sup> | 1 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | GitLab Rails | 2 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | | Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Object storage<sup>4</sup> | - | - | - | - | - | @@ -35,13 +35,14 @@ For a full list of reference architectures, see 1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB clusters](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo. - - [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and [Azure Database for PostgreSQL](https://azure.microsoft.com/en-gb/services/postgresql/) is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). + - [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and [Azure Database for PostgreSQL](https://azure.microsoft.com/en-gb/products/postgresql/#overview) is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). - Consul is primarily used for Omnibus PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - [Google Memorystore](https://cloud.google.com/memorystore) and [Amazon ElastiCache](https://aws.amazon.com/elasticache/) are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work. 4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section. +5. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to the [Large Repositories](#large-repositories) for more info. <!-- markdownlint-enable MD029 --> NOTE: @@ -94,10 +95,45 @@ Any "burstable" instance types are not recommended due to inconsistent performan ### Supported infrastructure -As a general guidance, GitLab should run on most infrastructure such as reputable Cloud Providers (AWS, GCP, Azure) and their services, or self managed (ESXi) that meet both the specs detailed above, as well as any requirements in this section. However, this does not constitute a guarantee for every potential permutation. +As a general guidance, GitLab should run on most infrastructure such as reputable Cloud Providers (AWS, GCP) and their services, +or self managed (ESXi) that meet both the specs detailed above, as well as any requirements in this section. +However, this does not constitute a guarantee for every potential permutation. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. +### Additional workloads + +The Reference Architectures have been [designed and tested](index.md#validation-and-test-results) for standard GitLab setups with +good headroom in mind to cover most scenarios. However, if any additional workloads are being added on the nodes, +such as security software, you may still need to adjust the specs accordingly to compensate. + +This also applies for some GitLab features where it's possible to run custom scripts, for example [server hooks](../server_hooks.md). + +As a general rule, it's recommended to have robust monitoring in place to measure the impact of +any additional workloads to inform any changes needed to be made. + +### Large repositories + +The Reference Architectures were tested with repositories of varying sizes that follow best practices. + +However, large repositories or monorepos (several gigabytes or more) can **significantly** impact the performance +of Git and in turn the environment itself if best practices aren't being followed such as not storing +binary or blob files in LFS. Repositories are at the core of any environment the consequences can be wide-ranging +when they are not optimized. Some examples of this impact include [Git packing operations](https://git-scm.com/book/en/v2/Git-Internals-Packfiles) +taking longer and consuming high CPU / Memory resources or Git checkouts taking longer that affect both users and +CI pipelines alike. + +As such, large repositories come with notable cost and typically will require more resources to handle, +significantly so in some cases. It's therefore **strongly** recommended then to review large repositories +to ensure they maintain good repo health and reduce their size wherever possible. + +NOTE: +If best practices aren't followed and large repositories are present on the environment, +increased Gitaly specs may be required to ensure stable performance. + +Refer to the [Managing large repositories documentation](../../user/project/repository/managing_large_repositories.md) +for more information and guidance. + ## Setup components To set up GitLab and its components to accommodate up to 2,000 users: @@ -127,8 +163,7 @@ In a multi-node GitLab configuration, you'll need a load balancer to route traffic to the application servers. The specifics on which load balancer to use or its exact configuration is beyond the scope of GitLab documentation. We assume that if you're managing multi-node systems like GitLab, you already have a load -balancer of choice and that the routing methods used are distributing calls evenly -between all nodes. Some load balancer examples include HAProxy (open-source), +balancer of choice. Some load balancer examples include HAProxy (open-source), F5 Big-IP LTM, and Citrix Net Scaler. This documentation outline the ports and protocols needed for use with GitLab. @@ -145,36 +180,13 @@ several different options: - [The load balancer terminates SSL with backend SSL](#load-balancer-terminates-ssl-with-backend-ssl) and communication is *secure* between the load balancer and the application node. -### Application node terminates SSL - -Configure your load balancer to pass connections on port 443 as `TCP` instead -of `HTTP(S)`. This will pass the connection unaltered to the application node's -NGINX service, which has the SSL certificate and listens to port 443. - -For details about managing SSL certificates and configuring NGINX, see the -[HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html) - -### Load balancer terminates SSL without backend SSL - -Configure your load balancer to use the `HTTP(S)` protocol instead of `TCP`. -The load balancer will be responsible for both managing SSL certificates and -terminating SSL. - -Due to communication between the load balancer and GitLab not being secure, -you'll need to complete some additional configuration. For details, see the -[proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/ssl.html#configure-a-reverse-proxy-or-load-balancer-ssl-termination). - -### Load balancer terminates SSL with backend SSL +### Balancing algorithm -Configure your load balancers (or single balancer, if you have only one) to use -the `HTTP(S)` protocol rather than `TCP`. The load balancers will be -responsible for the managing SSL certificates for end users. +We recommend that a least-connection load balancing algorithm or equivalent +is used wherever possible to ensure equal spread of calls to the nodes and good performance. -Traffic will be secure between the load balancers and NGINX in this scenario, -and there's no need to add a configuration for proxied SSL. However, you'll -need to add a configuration to GitLab to configure SSL certificates. For -details about managing SSL certificates and configuring NGINX, see the -[HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html). +We don't recommend the use of round-robin algorithms as they are known to not +spread connections equally in practice. ### Readiness checks @@ -186,56 +198,99 @@ connect. ### Ports -The basic load balancer ports you should use are described in the following -table: +The basic ports to be used are shown in the table below. -| Port | Backend Port | Protocol | +| LB Port | Backend Port | Protocol | | ------- | ------------ | ------------------------ | | 80 | 80 | HTTP (*1*) | | 443 | 443 | TCP or HTTPS (*1*) (*2*) | | 22 | 22 | TCP | -- (*1*): [Web terminal](../../ci/environments/index.md#web-terminals-deprecated) support - requires your load balancer to correctly handle WebSocket connections. - When using HTTP or HTTPS proxying, your load balancer must be configured - to pass through the `Connection` and `Upgrade` hop-by-hop headers. For - details, see the [web terminal](../integration/terminal.md) integration guide. -- (*2*): When using the HTTPS protocol for port 443, you'll need to add an SSL - certificate to the load balancers. If you need to terminate SSL at the - GitLab application server, use the TCP protocol. +- (*1*): [Web terminal](../../ci/environments/index.md#web-terminals-deprecated) support requires + your load balancer to correctly handle WebSocket connections. When using + HTTP or HTTPS proxying, this means your load balancer must be configured + to pass through the `Connection` and `Upgrade` hop-by-hop headers. See the + [web terminal](../integration/terminal.md) integration guide for + more details. +- (*2*): When using HTTPS protocol for port 443, you will need to add an SSL + certificate to the load balancers. If you wish to terminate SSL at the + GitLab application server instead, use TCP protocol. If you're using GitLab Pages with custom domain support you will need some -additional port configurations. GitLab Pages requires a separate virtual IP -address. Configure DNS to point the `pages_external_url` from -`/etc/gitlab/gitlab.rb` to the new virtual IP address. For more information, -see the [GitLab Pages documentation](../pages/index.md). +additional port configurations. +GitLab Pages requires a separate virtual IP address. Configure DNS to point the +`pages_external_url` from `/etc/gitlab/gitlab.rb` at the new virtual IP address. See the +[GitLab Pages documentation](../pages/index.md) for more information. -| Port | Backend Port | Protocol | +| LB Port | Backend Port | Protocol | | ------- | ------------- | --------- | | 80 | Varies (*1*) | HTTP | | 443 | Varies (*1*) | TCP (*2*) | - (*1*): The backend port for GitLab Pages depends on the `gitlab_pages['external_http']` and `gitlab_pages['external_https']` - settings. For details, see the [GitLab Pages documentation](../pages/index.md). -- (*2*): Port 443 for GitLab Pages must use the TCP protocol. Users can - configure custom domains with custom SSL, which wouldn't be possible if SSL - was terminated at the load balancer. + setting. See [GitLab Pages documentation](../pages/index.md) for more details. +- (*2*): Port 443 for GitLab Pages should always use the TCP protocol. Users can + configure custom domains with custom SSL, which would not be possible + if SSL was terminated at the load balancer. #### Alternate SSH Port Some organizations have policies against opening SSH port 22. In this case, -it may be helpful to configure an alternate SSH hostname that instead allows -users to use SSH over port 443. An alternate SSH hostname requires a new -virtual IP address compared to the previously described GitLab HTTP -configuration. +it may be helpful to configure an alternate SSH hostname that allows users +to use SSH on port 443. An alternate SSH hostname will require a new virtual IP address +compared to the other GitLab HTTP configuration above. -Configure DNS for an alternate SSH hostname, such as `altssh.gitlab.example.com`: +Configure DNS for an alternate SSH hostname such as `altssh.gitlab.example.com`. | LB Port | Backend Port | Protocol | | ------- | ------------ | -------- | | 443 | 22 | TCP | +### SSL + +The next question is how you will handle SSL in your environment. +There are several different options: + +- [The application node terminates SSL](#application-node-terminates-ssl). +- [The load balancer terminates SSL without backend SSL](#load-balancer-terminates-ssl-without-backend-ssl) + and communication is not secure between the load balancer and the application node. +- [The load balancer terminates SSL with backend SSL](#load-balancer-terminates-ssl-with-backend-ssl) + and communication is *secure* between the load balancer and the application node. + +#### Application node terminates SSL + +Configure your load balancer to pass connections on port 443 as `TCP` rather +than `HTTP(S)` protocol. This will pass the connection to the application node's +NGINX service untouched. NGINX will have the SSL certificate and listen on port 443. + +See the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html) +for details on managing SSL certificates and configuring NGINX. + +#### Load balancer terminates SSL without backend SSL + +Configure your load balancer to use the `HTTP(S)` protocol rather than `TCP`. +The load balancer will then be responsible for managing SSL certificates and +terminating SSL. + +Since communication between the load balancer and GitLab will not be secure, +there is some additional configuration needed. See the +[proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/ssl.html#configure-a-reverse-proxy-or-load-balancer-ssl-termination) +for details. + +#### Load balancer terminates SSL with backend SSL + +Configure your load balancers to use the 'HTTP(S)' protocol rather than 'TCP'. +The load balancers will be responsible for managing SSL certificates that +end users will see. + +Traffic will also be secure between the load balancers and NGINX in this +scenario. There is no need to add configuration for proxied SSL since the +connection will be secure all the way. However, configuration will need to be +added to GitLab to configure SSL certificates. See +the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html) +for details on managing SSL certificates and configuring NGINX. + <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> @@ -407,9 +462,15 @@ are supported and can be added if needed. specifically the number of projects and those projects' sizes. NOTE: -The Reference Architecture specs have been designed with good headroom in mind -but for Gitaly, increased specs or switching to Gitaly Cluster -may be required for notably large data sets or load. +Increased specs for Gitaly nodes may be required in some circumstances such as +significantly large repositories or if any [additional workloads](#additional-workloads), +such as [server hooks](../server_hooks.md), have been added. + +NOTE: +Gitaly has been designed and tested with repositories of varying sizes that follow best practices. +However, large repositories or monorepos not following these practices can significantly +impact Gitaly performance and requirements. +Refer to the [Large Repositories](#large-repositories) for more info. Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs @@ -878,7 +939,7 @@ GitLab has been tested on a number of object storage providers: - [Amazon S3](https://aws.amazon.com/s3/) - [Google Cloud Storage](https://cloud.google.com/storage) - [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces) -- [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) +- [Oracle Cloud Infrastructure](https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) - [OpenStack Swift (S3 compatibility mode)](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) - MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. @@ -962,7 +1023,9 @@ compute deployments. With this, _stateless_ components can benefit from cloud na workload management benefits while _stateful_ components are deployed in compute VMs with Omnibus to benefit from increased permanence. -The 2,000 reference architecture is not a highly-available setup. To achieve HA, you can follow a modified [3K reference architecture](3k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative). +Refer to the Helm charts [Advanced configuration](https://docs.gitlab.com/charts/advanced/) +documentation for setup instructions including guidance on what GitLab secrets to sync +between Kubernetes and the backend components. NOTE: This is an **advanced** setup. Running services in Kubernetes is well known @@ -971,6 +1034,10 @@ knowledge and experience in Kubernetes. The rest of this section assumes this. NOTE: +The 2,000 reference architecture is not a highly-available setup. To achieve HA, +you can follow a modified [3K reference architecture](3k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative). + +NOTE: **Gitaly Cluster is not supported to be run in Kubernetes**. Refer to [epic 6127](https://gitlab.com/groups/gitlab-org/-/epics/6127) for more details. @@ -1010,7 +1077,7 @@ services where applicable): 1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB clusters](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo. - - [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and [Azure Database for PostgreSQL](https://azure.microsoft.com/en-gb/services/postgresql/) is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). + - [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and [Azure Database for PostgreSQL](https://azure.microsoft.com/en-gb/products/postgresql/#overview) is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). - Consul is primarily used for Omnibus PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. - [Google Memorystore](https://cloud.google.com/memorystore) and [Amazon ElastiCache](https://aws.amazon.com/elasticache/) are known to work. diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index 7484fafe1b0..4fc6af3f72e 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -37,7 +37,7 @@ For a full list of reference architectures, see | PostgreSQL<sup>1</sup> | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | | Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | -| Gitaly<sup>5</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | +| Gitaly<sup>5 6</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | | Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | | Sidekiq | 4 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | @@ -59,6 +59,7 @@ For a full list of reference architectures, see - [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work. 4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. +6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to the [Large Repositories](#large-repositories) for more info. <!-- markdownlint-enable MD029 --> NOTE: @@ -164,10 +165,45 @@ Any "burstable" instance types are not recommended due to inconsistent performan ### Supported infrastructure -As a general guidance, GitLab should run on most infrastructure such as reputable Cloud Providers (AWS, GCP) and their services, or self managed (ESXi) that meet both the specs detailed above, as well as any requirements in this section. However, this does not constitute a guarantee for every potential permutation. +As a general guidance, GitLab should run on most infrastructure such as reputable Cloud Providers (AWS, GCP) and their services, +or self managed (ESXi) that meet both the specs detailed above, as well as any requirements in this section. +However, this does not constitute a guarantee for every potential permutation. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. +### Additional workloads + +The Reference Architectures have been [designed and tested](index.md#validation-and-test-results) for standard GitLab setups with +good headroom in mind to cover most scenarios. However, if any additional workloads are being added on the nodes, +such as security software, you may still need to adjust the specs accordingly to compensate. + +This also applies for some GitLab features where it's possible to run custom scripts, for example [server hooks](../server_hooks.md). + +As a general rule, it's recommended to have robust monitoring in place to measure the impact of +any additional workloads to inform any changes needed to be made. + +### Large repositories + +The Reference Architectures were tested with repositories of varying sizes that follow best practices. + +However, large repositories or monorepos (several gigabytes or more) can **significantly** impact the performance +of Git and in turn the environment itself if best practices aren't being followed such as not storing +binary or blob files in LFS. Repositories are at the core of any environment the consequences can be wide-ranging +when they are not optimized. Some examples of this impact include [Git packing operations](https://git-scm.com/book/en/v2/Git-Internals-Packfiles) +taking longer and consuming high CPU / Memory resources or Git checkouts taking longer that affect both users and +CI pipelines alike. + +As such, large repositories come with notable cost and typically will require more resources to handle, +significantly so in some cases. It's therefore **strongly** recommended then to review large repositories +to ensure they maintain good repo health and reduce their size wherever possible. + +NOTE: +If best practices aren't followed and large repositories are present on the environment, +increased Gitaly specs may be required to ensure stable performance. + +Refer to the [Managing large repositories documentation](../../user/project/repository/managing_large_repositories.md) +for more information and guidance. + ### Praefect PostgreSQL It's worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and @@ -244,8 +280,7 @@ In a multi-node GitLab configuration, you'll need a load balancer to route traffic to the application servers. The specifics on which load balancer to use or its exact configuration is beyond the scope of GitLab documentation. We assume that if you're managing multi-node systems like GitLab, you already have a load -balancer of choice and that the routing methods used are distributing calls evenly -between all nodes. Some load balancer examples include HAProxy (open-source), +balancer of choice. Some load balancer examples include HAProxy (open-source), F5 Big-IP LTM, and Citrix Net Scaler. This documentation outline the ports and protocols needed for use with GitLab. @@ -262,38 +297,13 @@ There are several different options: - [The load balancer terminates SSL with backend SSL](#load-balancer-terminates-ssl-with-backend-ssl) and communication is *secure* between the load balancer and the application node. -### Application node terminates SSL +### Balancing algorithm -Configure your load balancer to pass connections on port 443 as `TCP` rather -than `HTTP(S)` protocol. This will pass the connection to the application node's -NGINX service untouched. NGINX will have the SSL certificate and listen on port 443. - -See the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html) -for details on managing SSL certificates and configuring NGINX. - -### Load balancer terminates SSL without backend SSL +We recommend that a least-connection load balancing algorithm or equivalent +is used wherever possible to ensure equal spread of calls to the nodes and good performance. -Configure your load balancer to use the `HTTP(S)` protocol rather than `TCP`. -The load balancer will then be responsible for managing SSL certificates and -terminating SSL. - -Since communication between the load balancer and GitLab will not be secure, -there is some additional configuration needed. See the -[proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/ssl.html#configure-a-reverse-proxy-or-load-balancer-ssl-termination) -for details. - -### Load balancer terminates SSL with backend SSL - -Configure your load balancers to use the 'HTTP(S)' protocol rather than 'TCP'. -The load balancers will be responsible for managing SSL certificates that -end users will see. - -Traffic will also be secure between the load balancers and NGINX in this -scenario. There is no need to add configuration for proxied SSL since the -connection will be secure all the way. However, configuration will need to be -added to GitLab to configure SSL certificates. See the -[HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html) -for details on managing SSL certificates and configuring NGINX. +We don't recommend the use of round-robin algorithms as they are known to not +spread connections equally in practice. ### Readiness checks @@ -354,6 +364,50 @@ Configure DNS for an alternate SSH hostname such as `altssh.gitlab.example.com`. | ------- | ------------ | -------- | | 443 | 22 | TCP | +### SSL + +The next question is how you will handle SSL in your environment. +There are several different options: + +- [The application node terminates SSL](#application-node-terminates-ssl). +- [The load balancer terminates SSL without backend SSL](#load-balancer-terminates-ssl-without-backend-ssl) + and communication is not secure between the load balancer and the application node. +- [The load balancer terminates SSL with backend SSL](#load-balancer-terminates-ssl-with-backend-ssl) + and communication is *secure* between the load balancer and the application node. + +#### Application node terminates SSL + +Configure your load balancer to pass connections on port 443 as `TCP` rather +than `HTTP(S)` protocol. This will pass the connection to the application node's +NGINX service untouched. NGINX will have the SSL certificate and listen on port 443. + +See the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html) +for details on managing SSL certificates and configuring NGINX. + +#### Load balancer terminates SSL without backend SSL + +Configure your load balancer to use the `HTTP(S)` protocol rather than `TCP`. +The load balancer will then be responsible for managing SSL certificates and +terminating SSL. + +Since communication between the load balancer and GitLab will not be secure, +there is some additional configuration needed. See the +[proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/ssl.html#configure-a-reverse-proxy-or-load-balancer-ssl-termination) +for details. + +#### Load balancer terminates SSL with backend SSL + +Configure your load balancers to use the 'HTTP(S)' protocol rather than 'TCP'. +The load balancers will be responsible for managing SSL certificates that +end users will see. + +Traffic will also be secure between the load balancers and NGINX in this +scenario. There is no need to add configuration for proxied SSL since the +connection will be secure all the way. However, configuration will need to be +added to GitLab to configure SSL certificates. See +the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html) +for details on managing SSL certificates and configuring NGINX. + <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> @@ -418,8 +472,14 @@ backend praefect ``` Refer to your preferred Load Balancer's documentation for further guidance. -Also ensure that the routing methods used are distributing calls evenly across -all nodes. + +### Balancing algorithm + +We recommend that a least-connection load balancing algorithm or equivalent +is used wherever possible to ensure equal spread of calls to the nodes and good performance. + +We don't recommend the use of round-robin algorithms as they are known to not +spread connections equally in practice. <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -1114,6 +1174,12 @@ NOTE: Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). For implementations with sharded Gitaly, use the same Gitaly specs. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section. +NOTE: +Gitaly has been designed and tested with repositories of varying sizes that follow best practices. +However, large repositories or monorepos not following these practices can significantly +impact Gitaly performance and requirements. +Refer to the [Large Repositories](#large-repositories) for more info. + The recommended cluster setup includes the following components: - 3 Gitaly nodes: Replicated storage of Git repositories. @@ -1418,9 +1484,15 @@ The [Gitaly](../gitaly/index.md) server nodes that make up the cluster have requirements that are dependent on data and load. NOTE: -The Reference Architecture specs have been designed with good headroom in mind -but for Gitaly, increased specs or additional -Gitaly Cluster arrays may be required for notably large data sets or load. +Increased specs for Gitaly nodes may be required in some circumstances such as +significantly large repositories or if any [additional workloads](#additional-workloads), +such as [server hooks](../server_hooks.md), have been added. + +NOTE: +Gitaly has been designed and tested with repositories of varying sizes that follow best practices. +However, large repositories or monorepos not following these practices can significantly +impact Gitaly performance and requirements. +Refer to the [Large Repositories](#large-repositories) for more info. Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs @@ -2112,7 +2184,7 @@ GitLab has been tested on a number of object storage providers: - [Amazon S3](https://aws.amazon.com/s3/) - [Google Cloud Storage](https://cloud.google.com/storage) - [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces) -- [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) +- [Oracle Cloud Infrastructure](https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) - [OpenStack Swift (S3 compatibility mode)](https://docs.openstack.org/swift/latest/s3_compat.html) - MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. @@ -2197,7 +2269,6 @@ but with smaller performance requirements, several modifications can be consider - PostgreSQL and PgBouncer: PgBouncer nodes could be removed and instead be enabled on PostgreSQL nodes with the Internal Load Balancer pointing to them. However, to enable [Database Load Balancing](../postgresql/database_load_balancing.md), a separate PgBouncer array is still required. - Reducing the node counts: Some node types do not need consensus and can run with fewer nodes (but more than one for redundancy). This will also lead to reduced performance. - GitLab Rails and Sidekiq: Stateless services don't have a minimum node count. Two are enough for redundancy. - - Gitaly and Praefect: A quorum is not strictly necessary. Two Gitaly nodes and two Praefect nodes are enough for redundancy. - PostgreSQL and PgBouncer: A quorum is not strictly necessary. Two PostgreSQL nodes and two PgBouncer nodes are enough for redundancy. - Running select components in reputable Cloud PaaS solutions: Select components of the GitLab setup can instead be run on Cloud Provider PaaS solutions. By doing this, additional dependent components can also be removed: - PostgreSQL: Can be run on reputable Cloud PaaS solutions such as Google Cloud SQL or Amazon RDS. In this setup, the PgBouncer and Consul nodes are no longer required: @@ -2219,6 +2290,10 @@ compute deployments. With this, _stateless_ components can benefit from cloud na workload management benefits while _stateful_ components are deployed in compute VMs with Omnibus to benefit from increased permanence. +Refer to the Helm charts [Advanced configuration](https://docs.gitlab.com/charts/advanced/) +documentation for setup instructions including guidance on what GitLab secrets to sync +between Kubernetes and the backend components. + NOTE: This is an **advanced** setup. Running services in Kubernetes is well known to be complex. **This setup is only recommended** if you have strong working @@ -2260,7 +2335,7 @@ services where applicable): | PostgreSQL<sup>1</sup> | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | | Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | -| Gitaly<sup>5</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | +| Gitaly<sup>5 6</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | | Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | | Object storage<sup>4</sup> | - | - | - | - | @@ -2278,6 +2353,7 @@ services where applicable): - [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work. 4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. +6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to the [Large Repositories](#large-repositories) for more info. <!-- markdownlint-enable MD029 --> NOTE: diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index 88fc3649b3f..ca159d62f1f 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -28,7 +28,7 @@ full list of reference architectures, see | Internal load balancing node<sup>3</sup> | 1 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | | Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | | Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | -| Gitaly<sup>5</sup> | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` | `m5.16xlarge` | +| Gitaly<sup>5 6</sup> | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` | `m5.16xlarge` | | Praefect<sup>5</sup> | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | | Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | @@ -50,6 +50,7 @@ full list of reference architectures, see - [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work. 4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. +6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to the [Large Repositories](#large-repositories) for more info. <!-- markdownlint-enable MD029 --> NOTE: @@ -158,10 +159,45 @@ Any "burstable" instance types are not recommended due to inconsistent performan ### Supported infrastructure -As a general guidance, GitLab should run on most infrastructure such as reputable Cloud Providers (AWS, GCP) and their services, or self managed (ESXi) that meet both the specs detailed above, as well as any requirements in this section. However, this does not constitute a guarantee for every potential permutation. +As a general guidance, GitLab should run on most infrastructure such as reputable Cloud Providers (AWS, GCP, Azure) and their services, +or self managed (ESXi) that meet both the specs detailed above, as well as any requirements in this section. +However, this does not constitute a guarantee for every potential permutation. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. +### Additional workloads + +The Reference Architectures have been [designed and tested](index.md#validation-and-test-results) for standard GitLab setups with +good headroom in mind to cover most scenarios. However, if any additional workloads are being added on the nodes, +such as security software, you may still need to adjust the specs accordingly to compensate. + +This also applies for some GitLab features where it's possible to run custom scripts, for example [server hooks](../server_hooks.md). + +As a general rule, it's recommended to have robust monitoring in place to measure the impact of +any additional workloads to inform any changes needed to be made. + +### Large repositories + +The Reference Architectures were tested with repositories of varying sizes that follow best practices. + +However, large repositories or monorepos (Several gigabytes or more) can **significantly** impact the performance +of Git and in turn the environment itself if best practices aren't being followed such as not storing +binary or blob files in LFS. Repositories are at the core of any environment the consequences can be wide-ranging +when they are not optimized. Some examples of this impact include [Git packing operations](https://git-scm.com/book/en/v2/Git-Internals-Packfiles) +taking longer and consuming high CPU / Memory resources or Git checkouts taking longer that affect both users and +CI pipelines alike. + +As such, large repositories come with notable cost and typically will require more resources to handle, +significantly so in some cases. It's therefore **strongly** recommended then to review large repositories +to ensure they maintain good repo health and reduce their size wherever possible. + +NOTE: +If best practices aren't followed and large repositories are present on the environment, +increased Gitaly specs may be required to ensure stable performance. + +Refer to the [Managing large repositories documentation](../../user/project/repository/managing_large_repositories.md) +for more information and guidance. + ### Praefect PostgreSQL It's worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and @@ -250,8 +286,7 @@ In a multi-node GitLab configuration, you'll need a load balancer to route traffic to the application servers. The specifics on which load balancer to use or its exact configuration is beyond the scope of GitLab documentation. We assume that if you're managing multi-node systems like GitLab, you already have a load -balancer of choice and that the routing methods used are distributing calls evenly -between all nodes. Some load balancer examples include HAProxy (open-source), +balancer of choice. Some load balancer examples include HAProxy (open-source), F5 Big-IP LTM, and Citrix Net Scaler. This documentation outline the ports and protocols needed for use with GitLab. @@ -259,47 +294,13 @@ This architecture has been tested and validated with [HAProxy](https://www.hapro as the load balancer. Although other load balancers with similar feature sets could also be used, those load balancers have not been validated. -The next question is how you will handle SSL in your environment. -There are several different options: - -- [The application node terminates SSL](#application-node-terminates-ssl). -- [The load balancer terminates SSL without backend SSL](#load-balancer-terminates-ssl-without-backend-ssl) - and communication is not secure between the load balancer and the application node. -- [The load balancer terminates SSL with backend SSL](#load-balancer-terminates-ssl-with-backend-ssl) - and communication is *secure* between the load balancer and the application node. - -### Application node terminates SSL - -Configure your load balancer to pass connections on port 443 as `TCP` rather -than `HTTP(S)` protocol. This will pass the connection to the application node's -NGINX service untouched. NGINX will have the SSL certificate and listen on port 443. - -See the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html) -for details on managing SSL certificates and configuring NGINX. - -### Load balancer terminates SSL without backend SSL +### Balancing algorithm -Configure your load balancer to use the `HTTP(S)` protocol rather than `TCP`. -The load balancer will then be responsible for managing SSL certificates and -terminating SSL. +We recommend that a least-connection load balancing algorithm or equivalent +is used wherever possible to ensure equal spread of calls to the nodes and good performance. -Since communication between the load balancer and GitLab will not be secure, -there is some additional configuration needed. See the -[proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/ssl.html#configure-a-reverse-proxy-or-load-balancer-ssl-termination) -for details. - -### Load balancer terminates SSL with backend SSL - -Configure your load balancers to use the 'HTTP(S)' protocol rather than 'TCP'. -The load balancers will be responsible for managing SSL certificates that -end users will see. - -Traffic will also be secure between the load balancers and NGINX in this -scenario. There is no need to add configuration for proxied SSL since the -connection will be secure all the way. However, configuration will need to be -added to GitLab to configure SSL certificates. See the -[HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html) -for details on managing SSL certificates and configuring NGINX. +We don't recommend the use of round-robin algorithms as they are known to not +spread connections equally in practice. ### Readiness checks @@ -360,6 +361,50 @@ Configure DNS for an alternate SSH hostname such as `altssh.gitlab.example.com`. | ------- | ------------ | -------- | | 443 | 22 | TCP | +### SSL + +The next question is how you will handle SSL in your environment. +There are several different options: + +- [The application node terminates SSL](#application-node-terminates-ssl). +- [The load balancer terminates SSL without backend SSL](#load-balancer-terminates-ssl-without-backend-ssl) + and communication is not secure between the load balancer and the application node. +- [The load balancer terminates SSL with backend SSL](#load-balancer-terminates-ssl-with-backend-ssl) + and communication is *secure* between the load balancer and the application node. + +#### Application node terminates SSL + +Configure your load balancer to pass connections on port 443 as `TCP` rather +than `HTTP(S)` protocol. This will pass the connection to the application node's +NGINX service untouched. NGINX will have the SSL certificate and listen on port 443. + +See the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html) +for details on managing SSL certificates and configuring NGINX. + +#### Load balancer terminates SSL without backend SSL + +Configure your load balancer to use the `HTTP(S)` protocol rather than `TCP`. +The load balancer will then be responsible for managing SSL certificates and +terminating SSL. + +Since communication between the load balancer and GitLab will not be secure, +there is some additional configuration needed. See the +[proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/ssl.html#configure-a-reverse-proxy-or-load-balancer-ssl-termination) +for details. + +#### Load balancer terminates SSL with backend SSL + +Configure your load balancers to use the 'HTTP(S)' protocol rather than 'TCP'. +The load balancers will be responsible for managing SSL certificates that +end users will see. + +Traffic will also be secure between the load balancers and NGINX in this +scenario. There is no need to add configuration for proxied SSL since the +connection will be secure all the way. However, configuration will need to be +added to GitLab to configure SSL certificates. See +the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html) +for details on managing SSL certificates and configuring NGINX. + <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> @@ -424,8 +469,14 @@ backend praefect ``` Refer to your preferred Load Balancer's documentation for further guidance. -Also ensure that the routing methods used are distributing calls evenly across -all nodes. + +### Balancing algorithm + +We recommend that a least-connection load balancing algorithm or equivalent +is used wherever possible to ensure equal spread of calls to the nodes and good performance. + +We don't recommend the use of round-robin algorithms as they are known to not +spread connections equally in practice. <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -1181,6 +1232,12 @@ NOTE: Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). For implementations with sharded Gitaly, use the same Gitaly specs. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section. +NOTE: +Gitaly has been designed and tested with repositories of varying sizes that follow best practices. +However, large repositories or monorepos not following these practices can significantly +impact Gitaly performance and requirements. +Refer to the [Large Repositories](#large-repositories) for more info. + The recommended cluster setup includes the following components: - 3 Gitaly nodes: Replicated storage of Git repositories. @@ -1488,9 +1545,15 @@ The [Gitaly](../gitaly/index.md) server nodes that make up the cluster have requirements that are dependent on data and load. NOTE: -The Reference Architecture specs have been designed with good headroom in mind -but for Gitaly, increased specs or additional -Gitaly Cluster arrays may be required for notably large data sets or load. +Increased specs for Gitaly nodes may be required in some circumstances such as +significantly large repositories or if any [additional workloads](#additional-workloads), +such as [server hooks](../server_hooks.md), have been added. + +NOTE: +Gitaly has been designed and tested with repositories of varying sizes that follow best practices. +However, large repositories or monorepos not following these practices can significantly +impact Gitaly performance and requirements. +Refer to the [Large Repositories](#large-repositories) for more info. Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs @@ -2176,7 +2239,7 @@ GitLab has been tested on a number of object storage providers: - [Amazon S3](https://aws.amazon.com/s3/) - [Google Cloud Storage](https://cloud.google.com/storage) - [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces) -- [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) +- [Oracle Cloud Infrastructure](https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) - [OpenStack Swift (S3 compatibility mode)](https://docs.openstack.org/swift/latest/s3_compat.html) - MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. @@ -2258,6 +2321,10 @@ compute deployments. With this, _stateless_ components can benefit from cloud na workload management benefits while _stateful_ components are deployed in compute VMs with Omnibus to benefit from increased permanence. +Refer to the Helm charts [Advanced configuration](https://docs.gitlab.com/charts/advanced/) +documentation for setup instructions including guidance on what GitLab secrets to sync +between Kubernetes and the backend components. + NOTE: This is an **advanced** setup. Running services in Kubernetes is well known to be complex. **This setup is only recommended** if you have strong working @@ -2300,7 +2367,7 @@ services where applicable): | Internal load balancing node<sup>3</sup> | 1 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | | Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | | Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | -| Gitaly<sup>5</sup> | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` | `m5.16xlarge` | +| Gitaly<sup>5 6</sup> | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` | `m5.16xlarge` | | Praefect<sup>5</sup> | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | | Object storage<sup>4</sup> | - | - | - | - | @@ -2318,6 +2385,7 @@ services where applicable): - [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work. 4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. +6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to the [Large Repositories](#large-repositories) for more info. <!-- markdownlint-enable MD029 --> NOTE: diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index c8cf35a2e59..a2b92f9c300 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -34,7 +34,7 @@ costly-to-operate environment by using the | PostgreSQL<sup>1</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | | Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | -| Gitaly<sup>5</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | +| Gitaly<sup>5 6</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | | Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | | Sidekiq | 4 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | @@ -56,6 +56,7 @@ costly-to-operate environment by using the - [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work. 4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. +6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to the [Large Repositories](#large-repositories) for more info. <!-- markdownlint-enable MD029 --> NOTE: @@ -161,10 +162,45 @@ Any "burstable" instance types are not recommended due to inconsistent performan ### Supported infrastructure -As a general guidance, GitLab should run on most infrastructure such as reputable Cloud Providers (AWS, GCP) and their services, or self managed (ESXi) that meet both the specs detailed above, as well as any requirements in this section. However, this does not constitute a guarantee for every potential permutation. +As a general guidance, GitLab should run on most infrastructure such as reputable Cloud Providers (AWS, GCP) and their services, +or self managed (ESXi) that meet both the specs detailed above, as well as any requirements in this section. +However, this does not constitute a guarantee for every potential permutation. See [Recommended cloud providers and services](index.md#recommended-cloud-providers-and-services) for more information. +### Additional workloads + +The Reference Architectures have been [designed and tested](index.md#validation-and-test-results) for standard GitLab setups with +good headroom in mind to cover most scenarios. However, if any additional workloads are being added on the nodes, +such as security software, you may still need to adjust the specs accordingly to compensate. + +This also applies for some GitLab features where it's possible to run custom scripts, for example [server hooks](../server_hooks.md). + +As a general rule, it's recommended to have robust monitoring in place to measure the impact of +any additional workloads to inform any changes needed to be made. + +### Large repositories + +The Reference Architectures were tested with repositories of varying sizes that follow best practices. + +However, large repositories or monorepos (Several gigabytes or more) can **significantly** impact the performance +of Git and in turn the environment itself if best practices aren't being followed such as not storing +binary or blob files in LFS. Repositories are at the core of any environment the consequences can be wide-ranging +when they are not optimized. Some examples of this impact include [Git packing operations](https://git-scm.com/book/en/v2/Git-Internals-Packfiles) +taking longer and consuming high CPU / Memory resources or Git checkouts taking longer that affect both users and +CI pipelines alike. + +As such, large repositories come with notable cost and typically will require more resources to handle, +significantly so in some cases. It's therefore **strongly** recommended then to review large repositories +to ensure they maintain good repo health and reduce their size wherever possible. + +NOTE: +If best practices aren't followed and large repositories are present on the environment, +increased Gitaly specs may be required to ensure stable performance. + +Refer the [Managing large repositories documentation](../../user/project/repository/managing_large_repositories.md) +for more information and guidance. + ### Praefect PostgreSQL It's worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and @@ -237,12 +273,11 @@ The following list includes descriptions of each server and its assigned IP: ## Configure the external load balancer -In a multi-node GitLab configuration, you need a load balancer to route +In a multi-node GitLab configuration, you'll need a load balancer to route traffic to the application servers. The specifics on which load balancer to use or its exact configuration is beyond the scope of GitLab documentation. We assume that if you're managing multi-node systems like GitLab, you already have a load -balancer of choice and that the routing methods used are distributing calls evenly -between all nodes. Some load balancer examples include HAProxy (open-source), +balancer of choice. Some load balancer examples include HAProxy (open-source), F5 Big-IP LTM, and Citrix Net Scaler. This documentation outline the ports and protocols needed for use with GitLab. @@ -259,45 +294,20 @@ There are several different options: - [The load balancer terminates SSL with backend SSL](#load-balancer-terminates-ssl-with-backend-ssl) and communication is *secure* between the load balancer and the application node. -### Application node terminates SSL - -Configure your load balancer to pass connections on port 443 as `TCP` rather -than `HTTP(S)` protocol. This passes the connection to the application node's -NGINX service untouched. NGINX has the SSL certificate and listen on port 443. +### Balancing algorithm -See the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html) -for details on managing SSL certificates and configuring NGINX. +We recommend that a least-connection load balancing algorithm or equivalent +is used wherever possible to ensure equal spread of calls to the nodes and good performance. -### Load balancer terminates SSL without backend SSL - -Configure your load balancer to use the `HTTP(S)` protocol rather than `TCP`. -The load balancer is then responsible for managing SSL certificates and -terminating SSL. - -Since communication between the load balancer and GitLab is not secure, -there is some additional configuration needed. See the -[proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/ssl.html#configure-a-reverse-proxy-or-load-balancer-ssl-termination) -for details. - -### Load balancer terminates SSL with backend SSL - -Configure your load balancers to use the 'HTTP(S)' protocol rather than 'TCP'. -The load balancers are responsible for managing SSL certificates that -end users see. - -Traffic is also secure between the load balancers and NGINX in this -scenario. There is no need to add configuration for proxied SSL since the -connection is secure all the way. However, configuration needs to be -added to GitLab to configure SSL certificates. See the -[HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html) -for details on managing SSL certificates and configuring NGINX. +We don't recommend the use of round-robin algorithms as they are known to not +spread connections equally in practice. ### Readiness checks Ensure the external load balancer only routes to working services with built in monitoring endpoints. The [readiness checks](../../user/admin_area/monitoring/health_check.md) all require [additional configuration](../monitoring/ip_allowlist.md) -on the nodes being checked, otherwise, the external load balancer is not able to +on the nodes being checked, otherwise, the external load balancer will not be able to connect. ### Ports @@ -316,11 +326,11 @@ The basic ports to be used are shown in the table below. to pass through the `Connection` and `Upgrade` hop-by-hop headers. See the [web terminal](../integration/terminal.md) integration guide for more details. -- (*2*): When using HTTPS protocol for port 443, you need to add an SSL +- (*2*): When using HTTPS protocol for port 443, you will need to add an SSL certificate to the load balancers. If you wish to terminate SSL at the GitLab application server instead, use TCP protocol. -If you're using GitLab Pages with custom domain support you need some +If you're using GitLab Pages with custom domain support you will need some additional port configurations. GitLab Pages requires a separate virtual IP address. Configure DNS to point the `pages_external_url` from `/etc/gitlab/gitlab.rb` at the new virtual IP address. See the @@ -342,7 +352,7 @@ GitLab Pages requires a separate virtual IP address. Configure DNS to point the Some organizations have policies against opening SSH port 22. In this case, it may be helpful to configure an alternate SSH hostname that allows users -to use SSH on port 443. An alternate SSH hostname requires a new virtual IP address +to use SSH on port 443. An alternate SSH hostname will require a new virtual IP address compared to the other GitLab HTTP configuration above. Configure DNS for an alternate SSH hostname such as `altssh.gitlab.example.com`. @@ -351,6 +361,50 @@ Configure DNS for an alternate SSH hostname such as `altssh.gitlab.example.com`. | ------- | ------------ | -------- | | 443 | 22 | TCP | +### SSL + +The next question is how you will handle SSL in your environment. +There are several different options: + +- [The application node terminates SSL](#application-node-terminates-ssl). +- [The load balancer terminates SSL without backend SSL](#load-balancer-terminates-ssl-without-backend-ssl) + and communication is not secure between the load balancer and the application node. +- [The load balancer terminates SSL with backend SSL](#load-balancer-terminates-ssl-with-backend-ssl) + and communication is *secure* between the load balancer and the application node. + +#### Application node terminates SSL + +Configure your load balancer to pass connections on port 443 as `TCP` rather +than `HTTP(S)` protocol. This will pass the connection to the application node's +NGINX service untouched. NGINX will have the SSL certificate and listen on port 443. + +See the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html) +for details on managing SSL certificates and configuring NGINX. + +#### Load balancer terminates SSL without backend SSL + +Configure your load balancer to use the `HTTP(S)` protocol rather than `TCP`. +The load balancer will then be responsible for managing SSL certificates and +terminating SSL. + +Since communication between the load balancer and GitLab will not be secure, +there is some additional configuration needed. See the +[proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/ssl.html#configure-a-reverse-proxy-or-load-balancer-ssl-termination) +for details. + +#### Load balancer terminates SSL with backend SSL + +Configure your load balancers to use the 'HTTP(S)' protocol rather than 'TCP'. +The load balancers will be responsible for managing SSL certificates that +end users will see. + +Traffic will also be secure between the load balancers and NGINX in this +scenario. There is no need to add configuration for proxied SSL since the +connection will be secure all the way. However, configuration will need to be +added to GitLab to configure SSL certificates. See +the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html) +for details on managing SSL certificates and configuring NGINX. + <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> @@ -415,8 +469,14 @@ backend praefect ``` Refer to your preferred Load Balancer's documentation for further guidance. -Also ensure that the routing methods used are distributing calls evenly across -all nodes. + +### Balancing algorithm + +We recommend that a least-connection load balancing algorithm or equivalent +is used wherever possible to ensure equal spread of calls to the nodes and good performance. + +We don't recommend the use of round-robin algorithms as they are known to not +spread connections equally in practice. <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -1110,6 +1170,12 @@ NOTE: Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). For implementations with sharded Gitaly, use the same Gitaly specs. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section. +NOTE: +Gitaly has been designed and tested with repositories of varying sizes that follow best practices. +However, large repositories or monorepos not following these practices can significantly +impact Gitaly performance and requirements. +Refer to the [Large Repositories](#large-repositories) for more info. + The recommended cluster setup includes the following components: - 3 Gitaly nodes: Replicated storage of Git repositories. @@ -1415,9 +1481,15 @@ The [Gitaly](../gitaly/index.md) server nodes that make up the cluster have requirements that are dependent on data and load. NOTE: -The Reference Architecture specs have been designed with good headroom in mind -but for Gitaly, increased specs or additional -Gitaly Cluster arrays may be required for notably large data sets or load. +Increased specs for Gitaly nodes may be required in some circumstances such as +significantly large repositories or if any [additional workloads](#additional-workloads), +such as [server hooks](../server_hooks.md), have been added. + +NOTE: +Gitaly has been designed and tested with repositories of varying sizes that follow best practices. +However, large repositories or monorepos not following these practices can significantly +impact Gitaly performance and requirements. +Refer to the [Large Repositories](#large-repositories) for more info. Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs @@ -2111,7 +2183,7 @@ GitLab has been tested on a number of object storage providers: - [Amazon S3](https://aws.amazon.com/s3/) - [Google Cloud Storage](https://cloud.google.com/storage) - [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces) -- [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) +- [Oracle Cloud Infrastructure](https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) - [OpenStack Swift (S3 compatibility mode)](https://docs.openstack.org/swift/latest/s3_compat.html) - MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. @@ -2193,6 +2265,10 @@ compute deployments. With this, _stateless_ components can benefit from cloud na workload management benefits while _stateful_ components are deployed in compute VMs with Omnibus to benefit from increased permanence. +Refer to the Helm charts [Advanced configuration](https://docs.gitlab.com/charts/advanced/) +documentation for setup instructions including guidance on what GitLab secrets to sync +between Kubernetes and the backend components. + NOTE: This is an **advanced** setup. Running services in Kubernetes is well known to be complex. **This setup is only recommended** if you have strong working @@ -2234,7 +2310,7 @@ services where applicable): | PostgreSQL<sup>1</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | | Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | -| Gitaly<sup>5</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | +| Gitaly<sup>5 6</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | | Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | | Object storage<sup>4</sup> | - | - | - | - | @@ -2252,6 +2328,7 @@ services where applicable): - [Google Cloud Load Balancing](https://cloud.google.com/load-balancing) and [Amazon Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) are known to work. 4. Should be run on reputable Cloud Provider or Self Managed solutions. More information can be found in the [Configure the object storage](#configure-the-object-storage) section. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. +6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to the [Large Repositories](#large-repositories) for more info. <!-- markdownlint-enable MD029 --> NOTE: diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index 2cf9f2a1e83..467cc332e25 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -43,11 +43,6 @@ The following Cloud Native Hybrid reference architectures, where select recommen - [Up to 25,000 users](25k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) - [Up to 50,000 users](50k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) -A GitLab [Premium or Ultimate](https://about.gitlab.com/pricing/#self-managed) license is required -to get assistance from Support with troubleshooting the [2,000 users](2k_users.md) -and higher reference architectures. -[Read more about our definition of scaled architectures](https://about.gitlab.com/support/definitions/#definition-of-scaled-architecture). - ## Deciding which architecture to use The Reference Architectures are designed to strike a balance between two important factors--performance and resilience. @@ -109,18 +104,18 @@ Below you can find the above guidance in the form of a decision tree. It's recom ```mermaid %%{init: { 'theme': 'base' } }%% -graph TD +graph TD L1A(<b>What Reference Architecture should I use?</b>) --> L2A(More than 3000 users?) L2A -->|No| L3A("<a href=#do-you-need-high-availability-ha>Do you need HA?</a><br>(or Zero-Downtime Upgrades)") --> |Yes| L4A><b>Recommendation</b><br><br>3K architecture with HA<br>including supported modifications] L3A -->|No| L4B><b>Recommendation</b><br><br>Architecture closest to user<br>count with Backups] L2A -->|Yes| L3B[Do you have experience with<br/>and want additional resilience<br/>with select components in Kubernetes?] L3B -->|No| L4C><b>Recommendation</b><br><br>Architecture closest to user<br>count with HA] L3B -->|Yes| L4D><b>Recommendation</b><br><br>Cloud Native Hybrid architecture<br>closest to user count] - + L5A("<a href=#gitlab-geo-cross-regional-distribution-disaster-recovery>Do you need cross regional distribution or disaster recovery?"</a>) --> |Yes| L6A><b>Additional Recommendation</b><br><br> GitLab Geo] L4A -.- L5A - L4B -.- L5A - L4C -.- L5A + L4B -.- L5A + L4C -.- L5A L4D -.- L5A classDef default fill:#FCA326 @@ -211,7 +206,7 @@ When selecting a database service, it should run a standard, performant, and [su Several cloud provider services are known not to support the above or have been found to have other issues and aren't recommended: - [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is incompatible and not supported. See [14.4.0](../../update/index.md#1440) for more details. -- [Azure Database for PostgreSQL Single Server](https://azure.microsoft.com/en-gb/services/postgresql/#overview) (Single / Flexible) is **strongly not recommended** for use due to notable performance / stability issues or missing functionality. See [Recommendation Notes for Azure](#recommendation-notes-for-azure) for more details. +- [Azure Database for PostgreSQL Single Server](https://azure.microsoft.com/en-gb/products/postgresql/#overview) (Single / Flexible) is **strongly not recommended** for use due to notable performance / stability issues or missing functionality. See [Recommendation Notes for Azure](#recommendation-notes-for-azure) for more details. - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB clusters](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo. ### Recommendation notes for Azure @@ -222,7 +217,7 @@ In addition to the above, you should be aware of the additional specific guidanc - **We outright strongly do not recommend [Azure Database for PostgreSQL Single Server](https://learn.microsoft.com/en-us/azure/postgresql/single-server/overview-single-server)** specifically due to significant performance and stability issues found. **For GitLab 14.0 and higher the service is not supported** due to it only supporting up to PostgreSQL 11. - A new service, [Azure Database for Postgres Flexible Server](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/) has been released but due to it missing some functionality we don't recommend it at this time. -- [Azure Blob Storage](https://azure.microsoft.com/en-gb/services/storage/blobs/) has been found to have performance limits that can impact production use at certain times. However, this has only been seen in larger architectures. +- [Azure Blob Storage](https://azure.microsoft.com/en-gb/products/storage/blobs/) has been found to have performance limits that can impact production use at certain times. However, this has only been seen in larger architectures. ## Validation and test results diff --git a/doc/administration/reference_architectures/troubleshooting.md b/doc/administration/reference_architectures/troubleshooting.md deleted file mode 100644 index a9bf035a528..00000000000 --- a/doc/administration/reference_architectures/troubleshooting.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../configure.md' -remove_date: '2022-10-24' ---- - -This document was moved to [another location](../configure.md). - -<!-- This redirect file can be deleted after <2022-10-24>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
\ No newline at end of file diff --git a/doc/administration/repository_storage_paths.md b/doc/administration/repository_storage_paths.md index 492c5306b26..808659132f9 100644 --- a/doc/administration/repository_storage_paths.md +++ b/doc/administration/repository_storage_paths.md @@ -45,8 +45,8 @@ For more information on: WARNING: The following information is for configuring GitLab to directly access repositories. This -configuration option is deprecated in favor of using [Gitaly](gitaly/index.md) and is scheduled to -[be removed in GitLab 14.0](https://gitlab.com/gitlab-org/gitaly/-/issues/1690). +configuration option is deprecated in favor of using [Gitaly](gitaly/index.md). Gitaly issue +[1690](https://gitlab.com/gitlab-org/gitaly/-/issues/1690) proposes to remove this configuration option. To configure repository storage paths: diff --git a/doc/administration/server_hooks.md b/doc/administration/server_hooks.md index 6ab4f476e5e..448becb32dc 100644 --- a/doc/administration/server_hooks.md +++ b/doc/administration/server_hooks.md @@ -5,16 +5,18 @@ info: To determine the technical writer assigned to the Stage/Group associated w disqus_identifier: 'https://docs.gitlab.com/ee/administration/custom_hooks.html' --- -# Server hooks **(FREE SELF)** +# Git server hooks **(FREE SELF)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/196051) in GitLab 12.8 replacing Custom Hooks. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/196051) in GitLab 12.8 replacing Custom Hooks. +> - [Renamed](https://gitlab.com/gitlab-org/gitlab/-/issues/372991) from server hooks to Git server hooks in GitLab 15.6. -Server hooks run custom logic on the GitLab server. Users can use them to run Git-related tasks such as: +Git server hooks (not to be confused with [system hooks](system_hooks.md) or [file hooks](file_hooks.md)) run custom logic +on the GitLab server. You can use them to run Git-related tasks such as: - Enforcing specific commit policies. - Performing tasks based on the state of the repository. -Server hooks use `pre-receive`, `post-receive`, and `update` +Git server hooks use `pre-receive`, `post-receive`, and `update` [Git server-side hooks](https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks#_server_side_hooks). GitLab administrators configure server hooks on the file system of the GitLab server. If you don't have file system access, @@ -46,7 +48,7 @@ To create server hooks for a repository: - To create many server hooks, create a directory for the hooks that matches the hook type. For example, for a `pre-receive` server hook, the directory name should be `pre-receive.d`. Put the files for the hook in that directory. 1. Make the server hook files executable and ensure that they are owned by the Git user. -1. Write the code to make the server hook function as expected. Server hooks can be in any programming language. Ensure +1. Write the code to make the server hook function as expected. Git server hooks can be in any programming language. Ensure the [shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)) at the top reflects the language type. For example, if the script is in Ruby the shebang is probably `#!/usr/bin/env ruby`. 1. Make the hook file executable, ensure that it's owned by the Git user, and ensure it does not match the backup file @@ -87,7 +89,7 @@ To create a global server hook for all repositories: 1. On the GitLab server, go to the configured global server hook directory. 1. In the configured global server hook directory, create a directory for the hooks that matches the hook type. For example, for a `pre-receive` server hook, the directory name should be `pre-receive.d`. -1. Inside this new directory, add your server hooks. Server hooks can be in any programming language. Ensure the +1. Inside this new directory, add your server hooks. Git server hooks can be in any programming language. Ensure the [shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)) at the top reflects the language type. For example, if the script is in Ruby the shebang is probably `#!/usr/bin/env ruby`. 1. Make the hook file executable, ensure that it's owned by the Git user, and ensure it does not match the backup file diff --git a/doc/administration/sidekiq/sidekiq_memory_killer.md b/doc/administration/sidekiq/sidekiq_memory_killer.md index bbf95bc45ce..0876f98621d 100644 --- a/doc/administration/sidekiq/sidekiq_memory_killer.md +++ b/doc/administration/sidekiq/sidekiq_memory_killer.md @@ -64,5 +64,5 @@ The MemoryKiller is controlled using environment variables. If the process hard shutdown/restart is not performed by Sidekiq, the Sidekiq process is forcefully terminated after - `Sidekiq.options[:timeout] + 2` seconds. An external supervision mechanism + `Sidekiq[:timeout] + 2` seconds. An external supervision mechanism (for example, runit) must restart Sidekiq afterwards. diff --git a/doc/administration/snippets/index.md b/doc/administration/snippets/index.md index 89a571946af..7bf828afedd 100644 --- a/doc/administration/snippets/index.md +++ b/doc/administration/snippets/index.md @@ -26,7 +26,7 @@ content changes. ### Snippets size limit configuration This setting is not available through the [Admin Area settings](../../user/admin_area/settings/index.md). -In order to configure this setting, use either the Rails console +To configure this setting, use either the Rails console or the [Application settings API](../../api/settings.md). NOTE: diff --git a/doc/administration/system_hooks.md b/doc/administration/system_hooks.md index 56e73150616..038c26a9c2e 100644 --- a/doc/administration/system_hooks.md +++ b/doc/administration/system_hooks.md @@ -7,7 +7,8 @@ type: reference # System hooks **(FREE SELF)** -Your GitLab instance can perform HTTP POST requests on the following events: +System hooks (not to be confused with [server hooks](server_hooks.md) or [file hooks](file_hooks.md)) perform HTTP POST +requests and are triggered on the following events: - `group_create` - `group_destroy` @@ -31,21 +32,18 @@ Your GitLab instance can perform HTTP POST requests on the following events: - `user_update_for_group` - `user_update_for_team` -The triggers for most of these are self-explanatory, but `project_update` and -`project_rename` deserve some clarification: `project_update` is fired any time -an attribute of a project is changed (including name, description, and tags) -_unless_ the `path` attribute is also changed. In that case, a `project_rename` -is triggered instead (so that, for instance, if all you care about is the -repository URL, you can just listen for `project_rename`). +The triggers for most of these are self-explanatory, but `project_update` and `project_rename` require clarification: -`user_failed_login` is sent whenever a _blocked_ user attempts to sign in and is -denied access. +- `project_update` triggers when an attribute of a project is changed (including name, description, and tags) + **except** when the `path` attribute is also changed. +- `project_rename` triggers when an attribute of a project (including `path`) is changed. If you only care about the + repository URL, just listen for `project_rename`. -System hooks can be used, for example, for logging or changing information in an -LDAP server. +`user_failed_login` is sent whenever a **blocked** user attempts to sign in and is denied access. -In addition to these default events, you can enable triggers for other events, -such as push events, and disable the `repository_update` event +As an example, use system hooks for logging or changing information in an LDAP server. + +You can also enable triggers for other events, such as push events, and disable the `repository_update` event when you create a system hook. NOTE: @@ -136,7 +134,7 @@ X-Gitlab-Event: System Hook ``` Note that `project_rename` is not triggered if the namespace changes. -Please refer to `group_rename` and `user_rename` for that case. +Refer to `group_rename` and `user_rename` for that case. **Project transferred:** @@ -766,6 +764,6 @@ important to describe those, too. Think of things that may go wrong and include This is important to minimize requests for support, and to avoid doc comments with questions that you know someone might ask. -Each scenario can be a third-level heading, e.g. `### Getting error message X`. +Each scenario can be a third-level heading, for example `### Getting error message X`. If you have none to add when creating a doc, leave this section in place but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/terraform_state.md b/doc/administration/terraform_state.md index 61fda91ba71..02a95e28747 100644 --- a/doc/administration/terraform_state.md +++ b/doc/administration/terraform_state.md @@ -135,10 +135,11 @@ For GitLab 13.8 and earlier versions, you can use a workaround for the Rake task end ``` -You can optionally track progress and verify that all packages migrated successfully using the +You can optionally track progress and verify that all Terraform state files migrated successfully using the [PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database): -- `sudo gitlab-rails dbconsole` for Omnibus GitLab instances. +- `sudo gitlab-rails dbconsole` for Omnibus GitLab 14.1 and earlier. +- `sudo gitlab-rails dbconsole --database main` for Omnibus GitLab 14.2 and later. - `sudo -u git -H psql -d gitlabhq_production` for source-installed instances. Verify `objectstg` below (where `file_store=2`) has count of all states: diff --git a/doc/administration/troubleshooting/debug.md b/doc/administration/troubleshooting/debug.md deleted file mode 100644 index d5019f1aa4a..00000000000 --- a/doc/administration/troubleshooting/debug.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../reference_architectures/troubleshooting.md' -remove_date: '2022-10-19' ---- - -This document was moved to [another location](../reference_architectures/troubleshooting.md). - -<!-- This redirect file can be deleted after 2022-10-19. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/troubleshooting/defcon.md b/doc/administration/troubleshooting/defcon.md deleted file mode 100644 index f2554f523f0..00000000000 --- a/doc/administration/troubleshooting/defcon.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../../ci/troubleshooting.md#disaster-recovery' -remove_date: '2022-10-04' ---- - -This document was moved to [another location](../../ci/troubleshooting.md#disaster-recovery). - -<!-- This redirect file can be deleted after <2022-10-04>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index 20ce52a9094..6993d48b450 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -1,660 +1,11 @@ --- -stage: Systems -group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +redirect_to: 'index.md' +remove_date: '2023-02-01' --- -# GitLab Rails Console Cheat Sheet **(FREE SELF)** +This document was moved to [another location](index.md). -This is the GitLab Support Team's collection of information regarding the GitLab Rails -console, for use while troubleshooting. It is listed here for transparency, -and for users with experience with these tools. If you are currently -having an issue with GitLab, it is highly recommended that you first check -our guide on [our Rails console](../operations/rails_console.md), -and your [support options](https://about.gitlab.com/support/), before attempting to use -this information. - -WARNING: -Some of these scripts could be damaging if not run correctly, -or under the right conditions. We highly recommend running them under the -guidance of a Support Engineer, or running them in a test environment with a -backup of the instance ready to be restored, just in case. - -WARNING: -As GitLab changes, changes to the code are inevitable, -and so some scripts may not work as they once used to. These are not kept -up-to-date as these scripts/commands were added as they were found/needed. As -mentioned above, we recommend running these scripts under the supervision of a -Support Engineer, who can also verify that they continue to work as they -should and, if needed, update the script for the latest version of GitLab. - -## Attributes - -View available attributes, formatted using pretty print (`pp`). - -For example, determine what attributes contain users' names and email addresses: - -```ruby -u = User.find_by_username('someuser') -pp u.attributes -``` - -Partial output: - -```plaintext -{"id"=>1234, - "email"=>"someuser@example.com", - "sign_in_count"=>99, - "name"=>"S User", - "username"=>"someuser", - "first_name"=>nil, - "last_name"=>nil, - "bot_type"=>nil} -``` - -Then make use of the attributes, [testing SMTP, for example](https://docs.gitlab.com/omnibus/settings/smtp.html#testing-the-smtp-configuration): - -```ruby -e = u.email -n = u.name -Notify.test_email(e, "Test email for #{n}", 'Test email').deliver_now -# -Notify.test_email(u.email, "Test email for #{u.name}", 'Test email').deliver_now -``` - -## Imports and exports - -### Import a project - -```ruby -# Find the project and get the error -p = Project.find_by_full_path('<username-or-group>/<project-name>') - -p.import_error - -# To finish the import on GitLab running version before 11.6 -p.import_finish - -# To finish the import on GitLab running version 11.6 or after -p.import_state.mark_as_failed("Failed manually through console.") -``` - -### Rename imported repository - -In a specific situation, an imported repository needed to be renamed. The Support -Team was informed of a backup restore that failed on a single repository, which created -the project with an empty repository. The project was successfully restored to a development -instance, then exported, and imported into a new project under a different name. - -The Support Team was able to transfer the incorrectly named imported project into the -correctly named empty project using the steps below. - -Move the new repository to the empty repository: - -```shell -mv /var/opt/gitlab/git-data/repositories/<group>/<new-project> /var/opt/gitlab/git-data/repositories/<group>/<empty-project> -``` - -Make sure the permissions are correct: - -```shell -chown -R git:git <path-to-directory>.git -``` - -Clear the cache: - -```shell -sudo gitlab-rake cache:clear -``` - -### Export a project - -It's typically recommended to export a project through [the web interface](../../user/project/settings/import_export.md#export-a-project-and-its-data) or through [the API](../../api/project_import_export.md). In situations where this is not working as expected, it may be preferable to export a project directly via the Rails console: - -```ruby -user = User.find_by_username('<username>') -# Sufficient permissions needed -# Read https://docs.gitlab.com/ee/user/permissions.html#project-members-permissions - -project = Project.find_by_full_path('<username-or-group>/<project-name') -Projects::ImportExport::ExportService.new(project, user).execute -``` - -If this all runs successfully, you see an output like the following before being returned to the Rails console prompt: - -```ruby -=> nil -``` - -The exported project is located in a `.tar.gz` file in `/var/opt/gitlab/gitlab-rails/uploads/-/system/import_export_upload/export_file/`. - -If this fails, [enable verbose logging](../operations/rails_console.md#looking-up-database-persisted-objects), -repeat the above procedure after, -and report the output to -[GitLab Support](https://about.gitlab.com/support/). - -## Mirrors - -### Find mirrors with "bad decrypt" errors - -This content has been converted to a Rake task, see [verify database values can be decrypted using the current secrets](../raketasks/check.md#verify-database-values-can-be-decrypted-using-the-current-secrets). - -### Transfer mirror users and tokens to a single service account - -This content has been moved to [Troubleshooting Repository mirroring](../../user/project/repository/mirror/index.md#transfer-mirror-users-and-tokens-to-a-single-service-account-in-rails-console). - -## Users - -### Create new user - -```ruby -u = User.new(username: 'test_user', email: 'test@example.com', name: 'Test User', password: 'password', password_confirmation: 'password') -u.skip_confirmation! # Use it only if you wish user to be automatically confirmed. If skipped, user receives confirmation e-mail -u.save! -``` - -### Skip reconfirmation - -```ruby -user = User.find_by_username('<username>') -user.skip_reconfirmation! -``` - -### Disable 2fa for single user - -**In GitLab 13.5 and later:** - -Use the code under [Disable 2FA | For a single user](../../security/two_factor_authentication.md#for-a-single-user) so that the target user -is notified that 2FA has been disabled. - -**In GitLab 13.4 and earlier:** - -```ruby -user = User.find_by_username('<username>') -user.disable_two_factor! -``` - -### Active users & Historical users - -```ruby -# Active users on the instance, now -User.active.count - -# Users taking a seat on the instance -User.billable.count - -# The historical max on the instance as of the past year -::HistoricalData.max_historical_user_count(from: 1.year.ago.beginning_of_day, to: Time.current.end_of_day) -``` - -Using cURL and jq (up to a max 100, see [Pagination](../../api/index.md#pagination)): - -```shell -curl --silent --header "Private-Token: ********************" \ - "https://gitlab.example.com/api/v4/users?per_page=100&active" | jq --compact-output '.[] | [.id,.name,.username]' -``` - -### Update Daily Billable & Historical users - -```ruby -# Forces recount of historical (max) users -::HistoricalDataWorker.new.perform - -# Forces recount of daily billable users -identifier = Analytics::UsageTrends::Measurement.identifiers[:billable_users] -::Analytics::UsageTrends::CounterJobWorker.new.perform(identifier, User.minimum(:id), User.maximum(:id), Time.zone.now) -``` - -### Block or Delete Users that have no projects or groups - -```ruby -users = User.where('id NOT IN (select distinct(user_id) from project_authorizations)') - -# How many users are removed? -users.count - -# If that count looks sane: - -# You can either block the users: -users.each { |user| user.blocked? ? nil : user.block! } - -# Or you can delete them: - # need 'current user' (your user) for auditing purposes -current_user = User.find_by(username: '<your username>') - -users.each do |user| - DeleteUserWorker.perform_async(current_user.id, user.id) -end -``` - -### Deactivate Users that have no recent activity - -```ruby -days_inactive = 90 -inactive_users = User.active.where("last_activity_on <= ?", days_inactive.days.ago) - -inactive_users.each do |user| - puts "user '#{user.username}': #{user.last_activity_on}" - user.deactivate! -end -``` - -### Block Users that have no recent activity - -```ruby -days_inactive = 90 -inactive_users = User.active.where("last_activity_on <= ?", days_inactive.days.ago) - -inactive_users.each do |user| - puts "user '#{user.username}': #{user.last_activity_on}" - user.block! -end -``` - -### Find a user's max permissions for project/group - -```ruby -user = User.find_by_username 'username' -project = Project.find_by_full_path 'group/project' -user.max_member_access_for_project project.id -``` - -```ruby -user = User.find_by_username 'username' -group = Group.find_by_full_path 'group' -user.max_member_access_for_group group.id -``` - -## Merge requests - -### Close a merge request - -```ruby -u = User.find_by_username('<username>') -p = Project.find_by_full_path('<namespace/project>') -m = p.merge_requests.find_by(iid: <iid>) -MergeRequests::CloseService.new(project: p, current_user: u).execute(m) -``` - -### Delete a merge request - -```ruby -u = User.find_by_username('<username>') -p = Project.find_by_full_path('<namespace/project>') -m = p.merge_requests.find_by(iid: <iid>) -Issuable::DestroyService.new(project: m.project, current_user: u).execute(m) -``` - -### Rebase manually - -```ruby -u = User.find_by_username('<username>') -p = Project.find_by_full_path('<namespace/project>') -m = p.merge_requests.find_by(iid: <iid>) -MergeRequests::RebaseService.new(project: m.target_project, current_user: u).execute(m) -``` - -### Set a merge request as merged - -Use when a merge request was accepted and the changes merged into the Git repository, -but the merge request still shows as open. - -If the changes are not merged yet, this action causes the merge request to -incorrectly show `merged into <branch-name>`. - -```ruby -u = User.find_by_username('<username>') -p = Project.find_by_full_path('<namespace/project>') -m = p.merge_requests.find_by(iid: <iid>) -MergeRequests::PostMergeService.new(project: p, current_user: u).execute(m) -``` - -## CI - -### Cancel stuck pending pipelines - -For more information, see the [confidential issue](../../user/project/issues/confidential_issues.md) -`https://gitlab.com/gitlab-com/support-forum/issues/2449#note_41929707`. - -```ruby -Ci::Pipeline.where(project_id: p.id).where(status: 'pending').count -Ci::Pipeline.where(project_id: p.id).where(status: 'pending').each {|p| p.cancel if p.stuck?} -Ci::Pipeline.where(project_id: p.id).where(status: 'pending').count -``` - -### Remove artifacts more than a week old - -This section has been moved to the [job artifacts troubleshooting documentation](../job_artifacts.md#delete-job-artifacts-from-jobs-completed-before-a-specific-date). - -### Find reason failure (for when build trace is empty) (Introduced in 10.3.0) - -See <https://gitlab.com/gitlab-org/gitlab-foss/-/issues/41111>. - -```ruby -build = Ci::Build.find(78420) - -build.failure_reason - -build.dependencies.each do |d| { puts "status: #{d.status}, finished at: #{d.finished_at}, - completed: #{d.complete?}, artifacts_expired: #{d.artifacts_expired?}, erased: #{d.erased?}" } -``` - -### Try CI integration - -```ruby -p = Project.find_by_full_path('<project_path>') -m = project.merge_requests.find_by(iid: ) -m.project.try(:ci_integration) -``` - -### Validate the `.gitlab-ci.yml` - -```ruby -project = Project.find_by_full_path 'group/project' -content = project.repository.gitlab_ci_yml_for(project.repository.root_ref_sha) -Gitlab::Ci::Lint.new(project: project, current_user: User.first).validate(content) -``` - -### Disable AutoDevOps on Existing Projects - -```ruby -Project.all.each do |p| - p.auto_devops_attributes={"enabled"=>"0"} - p.save -end -``` - -### Obtain runners registration token - -```ruby -Gitlab::CurrentSettings.current_application_settings.runners_registration_token -``` - -### Seed runners registration token - -```ruby -appSetting = Gitlab::CurrentSettings.current_application_settings -appSetting.set_runners_registration_token('<new-runners-registration-token>') -appSetting.save! -``` - -### Run pipeline schedules manually - -You can run pipeline schedules manually through the Rails console to reveal any errors that are usually not visible. - -```ruby -# schedule_id can be obtained from Edit Pipeline Schedule page -schedule = Ci::PipelineSchedule.find_by(id: <schedule_id>) - -# Select the user that you want to run the schedule for -user = User.find_by_username('<username>') - -# Run the schedule -ps = Ci::CreatePipelineService.new(schedule.project, user, ref: schedule.ref).execute!(:schedule, ignore_skip_ci: true, save_on_errors: false, schedule: schedule) -``` - -## License - -### See current license information - -```ruby -# License information (name, company, email address) -License.current.licensee - -# Plan: -License.current.plan - -# Uploaded: -License.current.created_at - -# Started: -License.current.starts_at - -# Expires at: -License.current.expires_at - -# Is this a trial license? -License.current.trial? - -# License ID for lookup on CustomersDot -License.current.license_id - -# License data in Base64-encoded ASCII format -License.current.data -``` - -### Check if a project feature is available on the instance - -Features listed in <https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/models/license.rb>. - -```ruby -License.current.feature_available?(:jira_dev_panel_integration) -``` - -### Check if a project feature is available in a project - -Features listed in [`license.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/models/license.rb). - -```ruby -p = Project.find_by_full_path('<group>/<project>') -p.feature_available?(:jira_dev_panel_integration) -``` - -### Add a license through the console - -```ruby -key = "<key>" -license = License.new(data: key) -license.save -License.current # check to make sure it applied -``` - -This is needed for example in a known edge-case with -[expired license and multiple LDAP servers](../auth/ldap/ldap-troubleshooting.md#expired-license-causes-errors-with-multiple-ldap-servers). - -### Remove licenses - -To clean up the [License History table](../../user/admin_area/license_file.md#view-license-details-and-history): - -```ruby -TYPE = :trial? -# or :expired? - -License.select(&TYPE).each(&:destroy!) - -# or even License.all.each(&:destroy!) -``` - -## Registry - -### Registry Disk Space Usage by Project - -Find this content in the [Container Registry troubleshooting documentation](../packages/container_registry.md#registry-disk-space-usage-by-project). - -### Run the Cleanup policy now - -Find this content in the [Container Registry troubleshooting documentation](../packages/container_registry.md#run-the-cleanup-policy-now). - -## Sidekiq - -This content has been moved to [Troubleshooting Sidekiq](sidekiq.md). - -## LFS - -### Get information about LFS objects and associated project - -```ruby -o = LfsObject.find_by(oid: "<oid>") -p = Project.find(LfsObjectsProject.find_by_lfs_object_id(o.id).project_id) -``` - -You can then delete these records from the database with: - -```ruby -LfsObjectsProject.find_by_lfs_object_id(o.id).destroy -o.destroy -``` - -You would also want to combine this with deleting the LFS file in the LFS storage -area on disk. It remains to be seen exactly how or whether the deletion is useful, however. - -## Decryption Problems - -### Bad Decrypt Script (for encrypted variables) - -This content has been converted to a Rake task, see [verify database values can be decrypted using the current secrets](../raketasks/check.md#verify-database-values-can-be-decrypted-using-the-current-secrets). - -As an example of repairing, if `ProjectImportData Bad count:` is detected and the decision is made to delete the -encrypted credentials to allow manual reentry: - -```ruby - # Find the ids of the corrupt ProjectImportData objects - total = 0 - bad = [] - ProjectImportData.find_each do |data| - begin - total += 1 - data.credentials - rescue => e - bad << data.id - end - end - - puts "Bad count: #{bad.count} / #{total}" - - # See the bad ProjectImportData ids - bad - - # Remove the corrupted credentials - import_data = ProjectImportData.where(id: bad) - import_data.each do |data| - data.update_columns({ encrypted_credentials: nil, encrypted_credentials_iv: nil, encrypted_credentials_salt: nil}) - end -``` - -If `User OTP Secret Bad count:` is detected. For each user listed disable/enable -two-factor authentication. - -The following script searches in some of the tables for encrypted tokens that are -causing decryption errors, and update or reset as needed: - -```shell -wget -O /tmp/encrypted-tokens.rb https://gitlab.com/snippets/1876342/raw -gitlab-rails runner /tmp/encrypted-tokens.rb -``` - -### Decrypt Script for encrypted tokens - -This content has been converted to a Rake task, see [verify database values can be decrypted using the current secrets](../raketasks/check.md#verify-database-values-can-be-decrypted-using-the-current-secrets). - -## Geo - -### Reverify all uploads (or any SSF data type which is verified) - -1. SSH into a GitLab Rails node in the primary Geo site. -1. Open [Rails console](../operations/rails_console.md). -1. Mark all uploads as "pending verification": - - ```ruby - Upload.verification_state_table_class.each_batch do |relation| - relation.update_all(verification_state: 0) - end - ``` - -1. This will cause the primary to start checksumming all Uploads. -1. When a primary successfully checksums a record, then all secondaries rechecksum as well, and they compare the values. - -A similar thing can be done for all Models handled by the [Geo Self-Service Framework](../../development/geo/framework.md) which have implemented verification: - -- `LfsObject` -- `MergeRequestDiff` -- `Packages::PackageFile` -- `Terraform::StateVersion` -- `SnippetRepository` -- `Ci::PipelineArtifact` -- `PagesDeployment` -- `Upload` -- `Ci::JobArtifact` -- `Ci::SecureFile` - -NOTE: -`GroupWikiRepository` is not in the previous list since verification is not implemented. -There is an [issue to implement this functionality in the Admin UI](https://gitlab.com/gitlab-org/gitlab/-/issues/364729). - -### Artifacts - -Moved to [Geo replication troubleshooting](../geo/replication/troubleshooting.md#find-failed-artifacts). - -### Repository verification failures - -Moved to [Geo replication troubleshooting](../geo/replication/troubleshooting.md#repository-verification-failures). - -### Resync repositories - -Moved to [Geo replication troubleshooting](../geo/replication/troubleshooting.md#resync-repositories). - -### Blob types - -Moved to [Geo replication troubleshooting](../geo/replication/troubleshooting.md#blob-types). - -## Generate Service Ping - -The [Service Ping Guide](../../development/service_ping/index.md) in our developer documentation -has more information about Service Ping. - -### Generate or get the cached Service Ping - -```ruby -Gitlab::Usage::ServicePingReport.for(output: :all_metrics_values, cached: true) -``` - -### Generate a fresh new Service Ping - -This also refreshes the cached Service Ping displayed in the Admin Area - -```ruby -Gitlab::Usage::ServicePingReport.for(output: :all_metrics_values) -``` - -### Generate and print - -Generates Service Ping data in JSON format. - -```shell -rake gitlab:usage_data:generate -``` - -Generates Service Ping data in YAML format: - -```shell -rake gitlab:usage_data:dump_sql_in_yaml -``` - -### Generate and send Service Ping - -Prints the metrics saved in `conversational_development_index_metrics`. - -```shell -rake gitlab:usage_data:generate_and_send -``` - -## GraphQL - -Call a [GraphQL](../../api/graphql/getting_started.md) endpoint through the Rails console: - -```ruby -query = <<~EOQ -query securityGetProjects($search: String!) { - projects(search: $search) { - nodes { - path - } - } -} -EOQ - -variables = { "search": "gitlab" } - -result = GitlabSchema.execute(query, variables: variables, context: { current_user: current_user }) -result.to_h -``` +<!-- This redirect file can be deleted after 2023-02-01. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/troubleshooting/group_saml_scim.md b/doc/administration/troubleshooting/group_saml_scim.md deleted file mode 100644 index b5187504231..00000000000 --- a/doc/administration/troubleshooting/group_saml_scim.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../../user/group/saml_sso/example_saml_config.md' -remove_date: '2022-10-29' ---- - -This document was moved to [another location](../../user/group/saml_sso/example_saml_config.md). - -<!-- This redirect file can be deleted after <2022-10-29>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/troubleshooting/index.md b/doc/administration/troubleshooting/index.md index ce548f9e100..db572f202ea 100644 --- a/doc/administration/troubleshooting/index.md +++ b/doc/administration/troubleshooting/index.md @@ -9,19 +9,21 @@ info: To determine the technical writer assigned to the Stage/Group associated w This page documents a collection of resources to help you troubleshoot a GitLab installation. +This list is not necessarily comprehensive. If you don't find what you're looking +for in this list, you should search the documentation. + ## Troubleshooting guides - [SSL](ssl.md) - [Geo](../geo/replication/troubleshooting.md) -- [GitLab Rails console cheat sheet](gitlab_rails_cheat_sheet.md) -- [Example group SAML and SCIM configurations](../../user/group/saml_sso/example_saml_config.md) +- [SAML](../../user/group/saml_sso/troubleshooting.md) - [Kubernetes cheat sheet](https://docs.gitlab.com/charts/troubleshooting/kubernetes_cheat_sheet.html) - [Linux cheat sheet](linux_cheat_sheet.md) - [Parsing GitLab logs with `jq`](../logs/log_parsing.md) - [Diagnostics tools](diagnostics_tools.md) Some feature documentation pages also have a troubleshooting section at the end -that you can check for feature-specific help. +that you can check for feature-specific help, including helpful Rails commands. If you need a testing environment to troubleshoot, see the [apps for a testing environment](test_environments.md). diff --git a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md deleted file mode 100644 index 15ec8d5940b..00000000000 --- a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'https://docs.gitlab.com/charts/troubleshooting/kubernetes_cheat_sheet.html' -remove_date: '2022-10-05' ---- - -This document was moved to [another location](https://docs.gitlab.com/charts/troubleshooting/kubernetes_cheat_sheet.html). - -<!-- This redirect file can be deleted after 2022-10-05. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/troubleshooting/linux_cheat_sheet.md b/doc/administration/troubleshooting/linux_cheat_sheet.md index 0c6fcd31d1d..90cd1e24c79 100644 --- a/doc/administration/troubleshooting/linux_cheat_sheet.md +++ b/doc/administration/troubleshooting/linux_cheat_sheet.md @@ -210,7 +210,7 @@ using the `-s` or `--sort` flag. The number of results defaults to 25 processes, can be changed using the `-c`/`--count` option. See `--help` for full details. ```shell -$ ./strace-parser sidekiq_trace.txt summary -c15 -s=pid +$ ./strace-parser sidekiq_trace.txt summary -c15 -s=pid Top 15 PIDs by PID # ----------- @@ -244,7 +244,7 @@ processes using the `-p`/`--pid` for a specific process, or `-s`/`--stats` flags a sorted list. `--stats` takes the same sorting and count options as summary. ```shell -./strace-parser sidekiq_trace.txt p 16815 +./strace-parser sidekiq_trace.txt p 16815 PID 16815 diff --git a/doc/administration/troubleshooting/log_parsing.md b/doc/administration/troubleshooting/log_parsing.md deleted file mode 100644 index 929a49494be..00000000000 --- a/doc/administration/troubleshooting/log_parsing.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../logs/log_parsing.md' -remove_date: '2022-10-24' ---- - -This document was moved to [another location](../logs/log_parsing.md). - -<!-- This redirect file can be deleted after <2022-10-24>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md b/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md deleted file mode 100644 index 09a5cb8d185..00000000000 --- a/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../operations/rails_console.md' -remove_date: '2022-10-05' ---- - -This document was moved to [another location](../operations/rails_console.md). - -<!-- This redirect file can be deleted after <2022-10-05>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/troubleshooting/postgresql.md b/doc/administration/troubleshooting/postgresql.md index 8a76d4f88ab..829fed38060 100644 --- a/doc/administration/troubleshooting/postgresql.md +++ b/doc/administration/troubleshooting/postgresql.md @@ -58,59 +58,6 @@ This section is for links to information elsewhere in the GitLab documentation. some of which is absolutely not for production use. Including: - Understanding EXPLAIN plans. -### Troubleshooting/Fixes - -- [GitLab database requirements](../../install/requirements.md#database), - including - - Support for MySQL was removed in GitLab 12.1; [migrate to PostgreSQL](../../update/mysql_to_postgresql.md). - - Required extension: `pg_trgm` - - Required extension: `btree_gist` - -- Errors like this in the `production/sidekiq` log; see: - [Set `default_transaction_isolation` into read committed](https://docs.gitlab.com/omnibus/settings/database.html#set-default_transaction_isolation-into-read-committed): - - ```plaintext - ActiveRecord::StatementInvalid PG::TRSerializationFailure: ERROR: could not serialize access due to concurrent update - ``` - -- PostgreSQL HA [replication slot errors](https://docs.gitlab.com/omnibus/settings/database.html#troubleshooting-upgrades-in-an-ha-cluster): - - ```plaintext - pg_basebackup: could not create temporary replication slot "pg_basebackup_12345": ERROR: all replication slots are in use - HINT: Free one or increase max_replication_slots. - ``` - -- Geo [replication errors](../geo/replication/troubleshooting.md#fixing-replication-errors) including: - - ```plaintext - ERROR: replication slots can only be used if max_replication_slots > 0 - - FATAL: could not start WAL streaming: ERROR: replication slot "geo_secondary_my_domain_com" does not exist - - Command exceeded allowed execution time - - PANIC: could not write to file 'pg_xlog/xlogtemp.123': No space left on device - ``` - -- [Checking Geo configuration](../geo/replication/troubleshooting.md), including: - - Reconfiguring hosts/ports. - - Checking and fixing user/password mappings. - -- [Common Geo errors](../geo/replication/troubleshooting.md#fixing-common-errors). - -- Mismatch in `pg_dump` and `psql` versions: - - ```plaintext - Dumping PostgreSQL database gitlabhq_production ... pg_dump: error: server version: 13.3; pg_dump version: 14.2 - pg_dump: error: aborting because of server version mismatch - ``` - - To fix this, see [Backup and restore a non-packaged PostgreSQL database](https://docs.gitlab.com/omnibus/settings/database.html#backup-and-restore-a-non-packaged-postgresql-database). - -- Deploying PostgreSQL on Azure Database for PostgreSQL - Flexible Server may result in an error stating `extension "btree_gist" is not allow-listed for "azure_pg_admin" users in Azure Database for PostgreSQL` - - To resolve the above error, [allow-list the extension](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-extensions#how-to-use-postgresql-extensions) prior to install. - ## Support topics ### Database deadlocks @@ -236,3 +183,96 @@ To temporarily change the statement timeout: 1. Perform the action for which you need a different timeout (for example the backup or the Rails command). 1. Revert the edit in `/var/opt/gitlab/gitlab-rails/etc/database.yml`. + +## Troubleshooting + +### Database is not accepting commands to avoid wraparound data loss + +This error likely means that AUTOVACUUM is failing to complete its run: + +```plaintext +ERROR: database is not accepting commands to avoid wraparound data loss in database "gitlabhq_production" +``` + +To resolve the error, run `VACUUM` manually: + +1. Stop GitLab with the command `gitlab-ctl stop`. +1. Place the database in single-user mode with the command: + + ```shell + /opt/gitlab/embedded/bin/postgres --single -D /var/opt/gitlab/postgresql/data gitlabhq_production + ``` + +1. In the `backend>` prompt, run `VACUUM;`. This command can take several minutes to complete. +1. Wait for the command to complete, then press <kbd>Control</kbd> + <kbd>D</kbd> to exit. +1. Start GitLab with the command `gitlab-ctl start`. + +### GitLab database requirements + +The [database requirements](../../install/requirements.md#database) for GitLab include: + +- Support for MySQL was removed in GitLab 12.1; [migrate to PostgreSQL](../../update/mysql_to_postgresql.md). +- Review and install the [required extension list](../../install/postgresql_extensions.md). + +### Serialization errors in the `production/sidekiq` log + +If you receive errors like this example in your `production/sidekiq` log, read +about [setting `default_transaction_isolation` into read committed](https://docs.gitlab.com/omnibus/settings/database.html#set-default_transaction_isolation-into-read-committed) to fix the problem: + +```plaintext +ActiveRecord::StatementInvalid PG::TRSerializationFailure: ERROR: could not serialize access due to concurrent update +``` + +### PostgreSQL replication slot errors + +If you receive errors like this example, read about how to resolve PostgreSQL HA +[replication slot errors](https://docs.gitlab.com/omnibus/settings/database.html#troubleshooting-upgrades-in-an-ha-cluster): + +```plaintext +pg_basebackup: could not create temporary replication slot "pg_basebackup_12345": ERROR: all replication slots are in use +HINT: Free one or increase max_replication_slots. +``` + +### Geo replication errors + +If you receive errors like this example, read about how to resolve +[Geo replication errors](../geo/replication/troubleshooting.md#fixing-postgresql-database-replication-errors): + +```plaintext +ERROR: replication slots can only be used if max_replication_slots > 0 + +FATAL: could not start WAL streaming: ERROR: replication slot "geo_secondary_my_domain_com" does not exist + +Command exceeded allowed execution time + +PANIC: could not write to file 'pg_xlog/xlogtemp.123': No space left on device +``` + +### Review Geo configuration and common errors + +When troubleshooting problems with Geo, you should: + +- Review [common Geo errors](../geo/replication/troubleshooting.md#fixing-common-errors). +- [Review your Geo configuration](../geo/replication/troubleshooting.md), including: + - Reconfiguring hosts and ports. + - Reviewing and fixing the user and password mappings. + +### Mismatch in `pg_dump` and `psql` versions + +If you receive errors like this example, read about how to +[back up and restore a non-packaged PostgreSQL database](https://docs.gitlab.com/omnibus/settings/database.html#backup-and-restore-a-non-packaged-postgresql-database): + +```plaintext +Dumping PostgreSQL database gitlabhq_production ... pg_dump: error: server version: 13.3; pg_dump version: 14.2 +pg_dump: error: aborting because of server version mismatch +``` + +### Extension `btree_gist` is not allow-listed + +Deploying PostgreSQL on an Azure Database for PostgreSQL - Flexible Server may result in this error: + +```plaintext +extension "btree_gist" is not allow-listed for "azure_pg_admin" users in Azure Database for PostgreSQL +``` + +To resolve this error, [allow-list the extension](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-extensions#how-to-use-postgresql-extensions) prior to install. diff --git a/doc/administration/troubleshooting/ssl.md b/doc/administration/troubleshooting/ssl.md index 245ff9f4982..bf8e6b45fde 100644 --- a/doc/administration/troubleshooting/ssl.md +++ b/doc/administration/troubleshooting/ssl.md @@ -1,263 +1,11 @@ --- -stage: Systems -group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -type: reference +redirect_to: 'https://docs.gitlab.com/omnibus/settings/ssl/ssl_troubleshooting.html' +remove_date: '2023-10-27' --- -# Troubleshooting SSL **(FREE SELF)** +This document was moved to [another location](https://docs.gitlab.com/omnibus/settings/ssl/ssl_troubleshooting.html). -This page contains a list of common SSL-related errors and scenarios that you -may encounter while working with GitLab. It should serve as an addition to the -main SSL documentation: - -- [Omnibus SSL Configuration](https://docs.gitlab.com/omnibus/settings/ssl.html). -- [Self-signed certificates or custom Certification Authorities for GitLab Runner](https://docs.gitlab.com/runner/configuration/tls-self-signed.html). -- [Configure HTTPS manually](https://docs.gitlab.com/omnibus/settings/ssl.html#configure-https-manually). - -## Using an internal CA certificate with GitLab - -After configuring a GitLab instance with an internal CA certificate, you might -not be able to access it by using various CLI tools. You may experience the -following issues: - -- `curl` fails: - - ```shell - curl "https://gitlab.domain.tld" - curl: (60) SSL certificate problem: unable to get local issuer certificate - More details here: https://curl.haxx.se/docs/sslcerts.html - ``` - -- Testing by using the [rails console](../operations/rails_console.md#starting-a-rails-console-session) - also fails: - - ```ruby - uri = URI.parse("https://gitlab.domain.tld") - http = Net::HTTP.new(uri.host, uri.port) - http.use_ssl = true - http.verify_mode = 1 - response = http.request(Net::HTTP::Get.new(uri.request_uri)) - ... - Traceback (most recent call last): - 1: from (irb):5 - OpenSSL::SSL::SSLError (SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)) - ``` - -- The error `SSL certificate problem: unable to get local issuer certificate` - is displayed when setting up a [mirror](../../user/project/repository/mirror/index.md) - from this GitLab instance. -- `openssl` works when specifying the path to the certificate: - - ```shell - /opt/gitlab/embedded/bin/openssl s_client -CAfile /root/my-cert.crt -connect gitlab.domain.tld:443 -servername gitlab.domain.tld - ``` - -If you have the previously described issues, add your certificate to -`/etc/gitlab/trusted-certs`, and then run `sudo gitlab-ctl reconfigure`. - -## X.509 key values mismatch error - -After configuring your instance with a certificate bundle, NGINX may display -the following error message: - -`SSL: error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch` - -This error message means that the server certificate and key you have provided -don't match. You can confirm this by running the following command and then -comparing the output: - -```shell -openssl rsa -noout -modulus -in path/to/your/.key | openssl md5 -openssl x509 -noout -modulus -in path/to/your/.crt | openssl md5 -``` - -The following is an example of an md5 output between a matching key and -certificate. Note the matching md5 hashes: - -```shell -$ openssl rsa -noout -modulus -in private.key | openssl md5 -4f49b61b25225abeb7542b29ae20e98c -$ openssl x509 -noout -modulus -in public.crt | openssl md5 -4f49b61b25225abeb7542b29ae20e98c -``` - -This is an opposing output with a non-matching key and certificate which shows -different md5 hashes: - -```shell -$ openssl rsa -noout -modulus -in private.key | openssl md5 -d418865077299af27707b1d1fa83cd99 -$ openssl x509 -noout -modulus -in public.crt | openssl md5 -4f49b61b25225abeb7542b29ae20e98c -``` - -If the two outputs differ like the previous example, there's a mismatch between -the certificate and key. Contact the provider of the SSL certificate for -further support. - -## Using GitLab Runner with a GitLab instance configured with internal CA certificate or self-signed certificate - -Besides getting the errors mentioned in -[Using an internal CA certificate with GitLab](ssl.md#using-an-internal-ca-certificate-with-gitlab), -your CI pipelines may get stuck in `Pending` status. In the runner logs you may -see the following error message: - -```shell -Dec 6 02:43:17 runner-host01 gitlab-runner[15131]: #033[0;33mWARNING: Checking for jobs... failed -#033[0;m #033[0;33mrunner#033[0;m=Bfkz1fyb #033[0;33mstatus#033[0;m=couldn't execute POST against -https://gitlab.domain.tld/api/v4/jobs/request: Post https://gitlab.domain.tld/api/v4/jobs/request: -x509: certificate signed by unknown authority -``` - -Follow the details in [Self-signed certificates or custom Certification Authorities for GitLab Runner](https://docs.gitlab.com/runner/configuration/tls-self-signed.html). - -## Mirroring a remote GitLab repository that uses a self-signed SSL certificate - -When configuring a local GitLab instance to [mirror a repository](../../user/project/repository/mirror/index.md) -from a remote GitLab instance that uses a self-signed certificate, you may see -the `SSL certificate problem: self signed certificate` error message in the -user interface. - -The cause of the issue can be confirmed by checking if: - -- `curl` fails: - - ```shell - $ curl "https://gitlab.domain.tld" - curl: (60) SSL certificate problem: self signed certificate - More details here: https://curl.haxx.se/docs/sslcerts.html - ``` - -- Testing by using the Rails console also fails: - - ```ruby - uri = URI.parse("https://gitlab.domain.tld") - http = Net::HTTP.new(uri.host, uri.port) - http.use_ssl = true - http.verify_mode = 1 - response = http.request(Net::HTTP::Get.new(uri.request_uri)) - ... - Traceback (most recent call last): - 1: from (irb):5 - OpenSSL::SSL::SSLError (SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)) - ``` - -To fix this problem: - -- Add the self-signed certificate from the remote GitLab instance to the - `/etc/gitlab/trusted-certs` directory on the local GitLab instance, and then - run `sudo gitlab-ctl reconfigure` as per the instructions for - [installing custom public certificates](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). -- If your local GitLab instance was installed using the Helm Charts, you can - [add your self-signed certificate to your GitLab instance](https://docs.gitlab.com/runner/install/kubernetes.html#providing-a-custom-certificate-for-accessing-gitlab). - -You may also get another error message when trying to mirror a repository from -a remote GitLab instance that uses a self-signed certificate: - -```shell -2:Fetching remote upstream failed: fatal: unable to access &#39;https://gitlab.domain.tld/root/test-repo/&#39;: -SSL: unable to obtain common name from peer certificate -``` - -In this case, the problem can be related to the certificate itself: - -1. Validate that your self-signed certificate isn't missing a common name. If it - is, regenerate a valid certificate -1. Add the certificate to `/etc/gitlab/trusted-certs`. -1. Run `sudo gitlab-ctl reconfigure`. - -## Unable to perform Git operations due to an internal or self-signed certificate - -If your GitLab instance is using a self-signed certificate, or if the -certificate is signed by an internal certificate authority (CA), you might -experience the following errors when attempting to perform Git operations: - -```shell -$ git clone https://gitlab.domain.tld/group/project.git -Cloning into 'project'... -fatal: unable to access 'https://gitlab.domain.tld/group/project.git/': SSL certificate problem: self signed certificate -``` - -```shell -$ git clone https://gitlab.domain.tld/group/project.git -Cloning into 'project'... -fatal: unable to access 'https://gitlab.domain.tld/group/project.git/': server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none -``` - -To fix this problem: - -- If possible, use SSH remotes for all Git operations. This is considered more - secure and convenient to use. -- If you must use HTTPS remotes, you can try the following: - - Copy the self-signed certificate or the internal root CA certificate to a - local directory (for example, `~/.ssl`) and configure Git to trust your - certificate: - - ```shell - git config --global http.sslCAInfo ~/.ssl/gitlab.domain.tld.crt - ``` - - - Disable SSL verification in your Git client. This is intended as a - temporary measure, as it could be considered a security risk. - - ```shell - git config --global http.sslVerify false - ``` - -## SSL_connect wrong version number - -A misconfiguration may result in: - -- `gitlab-rails/exceptions_json.log` entries containing: - - ```plaintext - "exception.class":"Excon::Error::Socket","exception.message":"SSL_connect returned=1 errno=0 state=error: wrong version number (OpenSSL::SSL::SSLError)", - "exception.class":"Excon::Error::Socket","exception.message":"SSL_connect returned=1 errno=0 state=error: wrong version number (OpenSSL::SSL::SSLError)", - ``` - -- `gitlab-workhorse/current` containing: - - ```plaintext - http: server gave HTTP response to HTTPS client - http: server gave HTTP response to HTTPS client - ``` - -- `gitlab-rails/sidekiq.log` or `sidekiq/current` containing: - - ```plaintext - message: SSL_connect returned=1 errno=0 state=error: wrong version number (OpenSSL::SSL::SSLError) - message: SSL_connect returned=1 errno=0 state=error: wrong version number (OpenSSL::SSL::SSLError) - ``` - -Some of these errors come from the Excon Ruby gem, and could be generated in -circumstances where GitLab is configured to initiate an HTTPS session to a -remote server that is serving only HTTP. - -One scenario is that you're using [object storage](../object_storage.md), which -isn't served under HTTPS. GitLab is misconfigured and attempts a TLS handshake, -but the object storage responds with plain HTTP. - -## `schannel: SEC_E_UNTRUSTED_ROOT` - -If you're on Windows and get the following error: - -```plaintext -Fatal: unable to access 'https://gitlab.domain.tld/group/project.git': schannel: SEC_E_UNTRUSTED_ROOT (0x80090325) - The certificate chain was issued by an authority that is not trusted." -``` - -You must specify that Git should use OpenSSL: - -```shell -git config --system http.sslbackend openssl -``` - -Alternatively, you can ignore SSL verification by running: - -WARNING: -Proceed with caution when [ignoring SSL](https://git-scm.com/docs/git-config#Documentation/git-config.txt-httpsslVerify) -due to the potential security issues associated with disabling this option at global level. Use this option _only_ when troubleshooting, and reinstate SSL verification immediately after. - -```shell -git config --global http.sslVerify false -``` +<!-- This redirect file can be deleted after <2023-01-25>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/troubleshooting/test_environments.md b/doc/administration/troubleshooting/test_environments.md index a249d5bd133..c4d191f8c0f 100644 --- a/doc/administration/troubleshooting/test_environments.md +++ b/doc/administration/troubleshooting/test_environments.md @@ -20,13 +20,13 @@ are only available internally at GitLab. ## Docker The following were tested on Docker containers running in the cloud. Support Engineers, -please see [these docs](https://gitlab.com/gitlab-com/dev-resources/tree/master/dev-resources#running-docker-containers) +see [these docs](https://gitlab.com/gitlab-com/dev-resources/tree/master/dev-resources#running-docker-containers) on how to run Docker containers on `dev-resources`. Other setups haven't been tested, but contributions are welcome. ### GitLab -Please see [our official Docker installation method](../../install/docker.md) +See [our official Docker installation method](../../install/docker.md) for how to run GitLab on Docker. ### SAML diff --git a/doc/administration/user_settings.md b/doc/administration/user_settings.md index a767132db0f..c96a6311208 100644 --- a/doc/administration/user_settings.md +++ b/doc/administration/user_settings.md @@ -14,7 +14,7 @@ By default, new users can create top-level groups. To disable new users' ability to create top-level groups (does not affect existing users' setting), GitLab administrators can modify this setting: - In GitLab 15.5 and later, using either: - - The [GitLab UI](../user/admin_area/settings/account_and_limit_settings.md#prevent-users-from-creating-top-level-groups). + - The [GitLab UI](../user/admin_area/settings/account_and_limit_settings.md#prevent-new-users-from-creating-top-level-groups). - The [application setting API](../api/settings.md#change-application-settings). - In GitLab 15.4 and earlier, in a configuration file by following the steps in this section. |