diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2022-05-19 07:33:21 +0000 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2022-05-19 07:33:21 +0000 |
| commit | 36a59d088eca61b834191dacea009677a96c052f (patch) | |
| tree | e4f33972dab5d8ef79e3944a9f403035fceea43f /doc/administration | |
| parent | a1761f15ec2cae7c7f7bbda39a75494add0dfd6f (diff) | |
| download | gitlab-ce-36a59d088eca61b834191dacea009677a96c052f.tar.gz | |
Add latest changes from gitlab-org/gitlab@15-0-stable-eev15.0.0-rc42
Diffstat (limited to 'doc/administration')
86 files changed, 1685 insertions, 1072 deletions
diff --git a/doc/administration/audit_event_streaming.md b/doc/administration/audit_event_streaming.md index 6cf630ad079..07979e21038 100644 --- a/doc/administration/audit_event_streaming.md +++ b/doc/administration/audit_event_streaming.md @@ -132,13 +132,11 @@ Delete an event streaming destination by specifying an ID. Get the required ID b streaming destinations. ```graphql - -mutation{ +mutation { externalAuditEventDestinationDestroy(input: { id: destination }) { errors } } - ``` Destination is deleted if: @@ -158,10 +156,11 @@ the destination's value when [listing streaming destinations](#list-streaming-de ## Audit event streaming on Git operations -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332747) in GitLab 14.9 [with a flag](../administration/feature_flags.md) named `audit_event_streaming_git_operations`. Disabled by default. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332747) in GitLab 14.9 [with a flag](../administration/feature_flags.md) named `audit_event_streaming_git_operations`. Disabled by default. +> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/357211) in GitLab 15.0. FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](feature_flags.md) named `audit_event_streaming_git_operations`. On GitLab.com, this feature is not available. +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](feature_flags.md) named `audit_event_streaming_git_operations`. On GitLab.com, this feature is available. Streaming audit events can be sent when signed-in users push or pull a project's remote Git repositories: diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md index 36f3a71cd2e..6fac21ef889 100644 --- a/doc/administration/audit_events.md +++ b/doc/administration/audit_events.md @@ -10,6 +10,7 @@ GitLab offers a way to view the changes made within the GitLab server for owners on a [paid plan](https://about.gitlab.com/pricing/). GitLab system administrators can also view all audit events by accessing the [`audit_json.log` file](logs.md#audit_jsonlog). +The JSON audit log does not include events that are [only streamed](../development/audit_event_guide/index.md#event-streaming). You can: @@ -33,7 +34,7 @@ permission level, who added a new user, or who removed a user. ## Retention policy There is no retention policy in place for audit events. -See the [Specify a retention period for audit events](https://gitlab.com/gitlab-org/gitlab/-/issues/8137) for more information. +See the [Specify a retention period for audit events](https://gitlab.com/groups/gitlab-org/-/epics/7917) for more information. ## List of events @@ -114,6 +115,8 @@ From there, you can see the following actions: - Instance administrator started or stopped impersonation of a group member. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/300961) in GitLab 14.8. - Group deploy token was successfully created, revoked, or deleted. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/353452) in GitLab 14.9. - Failed attempt to create a group deploy token. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/353452) in GitLab 14.9. +- [IP restrictions](../user/group/index.md#restrict-group-access-by-ip-address) changed. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/358986) in GitLab 15.0 +- Changes to push rules. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/227629) in GitLab 15.0. Group events can also be accessed via the [Group Audit Events API](../api/audit_events.md#group-audit-events) @@ -178,6 +181,11 @@ From there, you can see the following actions: - Skipped pipelines are considered successful enabled or disabled ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9) - All discussions must be resolved enabled or disabled ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9) - Commit message suggestion is updated ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301124) in GitLab 14.9) +- Status check is added, edited, or deleted ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/355805) in GitLab 15.0) +- Merge commit message template is updated ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/355805) in GitLab 15.0) +- Squash commit message template is updated ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/355805) in GitLab 15.0) +- Default description template for merge requests is updated ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/355805) in GitLab 15.0) +- Project was scheduled for deletion due to inactivity ([introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/85689) in GitLab 15.0) Project events can also be accessed via the [Project Audit Events API](../api/audit_events.md#project-audit-events). @@ -258,36 +266,15 @@ Don't see the event you want in any of the epics linked above? You can either: request it. - [Add it yourself](../development/audit_event_guide/). -### Disabled events +### Removed events -#### Repository push (DEPRECATED) +> - Repositories push events was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/337993) in GitLab 14.3. +> - Repositories push events was [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/337993) in GitLab 15.0. -> [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/337993) in GitLab 14.3. +The repositories push events feature was: -WARNING: -This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/337993) in GitLab 14.3. - -The current architecture of audit events is not prepared to receive a very high amount of records. -It may make the user interface for your project or audit events very busy, and the disk space consumed by the -`audit_events` PostgreSQL table may increase considerably. It's disabled by default -to prevent performance degradations on GitLab instances with very high Git write traffic. - -If you still wish to enable **Repository push** events in your instance, follow -the steps below. - -**In Omnibus installations:** - -1. Enter the Rails console: - - ```shell - sudo gitlab-rails console - ``` - -1. Flip the switch and enable the feature flag: - - ```ruby - Feature.enable(:repository_push_audit_event) - ``` +- [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/337993) in GitLab 14.3. +- [Removed](https://gitlab.com/gitlab-org/gitlab/-/issues/337993) in GitLab 15.0. ## Search diff --git a/doc/administration/auditor_users.md b/doc/administration/auditor_users.md index b3304fd1cbd..1d0aff51a04 100644 --- a/doc/administration/auditor_users.md +++ b/doc/administration/auditor_users.md @@ -6,88 +6,52 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Auditor users **(PREMIUM SELF)** -Auditor users are given read-only access to all projects, groups, and other -resources on the GitLab instance. - -## Overview - -Auditor users are able to have both full access to their own resources -(including projects, groups, and snippets) and read-only access to _all_ other -resources, except the [Admin Area](../user/admin_area/index.md). These user -accounts are regular users who can be added to projects, create personal -snippets, and create milestones on their groups, while also having read-only -access to all projects on the server to which they haven't been explicitly -[given access](../user/permissions.md). - -The `Auditor` access level is _not_ a read-only version of the `Admin` access level. Auditor users -can't access the project or group settings pages, or the Admin Area. - -Assuming you have signed in as an Auditor user: - -- For a project the Auditor is not member of, the Auditor should have - read-only access. If the project is public or internal, they have the same - access as users that aren't members of that project or group. -- For a project the Auditor owns, the Auditor should have full access to - everything. -- For a project to which the Auditor is added as a member, the Auditor should - have the same access as their given [permissions](../user/permissions.md). - For example, if they were added as a Developer, they can push commits or - comment on issues. -- The Auditor can't view the Admin Area, or perform any administration actions. - -For more information about what an Auditor can or can't do, see the -[Permissions and restrictions of an Auditor user](#permissions-and-restrictions-of-an-auditor-user) -section. +Users with auditor access have read-only access to all groups, projects, and other resources except: + +- The [Admin Area](../user/admin_area/index.md). +- Project and group settings. -## Use cases +For more information, see [Auditor user permissions and restrictions](#auditor-user-permissions-and-restrictions) +section. -The following use cases describe some situations where Auditor users could be -helpful: +Situations where auditor access for users could be helpful include: - Your compliance department wants to run tests against the entire GitLab base to ensure users are complying with password, credit card, and other sensitive - data policies. With Auditor users, this can be achieved very without having - to give them user administration rights or using the API to add them to all projects. + data policies. You can achieve this with auditor access without giving the compliance department + user administration rights or adding them to all projects. - If particular users need visibility or access to most of all projects in your GitLab instance, instead of manually adding the user to all projects, - you can create an Auditor user and then share the credentials with those users - to which you want to grant access. + you can create an account with auditor access and then share the credentials + with those users to which you want to grant access. -## Add an Auditor user +## Add a user with auditor access -To create an Auditor user: +To create a new user account with auditor access (or change an existing user): + +To create a user account with auditor access: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Overview > Users**. -1. Create a new user or edit an existing one, and in the **Access** section - select Auditor. +1. Create a new user or edit an existing one. Set **Access Level** to **Auditor**. 1. If you created a user, select **Create user**. For an existing user, select **Save changes**. -To revoke Auditor permissions from a user, make them a Regular user by -following the previous steps. - -Additionally users can be set as an Auditor using [SAML groups](../integration/saml.md#auditor-groups). - -## Permissions and restrictions of an Auditor user - -An Auditor user should be able to access all projects and groups of a GitLab -instance, with the following permissions and restrictions: - -- Has read-only access to the API -- Can access projects that are: - - Private - - Public - - Internal -- Can read all files in a repository -- Can read issues and MRs -- Can read project snippets -- Cannot be Administrator and Auditor at the same time -- Cannot access the Admin Area -- In a group or project they're not a member of: - - Cannot access project settings - - Cannot access group settings - - Cannot commit to repository - - Cannot create or comment on issues and MRs - - Cannot create or modify files from the Web UI - - Cannot merge a merge request - - Cannot create project snippets +To revoke auditor access from a user, follow these steps but set **Access Level** to **Regular**. + +You can also give users auditor access using [SAML groups](../integration/saml.md#auditor-groups). + +## Auditor user permissions and restrictions + +Auditor access is _not_ a read-only version of administrator access because it doesn't permit access to the Admin Area. + +For access to their own resources and resources within a group or project where they are a member, +users with auditor access have the same [permissions](../user/permissions.md) as regular users. + +If you are signed in with auditor access, you: + +- Have full access to projects you own. +- Have read-only access to projects you aren't a member of. +- Have [permissions](../user/permissions.md) based on your role to projects you are a member of. For example, if you have the Developer role, + you can push commits or comment on issues. +- Can access the same resources using the GitLab UI or API. +- Can't view the Admin Area, or perform any administration actions. diff --git a/doc/administration/auth/ldap/index.md b/doc/administration/auth/ldap/index.md index a7e070b755a..9232e79b800 100644 --- a/doc/administration/auth/ldap/index.md +++ b/doc/administration/auth/ldap/index.md @@ -468,7 +468,7 @@ If initially your LDAP configuration looked like: password: '123' ``` -1. Edit `/etc/gitlab/gitlab.rb` and remove the settings for `user_dn` and `password`. +1. Edit `/etc/gitlab/gitlab.rb` and remove the settings for `bind_dn` and `password`. 1. [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -502,7 +502,7 @@ If initially your LDAP configuration looked like: password: '123' ``` -1. Edit `config/gitlab.yaml` and remove the settings for `user_dn` and `password`. +1. Edit `config/gitlab.yaml` and remove the settings for `bind_dn` and `password`. 1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect. @@ -531,16 +531,19 @@ However, these users can continue to use Git with SSH until the next time the To delete the account immediately, you can manually [block the user](../../../user/admin_area/moderate_users.md#block-a-user). -## Updating user email addresses +## Update user email addresses -Email addresses on the LDAP server are considered the source of truth for users when LDAP is used to sign in. Updating user email -addresses must be done on the LDAP server that manages the user. The email address for GitLab is updated either: +Email addresses on the LDAP server are considered the source of truth for users when LDAP is used to sign in. + +Updating user email addresses must be done on the LDAP server that manages the user. The email address for GitLab is updated either: - When the user next signs in. - When the next [user sync](ldap_synchronization.md#user-sync) is run. The updated user's previous email address becomes the secondary email address to preserve that user's commit history. +You can find more details on the expected behavior of user updates in our [LDAP troubleshooting section](ldap-troubleshooting.md#user-dn-orand-email-have-changed). + ## Google Secure LDAP > Introduced in GitLab 11.9. diff --git a/doc/administration/auth/ldap/ldap_synchronization.md b/doc/administration/auth/ldap/ldap_synchronization.md index d7ce54a47c0..0f0d301bfa9 100644 --- a/doc/administration/auth/ldap/ldap_synchronization.md +++ b/doc/administration/auth/ldap/ldap_synchronization.md @@ -190,8 +190,8 @@ to lock down user abilities to invite new members to a group. When enabled, the following applies: -- Only administrator can manage memberships of any group including access levels. -- Users are not allowed to share project with other groups or invite members to +- Only an administrator can manage memberships of any group including access levels. +- Users are not allowed to share a project with other groups or invite members to a project created in a group. To enable it, you must: diff --git a/doc/administration/configure.md b/doc/administration/configure.md index e42eb632f14..cba4a23d95a 100644 --- a/doc/administration/configure.md +++ b/doc/administration/configure.md @@ -30,7 +30,7 @@ The tables lists features on the left and provides their capabilities to the rig ## Geo Capabilities -If your availabity needs to span multiple zones or multiple locations, please read about [Geo](geo/index.md). +If your availability needs to span multiple zones or multiple locations, please read about [Geo](geo/index.md). | | Availability | Recoverability | Data Resiliency | Performance | Risks/Trade-offs| |-|--------------|----------------|-----------------|-------------|-----------------| @@ -44,7 +44,7 @@ The following table outlines failure modes and mitigation paths for the product | ----------- | -------------------------- | ----------------------------- | ---------------------------------- | ----- | | Single Gitaly Node | Downtime - Must restore from backup | Downtime - Must restore from Backup | Downtime - Must wait for outage to end | | | Single Gitaly Node + Geo Secondary | Downtime - Must restore from backup, can perform a manual failover to secondary | Downtime - Must restore from Backup, errors could have propagated to secondary | Manual intervention - failover to Geo secondary | | -| Sharded Gitaly Install | Partial Downtime - Only repos on impacted node affected, must restore from backup | Partial Downtime - Only repos on impacted node affected, must restore from backup | Downtime - Must wait for outage to end | | -| Sharded Gitaly Install + Geo Secondary | Partial Downtime - Only repos on impacted node affected, must restore from backup, could perform manual failover to secondary for impacted repos | Partial Downtime - Only repos on impacted node affected, must restore from backup, errors could have propagated to secondary | Manual intervention - failover to Geo secondary | | +| Sharded Gitaly Install | Partial Downtime - Only repositories on impacted node affected, must restore from backup | Partial Downtime - Only repositories on impacted node affected, must restore from backup | Downtime - Must wait for outage to end | | +| Sharded Gitaly Install + Geo Secondary | Partial Downtime - Only repositories on impacted node affected, must restore from backup, could perform manual failover to secondary for impacted repositories | Partial Downtime - Only repositories on impacted node affected, must restore from backup, errors could have propagated to secondary | Manual intervention - failover to Geo secondary | | | Gitaly Cluster Install* | No Downtime - will swap repository primary to another node after 10 seconds | N/A - All writes are voted on by multiple Gitaly Cluster nodes | Downtime - Must wait for outage to end | Snapshot backups for Gitaly Cluster nodes not supported at this time | | Gitaly Cluster Install* + Geo Secondary | No Downtime - will swap repository primary to another node after 10 seconds | N/A - All writes are voted on by multiple Gitaly Cluster nodes | Manual intervention - failover to Geo secondary | Snapshot backups for Gitaly Cluster nodes not supported at this time | diff --git a/doc/administration/encrypted_configuration.md b/doc/administration/encrypted_configuration.md index baa6e967507..5c943e56b98 100644 --- a/doc/administration/encrypted_configuration.md +++ b/doc/administration/encrypted_configuration.md @@ -11,7 +11,7 @@ type: reference GitLab can read settings for certain features from encrypted settings files. The supported features are: -- [LDAP `user_bn` and `password`](auth/ldap/index.md#use-encrypted-credentials). +- [LDAP `bind_dn` and `password`](auth/ldap/index.md#use-encrypted-credentials). - [SMTP `user_name` and `password`](raketasks/smtp.md#secrets). In order to enable the encrypted configuration settings, a new base key needs to be generated for diff --git a/doc/administration/environment_variables.md b/doc/administration/environment_variables.md index 22159b6e9db..1fe5b223f8b 100644 --- a/doc/administration/environment_variables.md +++ b/doc/administration/environment_variables.md @@ -37,31 +37,6 @@ You can use the following environment variables to override certain values: | `RAILS_ENV` | string | The Rails environment; can be one of `production`, `development`, `staging`, or `test`. | | `UNSTRUCTURED_RAILS_LOG` | string | Enables the unstructured log in addition to JSON logs (defaults to `true`). | -## Complete database variables - -The recommended method for specifying your database connection information is -to set the `DATABASE_URL` environment variable. This variable contains -connection information (`adapter`, `database`, `username`, `password`, `host`, -and `port`), but no behavior information (`encoding` or `pool`). If you don't -want to use `DATABASE_URL`, or want to set database behavior information, -either: - -- Copy the template file, `cp config/database.yml.env config/database.yml`. -- Set a value for some `GITLAB_DATABASE_XXX` variables. - -The list of `GITLAB_DATABASE_XXX` variables that you can set is: - -| Variable | Default value | Overridden by `DATABASE_URL`? | -|-----------------------------|--------------------------------|-------------------------------| -| `GITLAB_DATABASE_ADAPTER` | `postgresql` | **{check-circle}** Yes | -| `GITLAB_DATABASE_DATABASE` | `gitlab_#{ENV['RAILS_ENV']` | **{check-circle}** Yes | -| `GITLAB_DATABASE_ENCODING` | `unicode` | **{dotted-circle}** No | -| `GITLAB_DATABASE_HOST` | `localhost` | **{check-circle}** Yes | -| `GITLAB_DATABASE_PASSWORD` | _none_ | **{check-circle}** Yes | -| `GITLAB_DATABASE_POOL` | `10` | **{dotted-circle}** No | -| `GITLAB_DATABASE_PORT` | `5432` | **{check-circle}** Yes | -| `GITLAB_DATABASE_USERNAME` | `root` | **{check-circle}** Yes | - ## Adding more variables We welcome merge requests to make more settings configurable by using variables. diff --git a/doc/administration/geo/disaster_recovery/bring_primary_back.md b/doc/administration/geo/disaster_recovery/bring_primary_back.md index 64673b9ee8d..4e68cc39580 100644 --- a/doc/administration/geo/disaster_recovery/bring_primary_back.md +++ b/doc/administration/geo/disaster_recovery/bring_primary_back.md @@ -42,7 +42,7 @@ To bring the former **primary** site up to date: NOTE: If you [changed the DNS records](index.md#step-4-optional-updating-the-primary-domain-dns-record) for this site during disaster recovery procedure you may need to [block - all the writes to this site](planned_failover.md#prevent-updates-to-the-primary-node) + all the writes to this site](planned_failover.md#prevent-updates-to-the-primary-site) during this procedure. 1. [Set up database replication](../setup/database.md). In this case, the **secondary** site diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index 1725c283396..baece830318 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -100,15 +100,15 @@ Note the following when promoting a secondary: #### Promoting a **secondary** site running on a single node running GitLab 14.5 and later -1. SSH in to your **secondary** node and execute: +1. SSH in to your **secondary** site and execute: - - To promote the secondary node to primary: + - To promote the secondary site to primary: ```shell sudo gitlab-ctl geo promote ``` - - To promote the secondary node to primary **without any further confirmation**: + - To promote the secondary site to primary **without any further confirmation**: ```shell sudo gitlab-ctl geo promote --force @@ -122,10 +122,10 @@ Note the following when promoting a secondary: WARNING: The `gitlab-ctl promote-to-primary-node` and `gitlab-ctl promoted-db` commands are -deprecated in GitLab 14.5 and later, and are scheduled to [be removed in GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/345207). +deprecated in GitLab 14.5 and later, and [removed in GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/345207). Use `gitlab-ctl geo promote` instead. -1. SSH in to your **secondary** node and login as root: +1. SSH in to your **secondary** site and login as root: ```shell sudo -i @@ -142,7 +142,7 @@ Use `gitlab-ctl geo promote` instead. 1. Promote the **secondary** site to the **primary** site: - - To promote the secondary node to primary along with [preflight checks](planned_failover.md#preflight-checks): + - To promote the secondary site to primary along with [preflight checks](planned_failover.md#preflight-checks): ```shell gitlab-ctl promote-to-primary-node @@ -163,7 +163,7 @@ Use `gitlab-ctl geo promote` instead. shows that it is complete, you can skip the preflight checks to make the command complete promotion. This bug was fixed in GitLab 13.8 and later. - - To promote the secondary node to primary **without any further confirmation**, + - To promote the secondary site to primary **without any further confirmation**, even when preflight checks fail: ```shell @@ -178,13 +178,13 @@ Use `gitlab-ctl geo promote` instead. 1. SSH to every Sidekiq, PostgresSQL, and Gitaly node in the **secondary** site and run one of the following commands: - - To promote the secondary node to primary: + - To promote the node on the secondary site to primary: ```shell sudo gitlab-ctl geo promote ``` - - To promote the secondary node to primary **without any further confirmation**: + - To promote the secondary site to primary **without any further confirmation**: ```shell sudo gitlab-ctl geo promote --force @@ -192,13 +192,13 @@ Use `gitlab-ctl geo promote` instead. 1. SSH into each Rails node on your **secondary** site and run one of the following commands: - - To promote the secondary node to primary: + - To promote the secondary site to primary: ```shell sudo gitlab-ctl geo promote ``` - - To promote the secondary node to primary **without any further confirmation**: + - To promote the secondary site to primary **without any further confirmation**: ```shell sudo gitlab-ctl geo promote --force @@ -212,7 +212,7 @@ Use `gitlab-ctl geo promote` instead. WARNING: The `gitlab-ctl promote-to-primary-node` and `gitlab-ctl promoted-db` commands are -deprecated in GitLab 14.5 and later, and are scheduled to [be removed in GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/345207). +deprecated in GitLab 14.5 and later, and [removed in GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/345207). Use `gitlab-ctl geo promote` instead. The `gitlab-ctl promote-to-primary-node` command cannot be used yet in @@ -256,13 +256,13 @@ do this manually. 1. SSH to every Sidekiq, PostgresSQL, and Gitaly node in the **secondary** site and run one of the following commands: - - To promote the secondary node to primary: + - To promote the secondary site to primary: ```shell sudo gitlab-ctl geo promote ``` - - To promote the secondary node to primary **without any further confirmation**: + - To promote the secondary site to primary **without any further confirmation**: ```shell sudo gitlab-ctl geo promote --force @@ -270,13 +270,13 @@ do this manually. 1. SSH into each Rails node on your **secondary** site and run one of the following commands: - - To promote the secondary node to primary: + - To promote the secondary site to primary: ```shell sudo gitlab-ctl geo promote ``` - - To promote the secondary node to primary **without any further confirmation**: + - To promote the secondary site to primary **without any further confirmation**: ```shell sudo gitlab-ctl geo promote --force @@ -290,7 +290,7 @@ do this manually. WARNING: The `gitlab-ctl promote-to-primary-node` and `gitlab-ctl promoted-db` commands are -deprecated in GitLab 14.5 and later, and are scheduled to [be removed in GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/345207). +deprecated in GitLab 14.5 and later, and [removed in GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/345207). Use `gitlab-ctl geo promote` instead. The `gitlab-ctl promote-to-primary-node` command cannot be used yet in @@ -347,7 +347,7 @@ site first. - [Azure PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/howto-read-replicas-portal#stop-replication) - [Google Cloud SQL](https://cloud.google.com/sql/docs/mysql/replication/manage-replicas#promote-replica) - For other external PostgreSQL databases, save the following script in your - secondary node, for example `/tmp/geo_promote.sh`, and modify the connection + secondary site, for example `/tmp/geo_promote.sh`, and modify the connection parameters to match your environment. Then, execute it to promote the replica: ```shell @@ -370,13 +370,13 @@ site first. 1. SSH to every Sidekiq, PostgresSQL, and Gitaly node in the **secondary** site and run one of the following commands: - - To promote the secondary node to primary: + - To promote the secondary site to primary: ```shell sudo gitlab-ctl geo promote ``` - - To promote the secondary node to primary **without any further confirmation**: + - To promote the secondary site to primary **without any further confirmation**: ```shell sudo gitlab-ctl geo promote --force @@ -384,13 +384,13 @@ site first. 1. SSH into each Rails node on your **secondary** site and run one of the following commands: - - To promote the secondary node to primary: + - To promote the secondary site to primary: ```shell sudo gitlab-ctl geo promote ``` - - To promote the secondary node to primary **without any further confirmation**: + - To promote the secondary site to primary **without any further confirmation**: ```shell sudo gitlab-ctl geo promote --force @@ -404,7 +404,7 @@ site first. WARNING: The `gitlab-ctl promote-to-primary-node` and `gitlab-ctl promoted-db` commands are -deprecated in GitLab 14.5 and later, and are scheduled to [be removed in GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/345207). +deprecated in GitLab 14.5 and later, and [removed in GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/345207). Use `gitlab-ctl geo promote` instead. The `gitlab-ctl promote-to-primary-node` command cannot be used in conjunction with @@ -418,7 +418,7 @@ required: - [Azure PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/howto-read-replicas-portal#stop-replication) - [Google Cloud SQL](https://cloud.google.com/sql/docs/mysql/replication/manage-replicas#promote-replica) - For other external PostgreSQL databases, save the following script in your - secondary node, for example `/tmp/geo_promote.sh`, and modify the connection + secondary site, for example `/tmp/geo_promote.sh`, and modify the connection parameters to match your environment. Then, execute it to promote the replica: ```shell @@ -503,7 +503,7 @@ secondary domain, like changing Git remotes and API URLs. in `/etc/gitlab/gitlab.rb`. 1. For GitLab 12.0 through 12.7, you may need to update the **primary** - node's name in the database. This bug has been fixed in GitLab 12.8. + site's name in the database. This bug has been fixed in GitLab 12.8. To determine if you need to do this, search for the `gitlab_rails["geo_node_name"]` setting in your `/etc/gitlab/gitlab.rb` @@ -571,9 +571,9 @@ and after that you also need two extra steps. # Allow PostgreSQL client authentication from the primary and secondary IPs. These IPs may be # public or VPC addresses in CIDR format, for example ['198.51.100.1/32', '198.51.100.2/32'] ## - postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32'] + postgresql['md5_auth_cidr_addresses'] = ['<primary_site_ip>/32', '<secondary_site_ip>/32'] - # Every secondary server needs to have its own slot so specify the number of secondary nodes you're going to have + # Every secondary site needs to have its own slot so specify the number of secondary sites you're going to have postgresql['max_replication_slots'] = 1 ## @@ -643,7 +643,7 @@ avoid a split-brain situation where writes can occur in two different GitLab instances, complicating recovery efforts. So to prepare for the failover, you must disable the **primary** site: -- If you have access to the **primary** Kubernetes cluster, connect to it and disable the GitLab webservice and Sidekiq pods: +- If you have access to the **primary** Kubernetes cluster, connect to it and disable the GitLab `webservice` and `Sidekiq` pods: ```shell kubectl --namespace gitlab scale deploy gitlab-geo-webservice-default --replicas=0 @@ -662,7 +662,7 @@ must disable the **primary** site: - Revoke object storage permissions from the **primary** site. - Physically disconnect a machine. -### Step 2. Promote all **secondary** sites external to the cluster +### Step 2. Promote all **secondary** site nodes external to the cluster WARNING: If the secondary site [has been paused](../../geo/index.md#pausing-and-resuming-replication), this performs @@ -673,13 +673,13 @@ If you are running GitLab 14.5 and later: 1. For each node outside of the **secondary** Kubernetes cluster using Omnibus such as PostgreSQL or Gitaly, SSH into the node and run one of the following commands: - - To promote the secondary node to primary: + - To promote the **secondary** site node external to the Kubernetes cluster to primary: ```shell sudo gitlab-ctl geo promote ``` - - To promote the secondary node to primary **without any further confirmation**: + - To promote the **secondary** site node external to the Kubernetes cluster to primary **without any further confirmation**: ```shell sudo gitlab-ctl geo promote --force @@ -699,7 +699,7 @@ If you are running GitLab 14.5 and later: If you are running GitLab 14.4 and earlier: -1. SSH in to the database node in the **secondary** and trigger PostgreSQL to +1. SSH in to the database node in the **secondary** site and trigger PostgreSQL to promote to read-write: ```shell diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md index 2a21bda23d7..57bad6177d9 100644 --- a/doc/administration/geo/disaster_recovery/planned_failover.md +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -12,10 +12,10 @@ the event of unplanned outage, but it can be used in conjunction with a planned failover to migrate your GitLab instance between regions without extended downtime. -As replication between Geo nodes is asynchronous, a planned failover requires -a maintenance window in which updates to the **primary** node are blocked. The +As replication between Geo sites is asynchronous, a planned failover requires +a maintenance window in which updates to the **primary** site are blocked. The length of this window is determined by your replication capacity - once the -**secondary** node is completely synchronized with the **primary** node, the failover can occur without +**secondary** site is completely synchronized with the **primary** site, the failover can occur without data loss. This document assumes you already have a fully configured, working Geo setup. @@ -28,7 +28,7 @@ have a high degree of confidence in being able to perform them accurately. ## Not all data is automatically replicated If you are using any GitLab features that Geo [doesn't support](../replication/datatypes.md#limitations-on-replicationverification), -you must make separate provisions to ensure that the **secondary** node has an +you must make separate provisions to ensure that the **secondary** site has an up-to-date copy of any data associated with that feature. This may extend the required scheduled maintenance period significantly. @@ -36,7 +36,7 @@ A common strategy for keeping this period as short as possible for data stored in files is to use `rsync` to transfer the data. An initial `rsync` can be performed ahead of the maintenance window; subsequent `rsync`s (including a final transfer inside the maintenance window) then transfers only the -*changes* between the **primary** node and the **secondary** nodes. +*changes* between the **primary** site and the **secondary** sites. Repository-centric strategies for using `rsync` effectively can be found in the [moving repositories](../../operations/moving_repositories.md) documentation; these strategies can @@ -98,31 +98,31 @@ Doing so reduces both the length of the maintenance window, and the risk of data loss as a result of a poorly executed planned failover. In GitLab 12.4, you can optionally allow GitLab to manage replication of Object Storage for -**secondary** nodes. For more information, see [Object Storage replication](../replication/object_storage.md). +**secondary** sites. For more information, see [Object Storage replication](../replication/object_storage.md). -### Review the configuration of each **secondary** node +### Review the configuration of each **secondary** site -Database settings are automatically replicated to the **secondary** node, but the +Database settings are automatically replicated to the **secondary** site, but the `/etc/gitlab/gitlab.rb` file must be set up manually, and differs between -nodes. If features such as Mattermost, OAuth or LDAP integration are enabled -on the **primary** node but not the **secondary** node, they are lost during failover. +sites. If features such as Mattermost, OAuth or LDAP integration are enabled +on the **primary** site but not the **secondary** site, they are lost during failover. -Review the `/etc/gitlab/gitlab.rb` file for both nodes and ensure the **secondary** node -supports everything the **primary** node does **before** scheduling a planned failover. +Review the `/etc/gitlab/gitlab.rb` file for both sites and ensure the **secondary** site +supports everything the **primary** site does **before** scheduling a planned failover. ### Run system checks -Run the following on both **primary** and **secondary** nodes: +Run the following on both **primary** and **secondary** sites: ```shell gitlab-rake gitlab:check gitlab-rake gitlab:geo:check ``` -If any failures are reported on either node, they should be resolved **before** +If any failures are reported on either site, they should be resolved **before** scheduling a planned failover. -### Check that secrets match between nodes +### Check that secrets and SSH host keys match between nodes The SSH host keys and `/etc/gitlab/gitlab-secrets.json` files should be identical on all nodes. Check this by running the following on all nodes and @@ -132,8 +132,14 @@ comparing the output: sudo sha256sum /etc/ssh/ssh_host* /etc/gitlab/gitlab-secrets.json ``` -If any files differ, replace the content on the **secondary** node with the -content from the **primary** node. +If any files differ, [manually replicate GitLab secrets](../replication/configuration.md#step-1-manually-replicate-secret-gitlab-values) and [replicate SSH host keys](../replication/configuration.md#step-2-manually-replicate-the-primary-sites-ssh-host-keys) +to the **secondary** site as necessary. + +### Check that the correct certificates are installed for HTTPS + +This step can be safely skipped if the **primary** site and all external sites accessed by the **primary** site use public CA-issued certificates. + +If the **primary** site uses custom or self-signed TLS certificates to secure inbound connections or if the **primary** site connects to external services that use custom or self-signed certificates, the correct certificates should also be installed on the **secondary** site. Follow instructions for [using custom certificates](../replication/configuration.md#step-4-optional-using-custom-certificates) with **secondary** sites. ### Ensure Geo replication is up-to-date @@ -141,13 +147,13 @@ The maintenance window won't end until Geo replication and verification is completely finished. To keep the window as short as possible, you should ensure these processes are close to 100% as possible during active use. -On the **secondary** node: +On the **secondary** site: 1. On the top bar, select **Menu > Admin**. -1. On the left sidebar, select **Geo > Nodes**. +1. On the left sidebar, select **Geo > Sites**. Replicated objects (shown in green) should be close to 100%, and there should be no failures (shown in red). If a large proportion of - objects aren't yet replicated (shown in gray), consider giving the node more + objects aren't yet replicated (shown in gray), consider giving the site more time to complete  @@ -160,7 +166,7 @@ You can use the [Geo status API](../../../api/geo_nodes.md#retrieve-project-sync the reasons for failure. A common cause of replication failures is the data being missing on the -**primary** node - you can resolve these failures by restoring the data from backup, +**primary** site - you can resolve these failures by restoring the data from backup, or removing references to the missing data. ### Verify the integrity of replicated data @@ -169,21 +175,21 @@ This [content was moved to another location](background_verification.md). ### Notify users of scheduled maintenance -On the **primary** node: +On the **primary** site: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Messages**. 1. Add a message notifying users on the maintenance window. - You can check under **Geo > Nodes** to estimate how long it + You can check under **Geo > Sites** to estimate how long it takes to finish syncing. 1. Select **Add broadcast message**. -## Prevent updates to the **primary** node +## Prevent updates to the **primary** site To ensure that all data is replicated to a secondary site, updates (write requests) need to be disabled on the **primary** site: -1. Enable [maintenance mode](../../maintenance_mode/index.md) on the **primary** node. +1. Enable [maintenance mode](../../maintenance_mode/index.md) on the **primary** site. 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dashboard, select **Cron**. @@ -199,22 +205,22 @@ GitLab 13.9 through GitLab 14.3 are affected by a bug in which the Geo secondary 1. If you are manually replicating any data not managed by Geo, trigger the final replication process now. -1. On the **primary** node: +1. On the **primary** site: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dashboard, select **Queues**, and wait for all queues except those with `geo` in the name to drop to 0. These queues contain work that has been submitted by your users; failing over before it is completed, causes the work to be lost. - 1. On the left sidebar, select **Geo > Nodes** and wait for the - following conditions to be true of the **secondary** node you are failing over to: + 1. On the left sidebar, select **Geo > Sites** and wait for the + following conditions to be true of the **secondary** site you are failing over to: - All replication meters reach 100% replicated, 0% failures. - All verification meters reach 100% verified, 0% failures. - Database replication lag is 0ms. - The Geo log cursor is up to date (0 events behind). -1. On the **secondary** node: +1. On the **secondary** site: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dashboard, select **Queues**, and wait for all the `geo` @@ -222,16 +228,16 @@ GitLab 13.9 through GitLab 14.3 are affected by a bug in which the Geo secondary 1. [Run an integrity check](../../raketasks/check.md) to verify the integrity of CI artifacts, LFS objects, and uploads in file storage. -At this point, your **secondary** node contains an up-to-date copy of everything the -**primary** node has, meaning nothing was lost when you fail over. +At this point, your **secondary** site contains an up-to-date copy of everything the +**primary** site has, meaning nothing was lost when you fail over. -## Promote the **secondary** node +## Promote the **secondary** site -After the replication is finished, [promote the **secondary** node to a **primary** node](index.md). This process causes a brief outage on the **secondary** node, and users may need to log in again. If you follow the steps correctly, the old primary Geo site should still be disabled and user traffic should go to the newly-promoted site instead. +After the replication is finished, [promote the **secondary** site to a **primary** site](index.md). This process causes a brief outage on the **secondary** site, and users may need to log in again. If you follow the steps correctly, the old primary Geo site should still be disabled and user traffic should go to the newly-promoted site instead. -When the promotion is completed, the maintenance window is over, and your new **primary** node now +When the promotion is completed, the maintenance window is over, and your new **primary** site now begins to diverge from the old one. If problems do arise at this point, failing -back to the old **primary** node [is possible](bring_primary_back.md), but likely to result +back to the old **primary** site [is possible](bring_primary_back.md), but likely to result in the loss of any data uploaded to the new **primary** in the meantime. Don't forget to remove the broadcast message after the failover is complete. diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md index 9b3e0ecb427..59134df8e7f 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md @@ -69,7 +69,7 @@ GitLab 13.9 through GitLab 14.3 are affected by a bug in which the Geo secondary On the **secondary** site: 1. On the top bar, select **Menu > Admin**. -1. On the left sidebar, select **Geo > Nodes** to see its status. +1. On the left sidebar, select **Geo > Sites** to see its status. Replicated objects (shown in green) should be close to 100%, and there should be no failures (shown in red). If a large proportion of objects aren't yet replicated (shown in gray), consider giving the site more @@ -84,7 +84,7 @@ failed to replicate is **lost**. You can use the [Geo status API](../../../../api/geo_nodes.md#retrieve-project-sync-or-verification-failures-that-occurred-on-the-current-node) to review failed objects and the reasons for failure. -A common cause of replication failures is the data being missing on the +A common cause of replication failures is data that is missing on the **primary** site - you can resolve these failures by restoring the data from backup, or removing references to the missing data. @@ -100,22 +100,22 @@ follow these steps to avoid unnecessary data loss: **primary**. Your **secondary** site still needs read-only access to the **primary** site during the maintenance window: - 1. At the scheduled time, using your cloud provider or your node's firewall, block - all HTTP, HTTPS and SSH traffic to/from the **primary** node, **except** for your IP and - the **secondary** node's IP. + 1. At the scheduled time, using your cloud provider or your site's firewall, block + all HTTP, HTTPS and SSH traffic to/from the **primary** site, **except** for your IP and + the **secondary** site's IP. - For instance, you can run the following commands on the **primary** node: + For instance, you can run the following commands on the **primary** site: ```shell - sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 22 -j ACCEPT + sudo iptables -A INPUT -p tcp -s <secondary_site_ip> --destination-port 22 -j ACCEPT sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 22 -j ACCEPT sudo iptables -A INPUT --destination-port 22 -j REJECT - sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 80 -j ACCEPT + sudo iptables -A INPUT -p tcp -s <secondary_site_ip> --destination-port 80 -j ACCEPT sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 80 -j ACCEPT sudo iptables -A INPUT --tcp-dport 80 -j REJECT - sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 443 -j ACCEPT + sudo iptables -A INPUT -p tcp -s <secondary_site_ip> --destination-port 443 -j ACCEPT sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 443 -j ACCEPT sudo iptables -A INPUT --tcp-dport 443 -j REJECT ``` @@ -157,8 +157,8 @@ follow these steps to avoid unnecessary data loss: those with `geo` in the name to drop to 0. These queues contain work that has been submitted by your users; failing over before it is completed, causes the work to be lost. - 1. On the left sidebar, select **Geo > Nodes** and wait for the - following conditions to be true of the **secondary** node you are failing over to: + 1. On the left sidebar, select **Geo > Sites** and wait for the + following conditions to be true of the **secondary** site you are failing over to: - All replication meters reach 100% replicated, 0% failures. - All verification meters reach 100% verified, 0% failures. @@ -230,13 +230,13 @@ follow these steps to avoid unnecessary data loss: 1. SSH to every Sidekiq, PostgresSQL, and Gitaly node in the **secondary** site and run one of the following commands: - - To promote the secondary node to primary: + - To promote the secondary site to primary: ```shell sudo gitlab-ctl geo promote ``` - - To promote the secondary node to primary **without any further confirmation**: + - To promote the secondary site to primary **without any further confirmation**: ```shell sudo gitlab-ctl geo promote --force @@ -244,13 +244,13 @@ follow these steps to avoid unnecessary data loss: 1. SSH into each Rails node on your **secondary** site and run one of the following commands: - - To promote the secondary node to primary: + - To promote the secondary site to primary: ```shell sudo gitlab-ctl geo promote ``` - - To promote the secondary node to primary **without any further confirmation**: + - To promote the secondary site to primary **without any further confirmation**: ```shell sudo gitlab-ctl geo promote --force @@ -264,7 +264,7 @@ follow these steps to avoid unnecessary data loss: WARNING: The `gitlab-ctl promote-to-primary-node` and `gitlab-ctl promoted-db` commands are -deprecated in GitLab 14.5 and later, and are scheduled to [be removed in GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/345207). +deprecated in GitLab 14.5 and later, and [removed in GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/345207). Use `gitlab-ctl geo promote` instead. NOTE: diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md index fa30b11fe32..25f6999ce43 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md @@ -85,22 +85,22 @@ follow these steps to avoid unnecessary data loss: **primary**. Your **secondary** site still needs read-only access to the **primary** site during the maintenance window: - 1. At the scheduled time, using your cloud provider or your node's firewall, block - all HTTP, HTTPS and SSH traffic to/from the **primary** node, **except** for your IP and - the **secondary** node's IP. + 1. At the scheduled time, using your cloud provider or your site's firewall, block + all HTTP, HTTPS and SSH traffic to/from the **primary** site, **except** for your IP and + the **secondary** site's IP. - For instance, you can run the following commands on the **primary** node: + For instance, you can run the following commands on the **primary** site: ```shell - sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 22 -j ACCEPT + sudo iptables -A INPUT -p tcp -s <secondary_site_ip> --destination-port 22 -j ACCEPT sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 22 -j ACCEPT sudo iptables -A INPUT --destination-port 22 -j REJECT - sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 80 -j ACCEPT + sudo iptables -A INPUT -p tcp -s <secondary_site_ip> --destination-port 80 -j ACCEPT sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 80 -j ACCEPT sudo iptables -A INPUT --tcp-dport 80 -j REJECT - sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 443 -j ACCEPT + sudo iptables -A INPUT -p tcp -s <secondary_site_ip> --destination-port 443 -j ACCEPT sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 443 -j ACCEPT sudo iptables -A INPUT --tcp-dport 443 -j REJECT ``` @@ -120,7 +120,7 @@ follow these steps to avoid unnecessary data loss: 1. On the **primary** site: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Monitoring > Background Jobs**. - 1. On the Sidekiq dhasboard, select **Cron**. + 1. On the Sidekiq dashboard, select **Cron**. 1. Select `Disable All` to disable any non-Geo periodic background jobs. 1. Select `Enable` for the `geo_sidekiq_cron_config_worker` cron job. This job re-enables several other cron jobs that are essential for planned @@ -142,7 +142,7 @@ follow these steps to avoid unnecessary data loss: those with `geo` in the name to drop to 0. These queues contain work that has been submitted by your users; failing over before it is completed, causes the work to be lost. - 1. On the left sidebar, select **Geo > Nodes** and wait for the + 1. On the left sidebar, select **Geo > Sites** and wait for the following conditions to be true of the **secondary** site you are failing over to: - All replication meters reach 100% replicated, 0% failures. @@ -224,15 +224,15 @@ Note the following when promoting a secondary: To promote the secondary site running GitLab 14.5 and later: -1. SSH in to your **secondary** node and run one of the following commands: +1. SSH in to your **secondary** site and run one of the following commands: - - To promote the secondary node to primary: + - To promote the secondary site to primary: ```shell sudo gitlab-ctl geo promote ``` - - To promote the secondary node to primary **without any further confirmation**: + - To promote the secondary site to primary **without any further confirmation**: ```shell sudo gitlab-ctl geo promote --force @@ -247,7 +247,7 @@ To promote the secondary site running GitLab 14.4 and earlier: WARNING: The `gitlab-ctl promote-to-primary-node` and `gitlab-ctl promoted-db` commands are -deprecated in GitLab 14.5 and later, and are scheduled to [be removed in GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/345207). +deprecated in GitLab 14.5 and later, and [removed in GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/345207). Use `gitlab-ctl geo promote` instead. 1. SSH in to your **secondary** site and login as root: diff --git a/doc/administration/geo/glossary.md b/doc/administration/geo/glossary.md index a18e78a5e01..c6b3f26dc67 100644 --- a/doc/administration/geo/glossary.md +++ b/doc/administration/geo/glossary.md @@ -26,9 +26,9 @@ these definitions yet. | Single-node site | A specific configuration of GitLab that uses exactly one node. | GitLab | single-server, single-instance | Multi-node site | A specific configuration of GitLab that uses more than one node. | GitLab | multi-server, multi-instance, high availability | | Primary site | A GitLab site whose data is being replicated by at least one secondary site. There can only be a single primary site. | Geo-specific | Geo deployment, Primary node | -| Secondary site(s) | A GitLab site that is configured to replicate the data of a primary site. There can be one or more secondary sites. | Geo-specific | Geo deployment, Secondary node | +| Secondary site | A GitLab site that is configured to replicate the data of a primary site. There can be one or more secondary sites. | Geo-specific | Geo deployment, Secondary node | | Geo deployment | A collection of two or more GitLab sites with exactly one primary site being replicated by one or more secondary sites. | Geo-specific | | -| Reference architecture(s) | A [specified configuration of GitLab for a number of users](../reference_architectures/index.md), possibly including multiple nodes and multiple sites. | GitLab | | +| Reference architecture | A [specified configuration of GitLab for a number of users](../reference_architectures/index.md), possibly including multiple nodes and multiple sites. | GitLab | | | Promoting | Changing the role of a site from secondary to primary. | Geo-specific | | | Demoting | Changing the role of a site from primary to secondary. | Geo-specific | | | Failover | The entire process that shifts users from a primary Site to a secondary site. This includes promoting a secondary, but contains other parts as well. For example, scheduling maintenance. | Geo-specific | | diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md index 1b80e91c9c4..dc25be825ea 100644 --- a/doc/administration/geo/index.md +++ b/doc/administration/geo/index.md @@ -254,7 +254,7 @@ Omnibus GitLab-managed database. External databases are not supported. In some circumstances, like during [upgrades](replication/updating_the_geo_sites.md) or a [planned failover](disaster_recovery/planned_failover.md), it is desirable to pause replication between the primary and secondary. -Pausing and resuming replication is done via a command line tool from the a node in the secondary site where the `postgresql` service is enabled. +Pausing and resuming replication is done via a command line tool from the node in the secondary site where the `postgresql` service is enabled. If `postgresql` is on a standalone database node, ensure that `gitlab.rb` on that node contains the configuration line `gitlab_rails['geo_node_name'] = 'node_name'`, where `node_name` is the same as the `geo_name_name` on the application node. @@ -288,7 +288,7 @@ For more information on how to replicate the Container Registry, see [Docker Reg ### Geo secondary proxy -For more information on using Geo proxying on secondary nodes, see [Geo proxying for secondary sites](secondary_proxy/index.md). +For more information on using Geo proxying on secondary sites, see [Geo proxying for secondary sites](secondary_proxy/index.md). ### Security Review diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md index 16b4848e6d3..adf89c24a4e 100644 --- a/doc/administration/geo/replication/configuration.md +++ b/doc/administration/geo/replication/configuration.md @@ -129,7 +129,7 @@ keys must be manually replicated to the **secondary** site. ```shell chown root:root /etc/ssh/ssh_host_*_key* - chmod 0600 /etc/ssh/ssh_host_*_key* + chmod 0600 /etc/ssh/ssh_host_*_key ``` 1. To verify key fingerprint matches, execute the following command on both primary and secondary nodes on each site: @@ -241,15 +241,60 @@ that the **secondary** site can act on those notifications immediately. Be sure the _secondary_ site is running and accessible. You can sign in to the _secondary_ site with the same credentials as were used with the _primary_ site. -### Step 4. (Optional) Configuring the **secondary** site to trust the **primary** site +### Step 4. (Optional) Using custom certificates -You can safely skip this step if your **primary** site uses a CA-issued HTTPS certificate. +You can safely skip this step if: -If your **primary** site is using a self-signed certificate for *HTTPS* support, you -need to add that certificate to the **secondary** site's trust store. Retrieve the -certificate from the **primary** site and follow -[these instructions](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates) -on the **secondary** site. +- Your **primary** site uses a public CA-issued HTTPS certificate. +- Your **primary** site only connects to external services with CA-issued (not self-signed) HTTPS certificates. + +#### Custom or self-signed certificate for inbound connections + +If your GitLab Geo **primary** site uses a custom or [self-signed certificate to secure inbound HTTPS connections](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates), this certificate can either be single-domain certificate or multi-domain. + +Install the correct certificate based on your certificate type: + +- **Multi-domain certificate** that includes both primary and secondary site domains: Install the certificate at `/etc/gitlab/ssl` on all **Rails, Sidekiq, and Gitaly** nodes in the **secondary** site. +- **Single-domain certificate** where the certificates are specific to each Geo site domain: Generate a valid certificate for your **secondary** site's domain and install it at `/etc/gitlab/ssl` per [these instructions](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates) on all **Rails, Sidekiq, and Gitaly** nodes in the **secondary** site. + +#### Connecting to external services that use customer certificates + +A copy of the self-signed certificate for the external service needs to be added to the trust store on all the **primary** site's nodes that require access to the service. + +For the **secondary** site to be able to access the same external services, these certificates *must* be added to the **secondary** site's trust store. + +If your **primary** site is using a [custom or self-signed certificate for inbound HTTPS connections](#custom-or-self-signed-certificate-for-inbound-connections), the **primary** site's certificate needs to be added to the **secondary** site's trust store: + +1. SSH into each **Rails, Sidekiq, and Gitaly node on your secondary** site and login as root: + + ```shell + sudo -i + ``` + +1. Copy the trusted certs from the **primary** site: + + If you can access one of the nodes on your **primary** site serving SSH traffic using the root user: + + ```shell + scp root@<primary_site_node_fqdn>:/etc/gitlab/trusted-certs/* /etc/gitlab/trusted-certs + ``` + + If you only have access through a user with sudo privileges: + + ```shell + # Run this from the node on your primary site: + sudo tar --transform 's/.*\///g' -zcvf ~/geo-trusted-certs.tar.gz /etc/gitlab/trusted-certs/* + + # Run this on each node on your secondary site: + scp <user_with_sudo>@<primary_site_node_fqdn>:geo-trusted-certs.tar.gz . + tar zxvf ~/geo-trusted-certs.tar.gz -C /etc/gitlab/trusted-certs + ``` + +1. Reconfigure each updated **Rails, Sidekiq, and Gitaly node in your secondary** site: + + ```shell + sudo gitlab-ctl reconfigure + ``` ### Step 5. Enable Git access over HTTP/HTTPS diff --git a/doc/administration/geo/replication/multiple_servers.md b/doc/administration/geo/replication/multiple_servers.md index 87b1aa7fc44..7b800817461 100644 --- a/doc/administration/geo/replication/multiple_servers.md +++ b/doc/administration/geo/replication/multiple_servers.md @@ -119,7 +119,7 @@ NOTE: [NFS](../../nfs.md) can be used in place of Gitaly but is not recommended. -### Step 2: Configure Postgres streaming replication +### Step 2: Configure PostgreSQL streaming replication Follow the [Geo database replication instructions](../setup/database.md). @@ -261,7 +261,7 @@ nodes connect to the databases. NOTE: Make sure that current node's IP is listed in `postgresql['md5_auth_cidr_addresses']` setting of the read-replica database to -allow Rails on this node to connect to Postgres. +allow Rails on this node to connect to PostgreSQL. After making these changes [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect. diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index 871d6041066..5a29c5a3c54 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -419,6 +419,21 @@ sudo gitlab-ctl reconfigure To help us resolve this problem, consider commenting on [the issue](https://gitlab.com/gitlab-org/gitlab/-/issues/4489). +### Message: `FATAL: could not connect to the primary server: server certificate for "PostgreSQL" does not match host name` + +This happens because the PostgreSQL certificate that the Omnibus GitLab package automatically creates contains +the Common Name `PostgreSQL`, but the replication is connecting to a different host and GitLab attempts to use +the `verify-full` SSL mode by default. + +In order to fix this, you can either: + +- Use the `--sslmode=verify-ca` argument with the `replicate-geo-database` command. +- For an already replicated database, change `sslmode=verify-full` to `sslmode=verify-ca` + in `/var/opt/gitlab/postgresql/data/gitlab-geo.conf` and run `gitlab-ctl restart postgresql`. +- [Configure SSL for PostgreSQL](https://docs.gitlab.com/omnibus/settings/database.html#configuring-ssl) + with a custom certificate (including the host name that's used to connect to the database in the CN or SAN) + instead of using the automatically generated certificate. + ### Message: `LOG: invalid CIDR mask in address` This happens on wrongly-formatted addresses in `postgresql['md5_auth_cidr_addresses']`. @@ -637,9 +652,9 @@ to start again from scratch, there are a few steps that can help you: 1. Reset the Tracking Database. ```shell - gitlab-rake geo:db:drop # on a secondary app node - gitlab-ctl reconfigure # on the tracking database node - gitlab-rake geo:db:setup # on a secondary app node + gitlab-rake db:drop:geo # on a secondary app node + gitlab-ctl reconfigure # on the tracking database node + gitlab-rake db:migrate:geo # on a secondary app node ``` 1. Restart previously stopped services. @@ -977,7 +992,7 @@ On the **primary** node: 1. On the left sidebar, select **Geo > Nodes**. 1. Find the affected **secondary** site and select **Edit**. 1. Ensure the **URL** field matches the value found in `/etc/gitlab/gitlab.rb` - in `external_url "https://gitlab.example.com"` on the frontend server(s) of + in `external_url "https://gitlab.example.com"` on the frontend servers of the **secondary** node. ## Fixing common errors @@ -1042,7 +1057,7 @@ Make sure you follow the [Geo database replication](../setup/database.md) instru If you are using Omnibus GitLab installation, something might have failed during upgrade. You can: - Run `sudo gitlab-ctl reconfigure`. -- Manually trigger the database migration by running: `sudo gitlab-rake geo:db:migrate` as root on the **secondary** node. +- Manually trigger the database migration by running: `sudo gitlab-rake db:migrate:geo` as root on the **secondary** node. ### GitLab indicates that more than 100% of repositories were synced @@ -1101,12 +1116,70 @@ This is due to [Pages data not being managed by Geo](datatypes.md#limitations-on Find advice to resolve those error messages in the [Pages administration documentation](../../../administration/pages/index.md#404-error-after-promoting-a-geo-secondary-to-a-primary-node). +### Primary site returns 500 error when accessing `/admin/geo/replication/projects` + +Navigating to **Admin > Geo > Replication** (or `/admin/geo/replication/projects`) on a primary Geo site, shows a 500 error, while that same link on the secondary works fine. The primary's `production.log` has a similar entry to the following: + +```plaintext +Geo::TrackingBase::SecondaryNotConfigured: Geo secondary database is not configured + from ee/app/models/geo/tracking_base.rb:26:in `connection' + [..] + from ee/app/views/admin/geo/projects/_all.html.haml:1 +``` + +On a Geo primary site this error can be ignored. + +This happens because GitLab is attempting to display registries from the [Geo tracking database](../../../administration/geo/#geo-tracking-database) which doesn't exist on the primary site (only the original projects exist on the primary; no replicated projects are present, therefore no tracking database exists). + ## Fixing client errors -### Authorization errors from LFS HTTP(s) client requests +### Authorization errors from LFS HTTP(S) client requests You may have problems if you're running a version of [Git LFS](https://git-lfs.github.com/) before 2.4.2. As noted in [this authentication issue](https://github.com/git-lfs/git-lfs/issues/3025), requests redirected from the secondary to the primary node do not properly send the Authorization header. This may result in either an infinite `Authorization <-> Redirect` loop, or Authorization error messages. + +## Recovering from a partial failover + +The partial failover to a secondary Geo *site* may be the result of a temporary/transient issue. Therefore, first attempt to run the promote command again. + +1. SSH into every Sidekiq, PostgresSQL, Gitaly, and Rails node in the **secondary** site and run one of the following commands: + + - To promote the secondary node to primary: + + ```shell + sudo gitlab-ctl geo promote + ``` + + - To promote the secondary node to primary **without any further confirmation**: + + ```shell + sudo gitlab-ctl geo promote --force + ``` + +1. Verify you can connect to the newly-promoted **primary** site using the URL used previously for the **secondary** site. +1. If **successful**, the **secondary** site is now promoted to the **primary** site. + +If the above steps are **not successful**, proceed through the next steps: + +1. SSH to every Sidekiq, PostgresSQL, Gitaly and Rails node in the **secondary** site and perform the following operations: + + - Create a `/etc/gitlab/gitlab-cluster.json` file with the following content: + + ```shell + { + "primary": true, + "secondary": false + } + ``` + + - Reconfigure GitLab for the changes to take effect: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +1. Verify you can connect to the newly-promoted **primary** site using the URL used previously for the **secondary** site. +1. If successful, the **secondary** site is now promoted to the **primary** site. diff --git a/doc/administration/geo/replication/version_specific_updates.md b/doc/administration/geo/replication/version_specific_updates.md index b0797445890..6b617a21be8 100644 --- a/doc/administration/geo/replication/version_specific_updates.md +++ b/doc/administration/geo/replication/version_specific_updates.md @@ -12,9 +12,9 @@ for updating Geo sites. ## Updating to 14.9 -**DO NOT** update to GitLab 14.9.0. +**DO NOT** update to GitLab 14.9.0. Instead, use 14.9.1 or later. -We've discovered an issue with Geo's CI verification feature that may [cause job traces to be lost](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6664). This issue will be fixed in the next patch release. +We've discovered an issue with Geo's CI verification feature that may [cause job traces to be lost](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6664). This issue was fixed in [the GitLab 14.9.1 patch release](https://about.gitlab.com/releases/2022/03/23/gitlab-14-9-1-released/). If you have already updated to GitLab 14.9.0, you can disable the feature causing the issue by [disabling the `geo_job_artifact_replication` feature flag](../../feature_flags.md#how-to-enable-and-disable-features-behind-flags). diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md index 9c917be123e..52f4adc4e03 100644 --- a/doc/administration/geo/setup/database.md +++ b/doc/administration/geo/setup/database.md @@ -300,6 +300,24 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o need it when setting up the **secondary** node! The certificate is not sensitive data. + However, this certificate is created with a generic `PostgreSQL` Common Name. For this, + you must use the `verify-ca` mode when replicating the database, otherwise, + the hostname mismatch will cause errors. + +1. Optional. Generate your own SSL certificate and manually + [configure SSL for PostgreSQL](https://docs.gitlab.com/omnibus/settings/database.html#configuring-ssl), + instead of using the generated certificate. + + You will need at least the SSL certificate and key, and set the `postgresql['ssl_cert_file']` and + `postgresql['ssl_key_file']` values to their full paths, as per the Database SSL docs. + + This allows you to use the `verify-full` SSL mode when replicating the database + and get the extra benefit of verifying the full hostname in the CN. + + You can use this certificate (that you have also set in `postgresql['ssl_cert_file']`) instead + of the certificate from the point above going forward. This will allow you to use `verify-full` + without replication errors if the CN matches. + #### Step 2. Configure the **secondary** server 1. SSH into your GitLab **secondary** server and login as root: @@ -367,7 +385,13 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o -h <primary_node_ip> ``` - When prompted enter the password you set in the first step for the + NOTE: + If you are using manually generated certificates and plan on using + `sslmode=verify-full` to benefit of the full hostname verification, + make sure to replace `verify-ca` to `verify-full` when + running the command. + + When prompted enter the _plaintext_ password you set in the first step for the `gitlab_replicator` user. If all worked correctly, you should see the list of **primary** node's databases. @@ -455,6 +479,7 @@ data before running `pg_basebackup`. gitlab-ctl replicate-geo-database \ --slot-name=<secondary_node_name> \ --host=<primary_node_ip> + --sslmode=verify-ca ``` NOTE: @@ -463,6 +488,13 @@ data before running `pg_basebackup`. When prompted, enter the _plaintext_ password you set up for the `gitlab_replicator` user in the first step. + NOTE: + If you have generated custom PostgreSQL certificates, you will want to use + `--sslmode=verify-full` (or omit the `sslmode` line entirely), to benefit from the extra + validation of the full host name in the certificate CN / SAN for additional security. + Otherwise, using the automatically created certificate with `verify-full` will fail, + as it has a generic `PostgreSQL` CN which will not match the `--host` value in this command. + This command also takes a number of additional options. You can use `--help` to list them all, but here are a couple of tips: @@ -1061,7 +1093,7 @@ For each node running the `gitlab-rails`, `sidekiq`, and `geo-logcursor` service 1. Run the tracking database migrations: ```shell - gitlab-rake geo:db:migrate + gitlab-rake db:migrate:geo ``` ### Migrating a single tracking database node to Patroni diff --git a/doc/administration/geo/setup/external_database.md b/doc/administration/geo/setup/external_database.md index cd77647e7dc..58fd6b28797 100644 --- a/doc/administration/geo/setup/external_database.md +++ b/doc/administration/geo/setup/external_database.md @@ -245,11 +245,11 @@ the tracking database on port 5432. 1. The reconfigure should automatically create the database. If needed, you can perform this task manually. This task (whether run by itself or during reconfigure) requires the database user to be a superuser. ```shell - gitlab-rake geo:db:create + gitlab-rake db:create:geo ``` 1. The reconfigure should automatically migrate the database. You can migrate the database manually if needed, for example if `geo_secondary['auto_migrate'] = false`: ```shell - gitlab-rake geo:db:migrate + gitlab-rake db:migrate:geo ``` diff --git a/doc/administration/git_protocol.md b/doc/administration/git_protocol.md index 29156d9b3e1..3612487f456 100644 --- a/doc/administration/git_protocol.md +++ b/doc/administration/git_protocol.md @@ -111,4 +111,4 @@ URL to use SSH. ### Observe Git protocol version of connections For information on observing the Git protocol versions are being used in a production environment, -see the [relevant documentation](gitaly/index.md#useful-queries). +see the [relevant documentation](gitaly/monitoring.md#useful-queries). diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md index 0fb285c50d6..6888a2abe9a 100644 --- a/doc/administration/gitaly/configure_gitaly.md +++ b/doc/administration/gitaly/configure_gitaly.md @@ -2,7 +2,6 @@ stage: Create group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference --- # Configure Gitaly **(FREE SELF)** @@ -217,7 +216,12 @@ disable enforcement. For more information, see the documentation on configuring 1. Edit `/etc/gitlab/gitlab.rb`: - <!-- Updates to following example must also be made at https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab --> +<!-- +Updates to example must be made at: +- https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab +- https://gitlab.com/gitlab-org/gitlab/blob/master/doc/administration/gitaly/index.md#gitaly-server-configuration +- all reference architecture pages +--> ```ruby # Avoid running unnecessary services on the Gitaly server @@ -229,10 +233,11 @@ disable enforcement. For more information, see the documentation on configuring gitlab_workhorse['enable'] = false grafana['enable'] = false gitlab_exporter['enable'] = false + gitlab_kas['enable'] = false # If you run a separate monitoring node you can disable these services - alertmanager['enable'] = false prometheus['enable'] = false + alertmanager['enable'] = false # If you don't run a separate monitoring node you can # enable Prometheus access & disable these extra services. @@ -252,14 +257,14 @@ disable enforcement. For more information, see the documentation on configuring # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from Gitaly client to Gitaly server. gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' - # Authentication token to ensure only authorized servers can communicate with - # Gitaly server - gitaly['auth_token'] = 'AUTH_TOKEN' - # Make Gitaly accept connections on all network interfaces. You must use # firewalls to restrict access to this address/port. # Comment out following line if you only want to support TLS connections gitaly['listen_addr'] = "0.0.0.0:8075" + + # Authentication token to ensure only authorized servers can communicate with + # Gitaly server + gitaly['auth_token'] = 'AUTH_TOKEN' ``` 1. Append the following to `/etc/gitlab/gitlab.rb` for each respective Gitaly server: @@ -710,7 +715,7 @@ To configure Gitaly with TLS: ### Observe type of Gitaly connections For information on observing the type of Gitaly connections being served, see the -[relevant documentation](index.md#useful-queries). +[relevant documentation](monitoring.md#useful-queries). ## `gitaly-ruby` @@ -773,8 +778,8 @@ settings: Clone traffic can put a large strain on your Gitaly service. The bulk of the work gets done in the either of the following RPCs: -- `SSHUploadPack` (for Git SSH). -- `PostUploadPack` (for Git HTTP). +- `SSHUploadPackWithSidechannel` (for Git SSH). +- `PostUploadPackWithSidechannel` (for Git HTTP). To prevent such workloads from overwhelming your Gitaly server, you can set concurrency limits in Gitaly's configuration file. For example: @@ -784,26 +789,103 @@ Gitaly's configuration file. For example: gitaly['concurrency'] = [ { - 'rpc' => "/gitaly.SmartHTTPService/PostUploadPack", - 'max_per_repo' => 20 + 'rpc' => "/gitaly.SmartHTTPService/PostUploadPackWithSidechanel", + 'max_per_repo' => 20, + 'max_queue_time' => "1s", + 'max_queue_size' => 10 }, { - 'rpc' => "/gitaly.SSHService/SSHUploadPack", + 'rpc' => "/gitaly.SSHService/SSHUploadPackWithSidechannel", 'max_per_repo' => 20 + 'max_queue_time' => "1s", + 'max_queue_size' => 10 } ] ``` +- `rpc` is the name of the RPC to set a concurrency limit for per repository. +- `max_per_repo` is the maximum number of in-flight RPC calls for the given RPC per repository. +- `max_queue_time` is the maximum amount of time a request can wait in the concurrency queue to + be picked up by Gitaly. +- `max_queue_size` is the maximum size the concurrency queue can grow to before requests are rejected by + Gitaly. + This limits the number of in-flight RPC calls for the given RPCs. The limit is applied per repository. In the example above: -- Each repository served by the Gitaly server can have at most 20 simultaneous `PostUploadPack` RPC - calls in flight, and the same for `SSHUploadPack`. +- Each repository served by the Gitaly server can have at most 20 simultaneous `PostUploadPackWithSidechannel` and + `SSHUploadPackWithSidechannel` RPC calls in flight. - If another request comes in for a repository that has used up its 20 slots, that request gets queued. +- If a request waits in the queue for more than 1 second, it is rejected with an error. +- If the queue grows beyond 10, subsequent requests are rejected with an error. You can observe the behavior of this queue using the Gitaly logs and Prometheus. For more -information, see the [relevant documentation](index.md#monitor-gitaly). +information, see the [relevant documentation](monitoring.md#monitor-gitaly-concurrency-limiting). + +## Control groups + +> - Introduced in GitLab 13.10. + +Gitaly shells out to Git for many of its operations. Git can consume a lot of resources for certain operations, +especially for large repositories. + +Control groups (cgroups) in Linux allow limits to be imposed on how much memory and CPU can be consumed. +See the [`cgroups` Linux man page](https://man7.org/linux/man-pages/man7/cgroups.7.html) for more information. +cgroups can be useful for protecting the system against resource exhaustion because of overconsumption of memory and CPU. + +Gitaly has built-in cgroups control. When configured, Gitaly assigns Git +processes to a cgroup based on the repository the Git command is operating in. +Each cgroup has a memory and CPU limit. When a cgroup reaches its: + +- Memory limit, the kernel looks through the processes for a candidate to kill. +- CPU limit, processes are not killed, but the processes are prevented from consuming more CPU than allowed. + +The main reason to configure cgroups for your GitLab installation is that it +protects against system resource starvation due to a few large repositories or +bad actors. + +Some Git operations are expensive by nature. `git clone`, for instance, +spawns a `git-upload-pack` process on the server that can consume a lot of memory +for large repositories. For example, a client that keeps on cloning a +large repository over and over again. This situation could potentially use up all of the +memory on a server, causing other operations to fail for other users. + +There are many ways someone can create a repository that can consume large amounts of memory when cloned or downloaded. +Using cgroups allows the kernel to kill these operations before they hog up all system resources. + +### Configure cgroups in Gitaly + +To configure cgroups in Gitaly, add `gitaly['cgroups']` to `/etc/gitlab/gitlab.rb`. For +example: + +```ruby +# in /etc/gitlab/gitlab.rb +gitaly['cgroups_count'] = 1000 +gitaly['cgroups_mountpoint'] = "/sys/fs/cgroup" +gitaly['cgroups_hierarchy_root'] = "gitaly" +gitaly['cgroups_memory_limit'] = 32212254720 +gitaly['cgroups_memory_enabled'] = true +gitaly['cgroups_cpu_shares'] = 1024 +gitaly['cgroups_cpu_enabled'] = true + +``` + +- `cgroups_count` is the number of cgroups created. Each time a new + command is spawned, Gitaly assigns it to one of these cgroups based + on the command line arguments of the command. A circular hashing algorithm assigns + commands to these cgroups. +- `cgroups_mountpoint` is where the parent cgroup directory is mounted. Defaults to `/sys/fs/cgroup`. +- `cgroups_hierarchy_root` is the parent cgroup under which Gitaly creates groups, and + is expected to be owned by the user and group Gitaly runs as. Omnibus GitLab + creates the set of directories `mountpoint/<cpu|memory>/hierarchy_root` + when Gitaly starts. +- `cgroups_memory_enabled` enables or disables the memory limit on cgroups. +- `cgroups_memory_bytes` is the total memory limit each cgroup imposes on the processes added to it. +- `cgroups_cpu_enabled` enables or disables the CPU limit on cgroups. +- `cgroups_cpu_shares` is the CPU limit each cgroup imposes on the processes added to it. The maximum is 1024 shares, + which represents 100% of CPU. + which represents 100% of CPU. ## Background Repository Optimization @@ -858,7 +940,7 @@ server" and "Gitaly client" refers to the same machine. ### Verify authentication monitoring Before rotating a Gitaly authentication token, verify that you can -[monitor the authentication behavior](index.md#useful-queries) of your GitLab installation using +[monitor the authentication behavior](monitoring.md#useful-queries) of your GitLab installation using Prometheus. You can then continue the rest of the procedure. @@ -1068,7 +1150,7 @@ closed it. ### Observe the cache -The cache can be observed [using metrics](index.md#monitor-gitaly) and in the following logged +The cache can be observed [using metrics](monitoring.md#pack-objects-cache) and in the following logged information: |Message|Fields|Description| diff --git a/doc/administration/gitaly/faq.md b/doc/administration/gitaly/faq.md index b0a88e8adc9..a5c2c7d1469 100644 --- a/doc/administration/gitaly/faq.md +++ b/doc/administration/gitaly/faq.md @@ -2,7 +2,6 @@ stage: Create group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference --- # Frequently asked questions **(FREE SELF)** diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index 8e4464bba43..f43f9f5d6c1 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -2,7 +2,6 @@ stage: Create group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference --- # Gitaly and Gitaly Cluster **(FREE SELF)** @@ -57,7 +56,7 @@ If you have: - Not yet migrated to Gitaly Cluster and want to continue using NFS, remain on the service you are using. NFS is supported in 14.x releases but is [deprecated](../../update/deprecations.md#nfs-for-git-repository-storage). -Support for storing Git repository data on NFS will end for all versions of GitLab with the release of 15.0. + Support for storing Git repository data on NFS is scheduled to end for all versions of GitLab on November 22, 2022. - Not yet migrated to Gitaly Cluster but want to migrate away from NFS, you have two options: - A sharded Gitaly instance. - Gitaly Cluster. @@ -72,7 +71,7 @@ the current status of these issues, please refer to the referenced issues and ep | Issue | Summary | How to avoid | |:--------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------| | Gitaly Cluster + Geo - Issues retrying failed syncs | If Gitaly Cluster is used on a Geo secondary site, repositories that have failed to sync could continue to fail when Geo tries to resync them. Recovering from this state requires assistance from support to run manual steps. Work is in-progress to update Gitaly Cluster to [identify repositories with a unique and persistent identifier](https://gitlab.com/gitlab-org/gitaly/-/issues/3485), which is expected to resolve the issue. | No known solution at this time. | -| Praefect unable to insert data into the database due to migrations not being applied after an upgrade | If the database is not kept up to date with completed migrations, then the Praefect node is unable to perform normal operation. | Make sure the Praefect database is up and running with all migrations completed (For example: `/opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-migrate-status` should show a list of all applied migrations). Consider [requesting live upgrade assistance](https://about.gitlab.com/support/scheduling-live-upgrade-assistance.html) so your upgrade plan can be reviewed by support. | +| Praefect unable to insert data into the database due to migrations not being applied after an upgrade | If the database is not kept up to date with completed migrations, then the Praefect node is unable to perform normal operation. | Make sure the Praefect database is up and running with all migrations completed (For example: `/opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-migrate-status` should show a list of all applied migrations). Consider [requesting live upgrade assistance](https://about.gitlab.com/support/scheduling-upgrade-assistance.html) so your upgrade plan can be reviewed by support. | | Restoring a Gitaly Cluster node from a snapshot in a running cluster | Because the Gitaly Cluster runs with consistent state, introducing a single node that is behind will result in the cluster not being able to reconcile the nodes data and other nodes data | Don't restore a single Gitaly Cluster node from a backup snapshot. If you must restore from backup, it's best to snapshot all Gitaly Cluster nodes at the same time and take a database dump of the Praefect database. | ### Snapshot backup and recovery limitations @@ -233,9 +232,142 @@ variable replication factor is tracked in [this issue](https://gitlab.com/groups As with normal Gitaly storages, virtual storages can be sharded. +### Storage layout + +WARNING: +The storage layout is an internal detail of Gitaly Cluster and is not guaranteed to remain stable between releases. +The information here is only for informational purposes and to help with debugging. Performing changes in the +repositories directly on the disk is not supported and may lead to breakage or the changes being overwritten. + +Gitaly Cluster's virtual storages provide an abstraction that looks like a single storage but actually consists of +multiple physical storages. Gitaly Cluster has to replicate each operation to each physical storage. Operations +may succeed on some of the physical storages but fail on others. + +Partially applied operations can cause problems with other operations and leave the system in a state it can't recover from. +To avoid these types of problems, each operation should either fully apply or not apply at all. This property of operations is called +[atomicity](https://en.wikipedia.org/wiki/Atomicity_(database_systems)). + +GitLab controls the storage layout on the repository storages. GitLab instructs the repository storage where to create, +delete, and move repositories. These operations create atomicity issues when they are being applied to multiple physical storages. +For example: + +- GitLab deletes a repository while one of its replicas is unavailable. +- GitLab later recreates the repository. + +As a result, the stale replica that was unavailable at the time of deletion may cause conflicts and prevent +recreation of the repository. + +These atomicity issues have caused multiple problems in the past with: + +- Geo syncing to a secondary site with Gitaly Cluster. +- Backup restoration. +- Repository moves between repository storages. + +Gitaly Cluster provides atomicity for these operations by storing repositories on the disk in a special layout that prevents +conflicts that could occur due to partially applied operations. + +#### Client-generated replica paths + +Repositories are stored in the storages at the relative path determined by the [Gitaly client](#gitaly-architecture). These paths can be +identified by them not beginning with the `@cluster` prefix. The relative paths +follow the [hashed storage](../repository_storage_types.md#hashed-storage) schema. + +#### Praefect-generated replica paths (GitLab 15.0 and later) + +> Introduced in GitLab 15.0 behind [a feature flag](https://gitlab.com/gitlab-org/gitaly/-/issues/4218) named `gitaly_praefect_generated_replica_paths`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../feature_flags.md) +named `gitaly_praefect_generated_replica_paths`. On GitLab.com, this feature is available but can be configured by GitLab.com administrators only. The feature is not ready for production use. + +When Gitaly Cluster creates a repository, it assigns the repository a unique and permanent ID called the _repository ID_. The repository ID is +internal to Gitaly Cluster and doesn't relate to any IDs elsewhere in GitLab. If a repository is removed from Gitaly Cluster and later moved +back, the repository is assigned a new repository ID and is a different repository from Gitaly Cluster's perspective. The sequence of repository IDs +always increases, but there may be gaps in the sequence. + +The repository ID is used to derive a unique storage path called _replica path_ for each repository on the cluster. The replicas of +a repository are all stored at the same replica path on the storages. The replica path is distinct from the _relative path_: + +- The relative path is a name the Gitaly client uses to identify a repository, together with its virtual storage, that is unique to them. +- The replica path is the actual physical path in the physical storages. + +Praefect translates the repositories in the RPCs from the virtual `(virtual storage, relative path)` identifier into physical repository +`(storage, replica_path)` identifier when handling the client requests. + +The format of the replica path for: + +- Object pools is `@cluster/pools/<xx>/<xx>/<repository ID>`. Object pools are stored in a different directory than other repositories. + They must be identifiable by Gitaly to avoid pruning them as part of housekeeping. Pruning object pools can cause data loss in the linked + repositories. +- Other repositories is `@cluster/repositories/<xx>/<xx>/<repository ID>` + +For example, `@cluster/repositories/6f/96/54771`. + +The last component of the replica path, `54771`, is the repository ID. This can be used to identify the repository on the disk. + +`<xx>/<xx>` are the first four hex digits of the SHA256 hash of the string representation of the repository ID. This is used to balance +the repositories evenly into subdirectories to avoid overly large directories that might cause problems on some file +systems. In this case, `54771` hashes to `6f960ab01689464e768366d3315b3d3b2c28f38761a58a70110554eb04d582f7` so the +first four digits are `6f` and `96`. + +#### Identify repositories on disk + +Use the [`praefect metadata`](troubleshooting.md#view-repository-metadata) subcommand to: + +- Retrieve a repository's virtual storage and relative path from the metadata store. After you have the hashed storage path, you can use the Rails + console to retrieve the project path. +- Find where a repository is stored in the cluster with either: + - The virtual storage and relative path. + - The repository ID. + +The repository on disk also contains the project path in the Git configuration file. The configuration file can be used to determine +the project's location even if the repository's metadata has been deleted. Follow the +[instructions in hashed storage's documentation](../repository_storage_types.md#from-hashed-path-to-project-name). + +#### Atomicity of operations + +Gitaly Cluster uses the PostgreSQL metadata store with the storage layout to ensure atomicity of repository creation, +deletion, and move operations. The disk operations can't be atomically applied across multiple storages. However, PostgreSQL guarantees +the atomicity of the metadata operations. Gitaly Cluster models the operations in a manner that the failing operations always leave +the metadata consistent. The disks may contain stale state even after successful operations. This is expected and the leftover state +won't intefere with future operations but may use up disk space unnecessarily until a clean up is performed. + +There is on-going work on a [background crawler](https://gitlab.com/gitlab-org/gitaly/-/issues/3719) that cleans up the leftover +repositories from the storages. + +##### Repository creations + +When creating repositories, Praefect: + +1. Reserves a repository ID from PostgreSQL. This is atomic and no two creations receive the same ID. +1. Creates replicas on the Gitaly storages in the replica path derived from the repository ID. +1. Creates metadata records after the repository is successfully created on disk. + +Even if two concurrent operations create the same repository, they'd be stored in different directories on the storages and not +conflict. The first to complete creates the metadata record and the other operation fails with an "already exists" error. +The failing creation leaves leftover repositories on the storages. There is on-going work on a +[background crawler](https://gitlab.com/gitlab-org/gitaly/-/issues/3719) that clean up the leftover repositories from the storages. + +The repository IDs are generated from the `repositories_repository_id_seq` in PostgreSQL. In the above example, the failing operation took +one repository ID without successfully creating a repository with it. Failed repository creations are expected lead to gaps in the repository IDs. + +##### Repository deletions + +A repository is deleted by removing its metadata record. The repository ceases to logically exist as soon as the metadata record is deleted. +PostgreSQL guarantees the atomicity of the removal and a concurrent delete fails with a "not found" error. After successfully deleting +the metadata record, Praefect attempts to remove the replicas from the storages. This may fail and leave leftover state in the storages. +The leftover state is eventually cleaned up. + +##### Repository moves + +Unlike Gitaly, Gitaly Cluster doesn't move the repositories in the storages but only virtually moves the repository by updating the +relative path of the repository in the metadata store. + ### Moving beyond NFS -Engineering support for NFS for Git repositories is deprecated. Technical support is planned to be unavailable starting GitLab 15.0. Please see our [statement of support](https://about.gitlab.com/support/statement-of-support.html#gitaly-and-nfs) for more details. +Engineering support for NFS for Git repositories is deprecated. Technical support is planned to be unavailable starting +November 22, 2022. Please see our [statement of support](https://about.gitlab.com/support/statement-of-support.html#gitaly-and-nfs) +for more details. [Network File System (NFS)](https://en.wikipedia.org/wiki/Network_File_System) is not well suited to Git workloads which are CPU and IOPS sensitive. @@ -316,7 +448,7 @@ The primary node is chosen to serve the request if: - There are no up to date nodes. - Any other error occurs during node selection. -You can [monitor distribution of reads](#monitor-gitaly-cluster) using Prometheus. +You can [monitor distribution of reads](monitoring.md#monitor-gitaly-cluster) using Prometheus. #### Strong consistency @@ -344,7 +476,7 @@ Strong consistency: - The [13.12 documentation](https://docs.gitlab.com/13.12/ee/administration/gitaly/praefect.html#strong-consistency). - Is unavailable in GitLab 13.0 and earlier. -For more information on monitoring strong consistency, see the Gitaly Cluster [Prometheus metrics documentation](#monitor-gitaly-cluster). +For more information on monitoring strong consistency, see the Gitaly Cluster [Prometheus metrics documentation](monitoring.md#monitor-gitaly-cluster). #### Replication factor @@ -392,167 +524,6 @@ off Gitaly Cluster to a sharded Gitaly instance: 1. [Move the repositories](../operations/moving_repositories.md#move-repositories) to the newly created storage. You can move them by shard or by group, which gives you the opportunity to spread them over multiple Gitaly servers. -## Monitor Gitaly and Gitaly Cluster - -You can use the available logs and [Prometheus metrics](../monitoring/prometheus/index.md) to -monitor Gitaly and Gitaly Cluster (Praefect). - -Metric definitions are available: - -- Directly from Prometheus `/metrics` endpoint configured for Gitaly. -- Using [Grafana Explore](https://grafana.com/docs/grafana/latest/explore/) on a - Grafana instance configured against Prometheus. - -### Monitor Gitaly - -You can observe the behavior of [queued requests](configure_gitaly.md#limit-rpc-concurrency) using -the Gitaly logs and Prometheus: - -- In the [Gitaly logs](../logs.md#gitaly-logs), look for the string (or structured log field) - `acquire_ms`. Messages that have this field are reporting about the concurrency limiter. -- In Prometheus, look for the following metrics: - - `gitaly_rate_limiting_in_progress`. - - `gitaly_rate_limiting_queued`. - - `gitaly_rate_limiting_seconds`. - - Although the name of the Prometheus metric contains `rate_limiting`, it's a concurrency limiter, - not a rate limiter. If a Gitaly client makes 1,000 requests in a row very quickly, concurrency - doesn't exceed 1, and the concurrency limiter has no effect. - -The following [pack-objects cache](configure_gitaly.md#pack-objects-cache) metrics are available: - -- `gitaly_pack_objects_cache_enabled`, a gauge set to `1` when the cache is enabled. Available - labels: `dir` and `max_age`. -- `gitaly_pack_objects_cache_lookups_total`, a counter for cache lookups. Available label: `result`. -- `gitaly_pack_objects_generated_bytes_total`, a counter for the number of bytes written into the - cache. -- `gitaly_pack_objects_served_bytes_total`, a counter for the number of bytes read from the cache. -- `gitaly_streamcache_filestore_disk_usage_bytes`, a gauge for the total size of cache files. - Available label: `dir`. -- `gitaly_streamcache_index_entries`, a gauge for the number of entries in the cache. Available - label: `dir`. - -Some of these metrics start with `gitaly_streamcache` because they are generated by the -`streamcache` internal library package in Gitaly. - -Example: - -```plaintext -gitaly_pack_objects_cache_enabled{dir="/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache",max_age="300"} 1 -gitaly_pack_objects_cache_lookups_total{result="hit"} 2 -gitaly_pack_objects_cache_lookups_total{result="miss"} 1 -gitaly_pack_objects_generated_bytes_total 2.618649e+07 -gitaly_pack_objects_served_bytes_total 7.855947e+07 -gitaly_streamcache_filestore_disk_usage_bytes{dir="/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache"} 2.6200152e+07 -gitaly_streamcache_filestore_removed_total{dir="/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache"} 1 -gitaly_streamcache_index_entries{dir="/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache"} 1 -``` - -#### Useful queries - -The following are useful queries for monitoring Gitaly: - -- Use the following Prometheus query to observe the - [type of connections](configure_gitaly.md#enable-tls-support) Gitaly is serving a production - environment: - - ```prometheus - sum(rate(gitaly_connections_total[5m])) by (type) - ``` - -- Use the following Prometheus query to monitor the - [authentication behavior](configure_gitaly.md#observe-type-of-gitaly-connections) of your GitLab - installation: - - ```prometheus - sum(rate(gitaly_authentications_total[5m])) by (enforced, status) - ``` - - In a system where authentication is configured correctly and where you have live traffic, you - see something like this: - - ```prometheus - {enforced="true",status="ok"} 4424.985419441742 - ``` - - There may also be other numbers with rate 0, but you only have to take note of the non-zero numbers. - - The only non-zero number should have `enforced="true",status="ok"`. If you have other non-zero - numbers, something is wrong in your configuration. - - The `status="ok"` number reflects your current request rate. In the example above, Gitaly is - handling about 4000 requests per second. - -- Use the following Prometheus query to observe the [Git protocol versions](../git_protocol.md) - being used in a production environment: - - ```prometheus - sum(rate(gitaly_git_protocol_requests_total[1m])) by (grpc_method,git_protocol,grpc_service) - ``` - -### Monitor Gitaly Cluster - -To monitor Gitaly Cluster (Praefect), you can use these Prometheus metrics. There are two separate metrics -endpoints from which metrics can be scraped: - -- The default `/metrics` endpoint. -- `/db_metrics`, which contains metrics that require database queries. - -#### Default Prometheus `/metrics` endpoint - -The following metrics are available from the `/metrics` endpoint: - -- `gitaly_praefect_read_distribution`, a counter to track [distribution of reads](#distributed-reads). - It has two labels: - - - `virtual_storage`. - - `storage`. - - They reflect configuration defined for this instance of Praefect. - -- `gitaly_praefect_replication_latency_bucket`, a histogram measuring the amount of time it takes - for replication to complete after the replication job starts. Available in GitLab 12.10 and later. -- `gitaly_praefect_replication_delay_bucket`, a histogram measuring how much time passes between - when the replication job is created and when it starts. Available in GitLab 12.10 and later. -- `gitaly_praefect_node_latency_bucket`, a histogram measuring the latency in Gitaly returning - health check information to Praefect. This indicates Praefect connection saturation. Available in - GitLab 12.10 and later. - -To monitor [strong consistency](#strong-consistency), you can use the following Prometheus metrics: - -- `gitaly_praefect_transactions_total`, the number of transactions created and voted on. -- `gitaly_praefect_subtransactions_per_transaction_total`, the number of times nodes cast a vote for - a single transaction. This can happen multiple times if multiple references are getting updated in - a single transaction. -- `gitaly_praefect_voters_per_transaction_total`: the number of Gitaly nodes taking part in a - transaction. -- `gitaly_praefect_transactions_delay_seconds`, the server-side delay introduced by waiting for the - transaction to be committed. -- `gitaly_hook_transaction_voting_delay_seconds`, the client-side delay introduced by waiting for - the transaction to be committed. - -To monitor the number of repositories that have no healthy, up-to-date replicas: - -- `gitaly_praefect_unavailable_repositories` - -You can also monitor the [Praefect logs](../logs.md#praefect-logs). - -#### Database metrics `/db_metrics` endpoint - -> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/3286) in GitLab 14.5. - -The following metrics are available from the `/db_metrics` endpoint: - -- `gitaly_praefect_unavailable_repositories`, the number of repositories that have no healthy, up to date replicas. -- `gitaly_praefect_read_only_repositories`, the number of repositories in read-only mode in a virtual storage. - This metric is available for backwards compatibility reasons. `gitaly_praefect_unavailable_repositories` is more - accurate. -- `gitaly_praefect_replication_queue_depth`, the number of jobs in the replication queue. - -## Recover from failure - -Gitaly Cluster can [recover from certain types of failure](recovery.md). - ## Do not bypass Gitaly GitLab doesn't advise directly accessing Gitaly repositories stored on disk with a Git client, @@ -660,4 +631,4 @@ The second facet presents the only real solution. For this, we developed ## NFS deprecation notice Engineering support for NFS for Git repositories is deprecated. Technical support is planned to be -unavailable from GitLab 15.0. For further information, please see our [NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation) documentation. +unavailable beginning November 22, 2022. For further information, please see our [NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation) documentation. diff --git a/doc/administration/gitaly/monitoring.md b/doc/administration/gitaly/monitoring.md new file mode 100644 index 00000000000..17f94f912ee --- /dev/null +++ b/doc/administration/gitaly/monitoring.md @@ -0,0 +1,202 @@ +--- +stage: Create +group: Gitaly +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Monitoring Gitaly and Gitaly Cluster + +You can use the available logs and [Prometheus metrics](../monitoring/prometheus/index.md) to +monitor Gitaly and Gitaly Cluster (Praefect). + +Metric definitions are available: + +- Directly from Prometheus `/metrics` endpoint configured for Gitaly. +- Using [Grafana Explore](https://grafana.com/docs/grafana/latest/explore/) on a + Grafana instance configured against Prometheus. + +## Monitor Gitaly rate limiting + +Gitaly can be configured to limit requests based on: + +- Concurrency of requests. +- A rate limit. + +Monitor Gitaly request limiting with the `gitaly_requests_dropped_total` Prometheus metric. This metric provides a total count +of requests dropped due to request limiting. The `reason` label indicates why a request was dropped: + +- `rate`, due to rate limiting. +- `max_size`, because the concurrency queue size was reached. +- `max_time`, because the request exceeded the maximum queue wait time as configured in Gitaly. + +## Monitor Gitaly concurrency limiting + +You can observe specific behavior of [concurrency-queued requests](configure_gitaly.md#limit-rpc-concurrency) using +the Gitaly logs and Prometheus: + +- In the [Gitaly logs](../logs.md#gitaly-logs), look for the string (or structured log field) + `acquire_ms`. Messages that have this field are reporting about the concurrency limiter. +- In Prometheus, look for the following metrics: + - `gitaly_concurrency_limiting_in_progress` indicates how many concurrent requests are + being processed. + - `gitaly_concurrency_limiting_queued` indicates how many requests for an RPC for a given + repository are waiting due to the concurrency limit being reached. + - `gitaly_concurrency_limiting_acquiring_seconds` indicates how long a request has to + wait due to concurrency limits before being processed. + +## Monitor Gitaly cgroups + +You can observe the status of [control groups (cgroups)](configure_gitaly.md#control-groups) using Prometheus: + +- `gitaly_cgroups_memory_failed_total`, a gauge for the total number of times + the memory limit has been hit. This number resets each time a server is + restarted. +- `gitaly_cgroups_cpu_usage`, a gauge that measures CPU usage per cgroup. +- `gitaly_cgroup_procs_total`, a gauge that measures the total number of + processes Gitaly has spawned under the control of cgroups. + +## `pack-objects` cache + +The following [`pack-objects` cache](configure_gitaly.md#pack-objects-cache) metrics are available: + +- `gitaly_pack_objects_cache_enabled`, a gauge set to `1` when the cache is enabled. Available + labels: `dir` and `max_age`. +- `gitaly_pack_objects_cache_lookups_total`, a counter for cache lookups. Available label: `result`. +- `gitaly_pack_objects_generated_bytes_total`, a counter for the number of bytes written into the + cache. +- `gitaly_pack_objects_served_bytes_total`, a counter for the number of bytes read from the cache. +- `gitaly_streamcache_filestore_disk_usage_bytes`, a gauge for the total size of cache files. + Available label: `dir`. +- `gitaly_streamcache_index_entries`, a gauge for the number of entries in the cache. Available + label: `dir`. + +Some of these metrics start with `gitaly_streamcache` because they are generated by the +`streamcache` internal library package in Gitaly. + +Example: + +```plaintext +gitaly_pack_objects_cache_enabled{dir="/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache",max_age="300"} 1 +gitaly_pack_objects_cache_lookups_total{result="hit"} 2 +gitaly_pack_objects_cache_lookups_total{result="miss"} 1 +gitaly_pack_objects_generated_bytes_total 2.618649e+07 +gitaly_pack_objects_served_bytes_total 7.855947e+07 +gitaly_streamcache_filestore_disk_usage_bytes{dir="/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache"} 2.6200152e+07 +gitaly_streamcache_filestore_removed_total{dir="/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache"} 1 +gitaly_streamcache_index_entries{dir="/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache"} 1 +``` + +## Useful queries + +The following are useful queries for monitoring Gitaly: + +- Use the following Prometheus query to observe the + [type of connections](configure_gitaly.md#enable-tls-support) Gitaly is serving a production + environment: + + ```prometheus + sum(rate(gitaly_connections_total[5m])) by (type) + ``` + +- Use the following Prometheus query to monitor the + [authentication behavior](configure_gitaly.md#observe-type-of-gitaly-connections) of your GitLab + installation: + + ```prometheus + sum(rate(gitaly_authentications_total[5m])) by (enforced, status) + ``` + + In a system where authentication is configured correctly and where you have live traffic, you + see something like this: + + ```prometheus + {enforced="true",status="ok"} 4424.985419441742 + ``` + + There may also be other numbers with rate 0, but you only have to take note of the non-zero numbers. + + The only non-zero number should have `enforced="true",status="ok"`. If you have other non-zero + numbers, something is wrong in your configuration. + + The `status="ok"` number reflects your current request rate. In the example above, Gitaly is + handling about 4000 requests per second. + +- Use the following Prometheus query to observe the [Git protocol versions](../git_protocol.md) + being used in a production environment: + + ```prometheus + sum(rate(gitaly_git_protocol_requests_total[1m])) by (grpc_method,git_protocol,grpc_service) + ``` + +## Monitor Gitaly Cluster + +To monitor Gitaly Cluster (Praefect), you can use these Prometheus metrics. There are two separate metrics +endpoints from which metrics can be scraped: + +- The default `/metrics` endpoint. +- `/db_metrics`, which contains metrics that require database queries. + +### Default Prometheus `/metrics` endpoint + +The following metrics are available from the `/metrics` endpoint: + +- `gitaly_praefect_read_distribution`, a counter to track [distribution of reads](index.md#distributed-reads). + It has two labels: + + - `virtual_storage`. + - `storage`. + + They reflect configuration defined for this instance of Praefect. + +- `gitaly_praefect_replication_latency_bucket`, a histogram measuring the amount of time it takes + for replication to complete after the replication job starts. Available in GitLab 12.10 and later. +- `gitaly_praefect_replication_delay_bucket`, a histogram measuring how much time passes between + when the replication job is created and when it starts. Available in GitLab 12.10 and later. +- `gitaly_praefect_node_latency_bucket`, a histogram measuring the latency in Gitaly returning + health check information to Praefect. This indicates Praefect connection saturation. Available in + GitLab 12.10 and later. + +To monitor [strong consistency](index.md#strong-consistency), you can use the following Prometheus metrics: + +- `gitaly_praefect_transactions_total`, the number of transactions created and voted on. +- `gitaly_praefect_subtransactions_per_transaction_total`, the number of times nodes cast a vote for + a single transaction. This can happen multiple times if multiple references are getting updated in + a single transaction. +- `gitaly_praefect_voters_per_transaction_total`: the number of Gitaly nodes taking part in a + transaction. +- `gitaly_praefect_transactions_delay_seconds`, the server-side delay introduced by waiting for the + transaction to be committed. +- `gitaly_hook_transaction_voting_delay_seconds`, the client-side delay introduced by waiting for + the transaction to be committed. + +To monitor the number of repositories that have no healthy, up-to-date replicas: + +- `gitaly_praefect_unavailable_repositories` + +To monitor [repository verification](praefect.md#repository-verification), use the following Prometheus metrics: + +- `gitaly_praefect_verification_queue_depth`, the total number of replicas pending verification. This + metric is scraped from the database and is only available when Prometheus is scraping the database metrics. +- `gitaly_praefect_verification_jobs_dequeued_total`, the number of verification jobs picked up by the + worker. +- `gitaly_praefect_verification_jobs_completed_total`, the number of verification jobs completed by the + worker. The `result` label indicates the end result of the jobs: + - `valid` indicates the expected replica existed on the storage. + - `invalid` indicates the replica expected to exist did not exist on the storage. + - `error` indicates the job failed and has to be retried. +- `gitaly_praefect_stale_verification_leases_released_total`, the number of stale verification leases + released. + +You can also monitor the [Praefect logs](../logs.md#praefect-logs). + +### Database metrics `/db_metrics` endpoint + +> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/3286) in GitLab 14.5. + +The following metrics are available from the `/db_metrics` endpoint: + +- `gitaly_praefect_unavailable_repositories`, the number of repositories that have no healthy, up to date replicas. +- `gitaly_praefect_read_only_repositories`, the number of repositories in read-only mode in a virtual storage. + This metric is available for backwards compatibility reasons. `gitaly_praefect_unavailable_repositories` is more + accurate. +- `gitaly_praefect_replication_queue_depth`, the number of jobs in the replication queue. diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index 7dee12a6690..fb9f3e9c361 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -2,7 +2,6 @@ stage: Create group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference --- # Configure Gitaly Cluster **(FREE SELF)** @@ -326,7 +325,7 @@ pgbouncer['databases'] = { user: 'pgbouncer', password: PGBOUNCER_SQL_PASSWORD_HASH, pool_mode: 'transaction' - } + }, praefect_production_direct: { host: POSTGRESQL_HOST, # Use `pgbouncer` user to connect to database backend. @@ -338,6 +337,13 @@ pgbouncer['databases'] = { ... } + +# Allow the praefect user to connect to PgBouncer +pgbouncer['users'] = { + 'praefect': { + 'password': PRAEFECT_SQL_PASSWORD_HASH, + } +} ``` Both `praefect_production` and `praefect_production_direct` use the same database endpoint @@ -422,25 +428,33 @@ On the **Praefect** node: 1. Disable all other services by editing `/etc/gitlab/gitlab.rb`: +<!-- +Updates to example must be made at: +- https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/administration/gitaly/praefect.md +- all reference architecture pages +--> + ```ruby - # Disable all other services on the Praefect node + # Avoid running unnecessary services on the Praefect server + gitaly['enable'] = false postgresql['enable'] = false redis['enable'] = false nginx['enable'] = false - alertmanager['enable'] = false - prometheus['enable'] = false - grafana['enable'] = false puma['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false - gitaly['enable'] = false + prometheus['enable'] = false + alertmanager['enable'] = false + grafana['enable'] = false + gitlab_exporter['enable'] = false + gitlab_kas['enable'] = false # Enable only the Praefect service praefect['enable'] = true - # Disable database migrations to prevent database connections during 'gitlab-ctl reconfigure' - gitlab_rails['auto_migrate'] = false + # Prevent database migrations from running on upgrade automatically praefect['auto_migrate'] = false + gitlab_rails['auto_migrate'] = false ``` 1. Configure **Praefect** to listen on network interfaces by editing @@ -953,7 +967,7 @@ application. This is done by updating the `git_data_dirs`. Particular attention should be shown to: - the storage name added to `git_data_dirs` in this section must match the - storage name under `praefect['virtual_storages']` on the Praefect node(s). This + storage name under `praefect['virtual_storages']` on the Praefect nodes. This was set in the [Praefect](#praefect) section of this guide. This document uses `default` as the Praefect storage name. @@ -1209,6 +1223,110 @@ You can configure: If `default_replication_factor` is unset, the repositories are always replicated on every node defined in `virtual_storages`. If a new node is introduced to the virtual storage, both new and existing repositories are replicated to the node automatically. +## Repository verification + +> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/4080) in GitLab 15.0. + +Praefect stores metadata about the repositories in a database. If the repositories are modified on disk +without going through Praefect, the metadata can become inaccurate. Because the metadata is used for replication +and routing decisions, any inaccuracies may cause problems. Praefect contains a background worker that +periodically verifies the metadata against the actual state on the disks. The worker: + +1. Picks up a batch of replicas to verify on healthy storages. The replicas are either unverified or have exceeded + the configured verification interval. Replicas that have never been verified are prioritized, followed by + the other replicas ordered by longest time since the last successful verification. +1. Checks whether the replicas exist on their respective storages. If the: + - Replica exists, update its last successful verification time. + - Replica doesn't exist, remove its metadata record. + - Check failed, the replica is picked up for verification again when the next worker dequeues more work. + +The worker acquires an exclusive verification lease on each of the replicas it is about to verify. This avoids multiple +workers from verifying the same replica concurrently. The worker releases the leases when it has completed its check. +Praefect contains a background goroutine that releases stale leases every 10 seconds when workers are terminated for +some reason without releasing the lease. + +The worker logs each of the metadata removals prior to executing them. The `perform_deletions` key +indicates whether the invalid metadata records are actually deleted or not. For example: + +```json +{ + "level": "info", + "msg": "removing metadata records of non-existent replicas", + "perform_deletions": false, + "replicas": { + "default": { + "@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.git": [ + "praefect-internal-0" + ] + } + } +} +``` + +### Configure the verification worker + +The worker is enabled by default and verifies the metadata records every seven days. The verification +interval is configurable with any valid [Go duration string](https://pkg.go.dev/time#ParseDuration). + +To verify the metadata every three days: + +```ruby +praefect['background_verification_verification_interval'] = '72h' +``` + +Values of 0 and below disable the background verifier. + +```ruby +praefect['background_verification_verification_interval'] = '0' +``` + +#### Enable deletions + +WARNING: +Deletions are disabled by default due to a race condition with repository renames that can cause incorrect +deletions. This is especially prominent in Geo instances as Geo performs more renames than instances without Geo. +See [Handle repository creations, deletions and renames atomically](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4101) +for progress on a fix. We do not recommend enabling the deletions until this is fixed. + +By default, the worker does not delete invalid metadata records but simply logs them and outputs Prometheus +metrics for them. + +You can enable deleting invalid metadata records with: + +```ruby +praefect['background_verification_delete_invalid_records'] = true +``` + +### Prioritize verification manually + +You can prioritize verification of some replicas ahead of their next scheduled verification time. +This might be needed after a disk failure, for example, when the administrator knows that the disk contents may have +changed. Praefect would eventually verify the replicas again, but users may encounter errors in the meantime. + +To manually prioritize reverification of some replicas, use the `praefect verify` subcommand. The subcommand marks +replicas as unverified. Unverified replicas are prioritized by the background verification worker. The verification +worker must be enabled for the replicas to be verified. + +Prioritize verifying the replicas of a specific repository: + +```shell +sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml verify -repository-id=<repository-id> +``` + +Prioritize verifying all replicas stored on a virtual storage: + +```shell +sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml verify -virtual-storage=<virtual-storage> +``` + +Prioritize verifying all replicas stored on a storage: + +```shell +sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml verify -virtual-storage=<virtual-storage> -storage=<storage> +``` + +The output includes the number of replicas that were marked unverified. + ## Automatic failover and primary election strategies Praefect regularly checks the health of each Gitaly node. This is used to automatically fail over diff --git a/doc/administration/gitaly/recovery.md b/doc/administration/gitaly/recovery.md index a7166f7e62e..6de2acf1792 100644 --- a/doc/administration/gitaly/recovery.md +++ b/doc/administration/gitaly/recovery.md @@ -2,7 +2,6 @@ stage: Create group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference --- # Recovery options @@ -348,13 +347,23 @@ If this occurs, run `remove-repository` again. ### Manually list untracked repositories -> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3926) in GitLab 14.4. +> - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3926) in GitLab 14.4. +> - `older-than` option added in GitLab 15.0. The `list-untracked-repositories` Praefect sub-command lists repositories of the Gitaly Cluster that both: - Exist for at least one Gitaly storage. - Aren't tracked in the Praefect database. +Add the `-older-than` option to avoid showing repositories that are the process of being created and for which a record doesn't yet exist in the +Praefect database. Replace `<duration>` with a time duration (for example, `5s`, `10m`, or `1h`). Defaults to `6h`. + +```shell +sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml list-untracked-repositories -older-than <duration> +``` + +Only repositories with a creation time before the specified duration are considered. + The command outputs: - Result to `STDOUT` and the command's logs. diff --git a/doc/administration/gitaly/reference.md b/doc/administration/gitaly/reference.md index 9fe09be10a3..d1e802111cd 100644 --- a/doc/administration/gitaly/reference.md +++ b/doc/administration/gitaly/reference.md @@ -2,7 +2,6 @@ stage: Create group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference --- # Gitaly reference **(FREE SELF)** @@ -71,7 +70,7 @@ Remember to disable `transitioning` when you are done changing your token settings. All authentication attempts are counted in Prometheus under -the [`gitaly_authentications_total` metric](index.md#useful-queries). +the [`gitaly_authentications_total` metric](monitoring.md#useful-queries). ### TLS diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md index 1be0bf23ed5..c79ed1d1707 100644 --- a/doc/administration/gitaly/troubleshooting.md +++ b/doc/administration/gitaly/troubleshooting.md @@ -2,7 +2,6 @@ stage: Create group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference --- # Troubleshooting Gitaly and Gitaly Cluster **(FREE SELF)** @@ -356,7 +355,7 @@ The following sections provide possible solutions to Gitaly Cluster errors. ### Check cluster health -> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/) in GitLab 14.6. +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5688) in GitLab 14.5. The `check` Praefect sub-command runs a series of checks to determine the health of the Gitaly Cluster. @@ -417,6 +416,16 @@ If this check fails: 1. Check if there is a high load on the Praefect database. If the Praefect database is slow to respond, it can lead health checks failing to persist to the database, leading Praefect to think nodes are unhealthy. +#### Check clock synchronization + +> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4225) in GitLab 14.8. + +Authentication between Praefect and the Gitaly servers requires the server times to be +in sync so the token check succeeds. + +This check helps identify the root cause of `permission denied` +[errors being logged by Praefect](#permission-denied-errors-appearing-in-gitaly-or-praefect-logs-when-accessing-repositories). + ### Praefect errors in logs If you receive an error, check `/var/log/gitlab/gitlab-rails/production.log`. @@ -513,16 +522,19 @@ Replicas: Generation: 1, fully up to date Healthy: true Valid Primary: true + Verified At: 2021-04-01 10:04:20 +0000 UTC - Storage: "gitaly-2" Assigned: true Generation: 0, behind by 1 changes Healthy: true Valid Primary: false + Verified At: unverified - Storage: "gitaly-3" Assigned: true Generation: replica not yet created Healthy: false Valid Primary: false + Verified At: unverified ``` #### Available metadata @@ -548,6 +560,7 @@ For each replica, the following metadata is available: | `Generation` | Latest confirmed generation of the replica. It indicates:<br><br>- The replica is fully up to date if the generation matches the repository's generation.<br>- The replica is outdated if the replica's generation is less than the repository's generation.<br>- `replica not yet created` if the replica does not yet exist at all on the storage. | | `Healthy` | Indicates whether the Gitaly node that is hosting this replica is considered healthy by the consensus of Praefect nodes. | | `Valid Primary` | Indicates whether the replica is fit to serve as the primary node. If the repository's primary is not a valid primary, a failover occurs on the next write to the repository if there is another replica that is a valid primary. A replica is a valid primary if:<br><br>- It is stored on a healthy Gitaly node.<br>- It is fully up to date.<br>- It is not targeted by a pending deletion job from decreasing replication factor.<br>- It is assigned. | +| `Verified At` | Indicates last successful verification of the replica by the [verification worker](praefect.md#repository-verification). If the replica has not yet been verified, `unverified` is displayed in place of the last successful verification time. Introduced in GitLab 15.0. | ### Check that repositories are in sync diff --git a/doc/administration/inactive_project_deletion.md b/doc/administration/inactive_project_deletion.md new file mode 100644 index 00000000000..a2d2093c57b --- /dev/null +++ b/doc/administration/inactive_project_deletion.md @@ -0,0 +1,60 @@ +--- +stage: Manage +group: Compliance +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Inactive project deletion **(FREE SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/85689) in GitLab 15.0 [with a flag](../administration/feature_flags.md) named `inactive_projects_deletion`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to +[enable the feature flag](../administration/feature_flags.md) named `inactive_projects_deletion`. +On GitLab.com, this feature is not available. This feature is not ready for production use. + +Administrators of large GitLab instances can find that over time, projects become inactive and are no longer used. +These projects take up unnecessary disk space. With inactive project deletion, you can identify these projects, warn +the maintainers ahead of time, and then delete the projects if they remain inactive. When an inactive project is +deleted, the action generates an audit event that it was performed by the first active administrator. + +## Configure inactive project deletion + +You can configure inactive projects deletion or turn it off using the +[Application settings API](../api/settings.md#change-application-settings). + +The following options are available: + +- `delete_inactive_projects`: Enable or disable inactive project deletion. +- `inactive_projects_min_size_mb`: Minimum size (MB) of inactive projects to be considered for deletion. + Projects smaller in size than this threshold aren't considered inactive. +- `inactive_projects_delete_after_months`: Minimum duration (months) after which a project is scheduled for deletion if + it continues be inactive. +- `inactive_projects_send_warning_email_after_months`: Minimum duration (months) after which a deletion warning email is + sent if a project continues to be inactive. The warning email is sent to users with the Owner and Maintainer roles of + the inactive project. This duration should be less than the `inactive_projects_delete_after_months` duration. + +For example: + +- `delete_inactive_projects` enabled. +- `inactive_projects_min_size_mb` set to `50`. +- `inactive_projects_delete_after_months` set to `12`. +- `inactive_projects_send_warning_email_after_months` set to `6`. + +In this scenario, when a project's size is: + +- Less than 50 MB, the project is not considered inactive. +- Greater than 50 MB and it is inactive for: + - More than 6 months, a deletion warning is email is sent to users with the Owner and Maintainer role on the project + with the scheduled date of deletion. + - More than 12 months, the project is scheduled for deletion. + +## Determine when a project was last active + +You can view a project's activities and determine when the project was last active in the following ways: + +1. Go to the [activity page](../user/project/working_with_projects.md#view-project-activity) for the project and view + the date of the latest event. +1. View the `last_activity_at` attribute for the project using the [Projects API](../api/projects.md). +1. List the visible events for the project using the [Events API](../api/events.md#list-a-projects-visible-events). + View the `created_at` attribute of the latest event. diff --git a/doc/administration/incoming_email.md b/doc/administration/incoming_email.md index 63f8ce3c4cb..2fd57036169 100644 --- a/doc/administration/incoming_email.md +++ b/doc/administration/incoming_email.md @@ -302,7 +302,7 @@ gitlab_rails['incoming_email_mailbox_name'] = "inbox" # The IDLE command timeout. gitlab_rails['incoming_email_idle_timeout'] = 60 -# Whether to expunge (permanently remove) messages from the mailbox when they are deleted after delivery +# Whether to expunge (permanently remove) messages from the mailbox when they are marked as deleted after delivery gitlab_rails['incoming_email_expunge_deleted'] = true ``` @@ -340,7 +340,7 @@ incoming_email: # The IDLE command timeout. idle_timeout: 60 - # Whether to expunge (permanently remove) messages from the mailbox when they are deleted after delivery + # Whether to expunge (permanently remove) messages from the mailbox when they are marked as deleted after delivery expunge_deleted: true ``` @@ -384,7 +384,7 @@ gitlab_rails['incoming_email_mailbox_name'] = "inbox" # The IDLE command timeout. gitlab_rails['incoming_email_idle_timeout'] = 60 -# Whether to expunge (permanently remove) messages from the mailbox when they are deleted after delivery +# Whether to expunge (permanently remove) messages from the mailbox when they are marked as deleted after delivery gitlab_rails['incoming_email_expunge_deleted'] = true ``` @@ -422,7 +422,7 @@ incoming_email: # The IDLE command timeout. idle_timeout: 60 - # Whether to expunge (permanently remove) messages from the mailbox when they are deleted after delivery + # Whether to expunge (permanently remove) messages from the mailbox when they are marked as deleted after delivery expunge_deleted: true ``` @@ -463,6 +463,9 @@ gitlab_rails['incoming_email_host'] = "exchange.example.com" gitlab_rails['incoming_email_port'] = 993 # Whether the IMAP server uses SSL gitlab_rails['incoming_email_ssl'] = true + +# Whether to expunge (permanently remove) messages from the mailbox when they are marked as deleted after delivery +gitlab_rails['incoming_email_expunge_deleted'] = true ``` Example for source installs: @@ -491,6 +494,9 @@ incoming_email: port: 993 # Whether the IMAP server uses SSL ssl: true + + # Whether to expunge (permanently remove) messages from the mailbox when they are marked as deleted after delivery + expunge_deleted: true ``` ##### Dedicated email address @@ -521,6 +527,9 @@ gitlab_rails['incoming_email_host'] = "exchange.example.com" gitlab_rails['incoming_email_port'] = 993 # Whether the IMAP server uses SSL gitlab_rails['incoming_email_ssl'] = true + +# Whether to expunge (permanently remove) messages from the mailbox when they are marked as deleted after delivery +gitlab_rails['incoming_email_expunge_deleted'] = true ``` Example for source installs: @@ -545,6 +554,9 @@ incoming_email: port: 993 # Whether the IMAP server uses SSL ssl: true + + # Whether to expunge (permanently remove) messages from the mailbox when they are marked as deleted after delivery + expunge_deleted: true ``` #### Microsoft Office 365 @@ -599,6 +611,9 @@ gitlab_rails['incoming_email_host'] = "outlook.office365.com" gitlab_rails['incoming_email_port'] = 993 # Whether the IMAP server uses SSL gitlab_rails['incoming_email_ssl'] = true + +# Whether to expunge (permanently remove) messages from the mailbox when they are marked as deleted after delivery +gitlab_rails['incoming_email_expunge_deleted'] = true ``` This example for source installs assumes the mailbox `incoming@office365.example.com`: @@ -626,6 +641,9 @@ incoming_email: port: 993 # Whether the IMAP server uses SSL ssl: true + + # Whether to expunge (permanently remove) messages from the mailbox when they are marked as deleted after delivery + expunge_deleted: true ``` ##### Catch-all mailbox @@ -654,6 +672,9 @@ gitlab_rails['incoming_email_host'] = "outlook.office365.com" gitlab_rails['incoming_email_port'] = 993 # Whether the IMAP server uses SSL gitlab_rails['incoming_email_ssl'] = true + +# Whether to expunge (permanently remove) messages from the mailbox when they are marked as deleted after delivery +gitlab_rails['incoming_email_expunge_deleted'] = true ``` This example for source installs assumes the catch-all mailbox `incoming@office365.example.com`: @@ -681,6 +702,9 @@ incoming_email: port: 993 # Whether the IMAP server uses SSL ssl: true + + # Whether to expunge (permanently remove) messages from the mailbox when they are marked as deleted after delivery + expunge_deleted: true ``` ##### Dedicated email address @@ -708,6 +732,9 @@ gitlab_rails['incoming_email_host'] = "outlook.office365.com" gitlab_rails['incoming_email_port'] = 993 # Whether the IMAP server uses SSL gitlab_rails['incoming_email_ssl'] = true + +# Whether to expunge (permanently remove) messages from the mailbox when they are marked as deleted after delivery +gitlab_rails['incoming_email_expunge_deleted'] = true ``` This example for source installs assumes the dedicated email address `incoming@office365.example.com`: @@ -730,6 +757,9 @@ incoming_email: port: 993 # Whether the IMAP server uses SSL ssl: true + + # Whether to expunge (permanently remove) messages from the mailbox when they are marked as deleted after delivery + expunge_deleted: true ``` #### Microsoft Graph diff --git a/doc/administration/index.md b/doc/administration/index.md index 73ea4a8e1d0..1d8dcd34d68 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -203,13 +203,8 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Enable Performance Monitoring](monitoring/performance/gitlab_configuration.md): Enable GitLab Performance Monitoring. - [GitLab performance monitoring with Prometheus](monitoring/prometheus/index.md): Configure GitLab and Prometheus for measuring performance metrics. - [GitLab performance monitoring with Grafana](monitoring/performance/grafana_configuration.md): Configure GitLab to visualize time series metrics through graphs and dashboards. - - [Request Profiling](monitoring/performance/request_profiling.md): Get a detailed profile on slow requests. - [Performance Bar](monitoring/performance/performance_bar.md): Get performance information for the current page. -## Analytics - -- [Pseudonymizer](pseudonymizer.md): Export data from a GitLab database to CSV files in a secure way. - ## Troubleshooting - [Debugging tips](troubleshooting/debug.md): Tips to debug problems when things go wrong. diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md index 0a0c538caa2..8d3fec3b19e 100644 --- a/doc/administration/instance_limits.md +++ b/doc/administration/instance_limits.md @@ -636,7 +636,8 @@ If the limit value is set to zero, the limit is disabled. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276192) in GitLab 14.1, disabled by default. > - Enabled by default and [feature flag `ci_jobs_trace_size_limit` removed](https://gitlab.com/gitlab-org/gitlab/-/issues/335259) in GitLab 14.2. -The job log file size limit is 100 megabytes by default. Any job that exceeds this value is dropped. +The job log file size limit in GitLab is 100 megabytes by default. Any job that exceeds the +limit is marked as failed, and dropped by the runner. You can change the limit in the [GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session). Update `ci_jobs_trace_size_limit` with the new value in megabytes: @@ -645,6 +646,11 @@ Update `ci_jobs_trace_size_limit` with the new value in megabytes: Plan.default.actual_limits.update!(ci_jobs_trace_size_limit: 125) ``` +GitLab Runner also has an +[`output_limit` setting](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) +that configures the maximum log size in a runner. Jobs that exceed the runner limit +continue to run, but the log is truncated when it hits the limit. + ### Maximum number of active DAST profile schedules per project > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68551) in GitLab 14.3. @@ -876,6 +882,10 @@ See the limits in the [Add a design to an issue](../user/project/issues/design_m ## Push Event Limits +### Max push size + +The maximum allowed [push size](../user/admin_area/settings/account_and_limit_settings.md#max-push-size) is set to 5 GB. + ### Webhooks and Project Services > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/31009) in GitLab 12.4. @@ -886,7 +896,7 @@ than the specified limit, hooks are not executed. More information can be found in these docs: - [Webhooks push events](../user/project/integrations/webhook_events.md#push-events) -- [Project services push hooks limit](../user/project/integrations/overview.md#push-hooks-limit) +- [Project services push hooks limit](../user/project/integrations/index.md#push-hooks-limit) ### Activities diff --git a/doc/administration/integration/terminal.md b/doc/administration/integration/terminal.md index 792afc6c3d7..88b320941de 100644 --- a/doc/administration/integration/terminal.md +++ b/doc/administration/integration/terminal.md @@ -6,11 +6,15 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Web terminals (DEPRECATED) **(FREE)** -> [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. +> - [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. +> - [Disabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/353410) in GitLab 15.0. WARNING: This feature was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../../administration/feature_flags.md) named `certificate_based_clusters`. + - Read more about the non-deprecated [Web Terminals accessible through the Web IDE](../../user/project/web_ide/index.md). - Read more about the non-deprecated [Web Terminals accessible from a running CI job](../../ci/interactive_web_terminal/index.md). diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index 03a0162d526..2582df1b0c4 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -113,8 +113,6 @@ and then `object_store:`. On Omnibus GitLab installs they are prefixed by |---------------------|---------|-------------| | `enabled` | `false` | Enable or disable object storage. | | `remote_directory` | | The bucket name where Artifacts are stored. Use the name only, do not include the path. | -| `direct_upload` | `false` | Set to `true` to enable direct upload of Artifacts without the need of local shared storage. Option may be removed once we decide to support only single storage for all files. | -| `background_upload` | `true` | Set to `false` to disable automatic upload. Option may be removed once upload is direct to S3. | | `proxy_download` | `false` | Set to `true` to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data. | | `connection` | | Various connection options described below. | @@ -181,67 +179,6 @@ _The artifacts are stored by default in 1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. 1. [Migrate any existing local artifacts to the object storage](#migrating-to-object-storage). -### OpenStack example - -See [the available connection settings for OpenStack](object_storage.md#openstack-compatible-connection-settings). - -**In Omnibus installations:** - -_The uploads are stored by default in -`/var/opt/gitlab/gitlab-rails/shared/artifacts`._ - -1. Edit `/etc/gitlab/gitlab.rb` and add the following lines, substituting - the values you want: - - ```ruby - gitlab_rails['artifacts_enabled'] = true - gitlab_rails['artifacts_object_store_enabled'] = true - gitlab_rails['artifacts_object_store_remote_directory'] = "artifacts" - gitlab_rails['artifacts_object_store_connection'] = { - 'provider' => 'OpenStack', - 'openstack_username' => 'OS_USERNAME', - 'openstack_api_key' => 'OS_PASSWORD', - 'openstack_temp_url_key' => 'OS_TEMP_URL_KEY', - 'openstack_auth_url' => 'https://auth.cloud.ovh.net', - 'openstack_region' => 'GRA', - 'openstack_tenant_id' => 'OS_TENANT_ID', - } - ``` - -1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. [Migrate any existing local artifacts to the object storage](#migrating-to-object-storage). - ---- - -**In installations from source:** - -_The uploads are stored by default in -`/home/git/gitlab/shared/artifacts`._ - -1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following - lines: - - ```yaml - uploads: - object_store: - enabled: true - direct_upload: false - background_upload: true - proxy_download: false - remote_directory: "artifacts" - connection: - provider: OpenStack - openstack_username: OS_USERNAME - openstack_api_key: OS_PASSWORD - openstack_temp_url_key: OS_TEMP_URL_KEY - openstack_auth_url: 'https://auth.cloud.ovh.net' - openstack_region: GRA - openstack_tenant_id: OS_TENANT_ID - ``` - -1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. -1. [Migrate any existing local artifacts to the object storage](#migrating-to-object-storage). - ### Migrating to object storage After [configuring the object storage](#using-object-storage), use the following task to @@ -610,7 +547,7 @@ Bucket names that include folder paths are not supported with [consolidated obje For example, `bucket/path`. If a bucket name has a path in it, you might receive an error similar to: ```plaintext -WARNING: Uploading artifacts as "archive" to coordinator... POST https://gitlab.example.com/api/v4/jobs/job_id/artifacts?artifact_format=zip&artifact_type=archive&expire_in=1+day: 500 Internal Server Error (Missing file) +WARNING: Uploading artifacts as "archive" to coordinator... POST https://gitlab.example.com/api/v4/jobs/job_id/artifacts?artifact_format=zip&artifact_type=archive&expire_in=1+day: 500 Internal Server Error (Missing file) FATAL: invalid argument ``` diff --git a/doc/administration/job_logs.md b/doc/administration/job_logs.md index 5c6ea7f52eb..d2837bfa96e 100644 --- a/doc/administration/job_logs.md +++ b/doc/administration/job_logs.md @@ -177,7 +177,7 @@ Here is the detailed data flow: ### Limitations -- [Redis cluster is not supported](https://gitlab.com/gitlab-org/gitlab/-/issues/224171). +- [Redis Cluster is not supported](https://gitlab.com/gitlab-org/gitlab/-/issues/224171). - You must configure [object storage for CI/CD artifacts, logs, and builds](job_artifacts.md#object-storage-settings) before you enable the feature flag. After the flag is enabled, files cannot be written to disk, and there is no protection against misconfiguration. diff --git a/doc/administration/lfs/index.md b/doc/administration/lfs/index.md index 3fe6a94ef13..0d75880bdd1 100644 --- a/doc/administration/lfs/index.md +++ b/doc/administration/lfs/index.md @@ -89,8 +89,6 @@ The following general settings are supported. |---------------------|-------------|---------| | `enabled` | Enable/disable object storage. | `false` | | `remote_directory` | The bucket name where LFS objects are stored. | | -| `direct_upload` | Set to true to enable direct upload of LFS without the need of local shared storage. Option may be removed after we decide to support only single storage for all files. | `false` | -| `background_upload` | Set to false to disable automatic upload. Option may be removed once upload is direct to S3. | `true` | | `proxy_download` | Set to true to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data. | `false` | | `connection` | Various connection options described below. | | diff --git a/doc/administration/logs.md b/doc/administration/logs.md index 22e13270179..c89c485e7d3 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -385,7 +385,7 @@ Depending on your installation method, this file is located at: - Omnibus GitLab: `/var/log/gitlab/gitlab-rails/integrations_json.log` - Installations from source: `/home/git/gitlab/log/integrations_json.log` -It contains information about [integration](../user/project/integrations/overview.md) +It contains information about [integration](../user/project/integrations/index.md) activities, such as Jira, Asana, and irker services. It uses JSON format, like this example: @@ -1107,7 +1107,7 @@ in `/var/log/gitlab/gitlab-kas/`. For Omnibus GitLab installations, Praefect logs are in `/var/log/gitlab/praefect/`. -GitLab also tracks [Prometheus metrics for Praefect](gitaly/#monitor-gitaly-cluster). +GitLab also tracks [Prometheus metrics for Praefect](gitaly/monitoring.md#monitor-gitaly-cluster). ## Backup log diff --git a/doc/administration/maintenance_mode/index.md b/doc/administration/maintenance_mode/index.md index 50c0f0ecc63..41eed500594 100644 --- a/doc/administration/maintenance_mode/index.md +++ b/doc/administration/maintenance_mode/index.md @@ -116,11 +116,11 @@ For most JSON requests, POST, PUT, PATCH, and DELETE are blocked, and the API re | POST | `/users/sign_in` | To allow users to log in. | | POST | `/users/sign_out`| To allow users to log out. | | POST | `/oauth/token` | To allow users to log in to a Geo secondary for the first time. | -| POST | `/admin/session`, `/admin/session/destroy` | To allow [Administrator mode for GitLab administrators](https://gitlab.com/groups/gitlab-org/-/epics/2158) | +| POST | `/admin/session`, `/admin/session/destroy` | To allow [Admin Mode for GitLab administrators](https://gitlab.com/groups/gitlab-org/-/epics/2158) | | POST | Paths ending with `/compare`| Git revision routes. | | POST | `.git/git-upload-pack` | To allow Git pull/clone. | | POST | `/api/v4/internal` | [internal API routes](../../development/internal_api/index.md) | -| POST | `/admin/sidekiq` | To allow management of background jobs in the Admin UI | +| POST | `/admin/sidekiq` | To allow management of background jobs in the Admin Area | | POST | `/admin/geo` | To allow updating Geo Nodes in the administrator UI | | POST | `/api/v4/geo_replication`| To allow certain Geo-specific administrator UI actions on secondary sites | @@ -166,7 +166,7 @@ Package Registry allows you to install but not publish packages. Background jobs (cron jobs, Sidekiq) continue running as is, because background jobs are not automatically disabled. -[During a planned Geo failover](../geo/disaster_recovery/planned_failover.md#prevent-updates-to-the-primary-node), +[During a planned Geo failover](../geo/disaster_recovery/planned_failover.md#prevent-updates-to-the-primary-site), it is recommended that you disable all cron jobs except for those related to Geo. To monitor queues and disable jobs: @@ -210,4 +210,4 @@ For the same reason we don't automatically block background jobs when Maintenanc The resulting database writes are acceptable. Here, the trade-off is between more service degradation and the completion of replication. -However, during a planned failover, we [ask users to turn off cron jobs that are not related to Geo, manually](../geo/disaster_recovery/planned_failover.md#prevent-updates-to-the-primary-node). In the absence of new database writes and non-Geo cron jobs, new background jobs would either not be created at all or be minimal. +However, during a planned failover, we [ask users to turn off cron jobs that are not related to Geo, manually](../geo/disaster_recovery/planned_failover.md#prevent-updates-to-the-primary-site). In the absence of new database writes and non-Geo cron jobs, new background jobs would either not be created at all or be minimal. diff --git a/doc/administration/merge_request_diffs.md b/doc/administration/merge_request_diffs.md index 01576eb4abf..fe1c74b0e24 100644 --- a/doc/administration/merge_request_diffs.md +++ b/doc/administration/merge_request_diffs.md @@ -117,8 +117,6 @@ then `object_store:`. On Omnibus installations, they are prefixed by |---------|-------------|---------| | `enabled` | Enable/disable object storage | `false` | | `remote_directory` | The bucket name where external diffs are stored| | -| `direct_upload` | Set to `true` to enable direct upload of external diffs without the need of local shared storage. Option may be removed once we decide to support only single storage for all files. | `false` | -| `background_upload` | Set to `false` to disable automatic upload. Option may be removed once upload is direct to S3 | `true` | | `proxy_download` | Set to `true` to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` | | `connection` | Various connection options described below | | diff --git a/doc/administration/monitoring/performance/img/request_profile_result.png b/doc/administration/monitoring/performance/img/request_profile_result.png Binary files differdeleted file mode 100644 index 9176a0b49fd..00000000000 --- a/doc/administration/monitoring/performance/img/request_profile_result.png +++ /dev/null diff --git a/doc/administration/monitoring/performance/index.md b/doc/administration/monitoring/performance/index.md index 20fad8baf91..b063a20dc7f 100644 --- a/doc/administration/monitoring/performance/index.md +++ b/doc/administration/monitoring/performance/index.md @@ -17,7 +17,6 @@ documents to understand and properly configure GitLab Performance Monitoring: - [Prometheus documentation](../prometheus/index.md) - [Grafana Install/Configuration](grafana_configuration.md) - [Performance bar](performance_bar.md) -- [Request profiling](request_profiling.md) ## Introduction to GitLab Performance Monitoring diff --git a/doc/administration/monitoring/performance/request_profiling.md b/doc/administration/monitoring/performance/request_profiling.md index 6cd20132092..d0d05c16ea0 100644 --- a/doc/administration/monitoring/performance/request_profiling.md +++ b/doc/administration/monitoring/performance/request_profiling.md @@ -2,43 +2,11 @@ stage: Monitor group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +remove_date: '2022-07-26' +redirect_to: 'index.md' --- -# Request profiling (DEPRECATED) **(FREE SELF)** +# Request profiling (removed) **(FREE SELF)** -> [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/352488) in GitLab 14.8, and planned for removal in GitLab 15.0. - -WARNING: -This feature is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/352488) -in GitLab 14.8, and is planned for removal in GitLab 15.0. - -To profile a request: - -1. Sign in to GitLab as an Administrator or a user with the [Maintainer role](../../../user/permissions.md). -1. In the navigation bar, click **Admin area**. -1. Go to **Monitoring > Requests Profiles**. -1. In the **Requests Profiles** section, copy the token. -1. Pass the headers `X-Profile-Token: <token>` and `X-Profile-Mode: <mode>`(where - `<mode>` can be `execution` or `memory`) to the request you want to profile. When - passing headers, you can use: - - - Browser extensions such as the - [ModHeader](https://chrome.google.com/webstore/detail/modheader/idgpnmonknjnojddfkpgkljpfnnfcklj) - Chrome extension. - - `curl`. For example: - - ```shell - curl --header 'X-Profile-Token: <token>' --header 'X-Profile-Mode: <mode>' "https://gitlab.example.com/group/project" - ``` - - Profiled requests can take longer than usual. - -After the request completes, you can view the profiling output from the -**Monitoring > Requests Profiles** administration page: - - - -## Cleaning up profiled requests - -The output from profiled requests is cleared out once each day through a -Sidekiq worker. +This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/352488) in GitLab 14.8 +and [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/350152) in 15.0. diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index 08f7d5095da..4f8fbd0c07e 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -156,7 +156,7 @@ The following metrics can be controlled by feature flags: ## Praefect metrics You can [configure Praefect](../../gitaly/praefect.md#praefect) to report metrics. For information -on available metrics, see the [relevant documentation](../../gitaly/index.md#monitor-gitaly-cluster). +on available metrics, see the [relevant documentation](../../gitaly/monitoring.md#monitor-gitaly-cluster). ## Sidekiq metrics diff --git a/doc/administration/nfs.md b/doc/administration/nfs.md index ef5d26ac845..b0c50d63e78 100644 --- a/doc/administration/nfs.md +++ b/doc/administration/nfs.md @@ -24,9 +24,9 @@ file system performance, see Starting with GitLab version 14.0, support for NFS to store Git repository data is deprecated. Technical customer support and engineering support is available for the 14.x releases. Engineering is fixing bugs and security vulnerabilities consistent with our [release and maintenance policy](../policy/maintenance.md#security-releases). -Upon the release of GitLab 15.0 (tentatively May 22nd, 2022) technical and engineering support for using NFS to store Git repository data will be officially at end-of-life. There will be no product changes or troubleshooting provided via Engineering, Security or Paid Support channels after the release date of 15.0, regardless of your GitLab version. +Upon the release of GitLab 15.6 technical and engineering support for using NFS to store Git repository data will be officially at end-of-life. There will be no product changes or troubleshooting provided via Engineering, Security or Paid Support channels after the release date of 15.6, regardless of your GitLab version. -Until the release of 15.0, for customers running 14.x releases, we continue to help with Git related tickets from customers running one or more Gitaly servers with its data stored on NFS. Examples may include: +Until the release of 15.6, for customers running 14.x releases, we continue to help with Git related tickets from customers running one or more Gitaly servers with its data stored on NFS. Examples may include: - Performance issues or timeouts accessing Git data - Commits or branches vanish @@ -39,10 +39,10 @@ Assistance is limited to activities like: - Verifying that NFS client mount options match our [documented recommendations](#mount-options) - Analyzing the GitLab Workhorse and Rails logs, and determining that `500` errors being seen in the environment are caused by slow responses from Gitaly -GitLab support is unable to continue with the investigation if: +GitLab support is unable to continue with the investigation if both: -- The date of the request is on or after the release of GitLab version 15.0, and -- Support Engineers and Management determine that all reasonable non-NFS root causes have been exhausted +- The date of the request is on or after the release of GitLab version 15.6. +- Support Engineers and Management determine that all reasonable non-NFS root causes have been exhausted. If the issue is reproducible, or if it happens intermittently but regularly, GitLab Support can investigate providing the issue reproduces without the use of NFS. In order to reproduce without NFS, the affected repositories should be migrated to a different Gitaly shard, such as Gitaly cluster or a standalone Gitaly VM, backed with block storage. @@ -377,7 +377,7 @@ Any `Operation not permitted` errors means you should investigate your NFS serve ## NFS in a Firewalled Environment -If the traffic between your NFS server and NFS client(s) is subject to port filtering +If the traffic between your NFS server and NFS clients is subject to port filtering by a firewall, then you need to reconfigure that firewall to allow NFS communication. [This guide from The Linux Documentation Project (TDLP)](https://tldp.org/HOWTO/NFS-HOWTO/security.html#FIREWALLS) @@ -482,3 +482,10 @@ On Ubuntu 16.04, use: ```shell sudo perf trace --no-syscalls --event 'nfs4:*' -p $(pgrep -fd ',' puma) ``` + +### Warnings `garbage found: .../repositories/@hashed/...git/objects/pack/.nfs...` in Gitaly logs + +If you find any warnings like `garbage found: .../repositories/@hashed/...git/objects/pack/.nfs...` in [Gitaly logs](logs.md#gitaly-logs), +this problem occurs if `lookupcache=positive` is not set, which we recommend as an +[NFS mount option](#mount-options). +See [Gitaly issue #3175](https://gitlab.com/gitlab-org/gitaly/-/issues/3175) for more details. diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md index 6ea433f95e9..0560a8813df 100644 --- a/doc/administration/object_storage.md +++ b/doc/administration/object_storage.md @@ -19,7 +19,7 @@ GitLab has been tested by vendors and customers on a number of object storage pr - [Google Cloud Storage](https://cloud.google.com/storage) - [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces) - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) -- [OpenStack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) +- [OpenStack Swift (S3 compatible mode)](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) - On-premises hardware and appliances from various storage vendors, whose list is not officially established. - MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. @@ -62,7 +62,7 @@ Using the consolidated object storage configuration has a number of advantages: - It enables the use of [encrypted S3 buckets](#encrypted-s3-buckets). - It [uploads files to S3 with proper `Content-MD5` headers](https://gitlab.com/gitlab-org/gitlab-workhorse/-/issues/222). -Because [direct upload mode](../development/uploads/implementation.md#direct-upload) +Because [direct upload mode](../development/uploads/index.md#direct-upload) must be enabled, only the following providers can be used: - [Amazon S3-compatible providers](#s3-compatible-connection-settings) @@ -386,52 +386,6 @@ If you are using a custom Azure storage domain, configuration. This information is exchanged in an API call between GitLab Rails and Workhorse. -#### OpenStack-compatible connection settings - -Although OpenStack Swift provides S3 compatibility, some users may want to use -the [Swift API](https://docs.openstack.org/swift/latest/api/object_api_v1_overview.html). - -This isn't compatible with the consolidated object storage form. OpenStack Swift -is supported only with the storage-specific form. If you want to use the -consolidated form, see the [S3 settings](#s3-compatible-connection-settings). - -Here are the valid connection settings for the Swift API, provided by -[fog-openstack](https://github.com/fog/fog-openstack): - -| Setting | Description | Default | -|--------------------------|----------------------|---------| -| `provider` | Always `OpenStack` for compatible hosts. | `OpenStack` | -| `openstack_username` | OpenStack username. | | -| `openstack_api_key` | OpenStack API key. | | -| `openstack_temp_url_key` | OpenStack key for generating temporary URLs | | -| `openstack_auth_url` | OpenStack authentication endpoint | | -| `openstack_region` | OpenStack region. | | -| `openstack_tenant` | OpenStack tenant ID. | | - -#### Rackspace Cloud Files - -The following table describes the valid connection parameters for -Rackspace Cloud, provided by [fog-rackspace](https://github.com/fog/fog-rackspace/). - -This isn't compatible with the consolidated object storage form. -Rackspace Cloud is supported only with the storage-specific form. - -| Setting | Description | Example | -|--------------------------|----------------|-------------| -| `provider` | Provider name. | `Rackspace` | -| `rackspace_username` | Username of the Rackspace account with access to the container. | `joe.smith` | -| `rackspace_api_key` | API key of the Rackspace account with access to the container. | `ABC123DEF456ABC123DEF456ABC123DE` | -| `rackspace_region` | Rackspace storage region to use, a three letter code from the [list of service access endpoints](https://docs.rackspace.com/docs/cloud-files/v1/general-api-info/service-access/). | `iad` | -| `rackspace_temp_url_key` | Private key you set in the Rackspace API for [temporary URLs](https://docs.rackspace.com/docs/cloud-files/v1/use-cases/public-access-to-your-cloud-files-account/#tempurl). | `ABC123DEF456ABC123DEF456ABC123DE` | - -Regardless of whether the container has public access enabled or disabled, Fog -uses the TempURL method to grant access to LFS objects. If you see error -messages in logs that refer to instantiating storage with a `temp-url-key`, -be sure you have set the key properly both in the Rackspace API and in -`gitlab.rb`. You can verify the value of the key Rackspace has set by sending a -GET request with token header to the service access endpoint URL and comparing -the output of the returned headers. - ### Object-specific configuration The following YAML shows how the `object_store` section defines @@ -582,7 +536,6 @@ supported by consolidated configuration form, refer to the following guides: | [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| **{dotted-circle}** No | | [Packages](packages/index.md#using-object-storage) (optional feature) | **{check-circle}** Yes | | [Dependency Proxy](packages/dependency_proxy.md#using-object-storage) (optional feature) | **{check-circle}** Yes | -| [Pseudonymizer](pseudonymizer.md) (optional feature) | **{dotted-circle}** No | | [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | **{dotted-circle}** No | | [Terraform state files](terraform_state.md#using-object-storage) | **{check-circle}** Yes | | [Pages content](pages/index.md#using-object-storage) | **{check-circle}** Yes | @@ -616,7 +569,7 @@ There are plans to [enable the use of a single bucket](https://gitlab.com/gitlab in the future. Helm-based installs require separate buckets to -[handle backup restorations](https://docs.gitlab.com/charts/advanced/external-object-storage/#lfs-artifacts-uploads-packages-external-diffs-pseudonymizer). +[handle backup restorations](https://docs.gitlab.com/charts/advanced/external-object-storage/#lfs-artifacts-uploads-packages-external-diffs-terraform-state-dependency-proxy). ### S3 API compatibility issues @@ -799,3 +752,46 @@ to run the following command: ```ruby Feature.disable(:s3_multithreaded_uploads) ``` + +## Migrate objects to a different object storage provider + +You may need to migrate GitLab data in object storage to a different object storage provider. The following steps show you how do this using [Rclone](https://rclone.org/). + +The steps assume you are moving the `uploads` bucket, but the same process works for other buckets. + +Prerequisites: + +- Choose the computer to run Rclone on. Depending on how much data you are migrating, Rclone may have to run for a long time so you should avoid using a laptop or desktop computer that can go into power saving. You can use your GitLab server to run Rclone. + +1. [Install](https://rclone.org/downloads/) Rclone. +1. Configure Rclone by running the following: + + ```shell + rclone config + ``` + + The configuration process is interactive. Add at least two "remotes": one for the object storage provider your data is currently on (`old`), and one for the provider you are moving to (`new`). + +1. Verify that you can read the old data. The following example refers to the `uploads` bucket , but your bucket may have a different name: + + ```shell + rclone ls old:uploads | head + ``` + + This should print a partial list of the objects currently stored in your `uploads` bucket. If you get an error, or if + the list is empty, go back and update your Rclone configuration using `rclone config`. + +1. Perform an initial copy. You do not need to take your GitLab server offline for this step. + + ```shell + rclone sync -P old:uploads new:uploads + ``` + +1. After the first sync completes, use the web UI or command-line interface of your new object storage provider to + verify that there are objects in the new bucket. If there are none, or if you encounter an error while running `rclone + sync`, check your Rclone configuration and try again. + +After you have done at least one successful Rclone copy from the old location to the new location, schedule maintenance and take your GitLab server offline. During your maintenance window you must do two things: + +1. Perform a final `rclone sync` run, knowing that your users cannot add new objects so you will not leave any behind in the old bucket. +1. Update the object storage configuration of your GitLab server to use the new provider for `uploads`. diff --git a/doc/administration/operations/extra_sidekiq_processes.md b/doc/administration/operations/extra_sidekiq_processes.md index fd04efe8473..75f2ad5ed26 100644 --- a/doc/administration/operations/extra_sidekiq_processes.md +++ b/doc/administration/operations/extra_sidekiq_processes.md @@ -354,7 +354,7 @@ file is written, but this can be changed by passing the `--pidfile` option to ``` Keep in mind that the PID file contains the PID of the `sidekiq-cluster` -command and not the PID(s) of the started Sidekiq processes. +command and not the PIDs of the started Sidekiq processes. ### Environment diff --git a/doc/administration/operations/extra_sidekiq_routing.md b/doc/administration/operations/extra_sidekiq_routing.md index bb8eb184302..cd3a53b7c63 100644 --- a/doc/administration/operations/extra_sidekiq_routing.md +++ b/doc/administration/operations/extra_sidekiq_routing.md @@ -88,8 +88,8 @@ components: > [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/261) in GitLab 13.1 (`tags`). -Queue matching query works upon the worker attributes, described in [Sidekiq -style guide](../../development/sidekiq_style_guide.md). We support querying +Queue matching query works upon the worker attributes, described in +[Sidekiq style guide](../../development/sidekiq/index.md). We support querying based on a subset of worker attributes: - `feature_category` - the [GitLab feature diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md index 477a11dd899..a3240a6041b 100644 --- a/doc/administration/operations/fast_ssh_key_lookup.md +++ b/doc/administration/operations/fast_ssh_key_lookup.md @@ -1,6 +1,6 @@ --- -stage: Enablement -group: Distribution +stage: Create +group: Source Code info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -14,7 +14,7 @@ drop-in replacement. Regular SSH operations become slow as the number of users grows because OpenSSH searches for a key to authorize a user via a linear search. In the worst case, -such as when the user is not authorized to access GitLab, OpenSSH will scan the +such as when the user is not authorized to access GitLab, OpenSSH scans the entire file to search for a key. This can take significant time and disk I/O, which delays users attempting to push or pull to a repository. Making matters worse, if users add or remove keys frequently, the operating system may @@ -33,21 +33,22 @@ able to accept a fingerprint. Check the version of OpenSSH on your server with ` Unlike [Cloud Native GitLab](https://docs.gitlab.com/charts/), Omnibus GitLab by default manages an `authorized_keys` file that is located in the -`git` user's home directory. For most installations, this will be located under -`/var/opt/gitlab/.ssh/authorized_keys`, but you can use the following command to locate the `authorized_keys` on your system: +`git` user's home directory. For most installations, this file is located under +`/var/opt/gitlab/.ssh/authorized_keys`, but you can use the following command to +locate the `authorized_keys` on your system: ```shell getent passwd git | cut -d: -f6 | awk '{print $1"/.ssh/authorized_keys"}' ``` The `authorized_keys` file contains all the public SSH keys for users allowed to access GitLab. However, to maintain a -single source of truth, [Geo](../geo/index.md) needs to be configured to perform SSH fingerprint +single source of truth, [Geo](../geo/index.md) must be configured to perform SSH fingerprint lookups via database lookup. As part of [setting up Geo](../geo/index.md#setup-instructions), you are required to follow the steps outlined below for both the primary and -secondary nodes, but the `Write to "authorized keys" file` checkbox -only needs to be unchecked on the primary node since it is reflected +secondary nodes, but **Write to "authorized keys" file** +must be unchecked only on the primary node, because it is reflected automatically on the secondary if database replication is working. ## Setting up fast lookup via GitLab Shell @@ -56,8 +57,8 @@ GitLab Shell provides a way to authorize SSH users via a fast, indexed lookup to the GitLab database. GitLab Shell uses the fingerprint of the SSH key to check whether the user is authorized to access GitLab. -Add the following to your `sshd_config` file. This is usually located at -`/etc/ssh/sshd_config`, but it will be `/assets/sshd_config` if you're using +Add the following to your `sshd_config` file. This file is usually located at +`/etc/ssh/sshd_config`, but it is at `/assets/sshd_config` if you're using Omnibus Docker: ```plaintext @@ -84,17 +85,14 @@ file (start the line with a `#` to comment it), and from your local machine, att ssh -T git@gitlab.example.com ``` -A successful pull or [welcome message](../../user/ssh.md#verify-that-you-can-connect) would mean that GitLab was able to find the key in the database, -since it is not present in the file anymore. - -NOTE: -For Omnibus Docker, `AuthorizedKeysCommand` is setup by default in -GitLab 11.11 and later. +A successful pull or [welcome message](../../user/ssh.md#verify-that-you-can-connect) +means that GitLab was able to find the key in the database, +as it is not present in the file. NOTE: For Installations from source, the command would be located at `/home/git/gitlab-shell/bin/gitlab-shell-authorized-keys-check` if [the install from source](../../install/installation.md#install-gitlab-shell) instructions were followed. -You might want to consider creating a wrapper script somewhere else since this command needs to be +You might want to consider creating a wrapper script somewhere else, as this command must be owned by `root` and not be writable by group or others. You could also consider changing the ownership of this command as required, but that might require temporary ownership changes during `gitlab-shell` upgrades. @@ -123,7 +121,7 @@ or for users to re-add their keys. ## How to go back to using the `authorized_keys` file -This is a brief overview. Please refer to the above instructions for more context. +This overview is brief. Refer to the above instructions for more context. 1. [Rebuild the `authorized_keys` file](../raketasks/maintenance.md#rebuild-authorized_keys-file). 1. Enable writes to the `authorized_keys` file. @@ -136,27 +134,35 @@ This is a brief overview. Please refer to the above instructions for more contex ## Use `gitlab-sshd` instead of OpenSSH -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/299109) in GitLab 14.5. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/299109) in GitLab 14.5 as an **Alpha** release for self-managed customers. WARNING: `gitlab-sshd` is in [**Alpha**](../../policy/alpha-beta-support.md#alpha-features). It is not ready for production use. `gitlab-sshd` is [a standalone SSH server](https://gitlab.com/gitlab-org/gitlab-shell/-/tree/main/internal/sshd) - written in Go. It is provided as a part of `gitlab-shell` package. It has a lower memory - use as a OpenSSH alternative and supports - [group access restriction by IP address](../../user/group/index.md) for applications - running behind the proxy. +written in Go. It is provided as a part of the `gitlab-shell` package. It has a lower memory +use as a OpenSSH alternative, and supports +[group access restriction by IP address](../../user/group/index.md) for applications +running behind the proxy. + +`gitlab-sshd` is a lightweight alternative to OpenSSH for providing +[SSH operations](https://gitlab.com/gitlab-org/gitlab-shell/-/blob/71a7f34a476f778e62f8fe7a453d632d395eaf8f/doc/features.md). +While OpenSSH uses a restricted shell approach, `gitlab-sshd` behaves more like a +modern multi-threaded server application, responding to incoming requests. The major +difference is that OpenSSH uses SSH as a transport protocol while `gitlab-sshd` uses Remote Procedure Calls (RPCs). + +The capabilities of GitLab Shell are not limited to Git operations. If you are considering switching from OpenSSH to `gitlab-sshd`, consider these concerns: - The `gitlab-sshd` component is only available for [Cloud Native Helm Charts](https://docs.gitlab.com/charts/) deployments. - `gitlab-sshd` supports the PROXY protocol. It can run behind proxy servers that rely - on it, such as HAProxy. -- `gitlab-sshd` does not share a SSH port with the system administrator's OpenSSH, - and requires a bind to port 22. -- `gitlab-sshd` **does not** support SSH certificates. + on it, such as HAProxy. The PROXY protocol not enabled by default, but can be enabled with a Helm chart setting. +- By default, `gitlab-sshd` binds to port 22, but you can configure a different port in the Helm chart. +- `gitlab-sshd` **does not** support SSH certificates. For more details, read + [issue #495](https://gitlab.com/gitlab-org/gitlab-shell/-/issues/495). To switch from OpenSSH to `gitlab-sshd`: @@ -178,7 +184,11 @@ GitLab supports `authorized_keys` database lookups with [SELinux](https://en.wik Because the SELinux policy is static, GitLab doesn't support the ability to change internal webserver ports at the moment. Administrators would have to create a special `.te` -file for the environment, since it isn't generated dynamically. +file for the environment, as it isn't generated dynamically. + +### Additional documentation + +Additional technical documentation for `gitlab-sshd` may be found on the [GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell/-/blob/main/README.md) documentation page. ## Troubleshooting @@ -187,7 +197,8 @@ or causing high CPU load, be sure to check the size of `/var/log/btmp`, and ensu If this file is very large, GitLab SSH fast lookup can cause the bottleneck to be hit more frequently, thus decreasing performance even further. If you are able to, you may consider disabling [`UsePAM` in your `sshd_config`](https://linux.die.net/man/5/sshd_config) to avoid reading `/var/log/btmp` altogether. -Running `strace` and `lsof` on a running `sshd: git` process can return useful debugging information. To get an `strace` on an in-progress Git over SSH connection for IP `x.x.x.x`, run: +Running `strace` and `lsof` on a running `sshd: git` process returns debugging information. +To get an `strace` on an in-progress Git over SSH connection for IP `x.x.x.x`, run: ```plaintext sudo strace -s 10000 -p $(sudo netstat -tp | grep x.x.x.x | egrep 'ssh.*: git' | sed -e 's/.*ESTABLISHED *//' -e 's#/.*##') diff --git a/doc/administration/operations/puma.md b/doc/administration/operations/puma.md index c12f75989c3..12a8b2faadc 100644 --- a/doc/administration/operations/puma.md +++ b/doc/administration/operations/puma.md @@ -179,6 +179,46 @@ optimal configuration: - To force Rugged to be used with multi-threaded Puma, you can use a [feature flag](../../development/gitaly.md#legacy-rugged-code). +## Configuring Puma to listen over SSL + +Puma, when deployed with Omnibus GitLab, listens over a Unix socket by +default. To configure Puma to listen over an HTTPS port instead, follow the +steps below: + +1. Generate an SSL certificate key-pair for the address where Puma will + listen. For the example below, this is `127.0.0.1`. + + NOTE: + If using a self-signed certificate from a custom Certificate Authority (CA), + follow [the documentation](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates) + to make them trusted by other GitLab components. + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + puma['ssl_listen'] = '127.0.0.1' + puma['ssl_port'] = 9111 + puma['ssl_certificate'] = '<path_to_certificate>' + puma['ssl_certificate_key'] = '<path_to_key>' + + # Disable UNIX socket + puma['socket'] = "" + ``` + +1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +NOTE: +In addition to the Unix socket, Puma also listens over HTTP on port 8080 for +providing metrics to be scraped by Prometheus. It is not currently possible to +make Prometheus scrape them over HTTPS, and support for it is being discussed +[in this issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6811). +Hence, it is not technically possible to turn off this HTTP listener without +losing Prometheus metrics. + ## Switch from Unicorn to Puma NOTE: diff --git a/doc/administration/package_information/licensing.md b/doc/administration/package_information/licensing.md index a7bf5c52d7b..d27c1df0ccf 100644 --- a/doc/administration/package_information/licensing.md +++ b/doc/administration/package_information/licensing.md @@ -66,7 +66,7 @@ This software is based in part on the work of the Independent JPEG Group. ## Trademark Usage -Within the GitLab documentation, reference to third party technology(ies) and/or trademarks of third party entities, may be made. The inclusion of reference to third party technology and/or entities is solely for the purposes of example(s) of how GitLab software may interact with, or be used in conjunction with, such third party technology. +Within the GitLab documentation, reference to third-party technologies and/or trademarks of third-party entities may be made. The inclusion of reference to third-party technology and/or entities is solely for the purposes of examples of how GitLab software may interact with, or be used in conjunction with, such third-party technology. All trademarks, materials, documentation, and other intellectual property remain the property of any/all such third party. ### Trademark Requirements diff --git a/doc/administration/package_information/postgresql_versions.md b/doc/administration/package_information/postgresql_versions.md index c80437221c4..97f93663003 100644 --- a/doc/administration/package_information/postgresql_versions.md +++ b/doc/administration/package_information/postgresql_versions.md @@ -26,6 +26,7 @@ Read more about update policies and warnings in the PostgreSQL | GitLab version | PostgreSQL versions | Default version for fresh installs | Default version for upgrades | Notes | | -------------- | --------------------- | ---------------------------------- | ---------------------------- | ----- | +| 15.0 | 12.10, 13.6 | 13.6 | 13.6 | Users can manually upgrade to 13.6 following the [upgrade docs](https://docs.gitlab.com/omnibus/settings/database.html#gitlab-150-and-later). | | 14.1 | 12.7, 13.3 | 12.7 | 12.7 | PostgreSQL 13 available for fresh installations if not using [Geo](../geo/index.md#requirements-for-running-geo) or [Patroni](../postgresql/index.md#postgresql-replication-and-failover-with-omnibus-gitlab). | 14.0 | 12.7 | 12.7 | 12.7 | HA installations with repmgr are no longer supported and will be prevented from upgrading to Omnibus GitLab 14.0 | | 13.8 | 11.9, 12.4 | 12.4 | 12.4 | Package upgrades automatically performed PostgreSQL upgrade for nodes that are not part of a Geo or HA cluster.). | diff --git a/doc/administration/package_information/signed_packages.md b/doc/administration/package_information/signed_packages.md index d7bcfa113ff..24857dcfc27 100644 --- a/doc/administration/package_information/signed_packages.md +++ b/doc/administration/package_information/signed_packages.md @@ -14,7 +14,7 @@ These packages are produced by the GitLab CI process, as found in the [Omnibus ## GnuPG Public Keys -All packages are signed with [GnuPG](https://www.gnupg.org/), in a method appropriate for their format. The key used to sign these packages can be found on [pgp.mit.edu](https://pgp.mit.edu) at [0x3cfcf9baf27eab47](https://pgp.mit.edu/pks/lookup?op=vindex&search=0x3CFCF9BAF27EAB47) +All packages are signed with [GnuPG](https://www.gnupg.org/), in a method appropriate for their format. The key used to sign these packages can be found on [MIT PGP Public Key Server](https://pgp.mit.edu) at [0x3cfcf9baf27eab47](https://pgp.mit.edu/pks/lookup?op=vindex&search=0x3CFCF9BAF27EAB47) ## Verifying Signatures diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md index d6f5588ebc9..0edad83cc18 100644 --- a/doc/administration/packages/container_registry.md +++ b/doc/administration/packages/container_registry.md @@ -158,7 +158,7 @@ If your certificate provider provides the CA Bundle certificates, append them to An administrator may want the container registry listening on an arbitrary port such as `5678`. However, the registry and application server are behind an AWS application load balancer that only -listens on ports `80` and `443`. The admin may simply remove the port number for +listens on ports `80` and `443`. The administrator may simply remove the port number for `registry_external_url`, so HTTP or HTTPS is assumed. Then, the rules apply that map the load balancer to the registry from ports `80` or `443` to the arbitrary port. This is important if users rely on the `docker login` example in the container registry. Here's an example: @@ -1244,9 +1244,9 @@ unauthorized: authentication required GitLab has a default token expiration of 5 minutes for the registry. When pushing larger images, or images that take longer than 5 minutes to push, users may -encounter this error. +encounter this error. On GitLab.com, the expiration time is 15 minutes. -Administrators can increase the token duration in **Admin area > Settings > +Administrators can increase the token duration in **Admin Area > Settings > CI/CD > Container Registry > Authorization token duration (minutes)**. ### Docker login attempt fails with: 'token signed by untrusted key' diff --git a/doc/administration/packages/dependency_proxy.md b/doc/administration/packages/dependency_proxy.md index 4c50cfeeb0f..40c1b9d795a 100644 --- a/doc/administration/packages/dependency_proxy.md +++ b/doc/administration/packages/dependency_proxy.md @@ -140,8 +140,6 @@ This section describes the earlier configuration format. gitlab_rails['dependency_proxy_storage_path'] = "/var/opt/gitlab/gitlab-rails/shared/dependency_proxy" gitlab_rails['dependency_proxy_object_store_enabled'] = true gitlab_rails['dependency_proxy_object_store_remote_directory'] = "dependency_proxy" # The bucket name. - gitlab_rails['dependency_proxy_object_store_direct_upload'] = false # Use Object Storage directly for uploads instead of background uploads if enabled (Default: false). - gitlab_rails['dependency_proxy_object_store_background_upload'] = true # Temporary option to limit automatic upload (Default: true). gitlab_rails['dependency_proxy_object_store_proxy_download'] = false # Passthrough all downloads via GitLab instead of using Redirects to Object Storage. gitlab_rails['dependency_proxy_object_store_connection'] = { ## @@ -177,8 +175,6 @@ This section describes the earlier configuration format. object_store: enabled: false remote_directory: dependency_proxy # The bucket name. - # direct_upload: false # Use Object Storage directly for uploads instead of background uploads if enabled (Default: false). - # background_upload: true # Temporary option to limit automatic upload (Default: true). # proxy_download: false # Passthrough all downloads via GitLab instead of using Redirects to Object Storage. connection: ## @@ -248,23 +244,6 @@ Verify that there are no files on disk in the `dependency_proxy` folder: sudo find /var/opt/gitlab/gitlab-rails/shared/dependency_proxy -type f | grep -v tmp | wc -l ``` -## Disabling Authentication - -Authentication was introduced in 13.7 as part of [enabling private groups to use the -Dependency Proxy](https://gitlab.com/gitlab-org/gitlab/-/issues/11582). If you -previously used the Dependency Proxy without authentication and need to disable -this feature while you update your workflow to [authenticate with the Dependency -Proxy](../../user/packages/dependency_proxy/index.md#authenticate-with-the-dependency-proxy), -the following commands can be issued in a Rails console: - -```ruby -# Disable the authentication -Feature.disable(:dependency_proxy_for_private_groups) - -# Re-enable the authentication -Feature.enable(:dependency_proxy_for_private_groups) -``` - ## Changing the JWT expiration The Dependency Proxy follows the [Docker v2 token authentication flow](https://docs.docker.com/registry/spec/auth/token/), @@ -280,7 +259,7 @@ ApplicationSetting.update(container_registry_token_expire_delay: 30) The default expiration and the expiration on GitLab.com is 15 minutes. ## Using the dependency proxy behind a proxy - + 1. Edit `/etc/gitlab/gitlab.rb` and add the following lines: ```ruby diff --git a/doc/administration/packages/index.md b/doc/administration/packages/index.md index b122cb9db90..ef00127a70e 100644 --- a/doc/administration/packages/index.md +++ b/doc/administration/packages/index.md @@ -152,8 +152,6 @@ We recommend using the [consolidated object storage settings](../object_storage. gitlab_rails['packages_enabled'] = true gitlab_rails['packages_object_store_enabled'] = true gitlab_rails['packages_object_store_remote_directory'] = "packages" # The bucket name. - gitlab_rails['packages_object_store_direct_upload'] = false # Use Object Storage directly for uploads instead of background uploads if enabled (Default: false). - gitlab_rails['packages_object_store_background_upload'] = true # Temporary option to limit automatic upload (Default: true). gitlab_rails['packages_object_store_proxy_download'] = false # Passthrough all downloads via GitLab instead of using Redirects to Object Storage. gitlab_rails['packages_object_store_connection'] = { ## @@ -192,8 +190,6 @@ We recommend using the [consolidated object storage settings](../object_storage. object_store: enabled: false remote_directory: packages # The bucket name. - # direct_upload: false # Use Object Storage directly for uploads instead of background uploads if enabled (Default: false). - # background_upload: true # Temporary option to limit automatic upload (Default: true). # proxy_download: false # Passthrough all downloads via GitLab instead of using Redirects to Object Storage. connection: ## diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index 114894c01f0..f144c60fcfe 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -97,10 +97,9 @@ IPv6 address. If you don't have IPv6, you can omit the `AAAA` record. #### DNS configuration for custom domains -If support for custom domains is needed, the Pages root domain and its subdomains should point to -the secondary IP (which is dedicated for the Pages daemon). `<namespace>.<pages root domain>` should -point at Pages directly. Without this, users aren't able to use `CNAME` records to point their -custom domains to their GitLab Pages. +If support for custom domains is needed, all subdomains of the Pages root domain should point to the +secondary IP (which is dedicated for the Pages daemon). Without this configuration, users can't use +`CNAME` records to point their custom domains to their GitLab Pages. For example, an entry could look like this: @@ -295,7 +294,7 @@ control over how the Pages daemon runs and serves content in your environment. | `rate_limit_domain_burst` | Rate limit per domain maximum burst allowed per second. | | `server_read_timeout` | Maximum duration to read the request headers and body. For no timeout, set to `0` or a negative value. Default: `5s` | | `server_read_header_timeout` | Maximum duration to read the request headers. For no timeout, set to `0` or a negative value. Default: `1s` | -| `server_write_timeout` | Maximum duration to write all files in the response. Larger files require more time. For no timeout, set to `0` or a negative value. Default: `5m` | +| `server_write_timeout` | Maximum duration to write all files in the response. Larger files require more time. For no timeout, set to `0` or a negative value. Default: `0` | | `server_keep_alive` | The `Keep-Alive` period for network connections accepted by this listener. If `0`, `Keep-Alive` is enabled if supported by the protocol and operating system. If negative, `Keep-Alive` is disabled. Default: `15s` | ## Advanced configuration @@ -387,6 +386,12 @@ GitLab supports [custom domain verification](../../user/project/pages/custom_dom When adding a custom domain, users are required to prove they own it by adding a GitLab-controlled verification code to the DNS records for that domain. +WARNING: +Disabling domain verification is unsafe and can lead to various vulnerabilities. +If you *do* disable it, either ensure that the Pages root domain itself does not point to the +secondary IP or add the root domain as custom domain to a project; otherwise, any user can add this +domain as a custom domain to their project. + If your user base is private or otherwise trusted, you can disable the verification requirement: @@ -1135,7 +1140,7 @@ Rate limits are enforced using the following: - `rate_limit_source_ip`: Set the maximum threshold in number of requests per client IP per second. Set to 0 to disable this feature. - `rate_limit_source_ip_burst`: Sets the maximum threshold of number of requests allowed in an initial outburst of requests per client IP. For example, when you load a web page that loads a number of resources at the same time. -- `rate_limit_domain_ip`: Set the maximum threshold in number of requests per hosted pages domain per second. Set to 0 to disable this feature. +- `rate_limit_domain`: Set the maximum threshold in number of requests per hosted pages domain per second. Set to 0 to disable this feature. - `rate_limit_domain_burst`: Sets the maximum threshold of number of requests allowed in an initial outburst of requests per hosted pages domain. #### Enable source-IP rate limits @@ -1339,6 +1344,27 @@ To stop `systemd` from cleaning the Pages related content: sudo gitlab-ctl restart gitlab-pages ``` +### Unable to access GitLab Pages + +If you can't access your GitLab Pages (such as receiving `502 Bad Gateway` errors, or a login loop) +and in your Pages log shows this error: + +```plaintext +"error":"retrieval context done: context deadline exceeded","host":"root.docs-cit.otenet.gr","level":"error","msg":"could not fetch domain information from a source" +``` + +1. Add the following to `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_pages['internal_gitlab_server'] = 'http://localhost:8080' + ``` + +1. Restart GitLab Pages: + + ```shell + sudo gitlab-ctl restart gitlab-pages + ``` + ### 404 error after promoting a Geo secondary to a primary node Pages files are not among the diff --git a/doc/administration/postgresql/pgbouncer.md b/doc/administration/postgresql/pgbouncer.md index a666c1fab95..ed3c662eba3 100644 --- a/doc/administration/postgresql/pgbouncer.md +++ b/doc/administration/postgresql/pgbouncer.md @@ -219,12 +219,12 @@ the database. Each of the listed services below use the following formula to def - `puma` : `max_threads + headroom` (default `14`) - `max_threads` is configured via: `gitlab['puma']['max_threads']` (default: `4`) - - `headroom` can be configured via `DB_POOL_HEADROOM` env variable (default to `10`) + - `headroom` can be configured via `DB_POOL_HEADROOM` environment variable (default to `10`) - `sidekiq` : `max_concurrency + 1 + headroom` (default: `61`) - `max_concurrency` is configured via: `sidekiq['max_concurrency']` (default: `50`) - - `headroom` can be configured via `DB_POOL_HEADROOM` env variable (default to `10`) + - `headroom` can be configured via `DB_POOL_HEADROOM` environment variable (default to `10`) - `geo-logcursor`: `1+headroom` (default: `11`) - - `headroom` can be configured via `DB_POOL_HEADROOM` env variable (default to `10`) + - `headroom` can be configured via `DB_POOL_HEADROOM` environment variable (default to `10`) To calculate the `default_pool_size`, multiply the number of instances of `puma`, `sidekiq` and `geo-logcursor` by the number of connections each can consume as per listed above. The total will be the suggested `default_pool_size`. diff --git a/doc/administration/postgresql/replication_and_failover.md b/doc/administration/postgresql/replication_and_failover.md index 8c7151606a5..84122149cb8 100644 --- a/doc/administration/postgresql/replication_and_failover.md +++ b/doc/administration/postgresql/replication_and_failover.md @@ -1123,25 +1123,36 @@ postgresql['trust_auth_cidr_addresses'] = %w(123.123.123.123/32 <other_cidrs>) ### Reinitialize a replica -If replication is not occurring, it may be necessary to reinitialize a replica. +If a replica cannot start or rejoin the cluster, or when it lags behind and can not catch up, it might be necessary to reinitialize the replica: -1. On any server in the cluster, determine the Cluster and Member names, - and check the replication lag by running `gitlab-ctl patroni members`. Here is an example: +1. [Check the replication status](#check-replication-status) to confirm which server + needs to be reinitialized. For example: ```plaintext - + Cluster: postgresql-ha (6970678148837286213) ------+---------+---------+----+-----------+ - | Member | Host | Role | State | TL | Lag in MB | - +-------------------------------------+--------------+---------+---------+----+-----------+ - | gitlab-database-1.example.com | 172.18.0.111 | Replica | running | 5 | 0 | - | gitlab-database-2.example.com | 172.18.0.112 | Replica | running | 5 | 100 | - | gitlab-database-3.example.com | 172.18.0.113 | Leader | running | 5 | | - +-------------------------------------+--------------+---------+---------+----+-----------+ + + Cluster: postgresql-ha (6970678148837286213) ------+---------+--------------+----+-----------+ + | Member | Host | Role | State | TL | Lag in MB | + +-------------------------------------+--------------+---------+--------------+----+-----------+ + | gitlab-database-1.example.com | 172.18.0.111 | Replica | running | 55 | 0 | + | gitlab-database-2.example.com | 172.18.0.112 | Replica | start failed | | unknown | + | gitlab-database-3.example.com | 172.18.0.113 | Leader | running | 55 | | + +-------------------------------------+--------------+---------+--------------+----+-----------+ ``` -1. Reinitialize the affected replica server: +1. Sign in to the broken server and reinitialize the database and replication. Patroni will shut + down PostgreSQL on that server, remove the data directory, and reinitialize it from scratch: - ```plaintext - gitlab-ctl patroni reinitialize-replica postgresql-ha gitlab-database-2.example.com + ```shell + sudo gitlab-ctl patroni reinitialize-replica --member gitlab-database-2.example.com + ``` + + This can be run on any Patroni node, but be aware that `sudo gitlab-ctl patroni + reinitialize-replica` without `--member` will reinitialize the server it is run on. + It is recommended to run it locally on the broken server to reduce the risk of + unintended data loss. +1. Monitor the logs: + + ```shell + sudo gitlab-ctl tail patroni ``` ### Reset the Patroni state in Consul diff --git a/doc/administration/pseudonymizer.md b/doc/administration/pseudonymizer.md index 24d9792dcb0..ad4cfd11474 100644 --- a/doc/administration/pseudonymizer.md +++ b/doc/administration/pseudonymizer.md @@ -2,127 +2,14 @@ stage: Enablement group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +remove_date: '2022-08-22' +redirect_to: 'index.md' --- -# Pseudonymizer (DEPRECATED) **(ULTIMATE)** +# Pseudonymizer (removed) **(ULTIMATE)** -> [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/219952) in GitLab 14.7. +> [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/219952) in +> GitLab 14.7 and removed in 15.0. WARNING: This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/219952) in GitLab 14.7. - -Your GitLab database contains sensitive information. To protect sensitive information -when you run analytics on your database, you can use the Pseudonymizer service, which: - -1. Uses `HMAC(SHA256)` to mutate fields containing sensitive information. -1. Preserves references (referential integrity) between fields. -1. Exports your GitLab data, scrubbed of sensitive material. - -WARNING: -If the source data is available, users can compare and correlate the scrubbed data -with the original. - -To generate a pseudonymized data set: - -1. [Configure Pseudonymizer](#configure-pseudonymizer) fields and output location. -1. [Enable Pseudonymizer data collection](#enable-pseudonymizer-data-collection). -1. Optional. [Generate a data set manually](#generate-data-set-manually). - -## Configure Pseudonymizer - -To use the Pseudonymizer, configure both the fields you want to anonymize, and the location to -store the scrubbed data: - -1. **Create a manifest file**: This file describes the fields to include or pseudonymize. - - **Default manifest** - GitLab provides a default manifest in your GitLab installation - ([example `manifest.yml` file](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/pseudonymizer.yml)). - To use the example manifest file, use the `config/pseudonymizer.yml` relative path - when you configure connection parameters. - - **Custom manifest** - To use a custom manifest file, use the absolute path to - the file when you configure the connection parameters. -1. **Configure connection parameters**: In the configuration method appropriate for - your version of GitLab, specify the [object storage](object_storage.md) - connection parameters (`pseudonymizer.upload.connection`). - -**For Omnibus installations:** - -1. Edit `/etc/gitlab/gitlab.rb` and add the following lines by replacing with - the values you want: - - ```ruby - gitlab_rails['pseudonymizer_manifest'] = 'config/pseudonymizer.yml' - gitlab_rails['pseudonymizer_upload_remote_directory'] = 'gitlab-elt' # bucket name - gitlab_rails['pseudonymizer_upload_connection'] = { - 'provider' => 'AWS', - 'region' => 'eu-central-1', - 'aws_access_key_id' => 'AWS_ACCESS_KEY_ID', - 'aws_secret_access_key' => 'AWS_SECRET_ACCESS_KEY' - } - ``` - - If you are using AWS IAM profiles, omit the AWS access key and secret access key/value pairs. - - ```ruby - gitlab_rails['pseudonymizer_upload_connection'] = { - 'provider' => 'AWS', - 'region' => 'eu-central-1', - 'use_iam_profile' => true - } - ``` - -1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) - for the changes to take effect. - ---- - -**For installations from source:** - -1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following - lines: - - ```yaml - pseudonymizer: - manifest: config/pseudonymizer.yml - upload: - remote_directory: 'gitlab-elt' # bucket name - connection: - provider: AWS - aws_access_key_id: AWS_ACCESS_KEY_ID - aws_secret_access_key: AWS_SECRET_ACCESS_KEY - region: eu-central-1 - ``` - -1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) - for the changes to take effect. - -## Enable Pseudonymizer data collection - -To enable data collection: - -1. On the top bar, select **Menu > Admin**. -1. On the left sidebar, select **Settings > Metrics and Profiling**, then expand - **Pseudonymizer data collection**. -1. Select **Enable Pseudonymizer data collection**. -1. Select **Save changes**. - -## Generate data set manually - -You can also run the Pseudonymizer manually: - -1. Set these environment variables: - - `PSEUDONYMIZER_OUTPUT_DIR` - Where to store the output CSV files. Defaults to `/tmp`. - These commands produce CSV files that can be quite large. Make sure the directory - can store a file at least 10% of the size of your database. - - `PSEUDONYMIZER_BATCH` - The batch size when querying the database. Defaults to `100000`. -1. Run the command appropriate for your application: - - **Omnibus GitLab**: - `sudo gitlab-rake gitlab:db:pseudonymizer` - - **Installations from source**: - `sudo -u git -H bundle exec rake gitlab:db:pseudonymizer RAILS_ENV=production` - -After you run the command, upload the output CSV files to your configured object -storage. After the upload completes, delete the output file from the local disk. - -## Related topics - -- [Using object storage with GitLab](object_storage.md). diff --git a/doc/administration/raketasks/ldap.md b/doc/administration/raketasks/ldap.md index cae2465fad8..cdad323733d 100644 --- a/doc/administration/raketasks/ldap.md +++ b/doc/administration/raketasks/ldap.md @@ -175,7 +175,7 @@ bundle exec rake gitlab:ldap:secret:show RAILS_ENV=production ```plaintext main: password: '123' - user_bn: 'gitlab-adm' + bind_dn: 'gitlab-adm' ``` ### Edit secret diff --git a/doc/administration/raketasks/storage.md b/doc/administration/raketasks/storage.md index 912cf260a03..689ed5c5824 100644 --- a/doc/administration/raketasks/storage.md +++ b/doc/administration/raketasks/storage.md @@ -79,7 +79,7 @@ In GitLab 13.0, [hashed storage](../repository_storage_types.md#hashed-storage) is enabled by default and the legacy storage is deprecated. GitLab 14.0 eliminates support for legacy storage. If you're on GitLab 13.0 and later, switching new projects to legacy storage is not possible. -The option to choose between hashed and legacy storage in the admin area has +The option to choose between hashed and legacy storage in the Admin Area has been disabled. This task must be run on any machine that has Rails/Sidekiq configured, and the task @@ -132,7 +132,7 @@ In GitLab 13.0, [hashed storage](../repository_storage_types.md#hashed-storage) is enabled by default and the legacy storage is deprecated. GitLab 14.0 eliminates support for legacy storage. If you're on GitLab 13.0 and later, switching new projects to legacy storage is not possible. -The option to choose between hashed and legacy storage in the admin area has +The option to choose between hashed and legacy storage in the Admin Area has been disabled. This task schedules all your existing projects and associated attachments to be rolled back to the @@ -169,3 +169,99 @@ Any error or warning is logged in Sidekiq's log file. If you have a Geo setup, the rollback is not reflected automatically on the **secondary** node. You may need to wait for a backfill operation to kick-in and remove the remaining repositories from the special `@hashed/` folder manually. + +## Troubleshooting + +The Rake task might not be able to complete the migration to hashed storage. +Checks on the instance will continue to report that there is legacy data: + +```plaintext +* Found 1 projects using Legacy Storage +- janedoe/testproject (id: 1234) +``` + +If you have a subscription, [raise a ticket with GitLab support](https://support.gitlab.com) +as most of the fixes are relatively high risk, involving running code on the Rails console. + +### Read only projects + +If you have [set projects read only](../troubleshooting/gitlab_rails_cheat_sheet.md#make-a-project-read-only-can-only-be-done-in-the-console) +they might fail to migrate. + +1. [Start a Rails console](../operations/rails_console.md#starting-a-rails-console-session). + +1. Check if the project is read only: + + ```ruby + project = Project.find_by_full_path('janedoe/testproject') + project.repository_read_only + ``` + +1. If it returns `true` (not `nil` or `false`), set it writable: + + ```ruby + project.update!(repository_read_only: false) + ``` + +1. [Re-run the migration Rake task](#migrate-to-hashed-storage). + +1. Set the project read-only again: + + ```ruby + project.update!(repository_read_only: true) + ``` + +### Projects pending deletion + +Check the project details in the Admin Area. If deleting the project failed +it will show as `Marked For Deletion At ..`, `Scheduled Deletion At ..` and +`pending removal`, but the dates will not be recent. + +Delete the project using the Rails console: + +1. [Start a Rails console](../operations/rails_console.md#starting-a-rails-console-session). + +1. With the following code, select the project to be deleted and account to action it: + + ```ruby + project = Project.find_by_full_path('janedoe/testproject') + user = User.find_by_username('admin_handle') + puts "\nproject selected for deletion is:\nID: #{project.id}\nPATH: #{project.full_path}\nNAME: #{project.name}\n\n" + ``` + + - Replace `janedoe/testproject` with your project path from the Rake take output or from the Admin Area. + - Replace `admin_handle` with the handle of an instance administrator or with `root`. + - Verify the output before proceeding. **There are no other checks performed**. + +1. [Destroy the project](../troubleshooting/gitlab_rails_cheat_sheet.md#destroy-a-project) **immediately**: + + ```ruby + Projects::DestroyService.new(project, user).execute + ``` + +If destroying the project generates a stack trace relating to encryption or the error `OpenSSL::Cipher::CipherError`: + +1. [Verify your GitLab secrets](check.md#verify-database-values-can-be-decrypted-using-the-current-secrets). + +1. If the affected projects have secrets that cannot be decrypted it will be necessary to remove those specific secrets. + [Our documentation for dealing with lost secrets](../../raketasks/backup_restore.md#when-the-secrets-file-is-lost) + is for loss of all secrets, but it's possible for specific projects to be affected. For example, + to [reset specific runner registration tokens](../../raketasks/backup_restore.md#reset-runner-registration-tokens) + for a specific project ID: + + ```sql + UPDATE projects SET runners_token = null, runners_token_encrypted = null where id = 1234; + ``` + +### `Repository cannot be moved from` errors in Sidekiq log + +Projects might fail to migrate with errors in the Sidekiq log: + +```shell +# grep 'Repository cannot be moved' /var/log/gitlab/sidekiq/current +{"severity":"ERROR","time":"2021-02-29T02:29:02.021Z","message":"Repository cannot be moved from 'janedoe/testproject' to '@hashed<value>' (PROJECT_ID=1234)"} +``` + +This might be caused by [a bug](https://gitlab.com/gitlab-org/gitlab/-/issues/259605) in the original code for hashed storage migration. + +[There is a workaround for projects still affected by this issue](https://gitlab.com/-/snippets/2039252). diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index 2ca79bbeae4..f70912dbecb 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -39,7 +39,7 @@ full list of reference architectures, see <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> 1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. -2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS ElastiCache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. @@ -1351,11 +1351,18 @@ To configure the Praefect nodes, on each one: on the page. 1. Edit the `/etc/gitlab/gitlab.rb` file to configure Praefect: +<!-- +Updates to example must be made at: +- https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/administration/gitaly/praefect.md +- all reference architecture pages +--> + ```ruby # Avoid running unnecessary services on the Praefect server gitaly['enable'] = false postgresql['enable'] = false redis['enable'] = false + nginx['enable'] = false puma['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false @@ -1364,7 +1371,6 @@ To configure the Praefect nodes, on each one: grafana['enable'] = false gitlab_exporter['enable'] = false gitlab_kas['enable'] = false - nginx['enable'] = false # Praefect Configuration praefect['enable'] = true @@ -1391,8 +1397,8 @@ To configure the Praefect nodes, on each one: praefect['database_host'] = '10.6.0.141' praefect['database_port'] = 5432 # `no_proxy` settings must always be a direct connection for caching - praefect['database_host_no_proxy'] = '10.6.0.141' - praefect['database_port_no_proxy'] = 5432 + praefect['database_direct_host'] = '10.6.0.141' + praefect['database_direct_port'] = 5432 praefect['database_dbname'] = 'praefect_production' praefect['database_user'] = 'praefect' praefect['database_password'] = '<praefect_postgresql_password>' @@ -1495,10 +1501,18 @@ On each node: 1. Edit the Gitaly server node's `/etc/gitlab/gitlab.rb` file to configure storage paths, enable the network listener, and to configure the token: +<!-- +Updates to example must be made at: +- https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab +- https://gitlab.com/gitlab-org/gitlab/blob/master/doc/administration/gitaly/index.md#gitaly-server-configuration +- all reference architecture pages +--> + ```ruby # Avoid running unnecessary services on the Gitaly server postgresql['enable'] = false redis['enable'] = false + nginx['enable'] = false puma['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false @@ -1507,7 +1521,6 @@ On each node: grafana['enable'] = false gitlab_exporter['enable'] = false gitlab_kas['enable'] = false - nginx['enable'] = false # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -1693,11 +1706,18 @@ To configure the Sidekiq nodes, on each one: on the page. 1. Create or edit `/etc/gitlab/gitlab.rb` and use the following configuration: +<!-- +Updates to example must be made at: +- https://gitlab.com/gitlab-org/gitlab/blob/master/doc/administration/sidekiq.md +- all reference architecture pages +--> + ```ruby # Avoid running unnecessary services on the Sidekiq server gitaly['enable'] = false postgresql['enable'] = false redis['enable'] = false + nginx['enable'] = false puma['enable'] = false gitlab_workhorse['enable'] = false prometheus['enable'] = false @@ -1705,7 +1725,6 @@ To configure the Sidekiq nodes, on each one: grafana['enable'] = false gitlab_exporter['enable'] = false gitlab_kas['enable'] = false - nginx['enable'] = false # External URL ## This should match the URL of the external load balancer @@ -2169,7 +2188,7 @@ GitLab has been tested on a number of object storage providers: - [Google Cloud Storage](https://cloud.google.com/storage) - [Digital Ocean Spaces](http://www.digitalocean.com/products/spaces) - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) -- [OpenStack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) +- [OpenStack Swift (S3 compatibility mode)](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) - On-premises hardware and appliances from various storage vendors. - MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. @@ -2202,7 +2221,6 @@ on what features you intend to use: | [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | | [Packages](../packages/index.md#using-object-storage) (optional feature) | Yes | | [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature) | Yes | -| [Pseudonymizer](../pseudonymizer.md) (optional feature) | No | | [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | | [Terraform state files](../terraform_state.md#using-object-storage) | Yes | @@ -2281,7 +2299,7 @@ The following tables and diagram detail the hybrid environment using the same fo as the normal environment above. First are the components that run in Kubernetes. The recommendation at this time is to -use Google Cloud's Kubernetes Engine (GKE) and associated machine types, but the memory +use Google Cloud's Kubernetes Engine (GKE) or AWS Elastic Kubernetes Service (EKS) and associated machine types, but the memory and CPU requirements should translate to most other providers. We hope to update this in the future with further specific cloud provider details. @@ -2293,7 +2311,7 @@ future with further specific cloud provider details. - For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results) [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. -- Nodes configuration is shown as it is forced to ensure pod vcpu / memory ratios and avoid scaling during **performance testing**. +- Nodes configuration is shown as it is forced to ensure pod vCPU / memory ratios and avoid scaling during **performance testing**. - In production deployments, there is no need to assign pods to nodes. A minimum of three nodes in three different availability zones is strongly recommended to align with resilient cloud architecture practices. Next are the backend components that run on static compute VMs via Omnibus (or External PaaS @@ -2315,7 +2333,7 @@ services where applicable): <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> 1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. -2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS ElastiCache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. diff --git a/doc/administration/reference_architectures/1k_users.md b/doc/administration/reference_architectures/1k_users.md index 7213a1eb92b..8d5afd732d9 100644 --- a/doc/administration/reference_architectures/1k_users.md +++ b/doc/administration/reference_architectures/1k_users.md @@ -62,7 +62,7 @@ monitor .[#7FFFD4,norank]--> redis @enduml ``` -The diagram above shows that while GitLab can be installed on a single server, it is internally composed of multiple services. As a GitLab instance is scaled, each of these services are broken out and independently scaled according to the demands placed on them. In some cases PaaS can be leveraged for some services (e.g. Cloud Object Storage for some file systems). For the sake of redundancy some of the services become clusters of nodes storing the same data. In a horizontal configuration of GitLab there are various ancillary services required to coordinate clusters or discover of resources (e.g. PgBouncer for Postgres connection management, Consul for Prometheus end point discovery). +The diagram above shows that while GitLab can be installed on a single server, it is internally composed of multiple services. As a GitLab instance is scaled, each of these services are broken out and independently scaled according to the demands placed on them. In some cases PaaS can be leveraged for some services (e.g. Cloud Object Storage for some file systems). For the sake of redundancy some of the services become clusters of nodes storing the same data. In a horizontal configuration of GitLab there are various ancillary services required to coordinate clusters or discover of resources (e.g. PgBouncer for PostgreSQL connection management, Consul for Prometheus end point discovery). ## Requirements diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index 2a1d344508e..d9f349b59c7 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -39,7 +39,7 @@ full list of reference architectures, see <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> 1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. -2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS ElastiCache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. @@ -1355,11 +1355,18 @@ To configure the Praefect nodes, on each one: on the page. 1. Edit the `/etc/gitlab/gitlab.rb` file to configure Praefect: +<!-- +Updates to example must be made at: +- https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/administration/gitaly/praefect.md +- all reference architecture pages +--> + ```ruby # Avoid running unnecessary services on the Praefect server gitaly['enable'] = false postgresql['enable'] = false redis['enable'] = false + nginx['enable'] = false puma['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false @@ -1368,7 +1375,6 @@ To configure the Praefect nodes, on each one: grafana['enable'] = false gitlab_exporter['enable'] = false gitlab_kas['enable'] = false - nginx['enable'] = false # Praefect Configuration praefect['enable'] = true @@ -1395,8 +1401,8 @@ To configure the Praefect nodes, on each one: praefect['database_host'] = '10.6.0.141' praefect['database_port'] = 5432 # `no_proxy` settings must always be a direct connection for caching - praefect['database_host_no_proxy'] = '10.6.0.141' - praefect['database_port_no_proxy'] = 5432 + praefect['database_direct_host'] = '10.6.0.141' + praefect['database_direct_port'] = 5432 praefect['database_dbname'] = 'praefect_production' praefect['database_user'] = 'praefect' praefect['database_password'] = '<praefect_postgresql_password>' @@ -1499,10 +1505,18 @@ On each node: 1. Edit the Gitaly server node's `/etc/gitlab/gitlab.rb` file to configure storage paths, enable the network listener, and to configure the token: +<!-- +Updates to example must be made at: +- https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab +- https://gitlab.com/gitlab-org/gitlab/blob/master/doc/administration/gitaly/index.md#gitaly-server-configuration +- all reference architecture pages +--> + ```ruby # Avoid running unnecessary services on the Gitaly server postgresql['enable'] = false redis['enable'] = false + nginx['enable'] = false puma['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false @@ -1511,7 +1525,6 @@ On each node: grafana['enable'] = false gitlab_exporter['enable'] = false gitlab_kas['enable'] = false - nginx['enable'] = false # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -1697,11 +1710,18 @@ To configure the Sidekiq nodes, on each one: on the page. 1. Create or edit `/etc/gitlab/gitlab.rb` and use the following configuration: +<!-- +Updates to example must be made at: +- https://gitlab.com/gitlab-org/gitlab/blob/master/doc/administration/sidekiq.md +- all reference architecture pages +--> + ```ruby # Avoid running unnecessary services on the Sidekiq server gitaly['enable'] = false postgresql['enable'] = false redis['enable'] = false + nginx['enable'] = false puma['enable'] = false gitlab_workhorse['enable'] = false prometheus['enable'] = false @@ -1709,7 +1729,6 @@ To configure the Sidekiq nodes, on each one: grafana['enable'] = false gitlab_exporter['enable'] = false gitlab_kas['enable'] = false - nginx['enable'] = false # External URL ## This should match the URL of the external load balancer @@ -2173,7 +2192,7 @@ GitLab has been tested on a number of object storage providers: - [Google Cloud Storage](https://cloud.google.com/storage) - [Digital Ocean Spaces](http://www.digitalocean.com/products/spaces) - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) -- [OpenStack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) +- [OpenStack Swift (S3 compatibility mode)](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) - On-premises hardware and appliances from various storage vendors. - MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. @@ -2206,7 +2225,6 @@ on what features you intend to use: | [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | | [Packages](../packages/index.md#using-object-storage) (optional feature) | Yes | | [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature) | Yes | -| [Pseudonymizer](../pseudonymizer.md) (optional feature) | No | | [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | | [Terraform state files](../terraform_state.md#using-object-storage) | Yes | @@ -2279,7 +2297,7 @@ The following tables and diagram detail the hybrid environment using the same fo as the normal environment above. First are the components that run in Kubernetes. The recommendation at this time is to -use Google Cloud's Kubernetes Engine (GKE) and associated machine types, but the memory +use Google Cloud's Kubernetes Engine (GKE) or AWS Elastic Kubernetes Service (EKS) and associated machine types, but the memory and CPU requirements should translate to most other providers. We hope to update this in the future with further specific cloud provider details. @@ -2291,7 +2309,7 @@ future with further specific cloud provider details. - For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results) [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. -- Nodes configuration is shown as it is forced to ensure pod vcpu / memory ratios and avoid scaling during **performance testing**. +- Nodes configuration is shown as it is forced to ensure pod vCPU / memory ratios and avoid scaling during **performance testing**. - In production deployments, there is no need to assign pods to nodes. A minimum of three nodes in three different availability zones is strongly recommended to align with resilient cloud architecture practices. Next are the backend components that run on static compute VMs via Omnibus (or External PaaS @@ -2313,7 +2331,7 @@ services where applicable): <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> 1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. -2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS ElastiCache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index f417d1e2ab5..d029f356612 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -32,7 +32,7 @@ For a full list of reference architectures, see <!-- markdownlint-disable MD029 --> 1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. -2. Can be optionally run as reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +2. Can be optionally run as reputable third-party external PaaS Redis solutions. Google Memorystore and AWS ElastiCache are known to work. 3. Can be optionally run as reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. <!-- markdownlint-enable MD029 --> @@ -444,21 +444,18 @@ To configure the Gitaly server, on the server node you want to use for Gitaly: 1. Edit the Gitaly server node's `/etc/gitlab/gitlab.rb` file to configure storage paths, enable the network listener, and to configure the token: - <!-- Updates to following example must also be made at https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab --> +<!-- +Updates to example must be made at: +- https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab +- https://gitlab.com/gitlab-org/gitlab/blob/master/doc/administration/gitaly/index.md#gitaly-server-configuration +- all reference architecture pages +--> ```ruby - # /etc/gitlab/gitlab.rb - - # Gitaly and GitLab use two shared secrets for authentication, one to authenticate gRPC requests - # to Gitaly, and a second for authentication callbacks from GitLab-Shell to the GitLab internal API. - # The following two values must be the same as their respective values - # of the GitLab Rails application setup - gitaly['auth_token'] = 'gitalysecret' - gitlab_shell['secret_token'] = 'shellsecret' - # Avoid running unnecessary services on the Gitaly server postgresql['enable'] = false redis['enable'] = false + nginx['enable'] = false puma['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false @@ -467,7 +464,6 @@ To configure the Gitaly server, on the server node you want to use for Gitaly: grafana['enable'] = false gitlab_exporter['enable'] = false gitlab_kas['enable'] = false - nginx['enable'] = false # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -486,6 +482,13 @@ To configure the Gitaly server, on the server node you want to use for Gitaly: gitaly['listen_addr'] = "0.0.0.0:8075" gitaly['prometheus_listen_addr'] = "0.0.0.0:9236" + # Gitaly and GitLab use two shared secrets for authentication, one to authenticate gRPC requests + # to Gitaly, and a second for authentication callbacks from GitLab-Shell to the GitLab internal API. + # The following two values must be the same as their respective values + # of the GitLab Rails application setup + gitaly['auth_token'] = 'gitalysecret' + gitlab_shell['secret_token'] = 'shellsecret' + # Set the network addresses that the exporters used for monitoring will listen on node_exporter['listen_address'] = '0.0.0.0:9100' @@ -891,7 +894,7 @@ GitLab has been tested on a number of object storage providers: - [Google Cloud Storage](https://cloud.google.com/storage) - [Digital Ocean Spaces](http://www.digitalocean.com/products/spaces) - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) -- [OpenStack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) +- [OpenStack Swift (S3 compatibility mode)](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) - On-premises hardware and appliances from various storage vendors. - MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. @@ -924,7 +927,6 @@ on what features you intend to use: | [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | | [Packages](../packages/index.md#using-object-storage) (optional feature) | Yes | | [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature) | Yes | -| [Pseudonymizer](../pseudonymizer.md) (optional feature) | No | | [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | | [Terraform state files](../terraform_state.md#using-object-storage) | Yes | @@ -1000,7 +1002,7 @@ The following tables and diagram detail the hybrid environment using the same fo as the normal environment above. First are the components that run in Kubernetes. The recommendation at this time is to -use Google Cloud's Kubernetes Engine (GKE) and associated machine types, but the memory +use Google Cloud's Kubernetes Engine (GKE) or AWS Elastic Kubernetes Service (EKS) and associated machine types, but the memory and CPU requirements should translate to most other providers. We hope to update this in the future with further specific cloud provider details. @@ -1012,7 +1014,7 @@ future with further specific cloud provider details. - For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results) [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. -- Nodes configuration is shown as it is forced to ensure pod vcpu / memory ratios and avoid scaling during **performance testing**. +- Nodes configuration is shown as it is forced to ensure pod vCPU / memory ratios and avoid scaling during **performance testing**. - In production deployments, there is no need to assign pods to nodes. A minimum of three nodes in three different availability zones is strongly recommended to align with resilient cloud architecture practices. Next are the backend components that run on static compute VMs via Omnibus (or External PaaS @@ -1028,7 +1030,7 @@ services where applicable): <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> 1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. -2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS ElastiCache are known to work. 3. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. <!-- markdownlint-enable MD029 --> diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index 9d8be3e90b6..14b8982766c 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -48,7 +48,7 @@ For a full list of reference architectures, see <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> 1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. -2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS ElastiCache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. @@ -1295,11 +1295,18 @@ To configure the Praefect nodes, on each one: on the page. 1. Edit the `/etc/gitlab/gitlab.rb` file to configure Praefect: +<!-- +Updates to example must be made at: +- https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/administration/gitaly/praefect.md +- all reference architecture pages +--> + ```ruby # Avoid running unnecessary services on the Praefect server gitaly['enable'] = false postgresql['enable'] = false redis['enable'] = false + nginx['enable'] = false puma['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false @@ -1308,7 +1315,6 @@ To configure the Praefect nodes, on each one: grafana['enable'] = false gitlab_exporter['enable'] = false gitlab_kas['enable'] = false - nginx['enable'] = false # Praefect Configuration praefect['enable'] = true @@ -1335,8 +1341,8 @@ To configure the Praefect nodes, on each one: praefect['database_host'] = '10.6.0.141' praefect['database_port'] = 5432 # `no_proxy` settings must always be a direct connection for caching - praefect['database_host_no_proxy'] = '10.6.0.141' - praefect['database_port_no_proxy'] = 5432 + praefect['database_direct_host'] = '10.6.0.141' + praefect['database_direct_port'] = 5432 praefect['database_dbname'] = 'praefect_production' praefect['database_user'] = 'praefect' praefect['database_password'] = '<praefect_postgresql_password>' @@ -1439,10 +1445,18 @@ On each node: 1. Edit the Gitaly server node's `/etc/gitlab/gitlab.rb` file to configure storage paths, enable the network listener, and to configure the token: +<!-- +Updates to example must be made at: +- https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab +- https://gitlab.com/gitlab-org/gitlab/blob/master/doc/administration/gitaly/index.md#gitaly-server-configuration +- all reference architecture pages +--> + ```ruby # Avoid running unnecessary services on the Gitaly server postgresql['enable'] = false redis['enable'] = false + nginx['enable'] = false puma['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false @@ -1451,7 +1465,6 @@ On each node: grafana['enable'] = false gitlab_exporter['enable'] = false gitlab_kas['enable'] = false - nginx['enable'] = false # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -1639,11 +1652,18 @@ To configure the Sidekiq nodes, one each one: on the page. 1. Create or edit `/etc/gitlab/gitlab.rb` and use the following configuration: +<!-- +Updates to example must be made at: +- https://gitlab.com/gitlab-org/gitlab/blob/master/doc/administration/sidekiq.md +- all reference architecture pages +--> + ```ruby # Avoid running unnecessary services on the Sidekiq server gitaly['enable'] = false postgresql['enable'] = false redis['enable'] = false + nginx['enable'] = false puma['enable'] = false gitlab_workhorse['enable'] = false prometheus['enable'] = false @@ -1651,7 +1671,6 @@ To configure the Sidekiq nodes, one each one: grafana['enable'] = false gitlab_exporter['enable'] = false gitlab_kas['enable'] = false - nginx['enable'] = false # External URL ## This should match the URL of the external load balancer @@ -2108,7 +2127,7 @@ GitLab has been tested on a number of object storage providers: - [Google Cloud Storage](https://cloud.google.com/storage) - [Digital Ocean Spaces](http://www.digitalocean.com/products/spaces) - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) -- [OpenStack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) +- [OpenStack Swift (S3 compatibility mode)](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) - On-premises hardware and appliances from various storage vendors. - MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. @@ -2141,7 +2160,6 @@ on what features you intend to use: | [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | | [Packages](../packages/index.md#using-object-storage) (optional feature) | Yes | | [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature) | Yes | -| [Pseudonymizer](../pseudonymizer.md) (optional feature) | No | | [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | | [Terraform state files](../terraform_state.md#using-object-storage) | Yes | @@ -2239,7 +2257,7 @@ The following tables and diagram detail the hybrid environment using the same fo as the normal environment above. First are the components that run in Kubernetes. The recommendation at this time is to -use Google Cloud's Kubernetes Engine (GKE) and associated machine types, but the memory +use Google Cloud's Kubernetes Engine (GKE) or AWS Elastic Kubernetes Service (EKS) and associated machine types, but the memory and CPU requirements should translate to most other providers. We hope to update this in the future with further specific cloud provider details. @@ -2251,7 +2269,7 @@ future with further specific cloud provider details. - For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results) [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. -- Nodes configuration is shown as it is forced to ensure pod vcpu / memory ratios and avoid scaling during **performance testing**. +- Nodes configuration is shown as it is forced to ensure pod vCPU / memory ratios and avoid scaling during **performance testing**. - In production deployments, there is no need to assign pods to nodes. A minimum of three nodes in three different availability zones is strongly recommended to align with resilient cloud architecture practices. Next are the backend components that run on static compute VMs via Omnibus (or External PaaS @@ -2272,7 +2290,7 @@ services where applicable): <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> 1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. -2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS ElastiCache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index 6c4e2227b18..078e3a289cc 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -39,7 +39,7 @@ full list of reference architectures, see <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> 1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. -2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS ElastiCache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. @@ -86,7 +86,7 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 - + redis_cache -[hidden]-> redis_persistent } @@ -1364,11 +1364,18 @@ To configure the Praefect nodes, on each one: on the page. 1. Edit the `/etc/gitlab/gitlab.rb` file to configure Praefect: +<!-- +Updates to example must be made at: +- https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/administration/gitaly/praefect.md +- all reference architecture pages +--> + ```ruby # Avoid running unnecessary services on the Praefect server gitaly['enable'] = false postgresql['enable'] = false redis['enable'] = false + nginx['enable'] = false puma['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false @@ -1377,7 +1384,6 @@ To configure the Praefect nodes, on each one: grafana['enable'] = false gitlab_exporter['enable'] = false gitlab_kas['enable'] = false - nginx['enable'] = false # Praefect Configuration praefect['enable'] = true @@ -1404,8 +1410,8 @@ To configure the Praefect nodes, on each one: praefect['database_host'] = '10.6.0.141' praefect['database_port'] = 5432 # `no_proxy` settings must always be a direct connection for caching - praefect['database_host_no_proxy'] = '10.6.0.141' - praefect['database_port_no_proxy'] = 5432 + praefect['database_direct_host'] = '10.6.0.141' + praefect['database_direct_port'] = 5432 praefect['database_dbname'] = 'praefect_production' praefect['database_user'] = 'praefect' praefect['database_password'] = '<praefect_postgresql_password>' @@ -1508,10 +1514,18 @@ On each node: 1. Edit the Gitaly server node's `/etc/gitlab/gitlab.rb` file to configure storage paths, enable the network listener, and to configure the token: +<!-- +Updates to example must be made at: +- https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab +- https://gitlab.com/gitlab-org/gitlab/blob/master/doc/administration/gitaly/index.md#gitaly-server-configuration +- all reference architecture pages +--> + ```ruby # Avoid running unnecessary services on the Gitaly server postgresql['enable'] = false redis['enable'] = false + nginx['enable'] = false puma['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false @@ -1520,7 +1534,6 @@ On each node: grafana['enable'] = false gitlab_exporter['enable'] = false gitlab_kas['enable'] = false - nginx['enable'] = false # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -1706,11 +1719,18 @@ To configure the Sidekiq nodes, on each one: on the page. 1. Create or edit `/etc/gitlab/gitlab.rb` and use the following configuration: +<!-- +Updates to example must be made at: +- https://gitlab.com/gitlab-org/gitlab/blob/master/doc/administration/sidekiq.md +- all reference architecture pages +--> + ```ruby # Avoid running unnecessary services on the Sidekiq server gitaly['enable'] = false postgresql['enable'] = false redis['enable'] = false + nginx['enable'] = false puma['enable'] = false gitlab_workhorse['enable'] = false prometheus['enable'] = false @@ -1718,7 +1738,6 @@ To configure the Sidekiq nodes, on each one: grafana['enable'] = false gitlab_exporter['enable'] = false gitlab_kas['enable'] = false - nginx['enable'] = false # External URL ## This should match the URL of the external load balancer @@ -2189,7 +2208,7 @@ GitLab has been tested on a number of object storage providers: - [Google Cloud Storage](https://cloud.google.com/storage) - [Digital Ocean Spaces](http://www.digitalocean.com/products/spaces) - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) -- [OpenStack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) +- [OpenStack Swift (S3 compatibility mode)](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) - On-premises hardware and appliances from various storage vendors. - MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. @@ -2222,7 +2241,6 @@ on what features you intend to use: | [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | | [Packages](../packages/index.md#using-object-storage) (optional feature) | Yes | | [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature) | Yes | -| [Pseudonymizer](../pseudonymizer.md) (optional feature) | No | | [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | | [Terraform state files](../terraform_state.md#using-object-storage) | Yes | @@ -2295,7 +2313,7 @@ The following tables and diagram detail the hybrid environment using the same fo as the normal environment above. First are the components that run in Kubernetes. The recommendation at this time is to -use Google Cloud's Kubernetes Engine (GKE) and associated machine types, but the memory +use Google Cloud's Kubernetes Engine (GKE) or AWS Elastic Kubernetes Service (EKS) and associated machine types, but the memory and CPU requirements should translate to most other providers. We hope to update this in the future with further specific cloud provider details. @@ -2304,10 +2322,10 @@ future with further specific cloud provider details. | Webservice | 16 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `m5.8xlarge` | 510 vCPU, 472 GB memory | | Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | 15.5 vCPU, 50 GB memory | | Supporting services such as NGINX, Prometheus | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | 7.75 vCPU, 25 GB memory | - + - For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results) [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. -- Nodes configuration is shown as it is forced to ensure pod vcpu / memory ratios and avoid scaling during **performance testing**. +- Nodes configuration is shown as it is forced to ensure pod vCPU / memory ratios and avoid scaling during **performance testing**. - In production deployments, there is no need to assign pods to nodes. A minimum of three nodes in three different availability zones is strongly recommended to align with resilient cloud architecture practices. Next are the backend components that run on static compute VMs via Omnibus (or External PaaS @@ -2329,7 +2347,7 @@ services where applicable): <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> 1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. -2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS ElastiCache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. @@ -2377,7 +2395,7 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 - + redis_cache -[hidden]-> redis_persistent } diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index dd7209a3a3a..d18c77902f6 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -45,7 +45,7 @@ costly-to-operate environment by using the <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> 1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. -2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS ElastiCache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. @@ -1293,11 +1293,18 @@ To configure the Praefect nodes, on each one: on the page. 1. Edit the `/etc/gitlab/gitlab.rb` file to configure Praefect: +<!-- +Updates to example must be made at: +- https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/administration/gitaly/praefect.md +- all reference architecture pages +--> + ```ruby # Avoid running unnecessary services on the Praefect server gitaly['enable'] = false postgresql['enable'] = false redis['enable'] = false + nginx['enable'] = false puma['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false @@ -1306,7 +1313,6 @@ To configure the Praefect nodes, on each one: grafana['enable'] = false gitlab_exporter['enable'] = false gitlab_kas['enable'] = false - nginx['enable'] = false # Praefect Configuration praefect['enable'] = true @@ -1333,8 +1339,8 @@ To configure the Praefect nodes, on each one: praefect['database_host'] = '10.6.0.141' praefect['database_port'] = 5432 # `no_proxy` settings must always be a direct connection for caching - praefect['database_host_no_proxy'] = '10.6.0.141' - praefect['database_port_no_proxy'] = 5432 + praefect['database_direct_host'] = '10.6.0.141' + praefect['database_direct_port'] = 5432 praefect['database_dbname'] = 'praefect_production' praefect['database_user'] = 'praefect' praefect['database_password'] = '<praefect_postgresql_password>' @@ -1437,10 +1443,18 @@ On each node: 1. Edit the Gitaly server node's `/etc/gitlab/gitlab.rb` file to configure storage paths, enable the network listener, and to configure the token: +<!-- +Updates to example must be made at: +- https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab +- https://gitlab.com/gitlab-org/gitlab/blob/master/doc/administration/gitaly/index.md#gitaly-server-configuration +- all reference architecture pages +--> + ```ruby # Avoid running unnecessary services on the Gitaly server postgresql['enable'] = false redis['enable'] = false + nginx['enable'] = false puma['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false @@ -1449,7 +1463,6 @@ On each node: grafana['enable'] = false gitlab_exporter['enable'] = false gitlab_kas['enable'] = false - nginx['enable'] = false # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -1635,11 +1648,18 @@ To configure the Sidekiq nodes, one each one: on the page. 1. Create or edit `/etc/gitlab/gitlab.rb` and use the following configuration: +<!-- +Updates to example must be made at: +- https://gitlab.com/gitlab-org/gitlab/blob/master/doc/administration/sidekiq.md +- all reference architecture pages +--> + ```ruby # Avoid running unnecessary services on the Sidekiq server gitaly['enable'] = false postgresql['enable'] = false redis['enable'] = false + nginx['enable'] = false puma['enable'] = false gitlab_workhorse['enable'] = false prometheus['enable'] = false @@ -1647,7 +1667,6 @@ To configure the Sidekiq nodes, one each one: grafana['enable'] = false gitlab_exporter['enable'] = false gitlab_kas['enable'] = false - nginx['enable'] = false # External URL ## This should match the URL of the external load balancer @@ -2108,7 +2127,7 @@ GitLab has been tested on a number of object storage providers: - [Google Cloud Storage](https://cloud.google.com/storage) - [Digital Ocean Spaces](http://www.digitalocean.com/products/spaces) - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) -- [OpenStack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) +- [OpenStack Swift (S3 compatibility mode)](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) - On-premises hardware and appliances from various storage vendors. - MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. @@ -2141,7 +2160,6 @@ on what features you intend to use: | [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | | [Packages](../packages/index.md#using-object-storage) (optional feature) | Yes | | [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature) | Yes | -| [Pseudonymizer](../pseudonymizer.md) (optional feature) | No | | [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | | [Terraform state files](../terraform_state.md#using-object-storage) | Yes | @@ -2214,7 +2232,7 @@ The following tables and diagram detail the hybrid environment using the same fo as the normal environment above. First are the components that run in Kubernetes. The recommendation at this time is to -use Google Cloud's Kubernetes Engine (GKE) and associated machine types, but the memory +use Google Cloud's Kubernetes Engine (GKE) or AWS Elastic Kubernetes Service (EKS) and associated machine types, but the memory and CPU requirements should translate to most other providers. We hope to update this in the future with further specific cloud provider details. @@ -2226,7 +2244,7 @@ future with further specific cloud provider details. - For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results) [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. -- Nodes configuration is shown as it is forced to ensure pod vcpu / memory ratios and avoid scaling during **performance testing**. +- Nodes configuration is shown as it is forced to ensure pod vCPU / memory ratios and avoid scaling during **performance testing**. - In production deployments, there is no need to assign pods to nodes. A minimum of three nodes in three different availability zones is strongly recommended to align with resilient cloud architecture practices. Next are the backend components that run on static compute VMs via Omnibus (or External PaaS @@ -2247,7 +2265,7 @@ services where applicable): <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> 1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. -2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS ElastiCache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. 5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index bb741c39c08..089e8881d7d 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -157,7 +157,7 @@ table.test-coverage th { </tr> <tr> <th scope="row">1k</th> - <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/1k">Daily</a> (to be moved to Weekly)</td> + <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/1k">Weekly</a></td> <td></td> <td></td> <td></td> @@ -165,7 +165,7 @@ table.test-coverage th { </tr> <tr> <th scope="row">2k</th> - <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/2k">Daily</a> (to be moved to Weekly)</td> + <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/2k">Weekly</a></td> <td></td> <td></td> <td></td> @@ -176,7 +176,7 @@ table.test-coverage th { <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/3k">Weekly</a></td> <td></td> <td></td> - <td></td> + <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/3k_hybrid_aws_services">Weekly</a></td> <td></td> </tr> <tr> @@ -190,9 +190,9 @@ table.test-coverage th { <tr> <th scope="row">10k</th> <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/10k">Daily</a></td> - <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/10k-Cloud-Native-Hybrid">Ad-Hoc</a></td> - <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/10k">Ad-Hoc (inc Cloud Services)</a></td> - <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/10k-Cloud-Native-Hybrid">Ad-Hoc</a></td> + <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/10k_hybrid">Weekly</a></td> + <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/10k_aws">Weekly</a></td> + <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/10k_hybrid_aws_services">Weekly</a></td> <td><a href="https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/10k">Ad-Hoc</a></td> </tr> <tr> @@ -215,6 +215,8 @@ table.test-coverage th { ## Cost to run +The following table details the cost to run the different reference architectures across GCP, AWS, and Azure. Bare-metal costs are not included here as it varies widely depending on each customer configuration. + <table class="test-coverage"> <col> <colgroup span="2"></colgroup> @@ -356,7 +358,7 @@ Additionally, the following cloud provider services are validated and supported <tr> <td>Redis</td> <td></td> - <td>✅ <a href="https://aws.amazon.com/elasticache/" target="_blank" rel="noopener noreferrer">Elasticache</a></td> + <td>✅ <a href="https://aws.amazon.com/elasticache/" target="_blank" rel="noopener noreferrer">ElastiCache</a></td> <td></td> </tr> </tbody> diff --git a/doc/administration/repository_storage_types.md b/doc/administration/repository_storage_types.md index f33d494f638..1a593c8c6bd 100644 --- a/doc/administration/repository_storage_types.md +++ b/doc/administration/repository_storage_types.md @@ -75,7 +75,7 @@ translate between the human-readable project name and the hashed storage path. Y Administrators can look up a project's hashed path from its name or ID using: -- The [Admin area](../user/admin_area/index.md#administering-projects). +- The [Admin Area](../user/admin_area/index.md#administering-projects). - A Rails console. To look up a project's hash path in the Admin Area: @@ -154,6 +154,20 @@ WARNING: Do not run `git prune` or `git gc` in object pool repositories, which are stored in the `@pools` directory. This can cause data loss in the regular repositories that depend on the object pool. +### Group wiki storage + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13195) in GitLab 13.5. + +Unlike project wikis that are stored in the `@hashed` directory, group wikis are stored in a directory called `@groups`. +Like project wikis, group wikis follow the hashed storage folder convention, but use a hash of the group ID rather than the project ID. + +For example: + +```ruby +# group wiki paths +"@groups/#{hash[0..1]}/#{hash[2..3]}/#{hash}.wiki.git" +``` + ### Object storage support This table shows which storable objects are storable in each storage type: diff --git a/doc/administration/sidekiq.md b/doc/administration/sidekiq.md index b6c1c6a704e..d9031e2221b 100644 --- a/doc/administration/sidekiq.md +++ b/doc/administration/sidekiq.md @@ -20,21 +20,13 @@ To configure Sidekiq: 1. Edit `/etc/gitlab/gitlab.rb` with the following information and make sure to replace with your values: - ```ruby - ## - ## To maintain uniformity of links across nodes, the - ##`external_url` on the Sidekiq server should point to the external URL that users - ## use to access GitLab. This can be either: - ## - ## - The `external_url` set on your application server. - ## - The URL of a external load balancer, which routes traffic to the GitLab application server. - ## - - external_url 'https://gitlab.example.com' - - ## Prevent database migrations from running on upgrade automatically - gitlab_rails['auto_migrate'] = false +<!-- +Updates to example must be made at: +- https://gitlab.com/gitlab-org/gitlab/blob/master/doc/administration/sidekiq.md +- all reference architecture pages +--> + ```ruby ######################################## ##### Services Disabled ### ######################################## @@ -45,32 +37,27 @@ To configure Sidekiq: # Enable only the services you want to run on each # specific server, while disabling all others. # - nginx['enable'] = false - grafana['enable'] = false - prometheus['enable'] = false - gitlab_rails['auto_migrate'] = false - alertmanager['enable'] = false gitaly['enable'] = false - gitlab_workhorse['enable'] = false - nginx['enable'] = false - postgres_exporter['enable'] = false postgresql['enable'] = false redis['enable'] = false - redis_exporter['enable'] = false + nginx['enable'] = false puma['enable'] = false + gitlab_workhorse['enable'] = false + prometheus['enable'] = false + alertmanager['enable'] = false + grafana['enable'] = false gitlab_exporter['enable'] = false + gitlab_kas['enable'] = false - ####################################### - ### Sidekiq configuration ### - ####################################### - sidekiq['enable'] = true - sidekiq['listen_address'] = "0.0.0.0" - - ## Set number of Sidekiq queue processes to the same number as available CPUs - sidekiq['queue_groups'] = ['*'] * 4 - - ## Set number of Sidekiq threads per queue process to the recommend number of 10 - sidekiq['max_concurrency'] = 10 + ## + ## To maintain uniformity of links across nodes, the + ## `external_url` on the Sidekiq server should point to the external URL that users + ## use to access GitLab. This can be either: + ## + ## - The `external_url` set on your application server. + ## - The URL of a external load balancer, which routes traffic to the GitLab application server. + ## + external_url 'https://gitlab.example.com' ######################################## #### Redis ### @@ -89,9 +76,11 @@ To configure Sidekiq: ## Replace <gitaly_token> with the one you set up, see ## https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#about-the-gitaly-token git_data_dirs({ - 'default' => { 'gitaly_address' => 'tcp://gitaly:8075' }, + "default" => { + "gitaly_address" => "tcp://gitaly:8075", + "gitaly_token" => "<gitaly_token>" + } }) - gitlab_rails['gitaly_token'] = '<gitaly_token>' ####################################### ### Postgres ### @@ -99,16 +88,28 @@ To configure Sidekiq: # Replace <database_host> and <database_password> gitlab_rails['db_host'] = '<database_host>' - gitlab_rails['db_password'] = '<database_password>' gitlab_rails['db_port'] = '5432' - gitlab_rails['db_adapter'] = 'postgresql' - gitlab_rails['db_encoding'] = 'unicode' - gitlab_rails['auto_migrate'] = false + gitlab_rails['db_password'] = '<database_password>' # Add the Sidekiq nodes to PostgreSQL's trusted addresses. # In the following example, 10.10.1.30/32 is the private IP # of the Sidekiq server. postgresql['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 10.10.1.30/32) + + ## Prevent database migrations from running on upgrade automatically + gitlab_rails['auto_migrate'] = false + + ####################################### + ### Sidekiq configuration ### + ####################################### + sidekiq['enable'] = true + sidekiq['listen_address'] = "0.0.0.0" + + ## Set number of Sidekiq queue processes to the same number as available CPUs + sidekiq['queue_groups'] = ['*'] * 4 + + ## Set number of Sidekiq threads per queue process to the recommend number of 10 + sidekiq['max_concurrency'] = 10 ``` 1. Reconfigure GitLab: @@ -192,12 +193,8 @@ To configure the metrics server: ## Configure health checks -If you use health check probes to observe Sidekiq, -you can set a separate port for health checks. -Configuring health checks is only necessary if there is something that actually probes them. -For more information about health checks, see the [Sidekiq health check page](sidekiq_health_check.md). - -To enable health checks for Sidekiq: +If you use health check probes to observe Sidekiq, enable the Sidekiq health check server. +To make health checks available from `localhost:8092`: 1. Edit `/etc/gitlab/gitlab.rb`: @@ -207,16 +204,14 @@ To enable health checks for Sidekiq: sidekiq['health_checks_listen_port'] = "8092" ``` - NOTE: - If health check settings are not set, they default to the metrics exporter settings. - This default is deprecated and is set to be removed in [GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/347509). - 1. Reconfigure GitLab: ```shell sudo gitlab-ctl reconfigure ``` +For more information about health checks, see the [Sidekiq health check page](sidekiq_health_check.md). + ## Configure LDAP and user or group synchronization If you use LDAP for user and group management, you must add the LDAP configuration to your Sidekiq node as well as the LDAP diff --git a/doc/administration/sidekiq_health_check.md b/doc/administration/sidekiq_health_check.md index ae02711f46f..3ff17d3682b 100644 --- a/doc/administration/sidekiq_health_check.md +++ b/doc/administration/sidekiq_health_check.md @@ -21,8 +21,7 @@ The readiness probe checks whether the Sidekiq workers are ready to process jobs GET /readiness ``` -If you set Sidekiq's address as `localhost` and port as `8092`, -here's an example request: +If the server is bound to `localhost:8092`, the process cluster can be probed for readiness as follows: ```shell curl "http://localhost:8092/readiness" @@ -44,8 +43,7 @@ Checks whether the Sidekiq cluster is running. GET /liveness ``` -If you set Sidekiq's address as `localhost` and port as `8092`, -here's an example request: +If the server is bound to `localhost:8092`, the process cluster can be probed for liveness as follows: ```shell curl "http://localhost:8092/liveness" diff --git a/doc/administration/terraform_state.md b/doc/administration/terraform_state.md index 5e8f7cb18bb..14649b82ba8 100644 --- a/doc/administration/terraform_state.md +++ b/doc/administration/terraform_state.md @@ -18,7 +18,7 @@ The storage location of these files defaults to: These locations can be configured using the options described below. -Use [external object storage](https://docs.gitlab.com/charts/advanced/external-object-storage/#lfs-artifacts-uploads-packages-external-diffs-pseudonymizer-terraform-state-dependency-proxy) configuration for [GitLab Helm chart](https://docs.gitlab.com/charts/) installations. +Use [external object storage](https://docs.gitlab.com/charts/advanced/external-object-storage/#lfs-artifacts-uploads-packages-external-diffs-terraform-state-dependency-proxy) configuration for [GitLab Helm chart](https://docs.gitlab.com/charts/) installations. ## Disabling Terraform state @@ -144,7 +144,7 @@ You can optionally track progress and verify that all packages migrated successf Verify `objectstg` below (where `file_store=2`) has count of all states: ```shell -gitlabhq_production=# SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM terraform_states; +gitlabhq_production=# SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM terraform_state_versions; total | filesystem | objectstg ------+------------+----------- @@ -154,7 +154,7 @@ total | filesystem | objectstg Verify that there are no files on disk in the `terraform_state` folder: ```shell -sudo find /var/opt/gitlab/gitlab-rails/shared/terraform_state -type f | wc -l +sudo find /var/opt/gitlab/gitlab-rails/shared/terraform_state -type f | grep -v tmp | wc -l ``` ### S3-compatible connection settings diff --git a/doc/administration/troubleshooting/diagnostics_tools.md b/doc/administration/troubleshooting/diagnostics_tools.md index d510df5976c..479fdb963cb 100644 --- a/doc/administration/troubleshooting/diagnostics_tools.md +++ b/doc/administration/troubleshooting/diagnostics_tools.md @@ -26,4 +26,4 @@ and summarize raw `strace` data. ## kubesos -The [`kubesos`](https://gitlab.com/gitlab-com/support/toolbox/kubesos/) utiltity retrieves GitLab cluster configuration and logs from GitLab Cloud Native chart deployments. +The [`kubesos`](https://gitlab.com/gitlab-com/support/toolbox/kubesos/) utility retrieves GitLab cluster configuration and logs from GitLab Cloud Native chart deployments. diff --git a/doc/administration/troubleshooting/elasticsearch.md b/doc/administration/troubleshooting/elasticsearch.md index c45938ecd3f..97e625eb0a3 100644 --- a/doc/administration/troubleshooting/elasticsearch.md +++ b/doc/administration/troubleshooting/elasticsearch.md @@ -289,7 +289,7 @@ If the issue is: Go indexer was a beta indexer which can be optionally turned on/off, but in 12.3 it reached stable status and is now the default. - Not concerning the Go indexer, it is almost always an Elasticsearch-side issue. This means you should reach out to your Elasticsearch administrator - regarding the error(s) you are seeing. If you are unsure here, it never hurts to reach + regarding the errors you are seeing. If you are unsure here, it never hurts to reach out to GitLab support. Beyond that, review the error. If it is: diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index 54d934c8986..34481582d8b 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -83,7 +83,7 @@ Notify.test_email(u.email, "Test email for #{u.name}", 'Test email').deliver_now Adding a semicolon(`;`) and a follow-up statement at the end of a statement prevents the default implicit return output. This is useful if you are already explicitly printing details and potentially have a lot of return output: ```ruby -puts ActiveRecord::Base.descendants; :ok +puts ActiveRecord::Base.descendants; :ok Project.select(&:pages_deployed?).each {|p| puts p.pages_url }; true ``` @@ -753,6 +753,21 @@ group.members_with_parents.count parent.members_with_descendants.count ``` +### Find groups that are pending deletion + +```ruby +# +# This section lists all the groups which are pending deletion +# +Group.all.each do |g| + if g.marked_for_deletion? + puts "Group ID: #{g.id}" + puts "Group name: #{g.name}" + puts "Group path: #{g.full_path}" + end +end +``` + ### Delete a group ```ruby @@ -798,6 +813,39 @@ To find features that can be toggled, run `pp p.project_feature`. Available permission levels are listed in [concerns/featurable.rb](https://gitlab.com/gitlab-org/gitlab/blob/master/app/models/concerns/featurable.rb). +### Get all error messages associated with groups, subgroups, members, and requesters + +Collect error messages associated with groups, subgroups, members, and requesters. This +captures error messages that may not appear in the Web interface. This can be especially helpful +for troubleshooting issues with [LDAP group sync](../auth/ldap/ldap_synchronization.md#group-sync) +and unexpected behavior with users and their membership in groups and subgroups. + +```ruby +# Find the group and subgroup +group = Group.find_by_full_path("parent_group") +subgroup = Group.find_by_full_path("parent_group/child_group") + +# Group and subgroup errors +group.valid? +group.errors.map(&:full_messages) + +subgroup.valid? +subgroup.errors.map(&:full_messages) + +# Group and subgroup errors for the members AND requesters +group.requesters.map(&:valid?) +group.requesters.map(&:errors).map(&:full_messages) +group.members.map(&:valid?) +group.members.map(&:errors).map(&:full_messages) +group.members_and_requesters.map(&:errors).map(&:full_messages) + +subgroup.requesters.map(&:valid?) +subgroup.requesters.map(&:errors).map(&:full_messages) +subgroup.members.map(&:valid?) +subgroup.members.map(&:errors).map(&:full_messages) +subgroup.members_and_requesters.map(&:errors).map(&:full_messages) +``` + ## Authentication ### Re-enable standard web sign-in form @@ -1191,12 +1239,6 @@ This content has been converted to a Rake task, see [verify database values can Geo::JobArtifactRegistry.failed ``` -#### Download artifact - -```ruby -Gitlab::Geo::JobArtifactDownloader.new(:job_artifact, <artifact_id>).execute -``` - #### Get a count of the synced artifacts ```ruby diff --git a/doc/administration/troubleshooting/img/GoogleWorkspace-basic-SAML_v14_10.png b/doc/administration/troubleshooting/img/GoogleWorkspace-basic-SAML_v14_10.png Binary files differindex bc11e18fb6f..e002503ddc0 100644 --- a/doc/administration/troubleshooting/img/GoogleWorkspace-basic-SAML_v14_10.png +++ b/doc/administration/troubleshooting/img/GoogleWorkspace-basic-SAML_v14_10.png diff --git a/doc/administration/troubleshooting/img/GoogleWorkspace-claims_v14_10.png b/doc/administration/troubleshooting/img/GoogleWorkspace-claims_v14_10.png Binary files differindex 78bb1725e9c..2b827e122a8 100644 --- a/doc/administration/troubleshooting/img/GoogleWorkspace-claims_v14_10.png +++ b/doc/administration/troubleshooting/img/GoogleWorkspace-claims_v14_10.png diff --git a/doc/administration/troubleshooting/img/GoogleWorkspace-linkscert_v14_10.png b/doc/administration/troubleshooting/img/GoogleWorkspace-linkscert_v14_10.png Binary files differindex e665e23058c..8d6c5010670 100644 --- a/doc/administration/troubleshooting/img/GoogleWorkspace-linkscert_v14_10.png +++ b/doc/administration/troubleshooting/img/GoogleWorkspace-linkscert_v14_10.png diff --git a/doc/administration/troubleshooting/index.md b/doc/administration/troubleshooting/index.md index 75c5ce460e9..bc9c39d57ea 100644 --- a/doc/administration/troubleshooting/index.md +++ b/doc/administration/troubleshooting/index.md @@ -14,6 +14,7 @@ installation. - [SSL](ssl.md) - [Geo](../geo/replication/troubleshooting.md) - [Elasticsearch](elasticsearch.md) +- [Sidekiq](sidekiq.md) - [GitLab Rails console cheat sheet](gitlab_rails_cheat_sheet.md) - [Group SAML and SCIM troubleshooting](group_saml_scim.md) **(PREMIUM SAAS)** - [Kubernetes cheat sheet](kubernetes_cheat_sheet.md) diff --git a/doc/administration/troubleshooting/log_parsing.md b/doc/administration/troubleshooting/log_parsing.md index c5b1d302db2..9aa490f73ef 100644 --- a/doc/administration/troubleshooting/log_parsing.md +++ b/doc/administration/troubleshooting/log_parsing.md @@ -12,7 +12,7 @@ but if they are not available you can still quickly parse (the default in GitLab 12.0 and later) using [`jq`](https://stedolan.github.io/jq/). NOTE: -Spefically for summarising error events and basic usage statistics, +Specifically for summarizing error events and basic usage statistics, the GitLab Support Team provides the specialised [`fast-stats` tool](https://gitlab.com/gitlab-com/support/toolbox/fast-stats/#when-to-use-it). @@ -81,7 +81,7 @@ jq 'select(.status >= 500)' <FILE> #### Top 10 slowest requests ```shell -jq -s 'sort_by(-.duration) | limit(10; .[])' <FILE> +jq -s 'sort_by(-.duration_s) | limit(10; .[])' <FILE> ``` #### Find and pretty print all requests related to a project @@ -93,7 +93,7 @@ grep <PROJECT_NAME> <FILE> | jq . #### Find all requests with a total duration > 5 seconds ```shell -jq 'select(.duration > 5000)' <FILE> +jq 'select(.duration_s > 5000)' <FILE> ``` #### Find all project requests with more than 5 rugged calls @@ -105,13 +105,13 @@ grep <PROJECT_NAME> <FILE> | jq 'select(.rugged_calls > 5)' #### Find all requests with a Gitaly duration > 10 seconds ```shell -jq 'select(.gitaly_duration > 10000)' <FILE> +jq 'select(.gitaly_duration_s > 10000)' <FILE> ``` #### Find all requests with a queue duration > 10 seconds ```shell -jq 'select(.queue_duration > 10000)' <FILE> +jq 'select(.queue_duration_s > 10000)' <FILE> ``` #### Top 10 requests by # of Gitaly calls @@ -125,7 +125,7 @@ jq -s 'map(select(.gitaly_calls != null)) | sort_by(-.gitaly_calls) | limit(10; #### Print the top three controller methods by request volume and their three longest durations ```shell -jq -s -r 'group_by(.controller+.action) | sort_by(-length) | limit(3; .[]) | sort_by(-.duration) | "CT: \(length)\tMETHOD: \(.[0].controller)#\(.[0].action)\tDURS: \(.[0].duration), \(.[1].duration), \(.[2].duration)"' production_json.log +jq -s -r 'group_by(.controller+.action) | sort_by(-length) | limit(3; .[]) | sort_by(-.duration_s) | "CT: \(length)\tMETHOD: \(.[0].controller)#\(.[0].action)\tDURS: \(.[0].duration_s), \(.[1].duration_s), \(.[2].duration_s)"' production_json.log ``` **Example output** @@ -141,7 +141,7 @@ CT: 1328 METHOD: Projects::NotesController#index DURS: 403.99, 386.29, 384.3 #### Print top three routes with request count and their three longest durations ```shell -jq -s -r 'group_by(.route) | sort_by(-length) | limit(3; .[]) | sort_by(-.duration) | "CT: \(length)\tROUTE: \(.[0].route)\tDURS: \(.[0].duration), \(.[1].duration), \(.[2].duration)"' api_json.log +jq -s -r 'group_by(.route) | sort_by(-length) | limit(3; .[]) | sort_by(-.duration_s) | "CT: \(length)\tROUTE: \(.[0].route)\tDURS: \(.[0].duration_s), \(.[1].duration_s), \(.[2].duration_s)"' api_json.log ``` **Example output** diff --git a/doc/administration/troubleshooting/sidekiq.md b/doc/administration/troubleshooting/sidekiq.md index 62ea3bcfa3c..7a64bcc9b87 100644 --- a/doc/administration/troubleshooting/sidekiq.md +++ b/doc/administration/troubleshooting/sidekiq.md @@ -315,7 +315,7 @@ queue.each { |job| job.delete if <condition>} Have a look at the section below for cancelling running jobs. -In the method above, `<queue-name>` is the name of the queue that contains the job(s) you want to delete and `<condition>` decides which jobs get deleted. +In the method above, `<queue-name>` is the name of the queue that contains the jobs you want to delete and `<condition>` decides which jobs get deleted. Commonly, `<condition>` references the job arguments, which depend on the type of job in question. To find the arguments for a specific queue, you can have a look at the `perform` function of the related worker file, commonly found at `/app/workers/<queue-name>_worker.rb`. diff --git a/doc/administration/troubleshooting/test_environments.md b/doc/administration/troubleshooting/test_environments.md index 5fe493a536c..0821f952d61 100644 --- a/doc/administration/troubleshooting/test_environments.md +++ b/doc/administration/troubleshooting/test_environments.md @@ -52,6 +52,8 @@ gitlab/gitlab-ee:11.5.3-ee.0 #### SAML for Authentication +In the following examples, when replacing `<GITLAB_IP_OR_DOMAIN>` and `<SAML_IP_OR_DOMAIN>` it is important to prepend your IP or domain name, with the protocol (`http://` or `https://`) being used. + We can use the [`test-saml-idp` Docker image](https://hub.docker.com/r/jamedjo/test-saml-idp) to do the work for us: diff --git a/doc/administration/uploads.md b/doc/administration/uploads.md index aba7b5fc318..d66aa5945b6 100644 --- a/doc/administration/uploads.md +++ b/doc/administration/uploads.md @@ -67,8 +67,6 @@ For source installations the following settings are nested under `uploads:` and |---------|-------------|---------| | `enabled` | Enable/disable object storage | `false` | | `remote_directory` | The bucket name where Uploads will be stored| | -| `direct_upload` | Set to `true` to remove Puma from the Upload path. Workhorse handles the actual Artifact Upload to Object Storage while Puma does minimal processing to keep track of the upload. There is no need for local shared storage. The option may be removed if support for a single storage type for all files is introduced. Read more on [direct upload](../development/uploads/implementation.md#direct-upload). | `false` | -| `background_upload` | Set to `false` to disable automatic upload. Option may be removed once upload is direct to S3 (if `direct_upload` is set to `true` it will override `background_upload`) | `true` | | `proxy_download` | Set to `true` to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` | | `connection` | Various connection options described below | | @@ -130,60 +128,3 @@ _The uploads are stored by default in 1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. 1. Migrate any existing local uploads to the object storage using [`gitlab:uploads:migrate:all` Rake task](raketasks/uploads/migrate.md). - -#### OpenStack example - -**In Omnibus installations:** - -_The uploads are stored by default in -`/var/opt/gitlab/gitlab-rails/uploads`._ - -1. Edit `/etc/gitlab/gitlab.rb` and add the following lines by replacing with - the values you want: - - ```ruby - gitlab_rails['uploads_object_store_remote_directory'] = "OPENSTACK_OBJECT_CONTAINER_NAME" - gitlab_rails['uploads_object_store_connection'] = { - 'provider' => 'OpenStack', - 'openstack_username' => 'OPENSTACK_USERNAME', - 'openstack_api_key' => 'OPENSTACK_PASSWORD', - 'openstack_temp_url_key' => 'OPENSTACK_TEMP_URL_KEY', - 'openstack_auth_url' => 'https://auth.cloud.ovh.net/v2.0/', - 'openstack_region' => 'DE1', - 'openstack_tenant' => 'TENANT_ID', - } - ``` - -1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. Migrate any existing local uploads to the object storage using [`gitlab:uploads:migrate:all` Rake task](raketasks/uploads/migrate.md). - ---- - -**In installations from source:** - -_The uploads are stored by default in -`/home/git/gitlab/public/uploads`._ - -1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following - lines: - - ```yaml - uploads: - object_store: - enabled: true - direct_upload: false - background_upload: true - proxy_download: false - remote_directory: OPENSTACK_OBJECT_CONTAINER_NAME - connection: - provider: OpenStack - openstack_username: OPENSTACK_USERNAME - openstack_api_key: OPENSTACK_PASSWORD - openstack_temp_url_key: OPENSTACK_TEMP_URL_KEY - openstack_auth_url: 'https://auth.cloud.ovh.net/v2.0/' - openstack_region: DE1 - openstack_tenant: 'TENANT_ID' - ``` - -1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. Migrate any existing local uploads to the object storage using [`gitlab:uploads:migrate:all` Rake task](raketasks/uploads/migrate.md). |
