diff options
Diffstat (limited to 'doc/administration')
115 files changed, 3053 insertions, 3135 deletions
diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md index a7f4fc10655..f0c4d947668 100644 --- a/doc/administration/audit_events.md +++ b/doc/administration/audit_events.md @@ -6,7 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Audit Events **(PREMIUM)** -GitLab offers a way to view the changes made within the GitLab server for owners and administrators on a [paid plan](https://about.gitlab.com/pricing/). +GitLab offers a way to view the changes made within the GitLab server for owners and administrators +on a [paid plan](https://about.gitlab.com/pricing/). GitLab system administrators can also take advantage of the logs located on the file system. See [the logs system documentation](logs.md#audit_jsonlog) for more details. @@ -49,10 +50,18 @@ When a user is being [impersonated](../user/admin_area/index.md#user-impersonati ### Group events **(PREMIUM)** -A user with a Owner role (or above) can retrieve group audit events of all users. -A user with a Developer or Maintainer role is limited to group audit events based on their individual actions. +A user with: + +- Owner role (or above) can retrieve group audit events of all users. +- Developer or Maintainer role is limited to group audit events based on their individual actions. + +Group events do not include project audit events. + +To view a group's audit events: + +1. Go to the group. +1. On the left sidebar, select **Security & Compliance > Audit Events**. -To view a group's audit events, navigate to **Group > Security & Compliance > Audit Events**. From there, you can see the following actions: - Group name or path changed. @@ -82,7 +91,11 @@ Group events can also be accessed via the [Group Audit Events API](../api/audit_ A user with a Maintainer role (or above) can retrieve project audit events of all users. A user with a Developer role is limited to project audit events based on their individual actions. -To view a project's audit events, navigate to **Project > Security & Compliance > Audit Events**. +To view a project's audit events: + +1. Go to the project. +1. On the left sidebar, select **Security & Compliance > Audit Events**. + From there, you can see the following actions: - Added or removed deploy keys @@ -120,10 +133,14 @@ Server-wide audit events introduce the ability to observe user actions across the entire instance of your GitLab server, making it easy to understand who changed what and when for audit purposes. -To view the server-wide administrator log, visit **Admin Area > Monitoring > Audit Events**. +Instance events do not include group or project audit events. -In addition to the group and project events, the following user actions are also -recorded: +To view the server-wide audit events: + +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Monitoring > Audit Events**. + +The following user actions are recorded: - Sign-in events and the authentication type (such as standard, LDAP, or OmniAuth) - Failed sign-ins @@ -147,6 +164,17 @@ recorded: Instance events can also be accessed via the [Instance Audit Events API](../api/audit_events.md#instance-audit-events). +### Sign-in events **(FREE)** + +Successful sign-in events are the only Audit Events available at all tiers. To see +successful sign-in events: + +1. Select your avatar. +1. Select **Edit profile > Authentication log**. + +After upgrading from GitLab Free to a paid tier, successful sign-in events are the only Audit +Events visible in Audit Events views until more events are logged. + ### Missing events Some events are not tracked in Audit Events. See the following @@ -171,7 +199,7 @@ It may make the user interface for your project or audit events very busy, and t `audit_events` PostgreSQL table may increase considerably. It's disabled by default to prevent performance degradations on GitLab instances with very high Git write traffic. -In an upcoming release, Audit Events for Git push events will be enabled +In an upcoming release, Audit Events for Git push events are planned to be enabled by default. Follow our [Partitioning strategy for Audit Events epic](https://gitlab.com/groups/gitlab-org/-/epics/3206) for updates. If you still wish to enable **Repository push** events in your instance, follow @@ -213,11 +241,12 @@ Export to CSV allows customers to export the current filter view of your audit e CSV file, which stores tabular data in plain text. The data provides a comprehensive view with respect to audit events. -To export the Audit Events to CSV, navigate to -**{monitor}** **Admin Area > Monitoring > Audit Events** +To export the Audit Events to CSV: +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Monitoring > Audit Events**. 1. Select the available search [filters](#search). -1. Click **Export as CSV**. +1. Select **Export as CSV**. ### Sort diff --git a/doc/administration/audit_reports.md b/doc/administration/audit_reports.md index 2721ee39b60..6fa592b96db 100644 --- a/doc/administration/audit_reports.md +++ b/doc/administration/audit_reports.md @@ -5,10 +5,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w description: 'Learn how to create evidence artifacts typically requested by a 3rd party auditor.' --- -# Audit Reports +# Audit reports **(FREE)** GitLab can help owners and administrators respond to auditors by generating -comprehensive reports. These **Audit Reports** vary in scope, depending on the +comprehensive reports. These audit reports vary in scope, depending on the needs. ## Use cases diff --git a/doc/administration/auth/ldap/index.md b/doc/administration/auth/ldap/index.md index 364c7cebea3..bc6a854c518 100644 --- a/doc/administration/auth/ldap/index.md +++ b/doc/administration/auth/ldap/index.md @@ -7,7 +7,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # General LDAP setup **(FREE SELF)** -GitLab integrates with LDAP to support user authentication. +GitLab integrates with [LDAP](https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol) +to support user authentication. This integration works with most LDAP-compliant directory servers, including: @@ -20,58 +21,48 @@ This integration works with most LDAP-compliant directory servers, including: Users added through LDAP take a [licensed seat](../../../subscriptions/self_managed/index.md#billable-users). GitLab Enterprise Editions (EE) include enhanced integration, -including group membership syncing as well as multiple LDAP servers support. - -## Overview - -[LDAP](https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol) -stands for **Lightweight Directory Access Protocol**, which is a standard -application protocol for accessing and maintaining distributed directory -information services over an Internet Protocol (IP) network. +including group membership syncing and multiple LDAP server support. ## Security GitLab assumes that LDAP users: - Are not able to change their LDAP `mail`, `email`, or `userPrincipalName` attributes. - An LDAP user who is allowed to change their email on the LDAP server can potentially + An LDAP user allowed to change their email on the LDAP server can potentially [take over any account](#enabling-ldap-sign-in-for-existing-gitlab-users) on your GitLab server. -- Have unique email addresses, otherwise it is possible for LDAP users with the same +- Have unique email addresses. If not, it's possible for LDAP users with the same email address to share the same GitLab account. We recommend against using LDAP integration if your LDAP users are -allowed to change their 'mail', 'email' or 'userPrincipalName' attribute on -the LDAP server or share email addresses. +allowed to change their `mail`, `email` or `userPrincipalName` attributes on +the LDAP server, or share email addresses. ### User deletion -If a user is deleted from the LDAP server, they are also blocked in GitLab. -Users are immediately blocked from logging in. However, there is an -LDAP check cache time of one hour (see note) which means users that -are already logged in or are using Git over SSH are be able to access -GitLab for up to one hour. Manually block the user in the GitLab Admin Area to -immediately block all access. - -GitLab Enterprise Edition Premium supports a -[configurable sync time](#adjusting-ldap-user-sync-schedule). **(PREMIUM)** +Users deleted from the LDAP server are immediately blocked from signing in +to GitLab. However, there's an LDAP check cache time of one hour (which is +[configurable](#adjusting-ldap-user-sync-schedule) for GitLab Premium users). +This means users already signed-in or who are using Git over SSH can access +GitLab for up to one hour. Manually block the user in the GitLab Admin Area +to immediately block all access. ## Git password authentication -LDAP-enabled users can always authenticate with Git using their GitLab username -or email and LDAP password, even if password authentication for Git is disabled +LDAP-enabled users can authenticate with Git using their GitLab username or +email and LDAP password, even if password authentication for Git is disabled in the application settings. ## Enabling LDAP sign-in for existing GitLab users -When a user signs in to GitLab with LDAP for the first time, and their LDAP -email address is the primary email address of an existing GitLab user, then -the LDAP DN is associated with the existing user. If the LDAP email -attribute is not found in the GitLab user database, a new user is created. +When a user signs in to GitLab with LDAP for the first time and their LDAP +email address is the primary email address of an existing GitLab user, the +LDAP DN is associated with the existing user. If the LDAP email attribute +isn't found in the GitLab user database, a new user is created. In other words, if an existing GitLab user wants to enable LDAP sign-in for themselves, they should check that their GitLab email address matches their -LDAP email address, and then sign into GitLab via their LDAP credentials. +LDAP email address, and then sign into GitLab by using their LDAP credentials. ## Google Secure LDAP @@ -95,7 +86,8 @@ NOTE: The `encryption` value `simple_tls` corresponds to 'Simple TLS' in the LDAP library. `start_tls` corresponds to StartTLS, not to be confused with regular TLS. Normally, if you specify `simple_tls` it is on port 636, while `start_tls` (StartTLS) -would be on port 389. `plain` also operates on port 389. Removed values: `tls` was replaced with `start_tls` and `ssl` was replaced with `simple_tls`. +would be on port 389. `plain` also operates on port 389. Removed values: `tls` was replaced +with `start_tls` and `ssl` was replaced with `simple_tls`. LDAP users must have a set email address, regardless of whether or not it's used to sign in. @@ -165,23 +157,23 @@ production: ### Basic Configuration Settings -| Setting | Description | Required | Examples | -| ------- | ----------- | -------- | -------- | -| `label` | A human-friendly name for your LDAP server. It is displayed on your sign-in page. | yes | `'Paris'` or `'Acme, Ltd.'` | -| `host` | IP address or domain name of your LDAP server. | yes | `'ldap.mydomain.com'` | -| `port` | The port to connect with on your LDAP server. Always an integer, not a string. | yes | `389` or `636` (for SSL) | -| `uid` | LDAP attribute for username. Should be the attribute, not the value that maps to the `uid`. | yes | `'sAMAccountName'` or `'uid'` or `'userPrincipalName'` | -| `bind_dn` | The full DN of the user you bind with. | no | `'america\momo'` or `'CN=Gitlab,OU=Users,DC=domain,DC=com'` | -| `password` | The password of the bind user. | no | `'your_great_password'` | -| `encryption` | Encryption method. The `method` key is deprecated in favor of `encryption`. | yes | `'start_tls'` or `'simple_tls'` or `'plain'` | -| `verify_certificates` | Enables SSL certificate verification if encryption method is `start_tls` or `simple_tls`. Defaults to true. | no | boolean | -| `timeout` | Set a timeout, in seconds, for LDAP queries. This helps avoid blocking a request if the LDAP server becomes unresponsive. A value of `0` means there is no timeout. (default: `10`) | no | `10` or `30` | -| `active_directory` | This setting specifies if LDAP server is Active Directory LDAP server. For non-AD servers it skips the AD specific queries. If your LDAP server is not AD, set this to false. | no | boolean | -| `allow_username_or_email_login` | If enabled, GitLab ignores everything after the first `@` in the LDAP username submitted by the user on sign-in. If you are using `uid: 'userPrincipalName'` on ActiveDirectory you need to disable this setting, because the userPrincipalName contains an `@`. | no | boolean | -| `block_auto_created_users` | To maintain tight control over the number of billable users on your GitLab installation, enable this setting to keep new users blocked until they have been cleared by an administrator (default: false). | no | boolean | -| `base` | Base where we can search for users. | yes | `'ou=people,dc=gitlab,dc=example'` or `'DC=mydomain,DC=com'` | -| `user_filter` | Filter LDAP users. Format: [RFC 4515](https://tools.ietf.org/search/rfc4515) Note: GitLab does not support `omniauth-ldap`'s custom filter syntax. | no | For examples, read [Examples of user filters](#examples-of-user-filters). | -| `lowercase_usernames` | If lowercase_usernames is enabled, GitLab converts the name to lower case. | no | boolean | +| Setting | Description | Required | Examples | +|--------------------|-------------|----------|----------| +| `label` | A human-friendly name for your LDAP server. It is displayed on your sign-in page. | **{check-circle}** Yes | `'Paris'` or `'Acme, Ltd.'` | +| `host` | IP address or domain name of your LDAP server. | **{check-circle}** Yes | `'ldap.mydomain.com'` | +| `port` | The port to connect with on your LDAP server. Always an integer, not a string. | **{check-circle}** Yes | `389` or `636` (for SSL) | +| `uid` | LDAP attribute for username. Should be the attribute, not the value that maps to the `uid`. | **{check-circle}** Yes | `'sAMAccountName'` or `'uid'` or `'userPrincipalName'` | +| `bind_dn` | The full DN of the user you bind with. | **{dotted-circle}** No | `'america\momo'` or `'CN=Gitlab,OU=Users,DC=domain,DC=com'` | +| `password` | The password of the bind user. | **{dotted-circle}** No | `'your_great_password'` | +| `encryption` | Encryption method. The `method` key is deprecated in favor of `encryption`. | **{check-circle}** Yes | `'start_tls'` or `'simple_tls'` or `'plain'` | +| `verify_certificates` | Enables SSL certificate verification if encryption method is `start_tls` or `simple_tls`. Defaults to true. | **{dotted-circle}** No | boolean | +| `timeout` | Set a timeout, in seconds, for LDAP queries. This helps avoid blocking a request if the LDAP server becomes unresponsive. A value of `0` means there is no timeout. (default: `10`) | **{dotted-circle}** No | `10` or `30` | +| `active_directory` | This setting specifies if LDAP server is Active Directory LDAP server. For non-AD servers it skips the AD specific queries. If your LDAP server is not AD, set this to false. | **{dotted-circle}** No | boolean | +| `allow_username_or_email_login` | If enabled, GitLab ignores everything after the first `@` in the LDAP username submitted by the user on sign-in. If you are using `uid: 'userPrincipalName'` on ActiveDirectory you need to disable this setting, because the userPrincipalName contains an `@`. | **{dotted-circle}** No | boolean | +| `block_auto_created_users` | To maintain tight control over the number of billable users on your GitLab installation, enable this setting to keep new users blocked until they have been cleared by an administrator (default: false). | **{dotted-circle}** No | boolean | +| `base` | Base where we can search for users. | **{check-circle}** Yes | `'ou=people,dc=gitlab,dc=example'` or `'DC=mydomain,DC=com'` | +| `user_filter` | Filter LDAP users. Format: [RFC 4515](https://tools.ietf.org/search/rfc4515) Note: GitLab does not support `omniauth-ldap`'s custom filter syntax. | **{dotted-circle}** No | For examples, read [Examples of user filters](#examples-of-user-filters). | +| `lowercase_usernames` | If enabled, GitLab converts the name to lower case. | **{dotted-circle}** No | boolean | #### Examples of user filters @@ -192,41 +184,44 @@ Some examples of the `user_filter` field syntax: ### SSL Configuration Settings -| Setting | Description | Required | Examples | -| ------- | ----------- | -------- | -------- | -| `ca_file` | Specifies the path to a file containing a PEM-format CA certificate, for example, if you need to use an internal CA. | no | `'/etc/ca.pem'` | -| `ssl_version` | Specifies the SSL version for OpenSSL to use, if the OpenSSL default is not appropriate. | no | `'TLSv1_1'` | -| `ciphers` | Specific SSL ciphers to use in communication with LDAP servers. | no | `'ALL:!EXPORT:!LOW:!aNULL:!eNULL:!SSLv2'` | -| `cert` | Client certificate | no | `'-----BEGIN CERTIFICATE----- <REDACTED> -----END CERTIFICATE -----'` | -| `key` | Client private key | no | `'-----BEGIN PRIVATE KEY----- <REDACTED> -----END PRIVATE KEY -----'` | +| Setting | Description | Required | Examples | +|---------------|-------------|----------|----------| +| `ca_file` | Specifies the path to a file containing a PEM-format CA certificate, for example, if you need to use an internal CA. | **{dotted-circle}** No | `'/etc/ca.pem'` | +| `ssl_version` | Specifies the SSL version for OpenSSL to use, if the OpenSSL default is not appropriate. | **{dotted-circle}** No | `'TLSv1_1'` | +| `ciphers` | Specific SSL ciphers to use in communication with LDAP servers. | **{dotted-circle}** No | `'ALL:!EXPORT:!LOW:!aNULL:!eNULL:!SSLv2'` | +| `cert` | Client certificate. | **{dotted-circle}** No | `'-----BEGIN CERTIFICATE----- <REDACTED> -----END CERTIFICATE -----'` | +| `key` | Client private key. | **{dotted-circle}** No | `'-----BEGIN PRIVATE KEY----- <REDACTED> -----END PRIVATE KEY -----'` | ### Attribute Configuration Settings -LDAP attributes that GitLab uses to create an account for the LDAP user. The specified attribute can either be the attribute name as a string (for example, `'mail'`), or an array of attribute names to try in order (for example, `['mail', 'email']`). Note that the user's LDAP sign-in is the attribute specified as `uid` above. +LDAP attributes that GitLab uses to create an account for the LDAP user. The specified +attribute can either be the attribute name as a string (for example, `'mail'`), or an +array of attribute names to try in order (for example, `['mail', 'email']`). Note that +the user's LDAP sign-in is the attribute specified as `uid` above. -| Setting | Description | Required | Examples | -| ------- | ----------- | -------- | -------- | -| `username` | The username is used in paths for the user's own projects (like `gitlab.example.com/username/project`) and when mentioning them in issues, merge request and comments (like `@username`). If the attribute specified for `username` contains an email address, the GitLab username is part of the email address before the `@`. | no | `['uid', 'userid', 'sAMAccountName']` | -| `email` | LDAP attribute for user email. | no | `['mail', 'email', 'userPrincipalName']` | -| `name` | LDAP attribute for user display name. If `name` is blank, the full name is taken from the `first_name` and `last_name`. | no | Attributes `'cn'`, or `'displayName'` commonly carry full names. Alternatively, you can force the use of `first_name` and `last_name` by specifying an absent attribute such as `'somethingNonExistent'`. | -| `first_name` | LDAP attribute for user first name. Used when the attribute configured for `name` does not exist. | no | `'givenName'` | -| `last_name` | LDAP attribute for user last name. Used when the attribute configured for `name` does not exist. | no | `'sn'` | +| Setting | Description | Required | Examples | +|--------------|-------------|----------|----------| +| `username` | The username is used in paths for the user's own projects (like `gitlab.example.com/username/project`) and when mentioning them in issues, merge request and comments (like `@username`). If the attribute specified for `username` contains an email address, the GitLab username is part of the email address before the `@`. | **{dotted-circle}** No | `['uid', 'userid', 'sAMAccountName']` | +| `email` | LDAP attribute for user email. | **{dotted-circle}** No | `['mail', 'email', 'userPrincipalName']` | +| `name` | LDAP attribute for user display name. If `name` is blank, the full name is taken from the `first_name` and `last_name`. | **{dotted-circle}** No | Attributes `'cn'`, or `'displayName'` commonly carry full names. Alternatively, you can force the use of `first_name` and `last_name` by specifying an absent attribute such as `'somethingNonExistent'`. | +| `first_name` | LDAP attribute for user first name. Used when the attribute configured for `name` does not exist. | **{dotted-circle}** No | `'givenName'` | +| `last_name` | LDAP attribute for user last name. Used when the attribute configured for `name` does not exist. | **{dotted-circle}** No | `'sn'` | ### LDAP Sync Configuration Settings **(PREMIUM SELF)** -| Setting | Description | Required | Examples | -| ------- | ----------- | -------- | -------- | -| `group_base` | Base used to search for groups. | no | `'ou=groups,dc=gitlab,dc=example'` | -| `admin_group` | The CN of a group containing GitLab administrators. Note: Not `cn=administrators` or the full DN. | no | `'administrators'` | -| `external_groups` | An array of CNs of groups containing users that should be considered external. Note: Not `cn=interns` or the full DN. | no | `['interns', 'contractors']` | -| `sync_ssh_keys` | The LDAP attribute containing a user's public SSH key. | no | `'sshPublicKey'` or false if not set | +| Setting | Description | Required | Examples | +|-------------------|-------------|----------|----------| +| `group_base` | Base used to search for groups. | **{dotted-circle}** No | `'ou=groups,dc=gitlab,dc=example'` | +| `admin_group` | The CN of a group containing GitLab administrators. Note: Not `cn=administrators` or the full DN. | **{dotted-circle}** No | `'administrators'` | +| `external_groups` | An array of CNs of groups containing users that should be considered external. Note: Not `cn=interns` or the full DN. | **{dotted-circle}** No | `['interns', 'contractors']` | +| `sync_ssh_keys` | The LDAP attribute containing a user's public SSH key. | **{dotted-circle}** No | `'sshPublicKey'` or false if not set | ### Set up LDAP user filter If you want to limit all GitLab access to a subset of the LDAP users on your LDAP server, the first step should be to narrow the configured `base`. However, -it is sometimes necessary to filter users further. In this case, you can set up -an LDAP user filter. The filter must comply with +it's sometimes necessary to further filter users. In this case, you can set +up an LDAP user filter. The filter must comply with [RFC 4515](https://tools.ietf.org/search/rfc4515). **Omnibus configuration** @@ -252,7 +247,7 @@ production: ``` If you want to limit access to the nested members of an Active Directory -group, you can use the following syntax: +group, use the following syntax: ```plaintext (memberOf:1.2.840.113556.1.4.1941:=CN=My Group,DC=Example,DC=com) @@ -260,11 +255,10 @@ group, you can use the following syntax: For more information about this "LDAP_MATCHING_RULE_IN_CHAIN" filter, see the following [Microsoft Search Filter Syntax](https://docs.microsoft.com/en-us/windows/win32/adsi/search-filter-syntax) document. -Support for nested members in the user filter should not be confused with +Support for nested members in the user filter shouldn't be confused with [group sync nested groups support](#supported-ldap-group-typesattributes). **(PREMIUM SELF)** -Please note that GitLab does not support the custom filter syntax used by -OmniAuth LDAP. +GitLab does not support the custom filter syntax used by OmniAuth LDAP. #### Escaping special characters @@ -342,7 +336,7 @@ an alternative such as SAML is preferred. This allows LDAP to be used for group sync, while also allowing your SAML identity provider to handle additional checks like custom 2FA. -When LDAP web sign in is disabled, users don't see an **LDAP** tab on the sign in page. +When LDAP web sign in is disabled, users don't see an **LDAP** tab on the sign-in page. This does not disable [using LDAP credentials for Git access](#git-password-authentication). **Omnibus configuration** @@ -373,7 +367,7 @@ Instead of having the LDAP integration credentials stored in plaintext in the co use an encrypted file for the LDAP credentials. To use this feature, you first need to enable [GitLab encrypted configuration](../../encrypted_configuration.md). -The encrypted configuration for LDAP exists in an encrypted YAML file. By default the file will be created at +The encrypted configuration for LDAP exists in an encrypted YAML file. By default the file is created at `shared/encrypted_configuration/ldap.yaml.enc`. This location is configurable in the GitLab configuration. The unencrypted contents of the file should be a subset of the secret settings from your `servers` block in the LDAP @@ -520,7 +514,9 @@ gitlab_rails['ldap_servers'] = { } ``` -If you configure multiple LDAP servers, use a unique naming convention for the `label` section of each entry. That label is used as the display name of the tab shown on the sign-in page. +If you configure multiple LDAP servers, use a unique naming convention for the +`label` section of each entry. That label is used as the display name of the tab +shown on the sign-in page. ## User sync **(PREMIUM SELF)** @@ -545,13 +541,13 @@ For more information, see [Bitmask Searches in LDAP](https://ctovswild.com/2009/ <!-- vale gitlab.Spelling = YES --> The user is set to an `ldap_blocked` state in GitLab if the previous conditions -fail. This means the user is not able to sign in or push/pull code. +fail. This means the user cannot sign in or push or pull code. The process also updates the following user information: -- Email address. -- If `sync_ssh_keys` is set, SSH public keys. -- If Kerberos is enabled, Kerberos identity. +- Email address +- SSH public keys (if `sync_ssh_keys` is set) +- Kerberos identity (if Kerberos is enabled) The LDAP sync process: @@ -643,19 +639,22 @@ or more LDAP group links](#adding-group-links). ### Adding group links **(PREMIUM SELF)** -For information on adding group links via CNs and filters, refer to [the GitLab groups documentation](../../../user/group/index.md#manage-group-memberships-via-ldap). +For information on adding group links by using CNs and filters, refer to the +[GitLab groups documentation](../../../user/group/index.md#manage-group-memberships-via-ldap). ### Administrator sync **(PREMIUM SELF)** As an extension of group sync, you can automatically manage your global GitLab administrators. Specify a group CN for `admin_group` and all members of the -LDAP group will be given administrator privileges. The configuration looks +LDAP group are given administrator privileges. The configuration looks like the following. NOTE: Administrators are not synced unless `group_base` is also specified alongside `admin_group`. Also, only specify the CN of the `admin_group`, as opposed to the full DN. +Additionally, note that if an LDAP user has an `admin` role, but is not a member of the `admin_group` +group, GitLab revokes their `admin` role when syncing. **Omnibus configuration** @@ -705,8 +704,10 @@ When enabled, the following applies: To enable it you need to: 1. [Enable LDAP](#configuration) -1. Go to **Admin Area > Settings > Visibility and access controls**. -1. Make sure the **Lock memberships to LDAP synchronization** checkbox is selected. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > General**. +1. Expand the **Visibility and access controls** section. +1. Ensure the **Lock memberships to LDAP synchronization** checkbox is selected. ### Adjusting LDAP group sync schedule **(PREMIUM SELF)** @@ -717,13 +718,13 @@ The values shown are in cron format. If needed, you can use a WARNING: Do not start the sync process too frequently as this could lead to multiple syncs running concurrently. This is primarily a concern -for installations with a large number of LDAP users. Please review the +for installations with a large number of LDAP users. Review the [LDAP group sync benchmark metrics](#benchmarks) to see how your installation compares before proceeding. You can manually configure LDAP group sync times by setting the following configuration values. The example below shows how to set group -sync to run once every 2 hours at the top of the hour. +sync to run once every two hours at the top of the hour. **Omnibus installations** @@ -749,7 +750,7 @@ sync to run once every 2 hours at the top of the hour. ### External groups **(PREMIUM SELF)** -Using the `external_groups` setting will allow you to mark all users belonging +Using the `external_groups` setting allows you to mark all users belonging to these groups as [external users](../../../user/permissions.md#external-users). Group membership is checked periodically through the `LdapGroupSync` background task. @@ -786,15 +787,14 @@ task. ### Group sync technical details -There is a lot going on with group sync 'under the hood'. This section -outlines what LDAP queries are executed and what behavior you can expect -from group sync. +This section outlines what LDAP queries are executed and what behavior you +can expect from group sync. Group member access are downgraded from a higher level if their LDAP group -membership changes. For example, if a user has 'Owner' rights in a group and the -next group sync reveals they should only have 'Developer' privileges, their +membership changes. For example, if a user the Owner role in a group and the +next group sync reveals they should only have the Developer role, their access is adjusted accordingly. The only exception is if the user is the -*last* owner in a group. Groups need at least one owner to fulfill +last owner in a group. Groups need at least one owner to fulfill administrative duties. #### Supported LDAP group types/attributes @@ -805,18 +805,20 @@ GitLab supports LDAP groups that use member attributes: - `submember` - `uniquemember` - `memberof` -- `memberuid`. +- `memberuid` + +This means group sync supports (at least) LDAP groups with the following object +classes: -This means group sync supports, at least, LDAP groups with the following object classes: -`groupOfNames`, `posixGroup`, and `groupOfUniqueNames`. +- `groupOfNames` +- `posixGroup` +- `groupOfUniqueNames` -Other object classes should work fine as long as members -are defined as one of the mentioned attributes. This also means GitLab supports -Microsoft Active Directory, Apple Open Directory, Open LDAP, and 389 Server. -Other LDAP servers should work, too. +Other object classes should work if members are defined as one of the +mentioned attributes. -Active Directory also supports nested groups. Group sync recursively -resolves membership if `active_directory: true` is set in the configuration file. +Active Directory supports nested groups. Group sync recursively resolves +membership if `active_directory: true` is set in the configuration file. ##### Nested group memberships @@ -842,7 +844,7 @@ Group sync was written to be as performant as possible. Data is cached, database queries are optimized, and LDAP queries are minimized. The last benchmark run revealed the following metrics: -For 20000 LDAP users, 11000 LDAP groups and 1000 GitLab groups with 10 +For 20,000 LDAP users, 11,000 LDAP groups, and 1,000 GitLab groups with 10 LDAP group links each: - Initial sync (no existing members assigned in GitLab) took 1.8 hours @@ -855,4 +857,4 @@ network and LDAP server response time affects these metrics. ## Troubleshooting -Please see our [administrator guide to troubleshooting LDAP](ldap-troubleshooting.md). +See our [administrator guide to troubleshooting LDAP](ldap-troubleshooting.md). diff --git a/doc/administration/auth/ldap/ldap-troubleshooting.md b/doc/administration/auth/ldap/ldap-troubleshooting.md index 1e6684751ed..acafe52007b 100644 --- a/doc/administration/auth/ldap/ldap-troubleshooting.md +++ b/doc/administration/auth/ldap/ldap-troubleshooting.md @@ -20,7 +20,7 @@ or `encryption: 'simple_tls'` and `port: 636`. #### Connection times out -If GitLab cannot reach your LDAP endpoint, you will see a message like this: +If GitLab cannot reach your LDAP endpoint, you see a message like this: ```plaintext Could not authenticate you from Ldapmain because "Connection timed out - user specified timeout". @@ -79,7 +79,7 @@ adapter.ldap_search(options) ``` For examples of how this is run, -[review the `Adapter` module](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/ee/gitlab/auth/ldap/adapter.rb). +[review the `Adapter` module](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/ee/gitlab/auth/ldap/adapter.rb). ### User sign-ins @@ -145,7 +145,8 @@ may see the following message: `Access denied for your LDAP account`. We have a workaround, based on toggling the access level of affected users: -1. As an administrator, go to **Admin Area > Overview > Users**. +1. As an administrator, on the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Overview > Users**. 1. Select the name of the affected user. 1. In the user's administrative page, press **Edit** on the top right of the page. 1. Change the user's access level from `Regular` to `Admin` (or vice versa), @@ -192,6 +193,24 @@ This shows you which user has this email address. One of two steps must be taken The user can do either of these steps [in their profile](../../../user/profile/index.md#access-your-user-profile) or an administrator can do it. +#### Projects limit errors + +The following errors indicate that a limit or restriction is activated, but an associated data +field contains no data: + +- `Projects limit can't be blank`. +- `Projects limit is not a number`. + +To resolve this: + +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, go to **Settings > General**. +1. Expand both of the following: + - **Account and limit**. + - **Sign-up restrictions**. +1. Check, for example, the **Default projects limit** or **Allowed domains for sign-ups** + fields and ensure that a relevant value is configured. + #### Debug LDAP user filter [`ldapsearch`](#ldapsearch) allows you to test your configured @@ -329,8 +348,9 @@ things to check to debug the situation. group](index.md#adding-group-links). - Check that the user has an LDAP identity: 1. Sign in to GitLab as an administrator user. - 1. Go to **Admin area > Users**. - 1. Search for the user + 1. On the top bar, select **Menu >** **{admin}** **Admin**. + 1. On the left sidebar, select **Overview > Users**. + 1. Search for the user. 1. Open the user by clicking their name. Do not click **Edit**. 1. Select the **Identities** tab. There should be an LDAP identity with an LDAP DN as the 'Identifier'. If not, this user hasn't signed in with @@ -367,7 +387,7 @@ the following are true: - The configured `admin_group` in the `gitlab.rb` is a CN, rather than a DN or an array. - This CN falls under the scope of the configured `group_base`. - The members of the `admin_group` have already signed into GitLab with their LDAP - credentials. GitLab will only grant this administrator access to the users whose + credentials. GitLab only grants this administrator access to the users whose accounts are already connected to LDAP. If all the above are true and the users are still not getting access, [run a manual @@ -396,7 +416,7 @@ output](#example-console-output-after-a-group-sync). ##### Example console output after a group sync **(PREMIUM SELF)** Like the output from the user sync, the output from the [manual group -sync](#sync-all-groups) will also be very verbose. However, it contains lots +sync](#sync-all-groups) is also very verbose. However, it contains lots of helpful information. Indicates the point where syncing actually begins: @@ -407,7 +427,7 @@ Started syncing 'ldapmain' provider for 'my_group' group The following entry shows an array of all user DNs GitLab sees in the LDAP server. Note that these are the users for a single LDAP group, not a GitLab group. If -you have multiple LDAP groups linked to this GitLab group, you will see multiple +you have multiple LDAP groups linked to this GitLab group, you see multiple log entries like this - one for each LDAP group. If you don't see an LDAP user DN in this log entry, LDAP is not returning the user when we do the lookup. Verify the user is actually in the LDAP group. @@ -421,7 +441,7 @@ Members in 'ldap_group_1' LDAP group: ["uid=john0,ou=people,dc=example,dc=com", "uid=mary4,ou=people,dc=example,dc=com"] ``` -Shortly after each of the above entries, you will see a hash of resolved member +Shortly after each of the above entries, you see a hash of resolved member access levels. This hash represents all user DNs GitLab thinks should have access to this group, and at which access level (role). This hash is additive, and more DNs may be added, or existing entries modified, based on additional @@ -462,21 +482,21 @@ Finally, the following entry says syncing has finished for this group: Finished syncing all providers for 'my_group' group ``` -Once all the configured group links have been synchronized, GitLab will look +Once all the configured group links have been synchronized, GitLab looks for any Administrators or External users to sync: ```shell Syncing admin users for 'ldapmain' provider ``` -The output will look similar to what happens with a single group, and then -this line will indicate the sync is finished: +The output looks similar to what happens with a single group, and then +this line indicates the sync is finished: ```shell Finished syncing admin users for 'ldapmain' provider ``` -If [administrator sync](index.md#administrator-sync) is not configured, you'll see a message +If [administrator sync](index.md#administrator-sync) is not configured, you see a message stating as such: ```shell @@ -502,8 +522,8 @@ group = Group.find_by(name: 'my_gitlab_group') EE::Gitlab::Auth::Ldap::Sync::Group.execute_all_providers(group) ``` -The output will be similar to -[that you'd get from syncing all groups](#example-console-output-after-a-group-sync). +The output is similar to +[that you get from syncing all groups](#example-console-output-after-a-group-sync). #### Query a group in LDAP **(PREMIUM SELF)** @@ -524,24 +544,25 @@ ldap_group.member_uids When an LDAP user is created in GitLab, their LDAP DN is stored for later reference. -If GitLab cannot find a user by their DN, it will fall back -to finding the user by their email. If the lookup is successful, GitLab will -update the stored DN to the new value so both values will now match what's in +If GitLab cannot find a user by their DN, it falls back +to finding the user by their email. If the lookup is successful, GitLab +updates the stored DN to the new value so both values now match what's in LDAP. -If the email has changed and the DN has not, GitLab will find the user with +If the email has changed and the DN has not, GitLab finds the user with the DN and update its own record of the user's email to match the one in LDAP. -However, if the primary email _and_ the DN change in LDAP, then GitLab will -have no way of identifying the correct LDAP record of the user and, as a -result, the user will be blocked. To rectify this, the user's existing -profile will have to be updated with at least one of the new values (primary +However, if the primary email _and_ the DN change in LDAP, then GitLab +has no way of identifying the correct LDAP record of the user and, as a +result, the user is blocked. To rectify this, the user's existing +profile must be updated with at least one of the new values (primary email or DN) so the LDAP record can be found. -The following script will update the emails for all provided users so they -won't be blocked or unable to access their accounts. +The following script updates the emails for all provided users so they +aren't blocked or unable to access their accounts. ->**NOTE**: The following script will require that any new accounts with the new +NOTE: +The following script requires that any new accounts with the new email address are removed first. This is because emails have to be unique in GitLab. Go to the [rails console](#rails-console) and then run: @@ -588,23 +609,23 @@ users, [see what to do when no users are found](#no-users-are-found). ### GitLab logs If a user account is blocked or unblocked due to the LDAP configuration, a -message will be [logged to `application.log`](../../logs.md#applicationlog). +message is [logged to `application.log`](../../logs.md#applicationlog). If there is an unexpected error during an LDAP lookup (configuration error, -timeout), the sign-in is rejected and a message will be [logged to +timeout), the sign-in is rejected and a message is [logged to `production.log`](../../logs.md#productionlog). ### ldapsearch -`ldapsearch` is a utility that will allow you to query your LDAP server. You can +`ldapsearch` is a utility that allows you to query your LDAP server. You can use it to test your LDAP settings and ensure that the settings you're using -will get you the results you expect. +get you the results you expect. When using `ldapsearch`, be sure to use the same settings you've already specified in your `gitlab.rb` configuration so you can confirm what happens when those exact settings are used. -Running this command on the GitLab host will also help confirm that there's no +Running this command on the GitLab host also helps confirm that there's no obstruction between the GitLab host and LDAP. For example, consider the following GitLab configuration: @@ -685,9 +706,9 @@ For instructions about how to use the rails console, refer to this #### Enable debug output -This will provide debug output that will be useful to see -what GitLab is doing and with what. This value is not persisted, and will only -be enabled for this session in the rails console. +This provides debug output that is useful to see +what GitLab is doing and with what. This value is not persisted, and is only +enabled for this session in the rails console. To enable debug output in the rails console, [enter the rails console](#rails-console) and run: diff --git a/doc/administration/auth/okta.md b/doc/administration/auth/okta.md index 88e9180b103..64b42339d19 100644 --- a/doc/administration/auth/okta.md +++ b/doc/administration/auth/okta.md @@ -1,5 +1,6 @@ --- redirect_to: '../../integration/saml.md' +remove_date: '2021-06-15' --- This document was moved to [another location](../../integration/saml.md). diff --git a/doc/administration/clusters/kas.md b/doc/administration/clusters/kas.md index 1b9638411de..8e5c162001e 100644 --- a/doc/administration/clusters/kas.md +++ b/doc/administration/clusters/kas.md @@ -127,5 +127,5 @@ time="2020-10-29T04:44:14Z" level=warning msg="Config: failed to fetch" agent_id It means that the path to the configuration project is incorrect, or the path to `config.yaml` inside the project is not valid. -To fix this, ensure that the paths to the configuration repo and to the `config.yaml` file +To fix this, ensure that the paths to the configuration repository and to the `config.yaml` file are correct. diff --git a/doc/administration/compliance.md b/doc/administration/compliance.md index 470dc1b4f9e..6b80ddbcdb5 100644 --- a/doc/administration/compliance.md +++ b/doc/administration/compliance.md @@ -4,7 +4,7 @@ group: Compliance info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Compliance features +# Compliance features **(FREE)** You can configure the following GitLab features to help ensure that your GitLab instance meets common compliance standards. Click a feature name for additional @@ -13,20 +13,20 @@ documentation. The [security features](../security/README.md) in GitLab may also help you meet relevant compliance standards. -|Feature |GitLab tier |GitLab SaaS | Product level | -| ---------| :--------: | :-------: | :-----------: | -|**[Restrict SSH Keys](../security/ssh_keys_restrictions.md)**<br>Control the technology and key length of SSH keys used to access GitLab|Free+||Instance| -|**[Granular user roles and flexible permissions](../user/permissions.md)**<br>Manage access and permissions with five different user roles and settings for external users. Set permissions according to people's role, rather than either read or write access to a repository. Don't share the source code with people that only need access to the issue tracker.|Free+|✓|Instance, Group, Project| -|**[Enforce TOS acceptance](../user/admin_area/settings/terms.md)**<br>Enforce your users accepting new terms of service by blocking GitLab traffic.|Free+||Instance| -|**[Email all users of a project, group, or entire server](../tools/email.md)**<br>An administrator can email groups of users based on project or group membership, or email everyone using the GitLab instance. This is great for scheduled maintenance or upgrades.|Premium+||Instance| -|**[Omnibus package supports log forwarding](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-forwarding)**<br>Forward your logs to a central system.|Premium+||Instance| -|**[Lock project membership to group](../user/group/index.md#prevent-members-from-being-added-to-a-group)**<br>Group owners can prevent new members from being added to projects within a group.|Premium+|✓|Group| -|**[LDAP group sync](auth/ldap/index.md#group-sync)**<br>GitLab Enterprise Edition gives administrators the ability to automatically sync groups and manage SSH keys, permissions, and authentication, so you can focus on building your product, not configuring your tools.|Premium+||Instance| -|**[LDAP group sync filters](auth/ldap/index.md#group-sync)**<br>GitLab Enterprise Edition Premium gives more flexibility to synchronize with LDAP based on filters, meaning you can leverage LDAP attributes to map GitLab permissions.|Premium+||Instance| -|**[Audit events](audit_events.md)**<br>To maintain the integrity of your code, GitLab Enterprise Edition Premium gives administrators the ability to view any modifications made within the GitLab server in an advanced audit events system, so you can control, analyze, and track every change.|Premium+|✓|Instance, Group, Project| -|**[Auditor users](auditor_users.md)**<br>Auditor users are users who are given read-only access to all projects, groups, and other resources on the GitLab instance.|Premium+||Instance| -|**[Credentials inventory](../user/admin_area/credentials_inventory.md)**<br>With a credentials inventory, GitLab administrators can keep track of the credentials used by all of the users in their GitLab instance. |Ultimate||Instance| -|**Separation of Duties using [Protected branches](../user/project/protected_branches.md#protected-branches-approval-by-code-owners) and [custom CI Configuration Paths](../ci/pipelines/settings.md#custom-cicd-configuration-path)**<br> GitLab Premium users can leverage the GitLab cross-project YAML configurations to define deployers of code and developers of code. View the [Separation of Duties Deploy Project](https://gitlab.com/guided-explorations/separation-of-duties-deploy/blob/master/README.md) and [Separation of Duties Project](https://gitlab.com/guided-explorations/separation-of-duties/blob/master/README.md) to see how to use this set up to define these roles.|Premium+|✓|Project| -|**[Compliance frameworks](../user/project/settings/index.md#compliance-frameworks)**<br>Create a custom compliance framework at the group level to describe the type of compliance requirements any child project needs to follow. |Premium+|✓|Group| -|**[Compliance pipelines](../user/project/settings/index.md#compliance-pipeline-configuration)**<br>Define a pipeline configuration to run for any projects with a given compliance framework.|Ultimate|✓|Group| -|**[Compliance dashboard](../user/compliance/compliance_dashboard/index.md)**<br>Quickly get visibility into the compliance posture of your organization.|Ultimate|✓|Group| +| Feature | GitLab tier | GitLab SaaS | Product level | +|----------|:-----------:|:-----------:|:-------------:| +|**[Restrict SSH Keys](../security/ssh_keys_restrictions.md)**<br>Control the technology and key length of SSH keys used to access GitLab. | Free+ | **{dotted-circle}** No | Instance | +|**[Granular user roles and flexible permissions](../user/permissions.md)**<br>Manage access and permissions with five different user roles and settings for external users. Set permissions according to people's role, rather than either read or write access to a repository. Don't share the source code with people that only need access to the issue tracker. | Free+ | **{check-circle}** Yes | Instance, Group, Project | +|**[Enforce TOS acceptance](../user/admin_area/settings/terms.md)**<br>Enforce your users accepting new terms of service by blocking GitLab traffic. | Free+ | **{dotted-circle}** No | Instance | +|**[Email all users of a project, group, or entire server](../tools/email.md)**<br>An administrator can email groups of users based on project or group membership, or email everyone using the GitLab instance. This is great for scheduled maintenance or upgrades. | Premium+ | **{dotted-circle}** No | Instance | +|**[Omnibus package supports log forwarding](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-forwarding)**<br>Forward your logs to a central system. | Premium+ | **{dotted-circle}** No | Instance | +|**[Lock project membership to group](../user/group/index.md#prevent-members-from-being-added-to-a-group)**<br>Group owners can prevent new members from being added to projects within a group. | Premium+ | **{check-circle}** Yes | Group | +|**[LDAP group sync](auth/ldap/index.md#group-sync)**<br>GitLab Enterprise Edition gives administrators the ability to automatically sync groups and manage SSH keys, permissions, and authentication, so you can focus on building your product, not configuring your tools. | Premium+ | **{dotted-circle}** No | Instance | +|**[LDAP group sync filters](auth/ldap/index.md#group-sync)**<br>GitLab Enterprise Edition Premium gives more flexibility to synchronize with LDAP based on filters, meaning you can leverage LDAP attributes to map GitLab permissions. | Premium+ | **{dotted-circle}** No | Instance | +|**[Audit events](audit_events.md)**<br>To maintain the integrity of your code, GitLab Enterprise Edition Premium gives administrators the ability to view any modifications made within the GitLab server in an advanced audit events system, so you can control, analyze, and track every change. | Premium+ | **{check-circle}** Yes | Instance, Group, Project | +|**[Auditor users](auditor_users.md)**<br>Auditor users are users who are given read-only access to all projects, groups, and other resources on the GitLab instance. | Premium+ | **{dotted-circle}** No | Instance | +|**[Credentials inventory](../user/admin_area/credentials_inventory.md)**<br>With a credentials inventory, GitLab administrators can keep track of the credentials used by all of the users in their GitLab instance. | Ultimate | **{dotted-circle}** No | Instance | +|**Separation of Duties using [Protected branches](../user/project/protected_branches.md#protected-branches-approval-by-code-owners) and [custom CI Configuration Paths](../ci/pipelines/settings.md#custom-cicd-configuration-file)**<br> GitLab Premium users can leverage the GitLab cross-project YAML configurations to define deployers of code and developers of code. View the [Separation of Duties Deploy Project](https://gitlab.com/guided-explorations/separation-of-duties-deploy/blob/master/README.md) and [Separation of Duties Project](https://gitlab.com/guided-explorations/separation-of-duties/blob/master/README.md) to see how to use this set up to define these roles. | Premium+ | **{check-circle}** Yes | Project | +|**[Compliance frameworks](../user/project/settings/index.md#compliance-frameworks)**<br>Create a custom compliance framework at the group level to describe the type of compliance requirements any child project needs to follow. | Premium+ | **{check-circle}** Yes | Group | +|**[Compliance pipelines](../user/project/settings/index.md#compliance-pipeline-configuration)**<br>Define a pipeline configuration to run for any projects with a given compliance framework. | Ultimate | **{check-circle}** Yes | Group | +|**[Compliance dashboard](../user/compliance/compliance_dashboard/index.md)**<br>Quickly get visibility into the compliance posture of your organization. | Ultimate | **{check-circle}** Yes | Group | diff --git a/doc/administration/configure.md b/doc/administration/configure.md new file mode 100644 index 00000000000..12a8f721ccf --- /dev/null +++ b/doc/administration/configure.md @@ -0,0 +1,16 @@ +--- +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: reference +--- + +# Configure your GitLab installation + +Customize and configure your self-managed GitLab installation. + +- [Authentication](auth/README.md) +- [Configuration](../user/admin_area/index.md) +- [Repository storage](repository_storage_paths.md) +- [Geo](geo/index.md) +- [Packages](packages/index.md) diff --git a/doc/administration/consul.md b/doc/administration/consul.md index a748259aff0..c88047c4c61 100644 --- a/doc/administration/consul.md +++ b/doc/administration/consul.md @@ -15,11 +15,17 @@ turn communicate with the servers. GitLab Premium includes a bundled version of [Consul](https://www.consul.io/) a service networking solution that you can manage by using `/etc/gitlab/gitlab.rb`. +## Prerequisites + +Before configuring Consul: + +1. Review the [reference architecture](reference_architectures/index.md#available-reference-architectures) + documentation to determine the number of Consul server nodes you should have. +1. If necessary, ensure the [appropriate ports are open](https://docs.gitlab.com/omnibus/package-information/defaults.html#ports) in your firewall. + ## Configure the Consul nodes -After you review the [reference architecture](reference_architectures/index.md#available-reference-architectures) -documentation to determine the number of Consul server nodes you should have, -on _each_ Consul server node: +On _each_ Consul server node: 1. Follow the instructions to [install](https://about.gitlab.com/install/) GitLab by choosing your preferred platform, but do not supply the @@ -80,6 +86,15 @@ within each node. The command will return an empty array if the cluster is healt curl "http://127.0.0.1:8500/v1/health/state/critical" ``` +If the Consul version has changed, you'll see a notice at the end of `gitlab-ctl reconfigure` +informing you that Consul needs to be restarted for the new version to be used. + +Restart Consul one node at a time: + +```shell +sudo gitlab-ctl restart consul +``` + Consul nodes communicate using the raft protocol. If the current leader goes offline, there needs to be a leader election. A leader node must exist to facilitate synchronization across the cluster. If too many nodes go offline at the same time, diff --git a/doc/administration/database_load_balancing.md b/doc/administration/database_load_balancing.md index bd34a82f688..9c1ed9b3477 100644 --- a/doc/administration/database_load_balancing.md +++ b/doc/administration/database_load_balancing.md @@ -4,9 +4,10 @@ group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Database Load Balancing **(PREMIUM SELF)** +# Database Load Balancing **(FREE SELF)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/1283) in [GitLab Premium](https://about.gitlab.com/pricing/) 9.0. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/1283) in [GitLab Premium](https://about.gitlab.com/pricing/) 9.0. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/60894) from GitLab Premium to GitLab Free in 14.0. Distribute read-only queries among multiple database servers. @@ -21,8 +22,6 @@ component may increase reliability and availability through redundancy. When database load balancing is enabled in GitLab, the load is balanced using a simple round-robin algorithm, without any external dependencies such as Redis. -Load balancing is not enabled for Sidekiq as this would lead to consistency -problems, and Sidekiq mostly performs writes anyway. In the following image, you can see the load is balanced rather evenly among all the secondaries (`db4`, `db5`, `db6`). Because `SELECT` queries are not @@ -105,6 +104,32 @@ the following. This will balance the load between `host1.example.com` and 1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. +### Enable the load balancer for Sidekiq + +Sidekiq mostly writes to the database, which means that most of its traffic hits the +primary database. + +Some background jobs can use database replicas to read application state. +This allows to offload the primary database. + +Load balancing is disabled by default in Sidekiq. When enabled, we can define +[the data consistency](../development/sidekiq_style_guide.md#job-data-consistency) +requirements for a specific job. + +To enable it, define the `ENABLE_LOAD_BALANCING_FOR_SIDEKIQ` variable to the environment, as shown below. + +For Omnibus installations: + +```ruby +gitlab_rails['env'] = {"ENABLE_LOAD_BALANCING_FOR_SIDEKIQ" => "true"} +``` + +For installations from source: + +```shell +export ENABLE_LOAD_BALANCING_FOR_SIDEKIQ="true" +``` + ## Service Discovery > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/5883) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.0. @@ -171,28 +196,6 @@ Some nameservers (like [Consul](https://www.consul.io/docs/discovery/dns#udp-bas queried over UDP. To overcome this issue, you can use TCP for querying by setting `use_tcp` to `true`. -### Forking - -NOTE: -Starting with GitLab 13.0, Puma is the default web server used in GitLab -all-in-one package based installations as well as GitLab Helm chart deployments. - -If you use an application server that forks, such as Unicorn, you _have to_ -update your Unicorn configuration to start service discovery _after_ a fork. -Failure to do so leads to service discovery only running in the parent -process. If you are using Unicorn, then you can add the following to your -Unicorn configuration file: - -```ruby -after_fork do |server, worker| - defined?(Gitlab::Database::LoadBalancing) && - Gitlab::Database::LoadBalancing.start_service_discovery -end -``` - -This ensures that service discovery is started in both the parent and all -child processes. - ## Balancing queries Read-only `SELECT` queries balance among all the secondary hosts. diff --git a/doc/administration/environment_variables.md b/doc/administration/environment_variables.md index a168584e754..057abce0ed5 100644 --- a/doc/administration/environment_variables.md +++ b/doc/administration/environment_variables.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Environment variables +# Environment variables **(FREE SELF)** GitLab exposes certain environment variables which can be used to override their defaults values. @@ -32,8 +32,6 @@ You can use the following environment variables to override certain values: | `GITLAB_HOST` | string | The full URL of the GitLab server (including `http://` or `https://`). | | `GITLAB_ROOT_PASSWORD` | string | Sets the password for the `root` user on installation. | | `GITLAB_SHARED_RUNNERS_REGISTRATION_TOKEN` | string | Sets the initial registration token used for runners. | -| `GITLAB_UNICORN_MEMORY_MAX` | integer | The maximum memory threshold (in bytes) for the [unicorn-worker-killer](operations/unicorn.md#unicorn-worker-killer). | -| `GITLAB_UNICORN_MEMORY_MIN` | integer | The minimum memory threshold (in bytes) for the [unicorn-worker-killer](operations/unicorn.md#unicorn-worker-killer). | | `RAILS_ENV` | string | The Rails environment; can be one of `production`, `development`, `staging`, or `test`. | | `UNSTRUCTURED_RAILS_LOG` | string | Enables the unstructured log in addition to JSON logs (defaults to `true`). | diff --git a/doc/administration/external_pipeline_validation.md b/doc/administration/external_pipeline_validation.md index 89543e446ac..9fc65fdd0b5 100644 --- a/doc/administration/external_pipeline_validation.md +++ b/doc/administration/external_pipeline_validation.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference, howto --- @@ -17,7 +17,7 @@ data as payload. The response code from the external service determines if GitLa should accept or reject the pipeline. If the response is: - `200`, the pipeline is accepted. -- `4XX`, the pipeline is rejected. +- `406`, the pipeline is rejected. - Other codes, the pipeline is accepted and logged. If there's an error or the request times out, the pipeline is accepted. @@ -74,7 +74,9 @@ required number of seconds. "id": { "type": "integer" }, "username": { "type": "string" }, "email": { "type": "string" }, - "created_at": { "type": ["string", "null"], "format": "date-time" } + "created_at": { "type": ["string", "null"], "format": "date-time" }, + "current_sign_in_ip": { "type": ["string", "null"] }, + "last_sign_in_ip": { "type": ["string", "null"] } } }, "pipeline": { @@ -126,6 +128,17 @@ required number of seconds. "plan": { "type": "string" }, "trial": { "type": "boolean" } } + }, + "provisioning_group": { + "type": "object", + "required": [ + "plan", + "trial" + ], + "properties": { + "plan": { "type": "string" }, + "trial": { "type": "boolean" } + } } } } diff --git a/doc/administration/feature_flags.md b/doc/administration/feature_flags.md index 9ba50cfbf2e..44abf4a875d 100644 --- a/doc/administration/feature_flags.md +++ b/doc/administration/feature_flags.md @@ -127,6 +127,5 @@ Feature.disabled?(:my_awesome_feature) => false ``` -When the feature is ready, GitLab will remove the feature flag, the option for -enabling and disabling it will no longer exist, and the feature will become -available in all instances. +When the feature is ready, GitLab removes the feature flag, and the option for +enabling and disabling it no longer exists. The feature becomes available in all instances. diff --git a/doc/administration/file_hooks.md b/doc/administration/file_hooks.md index c60f0040496..f73c961f541 100644 --- a/doc/administration/file_hooks.md +++ b/doc/administration/file_hooks.md @@ -33,7 +33,7 @@ see the [system hooks](../system_hooks/system_hooks.md) documentation. The file hooks must be placed directly into the `file_hooks` directory, subdirectories are ignored. There is an -[`example` directory inside `file_hooks`](https://gitlab.com/gitlab-org/gitlab/tree/master/file_hooks/examples) +[`example` directory inside `file_hooks`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/file_hooks/examples) where you can find some basic examples. Follow the steps below to set up a custom hook: @@ -63,8 +63,11 @@ need to restart GitLab to apply a new file hook. If a file hook executes with non-zero exit code or GitLab fails to execute it, a message is logged to: -- `gitlab-rails/plugin.log` in an Omnibus installation. -- `log/plugin.log` in a source installation. +- `gitlab-rails/file_hook.log` in an Omnibus installation. +- `log/file_hook.log` in a source installation. + +NOTE: +Before 14.0 release, the file name was `plugin.log` ## Creating file hooks @@ -79,7 +82,7 @@ require 'json' require 'mail' # The incoming variables are in JSON format so we need to parse it first. -ARGS = JSON.parse(STDIN.read) +ARGS = JSON.parse($stdin.read) # We only want to trigger this file hook on the event project_create return unless ARGS['event_name'] == 'project_create' diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index 7c6f4a32b57..f6f88e9b193 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -7,17 +7,14 @@ type: howto # Disaster Recovery (Geo) **(PREMIUM SELF)** -Geo replicates your database, your Git repositories, and few other assets. -We will support and replicate more data in the future, that will enable you to -failover with minimal effort, in a disaster situation. - -See [Geo limitations](../index.md#limitations) for more information. +Geo replicates your database, your Git repositories, and few other assets, +but there are some [limitations](../index.md#limitations). WARNING: Disaster recovery for multi-secondary configurations is in **Alpha**. For the latest updates, check the [Disaster Recovery epic for complete maturity](https://gitlab.com/groups/gitlab-org/-/epics/3574). Multi-secondary configurations require the complete re-synchronization and re-configuration of all non-promoted secondaries and -will cause downtime. +causes downtime. ## Promoting a **secondary** Geo node in single-secondary configurations @@ -91,13 +88,16 @@ Note the following when promoting a secondary: before proceeding. If the secondary node [has been paused](../../geo/index.md#pausing-and-resuming-replication), the promotion performs a point-in-time recovery to the last known state. - Data that was created on the primary while the secondary was paused will be lost. + Data that was created on the primary while the secondary was paused is lost. - A new **secondary** should not be added at this time. If you want to add a new **secondary**, do this after you have completed the entire process of promoting the **secondary** to the **primary**. - If you encounter an `ActiveRecord::RecordInvalid: Validation failed: Name has already been taken` error message during this process, for more information, see this [troubleshooting advice](../replication/troubleshooting.md#fixing-errors-during-a-failover-or-when-promoting-a-secondary-to-a-primary-node). +- If you run into errors when using `--force` or `--skip-preflight-checks` before 13.5 during this process, + for more information, see this + [troubleshooting advice](../replication/troubleshooting.md#errors-when-using---skip-preflight-checks-or---force). #### Promoting a **secondary** node running on a single machine @@ -243,6 +243,7 @@ required: sets the database to read-write. The instructions vary depending on where your database is hosted: - [Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html#USER_ReadRepl.Promote) - [Azure PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/howto-read-replicas-portal#stop-replication) + - [Google Cloud SQL](https://cloud.google.com/sql/docs/mysql/replication/manage-replicas#promote-replica) - For other external PostgreSQL databases, save the following script in your secondary node, for example `/tmp/geo_promote.sh`, and modify the connection parameters to match your environment. Then, execute it to promote the replica: @@ -493,7 +494,7 @@ must disable the **primary** site: WARNING: If the secondary site [has been paused](../../geo/index.md#pausing-and-resuming-replication), this performs a point-in-time recovery to the last known state. -Data that was created on the primary while the secondary was paused will be lost. +Data that was created on the primary while the secondary was paused is lost. 1. SSH in to the database node in the **secondary** and trigger PostgreSQL to promote to read-write: @@ -509,7 +510,7 @@ Data that was created on the primary while the secondary was paused will be lost `geo_secondary_role`: NOTE: - Depending on your architecture these steps will need to be run on any GitLab node that is external to the **secondary** Kubernetes cluster. + Depending on your architecture, these steps need to run on any GitLab node that is external to the **secondary** Kubernetes cluster. ```ruby ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. @@ -537,13 +538,13 @@ Data that was created on the primary while the secondary was paused will be lost 1. Update the existing cluster configuration. - You can retrieve the existing config with Helm: + You can retrieve the existing configuration with Helm: ```shell helm --namespace gitlab get values gitlab-geo > gitlab.yaml ``` - The existing config will contain a section for Geo that should resemble: + The existing configuration contains a section for Geo that should resemble: ```yaml geo: @@ -560,9 +561,9 @@ Data that was created on the primary while the secondary was paused will be lost To promote the **secondary** cluster to a **primary** cluster, update `role: secondary` to `role: primary`. - You can remove the entire `psql` section if the cluster will remain as a primary site, this refers to the tracking database and will be ignored whilst the cluster is acting as a primary site. + If the cluster remains as a primary site, you can remove the entire `psql` section; it refers to the tracking database and is ignored whilst the cluster is acting as a primary site. - Update the cluster with the new config: + Update the cluster with the new configuration: ```shell helm upgrade --install --version <current Chart version> gitlab-geo gitlab/gitlab --namespace gitlab -f gitlab.yaml diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md index bd8467f5437..d50078da172 100644 --- a/doc/administration/geo/disaster_recovery/planned_failover.md +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -35,7 +35,7 @@ required scheduled maintenance period significantly. A common strategy for keeping this period as short as possible for data stored in files is to use `rsync` to transfer the data. An initial `rsync` can be performed ahead of the maintenance window; subsequent `rsync`s (including a -final transfer inside the maintenance window) will then transfer only the +final transfer inside the maintenance window) then transfers only the *changes* between the **primary** node and the **secondary** nodes. Repository-centric strategies for using `rsync` effectively can be found in the @@ -50,7 +50,7 @@ this command reports `ERROR - Replication is not up-to-date` even if replication is actually up-to-date. This bug was fixed in GitLab 13.8 and later. -Run this command to list out all preflight checks and automatically check if replication and verification are complete before scheduling a planned failover to ensure the process will go smoothly: +Run this command to list out all preflight checks and automatically check if replication and verification are complete before scheduling a planned failover to ensure the process goes smoothly: ```shell gitlab-ctl promotion-preflight-checks @@ -73,7 +73,7 @@ In GitLab 12.4, you can optionally allow GitLab to manage replication of Object Database settings are automatically replicated to the **secondary** node, but the `/etc/gitlab/gitlab.rb` file must be set up manually, and differs between nodes. If features such as Mattermost, OAuth or LDAP integration are enabled -on the **primary** node but not the **secondary** node, they will be lost during failover. +on the **primary** node but not the **secondary** node, they are lost during failover. Review the `/etc/gitlab/gitlab.rb` file for both nodes and ensure the **secondary** node supports everything the **primary** node does **before** scheduling a planned failover. @@ -119,7 +119,7 @@ time to complete If any objects are failing to replicate, this should be investigated before scheduling the maintenance window. Following a planned failover, anything that -failed to replicate will be **lost**. +failed to replicate is **lost**. You can use the [Geo status API](../../../api/geo_nodes.md#retrieve-project-sync-or-verification-failures-that-occurred-on-the-current-node) to review failed objects and the reasons for failure. @@ -136,9 +136,9 @@ This [content was moved to another location](background_verification.md). On the **primary** node, navigate to **Admin Area > Messages**, add a broadcast message. You can check under **Admin Area > Geo** to estimate how long it -will take to finish syncing. An example message would be: +takes to finish syncing. An example message would be: -> A scheduled maintenance will take place at XX:XX UTC. We expect it to take +> A scheduled maintenance takes place at XX:XX UTC. We expect it to take > less than 1 hour. ## Prevent updates to the **primary** node @@ -151,7 +151,7 @@ be disabled on the primary site: 1. Disable non-Geo periodic background jobs on the **primary** node by navigating to **Admin Area > Monitoring > Background Jobs > Cron**, pressing `Disable All`, and then pressing `Enable` for the `geo_sidekiq_cron_config_worker` cron job. - This job will re-enable several other cron jobs that are essential for planned + This job re-enables several other cron jobs that are essential for planned failover to complete successfully. ## Finish replicating and verifying all data @@ -161,7 +161,7 @@ be disabled on the primary site: 1. On the **primary** node, navigate to **Admin Area > Monitoring > Background Jobs > Queues** and wait for all queues except those with `geo` in the name to drop to 0. These queues contain work that has been submitted by your users; failing over - before it is completed will cause the work to be lost. + before it is completed, causes the work to be lost. 1. On the **primary** node, navigate to **Admin Area > Geo** and wait for the following conditions to be true of the **secondary** node you are failing over to: @@ -176,15 +176,15 @@ be disabled on the primary site: to verify the integrity of CI artifacts, LFS objects, and uploads in file storage. -At this point, your **secondary** node will contain an up-to-date copy of everything the -**primary** node has, meaning nothing will be lost when you fail over. +At this point, your **secondary** node contains an up-to-date copy of everything the +**primary** node has, meaning nothing was lost when you fail over. ## Promote the **secondary** node Finally, follow the [Disaster Recovery docs](index.md) to promote the -**secondary** node to a **primary** node. This process will cause a brief outage on the **secondary** node, and users may need to log in again. +**secondary** node to a **primary** node. This process causes a brief outage on the **secondary** node, and users may need to log in again. -Once it is completed, the maintenance window is over! Your new **primary** node will now +Once it is completed, the maintenance window is over! Your new **primary** node, now begin to diverge from the old one. If problems do arise at this point, failing back to the old **primary** node [is possible](bring_primary_back.md), but likely to result in the loss of any data uploaded to the new **primary** in the meantime. diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md index 780e391973c..295a448c432 100644 --- a/doc/administration/geo/index.md +++ b/doc/administration/geo/index.md @@ -27,7 +27,7 @@ to clone and fetch large repositories, speeding up development. For a video introduction to Geo, see [Introduction to GitLab Geo - GitLab Features](https://www.youtube.com/watch?v=-HDLxSjEh6w). -To make sure you're using the right version of the documentation, navigate to [the Geo page on GitLab.com](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/administration/geo/index.md) and choose the appropriate release from the **Switch branch/tag** dropdown. For example, [`v13.7.6-ee`](https://gitlab.com/gitlab-org/gitlab/-/blob/v13.7.6-ee/doc/administration/geo/index.md). +To make sure you're using the right version of the documentation, navigate to [the Geo page on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/administration/geo/index.md) and choose the appropriate release from the **Switch branch/tag** dropdown. For example, [`v13.7.6-ee`](https://gitlab.com/gitlab-org/gitlab/-/blob/v13.7.6-ee/doc/administration/geo/index.md). Geo uses a set of defined terms that is described in the [Geo Glossary](glossary.md), please familiarize yourself with those terms. @@ -56,11 +56,12 @@ Geo provides: ### Gitaly Cluster Geo should not be confused with [Gitaly Cluster](../gitaly/praefect.md). For more information about -the difference between Geo and Gitaly Cluster, see [Gitaly Cluster compared to Geo](../gitaly/index.md#gitaly-cluster-compared-to-geo). +the difference between Geo and Gitaly Cluster, see +[How does Gitaly Cluster compare to Geo?](../gitaly/faq.md#how-does-gitaly-cluster-compare-to-geo). ## How it works -Your Geo instance can be used for cloning and fetching projects, in addition to reading any data. This will make working with large repositories over large distances much faster. +Your Geo instance can be used for cloning and fetching projects, in addition to reading any data. This makes working with large repositories over large distances much faster.  @@ -121,7 +122,7 @@ The following are required to run Geo: The following operating systems are known to ship with a current version of OpenSSH: - [CentOS](https://www.centos.org) 7.4+ - [Ubuntu](https://ubuntu.com) 16.04+ -- PostgreSQL 11+ with [Streaming Replication](https://wiki.postgresql.org/wiki/Streaming_Replication) +- PostgreSQL 12+ with [Streaming Replication](https://wiki.postgresql.org/wiki/Streaming_Replication) - Git 2.9+ - Git-lfs 2.4.2+ on the user side when using LFS - All sites must run the same GitLab version. @@ -150,17 +151,17 @@ NOTE: When using HTTP or HTTPS proxying, your load balancer must be configured to pass through the `Connection` and `Upgrade` hop-by-hop headers. See the [web terminal](../integration/terminal.md) integration guide for more details. NOTE: -When using HTTPS protocol for port 443, you will need to add an SSL certificate to the load balancers. +When using HTTPS protocol for port 443, you need to add an SSL certificate to the load balancers. If you wish to terminate SSL at the GitLab application server instead, use TCP protocol. ### LDAP -We recommend that if you use LDAP on your **primary** site, you also set up secondary LDAP servers on each **secondary** site. Otherwise, users will not be able to perform Git operations over HTTP(s) on the **secondary** site using HTTP Basic Authentication. However, Git via SSH and personal access tokens will still work. +We recommend that if you use LDAP on your **primary** site, you also set up secondary LDAP servers on each **secondary** site. Otherwise, users are unable to perform Git operations over HTTP(s) on the **secondary** site using HTTP Basic Authentication. However, Git via SSH and personal access tokens still works. NOTE: -It is possible for all **secondary** sites to share an LDAP server, but additional latency can be an issue. Also, consider what LDAP server will be available in a [disaster recovery](disaster_recovery/index.md) scenario if a **secondary** site is promoted to be a **primary** site. +It is possible for all **secondary** sites to share an LDAP server, but additional latency can be an issue. Also, consider what LDAP server is available in a [disaster recovery](disaster_recovery/index.md) scenario if a **secondary** site is promoted to be a **primary** site. -Check for instructions on how to set up replication in your LDAP service. Instructions will be different depending on the software or service used. For example, OpenLDAP provides [these instructions](https://www.openldap.org/doc/admin24/replication.html). +Check for instructions on how to set up replication in your LDAP service. Instructions are different depending on the software or service used. For example, OpenLDAP provides [these instructions](https://www.openldap.org/doc/admin24/replication.html). ### Geo Tracking Database @@ -179,9 +180,9 @@ This daemon: - Reads a log of events replicated by the **primary** site to the **secondary** database instance. - Updates the Geo Tracking Database instance with changes that need to be executed. -When something is marked to be updated in the tracking database instance, asynchronous jobs running on the **secondary** site will execute the required operations and update the state. +When something is marked to be updated in the tracking database instance, asynchronous jobs running on the **secondary** site execute the required operations and update the state. -This new architecture allows GitLab to be resilient to connectivity issues between the sites. It doesn't matter how long the **secondary** site is disconnected from the **primary** site as it will be able to replay all the events in the correct order and become synchronized with the **primary** site again. +This new architecture allows GitLab to be resilient to connectivity issues between the sites. It doesn't matter how long the **secondary** site is disconnected from the **primary** site as it is able to replay all the events in the correct order and become synchronized with the **primary** site again. ## Limitations @@ -196,7 +197,7 @@ This list of limitations only reflects the latest version of GitLab. If you are - Object pools for forked project deduplication work only on the **primary** site, and are duplicated on the **secondary** site. - GitLab Runners cannot register with a **secondary** site. Support for this is [planned for the future](https://gitlab.com/gitlab-org/gitlab/-/issues/3294). - Configuring Geo **secondary** sites to [use high-availability configurations of PostgreSQL](https://gitlab.com/groups/gitlab-org/-/epics/2536) is currently in **alpha** support. -- [Selective synchronization](replication/configuration.md#selective-synchronization) only limits what repositories are replicated. The entire PostgreSQL data is still replicated. Selective synchronization is not built to accomodate compliance / export control use cases. +- [Selective synchronization](replication/configuration.md#selective-synchronization) only limits what repositories are replicated. The entire PostgreSQL data is still replicated. Selective synchronization is not built to accommodate compliance / export control use cases. ### Limitations on replication/verification @@ -280,7 +281,7 @@ For an example of how to set up a location-aware Git remote URL with AWS Route53 ### Backfill -Once a **secondary** site is set up, it will start replicating missing data from +Once a **secondary** site is set up, it starts replicating missing data from the **primary** site in a process known as **backfill**. You can monitor the synchronization process on each Geo site from the **primary** site's **Geo Nodes** dashboard in your browser. diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index e2f12cbd8dc..a1461a64518 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -47,8 +47,8 @@ verification methods: | Blobs | Container registry _(file system)_ | Geo with API/Docker API | _Not implemented_ | | Blobs | Container registry _(object storage)_ | Geo with API/Managed/Docker API (*2*) | _Not implemented_ | | Blobs | Package registry _(file system)_ | Geo with API | SHA256 checksum | -| Blobs | Package registry _(object storage)_ | Geo with API/Managed (*2*) | SHA256 checksum | -| Blobs | Versioned Terraform State _(file system)_ | Geo with API | _Not implemented_ | +| Blobs | Package registry _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | +| Blobs | Versioned Terraform State _(file system)_ | Geo with API | SHA256 checksum | | Blobs | Versioned Terraform State _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | | Blobs | External Merge Request Diffs _(file system)_ | Geo with API | _Not implemented_ | | Blobs | External Merge Request Diffs _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | @@ -180,7 +180,7 @@ successfully, you must replicate their data using some other means. |[Project wiki repository](../../../user/project/wiki/) | **Yes** (10.2) | **Yes** (10.7) | No | | |[Group wiki repository](../../../user/project/wiki/index.md#group-wikis) | [**Yes** (13.10)](https://gitlab.com/gitlab-org/gitlab/-/issues/208147) | No | No | Behind feature flag `geo_group_wiki_repository_replication`, enabled by default. | |[Uploads](../../uploads.md) | **Yes** (10.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | No | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. | -|[LFS objects](../../lfs/index.md) | **Yes** (10.2) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8922) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. GitLab versions 11.11.x and 12.0.x are affected by [a bug that prevents any new LFS objects from replicating](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). | +|[LFS objects](../../lfs/index.md) | **Yes** (10.2) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8922) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. GitLab versions 11.11.x and 12.0.x are affected by [a bug that prevents any new LFS objects from replicating](https://gitlab.com/gitlab-org/gitlab/-/issues/32696).<br /><br />Behind feature flag `geo_lfs_object_replication`, enabled by default. | |[Personal snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | No | | |[Project snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | No | | |[CI job artifacts (other than Job Logs)](../../../ci/pipelines/job_artifacts.md) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. | @@ -189,49 +189,25 @@ successfully, you must replicate their data using some other means. |[Object pools for forked project deduplication](../../../development/git_object_deduplication.md) | **Yes** | No | No | | |[Container Registry](../../packages/container_registry.md) | **Yes** (12.3) | No | No | Disabled by default. See [instructions](docker_registry.md) to enable. | |[Content in object storage (beta)](object_storage.md) | **Yes** (12.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/13845) | No | | -|[Project designs repository](../../../user/project/issues/design_management.md) | **Yes** (12.7) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/32467) | Via Object Storage provider if supported. Native Geo support (Beta). | | -|[Package Registry for npm](../../../user/packages/npm_registry/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | -|[Package Registry for Maven](../../../user/packages/maven_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | -|[Package Registry for Conan](../../../user/packages/conan_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | -|[Package Registry for NuGet](../../../user/packages/nuget_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | -|[Package Registry for PyPI](../../../user/packages/pypi_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | -|[Package Registry for Composer](../../../user/packages/composer_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | -|[Package Registry for generic packages](../../../user/packages/generic_packages/index.md) | **Yes** (13.5) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | -|[Versioned Terraform State](../../terraform_state.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_terraform_state_version_replication`, enabled by default. | -|[External merge request diffs](../../merge_request_diffs.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_merge_request_diff_replication`, enabled by default. | +|[Project designs repository](../../../user/project/issues/design_management.md) | **Yes** (12.7) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/32467) | No | Designs also require replication of LFS objects and Uploads. | +|[Package Registry for npm](../../../user/packages/npm_registry/index.md) | **Yes** (13.2) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Package Registry for Maven](../../../user/packages/maven_repository/index.md) | **Yes** (13.2) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Package Registry for Conan](../../../user/packages/conan_repository/index.md) | **Yes** (13.2) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Package Registry for NuGet](../../../user/packages/nuget_repository/index.md) | **Yes** (13.2) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Package Registry for PyPI](../../../user/packages/pypi_repository/index.md) | **Yes** (13.2) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Package Registry for Composer](../../../user/packages/composer_repository/index.md) | **Yes** (13.2) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Package Registry for generic packages](../../../user/packages/generic_packages/index.md) | **Yes** (13.5) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Versioned Terraform State](../../terraform_state.md) | **Yes** (13.5) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.12) | Via Object Storage provider if supported. Native Geo support (Beta). | Replication is behind the feature flag `geo_terraform_state_version_replication`, enabled by default. Verification was behind the feature flag `geo_terraform_state_version_verification`, which was removed in 14.0| +|[External merge request diffs](../../merge_request_diffs.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Replication is behind the feature flag `geo_merge_request_diff_replication`, enabled by default. Verification is under development, behind the feature flag `geo_merge_request_diff_verification`, introduced in 14.0.| |[Versioned snippets](../../../user/snippets.md#versioned-snippets) | [**Yes** (13.7)](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2810) | No | | |[Server-side Git hooks](../../server_hooks.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1867) | No | No | | |[Elasticsearch integration](../../../integration/elasticsearch.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/1186) | No | No | | -|[GitLab Pages](../../pages/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/589) | No | No | | +|[GitLab Pages](../../pages/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/589) | No | Via Object Storage provider if supported. **No** native Geo support (Beta). | | |[Dependency proxy images](../../../user/packages/dependency_proxy/index.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/259694) | No | No | Blocked on [Geo: Secondary Mimicry](https://gitlab.com/groups/gitlab-org/-/epics/1528). Note that replication of this cache is not needed for Disaster Recovery purposes because it can be recreated from external sources. | -|[Vulnerability Export](../../../user/application_security/vulnerability_report/#export-vulnerability-details) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Not planned because they are ephemeral and sensitive. They can be regenerated on demand. | +|[Vulnerability Export](../../../user/application_security/vulnerability_report/#export-vulnerability-details) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | | Not planned because they are ephemeral and sensitive. They can be regenerated on demand. | -#### LFS object replication using the self service framework +#### Limitation of verification for files in Object Storage -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276696) in GitLab 13.12. -> - [Deployed behind a feature flag](../../../user/feature_flags.md), enabled by default. -> - Not enabled on GitLab.com as Geo is not enabled. -> - Recommended for production use. -> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-lfs-object-replication-using-the-self-service-framework). +GitLab managed Object Storage replication support [is in beta](object_storage.md#enabling-gitlab-managed-object-storage-replication). -There can be [risks when disabling released features](../../../user/feature_flags.md#risks-when-disabling-released-features). -Refer to this feature's version history for more details. - -##### Enable or disable LFS object replication using the self service framework - -LFS object replication using the self service framework is under development but ready for production use. It is -deployed behind a feature flag that is **enabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) -can opt to disable it. - -To enable it: - -```ruby -Feature.enable(:geo_lfs_object_replication) -``` - -To disable it: - -```ruby -Feature.disable(:geo_lfs_object_replication) -``` +Locally stored files are verified but remote stored files are not. diff --git a/doc/administration/geo/replication/faq.md b/doc/administration/geo/replication/faq.md index a83a1c22db6..ef41b2ff172 100644 --- a/doc/administration/geo/replication/faq.md +++ b/doc/administration/geo/replication/faq.md @@ -17,13 +17,13 @@ On each **secondary** site, there is a read-only replicated copy of the GitLab d A **secondary** site also has a tracking database where it stores which projects have been synced. Geo compares the two databases to find projects that are not yet tracked. -At the start, this tracking database is empty, so Geo will start trying to update from every project that it can see in the GitLab database. +At the start, this tracking database is empty, so Geo tries to update from every project that it can see in the GitLab database. For each project to sync: -1. Geo will issue a `git fetch geo --mirror` to get the latest information from the **primary** site. - If there are no changes, the sync will be fast and end quickly. Otherwise, it will pull the latest commits. -1. The **secondary** site will update the tracking database to store the fact that it has synced projects A, B, C, etc. +1. Geo issues a `git fetch geo --mirror` to get the latest information from the **primary** site. + If there are no changes, the sync is fast. Otherwise, it has to pull the latest commits. +1. The **secondary** site updates the tracking database to store the fact that it has synced projects A, B, C, etc. 1. Repeat until all projects are synced. When someone pushes a commit to the **primary** site, it generates an event in the GitLab database that the repository has changed. @@ -70,4 +70,4 @@ Yes. See [Docker Registry for a **secondary** site](docker_registry.md). ## Can I login to a secondary site? -Yes, but secondary sites receive all authentication data (like user accounts and logins) from the primary instance. This means you will be re-directed to the primary for authentication and routed back afterwards. +Yes, but secondary sites receive all authentication data (like user accounts and logins) from the primary instance. This means you are re-directed to the primary for authentication and then routed back. diff --git a/doc/administration/geo/replication/geo_validation_tests.md b/doc/administration/geo/replication/geo_validation_tests.md index 8f67e70c9e2..c6b1078ddf0 100644 --- a/doc/administration/geo/replication/geo_validation_tests.md +++ b/doc/administration/geo/replication/geo_validation_tests.md @@ -43,7 +43,7 @@ The following are GitLab upgrade validation tests we performed. - Outcome: Partial success because we observed downtime during the upgrade of the primary and secondary sites. - Follow up issues/actions: - [Fix zero-downtime upgrade process/instructions for multi-node Geo deployments](https://gitlab.com/gitlab-org/gitlab/-/issues/225684) - - [Geo:check Rake task: Exclude AuthorizedKeysCommand check if node not running Puma/Unicorn](https://gitlab.com/gitlab-org/gitlab/-/issues/225454) + - [Geo:check Rake task: Exclude AuthorizedKeysCommand check if node not running Puma](https://gitlab.com/gitlab-org/gitlab/-/issues/225454) - [Update instructions in the next upgrade issue to include monitoring HAProxy dashboards](https://gitlab.com/gitlab-org/gitlab/-/issues/225359) [Upgrade Geo multi-node installation](https://gitlab.com/gitlab-org/gitlab/-/issues/208104): @@ -53,7 +53,7 @@ The following are GitLab upgrade validation tests we performed. - Outcome: Partial success because we did not run the looping pipeline during the demo to validate zero-downtime. - Follow up issues: - - [Clarify how Puma/Unicorn should include deploy node](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5460) + - [Clarify how Puma should include deploy node](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5460) - [Investigate MR creation failure after upgrade to 12.9.10](https://gitlab.com/gitlab-org/gitlab/-/issues/223282) Closed as false positive. ### February 2020 diff --git a/doc/administration/geo/replication/multiple_servers.md b/doc/administration/geo/replication/multiple_servers.md index 59bb3884a02..ea2488b65fb 100644 --- a/doc/administration/geo/replication/multiple_servers.md +++ b/doc/administration/geo/replication/multiple_servers.md @@ -17,9 +17,9 @@ described, it is possible to adapt these instructions to your needs. _[diagram source - GitLab employees only](https://docs.google.com/drawings/d/1z0VlizKiLNXVVVaERFwgsIOuEgjcUqDTWPdQYsE7Z4c/edit)_ -The topology above assumes that the **primary** and **secondary** Geo clusters +The topology above assumes the **primary** and **secondary** Geo clusters are located in two separate locations, on their own virtual network -with private IP addresses. The network is configured such that all machines within +with private IP addresses. The network is configured such that all machines in one geographic location can communicate with each other using their private IP addresses. The IP addresses given are examples and may be different depending on the network topology of your deployment. @@ -44,9 +44,10 @@ Support for PostgreSQL on **secondary** nodes in multi-node configuration Because of the additional complexity involved in setting up this configuration for PostgreSQL and Redis, it is not covered by this Geo multi-node documentation. -For more information about setting up a multi-node PostgreSQL cluster and Redis cluster using the omnibus package see the multi-node documentation for -[PostgreSQL](../../postgresql/replication_and_failover.md) and -[Redis](../../redis/replication_and_failover.md), respectively. +For more information on setting up a multi-node PostgreSQL cluster and Redis cluster using the Omnibus GitLab package, see: + +- [PostgreSQL multi-node documentation](../../postgresql/replication_and_failover.md) +- [Redis multi-node documentation](../../redis/replication_and_failover.md) NOTE: It is possible to use cloud hosted services for PostgreSQL and Redis, but this is beyond the scope of this document. @@ -60,8 +61,8 @@ you already have a working GitLab instance that is in-use, it can be used as a The second cluster serves as the **secondary** node. Again, use the [GitLab multi-node documentation](../../reference_architectures/index.md) to set this up. -It's a good idea to log in and test it, however, note that its data is -wiped out as part of the process of replicating from the **primary**. +It's a good idea to log in and test it. However, be aware that its data is +wiped out as part of the process of replicating from the **primary** node. ## Configure the GitLab cluster to be the **primary** node @@ -92,9 +93,9 @@ After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus NOTE: PostgreSQL and Redis should have already been disabled on the -application servers, and connections from the application servers to those -services on the backend servers configured, during normal GitLab multi-node set up. See -multi-node configuration documentation for +application servers during normal GitLab multi-node setup. Connections +from the application servers to services on the backend servers should +have also been configured. See multi-node configuration documentation for [PostgreSQL](../../postgresql/replication_and_failover.md#configuring-the-application-nodes) and [Redis](../../redis/replication_and_failover.md#example-configuration-for-the-gitlab-application). @@ -120,12 +121,12 @@ major differences: called the "tracking database", which tracks the synchronization state of various resources. -Therefore, we set up the multi-node components one-by-one, and include deviations -from the normal multi-node setup. However, we highly recommend first configuring a -brand-new cluster as if it were not part of a Geo setup so that it can be -tested and verified as a working cluster. And only then should it be modified -for use as a Geo **secondary**. This helps to separate problems that are related -and are not related to Geo setup. +Therefore, we set up the multi-node components one by one and include deviations +from the normal multi-node setup. However, we highly recommend configuring a +brand-new cluster first, as if it were not part of a Geo setup. This allows +verifying that it is a working cluster. And only then should it be modified +for use as a Geo **secondary**. This helps to separate Geo setup problems from +unrelated problems. ### Step 1: Configure the Redis and Gitaly services on the **secondary** node @@ -218,11 +219,10 @@ the **primary** database. Use the following as a guide. prometheus['enable'] = false redis['enable'] = false redis_exporter['enable'] = false - repmgr['enable'] = false + patroni['enable'] = false sidekiq['enable'] = false sidekiq_cluster['enable'] = false puma['enable'] = false - unicorn['enable'] = false ``` After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect. @@ -290,11 +290,10 @@ Configure the tracking database. prometheus['enable'] = false redis['enable'] = false redis_exporter['enable'] = false - repmgr['enable'] = false + patroni['enable'] = false sidekiq['enable'] = false sidekiq_cluster['enable'] = false puma['enable'] = false - unicorn['enable'] = false ``` After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect. @@ -366,10 +365,10 @@ then make the following modifications: ``` NOTE: -If you had set up PostgreSQL cluster using the omnibus package and you had set -up `postgresql['sql_user_password'] = 'md5 digest of secret'` setting, keep in +If you had set up PostgreSQL cluster using the omnibus package and had set +`postgresql['sql_user_password'] = 'md5 digest of secret'`, keep in mind that `gitlab_rails['db_password']` and `geo_secondary['db_password']` -mentioned above contains the plaintext passwords. This is used to let the Rails +contains the plaintext passwords. This is used to let the Rails servers connect to the databases. NOTE: @@ -438,9 +437,8 @@ application servers above, with some changes to run only the `sidekiq` service: prometheus['enable'] = false redis['enable'] = false redis_exporter['enable'] = false - repmgr['enable'] = false + patroni['enable'] = false puma['enable'] = false - unicorn['enable'] = false ## ## The unique identifier for the Geo node. diff --git a/doc/administration/geo/replication/remove_geo_node.md b/doc/administration/geo/replication/remove_geo_node.md index 697d8c6ae38..b72cd3cbb95 100644 --- a/doc/administration/geo/replication/remove_geo_node.md +++ b/doc/administration/geo/replication/remove_geo_node.md @@ -1,5 +1,6 @@ --- redirect_to: '../../geo/replication/remove_geo_site.md' +remove_date: '2021-06-01' --- This document was moved to [another location](../../geo/replication/remove_geo_site.md). diff --git a/doc/administration/geo/replication/security_review.md b/doc/administration/geo/replication/security_review.md index f84d7a2171d..ae41599311b 100644 --- a/doc/administration/geo/replication/security_review.md +++ b/doc/administration/geo/replication/security_review.md @@ -184,7 +184,7 @@ from [owasp.org](https://owasp.org/). ### What databases and application servers support the application? -- PostgreSQL >= 11, Redis, Sidekiq, Puma. +- PostgreSQL >= 12, Redis, Sidekiq, Puma. ### How will database connection strings, encryption keys, and other sensitive components be stored, accessed, and protected from unauthorized detection? diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index 6d990fd12ba..1fd923dbaf1 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -583,64 +583,6 @@ to start again from scratch, there are a few steps that can help you: gitlab-ctl start ``` -## Fixing errors during a PostgreSQL upgrade or downgrade - -### Message: `ERROR: psql: FATAL: role "gitlab-consul" does not exist` - -When -[upgrading PostgreSQL on a Geo instance](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance), you might encounter the -following error: - -```plaintext -$ sudo gitlab-ctl pg-upgrade --target-version=11 -Checking for an omnibus managed postgresql: OK -Checking if postgresql['version'] is set: OK -Checking if we already upgraded: NOT OK -Checking for a newer version of PostgreSQL to install -Upgrading PostgreSQL to 11.7 -Checking if PostgreSQL bin files are symlinked to the expected location: OK -Waiting 30 seconds to ensure tasks complete before PostgreSQL upgrade. -See https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server for details -If you do not want to upgrade the PostgreSQL server at this time, enter Ctrl-C and see the documentation for details - -Please hit Ctrl-C now if you want to cancel the operation. -..............................Detected an HA cluster. -Error running command: /opt/gitlab/embedded/bin/psql -qt -d gitlab_repmgr -h /var/opt/gitlab/postgresql -p 5432 -c "SELECT name FROM repmgr_gitlab_cluster.repl_nodes WHERE type='master' AND active != 'f'" -U gitlab-consul -ERROR: psql: FATAL: role "gitlab-consul" does not exist -Traceback (most recent call last): - 10: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `<main>' - 9: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load' - 8: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in `<top (required)>' - 7: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in `run' - 6: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in `block in add_command_under_category' - 5: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:171:in `block in load_file' - 4: from /opt/gitlab/embedded/service/omnibus-ctl-ee/lib/repmgr.rb:248:in `is_master?' - 3: from /opt/gitlab/embedded/service/omnibus-ctl-ee/lib/repmgr.rb:100:in `execute_psql' - 2: from /opt/gitlab/embedded/service/omnibus-ctl-ee/lib/repmgr.rb:113:in `cmd' - 1: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/mixlib-shellout-3.0.9/lib/mixlib/shellout.rb:287:in `error!' -/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/mixlib-shellout-3.0.9/lib/mixlib/shellout.rb:300:in `invalid!': Expected process to exit with [0], but received '2' (Mixlib::ShellOut::ShellCommandFailed) ----- Begin output of /opt/gitlab/embedded/bin/psql -qt -d gitlab_repmgr -h /var/opt/gitlab/postgresql -p 5432 -c "SELECT name FROM repmgr_gitlab_cluster.repl_nodes WHERE type='master' AND active != 'f'" -U gitlab-consul ---- -STDOUT: -STDERR: psql: FATAL: role "gitlab-consul" does not exist ----- End output of /opt/gitlab/embedded/bin/psql -qt -d gitlab_repmgr -h /var/opt/gitlab/postgresql -p 5432 -c "SELECT name FROM repmgr_gitlab_cluster.repl_nodes WHERE type='master' AND active != 'f'" -U gitlab-consul ---- -Ran /opt/gitlab/embedded/bin/psql -qt -d gitlab_repmgr -h /var/opt/gitlab/postgresql -p 5432 -c "SELECT name FROM repmgr_gitlab_cluster.repl_nodes WHERE type='master' AND active != 'f'" -U gitlab-consul returned 2 -``` - -If you are upgrading the PostgreSQL read-replica of a Geo secondary node, and -you are not using `consul` or `repmgr`, you may need to disable `consul` and/or -`repmgr` services in `gitlab.rb`: - -```ruby -consul['enable'] = false -repmgr['enable'] = false -``` - -Then reconfigure GitLab: - -```shell -sudo gitlab-ctl reconfigure -``` - ## Fixing errors during a failover or when promoting a secondary to a primary node The following are possible errors that might be encountered during failover or @@ -756,6 +698,30 @@ this command reports `ERROR - Replication is not up-to-date` even if replication is actually up-to-date. If replication and verification output shows that it is complete, you can add `--skip-preflight-checks` to make the command complete promotion. This bug was fixed in GitLab 13.8 and later. +### Errors when using `--skip-preflight-checks` or `--force` + +Before GitLab 13.5, you could bump into one of the following errors when using +`--skip-preflight-checks` or `--force`: + +```plaintext +get_ctl_options': invalid option: --skip-preflight-checks (OptionParser::InvalidOption) + +get_ctl_options': invalid option: --force (OptionParser::InvalidOption) +``` + +This can happen with XFS or filesystems that list files in lexical order, because the +load order of the Omnibus command files can be different than expected, and a global function would get redefined. +More details can be found in [the related issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6076). + +The workaround is to manually run the preflight checks and promote the database, by running +the following commands on the Geo secondary site: + +```shell +sudo gitlab-ctl promotion-preflight-checks +sudo /opt/gitlab/embedded/bin/gitlab-pg-ctl promote +sudo gitlab-ctl reconfigure +sudo gitlab-rake geo:set_secondary_as_primary + ## Expired artifacts If you notice for some reason there are more artifacts on the Geo @@ -854,6 +820,11 @@ To resolve this issue: the **primary** node using IPv4 in the `/etc/hosts` file. Alternatively, you should [enable IPv6 on the **primary** node](https://docs.gitlab.com/omnibus/settings/nginx.html#setting-the-nginx-listen-address-or-addresses). +### Geo Admin Area shows 'Unknown' for health status and 'Request failed with status code 401' + +If using a load balancer, ensure that the load balancer's URL is set as the `external_url` in the +`/etc/gitlab/gitlab.rb` of the nodes behind the load balancer. + ### GitLab Pages return 404 errors after promoting This is due to [Pages data not being managed by Geo](datatypes.md#limitations-on-replicationverification). diff --git a/doc/administration/geo/replication/using_a_geo_server.md b/doc/administration/geo/replication/using_a_geo_server.md index f8ce72ac3f8..04c30514a89 100644 --- a/doc/administration/geo/replication/using_a_geo_server.md +++ b/doc/administration/geo/replication/using_a_geo_server.md @@ -1,5 +1,6 @@ --- redirect_to: '../../geo/replication/usage.md' +remove_date: '2022-06-01' --- This document was moved to [another location](../../geo/replication/usage.md). diff --git a/doc/administration/geo/replication/version_specific_updates.md b/doc/administration/geo/replication/version_specific_updates.md index 4a101c52325..301be931b29 100644 --- a/doc/administration/geo/replication/version_specific_updates.md +++ b/doc/administration/geo/replication/version_specific_updates.md @@ -11,6 +11,10 @@ Review this page for update instructions for your version. These steps accompany the [general steps](updating_the_geo_nodes.md#general-update-steps) for updating Geo nodes. +## Updating to GitLab 13.11 + +We found an [issue with Git clone/pull through HTTP(s)](https://gitlab.com/gitlab-org/gitlab/-/issues/330787) on Geo secondaries and on any GitLab instance if maintenance mode is enabled. This was caused by a regression in GitLab Workhorse. This is fixed in the [GitLab 13.11.4 patch release](https://about.gitlab.com/releases/2021/05/14/gitlab-13-11-4-released/). To avoid this issue, upgrade to GitLab 13.11.4 or later. + ## Updating to GitLab 13.9 We've detected an issue [with a column rename](https://gitlab.com/gitlab-org/gitlab/-/issues/324160) @@ -78,6 +82,12 @@ paused fails. Do not pause replication before promoting a secondary. If the node is paused, be sure to resume before promoting. To avoid this issue, upgrade to GitLab 13.4 or later. +WARNING: +Promoting the database during a failover can fail on XFS and filesystems ordering files lexically, +when using `--force` or `--skip-preflight-checks`, due to [an issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6076) fixed in 13.5. +The [troubleshooting steps](troubleshooting.md#errors-when-using---skip-preflight-checks-or---force) +contain a workaround if you run into errors during the failover. + ## Updating to GitLab 13.2 In GitLab 13.2, promoting a secondary node to a primary while the secondary is diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md index b87a606e349..f6e72092a5f 100644 --- a/doc/administration/geo/setup/database.md +++ b/doc/administration/geo/setup/database.md @@ -9,7 +9,7 @@ type: howto NOTE: If your GitLab installation uses external (not managed by Omnibus) PostgreSQL -instances, the Omnibus roles will not be able to perform all necessary +instances, the Omnibus roles are unable to perform all necessary configuration steps. In this case, [follow the Geo with external PostgreSQL instances document instead](external_database.md). @@ -25,10 +25,23 @@ size. You are encouraged to first read through all the steps before executing them in your testing/production environment. -## PostgreSQL replication +## Single instance database replication -The GitLab **primary** node where the write operations happen will connect to -the **primary** database server, and **secondary** nodes will +A single instance database replication is easier to set up and still provides the same Geo capabilities +as a clusterized alternative. It's useful for setups running on a single machine +or trying to evaluate Geo for a future clusterized installation. + +A single instance can be expanded to a clusterized version using Patroni, which is recommended for a +highly available architecture. + +Follow below the instructions on how to set up PostgreSQL replication as a single instance database. +Alternatively, you can look at the [Multi-node database replication](#multi-node-database-replication) +instructions on setting up replication with a Patroni cluster. + +### PostgreSQL replication + +The GitLab **primary** node where the write operations happen connects to +the **primary** database server, and **secondary** nodes connect to their own database servers (which are also read-only). We recommend using [PostgreSQL replication slots](https://medium.com/@tk512/replication-slots-in-postgresql-b4b03d277c75) @@ -37,8 +50,8 @@ recover. See below for more details. The following guide assumes that: -- You are using Omnibus and therefore you are using PostgreSQL 11 or later - which includes the [`pg_basebackup` tool](https://www.postgresql.org/docs/11/app-pgbasebackup.html). +- You are using Omnibus and therefore you are using PostgreSQL 12 or later + which includes the [`pg_basebackup` tool](https://www.postgresql.org/docs/12/app-pgbasebackup.html). - You have a **primary** node already set up (the GitLab server you are replicating from), running Omnibus' PostgreSQL (or equivalent version), and you have a new **secondary** server set up with the same versions of the OS, @@ -48,7 +61,7 @@ WARNING: Geo works with streaming replication. Logical replication is not supported at this time. There is an [issue where support is being discussed](https://gitlab.com/gitlab-org/gitlab/-/issues/7420). -### Step 1. Configure the **primary** server +#### Step 1. Configure the **primary** server 1. SSH into your GitLab **primary** server and login as root: @@ -75,13 +88,9 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o gitlab-ctl set-geo-primary-node ``` - This command will use your defined `external_url` in `/etc/gitlab/gitlab.rb`. - -1. GitLab 10.4 and up only: Do the following to make sure the `gitlab` database user has a password defined: + This command uses your defined `external_url` in `/etc/gitlab/gitlab.rb`. - NOTE: - Until FDW settings are removed in GitLab version 14.0, avoid using single or double quotes in the - password for PostgreSQL as that will lead to errors when reconfiguring. +1. Define a password for the `gitlab` database user: Generate a MD5 hash of the desired password: @@ -103,18 +112,28 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o # must be present in all application nodes. gitlab_rails['db_password'] = '<your_password_here>' ``` + +1. Define a password for the database [replication user](https://wiki.postgresql.org/wiki/Streaming_Replication). -1. Omnibus GitLab already has a [replication user](https://wiki.postgresql.org/wiki/Streaming_Replication) - called `gitlab_replicator`. You must set the password for this user manually. - You will be prompted to enter a password: + We will use the username defined in `/etc/gitlab/gitlab.rb` under the `postgresql['sql_replication_user']` + setting. The default value is `gitlab_replicator`, but if you changed it to something else, adapt + the instructions below. + + Generate a MD5 hash of the desired password: ```shell - gitlab-ctl set-replication-password + gitlab-ctl pg-password-md5 gitlab_replicator + # Enter password: <your_password_here> + # Confirm password: <your_password_here> + # 950233c0dfc2f39c64cf30457c3b7f1e ``` - This command will also read the `postgresql['sql_replication_user']` Omnibus - setting in case you have changed `gitlab_replicator` username to something - else. + Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab_replicator` + postgresql['sql_replication_password'] = '<md5_hash_of_your_password>' + ``` If you are using an external database not managed by Omnibus GitLab, you need to create the replicator user and define a password to it manually: @@ -154,7 +173,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o echo "External address: $(curl --silent "ipinfo.io/ip")" ``` - In most cases, the following addresses will be used to configure GitLab + In most cases, the following addresses are used to configure GitLab Geo: | Configuration | Address | @@ -168,11 +187,11 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o `postgresql['md5_auth_cidr_addresses']` and `postgresql['listen_address']`. The `listen_address` option opens PostgreSQL up to network connections with the interface - corresponding to the given address. See [the PostgreSQL documentation](https://www.postgresql.org/docs/11/runtime-config-connection.html) + corresponding to the given address. See [the PostgreSQL documentation](https://www.postgresql.org/docs/12/runtime-config-connection.html) for more details. NOTE: - If you need to use `0.0.0.0` or `*` as the listen_address, you will also need to add + If you need to use `0.0.0.0` or `*` as the listen_address, you also need to add `127.0.0.1/32` to the `postgresql['md5_auth_cidr_addresses']` setting, to allow Rails to connect through `127.0.0.1`. For more information, see [omnibus-5258](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5258). @@ -190,7 +209,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ## Geo Primary role ## - configure dependent flags automatically to enable Geo ## - roles ['geo_primary_role'] + roles(['geo_primary_role']) ## ## Primary address @@ -226,7 +245,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ``` You may also want to edit the `wal_keep_segments` and `max_wal_senders` to match your - database replication requirements. Consult the [PostgreSQL - Replication documentation](https://www.postgresql.org/docs/11/runtime-config-replication.html) + database replication requirements. Consult the [PostgreSQL - Replication documentation](https://www.postgresql.org/docs/12/runtime-config-replication.html) for more information. 1. Save the file and reconfigure GitLab for the database listen changes and @@ -262,7 +281,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o `5432` to the **primary** server's private address. 1. A certificate was automatically generated when GitLab was reconfigured. This - will be used automatically to protect your PostgreSQL traffic from + is used automatically to protect your PostgreSQL traffic from eavesdroppers, but to protect against active ("man-in-the-middle") attackers, the **secondary** node needs a copy of the certificate. Make a copy of the PostgreSQL `server.crt` file on the **primary** node by running this command: @@ -272,10 +291,10 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ``` Copy the output into a clipboard or into a local file. You - will need it when setting up the **secondary** node! The certificate is not sensitive + need it when setting up the **secondary** node! The certificate is not sensitive data. -### Step 2. Configure the **secondary** server +#### Step 2. Configure the **secondary** server 1. SSH into your GitLab **secondary** server and login as root: @@ -325,7 +344,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o -T server.crt ~gitlab-psql/.postgresql/root.crt ``` - PostgreSQL will now only recognize that exact certificate when verifying TLS + PostgreSQL now only recognizes that exact certificate when verifying TLS connections. The certificate can only be replicated by someone with access to the private key, which is **only** present on the **primary** node. @@ -363,7 +382,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ## Geo Secondary role ## - configure dependent flags automatically to enable Geo ## - roles ['geo_secondary_role'] + roles(['geo_secondary_role']) ## ## Secondary address @@ -376,12 +395,13 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ## Database credentials password (defined previously in primary node) ## - replicate same values here as defined in primary node ## + postgresql['sql_replication_password'] = '<md5_hash_of_your_password>' postgresql['sql_user_password'] = '<md5_hash_of_your_password>' gitlab_rails['db_password'] = '<your_password_here>' ``` For external PostgreSQL instances, see [additional instructions](external_database.md). - If you bring a former **primary** node back online to serve as a **secondary** node, then you also need to remove `roles ['geo_primary_role']` or `geo_primary_role['enable'] = true`. + If you bring a former **primary** node back online to serve as a **secondary** node, then you also need to remove `roles(['geo_primary_role'])` or `geo_primary_role['enable'] = true`. 1. Reconfigure GitLab for the changes to take effect: @@ -395,7 +415,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o gitlab-ctl restart postgresql ``` -### Step 3. Initiate the replication process +#### Step 3. Initiate the replication process Below we provide a script that connects the database on the **secondary** node to the database on the **primary** node, replicates the database, and creates the @@ -423,7 +443,7 @@ data before running `pg_basebackup`. WARNING: Each Geo **secondary** node must have its own unique replication slot name. - Using the same slot name between two secondaries will break PostgreSQL replication. + Using the same slot name between two secondaries breaks PostgreSQL replication. ```shell gitlab-ctl replicate-geo-database \ @@ -441,57 +461,57 @@ data before running `pg_basebackup`. to list them all, but here are a couple of tips: - If PostgreSQL is listening on a non-standard port, add `--port=` as well. - - If your database is too large to be transferred in 30 minutes, you will need + - If your database is too large to be transferred in 30 minutes, you need to increase the timeout, e.g., `--backup-timeout=3600` if you expect the initial replication to take under an hour. - Pass `--sslmode=disable` to skip PostgreSQL TLS authentication altogether (e.g., you know the network path is secure, or you are using a site-to-site VPN). This is **not** safe over the public Internet! - You can read more details about each `sslmode` in the - [PostgreSQL documentation](https://www.postgresql.org/docs/11/libpq-ssl.html#LIBPQ-SSL-PROTECTION); + [PostgreSQL documentation](https://www.postgresql.org/docs/12/libpq-ssl.html#LIBPQ-SSL-PROTECTION); the instructions above are carefully written to ensure protection against both passive eavesdroppers and active "man-in-the-middle" attackers. - Change the `--slot-name` to the name of the replication slot - to be used on the **primary** database. The script will attempt to create the + to be used on the **primary** database. The script attempts to create the replication slot automatically if it does not exist. - - If you're repurposing an old server into a Geo **secondary** node, you'll need to + - If you're repurposing an old server into a Geo **secondary** node, you need to add `--force` to the command line. - When not in a production machine you can disable backup step if you really sure this is what you want by adding `--skip-backup` The replication process is now complete. -## PgBouncer support (optional) +### PgBouncer support (optional) [PgBouncer](https://www.pgbouncer.org/) may be used with GitLab Geo to pool -PostgreSQL connections. We recommend using PgBouncer if you use GitLab in a -high-availability configuration with a cluster of nodes supporting a Geo -**primary** site and two other clusters of nodes supporting a Geo **secondary** site. -One for the main database and the other for the tracking database. For more information, +PostgreSQL connections, which can improve performance even when using in a +single instance installation. + +We recommend using PgBouncer if you use GitLab in a highly available +configuration with a cluster of nodes supporting a Geo **primary** site and +two other clusters of nodes supporting a Geo **secondary** site. One for the +main database and the other for the tracking database. For more information, see [High Availability with Omnibus GitLab](../../postgresql/replication_and_failover.md). -## Patroni support +## Multi-node database replication -Support for Patroni is intended to replace `repmgr` as a -[highly available PostgreSQL solution](../../postgresql/replication_and_failover.md) -on the primary node, but it can also be used for PostgreSQL HA on a secondary -site. Similar to `repmgr`, using Patroni on a secondary node is optional. +In GitLab 14.0, Patroni replaced `repmgr` as the supported +[highly available PostgreSQL solution](../../postgresql/replication_and_failover.md). -Starting with GitLab 13.5, Patroni is available for _experimental_ use with Geo -primary and secondary sites. Due to its experimental nature, Patroni support is -subject to change without notice. +NOTE: +If you still haven't [migrated from repmgr to Patroni](#migrating-from-repmgr-to-patroni) you're highly advised to do so. -This experimental implementation has the following limitations: +### Patroni support -- Whenever `gitlab-ctl reconfigure` runs on a Patroni Leader instance, there's a - chance the node will be demoted due to the required short-time restart. To - avoid this, you can pause auto-failover by running `gitlab-ctl patroni pause`. - After a reconfigure, it resumes on its own. +Patroni is the official replication management solution for Geo. It +can be used to build a highly available cluster on the **primary** and a **secondary** Geo site. +Using Patroni on a **secondary** site is optional and you don't have to use the same amount of +nodes on each Geo site. For instructions about how to set up Patroni on the primary site, see the [PostgreSQL replication and failover with Omnibus GitLab](../../postgresql/replication_and_failover.md#patroni) page. -### Configuring Patroni cluster for a Geo secondary site +#### Configuring Patroni cluster for a Geo secondary site In a Geo secondary site, the main PostgreSQL database is a read-only replica of the primary site’s PostgreSQL database. @@ -503,7 +523,7 @@ configuration for the secondary site. The internal load balancer provides a sing endpoint for connecting to the Patroni cluster's leader whenever a new leader is elected. Be sure to use [password credentials](../../postgresql/replication_and_failover.md#database-authorization-for-patroni) and other database best practices. -#### Step 1. Configure Patroni permanent replication slot on the primary site +##### Step 1. Configure Patroni permanent replication slot on the primary site To set up database replication with Patroni on a secondary node, we need to configure a _permanent replication slot_ on the primary node's Patroni cluster, @@ -521,16 +541,16 @@ Leader instance**: 1. Edit `/etc/gitlab/gitlab.rb` and add the following: ```ruby - consul['enable'] = true + roles(['patroni_role']) + + consul['services'] = %w(postgresql) consul['configuration'] = { retry_join: %w[CONSUL_PRIMARY1_IP CONSUL_PRIMARY2_IP CONSUL_PRIMARY3_IP] } - - repmgr['enable'] = false - + # You need one entry for each secondary, with a unique name following PostgreSQL slot_name constraints: # - # Configuration syntax will be: 'unique_slotname' => { 'type' => 'physical' }, + # Configuration syntax is: 'unique_slotname' => { 'type' => 'physical' }, # We don't support setting a permanent replication slot for logical replication type patroni['replication_slots'] = { 'geo_secondary' => { 'type' => 'physical' } @@ -539,15 +559,18 @@ Leader instance**: patroni['use_pg_rewind'] = true patroni['postgresql']['max_wal_senders'] = 8 # Use double of the amount of patroni/reserved slots (3 patronis + 1 reserved slot for a Geo secondary). patroni['postgresql']['max_replication_slots'] = 8 # Use double of the amount of patroni/reserved slots (3 patronis + 1 reserved slot for a Geo secondary). + patroni['replication_password'] = 'PLAIN_TEXT_POSTGRESQL_REPLICATION_PASSWORD' - postgresql['md5_auth_cidr_addresses'] = [ - 'PATRONI_PRIMARY1_IP/32', 'PATRONI_PRIMARY2_IP/32', 'PATRONI_PRIMARY3_IP/32', 'PATRONI_PRIMARY_PGBOUNCER/32', - 'PATRONI_SECONDARY1_IP/32', 'PATRONI_SECONDARY2_IP/32', 'PATRONI_SECONDARY3_IP/32', 'PATRONI_SECONDARY_PGBOUNCER/32' # We list all secondary instances as they can all become a Standby Leader + # We list all secondary instances as they can all become a Standby Leader + postgresql['md5_auth_cidr_addresses'] = %w[ + PATRONI_PRIMARY1_IP/32 PATRONI_PRIMARY2_IP/32 PATRONI_PRIMARY3_IP/32 PATRONI_PRIMARY_PGBOUNCER/32 + PATRONI_SECONDARY1_IP/32 PATRONI_SECONDARY2_IP/32 PATRONI_SECONDARY3_IP/32 PATRONI_SECONDARY_PGBOUNCER/32 ] postgresql['pgbouncer_user_password'] = 'PGBOUNCER_PASSWORD_HASH' postgresql['sql_replication_password'] = 'POSTGRESQL_REPLICATION_PASSWORD_HASH' postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' + postgresql['listen_address'] = '0.0.0.0' # You can use a public or VPC address here instead ``` 1. Reconfigure GitLab for the changes to take effect: @@ -556,17 +579,17 @@ Leader instance**: gitlab-ctl reconfigure ``` -#### Step 2. Configure the internal load balancer on the primary site +##### Step 2. Configure the internal load balancer on the primary site To avoid reconfiguring the Standby Leader on the secondary site whenever a new -Leader is elected on the primary site, we'll need to set up a TCP internal load -balancer which will give a single endpoint for connecting to the Patroni +Leader is elected on the primary site, we need to set up a TCP internal load +balancer which gives a single endpoint for connecting to the Patroni cluster's Leader. The Omnibus GitLab packages do not include a Load Balancer. Here's how you could do it with [HAProxy](https://www.haproxy.org/). -The following IPs and names will be used as an example: +The following IPs and names are used as an example: - `10.6.0.21`: Patroni 1 (`patroni1.internal`) - `10.6.0.21`: Patroni 2 (`patroni2.internal`) @@ -600,7 +623,7 @@ backend postgresql Refer to your preferred Load Balancer's documentation for further guidance. -#### Step 3. Configure a PgBouncer node on the secondary site +##### Step 3. Configure a PgBouncer node on the secondary site A production-ready and highly available configuration requires at least three Consul nodes, a minimum of one PgBouncer node, but it’s recommended to have @@ -621,22 +644,26 @@ Follow the minimal configuration for the PgBouncer node: ```ruby # Disable all components except Pgbouncer and Consul agent - roles ['pgbouncer_role'] + roles(['pgbouncer_role']) # PgBouncer configuration + pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) pgbouncer['users'] = { + 'gitlab-consul': { + # Generate it with: `gitlab-ctl pg-password-md5 gitlab-consul` + password: 'GITLAB_CONSUL_PASSWORD_HASH' + }, 'pgbouncer': { + # Generate it with: `gitlab-ctl pg-password-md5 pgbouncer` password: 'PGBOUNCER_PASSWORD_HASH' } } # Consul configuration consul['watchers'] = %w(postgresql) - consul['configuration'] = { retry_join: %w[CONSUL_SECONDARY1_IP CONSUL_SECONDARY2_IP CONSUL_SECONDARY3_IP] } - consul['monitoring_service_discovery'] = true ``` @@ -652,17 +679,17 @@ Follow the minimal configuration for the PgBouncer node: gitlab-ctl write-pgpass --host 127.0.0.1 --database pgbouncer --user pgbouncer --hostuser gitlab-consul ``` -1. Restart the PgBouncer service: +1. Reload the PgBouncer service: ```shell - gitlab-ctl restart pgbouncer + gitlab-ctl hup pgbouncer ``` -#### Step 4. Configure a Standby cluster on the secondary site +##### Step 4. Configure a Standby cluster on the secondary site NOTE: If you are converting a secondary site to a Patroni Cluster, you must start -on the PostgreSQL instance. It will become the Patroni Standby Leader instance, +on the PostgreSQL instance. It becomes the Patroni Standby Leader instance, and then you can switchover to another replica if you need. For each Patroni instance on the secondary site: @@ -676,21 +703,18 @@ For each Patroni instance on the secondary site: 1. Edit `/etc/gitlab/gitlab.rb` and add the following: ```ruby - roles ['consul_role', 'postgres_role'] + roles(['consul_role', 'patroni_role']) consul['enable'] = true consul['configuration'] = { retry_join: %w[CONSUL_SECONDARY1_IP CONSUL_SECONDARY2_IP CONSUL_SECONDARY3_IP] } - repmgr['enable'] = false - postgresql['md5_auth_cidr_addresses'] = [ 'PATRONI_SECONDARY1_IP/32', 'PATRONI_SECONDARY2_IP/32', 'PATRONI_SECONDARY3_IP/32', 'PATRONI_SECONDARY_PGBOUNCER/32', # Any other instance that needs access to the database as per documentation ] - patroni['enable'] = false patroni['standby_cluster']['enable'] = true patroni['standby_cluster']['host'] = 'INTERNAL_LOAD_BALANCER_PRIMARY_IP' patroni['standby_cluster']['port'] = INTERNAL_LOAD_BALANCER_PRIMARY_PORT @@ -699,6 +723,15 @@ For each Patroni instance on the secondary site: patroni['use_pg_rewind'] = true patroni['postgresql']['max_wal_senders'] = 5 # A minimum of three for one replica, plus two for each additional replica patroni['postgresql']['max_replication_slots'] = 5 # A minimum of three for one replica, plus two for each additional replica + + postgresql['pgbouncer_user_password'] = 'PGBOUNCER_PASSWORD_HASH' + postgresql['sql_replication_password'] = 'POSTGRESQL_REPLICATION_PASSWORD_HASH' + postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' + postgresql['listen_address'] = '0.0.0.0' # You can use a public or VPC address here instead + + gitlab_rails['dbpassword'] = 'POSTGRESQL_PASSWORD' + gitlab_rails['enable'] = true + gitlab_rails['auto_migrate'] = false ``` 1. Reconfigure GitLab for the changes to take effect. @@ -708,33 +741,11 @@ For each Patroni instance on the secondary site: gitlab-ctl reconfigure ``` -1. Remove the PostgreSQL data directory: - - WARNING: - If you are converting a secondary site to a Patroni Cluster, you must skip - this step on the PostgreSQL instance. - - ```shell - rm -rf /var/opt/gitlab/postgresql/data - ``` - -1. Edit `/etc/gitlab/gitlab.rb` to enable Patroni: - - ```ruby - patroni['enable'] = true - ``` - -1. Reconfigure GitLab for the changes to take effect: - - ```shell - gitlab-ctl reconfigure - ``` - ### Migrating from repmgr to Patroni 1. Before migrating, it is recommended that there is no replication lag between the primary and secondary sites and that replication is paused. In GitLab 13.2 and later, you can pause and resume replication with `gitlab-ctl geo-replication-pause` and `gitlab-ctl geo-replication-resume` on a Geo secondary database node. 1. Follow the [instructions to migrate repmgr to Patroni](../../postgresql/replication_and_failover.md#switching-from-repmgr-to-patroni). When configuring Patroni on each primary site database node, add `patroni['replication_slots'] = { '<slot_name>' => 'physical' }` -to `gitlab.rb` where `<slot_name>` is the name of the replication slot for your Geo secondary. This will ensure that Patroni recognizes the replication slot as permanent and will not drop it upon restarting. +to `gitlab.rb` where `<slot_name>` is the name of the replication slot for your Geo secondary. This ensures that Patroni recognizes the replication slot as permanent and not drop it upon restarting. 1. If database replication to the secondary was paused before migration, resume replication once Patroni is confirmed working on the primary. ### Migrating a single PostgreSQL node to Patroni @@ -750,14 +761,14 @@ With Patroni it's now possible to support that. In order to migrate the existing 1. [Configure a Standby Cluster](#step-4-configure-a-standby-cluster-on-the-secondary-site) on that single node machine. -You will end up with a "Standby Cluster" with a single node. That allows you to later on add additional Patroni nodes +You end up with a "Standby Cluster" with a single node. That allows you to later on add additional Patroni nodes by following the same instructions above. ### Configuring Patroni cluster for the tracking PostgreSQL database Secondary sites use a separate PostgreSQL installation as a tracking database to keep track of replication status and automatically recover from potential replication issues. -Omnibus automatically configures a tracking database when `roles ['geo_secondary_role']` is set. +Omnibus automatically configures a tracking database when `roles(['geo_secondary_role'])` is set. If you want to run this database in a highly available configuration, follow the instructions below. A production-ready and secure setup requires at least three Consul nodes, three @@ -782,7 +793,7 @@ Follow the minimal configuration for the PgBouncer node for the tracking databas ```ruby # Disable all components except Pgbouncer and Consul agent - roles ['pgbouncer_role'] + roles(['pgbouncer_role']) # PgBouncer configuration pgbouncer['users'] = { @@ -844,7 +855,7 @@ For each Patroni instance on the secondary site for the tracking database: ```ruby # Disable all components except PostgreSQL, Patroni, and Consul - roles ['patroni_role'] + roles(['patroni_role']) # Consul configuration consul['services'] = %w(postgresql) @@ -875,6 +886,7 @@ For each Patroni instance on the secondary site for the tracking database: # GitLab database settings gitlab_rails['db_database'] = 'gitlabhq_geo_production' gitlab_rails['db_username'] = 'gitlab_geo' + gitlab_rails['enable'] = true # Disable automatic database migrations gitlab_rails['auto_migrate'] = false @@ -934,8 +946,8 @@ Patroni implementation on Omnibus that do not allow us to manage two different clusters on the same machine, we recommend setting up a new Patroni cluster for the tracking database by following the same instructions above. -The secondary nodes will backfill the new tracking database, and no data -synchronization will be required. +The secondary nodes backfill the new tracking database, and no data +synchronization is required. ## Troubleshooting diff --git a/doc/administration/geo/setup/external_database.md b/doc/administration/geo/setup/external_database.md index 1b0082687e6..9e187424afa 100644 --- a/doc/administration/geo/setup/external_database.md +++ b/doc/administration/geo/setup/external_database.md @@ -24,11 +24,19 @@ developed and tested. We aim to be compatible with most external sudo -i ``` -1. Edit `/etc/gitlab/gitlab.rb` and add a **unique** ID for your node (arbitrary value): +1. Edit `/etc/gitlab/gitlab.rb` and add: ```ruby - # The unique identifier for the Geo node. - gitlab_rails['geo_node_name'] = '<node_name_here>' + ## + ## Geo Primary role + ## - configure dependent flags automatically to enable Geo + ## + roles ['geo_primary_role'] + + ## + ## The unique identifier for the Geo site. + ## + gitlab_rails['geo_node_name'] = '<geo_site_name_here>' ``` 1. Reconfigure the **primary** node for the change to take effect: @@ -49,7 +57,7 @@ developed and tested. We aim to be compatible with most external To set up an external database, you can either: -- Set up [streaming replication](https://www.postgresql.org/docs/11/warm-standby.html#STREAMING-REPLICATION-SLOTS) yourself (for example AWS RDS, bare metal not managed by Omnibus, etc.). +- Set up [streaming replication](https://www.postgresql.org/docs/12/warm-standby.html#STREAMING-REPLICATION-SLOTS) yourself (for example AWS RDS, bare metal not managed by Omnibus, etc.). - Perform the Omnibus configuration manually as follows. #### Leverage your cloud provider's tools to replicate the primary database @@ -64,8 +72,9 @@ cloud providers: - Amazon RDS - [Creating a Read Replica](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html#USER_ReadRepl.Create) - Azure Database for PostgreSQL - [Create and manage read replicas in Azure Database for PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/howto-read-replicas-portal) +- Google Cloud SQL - [Creating read replicas](https://cloud.google.com/sql/docs/postgres/replication/create-replica) -Once your read-only replica is set up, you can skip to [configure you secondary application node](#configure-secondary-application-nodes-to-use-the-external-read-replica). +Once your read-only replica is set up, you can skip to [configure your secondary application node](#configure-secondary-application-nodes-to-use-the-external-read-replica). #### Manually configure the primary database for replication @@ -182,9 +191,12 @@ to grant additional roles to your tracking database user (by default, this is - Amazon RDS requires the [`rds_superuser`](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.html#Appendix.PostgreSQL.CommonDBATasks.Roles) role. - Azure Database for PostgreSQL requires the [`azure_pg_admin`](https://docs.microsoft.com/en-us/azure/postgresql/howto-create-users#how-to-create-additional-admin-users-in-azure-database-for-postgresql) role. +- Google Cloud SQL requires the [`cloudsqlsuperuser`](https://cloud.google.com/sql/docs/postgres/users#default-users) role. -If you have an external database ready to be used as the tracking database, -follow the instructions below to use it: +This is for the installation of extensions during installation and upgrades. As an alternative, +[ensure the extensions are installed manually, and read about the problems that may arise during future GitLab upgrades](../../../install/postgresql_extensions.md). + +To setup an external tracking database, follow the instructions below: NOTE: If you want to use AWS RDS as a tracking database, make sure it has access to @@ -193,8 +205,12 @@ outbound rules do not apply to RDS PostgreSQL databases. Therefore, you need to rule to the read-replica's security group allowing any TCP traffic from the tracking database on port 5432. -1. Ensure that your secondary node can communicate with your tracking database by - manually changing the `pg_hba.conf` that is associated with your tracking database. +1. Set up PostgreSQL according to the + [database requirements document](../../../install/requirements.md#database). +1. Set up a `gitlab_geo` user with a password of your choice, create the `gitlabhq_geo_production` database, and make the user an owner of the database. You can see an example of this setup in the [installation from source documentation](../../../install/installation.md#6-database). +1. If you are **not** using a cloud-managed PostgreSQL database, ensure that your secondary + node can communicate with your tracking database by manually changing the + `pg_hba.conf` that is associated with your tracking database. Remember to restart PostgreSQL afterwards for the changes to take effect: ```plaintext @@ -226,9 +242,14 @@ the tracking database on port 5432. 1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) -1. Run the tracking database migrations: +1. The reconfigure should automatically create the database. If needed, you can perform this task manually. Note that this task (whether run by itself or during reconfigure) requires the database user to be a superuser. ```shell gitlab-rake geo:db:create + ``` + +1. The reconfigure should automatically migrate the database. You can migrate the database manually if needed, for example if `gitlab_rails['auto_migrate'] = false`: + + ```shell gitlab-rake geo:db:migrate ``` diff --git a/doc/administration/git_annex.md b/doc/administration/git_annex.md index 741b2a78b85..d7fb8a37b9c 100644 --- a/doc/administration/git_annex.md +++ b/doc/administration/git_annex.md @@ -1,5 +1,6 @@ --- redirect_to: 'index.md' +remove_date: '2021-07-22' --- This document was moved to [another location](index.md). diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md index c9b6fd4c510..0b22df5a115 100644 --- a/doc/administration/gitaly/configure_gitaly.md +++ b/doc/administration/gitaly/configure_gitaly.md @@ -82,7 +82,7 @@ The following list depicts the network architecture of Gitaly: - Gitaly addresses must be specified in such a way that they resolve correctly for **all** Gitaly clients. - Gitaly clients are: - - Puma or Unicorn. + - Puma. - Sidekiq. - GitLab Workhorse. - GitLab Shell. @@ -247,7 +247,6 @@ disable enforcement. For more information, see the documentation on configuring # node_exporter['enable'] = false # Prevent database connections during 'gitlab-ctl reconfigure' - gitlab_rails['rake_cache_clear'] = false gitlab_rails['auto_migrate'] = false # Configure the gitlab-shell API callback URL. Without this, `git push` will diff --git a/doc/administration/gitaly/faq.md b/doc/administration/gitaly/faq.md new file mode 100644 index 00000000000..98a90925d32 --- /dev/null +++ b/doc/administration/gitaly/faq.md @@ -0,0 +1,90 @@ +--- +stage: Create +group: Gitaly +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: reference +--- + +# Frequently asked questions **(FREE SELF)** + +The following are answers to frequently asked questions about Gitaly and Gitaly Cluster. + +## How does Gitaly Cluster compare to Geo? + +Gitaly Cluster and [Geo](../geo/index.md) both provide redundancy. However the redundancy of: + +- Gitaly Cluster provides fault tolerance for data storage and is invisible to the user. Users are + not aware when Gitaly Cluster is used. +- Geo provides [replication](../geo/index.md) and [disaster recovery](../geo/disaster_recovery/index.md) for + an entire instance of GitLab. Users know when they are using Geo for + [replication](../geo/index.md). Geo [replicates multiple data types](../geo/replication/datatypes.md#limitations-on-replicationverification), + including Git data. + +The following table outlines the major differences between Gitaly Cluster and Geo: + +| Tool | Nodes | Locations | Latency tolerance | Failover | Consistency | Provides redundancy for | +|:---------------|:---------|:----------|:-------------------|:----------------------------------------------------------------------------|:-----------------------------------------|:------------------------| +| Gitaly Cluster | Multiple | Single | Approximately 1 ms | [Automatic](praefect.md#automatic-failover-and-primary-election-strategies) | [Strong](praefect.md#strong-consistency) | Data storage in Git | +| Geo | Multiple | Multiple | Up to one minute | [Manual](../geo/disaster_recovery/index.md) | Eventual | Entire GitLab instance | + +For more information, see: + +- Geo [use cases](../geo/index.md#use-cases). +- Geo [architecture](../geo/index.md#architecture). + +## Are there instructions for migrating to Gitaly Cluster? + +Yes! For more information, see [Migrate to Gitaly Cluster](praefect.md#migrate-to-gitaly-cluster). + +## What are some repository storage recommendations? + +The size of the required storage can vary between instances and depends on the set +[replication factor](praefect.md#replication-factor). You might want to include implementing +repository storage redundancy. + +For a replication factor: + +- Of `1`: NFS, Gitaly, and Gitaly Cluster have roughly the same storage requirements. +- More than `1`: The amount of required storage is `used space * replication factor`. `used space` + should include any planned future growth. + +## What are some Praefect database storage requirements? + +The requirements are relatively low because the database contains only metadata of: + +- Where repositories are located. +- Some queued work. + +It depends on the number of repositories, but a useful minimum is 5-10 GB, similar to the main +GitLab application database. + +## Can the GitLab application database and the Praefect database be on the same servers? + +Yes, however Praefect should have it's own database server when using Omnibus GitLab PostgreSQL. If +there is a failover, Praefect isn't aware and starts to fail as the database it's trying to use would +either: + +- Be unavailable. +- In read-only mode. + +A future solution may allow for Praefect and Omnibus GitLab databases on the same PostgreSQL server. +For more information, see the relevant: + +- [Omnibus GitLab issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919). +- [Gitaly issue](https://gitlab.com/gitlab-org/gitaly/-/issues/3398). + +## Is PgBouncer required for the Praefect database? + +No, because the number of connections Praefect makes is low. You can use the same PgBouncer instance +for both the GitLab application database and the Praefect database if you wish. + +## Are there any special considerations for Gitaly Cluster when PostgreSQL is upgraded? + +There are no special requirements. Gitaly Cluster requires PostgreSQL version 11 or later. + +## Praefect database tables are empty? + +These tables are created per the [specific configuration section](praefect.md#postgresql). + +If you find you have an empty Praefect database table, see the +[relevant troubleshooting section](index.md#relation-does-not-exist-errors). diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index 3935e990590..eaf9e21780d 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -235,29 +235,6 @@ A hybrid approach can be used in these instances, where each shard is configured cluster. [Variable replication factor](https://gitlab.com/groups/gitlab-org/-/epics/3372) is planned to provide greater flexibility for extremely large GitLab instances. -### Gitaly Cluster compared to Geo - -Gitaly Cluster and [Geo](../geo/index.md) both provide redundancy. However the redundancy of: - -- Gitaly Cluster provides fault tolerance for data storage and is invisible to the user. Users are - not aware when Gitaly Cluster is used. -- Geo provides [replication](../geo/index.md) and [disaster recovery](../geo/disaster_recovery/index.md) for - an entire instance of GitLab. Users know when they are using Geo for - [replication](../geo/index.md). Geo [replicates multiple data types](../geo/replication/datatypes.md#limitations-on-replicationverification), - including Git data. - -The following table outlines the major differences between Gitaly Cluster and Geo: - -| Tool | Nodes | Locations | Latency tolerance | Failover | Consistency | Provides redundancy for | -|:---------------|:---------|:----------|:-------------------|:----------------------------------------------------------------------------|:-----------------------------------------|:------------------------| -| Gitaly Cluster | Multiple | Single | Approximately 1 ms | [Automatic](praefect.md#automatic-failover-and-primary-election-strategies) | [Strong](praefect.md#strong-consistency) | Data storage in Git | -| Geo | Multiple | Multiple | Up to one minute | [Manual](../geo/disaster_recovery/index.md) | Eventual | Entire GitLab instance | - -For more information, see: - -- Geo [use cases](../geo/index.md#use-cases). -- Geo [architecture](../geo/index.md#architecture). - ### Architecture Praefect is a router and transaction manager for Gitaly, and a required @@ -346,7 +323,6 @@ When GitLab calls a function that has a "Rugged patch", it performs two checks: the GitLab use of "Rugged patch" code. - If the feature flag is not set, GitLab tries accessing the file system underneath the Gitaly server directly. If it can, it uses the "Rugged patch": - - If using Unicorn. - If using Puma and [thread count](../../install/requirements.md#puma-threads) is set to `1`. @@ -404,25 +380,36 @@ GitLab recommends: We welcome your feedback on this process: raise a support ticket, or [comment on the epic](https://gitlab.com/groups/gitlab-org/-/epics/4916). -## Troubleshooting Gitaly +## Troubleshooting + +Refer to the information below when troubleshooting Gitaly and Gitaly Cluster. + +Before troubleshooting, see the Gitaly and Gitaly Cluster +[frequently asked questions](faq.md). + +### Troubleshoot Gitaly + +The following sections provide possible solutions to Gitaly errors. -Check [Gitaly timeouts](../../user/admin_area/settings/gitaly_timeouts.md) when troubleshooting -Gitaly. +See also [Gitaly timeout](../../user/admin_area/settings/gitaly_timeouts.md) settings. -### Check versions when using standalone Gitaly servers +#### Check versions when using standalone Gitaly servers When using standalone Gitaly servers, you must make sure they are the same version -as GitLab to ensure full compatibility. Check **Admin Area > Overview > Gitaly Servers** on -your GitLab instance and confirm all Gitaly servers indicate that they are up to date. +as GitLab to ensure full compatibility: -### `gitaly-debug` +1. On the top bar, select **Menu >** **{admin}** **Admin** on your GitLab instance. +1. On the left sidebar, select **Overview > Gitaly Servers**. +1. Confirm all Gitaly servers indicate that they are up to date. + +#### Use `gitaly-debug` The `gitaly-debug` command provides "production debugging" tools for Gitaly and Git performance. It is intended to help production engineers and support engineers investigate Gitaly performance problems. If you're using GitLab 11.6 or newer, this tool should be installed on -your GitLab / Gitaly server already at `/opt/gitlab/embedded/bin/gitaly-debug`. +your GitLab or Gitaly server already at `/opt/gitlab/embedded/bin/gitaly-debug`. If you're investigating an older GitLab version you can compile this tool offline and copy the executable to your server: @@ -438,19 +425,19 @@ To see the help page of `gitaly-debug` for a list of supported sub-commands, run gitaly-debug -h ``` -### Commits, pushes, and clones return a 401 +#### Commits, pushes, and clones return a 401 ```plaintext remote: GitLab: 401 Unauthorized ``` -You need to sync your `gitlab-secrets.json` file with your Gitaly clients (GitLab -app nodes). +You need to sync your `gitlab-secrets.json` file with your GitLab +application nodes. -### Client side gRPC logs +#### Client side gRPC logs Gitaly uses the [gRPC](https://grpc.io/) RPC framework. The Ruby gRPC -client has its own log file which may contain debugging information when +client has its own log file which may contain useful information when you are seeing Gitaly errors. You can control the log level of the gRPC client with the `GRPC_LOG_LEVEL` environment variable. The default level is `WARN`. @@ -461,7 +448,7 @@ You can run a gRPC trace with: sudo GRPC_TRACE=all GRPC_VERBOSITY=DEBUG gitlab-rake gitlab:gitaly:check ``` -### Server side gRPC logs +#### Server side gRPC logs gRPC tracing can also be enabled in Gitaly itself with the `GODEBUG=http2debug` environment variable. To set this in an Omnibus GitLab install: @@ -476,7 +463,7 @@ environment variable. To set this in an Omnibus GitLab install: 1. [Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab. -### Correlating Git processes with RPCs +#### Correlating Git processes with RPCs Sometimes you need to find out which Gitaly RPC created a particular Git process. @@ -494,7 +481,7 @@ sudo cat /proc/$PID/environ | tr '\0' '\n' | grep ^CORRELATION_ID= This method isn't reliable for `git cat-file` processes, because Gitaly internally pools and re-uses those across RPCs. -### Observing `gitaly-ruby` traffic +#### Observing `gitaly-ruby` traffic [`gitaly-ruby`](configure_gitaly.md#gitaly-ruby) is an internal implementation detail of Gitaly, so, there's not that much visibility into what goes on inside @@ -502,12 +489,13 @@ so, there's not that much visibility into what goes on inside If you have Prometheus set up to scrape your Gitaly process, you can see request rates and error codes for individual RPCs in `gitaly-ruby` by -querying `grpc_client_handled_total`. Strictly speaking, this metric does -not differentiate between `gitaly-ruby` and other RPCs. However from GitLab 11.9, -all gRPC calls made by Gitaly itself are internal calls from the main Gitaly process to one of its -`gitaly-ruby` sidecars. +querying `grpc_client_handled_total`. + +- In theory, this metric does not differentiate between `gitaly-ruby` and other RPCs. +- In practice from GitLab 11.9, all gRPC calls made by Gitaly itself are internal calls from the + main Gitaly process to one of its `gitaly-ruby` sidecars. -Assuming your `grpc_client_handled_total` counter observes only Gitaly, +Assuming your `grpc_client_handled_total` counter only observes Gitaly, the following query shows you RPCs are (most likely) internally implemented as calls to `gitaly-ruby`: @@ -515,7 +503,7 @@ implemented as calls to `gitaly-ruby`: sum(rate(grpc_client_handled_total[5m])) by (grpc_method) > 0 ``` -### Repository changes fail with a `401 Unauthorized` error +#### Repository changes fail with a `401 Unauthorized` error If you run Gitaly on its own server and notice these conditions: @@ -541,7 +529,7 @@ Confirm the following are all true: - When any user adds or modifies a file from the repository using the GitLab UI, it immediately fails with a red `401 Unauthorized` banner. - Creating a new project and [initializing it with a README](../../user/project/working_with_projects.md#blank-projects) - successfully creates the project, but doesn't create the README. + successfully creates the project but doesn't create the README. - When [tailing the logs](https://docs.gitlab.com/omnibus/settings/logs.html#tail-logs-in-a-console-on-the-server) on a Gitaly client and reproducing the error, you get `401` errors when reaching the [`/api/v4/internal/allowed`](../../development/internal_api.md) endpoint: @@ -610,7 +598,7 @@ on the Gitaly server matches the one on Gitaly client. If it doesn't match, update the secrets file on the Gitaly server to match the Gitaly client, then [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure). -### Command line tools cannot connect to Gitaly +#### Command line tools cannot connect to Gitaly gRPC cannot reach your Gitaly server if: @@ -623,22 +611,24 @@ Verify you can reach Gitaly by using TCP: sudo gitlab-rake gitlab:tcp_check[GITALY_SERVER_IP,GITALY_LISTEN_PORT] ``` -If the TCP connection fails, check your network settings and your firewall rules. -If the TCP connection succeeds, your networking and firewall rules are correct. +If the TCP connection: + +- Fails, check your network settings and your firewall rules. +- Succeeds, your networking and firewall rules are correct. -If you use proxy servers in your command line environment, such as Bash, these -can interfere with your gRPC traffic. +If you use proxy servers in your command line environment such as Bash, these can interfere with +your gRPC traffic. -If you use Bash or a compatible command line environment, run the following commands -to determine whether you have proxy servers configured: +If you use Bash or a compatible command line environment, run the following commands to determine +whether you have proxy servers configured: ```shell echo $http_proxy echo $https_proxy ``` -If either of these variables have a value, your Gitaly CLI connections may be -getting routed through a proxy which cannot connect to Gitaly. +If either of these variables have a value, your Gitaly CLI connections may be getting routed through +a proxy which cannot connect to Gitaly. To remove the proxy setting, run the following commands (depending on which variables had values): @@ -647,7 +637,7 @@ unset http_proxy unset https_proxy ``` -### Permission denied errors appearing in Gitaly or Praefect logs when accessing repositories +#### Permission denied errors appearing in Gitaly or Praefect logs when accessing repositories You might see the following in Gitaly and Praefect logs: @@ -668,9 +658,87 @@ This is a GRPC call [error response code](https://grpc.github.io/grpc/core/md_doc_statuscodes.html). If this error occurs, even though -[the Gitaly auth tokens are correctly setup](../gitaly/praefect.md#debugging-praefect), +[the Gitaly auth tokens are set up correctly](#praefect-errors-in-logs), it's likely that the Gitaly servers are experiencing [clock drift](https://en.wikipedia.org/wiki/Clock_drift). Ensure the Gitaly clients and servers are synchronized, and use an NTP time server to keep them synchronized. + +#### Gitaly not listening on new address after reconfiguring + +When updating the `gitaly['listen_addr']` or `gitaly['prometheus_listen_addr']` values, Gitaly may +continue to listen on the old address after a `sudo gitlab-ctl reconfigure`. + +When this occurs, run `sudo gitlab-ctl restart` to resolve the issue. This should no longer be +necessary because [this issue](https://gitlab.com/gitlab-org/gitaly/-/issues/2521) is resolved. + +#### Permission denied errors appearing in Gitaly logs when accessing repositories from a standalone Gitaly node + +If this error occurs even though file permissions are correct, it's likely that the Gitaly node is +experiencing [clock drift](https://en.wikipedia.org/wiki/Clock_drift). + +Please ensure that the GitLab and Gitaly nodes are synchronized and use an NTP time +server to keep them synchronized if possible. + +### Troubleshoot Praefect (Gitaly Cluster) + +The following sections provide possible solutions to Gitaly Cluster errors. + +#### Praefect errors in logs + +If you receive an error, check `/var/log/gitlab/gitlab-rails/production.log`. + +Here are common errors and potential causes: + +- 500 response code + - **ActionView::Template::Error (7:permission denied)** + - `praefect['auth_token']` and `gitlab_rails['gitaly_token']` do not match on the GitLab server. + - **Unable to save project. Error: 7:permission denied** + - Secret token in `praefect['storage_nodes']` on GitLab server does not match the + value in `gitaly['auth_token']` on one or more Gitaly servers. +- 503 response code + - **GRPC::Unavailable (14:failed to connect to all addresses)** + - GitLab was unable to reach Praefect. + - **GRPC::Unavailable (14:all SubCons are in TransientFailure...)** + - Praefect cannot reach one or more of its child Gitaly nodes. Try running + the Praefect connection checker to diagnose. + +#### Determine primary Gitaly node + +To determine the current primary Gitaly node for a specific Praefect node: + +- Use the `Shard Primary Election` [Grafana chart](praefect.md#grafana) on the + [`Gitlab Omnibus - Praefect` dashboard](https://gitlab.com/gitlab-org/grafana-dashboards/-/blob/master/omnibus/praefect.json). + This is recommended. +- If you do not have Grafana set up, use the following command on each host of each + Praefect node: + + ```shell + curl localhost:9652/metrics | grep gitaly_praefect_primaries` + ``` + +#### Relation does not exist errors + +By default Praefect database tables are created automatically by `gitlab-ctl reconfigure` task. +However, if the `gitlab-ctl reconfigure` command isn't executed or there are errors during the +execution, the Praefect database tables are not created on initial reconfigure and can throw +errors that relations do not exist. + +For example: + +- `ERROR: relation "node_status" does not exist at character 13` +- `ERROR: relation "replication_queue_lock" does not exist at character 40` +- This error: + + ```json + {"level":"error","msg":"Error updating node: pq: relation \"node_status\" does not exist","pid":210882,"praefectName":"gitlab1x4m:0.0.0.0:2305","time":"2021-04-01T19:26:19.473Z","virtual_storage":"praefect-cluster-1"} + ``` + +To solve this, the database schema migration can be done using `sql-migrate` sub-command of +the `praefect` command: + +```shell +$ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-migrate +praefect sql-migrate: OK (applied 21 migrations) +``` diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index a1733d9d6ac..21e5360e27b 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -9,13 +9,21 @@ type: reference Configure Gitaly Cluster using either: -- The Gitaly Cluster configuration instructions available as part of - [reference architectures](../reference_architectures/index.md) for installations for more than - 2000 users. -- The advanced configuration instructions that follow on this page. +- Gitaly Cluster configuration instructions available as part of + [reference architectures](../reference_architectures/index.md) for installations of up to: + - [3000 users](../reference_architectures/3k_users.md#configure-gitaly-cluster). + - [5000 users](../reference_architectures/5k_users.md#configure-gitaly-cluster). + - [10,000 users](../reference_architectures/10k_users.md#configure-gitaly-cluster). + - [25,000 users](../reference_architectures/25k_users.md#configure-gitaly-cluster). + - [50,000 users](../reference_architectures/50k_users.md#configure-gitaly-cluster). +- The custom configuration instructions that follow on this page. Smaller GitLab installations may need only [Gitaly itself](index.md). +NOTE: +Upgrade instructions for Omnibus GitLab installations +[are available](https://docs.gitlab.com/omnibus/update/#gitaly-cluster). + ## Requirements for configuring a Gitaly Cluster The minimum recommended configuration for a Gitaly Cluster requires: @@ -161,6 +169,9 @@ node, using `psql` which is installed by Omnibus GitLab. The database used by Praefect is now configured. +If you see Praefect database errors after configuring PostgreSQL, see +[troubleshooting steps](index.md#relation-does-not-exist-errors). + #### PgBouncer To reduce PostgreSQL resource consumption, we recommend setting up and configuring @@ -223,8 +234,10 @@ PostgreSQL instances. Otherwise you should change the configuration parameter > [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/2634) in GitLab 13.4, Praefect nodes can no longer be designated as `primary`. -NOTE: -If there are multiple Praefect nodes, complete these steps for **each** node. +If there are multiple Praefect nodes: + +- Complete the following steps for **each** node. +- Designate one node as the "deploy node", and configure it first. To complete this section you need a [configured PostgreSQL server](#postgresql), including: @@ -259,8 +272,8 @@ application server, or a Gitaly node. praefect['enable'] = true # Prevent database connections during 'gitlab-ctl reconfigure' - gitlab_rails['rake_cache_clear'] = false gitlab_rails['auto_migrate'] = false + praefect['auto_migrate'] = false ``` 1. Configure **Praefect** to listen on network interfaces by editing @@ -381,6 +394,30 @@ application server, or a Gitaly node. gitlab-ctl reconfigure ``` +1. For: + + - The "deploy node": + 1. Enable Praefect auto-migration again by setting `praefect['auto_migrate'] = true` in + `/etc/gitlab/gitlab.rb`. + 1. To ensure database migrations are only run during reconfigure and not automatically on + upgrade, run: + + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure + ``` + + - The other nodes, you can leave the settings as they are. Though + `/etc/gitlab/skip-auto-reconfigure` isn't required, you may want to set it to prevent GitLab + running reconfigure automatically when running commands such as `apt-get update`. This way any + additional configuration changes can be done and then reconfigure can be run manually. + +1. Save the changes to `/etc/gitlab/gitlab.rb` and [reconfigure + Praefect](../restart_gitlab.md#omnibus-gitlab-reconfigure): + + ```shell + gitlab-ctl reconfigure + ``` + 1. To ensure that Praefect [has updated its Prometheus listen address](https://gitlab.com/gitlab-org/gitaly/-/issues/2734), [restart Praefect](../restart_gitlab.md#omnibus-gitlab-restart): @@ -599,7 +636,6 @@ documentation](configure_gitaly.md#configure-gitaly-servers). prometheus['enable'] = true # Prevent database connections during 'gitlab-ctl reconfigure' - gitlab_rails['rake_cache_clear'] = false gitlab_rails['auto_migrate'] = false ``` @@ -696,7 +732,7 @@ documentation](configure_gitaly.md#configure-gitaly-servers). **The steps above must be completed for each Gitaly node!** -After all Gitaly nodes are configured, you can run the Praefect connection +After all Gitaly nodes are configured, run the Praefect connection checker to verify Praefect can connect to all Gitaly servers in the Praefect configuration. @@ -865,9 +901,13 @@ Particular attention should be shown to: gitlab-rake gitlab:gitaly:check ``` -1. Check in **Admin Area > Settings > Repository > Repository storage** that the Praefect storage - is configured to store new repositories. Following this guide, the `default` storage should have - weight 100 to store all new repositories. +1. Check that the Praefect storage is configured to store new repositories: + + 1. On the top bar, select **Menu >** **{admin}** **Admin**. + 1. On the left sidebar, select **Settings > Repository**. + 1. Expand the **Repository storage** section. + + Following this guide, the `default` storage should have weight 100 to store all new repositories. 1. Verify everything is working by creating a new project. Check the "Initialize repository with a README" box so that there is content in the @@ -878,7 +918,7 @@ Particular attention should be shown to: When adding Gitaly Cluster to an existing Gitaly instance, the existing Gitaly storage must use a TCP address. If `gitaly_address` is not specified, then a Unix socket is used, -which will prevent the communication with the cluster. +which prevents the communication with the cluster. For example: @@ -1160,15 +1200,36 @@ To migrate existing clusters: a Praefect node to reattempt it. The migration only runs with `sql` election strategy configured. 1. Running two different election strategies side by side can cause a split brain, where different - Praefect nodes consider repositories to have different primaries. To avoid this, shut down - all Praefect nodes before changing the election strategy. + Praefect nodes consider repositories to have different primaries. This can be avoided either: + + - If a short downtime is acceptable: - Do this by running `gitlab-ctl stop praefect` on the Praefect nodes. + 1. Shut down all Praefect nodes before changing the election strategy. Do this by running `gitlab-ctl stop praefect` on the Praefect nodes. -1. On the Praefect nodes, configure the election strategy in `/etc/gitlab/gitlab.rb` with - `praefect['failover_election_strategy'] = 'per_repository'`. + 1. On the Praefect nodes, configure the election strategy in `/etc/gitlab/gitlab.rb` with `praefect['failover_election_strategy'] = 'per_repository'`. -1. Finally, run `gitlab-ctl reconfigure` to reconfigure and restart the Praefect nodes. + 1. Run `gitlab-ctl reconfigure && gitlab-ctl start` to reconfigure and start the Praefects. + + - If downtime is unacceptable: + + 1. Determine which Gitaly node is [the current primary](index.md#determine-primary-gitaly-node). + + 1. Comment out the secondary Gitaly nodes from the virtual storage's configuration in `/etc/gitlab/gitlab.rb` + on all Praefect nodes. This ensures there's only one Gitaly node configured, causing both of the election + strategies to elect the same Gitaly node as the primary. + + 1. Run `gitlab-ctl reconfigure` on all Praefect nodes. Wait until all Praefect processes have restarted and + the old processes have exited. This can take up to one minute. + + 1. On all Praefect nodes, configure the election strategy in `/etc/gitlab/gitlab.rb` with + `praefect['failover_election_strategy'] = 'per_repository'`. + + 1. Run `gitlab-ctl reconfigure` on all Praefect nodes. Wait until all of the Praefect processes have restarted and + the old processes have exited. This can take up to one minute. + + 1. Uncomment the secondary Gitaly node configuration commented out in the earlier step on all Praefect nodes. + + 1. Run `gitlab-ctl reconfigure` on all Praefect nodes to reconfigure and restart the Praefect processes. ### Deprecated election strategies @@ -1428,15 +1489,8 @@ or to move from single Gitaly nodes, the basic process involves: 1. Create and configure Gitaly Cluster. 1. [Move the repositories](#move-repositories). -The size of the required storage can vary between instances and depends on the set -[replication factor](#replication-factor). The migration to Gitaly Cluster might include -implementing repository storage redundancy. - -For a replication factor: - -- Of `1`: NFS, Gitaly, and Gitaly Cluster have roughly the same storage requirements. -- More than `1`: The amount of required storage is `used space * replication factor`. `used space` - should include any planned future growth. +When creating the storage, see some +[repository storage recommendations](faq.md#what-are-some-repository-storage-recommendations). ### Move Repositories @@ -1467,8 +1521,10 @@ After creating and configuring Gitaly Cluster: 1. [Schedule repository storage moves for all projects on a storage shard](../../api/project_repository_storage_moves.md#schedule-repository-storage-moves-for-all-projects-on-a-storage-shard) using the API. For example: ```shell - curl --request POST --header "Private-Token: <your_access_token>" --header "Content-Type: application/json" \ - --data '{"source_storage_name":"<original_storage_name>","destination_storage_name":"<cluster_storage_name>"}' "https://gitlab.example.com/api/v4/project_repository_storage_moves" + curl --request POST --header "Private-Token: <your_access_token>" \ + --header "Content-Type: application/json" \ + --data '{"source_storage_name":"<original_storage_name>","destination_storage_name":"<cluster_storage_name>"}' \ + "https://gitlab.example.com/api/v4/project_repository_storage_moves" ``` 1. [Query the most recent repository moves](../../api/project_repository_storage_moves.md#retrieve-all-project-repository-storage-moves) @@ -1500,8 +1556,10 @@ After creating and configuring Gitaly Cluster: 1. [Schedule repository storage moves for all snippets on a storage shard](../../api/snippet_repository_storage_moves.md#schedule-repository-storage-moves-for-all-snippets-on-a-storage-shard) using the API. For example: ```shell - curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" \ - --data '{"source_storage_name":"<original_storage_name>","destination_storage_name":"<cluster_storage_name>"}' "https://gitlab.example.com/api/v4/snippet_repository_storage_moves" + curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + --header "Content-Type: application/json" \ + --data '{"source_storage_name":"<original_storage_name>","destination_storage_name":"<cluster_storage_name>"}' \ + "https://gitlab.example.com/api/v4/snippet_repository_storage_moves" ``` 1. [Query the most recent repository moves](../../api/snippet_repository_storage_moves.md#retrieve-all-snippet-repository-storage-moves) @@ -1525,8 +1583,10 @@ After creating and configuring Gitaly Cluster: 1. [Schedule repository storage moves for all groups on a storage shard](../../api/group_repository_storage_moves.md#schedule-repository-storage-moves-for-all-groups-on-a-storage-shard) using the API. ```shell - curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" \ - --data '{"source_storage_name":"<original_storage_name>","destination_storage_name":"<cluster_storage_name>"}' "https://gitlab.example.com/api/v4/group_repository_storage_moves" + curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + --header "Content-Type: application/json" \ + --data '{"source_storage_name":"<original_storage_name>","destination_storage_name":"<cluster_storage_name>"}' \ + "https://gitlab.example.com/api/v4/group_repository_storage_moves" ``` 1. [Query the most recent repository moves](../../api/group_repository_storage_moves.md#retrieve-all-group-repository-storage-moves) @@ -1544,35 +1604,3 @@ After creating and configuring Gitaly Cluster: ``` 1. Repeat for each storage as required. - -## Debugging Praefect - -If you receive an error, check `/var/log/gitlab/gitlab-rails/production.log`. - -Here are common errors and potential causes: - -- 500 response code - - **ActionView::Template::Error (7:permission denied)** - - `praefect['auth_token']` and `gitlab_rails['gitaly_token']` do not match on the GitLab server. - - **Unable to save project. Error: 7:permission denied** - - Secret token in `praefect['storage_nodes']` on GitLab server does not match the - value in `gitaly['auth_token']` on one or more Gitaly servers. -- 503 response code - - **GRPC::Unavailable (14:failed to connect to all addresses)** - - GitLab was unable to reach Praefect. - - **GRPC::Unavailable (14:all SubCons are in TransientFailure...)** - - Praefect cannot reach one or more of its child Gitaly nodes. Try running - the Praefect connection checker to diagnose. - -### Determine primary Gitaly node - -To determine the current primary Gitaly node for a specific Praefect node: - -- Use the `Shard Primary Election` [Grafana chart](#grafana) on the [`Gitlab Omnibus - Praefect` dashboard](https://gitlab.com/gitlab-org/grafana-dashboards/-/blob/master/omnibus/praefect.json). - This is recommended. -- If you do not have Grafana set up, use the following command on each host of each - Praefect node: - - ```shell - curl localhost:9652/metrics | grep gitaly_praefect_primaries` - ``` diff --git a/doc/administration/incoming_email.md b/doc/administration/incoming_email.md index 9aa6bffdb98..56af5f56cfa 100644 --- a/doc/administration/incoming_email.md +++ b/doc/administration/incoming_email.md @@ -21,9 +21,8 @@ GitLab has several features based on receiving incoming emails: ## Requirements -It is **not** recommended to use an email address that receives any -messages not intended for the GitLab instance. Any incoming emails not intended -for GitLab receive a reject notice. +We recommend using an email address that receives **only** messages that are intended for +the GitLab instance. Any incoming emails not intended for GitLab receive a reject notice. Handling incoming emails requires an [IMAP](https://en.wikipedia.org/wiki/Internet_Message_Access_Protocol)-enabled email account. GitLab requires one of the following three strategies: @@ -131,6 +130,9 @@ list. ```shell sudo gitlab-ctl reconfigure + + # Needed when enabling or disabling for the first time but not for password changes. + # See https://gitlab.com/gitlab-org/gitlab-foss/-/issues/23560#note_61966788 sudo gitlab-ctl restart ``` @@ -469,11 +471,6 @@ This series of PowerShell commands enables [sub-addressing](#email-sub-addressin at the organization level in Office 365. This allows all mailboxes in the organization to receive sub-addressed mail: -NOTE: -This series of commands enables sub-addressing at the organization -level in Office 365. This allows all mailboxes in the organization -to receive sub-addressed mail. - ```powershell Set-ExecutionPolicy RemoteSigned -Scope CurrentUser diff --git a/doc/administration/index.md b/doc/administration/index.md index 1bc2084a23a..69e8689c589 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -54,7 +54,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Environment variables](environment_variables.md): Supported environment variables that can be used to override their default values to configure GitLab. -- [Plugins](file_hooks.md): With custom plugins, GitLab administrators can +- [File hooks](file_hooks.md): With custom file hooks, GitLab administrators can introduce custom integrations without modifying GitLab source code. - [Enforcing Terms of Service](../user/admin_area/settings/terms.md) - [Third party offers](../user/admin_area/settings/third_party_offers.md) @@ -103,7 +103,6 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [GitLab in maintenance mode](maintenance_mode/index.md): Put GitLab in maintenance mode. - [Update GitLab](../update/index.md): Update guides to upgrade your installation to a new version. - [Upgrading without downtime](../update/index.md#upgrading-without-downtime): Upgrade to a newer major, minor, or patch version of GitLab without taking your GitLab instance offline. -- [Migrate your GitLab CI/CD data to another version of GitLab](../migrate_ci_to_ce/README.md): If you have an old GitLab installation (older than 8.0), follow this guide to migrate your existing GitLab CI/CD data to another version of GitLab. ### Upgrading or downgrading GitLab @@ -172,7 +171,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [External Pipeline Validation](external_pipeline_validation.md): Enable, disable, and configure external pipeline validation. - [Job artifacts](job_artifacts.md): Enable, disable, and configure job artifacts (a set of files and directories which are outputted by a job when it completes successfully). - [Job logs](job_logs.md): Information about the job logs. -- [Register runners](../ci/runners/README.md#types-of-runners): Learn how to register and configure runners. +- [Register runners](../ci/runners/runners_scope.md): Learn how to register and configure runners. - [Shared runners pipelines quota](../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota): Limit the usage of pipeline minutes for shared runners. - [Enable/disable Auto DevOps](../topics/autodevops/index.md#enable-or-disable-auto-devops): Enable or disable Auto DevOps for your instance. diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md index 9ea76ff6151..9423045e3b5 100644 --- a/doc/administration/instance_limits.md +++ b/doc/administration/instance_limits.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# GitLab application limits +# GitLab application limits **(FREE SELF)** GitLab, like most large applications, enforces limits within certain features to maintain a minimum quality of performance. Allowing some features to be limitless could affect security, @@ -112,7 +112,7 @@ Limit the maximum daily member invitations allowed per group hierarchy. - GitLab.com: Free members may invite 20 members per day. - Self-managed: Invites are not limited. -### Webhook calls +### Webhook rate limit > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/61151) in GitLab 13.12. > - [Deployed behind a feature flag](../user/feature_flags.md), disabled by default. @@ -169,8 +169,8 @@ Read more about [Gitaly concurrency limits](gitaly/configure_gitaly.md#limit-rpc There's a limit to the number of comments that can be submitted on an issue, merge request, or commit. When the limit is reached, system notes can still be -added so that the history of events is not lost, but user-submitted comments -will fail. +added so that the history of events is not lost, but the user-submitted +comment fails. - **Max limit**: 5,000 comments. @@ -179,10 +179,10 @@ will fail. > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/61974) in GitLab 12.2. There is a limit to the size of comments and descriptions of issues, merge requests, and epics. -Attempting to add a body of text larger than the limit will result in an error, and the -item will not be created. +Attempting to add a body of text larger than the limit, results in an error, and the +item is also not created. -It's possible that this limit will be changed to a lower number in the future. +It's possible that this limit changes to a lower number in the future. - **Max size**: ~1 million characters / ~1 MB. @@ -191,8 +191,8 @@ It's possible that this limit will be changed to a lower number in the future. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/292039) in GitLab 13.9 Commits with arbitrarily large messages may be pushed to GitLab, but when -displaying commits, titles (the first line of the commit message) will be -limited to 1KiB, and descriptions (the rest of the message) will be limited to +displaying commits, titles (the first line of the commit message) +limits to 1KiB, and descriptions (the rest of the message) limits to 1MiB. ## Number of issues in the milestone overview @@ -316,7 +316,7 @@ each time a new pipeline is created. An active pipeline is any pipeline in one o - `running` If a new pipeline would cause the total number of jobs to exceed the limit, the pipeline -will fail with a `job_activity_limit_exceeded` error. +fails with a `job_activity_limit_exceeded` error. - GitLab SaaS subscribers have different limits [defined per plan](../user/gitlab_com/index.md#gitlab-cicd), and they affect all projects under that plan. @@ -367,7 +367,7 @@ The total number of subscriptions can be limited per project. This limit is checked each time a new subscription is created. If a new subscription would cause the total number of subscription to exceed the -limit, the subscription will be considered invalid. +limit, the subscription is considered invalid. - GitLab SaaS subscribers have different limits [defined per plan](../user/gitlab_com/index.md#gitlab-cicd), and they affect all projects under that plan. @@ -391,7 +391,7 @@ Set the limit to `0` to disable it. The total number of pipeline schedules can be limited per project. This limit is checked each time a new pipeline schedule is created. If a new pipeline schedule would cause the total number of pipeline schedules to exceed the limit, the -pipeline schedule will not be created. +pipeline schedule is not created. GitLab SaaS subscribers have different limits [defined per plan](../user/gitlab_com/index.md#gitlab-cicd), and they affect all projects under that plan. @@ -413,7 +413,7 @@ Plan.default.actual_limits.update!(ci_pipeline_schedules: 100) The total number of instance level CI/CD variables is limited at the instance level. This limit is checked each time a new instance level variable is created. If a new variable -would cause the total number of variables to exceed the limit, the new variable will not be created. +would cause the total number of variables to exceed the limit, the new variable is created. On self-managed instances this limit is defined for the `default` plan. By default, this limit is set to `25`. @@ -482,10 +482,9 @@ Plan.default.actual_limits.update!(ci_max_artifact_size_junit: 10) > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/321368) in GitLab 13.12. -The total number of registered runners is limited at the group and project -levels. Each time a new runner is registered, GitLab checks these limits. A -runner's registration fails if it exceeds the limit for the scope determined by -the runner registration token. +The total number of registered runners is limited at the group and project levels. Each time a new runner is registered, +GitLab checks these limits against runners that have been active in the last 3 months. +A runner's registration fails if it exceeds the limit for the scope determined by the runner registration token. - GitLab SaaS subscribers have different limits defined per plan, affecting all projects using that plan. - Self-managed GitLab Premium and Ultimate limits are defined by a default plan that affects all projects: @@ -574,7 +573,26 @@ See [Environment Dashboard](../ci/environments/environments_dashboard.md#adding- Pods and Deployments. However, data over 10 MB for a certain environment read from Kubernetes won't be shown. -## Merge request reports +## Merge requests + +### Diff limits + +GitLab has limits around: + +- The patch size for a single file. [This is configurable on self-managed instance](../user/admin_area/diff_limits.md). +- The total size of all the diffs for a merge request. + +An upper and lower limit applies to each of these: + +- The number of changed files. +- The number of changed lines. +- The cumulative size of the changes displayed. + +The lower limits result in additional diffs being collapsed. The higher limits +prevent any more changes from rendering. For more information about these limits, +[read the development documentation](../development/diffs.md#diff-limits). + +### Merge request reports size limit Reports that go over the 20 MB limit won't be loaded. Affected reports: @@ -589,8 +607,8 @@ Reports that go over the 20 MB limit won't be loaded. Affected reports: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/8638) in GitLab 13.3. You can set a limit on the content of repository files that are indexed in -Elasticsearch. Any files larger than this limit will not be indexed, and thus -will not be searchable. +Elasticsearch. Any files larger than this limit is neither indexed +nor searchable. Setting a limit helps reduce the memory usage of the indexing processes and the overall index size. This value defaults to `1024 KiB` (1 MiB) as any @@ -598,8 +616,8 @@ text files larger than this likely aren't meant to be read by humans. You must set a limit, as unlimited file sizes aren't supported. Setting this value to be greater than the amount of memory on GitLab Sidekiq nodes causes -the GitLab Sidekiq nodes to run out of memory, as they will pre-allocate this -amount of memory during indexing. +the GitLab Sidekiq nodes to run out of memory, as this amount of memory +is pre-allocated during indexing. ### Maximum field length @@ -607,18 +625,17 @@ amount of memory during indexing. You can set a limit on the content of text fields indexed for Advanced Search. Setting a maximum helps to reduce the load of the indexing processes. If any -text field exceeds this limit then the text will be truncated to this number of -characters and the rest will not be indexed and hence will not be searchable. -This is applicable to all indexed data except repository files that get -indexed, which have a separate limit (see [Maximum file size -indexed](#maximum-file-size-indexed)). - -- On GitLab.com, this is limited to 20,000 characters -- For self-managed installations, this is unlimited by default +text field exceeds this limit, then the text is truncated to this number of +characters. The rest of the text is not indexed, and not searchable. +This applies to all indexed data except repository files that get +indexed, which have a separate limit. For more information, read +[Maximum file size indexed](#maximum-file-size-indexed). -This limit can be configured for self-managed installations when [enabling -Elasticsearch](../integration/elasticsearch.md#enabling-advanced-search). +- On GitLab.com, the field length limit is 20,000 characters. +- For self-managed installations, the field length is unlimited by default. +You can configure this limit for self-managed installations when you +[enable Elasticsearch](../integration/elasticsearch.md#enabling-advanced-search). Set the limit to `0` to disable it. ## Wiki limits @@ -653,7 +670,7 @@ More information can be found in these docs: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/31007) in GitLab 12.4. Total number of changes (branches or tags) in a single push to determine whether -individual push events or bulk push event will be created. +individual push events or a bulk push event are created. More information can be found in the [Push event activities limit and bulk push events documentation](../user/admin_area/settings/push_event_activities_limit.md). diff --git a/doc/administration/integration/kroki.md b/doc/administration/integration/kroki.md index 9e9ea62c44e..e36b8a0be9d 100644 --- a/doc/administration/integration/kroki.md +++ b/doc/administration/integration/kroki.md @@ -6,10 +6,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Kroki diagrams **(FREE SELF)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/241744) in GitLab 13.7. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/241744) in GitLab 13.7. +> - Support for reStructuredText and Textile documents [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/324766) in GitLab 13.12. When [Kroki](https://kroki.io) integration is enabled and configured in -GitLab you can use it to create diagrams in AsciiDoc and Markdown documents. +GitLab you can use it to create diagrams in AsciiDoc, Markdown, reStructuredText, and Textile documents. ## Kroki Server @@ -55,7 +56,7 @@ read the [Kroki installation](https://docs.kroki.io/kroki/setup/install/#_images You need to enable Kroki integration from Settings under Admin Area. To do that, log in with an administrator account and follow these steps: -1. Select the Admin Area (**{admin}**) icon. +1. On the top bar, select **Menu >** **{admin}** **Admin**. 1. Go to **Settings > General**. 1. Expand the **Kroki** section. 1. Select **Enable Kroki** checkbox. @@ -85,13 +86,29 @@ your AsciiDoc or Markdown documentation using delimited blocks: .... ``` +- **reStructuredText** + + ```plaintext + .. code-block:: plantuml + + Bob->Alice : hello + Alice -> Bob : hi + ``` + +- **Textile** + + ```plaintext + bc[plantuml]. Bob->Alice : hello + Alice -> Bob : hi + ``` + The above blocks are converted to an HTML image tag with source pointing to the Kroki instance. If the Kroki server is correctly configured, this should render a nice diagram instead of the block:  -Kroki supports more than a dozen diagram libraries. Here's a few examples: +Kroki supports more than a dozen diagram libraries. Here's a few examples for AsciiDoc: **GraphViz** diff --git a/doc/administration/integration/plantuml.md b/doc/administration/integration/plantuml.md index 834f4047fdd..2b18efde95d 100644 --- a/doc/administration/integration/plantuml.md +++ b/doc/administration/integration/plantuml.md @@ -206,7 +206,7 @@ stop; After configuring your local PlantUML server, you're ready to enable the PlantUML integration: 1. Sign in to GitLab as an [Administrator](../../user/permissions.md) user. -1. In the top menu, click **{admin}** **Admin Area**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. 1. In the left sidebar, go to **Settings > General** and expand the **PlantUML** section. 1. Select the **Enable PlantUML** check box. 1. Set the PlantUML instance as `https://gitlab.example.com/-/plantuml/`, diff --git a/doc/administration/integration/terminal.md b/doc/administration/integration/terminal.md index 644e2d905ae..9302e9a1edc 100644 --- a/doc/administration/integration/terminal.md +++ b/doc/administration/integration/terminal.md @@ -6,8 +6,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Web terminals **(FREE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/7690) in GitLab 8.15. - With the introduction of the [Kubernetes integration](../../user/project/clusters/index.md), GitLab can store and use credentials for a Kubernetes cluster. GitLab uses these credentials to provide access to @@ -99,9 +97,10 @@ they receive a `Connection failed` message. ## Limiting WebSocket connection time -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/8413) in GitLab 8.17. +By default, terminal sessions do not expire. To limit the terminal session +lifetime in your GitLab instance: -Terminal sessions, by default, do not expire. -You can limit terminal session lifetime in your GitLab instance. To do so, -go to [**Admin Area > Settings > Web terminal**](../../user/admin_area/settings/index.md#general), -and set a `max session time`. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. Select + [**Settings > Web terminal**](../../user/admin_area/settings/index.md#general). +1. Set a `max session time`. diff --git a/doc/administration/issue_closing_pattern.md b/doc/administration/issue_closing_pattern.md index 3b9ec74e6ee..31e9944d9a4 100644 --- a/doc/administration/issue_closing_pattern.md +++ b/doc/administration/issue_closing_pattern.md @@ -20,7 +20,7 @@ in the project's default branch. In order to change the pattern you need to have access to the server that GitLab is installed on. -The default pattern can be located in [`gitlab.yml.example`](https://gitlab.com/gitlab-org/gitlab/blob/master/config/gitlab.yml.example) +The default pattern can be located in [`gitlab.yml.example`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/gitlab.yml.example) under the "Automatic issue closing" section. NOTE: diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index 187e98e9b43..99eb1395503 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Testing info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference, howto --- @@ -113,7 +113,7 @@ For source installations the following settings are nested under `artifacts:` an | Setting | Default | Description | |---------------------|---------|-------------| | `enabled` | `false` | Enable or disable object storage. | -| `remote_directory` | | The bucket name where Artifacts are stored. | +| `remote_directory` | | The bucket name where Artifacts are stored. Use the name only, do not include the path. | | `direct_upload` | `false` | Set to `true` to enable direct upload of Artifacts without the need of local shared storage. Option may be removed once we decide to support only single storage for all files. | | `background_upload` | `true` | Set to `false` to disable automatic upload. Option may be removed once upload is direct to S3. | | `proxy_download` | `false` | Set to `true` to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data. | diff --git a/doc/administration/job_logs.md b/doc/administration/job_logs.md index 93f83311cad..510da68442c 100644 --- a/doc/administration/job_logs.md +++ b/doc/administration/job_logs.md @@ -10,7 +10,7 @@ type: reference > [Renamed from job traces to job logs](https://gitlab.com/gitlab-org/gitlab/-/issues/29121) in GitLab 12.5. Job logs are sent by a runner while it's processing a job. You can see -logs in job pages, pipelines, email notifications, etc. +logs in job pages, pipelines, email notifications, and so on. ## Data flow @@ -20,9 +20,8 @@ In the following table you can see the phases a log goes through: | Phase | State | Condition | Data flow | Stored path | | -------------- | ------------ | ----------------------- | -----------------------------------------| ----------- | | 1: patching | log | When a job is running | Runner => Puma => file storage | `#{ROOT_PATH}/gitlab-ci/builds/#{YYYY_mm}/#{project_id}/#{job_id}.log` | -| 2: overwriting | log | When a job is finished | Runner => Puma => file storage | `#{ROOT_PATH}/gitlab-ci/builds/#{YYYY_mm}/#{project_id}/#{job_id}.log` | -| 3: archiving | archived log | After a job is finished | Sidekiq moves log to artifacts folder | `#{ROOT_PATH}/gitlab-rails/shared/artifacts/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/job.log` | -| 4: uploading | archived log | After a log is archived | Sidekiq moves archived log to [object storage](#uploading-logs-to-object-storage) (if configured) | `#{bucket_name}/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/job.log` | +| 2: archiving | archived log | After a job is finished | Sidekiq moves log to artifacts folder | `#{ROOT_PATH}/gitlab-rails/shared/artifacts/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/job.log` | +| 3: uploading | archived log | After a log is archived | Sidekiq moves archived log to [object storage](#uploading-logs-to-object-storage) (if configured) | `#{bucket_name}/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/job.log` | The `ROOT_PATH` varies per environment. For Omnibus GitLab it would be `/var/opt/gitlab`, and for installations from source @@ -128,6 +127,11 @@ This command permanently deletes the log files and is irreversible. find /var/opt/gitlab/gitlab-rails/shared/artifacts -name "job.log" -mtime +60 -delete ``` +NOTE: +After execution, broken file references can be reported when running +[`sudo gitlab-rake gitlab:artifacts:check`](raketasks/check.md#uploaded-files-integrity). +For more information, see [delete references to missing artifacts](raketasks/check.md#delete-references-to-missing-artifacts). + ## Incremental logging architecture > - [Deployed behind a feature flag](../user/feature_flags.md), disabled by default. diff --git a/doc/administration/libravatar.md b/doc/administration/libravatar.md index 552dc7095e2..b62e2ed4313 100644 --- a/doc/administration/libravatar.md +++ b/doc/administration/libravatar.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Using the Libravatar service with GitLab +# Using the Libravatar service with GitLab **(FREE SELF)** GitLab by default supports the [Gravatar](https://gravatar.com) avatar service. @@ -17,7 +17,7 @@ server. ## Configuration -In the [`gitlab.yml` gravatar section](https://gitlab.com/gitlab-org/gitlab/blob/672bd3902d86b78d730cea809fce312ec49d39d7/config/gitlab.yml.example#L122), set +In the [`gitlab.yml` gravatar section](https://gitlab.com/gitlab-org/gitlab/-/blob/672bd3902d86b78d730cea809fce312ec49d39d7/config/gitlab.yml.example#L122), set the configuration options as follows: ### For HTTP diff --git a/doc/administration/load_balancer.md b/doc/administration/load_balancer.md index ae96989a188..ca66791166e 100644 --- a/doc/administration/load_balancer.md +++ b/doc/administration/load_balancer.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Load Balancer for multi-node GitLab +# Load Balancer for multi-node GitLab **(FREE SELF)** In an multi-node GitLab configuration, you need a load balancer to route traffic to the application servers. The specifics on which load balancer to use diff --git a/doc/administration/logs.md b/doc/administration/logs.md index 289e2cb5362..cf9c2143d8c 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -21,6 +21,42 @@ including adjusting log retention, log forwarding, switching logs from JSON to plain text logging, and more. - [How to parse and analyze JSON logs](troubleshooting/log_parsing.md). +## Log Rotation + +The logs for a given service may be managed and rotated by: + +- `logrotate` +- `svlogd` (`runit`'s service logging daemon) +- `logrotate` and `svlogd` +- Or not at all + +The table below includes information about what is responsible for managing and rotating logs for +the included services. Logs +[managed by `svlogd`](https://docs.gitlab.com/omnibus/settings/logs.html#runit-logs) +are written to a file called `current`. The `logrotate` service built into GitLab +[manages all logs](https://docs.gitlab.com/omnibus/settings/logs.html#logrotate) +except those captured by `runit`. + +| Log Type | Managed by logrotate | Managed by svlogd/runit | +| ----------------------------------------------- | -------------------- | ----------------------- | +| [Alertmanager Logs](#alertmanager-logs) | N | Y | +| [Crond Logs](#crond-logs) | N | Y | +| [Gitaly](#gitaly-logs) | Y | Y | +| [GitLab Exporter For Omnibus](#gitlab-exporter) | N | Y | +| [GitLab Pages Logs](#pages-logs) | Y | Y | +| GitLab Rails | Y | N | +| [GitLab Shell Logs](#gitlab-shelllog) | Y | N | +| [Grafana Logs](#grafana-logs) | N | Y | +| [LogRotate Logs](#logrotate-logs) | N | Y | +| [Mailroom](#mail_room_jsonlog-default) | Y | Y | +| [NGINX](#nginx-logs) | Y | Y | +| [PostgreSQL Logs](#postgresql-logs) | N | Y | +| [Prometheus Logs](#prometheus-logs) | N | Y | +| [Puma](#puma-logs) | Y | Y | +| [Redis Logs](#redis-logs) | N | Y | +| [Registry Logs](#registry-logs) | N | Y | +| [Workhorse Logs](#workhorse-logs) | Y | Y | + ## `production_json.log` This file lives in `/var/log/gitlab/gitlab-rails/production_json.log` for @@ -275,7 +311,7 @@ installations from source. It contains the JSON version of the logs in `application.log` like the example below: -``` json +```json { "severity":"INFO", "time":"2020-01-14T13:35:15.466Z", @@ -302,7 +338,7 @@ It contains information about [integrations](../user/project/integrations/overvi { "severity":"ERROR", "time":"2018-09-06T14:56:20.439Z", - "service_class":"JiraService", + "service_class":"Integrations::Jira", "project_id":8, "project_path":"h5bp/html5-boilerplate", "message":"Error sending message", @@ -312,7 +348,7 @@ It contains information about [integrations](../user/project/integrations/overvi { "severity":"INFO", "time":"2018-09-06T17:15:16.365Z", - "service_class":"JiraService", + "service_class":"Integrations::Jira", "project_id":3, "project_path":"namespace2/project2", "message":"Successfully posted", @@ -611,41 +647,6 @@ This file lives in `/var/log/gitlab/puma/puma_stderr.log` for Omnibus GitLab packages, or in `/home/git/gitlab/log/puma_stderr.log` for installations from source. -## Unicorn Logs - -Starting with GitLab 13.0, Puma is the default web server used in GitLab -all-in-one package based installations, and GitLab Helm chart deployments. - -### `unicorn_stdout.log` - -This file lives in `/var/log/gitlab/unicorn/unicorn_stdout.log` for -Omnibus GitLab packages or in `/home/git/gitlab/log/unicorn_stdout.log` for -for installations from source. - -### `unicorn_stderr.log` - -This file lives in `/var/log/gitlab/unicorn/unicorn_stderr.log` for -Omnibus GitLab packages or in `/home/git/gitlab/log/unicorn_stderr.log` for -for installations from source. - -These logs contain all information about the state of Unicorn processes at any given time. - -```plaintext -I, [2015-02-13T06:14:46.680381 #9047] INFO -- : Refreshing Gem list -I, [2015-02-13T06:14:56.931002 #9047] INFO -- : listening on addr=127.0.0.1:8080 fd=12 -I, [2015-02-13T06:14:56.931381 #9047] INFO -- : listening on addr=/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket fd=13 -I, [2015-02-13T06:14:56.936638 #9047] INFO -- : master process ready -I, [2015-02-13T06:14:56.946504 #9092] INFO -- : worker=0 spawned pid=9092 -I, [2015-02-13T06:14:56.946943 #9092] INFO -- : worker=0 ready -I, [2015-02-13T06:14:56.947892 #9094] INFO -- : worker=1 spawned pid=9094 -I, [2015-02-13T06:14:56.948181 #9094] INFO -- : worker=1 ready -W, [2015-02-13T07:16:01.312916 #9094] WARN -- : #<Unicorn::HttpServer:0x0000000208f618>: worker (pid: 9094) exceeds memory limit (320626688 bytes > 247066940 bytes) -W, [2015-02-13T07:16:01.313000 #9094] WARN -- : Unicorn::WorkerKiller send SIGQUIT (pid: 9094) alive: 3621 sec (trial 1) -I, [2015-02-13T07:16:01.530733 #9047] INFO -- : reaped #<Process::Status: pid 9094 exit 0> worker=1 -I, [2015-02-13T07:16:01.534501 #13379] INFO -- : worker=1 spawned pid=13379 -I, [2015-02-13T07:16:01.534848 #13379] INFO -- : worker=1 ready -``` - ## `repocheck.log` This file lives in `/var/log/gitlab/gitlab-rails/repocheck.log` for @@ -774,7 +775,7 @@ When enabled, access logs are generated in packages or in `/home/git/gitlab/log/sidekiq_exporter.log` for installations from source. -If Prometheus metrics and the Web Exporter are both enabled, Puma/Unicorn +If Prometheus metrics and the Web Exporter are both enabled, Puma starts a Web server and listen to the defined port (default: `8083`), and access logs are generated: @@ -824,7 +825,7 @@ Line breaks have been added to the following example line for clarity: This file logs the information about exceptions being tracked by `Gitlab::ErrorTracking`, which provides a standard and consistent way of -[processing rescued exceptions](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/development/logging.md#exception-handling). This file is stored in: +[processing rescued exceptions](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/logging.md#exception-handling). This file is stored in: - `/var/log/gitlab/gitlab-rails/exceptions_json.log` for Omnibus GitLab packages. - `/home/git/gitlab/log/exceptions_json.log` for installations from source. diff --git a/doc/administration/maintenance_mode/index.md b/doc/administration/maintenance_mode/index.md index 7fd383eabae..c73a49287db 100644 --- a/doc/administration/maintenance_mode/index.md +++ b/doc/administration/maintenance_mode/index.md @@ -35,8 +35,8 @@ There are three ways to enable Maintenance Mode as an administrator: - [**Rails console**](../operations/rails_console.md#starting-a-rails-console-session): ```ruby - ::Gitlab::CurrentSettings.update_attributes!(maintenance_mode: true) - ::Gitlab::CurrentSettings.update_attributes!(maintenance_mode_message: "New message") + ::Gitlab::CurrentSettings.update!(maintenance_mode: true) + ::Gitlab::CurrentSettings.update!(maintenance_mode_message: "New message") ``` ## Disable Maintenance Mode @@ -143,25 +143,25 @@ is turned off. ### Deployments -Deployments won't go through because pipelines will be unfinished. +Deployments don't go through because pipelines are unfinished. It is recommended to disable auto deploys during Maintenance Mode, and enable them once it is disabled. #### Terraform integration -Terraform integration depends on running CI pipelines, hence it will be blocked. +Terraform integration depends on running CI pipelines, hence it is blocked. ### Container Registry -`docker push` will fail with this error: `denied: requested access to the resource is denied`, but `docker pull` will work. +`docker push` fails with this error: `denied: requested access to the resource is denied`, but `docker pull` works. ### Package Registry -Package Registry will allow you to install but not publish packages. +Package Registry allows you to install but not publish packages. ### Background jobs -Background jobs (cron jobs, Sidekiq) will continue running as is, because background jobs are not automatically disabled. +Background jobs (cron jobs, Sidekiq) continue running as is, because background jobs are not automatically disabled. [During a planned Geo failover](../geo/disaster_recovery/planned_failover.md#prevent-updates-to-the-primary-node), it is recommended that you disable all cron jobs except for those related to Geo. @@ -170,34 +170,34 @@ You can monitor queues and disable jobs in **Admin Area > Monitoring > Backgroun ### Incident management -[Incident management](../../operations/incident_management/index.md) functions will be limited. The creation of [alerts](../../operations/incident_management/alerts.md) and [incidents](../../operations/incident_management/incidents.md#incident-creation) will be paused entirely. Notifications and paging on alerts and incidents will therefore be disabled. +[Incident management](../../operations/incident_management/index.md) functions are limited. The creation of [alerts](../../operations/incident_management/alerts.md) and [incidents](../../operations/incident_management/incidents.md#incident-creation) are paused entirely. Notifications and paging on alerts and incidents are therefore disabled. ### Feature flags - [Development feature flags](../../development/feature_flags/index.md) cannot be turned on or off through the API, but can be toggled through the Rails console. -- [The feature flag service](../../operations/feature_flags.md) will respond to feature flag checks but feature flags cannot be toggled +- [The feature flag service](../../operations/feature_flags.md) responds to feature flag checks but feature flags cannot be toggled ### Geo secondaries -When primary is in Maintenance Mode, secondary will also automatically go into Maintenance Mode. +When primary is in Maintenance Mode, secondary also automatically goes into Maintenance Mode. It is important that you do not disable replication before enabling Maintenance Mode. -Replication and verification will continue to work but proxied Git pushes to primary will not work. +Replication and verification continues to work but proxied Git pushes to primary do not work. ### Secure features -Features that depend on creating issues or creating or approving Merge Requests, will not work. +Features that depend on creating issues or creating or approving Merge Requests, do not work. -Exporting a vulnerability list from a Vulnerability Report page will not work. +Exporting a vulnerability list from a Vulnerability Report page does not work. -Changing the status on a finding or vulnerability object will not work, even though no error is shown in the UI. +Changing the status on a finding or vulnerability object does not work, even though no error is shown in the UI. SAST and Secret Detection cannot be initiated because they depend on passing CI jobs to create artifacts. ## An example use case: a planned failover -In the use case of [a planned failover](../geo/disaster_recovery/planned_failover.md), a few writes in the primary database are acceptable, since they will be replicated quickly and are not significant in number. +In the use case of [a planned failover](../geo/disaster_recovery/planned_failover.md), a few writes in the primary database are acceptable, since they are replicated quickly and are not significant in number. For the same reason we don't automatically block background jobs when Maintenance Mode is enabled. diff --git a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md index 8d3c8555660..b1ec74c2f40 100644 --- a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md +++ b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md @@ -20,7 +20,8 @@ GitLab instance. All administrators at the time of creation of the project and group are added as maintainers of the group and project, and as an administrator, you can -add new members to the group to give them maintainer access to the project. +add new members to the group to give them the [Maintainer role](../../../user/permissions.md) for +the project. This project is used to self monitor your GitLab instance. The metrics dashboard of the project shows some basic resource usage charts, such as CPU and memory usage @@ -32,9 +33,12 @@ metrics exposed by the [GitLab exporter](../prometheus/gitlab_metrics.md#metrics ## Creating the self monitoring project -1. Go to **Admin Area > Settings > Metrics and profiling** and expand the **Self monitoring** section. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > Metrics and profiling** and expand **Self monitoring**. 1. Toggle the **Create Project** button on. -1. Once your GitLab instance creates the project, GitLab displays a link to the project in the text above the **Create Project** toggle. You can also find it under **Projects > Your projects**. +1. After your GitLab instance creates the project, GitLab displays a link to the + project in the text above the **Create Project** toggle. You can also find it + from the top bar by selecting **Menu > Project**, then selecting **Your projects**. ## Deleting the self monitoring project @@ -42,7 +46,8 @@ WARNING: Deleting the self monitoring project removes any changes made to the project. If you create the project again, it's created in its default state. -1. Go to **Admin Area > Settings > Metrics and profiling** and expand the **Self monitoring** section. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, go to **Settings > Metrics and profiling** and expand **Self monitoring**. 1. Toggle the **Create Project** button off. 1. In the confirmation dialog that opens, click **Delete project**. It can take a few seconds for it to be deleted. diff --git a/doc/administration/monitoring/performance/gitlab_configuration.md b/doc/administration/monitoring/performance/gitlab_configuration.md index 6e7557854ad..e8abe2708c7 100644 --- a/doc/administration/monitoring/performance/gitlab_configuration.md +++ b/doc/administration/monitoring/performance/gitlab_configuration.md @@ -9,7 +9,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w GitLab Performance Monitoring is disabled by default. To enable it and change any of its settings: -1. Go to **Admin Area > Settings > Metrics and profiling** +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > Metrics and profiling** (`/admin/application_settings/metrics_and_profiling`). 1. Add the necessary configuration changes. 1. Restart all GitLab for the changes to take effect: diff --git a/doc/administration/monitoring/performance/grafana_configuration.md b/doc/administration/monitoring/performance/grafana_configuration.md index ac322f3e1ef..3b82b0e4bb8 100644 --- a/doc/administration/monitoring/performance/grafana_configuration.md +++ b/doc/administration/monitoring/performance/grafana_configuration.md @@ -62,8 +62,9 @@ repository. After setting up Grafana, you can enable a link to access it easily from the GitLab sidebar: -1. Go to the **Admin Area > Settings > Metrics and profiling**. -1. Expand **Metrics - Grafana**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > Metrics and profiling** + and expand **Metrics - Grafana**. 1. Check the **Enable access to Grafana** checkbox. 1. Configure the **Grafana URL**: - *If Grafana is enabled through Omnibus GitLab and on the same server,* @@ -71,7 +72,7 @@ GitLab sidebar: - *Otherwise,* enter the full URL of the Grafana instance. 1. Click **Save changes**. -GitLab displays your link in the **Admin Area > Monitoring > Metrics Dashboard**. +GitLab displays your link in the **Menu > Admin > Monitoring > Metrics Dashboard**. ## Security Update diff --git a/doc/administration/monitoring/performance/index.md b/doc/administration/monitoring/performance/index.md index 15c54a36f6c..f3db6ac9f03 100644 --- a/doc/administration/monitoring/performance/index.md +++ b/doc/administration/monitoring/performance/index.md @@ -69,6 +69,5 @@ The following environment variables are recognized: - `DATABASE_SAMPLER_INTERVAL_SECONDS` - `ACTION_CABLE_SAMPLER_INTERVAL_SECONDS` - `PUMA_SAMPLER_INTERVAL_SECONDS` -- `UNICORN_SAMPLER_INTERVAL_SECONDS` - `THREADS_SAMPLER_INTERVAL_SECONDS` - `GLOBAL_SEARCH_SAMPLER_INTERVAL_SECONDS` diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md index dd43c7d6fbb..1125547f13f 100644 --- a/doc/administration/monitoring/performance/performance_bar.md +++ b/doc/administration/monitoring/performance/performance_bar.md @@ -6,7 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Performance Bar **(FREE SELF)** -> The **Stats** field [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/271551) in GitLab SaaS 13.9. +> The **Stats** field [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/271551) in GitLab 13.9. +> The **Memory** field [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/330736) in GitLab 14.0. You can display the GitLab Performance Bar to see statistics for the performance of a page. When activated, it looks as follows: @@ -40,9 +41,11 @@ From left to right, it displays: Time until something was visible to the user. - [**DomContentLoaded**](https://developers.google.com/web/fundamentals/performance/critical-rendering-path/measure-crp) Event. - **Total number of requests** the page loaded. -- **Trace**: If Jaeger is integrated, **Trace** links to a Jaeger tracing page +- **Memory**: the amount of memory consumed and objects allocated during the selected request. + Select it to display a window with more details. +- **Trace**: if Jaeger is integrated, **Trace** links to a Jaeger tracing page with the current request's `correlation_id` included. -- **+**: A link to add a request's details to the performance bar. The request +- **+**: a link to add a request's details to the performance bar. The request can be added by its full URL (authenticated as the current user), or by the value of its `X-Request-Id` header. - **Download**: a link to download the raw JSON used to generate the Performance Bar reports. @@ -52,6 +55,11 @@ From left to right, it displays: - **Stats** (optional): if the `GITLAB_PERFORMANCE_BAR_STATS_URL` environment variable is set, this URL is displayed in the bar. In GitLab 13.9 and later, used only in GitLab SaaS. +NOTE: +Not all indicators are available in all environments. For instance, the memory view +requires to run Ruby with [specific patches](https://gitlab.com/gitlab-org/gitlab-build-images/-/blob/master/patches/ruby/2.7.2/thread-memory-allocations-2.7.patch) applied. +When running GitLab locally using the GDK this is typically not the case and the memory view cannot be used. + ## Request warnings Requests that exceed predefined limits display a warning **{warning}** icon and @@ -68,12 +76,13 @@ appears next to requests with warnings. ## Enable the Performance Bar via the Admin Area -The GitLab Performance Bar is disabled by default. To enable it for a given group: +The GitLab Performance Bar is disabled by default for non-administrators. To enable it +for a given group: 1. Sign in as a user with Administrator [permissions](../../../user/permissions.md). -1. In the menu bar, click **Admin Area**. -1. Go to **Settings > Metrics and profiling** - (`admin/application_settings/metrics_and_profiling`), and expand the section +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > Metrics and profiling** + (`admin/application_settings/metrics_and_profiling`), and expand **Profiling - Performance bar**. 1. Click **Enable access to the Performance Bar**. 1. In the **Allowed group** field, provide the full path of the group allowed diff --git a/doc/administration/monitoring/performance/request_profiling.md b/doc/administration/monitoring/performance/request_profiling.md index 15a58456e05..ebdca8d3960 100644 --- a/doc/administration/monitoring/performance/request_profiling.md +++ b/doc/administration/monitoring/performance/request_profiling.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w To profile a request: -1. Sign in to GitLab as a user with Administrator or Maintainer [permissions](../../../user/permissions.md). +1. Sign in to GitLab as an Administrator or a user with the [Maintainer role](../../../user/permissions.md). 1. In the navigation bar, click **Admin area**. 1. Go to **Monitoring > Requests Profiles**. 1. In the **Requests Profiles** section, copy the token. diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index f29db9ead38..7e72f6ed7df 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -9,7 +9,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w To enable the GitLab Prometheus metrics: 1. Log into GitLab as a user with [administrator permissions](../../../user/permissions.md). -1. Go to **Admin Area > Settings > Metrics and profiling**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > Metrics and profiling**. 1. Find the **Metrics - Prometheus** section, and click **Enable Prometheus Metrics**. 1. [Restart GitLab](../../restart_gitlab.md#omnibus-gitlab-restart) for the changes to take effect. @@ -58,7 +59,7 @@ The following metrics are available: | `gitlab_transaction_cache_duration_total` | Counter | 10.2 | Counter for total time (seconds) spent in Rails cache calls (aggregate) | | | `gitlab_transaction_cache_read_hit_count_total` | Counter | 10.2 | Counter for cache hits for Rails cache calls | `controller`, `action` | | `gitlab_transaction_cache_read_miss_count_total` | Counter | 10.2 | Counter for cache misses for Rails cache calls | `controller`, `action` | -| `gitlab_transaction_duration_seconds` | Histogram | 10.2 | Duration for all transactions (`gitlab_transaction_*` metrics) | `controller`, `action` | +| `gitlab_transaction_duration_seconds` | Histogram | 10.2 | Duration for successful requests (`gitlab_transaction_*` metrics) | `controller`, `action` | | `gitlab_transaction_event_build_found_total` | Counter | 9.4 | Counter for build found for API /jobs/request | | | `gitlab_transaction_event_build_invalid_total` | Counter | 9.4 | Counter for build invalid due to concurrency conflict for API /jobs/request | | | `gitlab_transaction_event_build_not_found_cached_total` | Counter | 9.4 | Counter for cached response of build not found for API /jobs/request | | @@ -91,7 +92,7 @@ The following metrics are available: | `gitlab_transaction_view_duration_total` | Counter | 9.4 | Duration for views | `controller`, `action`, `view` | | `gitlab_view_rendering_duration_seconds` | Histogram | 10.2 | Duration for views (histogram) | `controller`, `action`, `view` | | `http_requests_total` | Counter | 9.4 | Rack request count | `method`, `status` | -| `http_request_duration_seconds` | Histogram | 9.4 | HTTP response time from rack middleware | `method` | +| `http_request_duration_seconds` | Histogram | 9.4 | HTTP response time from rack middleware for successful requests | `method` | | `gitlab_transaction_db_count_total` | Counter | 13.1 | Counter for total number of SQL calls | `controller`, `action` | | `gitlab_transaction_db_<role>_count_total` | Counter | 13.10 | Counter for total number of SQL calls, grouped by database roles (primary/replica) | `controller`, `action` | | `gitlab_transaction_db_write_count_total` | Counter | 13.1 | Counter for total number of write SQL calls | `controller`, `action` | @@ -130,6 +131,8 @@ The following metrics are available: | `gitlab_ci_pipeline_security_orchestration_policy_processing_duration_seconds` | Histogram | 13.12 | Time in seconds it takes to process Security Policies in CI/CD pipeline | | | `gitlab_ci_difference_live_vs_actual_minutes` | Histogram | 13.12 | Difference between CI minute consumption counted while jobs were running (live) vs when jobs are complete (actual). Used to enforce CI minute consumption limits on long running jobs. | `plan` | | `gitlab_spamcheck_request_duration_seconds` | Histogram | 13.12 | The duration for requests between Rails and the anti-spam engine | | +| `service_desk_thank_you_email` | Counter | 14.0 | Total number of email responses to new service desk emails | | +| `service_desk_new_note_email` | Counter | 14.0 | Total number of email notifications on new service desk comment | | ## Metrics controlled by a feature flag @@ -227,11 +230,15 @@ configuration option in `gitlab.yml`. These metrics are served from the | `global_search_bulk_cron_queue_size` | Gauge | 12.10 | Number of database records waiting to be synchronized to Elasticsearch | | | `global_search_awaiting_indexing_queue_size` | Gauge | 13.2 | Number of database updates waiting to be synchronized to Elasticsearch while indexing is paused | | | `geo_merge_request_diffs` | Gauge | 13.4 | Number of merge request diffs on primary | `url` | -| `geo_merge_request_diffs_checksummed` | Gauge | 13.4 | Number of merge request diffs checksummed on primary | `url` | +| `geo_merge_request_diffs_checksum_total` | Gauge | 13.12 | Number of merge request diffs tried to checksum on primary | `url` | +| `geo_merge_request_diffs_checksummed` | Gauge | 13.4 | Number of merge request diffs successfully checksummed on primary | `url` | | `geo_merge_request_diffs_checksum_failed` | Gauge | 13.4 | Number of merge request diffs failed to calculate the checksum on primary | `url` | | `geo_merge_request_diffs_synced` | Gauge | 13.4 | Number of syncable merge request diffs synced on secondary | `url` | | `geo_merge_request_diffs_failed` | Gauge | 13.4 | Number of syncable merge request diffs failed to sync on secondary | `url` | | `geo_merge_request_diffs_registry` | Gauge | 13.4 | Number of merge request diffs in the registry | `url` | +| `geo_merge_request_diffs_verification_total` | Gauge | 13.12 | Number of merge request diffs verifications tried on secondary | `url` | +| `geo_merge_request_diffs_verified` | Gauge | 13.12 | Number of merge request diffs verified on secondary | `url` | +| `geo_merge_request_diffs_verification_failed` | Gauge | 13.12 | Number of merge request diffs verifications failed on secondary | `url` | | `geo_snippet_repositories` | Gauge | 13.4 | Number of snippets on primary | `url` | | `geo_snippet_repositories_checksummed` | Gauge | 13.4 | Number of snippets checksummed on primary | `url` | | `geo_snippet_repositories_checksum_failed` | Gauge | 13.4 | Number of snippets failed to calculate the checksum on primary | `url` | @@ -308,20 +315,8 @@ Some basic Ruby runtime metrics are available: | `ruby_process_proportional_memory_bytes` | Gauge | 13.0 | Memory usage by process (PSS/Proportional Set Size) | | `ruby_process_start_time_seconds` | Gauge | 12.0 | UNIX timestamp of process start time | -## Unicorn Metrics - -Unicorn specific metrics, when Unicorn is used. - -| Metric | Type | Since | Description | -|:-----------------------------|:------|:------|:---------------------------------------------------| -| `unicorn_active_connections` | Gauge | 11.0 | The number of active Unicorn connections (workers) | -| `unicorn_queued_connections` | Gauge | 11.0 | The number of queued Unicorn connections | -| `unicorn_workers` | Gauge | 12.0 | The number of Unicorn workers | - ## Puma Metrics -When Puma is used instead of Unicorn, the following metrics are available: - | Metric | Type | Since | Description | |:--------------------------------- |:------- |:----- |:----------- | | `puma_workers` | Gauge | 12.0 | Total number of workers | @@ -352,8 +347,8 @@ instance (`cache`, `shared_state` etc.). ## Metrics shared directory The GitLab Prometheus client requires a directory to store metrics data shared between multi-process services. -Those files are shared among all instances running under Unicorn server. -The directory must be accessible to all running Unicorn's processes, or +Those files are shared among all instances running under Puma server. +The directory must be accessible to all running Puma's processes, or metrics can't function correctly. This directory's location is configured using environment variable `prometheus_multiproc_dir`. diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index 035c5b3ee7e..dd402f800e3 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -122,44 +122,28 @@ The steps below are the minimum necessary to configure a Monitoring node running 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby + roles ['monitoring_role'] + external_url 'http://gitlab.example.com' - # Enable Prometheus - prometheus['enable'] = true + # Prometheus prometheus['listen_address'] = '0.0.0.0:9090' prometheus['monitor_kubernetes'] = false - # Enable Login form - grafana['disable_login_form'] = false - - # Enable Grafana + # Grafana grafana['enable'] = true grafana['admin_password'] = 'toomanysecrets' + grafana['disable_login_form'] = false # Enable service discovery for Prometheus consul['enable'] = true consul['monitoring_service_discovery'] = true - - # The addresses can be IPs or FQDNs - consul['configuration'] = { - retry_join: %w(10.0.0.1 10.0.0.2 10.0.0.3), + consul['configuration'] = { + retry_join: %w(10.0.0.1 10.0.0.2 10.0.0.3), # The addresses can be IPs or FQDNs } - # Disable all other services - gitlab_rails['auto_migrate'] = false - alertmanager['enable'] = false - gitaly['enable'] = false - gitlab_exporter['enable'] = false - gitlab_workhorse['enable'] = false + # Nginx - For Grafana access nginx['enable'] = true - postgres_exporter['enable'] = false - postgresql['enable'] = false - redis['enable'] = false - redis_exporter['enable'] = false - sidekiq['enable'] = false - puma['enable'] = false - node_exporter['enable'] = false - gitlab_exporter['enable'] = false ``` 1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. @@ -227,7 +211,7 @@ To use an external Prometheus server: gitlab_rails['monitoring_whitelist'] = ['127.0.0.0/8', '192.168.0.1'] ``` -1. On **all** GitLab Rails(Puma/Unicorn, Sidekiq) servers, set the Prometheus server IP address and listen port. For example: +1. On **all** GitLab Rails(Puma, Sidekiq) servers, set the Prometheus server IP address and listen port. For example: ```ruby gitlab_rails['prometheus_address'] = '192.168.0.1:9090' diff --git a/doc/administration/nfs.md b/doc/administration/nfs.md index c49a2c20ed2..e53f2af3440 100644 --- a/doc/administration/nfs.md +++ b/doc/administration/nfs.md @@ -5,11 +5,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Using NFS with GitLab +# Using NFS with GitLab **(FREE SELF)** NFS can be used as an alternative for object storage but this isn't typically -recommended for performance reasons. Note however it is required for [GitLab -Pages](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/196). +recommended for performance reasons. For data objects such as LFS, Uploads, Artifacts, etc., an [Object Storage service](object_storage.md) is recommended over NFS where possible, due to better performance. @@ -17,7 +16,7 @@ is recommended over NFS where possible, due to better performance. File system performance can impact overall GitLab performance, especially for actions that read or write to Git repositories. For steps you can use to test file system performance, see -[File system Performance Benchmarking](operations/filesystem_benchmarking.md). +[File System Performance Benchmarking](operations/filesystem_benchmarking.md). ## Gitaly and NFS deprecation @@ -445,11 +444,11 @@ In case of NFS-related problems, it can be helpful to trace the file system requests that are being made by using `perf`: ```shell -sudo perf trace -e 'nfs4:*' -p $(pgrep -fd ',' puma && pgrep -fd ',' unicorn) +sudo perf trace -e 'nfs4:*' -p $(pgrep -fd ',' puma) ``` On Ubuntu 16.04, use: ```shell -sudo perf trace --no-syscalls --event 'nfs4:*' -p $(pgrep -fd ',' puma && pgrep -fd ',' unicorn) +sudo perf trace --no-syscalls --event 'nfs4:*' -p $(pgrep -fd ',' puma) ``` diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md index d133e8c3721..f1025bd1846 100644 --- a/doc/administration/object_storage.md +++ b/doc/administration/object_storage.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Object Storage +# Object storage **(FREE SELF)** GitLab supports using an object storage service for holding numerous types of data. It's recommended over NFS and @@ -29,7 +29,7 @@ GitLab has been tested on a number of object storage providers: - Dell EMC ECS: Prior to GitLab 13.3, there is a [known bug in GitLab Workhorse that prevents HTTP Range Requests from working with CI job artifacts](https://gitlab.com/gitlab-org/gitlab/-/issues/223806). - Be sure to upgrade to GitLab v13.3.0 or above if you use S3 storage with this hardware. + Be sure to upgrade to GitLab 13.3.0 or above if you use S3 storage with this hardware. - Ceph S3 prior to [Kraken 11.0.2](https://ceph.com/releases/kraken-11-0-2-released/) does not support the [Upload Copy Part API](https://gitlab.com/gitlab-org/gitlab/-/issues/300604). You may need to [disable multi-threaded copying](#multi-threaded-copying). @@ -47,7 +47,7 @@ For more information on the differences and to transition from one form to anoth ### Consolidated object storage configuration -> Introduced in [GitLab 13.2](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4368). +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4368) in GitLab 13.2. Using the consolidated object storage configuration has a number of advantages: @@ -209,13 +209,13 @@ gitlab_rails['object_store']['connection'] = { } ``` -| Setting | Description | -|---------|-------------| -| `enabled` | Enable/disable object storage | -| `proxy_download` | Set to `true` to [enable proxying all files served](#proxy-download). Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | -| `connection` | Various [connection options](#connection-settings) described below | -| `storage_options` | Options to use when saving new objects, such as [server side encryption](#server-side-encryption-headers). Introduced in GitLab 13.3 | -| `objects` | [Object-specific configuration](#object-specific-configuration) +| Setting | Description | +|-------------------|-----------------------------------| +| `enabled` | Enable or disable object storage. | +| `proxy_download` | Set to `true` to [enable proxying all files served](#proxy-download). Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data. | +| `connection` | Various [connection options](#connection-settings) described below. | +| `storage_options` | Options to use when saving new objects, such as [server side encryption](#server-side-encryption-headers). Introduced in GitLab 13.3. | +| `objects` | [Object-specific configuration](#object-specific-configuration). | ### Connection settings @@ -226,27 +226,27 @@ in the `connection` setting. The connection settings match those provided by [fog-aws](https://github.com/fog/fog-aws): -| Setting | Description | Default | -|---------|-------------|---------| -| `provider` | Always `AWS` for compatible hosts | `AWS` | -| `aws_access_key_id` | AWS credentials, or compatible | | -| `aws_secret_access_key` | AWS credentials, or compatible | | -| `aws_signature_version` | AWS signature version to use. `2` or `4` are valid options. Digital Ocean Spaces and other providers may need `2`. | `4` | -| `enable_signature_v4_streaming` | Set to `true` to enable HTTP chunked transfers with [AWS v4 signatures](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be `false`. | `true` | -| `region` | AWS region. | | -| `host` | S3 compatible host for when not using AWS, e.g. `localhost` or `storage.example.com`. HTTPS and port 443 is assumed. | `s3.amazonaws.com` | -| `endpoint` | Can be used when configuring an S3 compatible service such as [MinIO](https://min.io), by entering a URL such as `http://127.0.0.1:9000`. This takes precedence over `host`. | (optional) | -| `path_style` | Set to `true` to use `host/bucket_name/object` style paths instead of `bucket_name.host/object`. Leave as `false` for AWS S3. | `false` | -| `use_iam_profile` | Set to `true` to use IAM profile instead of access keys | `false` +| Setting | Description | Default | +|---------------------------------|------------------------------------|---------| +| `provider` | Always `AWS` for compatible hosts. | `AWS` | +| `aws_access_key_id` | AWS credentials, or compatible. | | +| `aws_secret_access_key` | AWS credentials, or compatible. | | +| `aws_signature_version` | AWS signature version to use. `2` or `4` are valid options. Digital Ocean Spaces and other providers may need `2`. | `4` | +| `enable_signature_v4_streaming` | Set to `true` to enable HTTP chunked transfers with [AWS v4 signatures](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be `false`. | `true` | +| `region` | AWS region. | | +| `host` | S3 compatible host for when not using AWS. For example, `localhost` or `storage.example.com`. HTTPS and port 443 is assumed. | `s3.amazonaws.com` | +| `endpoint` | Can be used when configuring an S3 compatible service such as [MinIO](https://min.io), by entering a URL such as `http://127.0.0.1:9000`. This takes precedence over `host`. | (optional) | +| `path_style` | Set to `true` to use `host/bucket_name/object` style paths instead of `bucket_name.host/object`. Leave as `false` for AWS S3. | `false`. | +| `use_iam_profile` | Set to `true` to use IAM profile instead of access keys. | `false` | #### Oracle Cloud S3 connection settings Note that Oracle Cloud S3 must be sure to use the following settings: -| Setting | Value | -|---------|-------| +| Setting | Value | +|---------------------------------|---------| | `enable_signature_v4_streaming` | `false` | -| `path_style` | `true` | +| `path_style` | `true` | If `enable_signature_v4_streaming` is set to `true`, you may see the following error in `production.log`: @@ -259,13 +259,13 @@ STREAMING-AWS4-HMAC-SHA256-PAYLOAD is not supported Here are the valid connection parameters for GCS: -| Setting | Description | example | -|---------|-------------|---------| -| `provider` | The provider name | `Google` | -| `google_project` | GCP project name | `gcp-project-12345` | -| `google_client_email` | The email address of the service account | `foo@gcp-project-12345.iam.gserviceaccount.com` | -| `google_json_key_location` | The JSON key path | `/path/to/gcp-project-12345-abcde.json` | -| `google_application_default` | Set to `true` to use [Google Cloud Application Default Credentials](https://cloud.google.com/docs/authentication/production#automatically) to locate service account credentials. | +| Setting | Description | Example | +|------------------------------|-------------------|---------| +| `provider` | Provider name. | `Google` | +| `google_project` | GCP project name. | `gcp-project-12345` | +| `google_client_email` | Email address of the service account. | `foo@gcp-project-12345.iam.gserviceaccount.com` | +| `google_json_key_location` | JSON key path. | `/path/to/gcp-project-12345-abcde.json` | +| `google_application_default` | Set to `true` to use [Google Cloud Application Default Credentials](https://cloud.google.com/docs/authentication/production#automatically) to locate service account credentials. | | The service account must have permission to access the bucket. Learn more in Google's @@ -328,12 +328,12 @@ The following are the valid connection parameters for Azure. Read the [Azure Blob storage documentation](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) to learn more. -| Setting | Description | Example | -|---------|-------------|---------| -| `provider` | Provider name | `AzureRM` | -| `azure_storage_account_name` | Name of the Azure Blob Storage account used to access the storage | `azuretest` | -| `azure_storage_access_key` | Storage account access key used to access the container. This is typically a secret, 512-bit encryption key encoded in base64. | `czV2OHkvQj9FKEgrTWJRZVRoV21ZcTN0Nnc5eiRDJkYpSkBOY1JmVWpYbjJy\nNHU3eCFBJUQqRy1LYVBkU2dWaw==\n` | -| `azure_storage_domain` | Domain name used to contact the Azure Blob Storage API (optional). Defaults to `blob.core.windows.net`. Set this if you are using Azure China, Azure Germany, Azure US Government, or some other custom Azure domain. | `blob.core.windows.net` | +| Setting | Description | Example | +|------------------------------|----------------|-----------| +| `provider` | Provider name. | `AzureRM` | +| `azure_storage_account_name` | Name of the Azure Blob Storage account used to access the storage. | `azuretest` | +| `azure_storage_access_key` | Storage account access key used to access the container. This is typically a secret, 512-bit encryption key encoded in base64. | `czV2OHkvQj9FKEgrTWJRZVRoV21ZcTN0Nnc5eiRDJkYpSkBOY1JmVWpYbjJy\nNHU3eCFBJUQqRy1LYVBkU2dWaw==\n` | +| `azure_storage_domain` | Domain name used to contact the Azure Blob Storage API (optional). Defaults to `blob.core.windows.net`. Set this if you are using Azure China, Azure Germany, Azure US Government, or some other custom Azure domain. | `blob.core.windows.net` | ##### Azure example (consolidated form) @@ -382,15 +382,15 @@ consolidated form, see the [S3 settings](#s3-compatible-connection-settings). Here are the valid connection settings for the Swift API, provided by [fog-openstack](https://github.com/fog/fog-openstack): -| Setting | Description | Default | -|---------|-------------|---------| -| `provider` | Always `OpenStack` for compatible hosts | `OpenStack` | -| `openstack_username` | OpenStack username | | -| `openstack_api_key` | OpenStack API key | | +| Setting | Description | Default | +|--------------------------|----------------------|---------| +| `provider` | Always `OpenStack` for compatible hosts. | `OpenStack` | +| `openstack_username` | OpenStack username. | | +| `openstack_api_key` | OpenStack API key. | | | `openstack_temp_url_key` | OpenStack key for generating temporary URLs | | -| `openstack_auth_url` | OpenStack authentication endpoint | | -| `openstack_region` | OpenStack region | | -| `openstack_tenant` | OpenStack tenant ID | +| `openstack_auth_url` | OpenStack authentication endpoint | | +| `openstack_region` | OpenStack region. | | +| `openstack_tenant` | OpenStack tenant ID. | | #### Rackspace Cloud Files @@ -400,13 +400,13 @@ Rackspace Cloud, provided by [fog-rackspace](https://github.com/fog/fog-rackspac This isn't compatible with the consolidated object storage form. Rackspace Cloud is supported only with the storage-specific form. -| Setting | Description | example | -|---------|-------------|---------| -| `provider` | The provider name | `Rackspace` | -| `rackspace_username` | The username of the Rackspace account with access to the container | `joe.smith` | -| `rackspace_api_key` | The API key of the Rackspace account with access to the container | `ABC123DEF456ABC123DEF456ABC123DE` | -| `rackspace_region` | The Rackspace storage region to use, a three letter code from the [list of service access endpoints](https://docs.rackspace.com/docs/cloud-files/v1/general-api-info/service-access/) | `iad` | -| `rackspace_temp_url_key` | The private key you have set in the Rackspace API for [temporary URLs](https://docs.rackspace.com/docs/cloud-files/v1/use-cases/public-access-to-your-cloud-files-account/#tempurl). | `ABC123DEF456ABC123DEF456ABC123DE` | +| Setting | Description | Example | +|--------------------------|----------------|-------------| +| `provider` | Provider name. | `Rackspace` | +| `rackspace_username` | Username of the Rackspace account with access to the container. | `joe.smith` | +| `rackspace_api_key` | API key of the Rackspace account with access to the container. | `ABC123DEF456ABC123DEF456ABC123DE` | +| `rackspace_region` | Rackspace storage region to use, a three letter code from the [list of service access endpoints](https://docs.rackspace.com/docs/cloud-files/v1/general-api-info/service-access/). | `iad` | +| `rackspace_temp_url_key` | Private key you set in the Rackspace API for [temporary URLs](https://docs.rackspace.com/docs/cloud-files/v1/use-cases/public-access-to-your-cloud-files-account/#tempurl). | `ABC123DEF456ABC123DEF456ABC123DE` | Regardless of whether the container has public access enabled or disabled, Fog uses the TempURL method to grant access to LFS objects. If you see error @@ -465,24 +465,24 @@ gitlab_rails['object_store']['objects']['pages']['bucket'] = 'pages' This is the list of valid `objects` that can be used: -| Type | Description | -|--------------------|---------------| -| `artifacts` | [CI artifacts](job_artifacts.md) | -| `external_diffs` | [Merge request diffs](merge_request_diffs.md) | -| `uploads` | [User uploads](uploads.md) | -| `lfs` | [Git Large File Storage objects](lfs/index.md) | -| `packages` | [Project packages (e.g. PyPI, Maven, NuGet, etc.)](packages/index.md) | -| `dependency_proxy` | [GitLab Dependency Proxy](packages/dependency_proxy.md) | -| `terraform_state` | [Terraform state files](terraform_state.md) | -| `pages` | [GitLab Pages](pages/index.md) | +| Type | Description | +|--------------------|----------------------------------------------------------------------------| +| `artifacts` | [CI artifacts](job_artifacts.md) | +| `external_diffs` | [Merge request diffs](merge_request_diffs.md) | +| `uploads` | [User uploads](uploads.md) | +| `lfs` | [Git Large File Storage objects](lfs/index.md) | +| `packages` | [Project packages (for example, PyPI, Maven, or NuGet)](packages/index.md) | +| `dependency_proxy` | [GitLab Dependency Proxy](packages/dependency_proxy.md) | +| `terraform_state` | [Terraform state files](terraform_state.md) | +| `pages` | [GitLab Pages](pages/index.md) | Within each object type, three parameters can be defined: -| Setting | Required? | Description | -|------------------|-----------|-------------| -| `bucket` | Yes | The bucket name for the object storage. | -| `enabled` | No | Overrides the common parameter | -| `proxy_download` | No | Overrides the common parameter | +| Setting | Required? | Description | +|------------------|------------------------|-------------------------------------| +| `bucket` | **{check-circle}** Yes | Bucket name for the object storage. | +| `enabled` | **{dotted-circle}** No | Overrides the common parameter. | +| `proxy_download` | **{dotted-circle}** No | Overrides the common parameter. | #### Selectively disabling object storage @@ -542,21 +542,21 @@ original configuration (for example, `artifacts_object_store_enabled`, or For configuring object storage in GitLab 13.1 and earlier, or for storage types not supported by consolidated configuration form, refer to the following guides: -|Object storage type|Supported by consolidated configuration?| -|-------------------|----------------------------------------| -| [Backups](../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage) | No | -| [Job artifacts](job_artifacts.md#using-object-storage) including archived job logs | Yes | -| [LFS objects](lfs/index.md#storing-lfs-objects-in-remote-object-storage) | Yes | -| [Uploads](uploads.md#using-object-storage) | Yes | -| [Container Registry](packages/container_registry.md#use-object-storage) (optional feature) | No | -| [Merge request diffs](merge_request_diffs.md#using-object-storage) | Yes | -| [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | -| [Packages](packages/index.md#using-object-storage) (optional feature) | Yes | -| [Dependency Proxy](packages/dependency_proxy.md#using-object-storage) (optional feature) **(PREMIUM SELF)** | Yes | -| [Pseudonymizer](pseudonymizer.md#configuration) (optional feature) **(ULTIMATE SELF)** | No | -| [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | -| [Terraform state files](terraform_state.md#using-object-storage) | Yes | -| [GitLab Pages content](pages/index.md#using-object-storage) | Yes | +| Object storage type | Supported by consolidated configuration? | +|---------------------|------------------------------------------| +| [Backups](../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage) | **{dotted-circle}** No | +| [Job artifacts](job_artifacts.md#using-object-storage) including archived job logs | **{check-circle}** Yes | +| [LFS objects](lfs/index.md#storing-lfs-objects-in-remote-object-storage) | **{check-circle}** Yes | +| [Uploads](uploads.md#using-object-storage) | **{check-circle}** Yes | +| [Container Registry](packages/container_registry.md#use-object-storage) (optional feature) | **{dotted-circle}** No | +| [Merge request diffs](merge_request_diffs.md#using-object-storage) | **{check-circle}** Yes | +| [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| **{dotted-circle}** No | +| [Packages](packages/index.md#using-object-storage) (optional feature) | **{check-circle}** Yes | +| [Dependency Proxy](packages/dependency_proxy.md#using-object-storage) (optional feature) **(PREMIUM SELF)** | **{check-circle}** Yes | +| [Pseudonymizer](pseudonymizer.md#configuration) (optional feature) **(ULTIMATE SELF)** | **{dotted-circle}** No | +| [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | **{dotted-circle}** No | +| [Terraform state files](terraform_state.md#using-object-storage) | **{check-circle}** Yes | +| [GitLab Pages content](pages/index.md#using-object-storage) | **{check-circle}** Yes | ### Other alternatives to file system storage @@ -618,20 +618,20 @@ This can result in some of the following problems: - If GitLab is using non-secure HTTP to access the object storage, clients may generate `https->http` downgrade errors and refuse to process the redirect. The solution to this -is for GitLab to use HTTPS. LFS, for example, will generate this error: +is for GitLab to use HTTPS. LFS, for example, generates this error: ```plaintext LFS: lfsapi/client: refusing insecure redirect, https->http ``` -- Clients will need to trust the certificate authority that issued the object storage +- Clients need to trust the certificate authority that issued the object storage certificate, or may return common TLS errors such as: ```plaintext x509: certificate signed by unknown authority ``` -- Clients will need network access to the object storage. +- Clients need network access to the object storage. Network firewalls could block access. Errors that might result if this access is not in place include: @@ -667,7 +667,7 @@ The first option is recommended for MinIO. Otherwise, the is to use the `--compat` parameter on the server. Without consolidated object store configuration or instance profiles enabled, -GitLab Workhorse will upload files to S3 using pre-signed URLs that do +GitLab Workhorse uploads files to S3 using pre-signed URLs that do not have a `Content-MD5` HTTP header computed for them. To ensure data is not corrupted, Workhorse checks that the MD5 hash of the data sent equals the ETag header returned from the S3 server. When encryption is @@ -683,7 +683,7 @@ eliminates the need to compare ETag headers returned from the S3 server. Instead of supplying AWS access and secret keys in object storage configuration, GitLab can be configured to use IAM roles to set up an [Amazon instance profile](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). -When this is used, GitLab will fetch temporary credentials each time an +When this is used, GitLab fetches temporary credentials each time an S3 bucket is accessed, so no hard-coded values are needed in the configuration. @@ -709,10 +709,10 @@ only encrypted objects are uploaded](https://aws.amazon.com/premiumsupport/knowl To do this, you must configure GitLab to send the proper encryption headers in the `storage_options` configuration section: -| Setting | Description | -|-------------------------------------|-------------| -| `server_side_encryption` | Encryption mode (`AES256` or `aws:kms`) | -| `server_side_encryption_kms_key_id` | Amazon Resource Name. Only needed when `aws:kms` is used in `server_side_encryption`. See the [Amazon documentation on using KMS encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html) | +| Setting | Description | +|-------------------------------------|------------------------------------------| +| `server_side_encryption` | Encryption mode (`AES256` or `aws:kms`). | +| `server_side_encryption_kms_key_id` | Amazon Resource Name. Only needed when `aws:kms` is used in `server_side_encryption`. See the [Amazon documentation on using KMS encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html). | As with the case for default encryption, these options only work when the Workhorse S3 client is enabled. One of the following two conditions @@ -721,7 +721,7 @@ must be fulfilled: - `use_iam_profile` is `true` in the connection settings. - Consolidated object storage settings are in use. -[ETag mismatch errors](#etag-mismatch) will occur if server side +[ETag mismatch errors](#etag-mismatch) occur if server side encryption headers are used without enabling the Workhorse S3 client. ##### Disabling the feature diff --git a/doc/administration/operations/cleaning_up_redis_sessions.md b/doc/administration/operations/cleaning_up_redis_sessions.md index 6513a4ed4c8..194dd8f39e2 100644 --- a/doc/administration/operations/cleaning_up_redis_sessions.md +++ b/doc/administration/operations/cleaning_up_redis_sessions.md @@ -25,7 +25,7 @@ NOTE: The instructions below must be modified in accordance with your configuration settings if you have used the advanced Redis settings outlined in -[Configuration Files Documentation](https://gitlab.com/gitlab-org/gitlab/blob/master/config/README.md). +[Configuration Files Documentation](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/README.md). First we define a shell function with the proper Redis connection details. diff --git a/doc/administration/operations/extra_sidekiq_processes.md b/doc/administration/operations/extra_sidekiq_processes.md index 6b8d6f3bf1e..ed89d11da75 100644 --- a/doc/administration/operations/extra_sidekiq_processes.md +++ b/doc/administration/operations/extra_sidekiq_processes.md @@ -18,8 +18,8 @@ The information in this page applies only to Omnibus GitLab. For a list of the existing Sidekiq queues, check the following files: -- [Queues for both GitLab Community and Enterprise Editions](https://gitlab.com/gitlab-org/gitlab/blob/master/app/workers/all_queues.yml) -- [Queues for GitLab Enterprise Editions only](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/app/workers/all_queues.yml) +- [Queues for both GitLab Community and Enterprise Editions](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/all_queues.yml) +- [Queues for GitLab Enterprise Editions only](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/all_queues.yml) Each entry in the above files represents a queue on which Sidekiq processes can be started. @@ -38,11 +38,11 @@ To start multiple processes: process, and values in each item determine the queues it works on. For example, the following setting creates three Sidekiq processes, one to run on - `elastic_indexer`, one to run on `mailers`, and one process running on all queues: + `elastic_commit_indexer`, one to run on `mailers`, and one process running on all queues: ```ruby sidekiq['queue_groups'] = [ - "elastic_indexer", + "elastic_commit_indexer", "mailers", "*" ] @@ -53,7 +53,7 @@ To start multiple processes: ```ruby sidekiq['queue_groups'] = [ - "elastic_indexer, elastic_commit_indexer", + "elastic_commit_indexer, elastic_association_indexer", "mailers", "*" ] @@ -70,11 +70,11 @@ To start multiple processes: ] ``` - `*` cannot be combined with concrete queue names - `*, mailers` will - just handle the `mailers` queue. + `*` cannot be combined with concrete queue names - `*, mailers` + just handles the `mailers` queue. When `sidekiq-cluster` is only running on a single node, make sure that at least - one process is running on all queues using `*`. This means a process will + one process is running on all queues using `*`. This means a process is This includes queues that have dedicated processes. If `sidekiq-cluster` is running on more than one node, you can also use @@ -116,83 +116,10 @@ you list: > - [Sidekiq cluster, including queue selector, moved](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/181) to GitLab Free in 12.10. > - [Renamed from `experimental_queue_selector` to `queue_selector`](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/147) in GitLab 13.6. -In addition to selecting queues by name, as above, the `queue_selector` -option allows queue groups to be selected in a more general way using -the following components: - -- Attributes that can be selected. -- Operators used to construct a query. - -When `queue_selector` is set, all `queue_groups` must be in the queue -selector syntax. - -### Available attributes - -- [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/261) in GitLab 13.1, `tags`. - -From the [list of all available -attributes](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/all_queues.yml), -`queue_selector` allows selecting of queues by the following attributes: - -- `feature_category` - the [GitLab feature - category](https://about.gitlab.com/direction/maturity/#category-maturity) the - queue belongs to. For example, the `merge` queue belongs to the - `source_code_management` category. -- `has_external_dependencies` - whether or not the queue connects to external - services. For example, all importers have this set to `true`. -- `urgency` - how important it is that this queue's jobs run - quickly. Can be `high`, `low`, or `throttled`. For example, the - `authorized_projects` queue is used to refresh user permissions, and - is high urgency. -- `worker_name` - the worker name. The other attributes are typically more useful as - they are more general, but this is available in case a particular worker needs - to be selected. -- `name` - the queue name. Similiarly, this is available in case a particular queue needs - to be selected. -- `resource_boundary` - if the queue is bound by `cpu`, `memory`, or - `unknown`. For example, the `project_export` queue is memory bound as it has - to load data in memory before saving it for export. -- `tags` - short-lived annotations for queues. These are expected to frequently - change from release to release, and may be removed entirely. - -`has_external_dependencies` is a boolean attribute: only the exact -string `true` is considered true, and everything else is considered -false. - -`tags` is a set, which means that `=` checks for intersecting sets, and -`!=` checks for disjoint sets. For example, `tags=a,b` selects queues -that have tags `a`, `b`, or both. `tags!=a,b` selects queues that have -neither of those tags. - -### Available operators - -`queue_selector` supports the following operators, listed from highest -to lowest precedence: - -- `|` - the logical OR operator. For example, `query_a|query_b` (where `query_a` - and `query_b` are queries made up of the other operators here) will include - queues that match either query. -- `&` - the logical AND operator. For example, `query_a&query_b` (where - `query_a` and `query_b` are queries made up of the other operators here) will - only include queues that match both queries. -- `!=` - the NOT IN operator. For example, `feature_category!=issue_tracking` - excludes all queues from the `issue_tracking` feature category. -- `=` - the IN operator. For example, `resource_boundary=cpu` includes all - queues that are CPU bound. -- `,` - the concatenate set operator. For example, - `feature_category=continuous_integration,pages` includes all queues from - either the `continuous_integration` category or the `pages` category. This - example is also possible using the OR operator, but allows greater brevity, as - well as being lower precedence. - -The operator precedence for this syntax is fixed: it's not possible to make AND -have higher precedence than OR. - -[In GitLab 12.9](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26594) and -later, as with the standard queue group syntax above, a single `*` as the -entire queue group selects all queues. - -### Example queries +In addition to selecting queues by name, as above, the `queue_selector` option +allows queue groups to be selected in a more general way using a [worker matching +query](extra_sidekiq_routing.md#worker-matching-query). After `queue_selector` +is set, all `queue_groups` must follow the aforementioned syntax. In `/etc/gitlab/gitlab.rb`: @@ -215,7 +142,7 @@ WARNING: Sidekiq cluster is [scheduled](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/240) to be the only way to start Sidekiq in GitLab 14.0. -By default, the Sidekiq service will run `sidekiq-cluster`. To disable this behavior, +By default, the Sidekiq service runs `sidekiq-cluster`. To disable this behavior, add the following to the Sidekiq configuration: ```ruby @@ -224,7 +151,7 @@ sidekiq['cluster'] = false ``` All of the aforementioned configuration options for `sidekiq` -are available. By default, they will be configured as follows: +are available. By default, they are configured as follows: ```ruby sidekiq['queue_selector'] = false @@ -241,14 +168,14 @@ cluster as above. When disabling `sidekiq_cluster`, you must copy your configuration for `sidekiq_cluster`over to `sidekiq`. Anything configured for -`sidekiq_cluster` will be overridden by the options for `sidekiq` when +`sidekiq_cluster` is overridden by the options for `sidekiq` when setting `sidekiq['cluster'] = true`. -When using this feature, the service called `sidekiq` will now be +When using this feature, the service called `sidekiq` is now running `sidekiq-cluster`. The [concurrency](#manage-concurrency) and other options configured -for Sidekiq will be respected. +for Sidekiq are respected. By default, logs for `sidekiq-cluster` go to `/var/log/gitlab/sidekiq` like regular Sidekiq logs. @@ -293,7 +220,7 @@ use all of its resources to perform those operations. To set up a separate Each process defined under `sidekiq` starts with a number of threads that equals the number of queues, plus one spare thread. For example, a process that handles the `process_commit` and `post_receive` -queues will use three threads in total. +queues uses three threads in total. ## Manage concurrency @@ -324,16 +251,16 @@ Running Sidekiq cluster is the default in GitLab 13.0 and later. ``` `min_concurrency` and `max_concurrency` are independent; one can be set without -the other. Setting `min_concurrency` to `0` will disable the limit. +the other. Setting `min_concurrency` to `0` disables the limit. For each queue group, let `N` be one more than the number of queues. The -concurrency factor will be set to: +concurrency factor are set to: 1. `N`, if it's between `min_concurrency` and `max_concurrency`. 1. `max_concurrency`, if `N` exceeds this value. 1. `min_concurrency`, if `N` is less than this value. -If `min_concurrency` is equal to `max_concurrency`, then this value will be used +If `min_concurrency` is equal to `max_concurrency`, then this value is used regardless of the number of queues. When `min_concurrency` is greater than `max_concurrency`, it is treated as @@ -360,7 +287,7 @@ Running Sidekiq directly is scheduled to be removed in GitLab sudo gitlab-ctl reconfigure ``` -This will set the concurrency (number of threads) for the Sidekiq process. +This sets the concurrency (number of threads) for the Sidekiq process. ## Modify the check interval @@ -426,21 +353,21 @@ you'd use the following: ### Monitor the `sidekiq-cluster` command -The `sidekiq-cluster` command will not terminate once it has started the desired -amount of Sidekiq processes. Instead, the process will continue running and +The `sidekiq-cluster` command does not terminate once it has started the desired +amount of Sidekiq processes. Instead, the process continues running and forward any signals to the child processes. This makes it easy to stop all Sidekiq processes as you simply send a signal to the `sidekiq-cluster` process, instead of having to send it to the individual processes. If the `sidekiq-cluster` process crashes or receives a `SIGKILL`, the child -processes will terminate themselves after a few seconds. This ensures you don't +processes terminate themselves after a few seconds. This ensures you don't end up with zombie Sidekiq processes. All of this makes monitoring the processes fairly easy. Simply hook up `sidekiq-cluster` to your supervisor of choice (for example, runit) and you're good to go. -If a child process died the `sidekiq-cluster` command will signal all remaining +If a child process died the `sidekiq-cluster` command signals all remaining process to terminate, then terminate itself. This removes the need for `sidekiq-cluster` to re-implement complex process monitoring/restarting code. Instead you should make sure your supervisor restarts the `sidekiq-cluster` @@ -456,7 +383,7 @@ file is written, but this can be changed by passing the `--pidfile` option to /opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster --pidfile /var/run/gitlab/sidekiq_cluster.pid process_commit ``` -Keep in mind that the PID file will contain the PID of the `sidekiq-cluster` +Keep in mind that the PID file contains the PID of the `sidekiq-cluster` command and not the PID(s) of the started Sidekiq processes. ### Environment diff --git a/doc/administration/operations/extra_sidekiq_routing.md b/doc/administration/operations/extra_sidekiq_routing.md new file mode 100644 index 00000000000..93cf8bd4f43 --- /dev/null +++ b/doc/administration/operations/extra_sidekiq_routing.md @@ -0,0 +1,164 @@ +--- +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Queue routing rules **(FREE SELF)** + +When the number of Sidekiq jobs increases to a certain scale, the system faces +some scalability issues. One of them is that the length of the queue tends to get +longer. High-urgency jobs have to wait longer until other less urgent jobs +finish. This head-of-line blocking situation may eventually affect the +responsiveness of the system, especially critical actions. In another scenario, +the performance of some jobs is degraded due to other long running or CPU-intensive jobs +(computing or rendering ones) in the same machine. + +To counter the aforementioned issues, one effective solution is to split +Sidekiq jobs into different queues and assign machines handling each queue +exclusively. For example, all CPU-intensive jobs could be routed to the +`cpu-bound` queue and handled by a fleet of CPU optimized instances. The queue +topology differs between companies depending on the workloads and usage +patterns. Therefore, GitLab supports a flexible mechanism for the +administrator to route the jobs based on their characteristics. + +As an alternative to [Queue selector](extra_sidekiq_processes.md#queue-selector), which +configures Sidekiq cluster to listen to a specific set of workers or queues, +GitLab also supports routing a job from a worker to the desired queue when it +is scheduled. Sidekiq clients try to match a job against a configured list of +routing rules. Rules are evaluated from first to last, and as soon as we find a +match for a given worker we stop processing for that worker (first match wins). +If the worker doesn't match any rule, it falls back to the queue name generated +from the worker name. + +By default, if the routing rules are not configured (or denoted with an empty +array), all the jobs are routed to the queue generated from the worker name. + +## Example configuration + +In `/etc/gitlab/gitlab.rb`: + +```ruby +sidekiq['routing_rules'] = [ + # Route all non-CPU-bound workers that are high urgency to `high-urgency` queue + ['resource_boundary!=cpu&urgency=high', 'high-urgency'], + # Route all database, gitaly and global search workers that are throttled to `throttled` queue + ['feature_category=database,gitaly,global_search&urgency=throttled', 'throttled'], + # Route all workers having contact with outside work to a `network-intenstive` queue + ['has_external_dependencies=true|feature_category=hooks|tags=network', 'network-intensive'], + # Route all import workers to the queues generated by the worker name, for + # example, JiraImportWorker to `jira_import`, SVNWorker to `svn_worker` + ['feature_category=import', nil], + # Wildcard matching, route the rest to `default` queue + ['*', 'default'] +] +``` + +The routing rules list is an order-matter array of tuples of query and +corresponding queue: + +- The query is following a [worker matching query](#worker-matching-query) syntax. +- The `<queue_name>` must be a valid Sidekiq queue name. If the queue name + is `nil`, or an empty string, the worker is routed to the queue generated + by the name of the worker instead. + +The query supports wildcard matching `*`, which matches all workers. As a +result, the wildcard query must stay at the end of the list or the rules after it +are ignored. + +NOTE: +Mixing queue routing rules and queue selectors requires care to +ensure all jobs that are scheduled and picked up by appropriate Sidekiq +workers. + +## Worker matching query + +GitLab provides a simple query syntax to match a worker based on its +attributes. This query syntax is employed by both [Queue routing +rules](#queue-routing-rules) and [Queue +selector](extra_sidekiq_processes.md#queue-selector). A query includes two +components: + +- Attributes that can be selected. +- Operators used to construct a query. + +### Available attributes + +> [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/261) in GitLab 13.1 (`tags`). + +Queue matching query works upon the worker attributes, described in [Sidekiq +style guide](../../development/sidekiq_style_guide.md). We support querying +based on a subset of worker attributes: + +- `feature_category` - the [GitLab feature + category](https://about.gitlab.com/direction/maturity/#category-maturity) the + queue belongs to. For example, the `merge` queue belongs to the + `source_code_management` category. +- `has_external_dependencies` - whether or not the queue connects to external + services. For example, all importers have this set to `true`. +- `urgency` - how important it is that this queue's jobs run + quickly. Can be `high`, `low`, or `throttled`. For example, the + `authorized_projects` queue is used to refresh user permissions, and + is high urgency. +- `worker_name` - the worker name. The other attributes are typically more useful as + they are more general, but this is available in case a particular worker needs + to be selected. +- `name` - the queue name. The other attributes are typically more useful as + they are more general, but this is available in case a particular queue needs + to be selected. +- `resource_boundary` - if the queue is bound by `cpu`, `memory`, or + `unknown`. For example, the `ProjectExportWorker` is memory bound as it has + to load data in memory before saving it for export. +- `tags` - short-lived annotations for queues. These are expected to frequently + change from release to release, and may be removed entirely. + +`has_external_dependencies` is a boolean attribute: only the exact +string `true` is considered true, and everything else is considered +false. + +`tags` is a set, which means that `=` checks for intersecting sets, and +`!=` checks for disjoint sets. For example, `tags=a,b` selects queues +that have tags `a`, `b`, or both. `tags!=a,b` selects queues that have +neither of those tags. + +The attributes of each worker are hard-coded in the source code. For +convenience, we generate a [list of all available attributes in +GitLab Community Edition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/all_queues.yml) +and a [list of all available attributes in +GitLab Enterprise Edition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/all_queues.yml). + +### Available operators + +`queue_selector` supports the following operators, listed from highest +to lowest precedence: + +- `|` - the logical OR operator. For example, `query_a|query_b` (where `query_a` + and `query_b` are queries made up of the other operators here) will include + queues that match either query. +- `&` - the logical AND operator. For example, `query_a&query_b` (where + `query_a` and `query_b` are queries made up of the other operators here) will + only include queues that match both queries. +- `!=` - the NOT IN operator. For example, `feature_category!=issue_tracking` + excludes all queues from the `issue_tracking` feature category. +- `=` - the IN operator. For example, `resource_boundary=cpu` includes all + queues that are CPU bound. +- `,` - the concatenate set operator. For example, + `feature_category=continuous_integration,pages` includes all queues from + either the `continuous_integration` category or the `pages` category. This + example is also possible using the OR operator, but allows greater brevity, as + well as being lower precedence. + +The operator precedence for this syntax is fixed: it's not possible to make AND +have higher precedence than OR. + +[In GitLab 12.9](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26594) and +later, as with the standard queue group syntax above, a single `*` as the +entire queue group selects all queues. + +### Migration + +After the Sidekiq routing rules are changed, administrators need to take care +with the migration to avoid losing jobs entirely, especially in a system with +long queues of jobs. The migration can be done by following the migration steps +mentioned in [Sidekiq job +migration](../../raketasks/sidekiq_job_migration.md) diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md index 980db9713ee..8acc40da4ab 100644 --- a/doc/administration/operations/fast_ssh_key_lookup.md +++ b/doc/administration/operations/fast_ssh_key_lookup.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Fast lookup of authorized SSH keys in the database +# Fast lookup of authorized SSH keys in the database **(FREE SELF)** NOTE: This document describes a drop-in replacement for the @@ -34,8 +34,15 @@ feature for CentOS 6, follow [the instructions on how to build and install a cus ## Fast lookup is required for Geo **(PREMIUM)** -By default, GitLab manages an `authorized_keys` file, which contains all the -public SSH keys for users allowed to access GitLab. However, to maintain a +By default, GitLab manages an `authorized_keys` file that is located in the +`git` user's home directory. For most installations, this will be located under +`/var/opt/gitlab/.ssh/authorized_keys`, but you can use the following command to locate the `authorized_keys` on your system.: + +```shell +getent passwd git | cut -d: -f6 | awk '{print $1"/.ssh/authorized_keys"}' +``` + +The `authorized_keys` file contains all the public SSH keys for users allowed to access GitLab. However, to maintain a single source of truth, [Geo](../geo/index.md) needs to be configured to perform SSH fingerprint lookups via database lookup. @@ -73,7 +80,7 @@ sudo service sshd reload ``` Confirm that SSH is working by commenting out your user's key in the `authorized_keys` -(start the line with a `#` to comment it), and attempting to pull a repository. +file (start the line with a `#` to comment it), and attempting to pull a repository. A successful pull would mean that GitLab was able to find the key in the database, since it is not present in the file anymore. @@ -219,5 +226,5 @@ the database. The following instructions can be used to build OpenSSH 7.5: GitLab supports `authorized_keys` database lookups with [SELinux](https://en.wikipedia.org/wiki/Security-Enhanced_Linux). Because the SELinux policy is static, GitLab doesn't support the ability to change -internal Unicorn ports at the moment. Administrators would have to create a special `.te` +internal webserver ports at the moment. Administrators would have to create a special `.te` file for the environment, since it isn't generated dynamically. diff --git a/doc/administration/operations/filesystem_benchmarking.md b/doc/administration/operations/filesystem_benchmarking.md index ffce104e1ad..a0ad2e24a4c 100644 --- a/doc/administration/operations/filesystem_benchmarking.md +++ b/doc/administration/operations/filesystem_benchmarking.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# File system Performance Benchmarking +# File system performance benchmarking **(FREE SELF)** File system performance has a big impact on overall GitLab performance, especially for actions that read or write to Git repositories. This information diff --git a/doc/administration/operations/index.md b/doc/administration/operations/index.md index 708861d8529..4b16c3b3a7e 100644 --- a/doc/administration/operations/index.md +++ b/doc/administration/operations/index.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Performing Operations in GitLab +# Performing operations in GitLab **(FREE SELF)** Keep your GitLab instance up and running smoothly. @@ -21,8 +21,8 @@ Keep your GitLab instance up and running smoothly. - [Sidekiq MemoryKiller](sidekiq_memory_killer.md): Configure Sidekiq MemoryKiller to restart Sidekiq. - [Multiple Sidekiq processes](extra_sidekiq_processes.md): Configure multiple Sidekiq processes to ensure certain queues always have dedicated workers, no matter the number of jobs that need to be processed. **(FREE SELF)** +- [Sidekiq routing rules](extra_sidekiq_routing.md): Configure the routing rules to route a job from a worker to a desirable queue. **(FREE SELF)** - [Puma](puma.md): Understand Puma and puma-worker-killer. -- [Unicorn](unicorn.md): Understand Unicorn and unicorn-worker-killer. - Speed up SSH operations by [Authorizing SSH users via a fast, indexed lookup to the GitLab database](fast_ssh_key_lookup.md), and/or by [doing away with user SSH keys stored on GitLab entirely in favor diff --git a/doc/administration/operations/puma.md b/doc/administration/operations/puma.md index 3b676010bfe..fffff78b9d6 100644 --- a/doc/administration/operations/puma.md +++ b/doc/administration/operations/puma.md @@ -4,11 +4,10 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Switching to Puma +# Switching to Puma **(FREE SELF)** As of GitLab 12.9, [Puma](https://github.com/puma/puma) has replaced [Unicorn](https://yhbt.net/unicorn/) -as the default web server. From GitLab 13.0, the following run Puma instead of Unicorn unless -explicitly configured not to: +as the default web server. From GitLab 14.0, the following run Puma: - All-in-one package-based installations. - Helm chart-based installations. @@ -25,7 +24,7 @@ Multi-threaded Puma can therefore still serve more requests than a single proces ## Configuring Puma to replace Unicorn -Beginning with GitLab 13.0, Puma is the default application server. We plan to remove support for +Beginning with GitLab 13.0, Puma is the default application server. We removed support for Unicorn in GitLab 14.0. When switching to Puma, Unicorn server configuration diff --git a/doc/administration/operations/sidekiq_memory_killer.md b/doc/administration/operations/sidekiq_memory_killer.md index c7f00d05213..d3019e2c580 100644 --- a/doc/administration/operations/sidekiq_memory_killer.md +++ b/doc/administration/operations/sidekiq_memory_killer.md @@ -19,7 +19,7 @@ _only_ for Omnibus packages. The reason for this is that the MemoryKiller relies on runit to restart Sidekiq after a memory-induced shutdown and GitLab installations from source do not all use runit or an equivalent. -With the default settings, the MemoryKiller will cause a Sidekiq restart no +With the default settings, the MemoryKiller causes a Sidekiq restart no more often than once every 15 minutes, with the restart causing about one minute of delay for incoming background jobs. @@ -48,13 +48,13 @@ The MemoryKiller is controlled using environment variables. `SIDEKIQ_MEMORY_KILLER_MAX_RSS` defines the Sidekiq process allowed RSS. In _legacy_ mode, if the Sidekiq process exceeds the allowed RSS then an irreversible - delayed graceful restart will be triggered. The restart of Sidekiq will happen + delayed graceful restart is triggered. The restart of Sidekiq happens after `SIDEKIQ_MEMORY_KILLER_GRACE_TIME` seconds. In _daemon_ mode, if the Sidekiq process exceeds the allowed RSS for longer than - `SIDEKIQ_MEMORY_KILLER_GRACE_TIME` the graceful restart will be triggered. If the + `SIDEKIQ_MEMORY_KILLER_GRACE_TIME` the graceful restart is triggered. If the Sidekiq process go below the allowed RSS within `SIDEKIQ_MEMORY_KILLER_GRACE_TIME`, - the restart will be aborted. + the restart is aborted. The default value for Omnibus packages is set [in the Omnibus GitLab @@ -71,13 +71,13 @@ The MemoryKiller is controlled using environment variables. The usage of this variable is described as part of `SIDEKIQ_MEMORY_KILLER_MAX_RSS`. - `SIDEKIQ_MEMORY_KILLER_SHUTDOWN_WAIT`: defaults to 30 seconds. This defines the - maximum time allowed for all Sidekiq jobs to finish. No new jobs will be accepted - during that time, and the process will exit as soon as all jobs finish. + maximum time allowed for all Sidekiq jobs to finish. No new jobs are accepted + during that time, and the process exits as soon as all jobs finish. - If jobs do not finish during that time, the MemoryKiller will interrupt all currently + If jobs do not finish during that time, the MemoryKiller interrupts all currently running jobs by sending `SIGTERM` to the Sidekiq process. If the process hard shutdown/restart is not performed by Sidekiq, - the Sidekiq process will be forcefully terminated after + the Sidekiq process is forcefully terminated after `Sidekiq.options[:timeout] + 2` seconds. An external supervision mechanism (e.g. runit) must restart Sidekiq afterwards. diff --git a/doc/administration/operations/ssh_certificates.md b/doc/administration/operations/ssh_certificates.md index cc09ad95dce..508d284b0bd 100644 --- a/doc/administration/operations/ssh_certificates.md +++ b/doc/administration/operations/ssh_certificates.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# User lookup via OpenSSH's AuthorizedPrincipalsCommand +# User lookup via OpenSSH's AuthorizedPrincipalsCommand **(FREE SELF)** > [Available in](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/19911) GitLab > Community Edition 11.2. diff --git a/doc/administration/operations/unicorn.md b/doc/administration/operations/unicorn.md index 03995ee05ba..6cee19186f9 100644 --- a/doc/administration/operations/unicorn.md +++ b/doc/administration/operations/unicorn.md @@ -1,115 +1,9 @@ --- -stage: Enablement -group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: 'puma.md' +remove_date: '2021-08-26' --- -# Understanding Unicorn and unicorn-worker-killer +This file was moved to [another location](puma.md). -NOTE: -Starting with GitLab 13.0, Puma is the default web server used in GitLab -all-in-one package based installations as well as GitLab Helm chart deployments. - -## Unicorn - -GitLab uses [Unicorn](https://yhbt.net/unicorn/), a pre-forking Ruby web -server, to handle web requests (web browsers and Git HTTP clients). Unicorn is -a daemon written in Ruby and C that can load and run a Ruby on Rails -application; in our case the Rails application is GitLab Community Edition or -GitLab Enterprise Edition. - -Unicorn has a multi-process architecture to make better use of available CPU -cores (processes can run on different cores) and to have stronger fault -tolerance (most failures stay isolated in only one process and cannot take down -GitLab entirely). On startup, the Unicorn 'master' process loads a clean Ruby -environment with the GitLab application code, and then spawns 'workers' which -inherit this clean initial environment. The 'master' never handles any -requests, that is left to the workers. The operating system network stack -queues incoming requests and distributes them among the workers. - -In a perfect world, the master would spawn its pool of workers once, and then -the workers handle incoming web requests one after another until the end of -time. In reality, worker processes can crash or time out: if the master notices -that a worker takes too long to handle a request it will terminate the worker -process with SIGKILL ('kill -9'). No matter how the worker process ended, the -master process will replace it with a new 'clean' process again. Unicorn is -designed to be able to replace 'crashed' workers without dropping user -requests. - -This is what a Unicorn worker timeout looks like in `unicorn_stderr.log`. The -master process has PID 56227 below. - -```plaintext -[2015-06-05T10:58:08.660325 #56227] ERROR -- : worker=10 PID:53009 timeout (61s > 60s), killing -[2015-06-05T10:58:08.699360 #56227] ERROR -- : reaped #<Process::Status: pid 53009 SIGKILL (signal 9)> worker=10 -[2015-06-05T10:58:08.708141 #62538] INFO -- : worker=10 spawned pid=62538 -[2015-06-05T10:58:08.708824 #62538] INFO -- : worker=10 ready -``` - -### Tunable options - -The main tunable options for Unicorn are the number of worker processes and the -request timeout after which the Unicorn master terminates a worker process. -See the [Omnibus GitLab Unicorn settings -documentation](https://docs.gitlab.com/omnibus/settings/unicorn.html) -if you want to adjust these settings. - -## unicorn-worker-killer - -GitLab has memory leaks. These memory leaks manifest themselves in long-running -processes, such as Unicorn workers. (The Unicorn master process is not known to -leak memory, probably because it does not handle user requests.) - -To make these memory leaks manageable, GitLab comes with the -[unicorn-worker-killer gem](https://github.com/kzk/unicorn-worker-killer). This -gem [monkey-patches](https://en.wikipedia.org/wiki/Monkey_patch) the Unicorn -workers to do a memory self-check after every 16 requests. If the memory of the -Unicorn worker exceeds a pre-set limit then the worker process exits. The -Unicorn master then automatically replaces the worker process. - -This is a robust way to handle memory leaks: Unicorn is designed to handle -workers that 'crash' so no user requests will be dropped. The -unicorn-worker-killer gem is designed to only terminate a worker process _in -between requests_, so no user requests are affected. You can set the minimum and -maximum memory threshold (in bytes) for the Unicorn worker killer by -setting the following values `/etc/gitlab/gitlab.rb`: - -- For GitLab **12.7** and newer: - - ```ruby - unicorn['worker_memory_limit_min'] = "1024 * 1 << 20" - unicorn['worker_memory_limit_max'] = "1280 * 1 << 20" - ``` - -- For GitLab **12.6** and older: - - ```ruby - unicorn['worker_memory_limit_min'] = "400 * 1 << 20" - unicorn['worker_memory_limit_max'] = "650 * 1 << 20" - ``` - -Otherwise, you can set the `GITLAB_UNICORN_MEMORY_MIN` and `GITLAB_UNICORN_MEMORY_MAX` -[environment variables](../environment_variables.md). - -This is what a Unicorn worker memory restart looks like in unicorn_stderr.log. -You see that worker 4 (PID 125918) is inspecting itself and decides to exit. -The threshold memory value was 254802235 bytes, about 250MB. With GitLab this -threshold is a random value between 200 and 250 MB. The master process (PID -117565) then reaps the worker process and spawns a new 'worker 4' with PID -127549. - -```plaintext -[2015-06-05T12:07:41.828374 #125918] WARN -- : #<Unicorn::HttpServer:0x00000002734770>: worker (pid: 125918) exceeds memory limit (256413696 bytes > 254802235 bytes) -[2015-06-05T12:07:41.828472 #125918] WARN -- : Unicorn::WorkerKiller send SIGQUIT (pid: 125918) alive: 23 sec (trial 1) -[2015-06-05T12:07:42.025916 #117565] INFO -- : reaped #<Process::Status: pid 125918 exit 0> worker=4 -[2015-06-05T12:07:42.034527 #127549] INFO -- : worker=4 spawned pid=127549 -[2015-06-05T12:07:42.035217 #127549] INFO -- : worker=4 ready -``` - -One other thing that stands out in the log snippet above, taken from -GitLab.com, is that 'worker 4' was serving requests for only 23 seconds. This -is a normal value for our current GitLab.com setup and traffic. - -The high frequency of Unicorn memory restarts on some GitLab sites can be a -source of confusion for administrators. Usually they are a [red -herring](https://en.wikipedia.org/wiki/Red_herring). +<!-- This redirect file can be deleted after <2021-08-26>. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md index 14e54513536..a6829b90f18 100644 --- a/doc/administration/packages/container_registry.md +++ b/doc/administration/packages/container_registry.md @@ -44,7 +44,7 @@ If you have installed GitLab from source: 1. After the installation is complete, to enable it, you must configure the Registry's settings in `gitlab.yml`. 1. Use the sample NGINX configuration file from under - [`lib/support/nginx/registry-ssl`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/support/nginx/registry-ssl) and edit it to match the + [`lib/support/nginx/registry-ssl`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/support/nginx/registry-ssl) and edit it to match the `host`, `port`, and TLS certificate paths. The contents of `gitlab.yml` are: @@ -417,8 +417,27 @@ To configure the `s3` storage driver in Omnibus: } ``` - - `regionendpoint` is only required when configuring an S3 compatible service such as MinIO. It takes a URL such as `http://127.0.0.1:9000`. + If using with an [AWS S3 VPC endpoint](https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-s3.html), + then set `regionendpoint` to your VPC endpoint address and set `path_style` to false: + + ```ruby + registry['storage'] = { + 's3' => { + 'accesskey' => 's3-access-key', + 'secretkey' => 's3-secret-key-for-access-key', + 'bucket' => 'your-s3-bucket', + 'region' => 'your-s3-region', + 'regionendpoint' => 'your-s3-vpc-endpoint', + 'path_style' => false + } + } + ``` + + - `regionendpoint` is only required when configuring an S3 compatible service such as MinIO, or + when using an AWS S3 VPC Endpoint. - `your-s3-bucket` should be the name of a bucket that exists, and can't include subdirectories. + - `path_style` should be set to true to use `host/bucket_name/object` style paths instead of + `bucket_name.host/object`. [Set to false for AWS S3](https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/). 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -1274,6 +1293,88 @@ curl "localhost:5001/debug/health" curl "localhost:5001/debug/vars" ``` +### Access old schema v1 Docker images + +Support for the [Docker registry v1 API](https://www.docker.com/blog/registry-v1-api-deprecation/), +including [schema V1 image manifests](https://docs.docker.com/registry/spec/manifest-v2-1/), +was: + +- [Deprecated in GitLab 13.7](https://about.gitlab.com/releases/2020/12/22/gitlab-13-7-released/#deprecate-pulls-that-use-v1-of-the-docker-registry-api) +- [Removed in GitLab 13.9](https://about.gitlab.com/releases/2021/02/22/gitlab-13-9-released/#deprecate-pulls-that-use-v1-of-the-docker-registry-api) + +It's no longer possible to push or pull v1 images from the GitLab Container Registry. + +If you had v1 images in the GitLab Container Registry, but you did not upgrade them (following the +[steps Docker recommends](https://docs.docker.com/registry/spec/deprecated-schema-v1/)) +ahead of the GitLab 13.9 upgrade, these images are no longer accessible. If you try to pull them, +this error appears: + +- `Error response from daemon: manifest invalid: Schema 1 manifest not supported` + +For Self-Managed GitLab instances, you can regain access to these images by temporarily downgrading +the GitLab Container Registry to a version lower than `v3.0.0-gitlab`. Follow these steps to regain +access to these images: + +1. Downgrade the Container Registry to [`v2.13.1-gitlab`](https://gitlab.com/gitlab-org/container-registry/-/releases/v2.13.1-gitlab). +1. Upgrade any v1 images. +1. Revert the Container Registry downgrade. + +There's no need to put the registry in read-only mode during the image upgrade process. Ensure that +you are not relying on any new feature introduced since `v3.0.0-gitlab`. Such features are +unavailable during the upgrade process. See the [complete registry changelog](https://gitlab.com/gitlab-org/container-registry/-/blob/master/CHANGELOG.md) +for more information. + +The following sections provide additional details about each installation method. + +#### Helm chart installations + +For Helm chart installations: + +1. Override the [`image.tag`](https://docs.gitlab.com/charts/charts/registry/#configuration) + configuration parameter with `v2.13.1-gitlab`. +1. Restart. +1. Performing the [images upgrade](#images-upgrade)) steps. +1. Revert the `image.tag` parameter to the previous value. + +No other registry configuration changes are required. + +#### Omnibus installations + +For Omnibus installations: + +1. Temporarily replace the registry binary that ships with GitLab 13.9+ for one prior to + `v3.0.0-gitlab`. To do so, pull a previous version of the Docker image for the GitLab Container + Registry, such as `v2.13.1-gitlab`. You can then grab the `registry` binary from within this + image, located at `/bin/registry`: + + ```shell + id=$(docker create registry.gitlab.com/gitlab-org/build/cng/gitlab-container-registry:v2.13.1-gitlab) + docker cp $id:/bin/registry registry-2.13.1-gitlab + docker rm $id + ``` + +1. Replace the binary embedded in the Omnibus install, located at + `/opt/gitlab/embedded/bin/registry`, with `registry-2.13.1-gitlab`. Make sure to start by backing + up the original binary embedded in Omnibus, and restore it after performing the + [image upgrade](#images-upgrade)) steps. You should [stop](https://docs.gitlab.com/omnibus/maintenance/#starting-and-stopping) + the registry service before replacing its binary and start it right after. No registry + configuration changes are required. + +#### Source installations + +For source installations, locate your `registry` binary and temporarily replace it with the one +obtained from `v3.0.0-gitlab`, as explained for [Omnibus installations](#omnibus-installations). +Make sure to start by backing up the original registry binary, and restore it after performing the +[images upgrade](#images-upgrade)) +steps. + +#### Images upgrade + +Follow the [steps that Docker recommends to upgrade v1 images](https://docs.docker.com/registry/spec/deprecated-schema-v1/). +The most straightforward option is to pull those images and push them once again to the registry, +using a Docker client version above v1.12. Docker converts images automatically before pushing them +to the registry. Once done, all your v1 images should now be available as v2 images. + ### Advanced Troubleshooting We use a concrete example to illustrate how to diff --git a/doc/administration/packages/dependency_proxy.md b/doc/administration/packages/dependency_proxy.md index b454728cc8b..c4906ef6d8e 100644 --- a/doc/administration/packages/dependency_proxy.md +++ b/doc/administration/packages/dependency_proxy.md @@ -32,6 +32,23 @@ To enable the dependency proxy feature: 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab") for the changes to take effect. 1. Enable the [Puma web server](https://docs.gitlab.com/omnibus/settings/puma.html). +**Helm chart installations** + +1. After the installation is complete, update the global `appConfig` to enable the feature: + + ```yaml + global: + appConfig: + dependencyProxy: + enabled: true + bucket: gitlab-dependency-proxy + connection: {} + secret: + key: + ``` + +For more information, see [Configure Charts using Globals](https://docs.gitlab.com/charts/charts/globals.html#configure-appconfig-settings). + **Installations from source** 1. After the installation is complete, configure the `dependency_proxy` diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index c93ae90deb6..b9637f1b6f5 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -246,7 +246,7 @@ control over how the Pages daemon runs and serves content in your environment. | `gitlab_retrieval_timeout` | The maximum time to wait for a response from the GitLab API per request (default: 30s). | | `gitlab_retrieval_interval` | The interval to wait before retrying to resolve a domain's configuration via the GitLab API (default: 1s). | | `gitlab_retrieval_retries` | The maximum number of times to retry to resolve a domain's configuration via the API (default: 3). | -| `domain_config_source` | Domain configuration source (default: `auto`) | +| `domain_config_source` | This parameter was removed in 14.0, on earlier versions it can be used to enable and test API domain configuration source | | `gitlab_id` | The OAuth application public ID. Leave blank to automatically fill when Pages authenticates with GitLab. | | `gitlab_secret` | The OAuth application secret. Leave blank to automatically fill when Pages authenticates with GitLab. | | `auth_scope` | The OAuth application scope to use for authentication. Must match GitLab Pages OAuth application settings. Leave blank to use `api` scope by default. | @@ -281,6 +281,7 @@ control over how the Pages daemon runs and serves content in your environment. | **`pages_nginx[]`** | | | `enable` | Include a virtual host `server{}` block for Pages inside NGINX. Needed for NGINX to proxy traffic back to the Pages daemon. Set to `false` if the Pages daemon should directly receive all requests, for example, when using [custom domains](index.md#custom-domains). | | `FF_ENABLE_REDIRECTS` | Feature flag to disable redirects (enabled by default). Read the [redirects documentation](../../user/project/pages/redirects.md#disable-redirects) for more information. | +| `use_legacy_storage` | Temporarily-introduced parameter allowing to use legacy domain configuration source and storage. [Will be removed in GitLab 14.3](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6166). | --- @@ -756,51 +757,37 @@ Pages server. ## Domain source configuration -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217912) in GitLab 13.3. +When GitLab Pages daemon serves pages requests it firstly needs to identify which project should be used to +serve the requested URL and how its content is stored. -GitLab Pages can use different sources to get domain configuration. -The default value for Omnibus installations is `nil`. +Before GitLab 13.3, all pages content was extracted to the special shared directory, +and each project had a special configuration file. +The Pages daemon was reading these configuration files and storing their content in memory. - ```ruby - gitlab_pages['domain_config_source'] = nil - ``` +This approach had several disadvantages and was replaced with GitLab Pages using the internal GitLab API +every time a new domain is requested. +The domain information is also cached by the Pages daemon to speed up subsequent requests. -If left unchanged, GitLab Pages tries to use any available source (either `gitlab` or `disk`). The -preferred source is `gitlab`, which uses [API-based configuration](#gitlab-api-based-configuration). +From [GitLab 13.3 to GitLab 13.12](#domain-source-configuration-before-140) GitLab Pages supported both ways of obtaining domain information. -On large GitLab instances, using the API-based configuration significantly improves the pages daemon startup time, as there is no need to load all custom domains configuration into memory. +Starting from [GitLab 14.0](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5993) GitLab Pages uses API +by default and fails to start if it can't connect to it. +For common issues, see the [troubleshooting section](#failed-to-connect-to-the-internal-gitlab-api). For more details see this [blog post](https://about.gitlab.com/blog/2020/08/03/how-gitlab-pages-uses-the-gitlab-api-to-serve-content/). -### Deprecated `domain_config_source` - -WARNING: -The flag `gitlab_pages['domain_config_source']` is deprecated for use in [GitLab 13.9](https://gitlab.com/gitlab-org/gitlab/-/issues/217913), -and is planned for removal in GitLab 14.0. - -GitLab 13.0 introduced the special flag `domain_config_source` to support manual opt-in to -[API-based configuration](#gitlab-api-based-configuration). -GitLab 13.7 introduced the [`auto` value](https://gitlab.com/gitlab-org/gitlab/-/issues/218358) -to support a smoother transition to API-based configuration. +### Domain source configuration before 14.0 -Starting with GitLab 14.0, GitLab Pages only supports API-based configuration, and -[disk source configuration is removed](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/382). -Therefore, GitLab 14.0 also removes `domain_config_source`. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217912) in GitLab 13.3. -GitLab Pages fails to start if it can't connect to the GitLab API. For other common issues, see the -[troubleshooting section](#failed-to-connect-to-the-internal-gitlab-api) -or report an issue. +WARNING: +`domain_config_source` parameter is removed and has no effect starting from [GitLab 14.0](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5993) -### GitLab API-based configuration +From [GitLab 13.3](https://gitlab.com/gitlab-org/gitlab/-/issues/217912) to [GitLab 13.12](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5993) GitLab Pages can either use `disk` or `gitlab` domain configuration source. -WARNING: -The flag `gitlab_pages['domain_config_source']` is deprecated for use in [GitLab 13.9](https://gitlab.com/gitlab-org/gitlab/-/issues/217913), -and is planned for removal in GitLab 14.0. In GitLab 14.0 and later, GitLab Pages attempts to -connect to the API automatically, without requiring the manual configuration steps shown here. Pages -fails to start if this automatic connection fails. +We highly advise you to use `gitlab` configuration source as it will make transition to newer versions easier. -GitLab Pages can use an API-based configuration. This replaces disk source configuration, which -was used prior to GitLab 13.0. Follow these steps to enable it: +To explicitly enable API source: 1. Add the following to your `/etc/gitlab/gitlab.rb` file: @@ -810,14 +797,15 @@ was used prior to GitLab 13.0. Follow these steps to enable it: 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -If you encounter an issue, you can disable it by choosing `disk`: +Or if you want to use legacy confiration source you can: -```ruby -gitlab_pages['domain_config_source'] = "disk" -``` +1. Add the following to your `/etc/gitlab/gitlab.rb` file: + + ```ruby + gitlab_pages['domain_config_source'] = "disk" + ``` -For other common issues, see the [troubleshooting section](#failed-to-connect-to-the-internal-gitlab-api) -or report an issue. +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. ### GitLab API cache configuration @@ -831,8 +819,8 @@ or persistent errors, or the Pages Daemon serving old content. NOTE: Expiry, interval and timeout flags use [Golang's duration formatting](https://golang.org/pkg/time/#ParseDuration). A duration string is a possibly signed sequence of decimal numbers, -each with optional fraction and a unit suffix, such as "300ms", "1.5h" or "2h45m". -Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". +each with optional fraction and a unit suffix, such as `300ms`, `1.5h` or `2h45m`. +Valid time units are `ns`, `us` (or `µs`), `ms`, `s`, `m`, `h`. Examples: @@ -938,7 +926,7 @@ In installations from source: ## ZIP storage -In GitLab 14.0 the underlaying storage format of GitLab Pages is changing from +In GitLab 14.0 the underlying storage format of GitLab Pages is changing from files stored directly in disk to a single ZIP archive per project. These ZIP archives can be stored either locally on disk storage or on the [object storage](#using-object-storage) if it is configured. @@ -1052,11 +1040,12 @@ To migrate GitLab Pages to GitLab 14.0: 1. If your current GitLab version is lower than 13.12, then you first need to upgrade to 13.12. Upgrading directly to 14.0 may cause downtime for some web-sites hosted on GitLab Pages until you finish the following steps. -1. Enable the [API-based configuration](#gitlab-api-based-configuration), which +1. Set [`domain_config_source` to `gitlab`](#domain-source-configuration-before-140), which is the default starting from GitLab 14.0. Skip this step if you're already running GitLab 14.0 or above. 1. If you want to store your pages content in the [object storage](#using-object-storage), make sure to configure it. If you want to store the pages content locally or continue using an NFS server, skip this step. 1. [Migrate legacy storage to ZIP storage.](#migrate-legacy-storage-to-zip-storage) +1. Upgrade GitLab to 14.0. ## Backup @@ -1081,6 +1070,16 @@ but commented out to help encourage others to add to it in the future. --> ## Troubleshooting +### How to see GitLab Pages logs + +You can see Pages daemon logs by running: + +```shell +sudo gitlab-ctl tail gitlab-pages +``` + +You can also find the log file in `/var/log/gitlab/gitlab-pages/current`. + ### `open /etc/ssl/ca-bundle.pem: permission denied` GitLab Pages runs inside a `chroot` jail, usually in a uniquely numbered directory like @@ -1210,12 +1209,12 @@ These are due to the Pages files not being among the It is possible to copy the subfolders and files in the [Pages path](#change-storage-path) to the new primary node to resolve this. For example, you can adapt the `rsync` strategy from the -[moving repositories documenation](../operations/moving_repositories.md). +[moving repositories documentation](../operations/moving_repositories.md). Alternatively, run the CI pipelines of those projects that contain a `pages` job again. ### Failed to connect to the internal GitLab API -If you have enabled [API-based configuration](#gitlab-api-based-configuration) and see the following error: +If you see the following error: ```plaintext ERRO[0010] Failed to connect to the internal GitLab API after 0.50s error="failed to connect to internal Pages API: HTTP status: 401" @@ -1236,11 +1235,6 @@ error="failed to connect to internal Pages API: Get \"https://gitlab.example.com ### Pages cannot communicate with an instance of the GitLab API -WARNING: -The flag `gitlab_pages['domain_config_source']` is [deprecated](#deprecated-domain_config_source) -for use in [GitLab 13.9](https://gitlab.com/gitlab-org/gitlab/-/issues/217913), -and is planned for removal in GitLab 14.0. - If you use the default value for `domain_config_source=auto` and run multiple instances of GitLab Pages, you may see intermittent 502 error responses while serving Pages content. You may also see the following warning in the Pages logs: @@ -1321,3 +1315,25 @@ To enable disk access: ``` 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +### GitLab Pages doesn't work after upgrading to GitLab 14.0 or above + +GitLab 14.0 introduces a number of changes to GitLab Pages which may require manual intervention. + +1. Firstly [follow the migration guide](#migrate-gitlab-pages-to-140). +1. If it doesn't work, see [GitLab Pages logs](#how-to-see-gitlab-pages-logs), and if you see any errors there then search them on this page. + +WARNING: +As the last resort you can temporarily enable legacy storage and configuration mechanisms. Support for them [will be removed in GitLab 14.3](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6166), so GitLab Pages will stop working if don't resolve the underlying issue. + +To do that: + +1. Please describe the issue you're seeing in [here](https://gitlab.com/gitlab-org/gitlab/-/issues/331699). + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_pages['use_legacy_storage'] = true + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). diff --git a/doc/administration/polling.md b/doc/administration/polling.md index f66df70a163..f6732b8edc6 100644 --- a/doc/administration/polling.md +++ b/doc/administration/polling.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Polling configuration +# Polling configuration **(FREE SELF)** The GitLab UI polls for updates for different resources (issue notes, issue titles, pipeline statuses, etc.) on a schedule appropriate to the resource. diff --git a/doc/administration/postgresql/external.md b/doc/administration/postgresql/external.md index a9d0af952a0..8f0fe0ace87 100644 --- a/doc/administration/postgresql/external.md +++ b/doc/administration/postgresql/external.md @@ -22,6 +22,10 @@ If you use a cloud-managed service, or provide your own PostgreSQL instance: roles to your `gitlab` user: - Amazon RDS requires the [`rds_superuser`](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.html#Appendix.PostgreSQL.CommonDBATasks.Roles) role. - Azure Database for PostgreSQL requires the [`azure_pg_admin`](https://docs.microsoft.com/en-us/azure/postgresql/howto-create-users#how-to-create-additional-admin-users-in-azure-database-for-postgresql) role. + - Google Cloud SQL requires the [`cloudsqlsuperuser`](https://cloud.google.com/sql/docs/postgres/users#default-users) role. + + This is for the installation of extensions during installation and upgrades. As an alternative, + [ensure the extensions are installed manually, and read about the problems that may arise during future GitLab upgrades](../../install/postgresql_extensions.md). 1. Configure the GitLab application servers with the appropriate connection details for your external PostgreSQL service in your `/etc/gitlab/gitlab.rb` file: diff --git a/doc/administration/postgresql/img/pg_ha_architecture.png b/doc/administration/postgresql/img/pg_ha_architecture.png Binary files differindex ef870f652ae..5d2a4a584bf 100644 --- a/doc/administration/postgresql/img/pg_ha_architecture.png +++ b/doc/administration/postgresql/img/pg_ha_architecture.png diff --git a/doc/administration/postgresql/index.md b/doc/administration/postgresql/index.md index eabb396aeab..bce78bbccff 100644 --- a/doc/administration/postgresql/index.md +++ b/doc/administration/postgresql/index.md @@ -16,7 +16,7 @@ There are essentially three setups to choose from. This setup is for when you have installed GitLab using the [Omnibus GitLab **Enterprise Edition** (EE) package](https://about.gitlab.com/install/?version=ee). -All the tools that are needed like PostgreSQL, PgBouncer, Patroni, and repmgr are bundled in +All the tools that are needed like PostgreSQL, PgBouncer, and Patroni are bundled in the package, so you can it to set up the whole PostgreSQL infrastructure (primary, replica). [> Read how to set up PostgreSQL replication and failover using Omnibus GitLab](replication_and_failover.md) diff --git a/doc/administration/postgresql/pgbouncer.md b/doc/administration/postgresql/pgbouncer.md index fc7da04b960..e481fcb71f4 100644 --- a/doc/administration/postgresql/pgbouncer.md +++ b/doc/administration/postgresql/pgbouncer.md @@ -164,7 +164,7 @@ and [GitLab upgrades](https://docs.gitlab.com/omnibus/update/README.html#use-pos 1. To find the primary node, run the following on a database node: ```shell - sudo gitlab-ctl repmgr cluster show + sudo gitlab-ctl patroni members ``` 1. Edit `/etc/gitlab/gitlab.rb` on the application node you're performing the task on, and update diff --git a/doc/administration/postgresql/replication_and_failover.md b/doc/administration/postgresql/replication_and_failover.md index 878d2b536cb..b6d2e36851d 100644 --- a/doc/administration/postgresql/replication_and_failover.md +++ b/doc/administration/postgresql/replication_and_failover.md @@ -36,8 +36,8 @@ to avoid the network becoming a single point of failure. NOTE: As of GitLab 13.3, PostgreSQL 12 is shipped with Omnibus GitLab. Clustering for PostgreSQL 12 is only supported with -Patroni. See the [Patroni](#patroni) section for further details. The support for repmgr will not be extended beyond -PostgreSQL 11. +Patroni. See the [Patroni](#patroni) section for further details. Starting with GitLab 14.0, only PostgreSQL 12 is +shipped with Omnibus GitLab and thus Patroni becomes mandatory for replication and failover. ### Database node @@ -118,26 +118,27 @@ When using default setup, minimum configuration requires: Few notes on the service itself: - The service runs under a system account, by default `gitlab-consul`. - - If you are using a different username, you will have to specify it. We - will refer to it with `CONSUL_USERNAME`, -- There will be a database user created with read-only access to the repmgr - database -- Passwords will be stored in the following locations: + - If you are using a different username, you have to specify it through the + `CONSUL_USERNAME` variable. +- Passwords are stored in the following locations: - `/etc/gitlab/gitlab.rb`: hashed - `/var/opt/gitlab/pgbouncer/pg_auth`: hashed - `/var/opt/gitlab/consul/.pgpass`: plaintext #### PostgreSQL information -When configuring PostgreSQL, we will set `max_wal_senders` to one more than -the number of database nodes in the cluster. -This is used to prevent replication from using up all of the -available database connections. +When configuring PostgreSQL, we do the following: + +- Set `max_replication_slots` to double the number of database nodes. + Patroni uses one extra slot per node when initiating the replication. +- Set `max_wal_senders` to one more than the allocated number of replication slots in the cluster. + This prevents replication from using up all of the available database connections. In this document we are assuming 3 database nodes, which makes this configuration: ```ruby -patroni['postgresql']['max_wal_senders'] = 4 +patroni['postgresql']['max_replication_slots'] = 6 +patroni['postgresql']['max_wal_senders'] = 7 ``` As previously mentioned, you'll have to prepare the network subnets that will @@ -176,9 +177,7 @@ Few notes on the service itself: - The service runs as the same system account as the database - In the package, this is by default `gitlab-psql` - If you use a non-default user account for PgBouncer service (by default `pgbouncer`), you will have to specify this username. We will refer to this requirement with `PGBOUNCER_USERNAME`. -- The service will have a regular database user account generated for it - - This defaults to `repmgr` -- Passwords will be stored in the following locations: +- Passwords are stored in the following locations: - `/etc/gitlab/gitlab.rb`: hashed, and in plain text - `/var/opt/gitlab/pgbouncer/pg_auth`: hashed @@ -198,8 +197,7 @@ When installing the GitLab package, do not supply `EXTERNAL_URL` value. #### Configuring Patroni cluster -You must enable Patroni explicitly to be able to use it (with `patroni['enable'] = true`). When Patroni is enabled -repmgr will be disabled automatically. +You must enable Patroni explicitly to be able to use it (with `patroni['enable'] = true`). Any PostgreSQL configuration item that controls replication, for example `wal_level`, `max_wal_senders`, etc, are strictly controlled by Patroni and will override the original settings that you make with the `postgresql[...]` configuration key. @@ -210,17 +208,14 @@ configuration key. NOTE: The configuration of a Patroni node is very similar to a repmgr but shorter. When Patroni is enabled, first you can ignore -any replication setting of PostgreSQL (it will be overwritten anyway). Then you can remove any `repmgr[...]` or +any replication setting of PostgreSQL (it is overwritten anyway). Then you can remove any `repmgr[...]` or repmgr-specific configuration as well. Especially, make sure that you remove `postgresql['shared_preload_libraries'] = 'repmgr_funcs'`. -Here is an example similar to [the one that was done with repmgr](#configuring-repmgr-nodes): +Here is an example: ```ruby -# Disable all components except PostgreSQL, Patroni (or Repmgr), and Consul -roles['postgres_role'] - -# Enable Patroni (which automatically disables Repmgr). -patroni['enable'] = true +# Disable all components except Patroni and Consul +roles(['patroni_role']) # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' @@ -236,13 +231,20 @@ consul['services'] = %w(postgresql) # # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value postgresql['pgbouncer_user_password'] = 'PGBOUNCER_PASSWORD_HASH' +# Replace POSTGRESQL_REPLICATION_PASSWORD_HASH with a generated md5 value +postgresql['sql_replication_password'] = 'POSTGRESQL_REPLICATION_PASSWORD_HASH' # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' -# Replace X with value of number of db nodes + 1 (OPTIONAL the default value is 5) -patroni['postgresql']['max_wal_senders'] = X +# Sets `max_replication_slots` to double the number of database nodes. +# Patroni uses one extra slot per node when initiating the replication. patroni['postgresql']['max_replication_slots'] = X +# Set `max_wal_senders` to one more than the number of replication slots in the cluster. +# This is used to prevent replication from using up all of the +# available database connections. +patroni['postgresql']['max_wal_senders'] = X+1 + # Replace XXX.XXX.XXX.XXX/YY with Network Address postgresql['trust_auth_cidr_addresses'] = %w(XXX.XXX.XXX.XXX/YY) @@ -267,10 +269,6 @@ Generally, when Consul cluster is ready, the first node that [reconfigures](../r becomes the leader. You do not need to sequence the nodes reconfiguration. You can run them in parallel or in any order. If you choose an arbitrary order you do not have any predetermined master. -NOTE: -As opposed to repmgr, once the nodes are reconfigured you do not need any further action or additional command to join -the replicas. - #### Enable Monitoring > [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3786) in GitLab 12.0. @@ -298,7 +296,7 @@ If you enable Monitoring, it must be enabled on **all** database servers. ```ruby # Disable all components except PgBouncer and Consul agent - roles ['pgbouncer_role'] + roles(['pgbouncer_role']) # Configure PgBouncer pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) @@ -348,7 +346,7 @@ If you enable Monitoring, it must be enabled on **all** database servers. 1. Ensure each node is talking to the current master: ```shell - gitlab-ctl pgb-console # You will be prompted for PGBOUNCER_PASSWORD + gitlab-ctl pgb-console # Supply PGBOUNCER_PASSWORD when prompted ``` If there is an error `psql: ERROR: Auth failed` after typing in the @@ -415,7 +413,7 @@ Refer to your preferred Load Balancer's documentation for further guidance. ### Configuring the Application nodes -These will be the nodes running the `gitlab-rails` service. You may have other +Application nodes run the `gitlab-rails` service. You may have other attributes set, but the following need to be set. 1. Edit `/etc/gitlab/gitlab.rb`: @@ -448,7 +446,7 @@ in the Troubleshooting section before proceeding. ### Backups -Do not backup or restore GitLab through a PgBouncer connection: this will cause a GitLab outage. +Do not backup or restore GitLab through a PgBouncer connection: this causes a GitLab outage. [Read more about this and how to reconfigure backups](../../raketasks/backup_restore.md#backup-and-restore-for-installations-using-pgbouncer). @@ -495,7 +493,7 @@ On each server edit `/etc/gitlab/gitlab.rb`: ```ruby # Disable all components except Consul -roles ['consul_role'] +roles(['consul_role']) consul['configuration'] = { server: true, @@ -512,7 +510,7 @@ On each server edit `/etc/gitlab/gitlab.rb`: ```ruby # Disable all components except Pgbouncer and Consul agent -roles ['pgbouncer_role'] +roles(['pgbouncer_role']) # Configure PgBouncer pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) @@ -527,7 +525,6 @@ pgbouncer['users'] = { } consul['watchers'] = %w(postgresql) -consul['enable'] = true consul['configuration'] = { retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } @@ -545,29 +542,26 @@ An internal load balancer (TCP) is then required to be setup to serve each PgBou On database nodes edit `/etc/gitlab/gitlab.rb`: ```ruby -# Disable all components except PostgreSQL, Patroni (or Repmgr), and Consul -roles ['postgres_role'] +# Disable all components except Patroni and Consul +roles(['patroni_role']) # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' postgresql['hot_standby'] = 'on' postgresql['wal_level'] = 'replica' -# Enable Patroni (which automatically disables Repmgr). -patroni['enable'] = true - # Disable automatic database migrations gitlab_rails['auto_migrate'] = false postgresql['pgbouncer_user_password'] = '771a8625958a529132abe6f1a4acb19c' postgresql['sql_user_password'] = '450409b85a0223a214b5fb1484f34d0f' -patroni['postgresql']['max_wal_senders'] = 4 +patroni['postgresql']['max_replication_slots'] = 6 +patroni['postgresql']['max_wal_senders'] = 7 postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/16) # Configure the Consul agent consul['services'] = %w(postgresql) -consul['enable'] = true consul['configuration'] = { retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } @@ -586,19 +580,6 @@ After deploying the configuration follow these steps: gitlab-ctl get-postgresql-primary ``` -1. On the primary database node: - - Enable the `pg_trgm` and `btree_gist` extensions: - - ```shell - gitlab-psql -d gitlabhq_production - ``` - - ```shell - CREATE EXTENSION pg_trgm; - CREATE EXTENSION btree_gist; - ``` - 1. On `10.6.0.41`, our application server: Set `gitlab-consul` user's PgBouncer password to `toomanysecrets`: @@ -640,17 +621,14 @@ Please note that after the initial configuration, if a failover occurs, the Post On database nodes edit `/etc/gitlab/gitlab.rb`: ```ruby -# Disable all components except PostgreSQL, Repmgr, and Consul -roles ['postgres_role'] +# Disable all components except Patroni and Consul +roles(['patroni_role']) # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' postgresql['hot_standby'] = 'on' postgresql['wal_level'] = 'replica' -# Enable Patroni (which automatically disables Repmgr). -patroni['enable'] = true - # Disable automatic database migrations gitlab_rails['auto_migrate'] = false @@ -659,7 +637,15 @@ consul['services'] = %w(postgresql) postgresql['pgbouncer_user_password'] = '771a8625958a529132abe6f1a4acb19c' postgresql['sql_user_password'] = '450409b85a0223a214b5fb1484f34d0f' -patroni['postgresql']['max_wal_senders'] = 4 + +# Sets `max_replication_slots` to double the number of database nodes. +# Patroni uses one extra slot per node when initiating the replication. +patroni['postgresql']['max_replication_slots'] = 6 + +# Set `max_wal_senders` to one more than the number of replication slots in the cluster. +# This is used to prevent replication from using up all of the +# available database connections. +patroni['postgresql']['max_wal_senders'] = 7 postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/16) @@ -716,22 +702,17 @@ The manual steps for this configuration are the same as for the [example recomme ## Patroni NOTE: -Using Patroni instead of Repmgr is supported for PostgreSQL 11 and required for PostgreSQL 12. +Using Patroni instead of Repmgr is supported for PostgreSQL 11 and required for PostgreSQL 12. Starting with GitLab 14.0, only PostgreSQL 12 is available and hence Patroni is mandatory to achieve failover and replication. Patroni is an opinionated solution for PostgreSQL high-availability. It takes the control of PostgreSQL, overrides its -configuration and manages its lifecycle (start, stop, restart). This is a more active approach when compared to repmgr. -Both repmgr and Patroni are both supported and available. But Patroni will be the default (and perhaps the only) option -for PostgreSQL 12 clustering and cascading replication for Geo deployments. +configuration and manages its lifecycle (start, stop, restart). Patroni is the only option for PostgreSQL 12 clustering and for cascading replication for Geo deployments. The [architecture](#example-recommended-setup-manual-steps) (that was mentioned above) does not change for Patroni. You do not need any special consideration for Patroni while provisioning your database nodes. Patroni heavily relies on Consul to store the state of the cluster and elect a leader. Any failure in Consul cluster and its leader election will propagate to Patroni cluster as well. -Similar to repmgr, Patroni monitors the cluster and handles failover. When the primary node fails it works with Consul -to notify PgBouncer. However, as opposed to repmgr, on failure, Patroni handles the transitioning of the old primary to -a replica and rejoins it to the cluster automatically. So you do not need any manual operation for recovering the -cluster as you do with repmgr. +Patroni monitors the cluster and handles failover. When the primary node fails it works with Consul to notify PgBouncer. On failure, Patroni handles the transitioning of the old primary to a replica and rejoins it to the cluster automatically. With Patroni the connection flow is slightly different. Patroni on each node connects to Consul agent to join the cluster. Only after this point it decides if the node is the primary or a replica. Based on this decision, it configures @@ -829,7 +810,7 @@ For further details on this subject, see the #### Geo secondary site considerations -Similar to `repmgr`, when a Geo secondary site is replicating from a primary site that uses `Patroni` and `PgBouncer`, [replicating through PgBouncer is not supported](https://github.com/pgbouncer/pgbouncer/issues/382#issuecomment-517911529) and the secondary must replicate directly from the leader node in the `Patroni` cluster. Therefore, when there is an automatic or manual failover in the `Patroni` cluster, you will need to manually re-point your secondary site to replicate from the new leader with: +When a Geo secondary site is replicating from a primary site that uses `Patroni` and `PgBouncer`, [replicating through PgBouncer is not supported](https://github.com/pgbouncer/pgbouncer/issues/382#issuecomment-517911529) and the secondary must replicate directly from the leader node in the `Patroni` cluster. Therefore, when there is an automatic or manual failover in the `Patroni` cluster, you will need to manually re-point your secondary site to replicate from the new leader with: ```shell sudo gitlab-ctl replicate-geo-database --host=<new_leader_ip> --replication-slot=<slot_name> @@ -891,8 +872,8 @@ You can switch an exiting database cluster to use Patroni instead of repmgr with 1. On the primary node, [configure Patroni](#configuring-patroni-cluster). Remove `repmgr` and any other repmgr-specific configuration. Also remove any configuration that is related to PostgreSQL replication. -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) on the primary node. It will become - the leader. You can check this with: +1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) on the primary node. + It makes it the leader. You can check this with: ```shell sudo gitlab-ctl tail patroni @@ -920,13 +901,13 @@ upgrading PostgreSQL. Here are a few key facts that you must consider before upgrading PostgreSQL: - The main point is that you will have to **shut down the Patroni cluster**. This means that your - GitLab deployment will be down for the duration of database upgrade or, at least, as long as your leader + GitLab deployment is down for the duration of database upgrade or, at least, as long as your leader node is upgraded. This can be **a significant downtime depending on the size of your database**. - Upgrading PostgreSQL creates a new data directory with a new control data. From Patroni's perspective this is a new cluster that needs to be bootstrapped again. Therefore, as part of the upgrade procedure, - the cluster state, which is stored in Consul, will be wiped out. Once the upgrade is completed, Patroni - will be instructed to bootstrap a new cluster. **Note that this will change your _cluster ID_**. + the cluster state (stored in Consul) is wiped out. Once the upgrade is completed, Patroni + bootstraps a new cluster. **Note that this changes your _cluster ID_**. - The procedures for upgrading leader and replicas are not the same. That is why it is important to use the right procedure on each node. @@ -996,389 +977,6 @@ Reverting PostgreSQL upgrade with `gitlab-ctl revert-pg-upgrade` has the same co `gitlab-ctl pg-upgrade`. You should follow the same procedure by first stopping the replicas, then reverting the leader, and finally reverting the replicas. -## Repmgr - -NOTE: -Using Patroni instead of Repmgr is supported for PostgreSQL 11 and required for PostgreSQL 12. - -### Configuring Repmgr Nodes - -1. On the master database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: - - ```ruby - # Disable all components except PostgreSQL and Repmgr and Consul - roles ['postgres_role'] - - # PostgreSQL configuration - postgresql['listen_address'] = '0.0.0.0' - postgresql['hot_standby'] = 'on' - postgresql['wal_level'] = 'replica' - postgresql['shared_preload_libraries'] = 'repmgr_funcs' - - # Disable automatic database migrations - gitlab_rails['auto_migrate'] = false - - # Configure the Consul agent - consul['services'] = %w(postgresql) - - # START user configuration - # Please set the real values as explained in Required Information section - # - # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value - postgresql['pgbouncer_user_password'] = 'PGBOUNCER_PASSWORD_HASH' - # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value - postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' - # Replace X with value of number of db nodes + 1 - postgresql['max_wal_senders'] = X - postgresql['max_replication_slots'] = X - - # Replace XXX.XXX.XXX.XXX/YY with Network Address - postgresql['trust_auth_cidr_addresses'] = %w(XXX.XXX.XXX.XXX/YY) - repmgr['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 XXX.XXX.XXX.XXX/YY) - - # Replace placeholders: - # - # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z - # with the addresses gathered for CONSUL_SERVER_NODES - consul['configuration'] = { - retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z) - } - # - # END user configuration - ``` - - > `postgres_role` was introduced with GitLab 10.3 - -1. On secondary nodes, add all the configuration specified above for primary node - to `/etc/gitlab/gitlab.rb`. In addition, append the following configuration - to inform `gitlab-ctl` that they are standby nodes initially and it need not - attempt to register them as primary node - - ```ruby - # Specify if a node should attempt to be master on initialization - repmgr['master_on_initialization'] = false - ``` - -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. [Enable Monitoring](#enable-monitoring) - -> Please note: -> -> - If you want your database to listen on a specific interface, change the configuration: -> `postgresql['listen_address'] = '0.0.0.0'`. -> - If your PgBouncer service runs under a different user account, -> you also need to specify: `postgresql['pgbouncer_user'] = PGBOUNCER_USERNAME` in -> your configuration. - -#### Database nodes post-configuration - -##### Primary node - -Select one node as a primary node. - -1. Open a database prompt: - - ```shell - gitlab-psql -d gitlabhq_production - ``` - -1. Enable the `pg_trgm` extension: - - ```shell - CREATE EXTENSION pg_trgm; - ``` - -1. Enable the `btree_gist` extension: - - ```shell - CREATE EXTENSION btree_gist; - ``` - -1. Exit the database prompt by typing `\q` and Enter. - -1. Verify the cluster is initialized with one node: - - ```shell - gitlab-ctl repmgr cluster show - ``` - - The output should be similar to the following: - - ```plaintext - Role | Name | Upstream | Connection String - ----------+----------|----------|---------------------------------------- - * master | HOSTNAME | | host=HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr - ``` - -1. Note down the hostname or IP address in the connection string: `host=HOSTNAME`. We will - refer to the hostname in the next section as `MASTER_NODE_NAME`. If the value - is not an IP address, it will need to be a resolvable name (via DNS or - `/etc/hosts`) - -##### Secondary nodes - -1. Set up the repmgr standby: - - ```shell - gitlab-ctl repmgr standby setup MASTER_NODE_NAME - ``` - - Do note that this will remove the existing data on the node. The command - has a wait time. - - The output should be similar to the following: - - ```console - # gitlab-ctl repmgr standby setup MASTER_NODE_NAME - Doing this will delete the entire contents of /var/opt/gitlab/postgresql/data - If this is not what you want, hit Ctrl-C now to exit - To skip waiting, rerun with the -w option - Sleeping for 30 seconds - Stopping the database - Removing the data - Cloning the data - Starting the database - Registering the node with the cluster - ok: run: repmgrd: (pid 19068) 0s - ``` - -1. Verify the node now appears in the cluster: - - ```shell - gitlab-ctl repmgr cluster show - ``` - - The output should be similar to the following: - - ```plaintext - Role | Name | Upstream | Connection String - ----------+---------|-----------|------------------------------------------------ - * master | MASTER | | host=MASTER_NODE_NAME user=gitlab_repmgr dbname=gitlab_repmgr - standby | STANDBY | MASTER | host=STANDBY_HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr - ``` - -Repeat the above steps on all secondary nodes. - -#### Database checkpoint - -Before moving on, make sure the databases are configured correctly. Run the -following command on the **primary** node to verify that replication is working -properly: - -```shell -gitlab-ctl repmgr cluster show -``` - -The output should be similar to: - -```plaintext -Role | Name | Upstream | Connection String -----------+--------------|--------------|-------------------------------------------------------------------- -* master | MASTER | | host=MASTER port=5432 user=gitlab_repmgr dbname=gitlab_repmgr - standby | STANDBY | MASTER | host=STANDBY port=5432 user=gitlab_repmgr dbname=gitlab_repmgr -``` - -If the 'Role' column for any node says "FAILED", check the -[Troubleshooting section](#troubleshooting) before proceeding. - -Also, check that the check master command works successfully on each node: - -```shell -su - gitlab-consul -gitlab-ctl repmgr-check-master || echo 'This node is a standby repmgr node' -``` - -This command relies on exit codes to tell Consul whether a particular node is a master -or secondary. The most important thing here is that this command does not produce errors. -If there are errors it's most likely due to incorrect `gitlab-consul` database user permissions. -Check the [Troubleshooting section](#troubleshooting) before proceeding. - -### Repmgr failover procedure - -By default, if the master database fails, `repmgrd` should promote one of the -standby nodes to master automatically, and Consul will update PgBouncer with -the new master. - -If you need to failover manually, you have two options: - -**Shutdown the current master database** - -Run: - -```shell -gitlab-ctl stop postgresql -``` - -The automated failover process will see this and failover to one of the -standby nodes. - -**Or perform a manual failover** - -1. Ensure the old master node is not still active. -1. Login to the server that should become the new master and run: - - ```shell - gitlab-ctl repmgr standby promote - ``` - -1. If there are any other standby servers in the cluster, have them follow - the new master server: - - ```shell - gitlab-ctl repmgr standby follow NEW_MASTER - ``` - -#### Geo secondary site considerations - -When a Geo secondary site is replicating from a primary site that uses `repmgr` and `PgBouncer`, [replicating through PgBouncer is not supported](https://github.com/pgbouncer/pgbouncer/issues/382#issuecomment-517911529) and the secondary must replicate directly from the leader node in the `repmgr` cluster. Therefore, when there is a failover in the `repmgr` cluster, you will need to manually re-point your secondary site to replicate from the new leader with: - -```shell -sudo gitlab-ctl replicate-geo-database --host=<new_leader_ip> --replication-slot=<slot_name> -``` - -Otherwise, the replication will not happen anymore, even if the original node gets re-added as a follower node. This will re-sync your secondary site database and may take a long time depending on the amount of data to sync. You may also need to run `gitlab-ctl reconfigure` if replication is still not working after re-syncing. - -### Repmgr Restore procedure - -If a node fails, it can be removed from the cluster, or added back as a standby -after it has been restored to service. - -#### Remove a standby from the cluster - - From any other node in the cluster, run: - - ```shell - gitlab-ctl repmgr standby unregister --node=X - ``` - - where X is the value of node in `repmgr.conf` on the old server. - - To find this, you can use: - - ```shell - awk -F = '$1 == "node" { print $2 }' /var/opt/gitlab/postgresql/repmgr.conf - ``` - - It will output something like: - - ```plaintext - 959789412 - ``` - - Then you will use this ID to unregister the node: - - ```shell - gitlab-ctl repmgr standby unregister --node=959789412 - ``` - -#### Add a node as a standby server - - From the standby node, run: - - ```shell - gitlab-ctl repmgr standby follow NEW_MASTER - gitlab-ctl restart repmgrd - ``` - - WARNING: - When the server is brought back online, and before - you switch it to a standby node, repmgr will report that there are two masters. - If there are any clients that are still attempting to write to the old master, - this will cause a split, and the old master will need to be resynced from - scratch by performing a `gitlab-ctl repmgr standby setup NEW_MASTER`. - -#### Add a failed master back into the cluster as a standby node - - Once `repmgrd` and PostgreSQL are running, the node will need to follow the new - as a standby node. - - ```shell - gitlab-ctl repmgr standby follow NEW_MASTER - ``` - - Once the node is following the new master as a standby, the node needs to be - [unregistered from the cluster on the new master node](#remove-a-standby-from-the-cluster). - - Once the old master node has been unregistered from the cluster, it will need - to be setup as a new standby: - - ```shell - gitlab-ctl repmgr standby setup NEW_MASTER - ``` - - Failure to unregister and read the old master node can lead to subsequent failovers - not working. - -### Alternate configurations - -#### Database authorization - -By default, we give any host on the database network the permission to perform -repmgr operations using PostgreSQL's `trust` method. If you do not want this -level of trust, there are alternatives. - -You can trust only the specific nodes that will be database clusters, or you -can require md5 authentication. - -#### Trust specific addresses - -If you know the IP address, or FQDN of all database and PgBouncer nodes in the -cluster, you can trust only those nodes. - -In `/etc/gitlab/gitlab.rb` on all of the database nodes, set -`repmgr['trust_auth_cidr_addresses']` to an array of strings containing all of -the addresses. - -If setting to a node's FQDN, they must have a corresponding PTR record in DNS. -If setting to a node's IP address, specify it as `XXX.XXX.XXX.XXX/32`. - -For example: - -```ruby -repmgr['trust_auth_cidr_addresses'] = %w(192.168.1.44/32 db2.example.com) -``` - -#### MD5 Authentication - -If you are running on an untrusted network, repmgr can use md5 authentication -with a [`.pgpass` file](https://www.postgresql.org/docs/11/libpq-pgpass.html) -to authenticate. - -You can specify by IP address, FQDN, or by subnet, using the same format as in -the previous section: - -1. On the current master node, create a password for the `gitlab` and - `gitlab_repmgr` user: - - ```shell - gitlab-psql -d template1 - template1=# \password gitlab_repmgr - Enter password: **** - Confirm password: **** - template1=# \password gitlab - ``` - -1. On each database node: - - 1. Edit `/etc/gitlab/gitlab.rb`: - 1. Ensure `repmgr['trust_auth_cidr_addresses']` is **not** set - 1. Set `postgresql['md5_auth_cidr_addresses']` to the desired value - 1. Set `postgresql['sql_replication_user'] = 'gitlab_repmgr'` - 1. Reconfigure with `gitlab-ctl reconfigure` - 1. Restart PostgreSQL with `gitlab-ctl restart postgresql` - - 1. Create a `.pgpass` file. Enter the `gitlab_repmgr` password twice to - when asked: - - ```shell - gitlab-ctl write-pgpass --user gitlab_repmgr --hostuser gitlab-psql --database '*' - ``` - -1. On each PgBouncer node, edit `/etc/gitlab/gitlab.rb`: - 1. Ensure `gitlab_rails['db_password']` is set to the plaintext password for - the `gitlab` database user - 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect - ## Troubleshooting ### Consul and PostgreSQL changes not taking effect @@ -1387,25 +985,10 @@ Due to the potential impacts, `gitlab-ctl reconfigure` only reloads Consul and P To restart either service, run `gitlab-ctl restart SERVICE` -For PostgreSQL, it is usually safe to restart the master node by default. Automatic failover defaults to a 1 minute timeout. Provided the database returns before then, nothing else needs to be done. To be safe, you can stop `repmgrd` on the standby nodes first with `gitlab-ctl stop repmgrd`, then start afterwards with `gitlab-ctl start repmgrd`. +For PostgreSQL, it is usually safe to restart the master node by default. Automatic failover defaults to a 1 minute timeout. Provided the database returns before then, nothing else needs to be done. On the Consul server nodes, it is important to [restart the Consul service](../consul.md#restart-consul) in a controlled manner. -### `gitlab-ctl repmgr-check-master` command produces errors - -If this command displays errors about database permissions it is likely that something failed during -install, resulting in the `gitlab-consul` database user getting incorrect permissions. Follow these -steps to fix the problem: - -1. On the master database node, connect to the database prompt - `gitlab-psql -d template1` -1. Delete the `gitlab-consul` user - `DROP USER "gitlab-consul";` -1. Exit the database prompt - `\q` -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) and the user will be re-added with the proper permissions. -1. Change to the `gitlab-consul` user - `su - gitlab-consul` -1. Try the check command again - `gitlab-ctl repmgr-check-master`. - -Now there should not be errors. If errors still occur then there is another problem. - ### PgBouncer error `ERROR: pgbouncer cannot connect to server` You may get this error when running `gitlab-rake gitlab:db:configure` or you diff --git a/doc/administration/postgresql/standalone.md b/doc/administration/postgresql/standalone.md index cca46a2ea8c..b21625acb56 100644 --- a/doc/administration/postgresql/standalone.md +++ b/doc/administration/postgresql/standalone.md @@ -17,8 +17,8 @@ together with Omnibus GitLab. This is recommended as part of our 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package you want using *steps 1 and 2* from the GitLab downloads page. - Do not complete any other steps on the download page. -1. Generate a password hash for PostgreSQL. This assumes you will use the default - username of `gitlab` (recommended). The command will request a password +1. Generate a password hash for PostgreSQL. This assumes you are using the default + username of `gitlab` (recommended). The command requests a password and confirmation. Use the value that is output by this command in the next step as the value of `POSTGRESQL_PASSWORD_HASH`. @@ -31,14 +31,12 @@ together with Omnibus GitLab. This is recommended as part of our - `POSTGRESQL_PASSWORD_HASH` - The value output from the previous step - `APPLICATION_SERVER_IP_BLOCKS` - A space delimited list of IP subnets or IP - addresses of the GitLab application servers that will connect to the + addresses of the GitLab application servers that connect to the database. Example: `%w(123.123.123.123/32 123.123.123.234/32)` ```ruby # Disable all components except PostgreSQL - roles ['postgres_role'] - repmgr['enable'] = false - consul['enable'] = false + roles(['postgres_role']) prometheus['enable'] = false alertmanager['enable'] = false pgbouncer_exporter['enable'] = false @@ -59,12 +57,9 @@ together with Omnibus GitLab. This is recommended as part of our gitlab_rails['auto_migrate'] = false ``` - NOTE: - The role `postgres_role` was introduced with GitLab 10.3 - 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Note the PostgreSQL node's IP address or hostname, port, and - plain text password. These will be necessary when configuring the GitLab + plain text password. These are necessary when configuring the GitLab application servers later. 1. [Enable monitoring](replication_and_failover.md#enable-monitoring) diff --git a/doc/administration/pseudonymizer.md b/doc/administration/pseudonymizer.md index 4aa73212e43..533ebe0ad2f 100644 --- a/doc/administration/pseudonymizer.md +++ b/doc/administration/pseudonymizer.md @@ -27,7 +27,7 @@ be textually exported. This ensures that: To configure the Pseudonymizer, you need to: - Provide a manifest file that describes which fields should be included or - pseudonymized ([example `manifest.yml` file](https://gitlab.com/gitlab-org/gitlab/tree/master/config/pseudonymizer.yml)). + pseudonymized ([example `manifest.yml` file](https://gitlab.com/gitlab-org/gitlab/-/tree/master/config/pseudonymizer.yml)). A default manifest is provided with the GitLab installation, using a relative file path that resolves from the Rails root. Alternatively, you can use an absolute file path. - Use an object storage and specify the connection parameters in the `pseudonymizer.upload.connection` configuration option. diff --git a/doc/administration/raketasks/check.md b/doc/administration/raketasks/check.md index 8d2ca103c82..7f344a00f72 100644 --- a/doc/administration/raketasks/check.md +++ b/doc/administration/raketasks/check.md @@ -34,10 +34,11 @@ exactly which repositories are causing the trouble. - Receiving an error when trying to push code - `remote: error: cannot lock ref` - A 500 error when viewing the GitLab dashboard or when accessing a specific project. -### Check all GitLab repositories +### Check project code repositories -This task loops through all repositories on the GitLab server and runs the -integrity check described previously. +This task loops through the project code repositories and runs the integrity check +described previously. If a project uses a pool repository, that will also be checked. +Other types of Git repositories [are not checked](https://gitlab.com/gitlab-org/gitaly/-/issues/3643). **Omnibus Installation** @@ -246,6 +247,41 @@ end p "#{uploads_deleted} remote objects were destroyed." ``` +### Delete references to missing artifacts + +`gitlab-rake gitlab:artifacts:check VERBOSE=1` detects when artifacts (or `job.log` files): + +- Are deleted outside of GitLab. +- Have references still in the GitLab database. + +When this scenario is detected, the Rake task displays an error message. For example: + +```shell +Checking integrity of Job artifacts +- 3..8: Failures: 2 + - Job artifact: 3: #<Errno::ENOENT: No such file or directory @ rb_sysopen - /var/opt/gitlab/gitlab-rails/shared/artifacts/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce/2021_05_26/5/3/job.log> + - Job artifact: 8: #<Errno::ENOENT: No such file or directory @ rb_sysopen - /var/opt/gitlab/gitlab-rails/shared/artifacts/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce/2021_05_26/6/8/job.log> +Done! + +``` + +To delete these references to missing local artifacts (`job.log` files): + +1. Open the [GitLab Rails Console](../operations/rails_console.md#starting-a-rails-console-session). +1. Run the following Ruby code: + + ```ruby + artifacts_deleted = 0 + ::Ci::JobArtifact.all.each do |artifact| ### Iterate artifacts + # next if artifact.file.filename != "job.log" ### Uncomment if only `job.log` files' references are to be processed + next if artifact.file.exists? ### Skip if the file reference is valid + artifacts_deleted += 1 + puts "#{artifact.id} #{artifact.file.path} is missing." ### Allow verification before destroy + # artifact.destroy! ### Uncomment to actually destroy + end + puts "Count of identified/destroyed invalid references: #{artifacts_deleted}" + ``` + ### Delete references to missing LFS objects If `gitlab-rake gitlab:lfs:check VERBOSE=1` detects LFS objects that exist in the database diff --git a/doc/administration/read_only_gitlab.md b/doc/administration/read_only_gitlab.md index 698da80a07c..c8931eb79df 100644 --- a/doc/administration/read_only_gitlab.md +++ b/doc/administration/read_only_gitlab.md @@ -19,10 +19,10 @@ The configuration for doing so depends on your desired outcome. The first thing you'll want to accomplish is to ensure that no changes can be made to your repositories. There's two ways you can accomplish that: -- Either stop Unicorn/Puma to make the internal API unreachable: +- Either stop Puma to make the internal API unreachable: ```shell - sudo gitlab-ctl stop puma # or unicorn + sudo gitlab-ctl stop puma ``` - Or, open up a Rails console: @@ -46,19 +46,19 @@ made to your repositories. There's two ways you can accomplish that: ## Shut down the GitLab UI If you don't mind shutting down the GitLab UI, then the easiest approach is to -stop `sidekiq` and `puma`/`unicorn`, and you'll effectively ensure that no +stop `sidekiq` and `puma`, and you'll effectively ensure that no changes can be made to GitLab: ```shell sudo gitlab-ctl stop sidekiq -sudo gitlab-ctl stop puma # or unicorn +sudo gitlab-ctl stop puma ``` When you're ready to revert this: ```shell sudo gitlab-ctl start sidekiq -sudo gitlab-ctl start puma # or unicorn +sudo gitlab-ctl start puma ``` ## Make the database read-only diff --git a/doc/administration/redis/replication_and_failover.md b/doc/administration/redis/replication_and_failover.md index 20a9fbd7d68..9fde91903e8 100644 --- a/doc/administration/redis/replication_and_failover.md +++ b/doc/administration/redis/replication_and_failover.md @@ -42,7 +42,7 @@ There should be no more than one Sentinel on the same machine though. You also need to take into consideration the underlying network topology, making sure you have redundant connectivity between Redis / Sentinel and -GitLab instances, otherwise the networks will become a single point of +GitLab instances, otherwise the networks become a single point of failure. Running Redis in a scaled environment requires a few things: @@ -73,7 +73,7 @@ whole cluster down, invalidating the failover effort. ## Recommended setup -For a minimal setup, you will install the Omnibus GitLab package in `3` +For a minimal setup, you need to install the Omnibus GitLab package in `3` **independent** machines, both with **Redis** and **Sentinel**: - Redis Primary + Sentinel @@ -84,7 +84,7 @@ If you are not sure or don't understand why and where the amount of nodes come from, read [Redis setup overview](#redis-setup-overview) and [Sentinel setup overview](#sentinel-setup-overview). -For a recommended setup that can resist more failures, you will install +For a recommended setup that can resist more failures, you need to install the Omnibus GitLab package in `5` **independent** machines, both with **Redis** and **Sentinel**: @@ -99,9 +99,9 @@ the Omnibus GitLab package in `5` **independent** machines, both with You must have at least `3` Redis servers: `1` primary, `2` Replicas, and they need to each be on independent machines (see explanation above). -You can have additional Redis nodes, that will help survive a situation +You can have additional Redis nodes, that helps to survive a situation where more nodes goes down. Whenever there is only `2` nodes online, a failover -will not be initiated. +is not initiated. As an example, if you have `6` Redis nodes, a maximum of `3` can be simultaneously down. @@ -117,7 +117,7 @@ in a failover situation, any **Replica** can be promoted as the new **Primary** the Sentinel servers. The replication requires authentication, so you need to define a password to -protect all Redis nodes and the Sentinels. They will all share the same +protect all Redis nodes and the Sentinels. All of them share the same password, and all instances must be able to talk to each other over the network. @@ -130,7 +130,7 @@ of Sentinels agreeing a node is down) to be able to start a failover. Whenever the **quorum** is met, the **majority** of all known Sentinel nodes need to be available and reachable, so that they can elect the Sentinel **leader** -who will take all the decisions to restore the service availability by: +who takes all the decisions to restore the service availability by: - Promoting a new **Primary** - Reconfiguring the other **Replicas** and make them point to the new **Primary** @@ -150,7 +150,7 @@ consensus algorithm to be effective in the case of a failure. In a `3` nodes topology, you can only afford `1` Sentinel node going down. Whenever the **majority** of the Sentinels goes down, the network partition -protection prevents destructive actions and a failover **will not be started**. +protection prevents destructive actions and a failover **is not started**. Here are some examples: @@ -159,11 +159,11 @@ Here are some examples: The **Leader** election can sometimes fail the voting round when **consensus** is not achieved (see the odd number of nodes requirement above). In that case, -a new attempt will be made after the amount of time defined in +a new attempt is made after the amount of time defined in `sentinel['failover_timeout']` (in milliseconds). NOTE: -We will see where `sentinel['failover_timeout']` is defined later. +We can see where `sentinel['failover_timeout']` is defined later. The `failover_timeout` variable has a lot of different use cases. According to the official documentation: @@ -183,7 +183,7 @@ the official documentation: - The maximum time a failover in progress waits for all the replicas to be reconfigured as replicas of the new primary. However even after this time - the replicas will be reconfigured by the Sentinels anyway, but not with + the replicas are reconfigured by the Sentinels anyway, but not with the exact parallel-syncs progression as specified. ## Configuring Redis @@ -195,7 +195,7 @@ If you already have Redis installed and running, read how to [switch from a single-machine installation](#switching-from-an-existing-single-machine-installation). NOTE: -Redis nodes (both primary and replica) will need the same password defined in +Redis nodes (both primary and replica) need the same password defined in `redis['password']`. At any time during a failover the Sentinels can reconfigure a node and change its status from primary to replica and vice versa. @@ -218,14 +218,14 @@ The requirements for a Redis setup are the following: ### Switching from an existing single-machine installation -If you already have a single-machine GitLab install running, you will need to +If you already have a single-machine GitLab install running, you need to replicate from this machine first, before de-activating the Redis instance inside it. -Your single-machine install will be the initial **Primary**, and the `3` others +Your single-machine install is the initial **Primary**, and the `3` others should be configured as **Replica** pointing to this machine. -After replication catches up, you will need to stop services in the +After replication catches up, you need to stop services in the single-machine install, to rotate the **Primary** to one of the new nodes. Make the required changes in configuration and restart the new nodes again. @@ -259,7 +259,7 @@ If you fail to replicate first, you may loose data (unprocessed background jobs) # sure you add extra firewall rules to prevent unauthorized access. redis['bind'] = '10.0.0.1' - # Define a port so Redis can listen for TCP requests which will allow other + # Define a port so Redis can listen for TCP requests which allows other # machines to connect to it. redis['port'] = 6379 @@ -303,7 +303,7 @@ Read more about [roles](https://docs.gitlab.com/omnibus/roles/). # sure you add extra firewall rules to prevent unauthorized access. redis['bind'] = '10.0.0.2' - # Define a port so Redis can listen for TCP requests which will allow other + # Define a port so Redis can listen for TCP requests which allows other # machines to connect to it. redis['port'] = 6379 @@ -333,8 +333,8 @@ You can specify multiple roles like sentinel and Redis as: Read more about [roles](https://docs.gitlab.com/omnibus/roles/). These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after -a failover, as the nodes will be managed by the Sentinels, and even after a -`gitlab-ctl reconfigure`, they will get their configuration restored by +a failover, as the nodes are managed by the Sentinels, and even after a +`gitlab-ctl reconfigure`, they get their configuration restored by the same Sentinels. ### Step 3. Configuring the Redis Sentinel instances @@ -342,7 +342,7 @@ the same Sentinels. NOTE: If you are using an external Redis Sentinel instance, be sure to exclude the `requirepass` parameter from the Sentinel -configuration. This parameter will cause clients to report `NOAUTH +configuration. This parameter causes clients to report `NOAUTH Authentication required.`. [Redis Sentinel 3.2.x does not support password authentication](https://github.com/antirez/redis/issues/3279). @@ -362,8 +362,8 @@ multiple machines with the Sentinel daemon. --- -1. SSH into the server that will host Redis Sentinel. -1. **You can omit this step if the Sentinels will be hosted in the same node as +1. SSH into the server that hosts Redis Sentinel. +1. **You can omit this step if the Sentinels is hosted in the same node as the other Redis instances.** [Download/install](https://about.gitlab.com/install/) the @@ -389,7 +389,7 @@ multiple machines with the Sentinel daemon. # The IP of the primary Redis node. redis['master_ip'] = '10.0.0.1' - # Define a port so Redis can listen for TCP requests which will allow other + # Define a port so Redis can listen for TCP requests which allows other # machines to connect to it. redis['port'] = 6379 @@ -437,7 +437,7 @@ multiple machines with the Sentinel daemon. ## ## - The maximum time a failover in progress waits for all the replica to be ## reconfigured as replicas of the new primary. However even after this time - ## the replicas will be reconfigured by the Sentinels anyway, but not with + ## the replicas are reconfigured by the Sentinels anyway, but not with ## the exact parallel-syncs progression as specified. # sentinel['failover_timeout'] = 60000 ``` @@ -511,7 +511,7 @@ If you enable Monitoring, it must be enabled on **all** Redis servers. retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z), } - # Set the network addresses that the exporters will listen on + # Set the network addresses that the exporters listen on node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' ``` @@ -528,7 +528,7 @@ In a real world usage, you would also set up firewall rules to prevent unauthorized access from other machines and block traffic from the outside (Internet). -We will use the same `3` nodes with **Redis** + **Sentinel** topology +We use the same `3` nodes with **Redis** + **Sentinel** topology discussed in [Redis setup overview](#redis-setup-overview) and [Sentinel setup overview](#sentinel-setup-overview) documentation. @@ -540,11 +540,11 @@ Here is a list and description of each **machine** and the assigned **IP**: - `10.0.0.4`: GitLab application Please note that after the initial configuration, if a failover is initiated -by the Sentinel nodes, the Redis nodes will be reconfigured and the **Primary** -will change permanently (including in `redis.conf`) from one node to the other, +by the Sentinel nodes, the Redis nodes are reconfigured and the **Primary** +changes permanently (including in `redis.conf`) from one node to the other, until a new failover is initiated again. -The same thing will happen with `sentinel.conf` that will be overridden after the +The same thing happens with `sentinel.conf` that is overridden after the initial execution, after any new sentinel node starts watching the **Primary**, or a failover promotes a different **Primary** node. @@ -691,7 +691,7 @@ To make this work with Sentinel: ``` NOTE: -For each persistence class, GitLab will default to using the +For each persistence class, GitLab defaults to using the configuration specified in `gitlab_rails['redis_sentinels']` unless overridden by the previously described settings. @@ -726,7 +726,7 @@ redis_replica_role['enable'] = true # enable only one of them # When Redis primary or Replica role are enabled, the following services are # enabled/disabled. Note that if Redis and Sentinel roles are combined, both -# services will be enabled. +# services are enabled. # The following services are disabled sentinel['enable'] = false diff --git a/doc/administration/redis/replication_and_failover_external.md b/doc/administration/redis/replication_and_failover_external.md index 2716d9bba37..141da2f79ec 100644 --- a/doc/administration/redis/replication_and_failover_external.md +++ b/doc/administration/redis/replication_and_failover_external.md @@ -21,7 +21,7 @@ The following are the requirements for providing your own Redis instance: [requirements page](../../install/requirements.md). - Standalone Redis or Redis high availability with Sentinel are supported. Redis Cluster is not supported. -- Managed Redis from cloud providers such as AWS ElastiCache will work. If these +- Managed Redis from cloud providers such as AWS ElastiCache works fine. If these services support high availability, be sure it is **not** the Redis Cluster type. Note the Redis node's IP address or hostname, port, and password (if required). @@ -53,13 +53,13 @@ Note the Redis node's IP address or hostname, port, and password (if required). This is the documentation for configuring a scalable Redis setup when you have installed Redis all by yourself and not using the bundled one that comes with the Omnibus packages, although using the Omnibus GitLab packages is -highly recommend as we optimize them specifically for GitLab, and we will take +highly recommend as we optimize them specifically for GitLab, and we take care of upgrading Redis to the latest supported version. Note also that you may elect to override all references to `/home/git/gitlab/config/resque.yml` in accordance with the advanced Redis settings outlined in -[Configuration Files Documentation](https://gitlab.com/gitlab-org/gitlab/blob/master/config/README.md). +[Configuration Files Documentation](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/README.md). We cannot stress enough the importance of reading the [replication and failover](replication_and_failover.md) documentation of the @@ -76,7 +76,7 @@ requirements: (e.g., one from an internal network). - Since Redis 3.2, you must define a password to receive external connections (`requirepass`). -- If you are using Redis with Sentinel, you will also need to define the same +- If you are using Redis with Sentinel, you also need to define the same password for the replica password definition (`masterauth`) in the same instance. In addition, read the prerequisites as described in the @@ -176,7 +176,7 @@ primary with IP `10.0.0.1` (some settings might overlap with the primary): sentinel monitor gitlab-redis 10.0.0.1 6379 2 ## Define with `sentinel down-after-milliseconds` the time in `ms` - ## that an unresponsive server will be considered down. + ## that an unresponsive server is considered down. sentinel down-after-milliseconds gitlab-redis 10000 ## Define a value for `sentinel failover_timeout` in `ms`. This has multiple @@ -197,7 +197,7 @@ primary with IP `10.0.0.1` (some settings might overlap with the primary): ## ## * The maximum time a failover in progress waits for all the replicas to be ## reconfigured as replicas of the new primary. However even after this time - ## the replicas will be reconfigured by the Sentinels anyway, but not with + ## the replicas are reconfigured by the Sentinels anyway, but not with ## the exact parallel-syncs progression as specified. sentinel failover_timeout 30000 ``` @@ -218,7 +218,7 @@ The following steps should be performed in the GitLab application server which ideally should not have Redis or Sentinels in the same machine: 1. Edit `/home/git/gitlab/config/resque.yml` following the example in - [resque.yml.example](https://gitlab.com/gitlab-org/gitlab/blob/master/config/resque.yml.example), and uncomment the Sentinel lines, pointing to + [resque.yml.example](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/resque.yml.example), and uncomment the Sentinel lines, pointing to the correct server credentials: ```yaml @@ -249,7 +249,7 @@ In a real world usage, you would also set up firewall rules to prevent unauthorized access from other machines, and block traffic from the outside ([Internet](https://gitlab.com/gitlab-org/gitlab-foss/uploads/c4cc8cd353604bd80315f9384035ff9e/The_Internet_IT_Crowd.png)). -For this example, **Sentinel 1** will be configured in the same machine as the +For this example, **Sentinel 1** is configured in the same machine as the **Redis Primary**, **Sentinel 2** and **Sentinel 3** in the same machines as the **Replica 1** and **Replica 2** respectively. @@ -261,11 +261,11 @@ Here is a list and description of each **machine** and the assigned **IP**: - `10.0.0.4`: GitLab application Please note that after the initial configuration, if a failover is initiated -by the Sentinel nodes, the Redis nodes will be reconfigured and the **Primary** -will change permanently (including in `redis.conf`) from one node to the other, +by the Sentinel nodes, the Redis nodes are reconfigured and the **Primary** +changes permanently (including in `redis.conf`) from one node to the other, until a new failover is initiated again. -The same thing will happen with `sentinel.conf` that will be overridden after the +The same thing happens with `sentinel.conf` that is overridden after the initial execution, after any new sentinel node starts watching the **Primary**, or a failover promotes a different **Primary** node. diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index e4a699b962c..4627b27a45e 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -17,29 +17,34 @@ full list of reference architectures, see | Service | Nodes | Configuration | GCP | AWS | Azure | |--------------------------------------------|-------------|-------------------------|------------------|--------------|-----------| -| External load balancing node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Consul* | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| PostgreSQL* | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | `D8s v3` | -| PgBouncer* | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Redis - Cache** | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis - Queues / Shared State** | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis Sentinel - Cache** | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | -| Redis Sentinel - Queues / Shared State** | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | +| External load balancing node(3) | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Consul(1) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| PostgreSQL(1) | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | `D8s v3` | +| PgBouncer(1) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Internal load balancing node(3) | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Redis - Cache(2) | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis - Queues / Shared State(2) | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis Sentinel - Cache(2) | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | +| Redis Sentinel - Queues / Shared State(2) | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | | Gitaly | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | `D16s v3` | | Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Praefect PostgreSQL* | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Praefect PostgreSQL(1) | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | GitLab Rails | 3 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` | | Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | -| Object storage | n/a | n/a | n/a | n/a | n/a | -| NFS server | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| Object storage(4) | n/a | n/a | n/a | n/a | n/a | +| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | + +<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> +<!-- markdownlint-disable MD029 --> +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. +4. Should be run on reputable third party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +<!-- markdownlint-enable MD029 --> NOTE: -Components marked with * can be optionally run on reputable -third party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work. -Components marked with ** can be optionally run on reputable -third party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +For all PaaS solutions that involve configuring instances, it is strongly recommended to implement a minimum of three nodes in three different availability zones to align with resilient cloud architecture practices. ```plantuml @startuml 10k @@ -157,7 +162,7 @@ To set up GitLab and its components to accommodate up to 10,000 users: provides access to the Git repositories. 1. [Configure Sidekiq](#configure-sidekiq). 1. [Configure the main GitLab Rails application](#configure-gitlab-rails) - to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend + to run Puma, Workhorse, GitLab Shell, and to serve all frontend requests (which include UI, API, and Git over HTTP/SSH). 1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab environment. @@ -411,11 +416,6 @@ The following IPs will be used as an example: - `10.6.0.12`: Consul 2 - `10.6.0.13`: Consul 3 -NOTE: -The configuration processes for the other servers in your reference architecture will -use the `/etc/gitlab/gitlab-secrets.json` file from your Consul server to connect -with the other servers. - To configure Consul: 1. SSH in to the server that will host Consul. @@ -426,10 +426,9 @@ To configure Consul: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - roles ['consul_role'] + roles(['consul_role']) ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -446,7 +445,11 @@ To configure Consul: gitlab_rails['auto_migrate'] = false ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Go through the steps again for all the other Consul nodes, and make sure you set up the correct IPs. @@ -538,6 +541,15 @@ in the second step, do not supply the `EXTERNAL_URL` value. sudo gitlab-ctl pg-password-md5 pgbouncer ``` +1. Generate a password hash for the PostgreSQL replication username/password pair. This assumes you will use the default + username of `gitlab_replicator` (recommended). The command will request a password + and a confirmation. Use the value that is output by this command in the next step + as the value of `<postgresql_replication_password_hash>`: + + ```shell + sudo gitlab-ctl pg-password-md5 gitlab_replicator + ``` + 1. Generate a password hash for the Consul database username/password pair. This assumes you will use the default username of `gitlab-consul` (recommended). The command will request a password and confirmation. Use the value that is output by this command in the next @@ -550,19 +562,21 @@ in the second step, do not supply the `EXTERNAL_URL` value. 1. On every database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: ```ruby - # Disable all components except PostgreSQL, Patroni, and Consul - roles ['postgres_role'] + # Disable all components except Patroni and Consul + roles(['patroni_role']) # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' - # Enable Patroni - patroni['enable'] = true - # Set `max_wal_senders` to one more than the number of database nodes in the cluster. + # Sets `max_replication_slots` to double the number of database nodes. + # Patroni uses one extra slot per node when initiating the replication. + patroni['postgresql']['max_replication_slots'] = 8 + + # Set `max_wal_senders` to one more than the number of replication slots in the cluster. # This is used to prevent replication from using up all of the # available database connections. - patroni['postgresql']['max_wal_senders'] = 4 - patroni['postgresql']['max_replication_slots'] = 4 + patroni['postgresql']['max_wal_senders'] = 9 + # Incoming recommended value for max connections is 500. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691. patroni['postgresql']['max_connections'] = 500 @@ -570,7 +584,6 @@ in the second step, do not supply the `EXTERNAL_URL` value. gitlab_rails['auto_migrate'] = false # Configure the Consul agent - consul['enable'] = true consul['services'] = %w(postgresql) ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -580,6 +593,8 @@ in the second step, do not supply the `EXTERNAL_URL` value. # # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value postgresql['pgbouncer_user_password'] = '<pgbouncer_password_hash>' + # Replace POSTGRESQL_REPLICATION_PASSWORD_HASH with a generated md5 value + postgresql['sql_replication_password'] = '<postgresql_replication_password_hash>' # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value postgresql['sql_user_password'] = '<postgresql_password_hash>' @@ -603,9 +618,8 @@ PostgreSQL, with Patroni managing its failover, will default to use `pg_rewind` Like most failover handling methods, this has a small chance of leading to data loss. Learn more about the various [Patroni replication methods](../postgresql/replication_and_failover.md#selecting-the-appropriate-patroni-replication-method). -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -620,21 +634,7 @@ are supported and can be added if needed. #### PostgreSQL post-configuration -SSH in to the **primary node**: - -1. Open a database prompt: - - ```shell - gitlab-psql -d gitlabhq_production - ``` - -1. Make sure the `pg_trgm` extension is enabled (it might already be): - - ```shell - CREATE EXTENSION pg_trgm; - ``` - -1. Exit the database prompt by typing `\q` and Enter. +SSH in to any of the Patroni nodes on the **primary site**: 1. Check the status of the leader and cluster: @@ -676,7 +676,7 @@ The following IPs will be used as an example: ```ruby # Disable all components except Pgbouncer and Consul agent - roles ['pgbouncer_role'] + roles(['pgbouncer_role']) # Configure PgBouncer pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) @@ -693,7 +693,6 @@ The following IPs will be used as an example: # Configure Consul agent consul['watchers'] = %w(postgresql) - consul['enable'] = true consul['configuration'] = { retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } @@ -705,9 +704,8 @@ The following IPs will be used as an example: node_exporter['listen_address'] = '0.0.0.0:9100' ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -826,8 +824,8 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' - roles ['redis_master_role'] + # Specify server role as 'redis_master_role' and enable Consul agent + roles(['redis_master_role', 'consul_role']) # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -849,7 +847,6 @@ a node and change its status from primary to replica (and vice versa). redis['maxmemory_samples'] = 5 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -861,19 +858,22 @@ a node and change its status from primary to replica (and vice versa). # Set the network addresses that the exporters will listen on node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' + redis_exporter['flags'] = { + 'redis.addr' => 'redis://10.6.0.51:6379', + 'redis.password' => 'redis-password-goes-here', + } # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. You can specify multiple roles, like sentinel and Redis, as: -`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about [roles](https://docs.gitlab.com/omnibus/roles/). #### Configure the replica Redis Cache nodes @@ -886,8 +886,8 @@ You can specify multiple roles, like sentinel and Redis, as: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_replica_role' - roles ['redis_replica_role'] + # Specify server role as 'redis_replica_role' and enable Consul agent + roles(['redis_replica_role', 'consul_role']) # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -916,7 +916,6 @@ You can specify multiple roles, like sentinel and Redis, as: redis['maxmemory_samples'] = 5 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -928,21 +927,24 @@ You can specify multiple roles, like sentinel and Redis, as: # Set the network addresses that the exporters will listen on node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' + redis_exporter['flags'] = { + 'redis.addr' => 'redis://10.6.0.52:6379', + 'redis.password' => 'redis-password-goes-here', + } # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. You can specify multiple roles, like sentinel and Redis, as: -`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about [roles](https://docs.gitlab.com/omnibus/roles/). These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after @@ -984,7 +986,7 @@ To configure the Sentinel Cache server: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - roles ['redis_sentinel_role'] + roles(['redis_sentinel_role', 'consul_role']) ## Must be the same in every sentinel node redis['master_name'] = 'gitlab-redis-cache' @@ -1048,7 +1050,6 @@ To configure the Sentinel Cache server: #sentinel['failover_timeout'] = 60000 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -1065,9 +1066,8 @@ To configure the Sentinel Cache server: gitlab_rails['auto_migrate'] = false ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Go through the steps again for all the other Consul/Sentinel nodes, and @@ -1097,8 +1097,8 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' - roles ['redis_master_role'] + # Specify server role as 'redis_master_role' and enable Consul agent + roles(['redis_master_role', 'consul_role']) # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -1114,7 +1114,6 @@ a node and change its status from primary to replica (and vice versa). redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -1131,14 +1130,13 @@ a node and change its status from primary to replica (and vice versa). gitlab_rails['auto_migrate'] = false ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. You can specify multiple roles, like sentinel and Redis, as: -`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about [roles](https://docs.gitlab.com/omnibus/roles/). #### Configure the replica Redis Queues nodes @@ -1151,8 +1149,8 @@ You can specify multiple roles, like sentinel and Redis, as: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_replica_role' - roles ['redis_replica_role'] + # Specify server role as 'redis_replica_role' and enable Consul agent + roles(['redis_replica_role', 'consul_role']) # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -1175,7 +1173,6 @@ You can specify multiple roles, like sentinel and Redis, as: #redis['master_port'] = 6379 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -1192,16 +1189,15 @@ You can specify multiple roles, like sentinel and Redis, as: gitlab_rails['auto_migrate'] = false ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. You can specify multiple roles, like sentinel and Redis, as: -`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about [roles](https://docs.gitlab.com/omnibus/roles/). These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after @@ -1243,7 +1239,7 @@ To configure the Sentinel Queues server: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - roles ['redis_sentinel_role'] + roles(['redis_sentinel_role', 'consul_role']) ## Must be the same in every sentinel node redis['master_name'] = 'gitlab-redis-persistent' @@ -1307,7 +1303,6 @@ To configure the Sentinel Queues server: #sentinel['failover_timeout'] = 60000 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -1324,17 +1319,8 @@ To configure the Sentinel Queues server: gitlab_rails['auto_migrate'] = false ``` -1. To prevent database migrations from running on upgrade, run: - - ```shell - sudo touch /etc/gitlab/skip-auto-reconfigure - ``` - - Only the primary GitLab application server should handle migrations. - -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Go through the steps again for all the other Sentinel nodes, and @@ -1397,9 +1383,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. ```ruby # Disable all components except PostgreSQL and Consul - roles ['postgres_role'] - repmgr['enable'] = false - patroni['enable'] = false + roles(['postgres_role', 'consul_role']) # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' @@ -1409,7 +1393,6 @@ in the second step, do not supply the `EXTERNAL_URL` value. gitlab_rails['auto_migrate'] = false # Configure the Consul agent - consul['enable'] = true ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -1435,7 +1418,11 @@ in the second step, do not supply the `EXTERNAL_URL` value. # END user configuration ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Follow the [post configuration](#praefect-postgresql-post-configuration). <div align="right"> @@ -1533,19 +1520,18 @@ To configure the Praefect nodes, on each one: 1. Edit the `/etc/gitlab/gitlab.rb` file to configure Praefect: ```ruby - # Avoid running unnecessary services on the Gitaly server + # Avoid running unnecessary services on the Praefect server + gitaly['enable'] = false postgresql['enable'] = false redis['enable'] = false - nginx['enable'] = false puma['enable'] = false - unicorn['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false - grafana['enable'] = false - - # If you run a separate monitoring node you can disable these services - alertmanager['enable'] = false prometheus['enable'] = false + alertmanager['enable'] = false + grafana['enable'] = false + gitlab_exporter['enable'] = false + nginx['enable'] = false # Praefect Configuration praefect['enable'] = true @@ -1583,19 +1569,20 @@ To configure the Praefect nodes, on each one: # server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1') praefect['virtual_storages'] = { 'default' => { - 'gitaly-1' => { - 'address' => 'tcp://10.6.0.91:8075', - 'token' => '<praefect_internal_token>', - 'primary' => true - }, - 'gitaly-2' => { - 'address' => 'tcp://10.6.0.92:8075', - 'token' => '<praefect_internal_token>' - }, - 'gitaly-3' => { - 'address' => 'tcp://10.6.0.93:8075', - 'token' => '<praefect_internal_token>' - }, + 'nodes' => { + 'gitaly-1' => { + 'address' => 'tcp://10.6.0.91:8075', + 'token' => '<praefect_internal_token>' + }, + 'gitaly-2' => { + 'address' => 'tcp://10.6.0.92:8075', + 'token' => '<praefect_internal_token>' + }, + 'gitaly-3' => { + 'address' => 'tcp://10.6.0.93:8075', + 'token' => '<praefect_internal_token>' + }, + } } } @@ -1612,11 +1599,25 @@ To configure the Praefect nodes, on each one: # END user configuration ``` - 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and - then replace the file of the same name on this server. If that file isn't on - this server, add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + +1. Praefect requires to run some database migrations, much like the main GitLab application. For this + you should select **one Praefect node only to run the migrations**, AKA the _Deploy Node_. This node + must be configured first before the others as follows: + + 1. In the `/etc/gitlab/gitlab.rb` file, change the `praefect['auto_migrate']` setting value from `false` to `true` + + 1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: + + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure + ``` + + 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect and + to run the Praefect database migrations. - 1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. On all other Praefect nodes, [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. ### Configure Gitaly @@ -1660,21 +1661,17 @@ On each node: storage paths, enable the network listener, and to configure the token: ```ruby - # /etc/gitlab/gitlab.rb - # Avoid running unnecessary services on the Gitaly server postgresql['enable'] = false redis['enable'] = false - nginx['enable'] = false puma['enable'] = false - unicorn['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false - grafana['enable'] = false - - # If you run a separate monitoring node you can disable these services - alertmanager['enable'] = false prometheus['enable'] = false + alertmanager['enable'] = false + grafana['enable'] = false + gitlab_exporter['enable'] = false + nginx['enable'] = false # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -1682,9 +1679,11 @@ On each node: # Configure the gitlab-shell API callback URL. Without this, `git push` will # fail. This can be your 'front door' GitLab URL or an internal load # balancer. - # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server. gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' + # Gitaly + gitaly['enable'] = true + # Make Gitaly accept connections on all network interfaces. You must use # firewalls to restrict access to this address/port. # Comment out following line if you only want to support TLS connections @@ -1726,9 +1725,8 @@ On each node: }) ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and - then replace the file of the same name on this server. If that file isn't on - this server, add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -1835,28 +1833,19 @@ To configure the Sidekiq nodes, on each one: 1. Open `/etc/gitlab/gitlab.rb` with your editor: ```ruby - ######################################## - ##### Services Disabled ### - ######################################## - - nginx['enable'] = false - grafana['enable'] = false - prometheus['enable'] = false - alertmanager['enable'] = false + # Avoid running unnecessary services on the Sidekiq server gitaly['enable'] = false - gitlab_workhorse['enable'] = false - nginx['enable'] = false - puma['enable'] = false - postgres_exporter['enable'] = false postgresql['enable'] = false redis['enable'] = false - redis_exporter['enable'] = false + puma['enable'] = false + gitlab_workhorse['enable'] = false + prometheus['enable'] = false + alertmanager['enable'] = false + grafana['enable'] = false gitlab_exporter['enable'] = false + nginx['enable'] = false - ######################################## - #### Redis ### - ######################################## - + # Redis ## Redis connection details ## First cluster that will host the cache gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' @@ -1888,13 +1877,10 @@ To configure the Sidekiq nodes, on each one: {host: '10.6.0.83', port: 26379}, ] - ####################################### - ### Gitaly ### - ####################################### - - # git_data_dirs get configured for the Praefect virtual storage - # Address is Internal Load Balancer for Praefect - # Token is praefect_external_token + # Gitaly Cluster + ## git_data_dirs get configured for the Praefect virtual storage + ## Address is Internal Load Balancer for Praefect + ## Token is praefect_external_token git_data_dirs({ "default" => { "gitaly_address" => "tcp://10.6.0.40:2305", # internal load balancer IP @@ -1902,20 +1888,17 @@ To configure the Sidekiq nodes, on each one: } }) - ####################################### - ### Postgres ### - ####################################### + # PostgreSQL gitlab_rails['db_host'] = '10.6.0.40' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '<postgresql_user_password>' gitlab_rails['db_adapter'] = 'postgresql' gitlab_rails['db_encoding'] = 'unicode' - # Prevent database migrations from running on upgrade automatically + ## Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false - ####################################### - ### Sidekiq configuration ### - ####################################### + # Sidekiq + sidekiqp['enable'] = true sidekiq['listen_address'] = "0.0.0.0" # Set number of Sidekiq queue processes to the same number as available CPUs @@ -1924,9 +1907,7 @@ To configure the Sidekiq nodes, on each one: # Set number of Sidekiq threads per queue process to the recommend number of 10 sidekiq['max_concurrency'] = 10 - ####################################### - ### Monitoring configuration ### - ####################################### + # Monitoring consul['enable'] = true consul['monitoring_service_discovery'] = true @@ -1934,18 +1915,15 @@ To configure the Sidekiq nodes, on each one: retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } - # Set the network addresses that the exporters will listen on + ## Set the network addresses that the exporters will listen on node_exporter['listen_address'] = '0.0.0.0:9100' - # Rails Status for prometheus + ## Add the monitoring node's IP address to the monitoring whitelist gitlab_rails['monitoring_whitelist'] = ['10.6.0.151/32', '127.0.0.0/8'] - ############################# - ### Object storage ### - ############################# - - # This is an example for configuring Object Storage on GCP - # Replace this config with your chosen Object Storage provider as desired + # Object Storage + ## This is an example for configuring Object Storage on GCP + ## Replace this config with your chosen Object Storage provider as desired gitlab_rails['object_store']['connection'] = { 'provider' => 'Google', 'google_project' => '<gcp-project-name>', @@ -1958,11 +1936,26 @@ To configure the Sidekiq nodes, on each one: gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-packages-bucket-name>" gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>" gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>" + + gitlab_rails['backup_upload_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } + gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" + ``` + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + +1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: + + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. + Only a single designated node should handle migrations as detailed in the + [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section. 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -1993,9 +1986,6 @@ On each node perform the following: 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to follow _only_ installation steps 1 and 2 on the page. -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. 1. Edit `/etc/gitlab/gitlab.rb` and use the following configuration. To maintain uniformity of links across nodes, the `external_url` @@ -2017,7 +2007,7 @@ On each node perform the following: }) ## Disable components that will not be on the GitLab application server - roles ['application_role'] + roles(['application_role']) gitaly['enable'] = false nginx['enable'] = true sidekiq['enable'] = false @@ -2090,9 +2080,15 @@ On each node perform the following: gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-packages-bucket-name>" gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>" gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>" + + gitlab_rails['backup_upload_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } + gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). 1. If you're using [Gitaly with TLS support](#gitaly-cluster-tls-support), make sure the `git_data_dirs` entry is configured with `tls` instead of `tcp`: @@ -2111,6 +2107,20 @@ On each node perform the following: sudo cp cert.pem /etc/gitlab/trusted-certs/ ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + +1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: + + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure + ``` + + Only a single designated node should handle migrations as detailed in the + [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section. + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. If you're [using NFS](#configure-nfs-optional): 1. If necessary, install the NFS client utility packages using the following commands: @@ -2150,7 +2160,8 @@ On each node perform the following: registry['gid'] = 9002 ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + 1. Confirm the node can connect to Gitaly: ```shell @@ -2214,55 +2225,34 @@ To configure the Monitoring node: 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to follow _only_ installation steps 1 and 2 on the page. -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - external_url 'http://gitlab.example.com' + roles(['monitoring_role', 'consul_role']) - # Disable all other services - alertmanager['enable'] = false - gitaly['enable'] = false - gitlab_exporter['enable'] = false - gitlab_workhorse['enable'] = false - nginx['enable'] = true - postgres_exporter['enable'] = false - postgresql['enable'] = false - redis['enable'] = false - redis_exporter['enable'] = false - sidekiq['enable'] = false - puma['enable'] = false - unicorn['enable'] = false - node_exporter['enable'] = false - gitlab_exporter['enable'] = false + external_url 'http://gitlab.example.com' - # Enable Prometheus - prometheus['enable'] = true + # Prometheus prometheus['listen_address'] = '0.0.0.0:9090' prometheus['monitor_kubernetes'] = false - # Enable Login form - grafana['disable_login_form'] = false - - # Enable Grafana - grafana['enable'] = true + # Grafana grafana['admin_password'] = '<grafana_password>' + grafana['disable_login_form'] = false # Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true consul['configuration'] = { retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } - # Prevent database migrations from running on upgrade automatically - gitlab_rails['auto_migrate'] = false + # Nginx - For Grafana access + nginx['enable'] = true ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. In the GitLab UI, set `admin/application_settings/metrics_and_profiling` > Metrics - Grafana to `/-/grafana` to `http[s]://<MONITOR NODE>/-/grafana` @@ -2395,35 +2385,46 @@ time use Google Cloud’s Kubernetes Engine (GKE) and associated machine types, and CPU requirements should translate to most other providers. We hope to update this in the future with further specific cloud provider details. -| Service | Nodes | Configuration | GCP | Allocatable CPUs and Memory | -|-------------------------------------------------------|-------|-------------------------|------------------|-----------------------------| -| Webservice | 4 | 32 vCPU, 28.8 GB memory | `n1-standard-32` | 127.5 vCPU, 118 GB memory | -| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | 15.5 vCPU, 50 GB memory | -| Supporting services such as NGINX, Prometheus, etc... | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | 7.75 vCPU, 25 GB memory | +| Service | Nodes(1) | Configuration | GCP | Allocatable CPUs and Memory | +|-------------------------------------------------------|----------|-------------------------|------------------|-----------------------------| +| Webservice | 4 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | 127.5 vCPU, 118 GB memory | +| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | 15.5 vCPU, 50 GB memory | +| Supporting services such as NGINX, Prometheus, etc. | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | 7.75 vCPU, 25 GB memory | + +<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> +<!-- markdownlint-disable MD029 --> +1. Nodes configuration is shown as it is forced to ensure pod vcpu / memory ratios and avoid scaling during **performance testing**. + In production deployments there is no need to assign pods to nodes. A minimum of three nodes in three different availability zones is strongly recommended to align with resilient cloud architecture practices. +<!-- markdownlint-enable MD029 --> Next are the backend components that run on static compute VMs via Omnibus (or External PaaS services where applicable): | Service | Nodes | Configuration | GCP | |--------------------------------------------|-------|-------------------------|------------------| -| Consul* | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| PostgreSQL* | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | -| PgBouncer* | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Redis - Cache** | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Redis - Queues / Shared State** | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Redis Sentinel - Cache** | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | -| Redis Sentinel - Queues / Shared State** | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | +| Consul(1) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| PostgreSQL(1) | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | +| PgBouncer(1) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| Internal load balancing node(3) | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| Redis - Cache(2) | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | +| Redis - Queues / Shared State(2) | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | +| Redis Sentinel - Cache(2) | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | +| Redis Sentinel - Queues / Shared State(2) | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | | Gitaly | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | | Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Praefect PostgreSQL* | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Object storage | n/a | n/a | n/a | +| Praefect PostgreSQL(1) | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| Object storage(4) | n/a | n/a | n/a | + +<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> +<!-- markdownlint-disable MD029 --> +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. +4. Should be run on reputable third party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +<!-- markdownlint-enable MD029 --> NOTE: -Components marked with * can be optionally run on reputable -third party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work. -Components marked with ** can be optionally run on reputable -third party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +For all PaaS solutions that involve configuring instances, it is strongly recommended to implement a minimum of three nodes in three different availability zones to align with resilient cloud architecture practices. ```plantuml @startuml 10k diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index 129386d6ce5..1f72c45c2b7 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -17,29 +17,34 @@ full list of reference architectures, see | Service | Nodes | Configuration | GCP | AWS | Azure | |------------------------------------------|-------------|-------------------------|------------------|--------------|-----------| -| External load balancing node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | -| Consul* | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| PostgreSQL* | 3 | 16 vCPU, 60 GB memory | `n1-standard-1` | `m5.4xlarge` | `D16s v3` | -| PgBouncer* | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Internal load balancing node | 1 | 4 vCPU, 3.6GB memory | `n1-highcpu-4` | `c5.large` | `F2s v2` | -| Redis - Cache** | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis - Queues / Shared State** | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis Sentinel - Cache** | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | -| Redis Sentinel - Queues / Shared State** | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | +| External load balancing node(3) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| Consul(1) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| PostgreSQL(1) | 3 | 16 vCPU, 60 GB memory | `n1-standard-1` | `m5.4xlarge` | `D16s v3` | +| PgBouncer(1) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Internal load balancing node(3) | 1 | 4 vCPU, 3.6GB memory | `n1-highcpu-4` | `c5.large` | `F2s v2` | +| Redis - Cache(2) | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis - Queues / Shared State(2) | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis Sentinel - Cache(2) | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | +| Redis Sentinel - Queues / Shared State(2)| 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | | Gitaly | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | `D32s v3` | | Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | -| Praefect PostgreSQL* | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Praefect PostgreSQL(1) | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | GitLab Rails | 5 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` | | Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | -| Object storage | n/a | n/a | n/a | n/a | n/a | -| NFS server | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| Object storage(4) | n/a | n/a | n/a | n/a | n/a | +| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | + +<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> +<!-- markdownlint-disable MD029 --> +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. +4. Should be run on reputable third party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +<!-- markdownlint-enable MD029 --> NOTE: -Components marked with * can be optionally run on reputable -third party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work. -Components marked with ** can be optionally run on reputable -third party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +For all PaaS solutions that involve configuring instances, it is strongly recommended to implement a minimum of three nodes in three different availability zones to align with resilient cloud architecture practices. ```plantuml @startuml 25k @@ -157,7 +162,7 @@ To set up GitLab and its components to accommodate up to 25,000 users: provides access to the Git repositories. 1. [Configure Sidekiq](#configure-sidekiq). 1. [Configure the main GitLab Rails application](#configure-gitlab-rails) - to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend + to run Puma, Workhorse, GitLab Shell, and to serve all frontend requests (which include UI, API, and Git over HTTP/SSH). 1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab environment. @@ -413,11 +418,6 @@ The following IPs will be used as an example: - `10.6.0.12`: Consul 2 - `10.6.0.13`: Consul 3 -NOTE: -The configuration processes for the other servers in your reference architecture will -use the `/etc/gitlab/gitlab-secrets.json` file from your Consul server to connect -with the other servers. - To configure Consul: 1. SSH in to the server that will host Consul. @@ -428,10 +428,9 @@ To configure Consul: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - roles ['consul_role'] + roles(['consul_role']) ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -448,7 +447,11 @@ To configure Consul: gitlab_rails['auto_migrate'] = false ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Go through the steps again for all the other Consul nodes, and make sure you set up the correct IPs. @@ -540,6 +543,15 @@ in the second step, do not supply the `EXTERNAL_URL` value. sudo gitlab-ctl pg-password-md5 pgbouncer ``` +1. Generate a password hash for the PostgreSQL replication username/password pair. This assumes you will use the default + username of `gitlab_replicator` (recommended). The command will request a password + and a confirmation. Use the value that is output by this command in the next step + as the value of `<postgresql_replication_password_hash>`: + + ```shell + sudo gitlab-ctl pg-password-md5 gitlab_replicator + ``` + 1. Generate a password hash for the Consul database username/password pair. This assumes you will use the default username of `gitlab-consul` (recommended). The command will request a password and confirmation. Use the value that is output by this command in the next @@ -552,19 +564,21 @@ in the second step, do not supply the `EXTERNAL_URL` value. 1. On every database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: ```ruby - # Disable all components except PostgreSQL, Patroni, and Consul - roles ['postgres_role'] + # Disable all components except Patroni and Consul + roles(['patroni_role']) # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' - # Enable Patroni - patroni['enable'] = true - # Set `max_wal_senders` to one more than the number of database nodes in the cluster. + # Sets `max_replication_slots` to double the number of database nodes. + # Patroni uses one extra slot per node when initiating the replication. + patroni['postgresql']['max_replication_slots'] = 8 + + # Set `max_wal_senders` to one more than the number of replication slots in the cluster. # This is used to prevent replication from using up all of the # available database connections. - patroni['postgresql']['max_wal_senders'] = 4 - patroni['postgresql']['max_replication_slots'] = 4 + patroni['postgresql']['max_wal_senders'] = 9 + # Incoming recommended value for max connections is 500. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691. patroni['postgresql']['max_connections'] = 500 @@ -572,7 +586,6 @@ in the second step, do not supply the `EXTERNAL_URL` value. gitlab_rails['auto_migrate'] = false # Configure the Consul agent - consul['enable'] = true consul['services'] = %w(postgresql) ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -582,6 +595,8 @@ in the second step, do not supply the `EXTERNAL_URL` value. # # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value postgresql['pgbouncer_user_password'] = '<pgbouncer_password_hash>' + # Replace POSTGRESQL_REPLICATION_PASSWORD_HASH with a generated md5 value + postgresql['sql_replication_password'] = '<postgresql_replication_password_hash>' # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value postgresql['sql_user_password'] = '<postgresql_password_hash>' @@ -605,9 +620,8 @@ PostgreSQL, with Patroni managing its failover, will default to use `pg_rewind` Like most failover handling methods, this has a small chance of leading to data loss. Learn more about the various [Patroni replication methods](../postgresql/replication_and_failover.md#selecting-the-appropriate-patroni-replication-method). -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -622,21 +636,7 @@ are supported and can be added if needed. #### PostgreSQL post-configuration -SSH in to the **primary node**: - -1. Open a database prompt: - - ```shell - gitlab-psql -d gitlabhq_production - ``` - -1. Make sure the `pg_trgm` extension is enabled (it might already be): - - ```shell - CREATE EXTENSION pg_trgm; - ``` - -1. Exit the database prompt by typing `\q` and Enter. +SSH in to any of the Patroni nodes on the **primary site**: 1. Check the status of the leader and cluster: @@ -678,7 +678,7 @@ The following IPs will be used as an example: ```ruby # Disable all components except Pgbouncer and Consul agent - roles ['pgbouncer_role'] + roles(['pgbouncer_role']) # Configure PgBouncer pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) @@ -695,7 +695,6 @@ The following IPs will be used as an example: # Configure Consul agent consul['watchers'] = %w(postgresql) - consul['enable'] = true consul['configuration'] = { retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } @@ -707,9 +706,8 @@ The following IPs will be used as an example: node_exporter['listen_address'] = '0.0.0.0:9100' ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -828,8 +826,8 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' - roles ['redis_master_role'] + # Specify server role as 'redis_master_role' and enable Consul agent + roles(['redis_master_role', 'consul_role'] # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -851,7 +849,6 @@ a node and change its status from primary to replica (and vice versa). redis['maxmemory_samples'] = 5 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -863,19 +860,22 @@ a node and change its status from primary to replica (and vice versa). # Set the network addresses that the exporters will listen on node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' + redis_exporter['flags'] = { + 'redis.addr' => 'redis://10.6.0.51:6379', + 'redis.password' => 'redis-password-goes-here', + } # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. You can specify multiple roles, like sentinel and Redis, as: -`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about [roles](https://docs.gitlab.com/omnibus/roles/). #### Configure the replica Redis Cache nodes @@ -888,8 +888,8 @@ You can specify multiple roles, like sentinel and Redis, as: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_replica_role' - roles ['redis_replica_role'] + # Specify server role as 'redis_replica_role' and enable Consul agent + roles(['redis_replica_role', 'consul_role'] # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -918,7 +918,6 @@ You can specify multiple roles, like sentinel and Redis, as: redis['maxmemory_samples'] = 5 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -930,21 +929,25 @@ You can specify multiple roles, like sentinel and Redis, as: # Set the network addresses that the exporters will listen on node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' + redis_exporter['flags'] = { + 'redis.addr' => 'redis://10.6.0.52:6379', + 'redis.password' => 'redis-password-goes-here', + } # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. You can specify multiple roles, like sentinel and Redis, as: -`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about [roles](https://docs.gitlab.com/omnibus/roles/). These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after @@ -986,7 +989,7 @@ To configure the Sentinel Cache server: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - roles ['redis_sentinel_role'] + roles(['redis_sentinel_role', 'consul_role']) ## Must be the same in every sentinel node redis['master_name'] = 'gitlab-redis-cache' @@ -1050,7 +1053,6 @@ To configure the Sentinel Cache server: #sentinel['failover_timeout'] = 60000 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -1067,11 +1069,11 @@ To configure the Sentinel Cache server: gitlab_rails['auto_migrate'] = false ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Go through the steps again for all the other Consul/Sentinel nodes, and make sure you set up the correct IPs. @@ -1099,8 +1101,8 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' - roles ['redis_master_role'] + # Specify server role as 'redis_master_role' and enable Consul agent + roles(['redis_master_role', 'consul_role']) # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -1116,7 +1118,6 @@ a node and change its status from primary to replica (and vice versa). redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -1133,14 +1134,13 @@ a node and change its status from primary to replica (and vice versa). gitlab_rails['auto_migrate'] = false ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. You can specify multiple roles, like sentinel and Redis, as: -`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about [roles](https://docs.gitlab.com/omnibus/roles/). #### Configure the replica Redis Queues nodes @@ -1153,8 +1153,8 @@ You can specify multiple roles, like sentinel and Redis, as: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_replica_role' - roles ['redis_replica_role'] + # Specify server role as 'redis_replica_role' and enable Consul agent + roles(['redis_replica_role', 'consul_role']) # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -1177,7 +1177,6 @@ You can specify multiple roles, like sentinel and Redis, as: #redis['master_port'] = 6379 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -1189,21 +1188,25 @@ You can specify multiple roles, like sentinel and Redis, as: # Set the network addresses that the exporters will listen on node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' + redis_exporter['flags'] = { + 'redis.addr' => 'redis://10.6.0.62:6379', + 'redis.password' => 'redis-password-goes-here', + } # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. You can specify multiple roles, like sentinel and Redis, as: -`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about [roles](https://docs.gitlab.com/omnibus/roles/). These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after @@ -1245,7 +1248,7 @@ To configure the Sentinel Queues server: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - roles ['redis_sentinel_role'] + roles(['redis_sentinel_role', 'consul_role']) ## Must be the same in every sentinel node redis['master_name'] = 'gitlab-redis-persistent' @@ -1309,7 +1312,6 @@ To configure the Sentinel Queues server: #sentinel['failover_timeout'] = 60000 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -1326,7 +1328,10 @@ To configure the Sentinel Queues server: gitlab_rails['auto_migrate'] = false ``` -1. To prevent database migrations from running on upgrade, run: +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + +1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: ```shell sudo touch /etc/gitlab/skip-auto-reconfigure @@ -1334,11 +1339,8 @@ To configure the Sentinel Queues server: Only the primary GitLab application server should handle migrations. -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. - 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Go through the steps again for all the other Sentinel nodes, and make sure you set up the correct IPs. @@ -1399,9 +1401,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. ```ruby # Disable all components except PostgreSQL and Consul - roles ['postgres_role'] - repmgr['enable'] = false - patroni['enable'] = false + roles(['postgres_role', 'consul_role']) # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' @@ -1411,7 +1411,6 @@ in the second step, do not supply the `EXTERNAL_URL` value. gitlab_rails['auto_migrate'] = false # Configure the Consul agent - consul['enable'] = true ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -1437,7 +1436,11 @@ in the second step, do not supply the `EXTERNAL_URL` value. # END user configuration ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Follow the [post configuration](#praefect-postgresql-post-configuration). <div align="right"> @@ -1535,19 +1538,18 @@ To configure the Praefect nodes, on each one: 1. Edit the `/etc/gitlab/gitlab.rb` file to configure Praefect: ```ruby - # Avoid running unnecessary services on the Gitaly server + # Avoid running unnecessary services on the Praefect server + gitaly['enable'] = false postgresql['enable'] = false redis['enable'] = false - nginx['enable'] = false puma['enable'] = false - unicorn['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false - grafana['enable'] = false - - # If you run a separate monitoring node you can disable these services - alertmanager['enable'] = false prometheus['enable'] = false + alertmanager['enable'] = false + grafana['enable'] = false + gitlab_exporter['enable'] = false + nginx['enable'] = false # Praefect Configuration praefect['enable'] = true @@ -1585,19 +1587,20 @@ To configure the Praefect nodes, on each one: # server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1') praefect['virtual_storages'] = { 'default' => { - 'gitaly-1' => { - 'address' => 'tcp://10.6.0.91:8075', - 'token' => '<praefect_internal_token>', - 'primary' => true - }, - 'gitaly-2' => { - 'address' => 'tcp://10.6.0.92:8075', - 'token' => '<praefect_internal_token>' - }, - 'gitaly-3' => { - 'address' => 'tcp://10.6.0.93:8075', - 'token' => '<praefect_internal_token>' - }, + 'nodes' => { + 'gitaly-1' => { + 'address' => 'tcp://10.6.0.91:8075', + 'token' => '<praefect_internal_token>' + }, + 'gitaly-2' => { + 'address' => 'tcp://10.6.0.92:8075', + 'token' => '<praefect_internal_token>' + }, + 'gitaly-3' => { + 'address' => 'tcp://10.6.0.93:8075', + 'token' => '<praefect_internal_token>' + }, + } } } @@ -1614,11 +1617,25 @@ To configure the Praefect nodes, on each one: # END user configuration ``` - 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and - then replace the file of the same name on this server. If that file isn't on - this server, add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace +the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + +1. Praefect requires to run some database migrations, much like the main GitLab application. For this + you should select **one Praefect node only to run the migrations**, AKA the _Deploy Node_. This node + must be configured first before the others as follows: + + 1. In the `/etc/gitlab/gitlab.rb` file, change the `praefect['auto_migrate']` setting value from `false` to `true` + + 1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: - 1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure + ``` + + 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect and + to run the Praefect database migrations. + +1. On all other Praefect nodes, [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. ### Configure Gitaly @@ -1662,21 +1679,17 @@ On each node: storage paths, enable the network listener, and to configure the token: ```ruby - # /etc/gitlab/gitlab.rb - # Avoid running unnecessary services on the Gitaly server postgresql['enable'] = false redis['enable'] = false - nginx['enable'] = false puma['enable'] = false - unicorn['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false - grafana['enable'] = false - - # If you run a separate monitoring node you can disable these services - alertmanager['enable'] = false prometheus['enable'] = false + alertmanager['enable'] = false + grafana['enable'] = false + gitlab_exporter['enable'] = false + nginx['enable'] = false # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -1684,9 +1697,11 @@ On each node: # Configure the gitlab-shell API callback URL. Without this, `git push` will # fail. This can be your 'front door' GitLab URL or an internal load # balancer. - # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server. gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' + # Gitaly + gitaly['enable'] = true + # Make Gitaly accept connections on all network interfaces. You must use # firewalls to restrict access to this address/port. # Comment out following line if you only want to support TLS connections @@ -1728,9 +1743,8 @@ On each node: }) ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and - then replace the file of the same name on this server. If that file isn't on - this server, add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -1837,28 +1851,19 @@ To configure the Sidekiq nodes, on each one: 1. Open `/etc/gitlab/gitlab.rb` with your editor: ```ruby - ######################################## - ##### Services Disabled ### - ######################################## - - nginx['enable'] = false - grafana['enable'] = false - prometheus['enable'] = false - alertmanager['enable'] = false + # Avoid running unnecessary services on the Sidekiq server gitaly['enable'] = false - gitlab_workhorse['enable'] = false - nginx['enable'] = false - puma['enable'] = false - postgres_exporter['enable'] = false postgresql['enable'] = false redis['enable'] = false - redis_exporter['enable'] = false + puma['enable'] = false + gitlab_workhorse['enable'] = false + prometheus['enable'] = false + alertmanager['enable'] = false + grafana['enable'] = false gitlab_exporter['enable'] = false + nginx['enable'] = false - ######################################## - #### Redis ### - ######################################## - + # Redis ## Redis connection details ## First cluster that will host the cache gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' @@ -1890,13 +1895,10 @@ To configure the Sidekiq nodes, on each one: {host: '10.6.0.83', port: 26379}, ] - ####################################### - ### Gitaly ### - ####################################### - - # git_data_dirs get configured for the Praefect virtual storage - # Address is Internal Load Balancer for Praefect - # Token is praefect_external_token + # Gitaly Cluster + ## git_data_dirs get configured for the Praefect virtual storage + ## Address is Internal Load Balancer for Praefect + ## Token is praefect_external_token git_data_dirs({ "default" => { "gitaly_address" => "tcp://10.6.0.40:2305", # internal load balancer IP @@ -1904,20 +1906,17 @@ To configure the Sidekiq nodes, on each one: } }) - ####################################### - ### Postgres ### - ####################################### + # PostgreSQL gitlab_rails['db_host'] = '10.6.0.20' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '<postgresql_user_password>' gitlab_rails['db_adapter'] = 'postgresql' gitlab_rails['db_encoding'] = 'unicode' - # Prevent database migrations from running on upgrade automatically + ## Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false - ####################################### - ### Sidekiq configuration ### - ####################################### + # Sidekiq + sidekiq['enable'] = true sidekiq['listen_address'] = "0.0.0.0" # Set number of Sidekiq queue processes to the same number as available CPUs @@ -1926,9 +1925,7 @@ To configure the Sidekiq nodes, on each one: # Set number of Sidekiq threads per queue process to the recommend number of 10 sidekiq['max_concurrency'] = 10 - ####################################### - ### Monitoring configuration ### - ####################################### + # Monitoring consul['enable'] = true consul['monitoring_service_discovery'] = true @@ -1936,16 +1933,13 @@ To configure the Sidekiq nodes, on each one: retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } - # Set the network addresses that the exporters will listen on + ## Set the network addresses that the exporters will listen on node_exporter['listen_address'] = '0.0.0.0:9100' - # Rails Status for prometheus + ## Add the monitoring node's IP address to the monitoring whitelist gitlab_rails['monitoring_whitelist'] = ['10.6.0.151/32', '127.0.0.0/8'] - ############################# - ### Object storage ### - ############################# - + # Object Storage # This is an example for configuring Object Storage on GCP # Replace this config with your chosen Object Storage provider as desired gitlab_rails['object_store']['connection'] = { @@ -1960,11 +1954,26 @@ To configure the Sidekiq nodes, on each one: gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-packages-bucket-name>" gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>" gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>" + + gitlab_rails['backup_upload_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } + gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" + ``` + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + +1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: + + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. + Only a single designated node should handle migrations as detailed in the + [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section. 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -1997,9 +2006,6 @@ On each node perform the following: 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to follow _only_ installation steps 1 and 2 on the page. -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. 1. Edit `/etc/gitlab/gitlab.rb` and use the following configuration. To maintain uniformity of links across nodes, the `external_url` @@ -2021,7 +2027,7 @@ On each node perform the following: }) ## Disable components that will not be on the GitLab application server - roles ['application_role'] + roles(['application_role']) gitaly['enable'] = false nginx['enable'] = true sidekiq['enable'] = false @@ -2094,9 +2100,15 @@ On each node perform the following: gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-packages-bucket-name>" gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>" gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>" + + gitlab_rails['backup_upload_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } + gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). 1. If you're using [Gitaly with TLS support](#gitaly-cluster-tls-support), make sure the `git_data_dirs` entry is configured with `tls` instead of `tcp`: @@ -2115,6 +2127,20 @@ On each node perform the following: sudo cp cert.pem /etc/gitlab/trusted-certs/ ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + +1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: + + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure + ``` + + Only a single designated node should handle migrations as detailed in the + [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section. + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. If you're [using NFS](#configure-nfs-optional): 1. If necessary, install the NFS client utility packages using the following commands: @@ -2154,7 +2180,7 @@ On each node perform the following: registry['gid'] = 9002 ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). 1. Confirm the node can connect to Gitaly: ```shell @@ -2218,52 +2244,30 @@ To configure the Monitoring node: 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to follow _only_ installation steps 1 and 2 on the page. -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - external_url 'http://gitlab.example.com' + roles(['monitoring_role', 'consul_role']) - # Disable all other services - alertmanager['enable'] = false - gitaly['enable'] = false - gitlab_exporter['enable'] = false - gitlab_workhorse['enable'] = false - nginx['enable'] = true - postgres_exporter['enable'] = false - postgresql['enable'] = false - redis['enable'] = false - redis_exporter['enable'] = false - sidekiq['enable'] = false - puma['enable'] = false - unicorn['enable'] = false - node_exporter['enable'] = false - gitlab_exporter['enable'] = false + external_url 'http://gitlab.example.com' - # Enable Prometheus - prometheus['enable'] = true + # Prometheus prometheus['listen_address'] = '0.0.0.0:9090' prometheus['monitor_kubernetes'] = false - # Enable Login form - grafana['disable_login_form'] = false - - # Enable Grafana - grafana['enable'] = true + # Grafana grafana['admin_password'] = '<grafana_password>' + grafana['disable_login_form'] = false # Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true consul['configuration'] = { retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } - # Prevent database migrations from running on upgrade automatically - gitlab_rails['auto_migrate'] = false + # Nginx - For Grafana access + nginx['enable'] = true ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index 69e261cfbe6..7db3a343e0b 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -18,20 +18,24 @@ For a full list of reference architectures, see | Service | Nodes | Configuration | GCP | AWS | Azure | |------------------------------------------|--------|-------------------------|-----------------|--------------|----------| -| Load balancer | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| PostgreSQL* | 1 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | -| Redis** | 1 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `m5.large` | `D2s v3` | +| Load balancer(3) | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| PostgreSQL(1) | 1 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | +| Redis(2) | 1 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `m5.large` | `D2s v3` | | Gitaly | 1 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | GitLab Rails | 2 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | | Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Object storage | n/a | n/a | n/a | n/a | n/a | +| Object storage(4) | n/a | n/a | n/a | n/a | n/a | | NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +<!-- markdownlint-disable MD029 --> +1. Can be optionally run on reputable third party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +2. Can be optionally run as reputable third party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +3. Can be optionally run as reputable third party load balancing services (LB PaaS). AWS ELB is known to work. +4. Should be run on reputable third party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +<!-- markdownlint-enable MD029 --> + NOTE: -Components marked with * can be optionally run on reputable -third party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work. -Components marked with ** can be optionally run on reputable -third party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +For all PaaS solutions that involve configuring instances, it is strongly recommended to implement a minimum of three nodes in three different availability zones to align with resilient cloud architecture practices. ```plantuml @startuml 2k @@ -84,7 +88,7 @@ To set up GitLab and its components to accommodate up to 2,000 users: 1. [Configure Gitaly](#configure-gitaly), which provides access to the Git repositories. 1. [Configure the main GitLab Rails application](#configure-gitlab-rails) - to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend + to run Puma, Workhorse, GitLab Shell, and to serve all frontend requests (which include UI, API, and Git over HTTP/SSH). 1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab environment. @@ -265,10 +269,8 @@ further configuration steps. database. Example: `%w(123.123.123.123/32 123.123.123.234/32)` ```ruby - # Disable all components except PostgreSQL - roles ['postgres_role'] - patroni['enable'] = false - consul['enable'] = false + # Disable all components except PostgreSQL related ones + roles(['postgres_role']) prometheus['enable'] = false alertmanager['enable'] = false pgbouncer_exporter['enable'] = false @@ -295,6 +297,9 @@ further configuration steps. gitlab_rails['auto_migrate'] = false ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Note the PostgreSQL node's IP address or hostname, port, and plain text password. These will be necessary when configuring the [GitLab @@ -346,20 +351,18 @@ Omnibus: ```ruby ## Enable Redis redis['enable'] = true - - ## Disable all other services + + # Avoid running unnecessary services on the Redis server + gitaly['enable'] = false + postgresql['enable'] = false + puma['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false - puma['enable'] = false - unicorn['enable'] = false - postgresql['enable'] = false - nginx['enable'] = false prometheus['enable'] = false alertmanager['enable'] = false - pgbouncer_exporter['enable'] = false - gitlab_exporter['enable'] = false - gitaly['enable'] = false grafana['enable'] = false + gitlab_exporter['enable'] = false + nginx['enable'] = false redis['bind'] = '0.0.0.0' redis['port'] = 6379 @@ -376,7 +379,11 @@ Omnibus: } ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Note the Redis node's IP address or hostname, port, and Redis password. These will be necessary when [configuring the GitLab application servers](#configure-gitlab-rails) later. @@ -455,16 +462,14 @@ To configure the Gitaly server, on the server node you want to use for Gitaly: # Avoid running unnecessary services on the Gitaly server postgresql['enable'] = false redis['enable'] = false - nginx['enable'] = false puma['enable'] = false - unicorn['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false - grafana['enable'] = false - - # If you run a separate monitoring node you can disable these services - alertmanager['enable'] = false prometheus['enable'] = false + alertmanager['enable'] = false + grafana['enable'] = false + gitlab_exporter['enable'] = false + nginx['enable'] = false # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -472,9 +477,11 @@ To configure the Gitaly server, on the server node you want to use for Gitaly: # Configure the gitlab-shell API callback URL. Without this, `git push` will # fail. This can be your 'front door' GitLab URL or an internal load # balancer. - # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server. gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' + # Gitaly + gitaly['enable'] = true + # Make Gitaly accept connections on all network interfaces. You must use # firewalls to restrict access to this address/port. # Comment out following line if you only want to support TLS connections @@ -494,7 +501,11 @@ To configure the Gitaly server, on the server node you want to use for Gitaly: }) ``` -1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Confirm that Gitaly can perform callbacks to the internal API: ```shell @@ -629,7 +640,7 @@ On each node perform the following: }) ## Disable components that will not be on the GitLab application server - roles ['application_role'] + roles(['application_role']) gitaly['enable'] = false nginx['enable'] = true @@ -658,10 +669,7 @@ On each node perform the following: gitlab_rails['monitoring_whitelist'] = ['<MONITOR NODE IP>/32', '127.0.0.0/8'] nginx['status']['options']['allow'] = ['<MONITOR NODE IP>/32', '127.0.0.0/8'] - ############################# - ### Object storage ### - ############################# - + # Object Storage # This is an example for configuring Object Storage on GCP # Replace this config with your chosen Object Storage provider as desired gitlab_rails['object_store']['connection'] = { @@ -677,6 +685,13 @@ On each node perform the following: gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>" gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>" + gitlab_rails['backup_upload_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } + gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" + ## Uncomment and edit the following options if you have set up NFS ## ## Prevent GitLab from starting if NFS data mounts are not available @@ -710,7 +725,20 @@ On each node perform the following: sudo cp cert.pem /etc/gitlab/trusted-certs/ ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + +1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: + + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure + ``` + + Only a single designated node should handle migrations as detailed in the + [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section. + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Run `sudo gitlab-rake gitlab:gitaly:check` to confirm the node can connect to Gitaly. 1. Tail the logs to see the requests: @@ -718,11 +746,6 @@ On each node perform the following: sudo gitlab-ctl tail gitaly ``` -1. Save the `/etc/gitlab/gitlab-secrets.json` file from one of the two - application nodes and install it on the other application node and the - [Gitaly node](#configure-gitaly) and - [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). - When you specify `https` in the `external_url`, as in the previous example, GitLab expects that the SSL certificates are in `/etc/gitlab/ssl/`. If the certificates aren't present, NGINX will fail to start. For more information, see @@ -765,38 +788,21 @@ running [Prometheus](../monitoring/prometheus/index.md) and 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby + roles(['monitoring_role']) + external_url 'http://gitlab.example.com' - # Enable Prometheus - prometheus['enable'] = true + # Prometheus prometheus['listen_address'] = '0.0.0.0:9090' prometheus['monitor_kubernetes'] = false - # Enable Login form - grafana['disable_login_form'] = false - - # Enable Grafana + # Grafana grafana['enable'] = true - grafana['admin_password'] = 'toomanysecrets' + grafana['admin_password'] = '<grafana_password>' + grafana['disable_login_form'] = false - # Disable all other services - alertmanager['enable'] = false - gitaly['enable'] = false - gitlab_exporter['enable'] = false - gitlab_workhorse['enable'] = false + # Nginx - For Grafana access nginx['enable'] = true - postgres_exporter['enable'] = false - postgresql['enable'] = false - redis['enable'] = false - redis_exporter['enable'] = false - sidekiq['enable'] = false - puma['enable'] = false - unicorn['enable'] = false - node_exporter['enable'] = false - gitlab_exporter['enable'] = false - - # Prevent database migrations from running on upgrade automatically - gitlab_rails['auto_migrate'] = false ``` 1. Prometheus also needs some scrape configurations to pull all the data from the various diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index d81f98a35d4..bca5e4c3dab 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -27,26 +27,31 @@ For a full list of reference architectures, see | Service | Nodes | Configuration | GCP | AWS | Azure | |--------------------------------------------|-------------|-----------------------|-----------------|--------------|----------| -| External load balancing node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Redis** | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | -| Consul* + Sentinel** | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| PostgreSQL* | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | -| PgBouncer* | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| External load balancing node(3) | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Redis(2) | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | +| Consul(1) + Sentinel(2) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| PostgreSQL(1) | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | +| PgBouncer(1) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Internal load balancing node(3) | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Gitaly | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Praefect PostgreSQL* | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Praefect PostgreSQL(1) | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Sidekiq | 4 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | | GitLab Rails | 3 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | | Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Object storage | n/a | n/a | n/a | n/a | n/a | +| Object storage(4) | n/a | n/a | n/a | n/a | n/a | | NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> +<!-- markdownlint-disable MD029 --> +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. +4. Should be run on reputable third party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +<!-- markdownlint-enable MD029 --> + NOTE: -Components marked with * can be optionally run on reputable -third party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work. -Components marked with ** can be optionally run on reputable -third party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +For all PaaS solutions that involve configuring instances, it is strongly recommended to implement a minimum of three nodes in three different availability zones to align with resilient cloud architecture practices. ```plantuml @startuml 3k @@ -169,7 +174,7 @@ To set up GitLab and its components to accommodate up to 3,000 users: provides access to the Git repositories. 1. [Configure Sidekiq](#configure-sidekiq). 1. [Configure the main GitLab Rails application](#configure-gitlab-rails) - to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend + to run Puma, Workhorse, GitLab Shell, and to serve all frontend requests (which include UI, API, and Git over HTTP/SSH). 1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab environment. @@ -470,8 +475,8 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' - roles ['redis_master_role'] + # Specify server role as 'redis_master_role' and enable Consul agent + roles(['redis_master_role', 'consul_role']) # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -487,7 +492,6 @@ a node and change its status from primary to replica (and vice versa). redis['password'] = 'redis-password-goes-here' ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -508,6 +512,9 @@ a node and change its status from primary to replica (and vice versa). gitlab_rails['auto_migrate'] = false ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. You can specify multiple roles, like sentinel and Redis, as: @@ -546,8 +553,8 @@ run: redis-exporter: (pid 30075) 76861s; run: log: (pid 29674) 76896s 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_replica_role' - roles ['redis_replica_role'] + # Specify server role as 'redis_replica_role' and enable Consul agent + roles(['redis_replica_role', 'consul_role']) # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -570,7 +577,6 @@ run: redis-exporter: (pid 30075) 76861s; run: log: (pid 29674) 76896s #redis['master_port'] = 6379 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -591,12 +597,15 @@ run: redis-exporter: (pid 30075) 76861s; run: log: (pid 29674) 76896s gitlab_rails['auto_migrate'] = false ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. You can specify multiple roles, like sentinel and Redis, as: -`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about [roles](https://docs.gitlab.com/omnibus/roles/). These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after @@ -638,7 +647,7 @@ To configure the Sentinel: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - roles ['redis_sentinel_role', 'consul_role'] + roles(['redis_sentinel_role', 'consul_role']) # Must be the same in every sentinel node redis['master_name'] = 'gitlab-redis' @@ -702,7 +711,6 @@ To configure the Sentinel: # sentinel['failover_timeout'] = 60000 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -720,7 +728,11 @@ To configure the Sentinel: gitlab_rails['auto_migrate'] = false ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Go through the steps again for all the other Consul/Sentinel nodes, and make sure you set up the correct IPs. @@ -813,6 +825,15 @@ in the second step, do not supply the `EXTERNAL_URL` value. sudo gitlab-ctl pg-password-md5 pgbouncer ``` +1. Generate a password hash for the PostgreSQL replication username/password pair. This assumes you will use the default + username of `gitlab_replicator` (recommended). The command will request a password + and a confirmation. Use the value that is output by this command in the next step + as the value of `<postgresql_replication_password_hash>`: + + ```shell + sudo gitlab-ctl pg-password-md5 gitlab_replicator + ``` + 1. Generate a password hash for the Consul database username/password pair. This assumes you will use the default username of `gitlab-consul` (recommended). The command will request a password and confirmation. Use the value that is output by this command in the next @@ -825,27 +846,28 @@ in the second step, do not supply the `EXTERNAL_URL` value. 1. On every database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: ```ruby - # Disable all components except PostgreSQL, Patroni, and Consul - roles ['postgres_role'] - + # Disable all components except Patroni and Consul + roles(['patroni_role']) + # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' - # Enable Patroni - patroni['enable'] = true - # Set `max_wal_senders` to one more than the number of database nodes in the cluster. + # Sets `max_replication_slots` to double the number of database nodes. + # Patroni uses one extra slot per node when initiating the replication. + patroni['postgresql']['max_replication_slots'] = 6 + + # Set `max_wal_senders` to one more than the number of replication slots in the cluster. # This is used to prevent replication from using up all of the # available database connections. - patroni['postgresql']['max_wal_senders'] = 4 - patroni['postgresql']['max_replication_slots'] = 4 + patroni['postgresql']['max_wal_senders'] = 7 + # Incoming recommended value for max connections is 500. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691. patroni['postgresql']['max_connections'] = 500 # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false - + # Configure the Consul agent - consul['enable'] = true consul['services'] = %w(postgresql) ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -855,6 +877,8 @@ in the second step, do not supply the `EXTERNAL_URL` value. # # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value postgresql['pgbouncer_user_password'] = '<pgbouncer_password_hash>' + # Replace POSTGRESQL_REPLICATION_PASSWORD_HASH with a generated md5 value + postgresql['sql_replication_password'] = '<postgresql_replication_password_hash>' # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value postgresql['sql_user_password'] = '<postgresql_password_hash>' @@ -878,9 +902,8 @@ PostgreSQL, with Patroni managing its failover, will default to use `pg_rewind` Like most failover handling methods, this has a small chance of leading to data loss. Learn more about the various [Patroni replication methods](../postgresql/replication_and_failover.md#selecting-the-appropriate-patroni-replication-method). -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -895,22 +918,7 @@ are supported and can be added if needed. #### PostgreSQL post-configuration -SSH in to the **primary node**: - -1. Open a database prompt: - - ```shell - gitlab-psql -d gitlabhq_production - ``` - -1. Enable the `pg_trgm` and `btree_gist` extensions: - - ```shell - CREATE EXTENSION pg_trgm; - CREATE EXTENSION btree_gist; - ``` - -1. Exit the database prompt by typing `\q` and Enter. +SSH in to any of the Patroni nodes on the **primary site**: 1. Check the status of the leader and cluster: @@ -952,7 +960,7 @@ The following IPs will be used as an example: ```ruby # Disable all components except Pgbouncer and Consul agent - roles ['pgbouncer_role'] + roles(['pgbouncer_role']) # Configure PgBouncer pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) @@ -969,7 +977,6 @@ The following IPs will be used as an example: # Configure Consul agent consul['watchers'] = %w(postgresql) - consul['enable'] = true consul['configuration'] = { retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } @@ -982,6 +989,9 @@ The following IPs will be used as an example: pgbouncer_exporter['listen_address'] = '0.0.0.0:9188' ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Create a `.pgpass` file so Consul is able to @@ -1097,9 +1107,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. ```ruby # Disable all components except PostgreSQL and Consul - roles ['postgres_role'] - repmgr['enable'] = false - patroni['enable'] = false + roles(['postgres_role', 'consul_role']) # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' @@ -1109,7 +1117,6 @@ in the second step, do not supply the `EXTERNAL_URL` value. gitlab_rails['auto_migrate'] = false # Configure the Consul agent - consul['enable'] = true ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -1135,6 +1142,9 @@ in the second step, do not supply the `EXTERNAL_URL` value. # END user configuration ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Follow the [post configuration](#praefect-postgresql-post-configuration). @@ -1213,7 +1223,7 @@ Praefect requires several secret tokens to secure communications across the Clus Gitaly Cluster nodes are configured in Praefect via a `virtual storage`. Each storage contains the details of each Gitaly node that makes up the cluster. Each storage is also given a name -and this name is used in several areas of the config. In this guide, the name of the storage will be +and this name is used in several areas of the configuration. In this guide, the name of the storage will be `default`. Also, this guide is geared towards new installs, if upgrading an existing environment to use Gitaly Cluster, you may need to use a different name. Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more info. @@ -1233,19 +1243,18 @@ To configure the Praefect nodes, on each one: 1. Edit the `/etc/gitlab/gitlab.rb` file to configure Praefect: ```ruby - # Avoid running unnecessary services on the Gitaly server + # Avoid running unnecessary services on the Praefect server + gitaly['enable'] = false postgresql['enable'] = false redis['enable'] = false - nginx['enable'] = false puma['enable'] = false - unicorn['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false - grafana['enable'] = false - - # If you run a separate monitoring node you can disable these services - alertmanager['enable'] = false prometheus['enable'] = false + alertmanager['enable'] = false + grafana['enable'] = false + gitlab_exporter['enable'] = false + nginx['enable'] = false # Praefect Configuration praefect['enable'] = true @@ -1283,19 +1292,20 @@ To configure the Praefect nodes, on each one: # server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1') praefect['virtual_storages'] = { 'default' => { - 'gitaly-1' => { - 'address' => 'tcp://10.6.0.91:8075', - 'token' => '<praefect_internal_token>', - 'primary' => true - }, - 'gitaly-2' => { - 'address' => 'tcp://10.6.0.92:8075', - 'token' => '<praefect_internal_token>' - }, - 'gitaly-3' => { - 'address' => 'tcp://10.6.0.93:8075', - 'token' => '<praefect_internal_token>' - }, + 'nodes' => { + 'gitaly-1' => { + 'address' => 'tcp://10.6.0.91:8075', + 'token' => '<praefect_internal_token>' + }, + 'gitaly-2' => { + 'address' => 'tcp://10.6.0.92:8075', + 'token' => '<praefect_internal_token>' + }, + 'gitaly-3' => { + 'address' => 'tcp://10.6.0.93:8075', + 'token' => '<praefect_internal_token>' + }, + } } } @@ -1312,11 +1322,25 @@ To configure the Praefect nodes, on each one: # END user configuration ``` - 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and - then replace the file of the same name on this server. If that file isn't on - this server, add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace +the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. - 1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Praefect requires to run some database migrations, much like the main GitLab application. For this + you should select **one Praefect node only to run the migrations**, AKA the _Deploy Node_. This node + must be configured first before the others as follows: + + 1. In the `/etc/gitlab/gitlab.rb` file, change the `praefect['auto_migrate']` setting value from `false` to `true` + + 1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: + + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure + ``` + + 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect and + to run the Praefect database migrations. + +1. On all other Praefect nodes, [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. ### Configure Gitaly @@ -1360,29 +1384,27 @@ On each node: storage paths, enable the network listener, and to configure the token: ```ruby - # /etc/gitlab/gitlab.rb - # Avoid running unnecessary services on the Gitaly server postgresql['enable'] = false redis['enable'] = false - nginx['enable'] = false puma['enable'] = false - unicorn['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false - grafana['enable'] = false - - # If you run a separate monitoring node you can disable these services - alertmanager['enable'] = false prometheus['enable'] = false + alertmanager['enable'] = false + grafana['enable'] = false + gitlab_exporter['enable'] = false + nginx['enable'] = false # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false + # Gitaly + gitaly['enable'] = true + # Configure the gitlab-shell API callback URL. Without this, `git push` will # fail. This can be your 'front door' GitLab URL or an internal load # balancer. - # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server. gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' # Make Gitaly accept connections on all network interfaces. You must use @@ -1426,9 +1448,8 @@ On each node: }) ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and - then replace the file of the same name on this server. If that file isn't on - this server, add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -1537,29 +1558,19 @@ To configure the Sidekiq nodes, one each one: 1. Open `/etc/gitlab/gitlab.rb` with your editor: ```ruby - ######################################## - ##### Services Disabled ### - ######################################## - - nginx['enable'] = false - grafana['enable'] = false - prometheus['enable'] = false - alertmanager['enable'] = false + # Avoid running unnecessary services on the Sidekiq server gitaly['enable'] = false - gitlab_workhorse['enable'] = false - nginx['enable'] = false - puma['enable'] = false - postgres_exporter['enable'] = false postgresql['enable'] = false redis['enable'] = false - redis_exporter['enable'] = false + puma['enable'] = false + gitlab_workhorse['enable'] = false + prometheus['enable'] = false + alertmanager['enable'] = false + grafana['enable'] = false gitlab_exporter['enable'] = false + nginx['enable'] = false - ######################################## - #### Redis ### - ######################################## - - ## Must be the same in every sentinel node + # Redis redis['master_name'] = 'gitlab-redis' ## The same password for Redis authentication you set up for the master node. @@ -1572,13 +1583,10 @@ To configure the Sidekiq nodes, one each one: {'host' => '10.6.0.13', 'port' => 26379}, ] - ####################################### - ### Gitaly ### - ####################################### - - # git_data_dirs get configured for the Praefect virtual storage - # Address is Internal Load Balancer for Praefect - # Token is praefect_external_token + # Gitaly Cluster + ## git_data_dirs get configured for the Praefect virtual storage + ## Address is Internal Load Balancer for Praefect + ## Token is praefect_external_token git_data_dirs({ "default" => { "gitaly_address" => "tcp://10.6.0.40:2305", # internal load balancer IP @@ -1586,31 +1594,26 @@ To configure the Sidekiq nodes, one each one: } }) - ####################################### - ### Postgres ### - ####################################### + # PostgreSQL gitlab_rails['db_host'] = '10.6.0.40' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '<postgresql_user_password>' gitlab_rails['db_adapter'] = 'postgresql' gitlab_rails['db_encoding'] = 'unicode' - # Prevent database migrations from running on upgrade automatically + ## Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false - ####################################### - ### Sidekiq configuration ### - ####################################### + # Sidekiq + sidekiq['enable'] = true sidekiq['listen_address'] = "0.0.0.0" - # Set number of Sidekiq queue processes to the same number as available CPUs + ## Set number of Sidekiq queue processes to the same number as available CPUs sidekiq['queue_groups'] = ['*'] * 2 - # Set number of Sidekiq threads per queue process to the recommend number of 10 + ## Set number of Sidekiq threads per queue process to the recommend number of 10 sidekiq['max_concurrency'] = 10 - ####################################### - ### Monitoring configuration ### - ####################################### + # Monitoring consul['enable'] = true consul['monitoring_service_discovery'] = true @@ -1618,19 +1621,16 @@ To configure the Sidekiq nodes, one each one: retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } - # Set the network addresses that the exporters will listen on + ## Set the network addresses that the exporters will listen on node_exporter['listen_address'] = '0.0.0.0:9100' - # Rails Status for prometheus + ## Add the monitoring node's IP address to the monitoring whitelist gitlab_rails['monitoring_whitelist'] = ['10.6.0.81/32', '127.0.0.0/8'] gitlab_rails['prometheus_address'] = '10.6.0.81:9090' - ############################# - ### Object storage ### - ############################# - - # This is an example for configuring Object Storage on GCP - # Replace this config with your chosen Object Storage provider as desired + # Object Storage + ## This is an example for configuring Object Storage on GCP + ## Replace this config with your chosen Object Storage provider as desired gitlab_rails['object_store']['connection'] = { 'provider' => 'Google', 'google_project' => '<gcp-project-name>', @@ -1644,9 +1644,28 @@ To configure the Sidekiq nodes, one each one: gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>" gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>" + gitlab_rails['backup_upload_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } + gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + +1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: + + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure + ``` + + Only a single designated node should handle migrations as detailed in the + [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section. + 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + 1. Verify the GitLab services are running: ```shell @@ -1728,7 +1747,7 @@ On each node perform the following: }) ## Disable components that will not be on the GitLab application server - roles ['application_role'] + roles(['application_role']) gitaly['enable'] = false nginx['enable'] = true sidekiq['enable'] = false @@ -1793,10 +1812,7 @@ On each node perform the following: #registry['uid'] = 9002 #registry['gid'] = 9002 - ############################# - ### Object storage ### - ############################# - + # Object storage # This is an example for configuring Object Storage on GCP # Replace this config with your chosen Object Storage provider as desired gitlab_rails['object_store']['connection'] = { @@ -1811,6 +1827,13 @@ On each node perform the following: gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-packages-bucket-name>" gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>" gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>" + + gitlab_rails['backup_upload_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } + gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" ``` 1. If you're using [Gitaly with TLS support](#gitaly-cluster-tls-support), make sure the @@ -1831,7 +1854,20 @@ On each node perform the following: sudo cp cert.pem /etc/gitlab/trusted-certs/ ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + +1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: + + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure + ``` + + Only a single designated node should handle migrations as detailed in the + [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section. + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Run `sudo gitlab-rake gitlab:gitaly:check` to confirm the node can connect to Gitaly. 1. Tail the logs to see the requests: @@ -1839,11 +1875,6 @@ On each node perform the following: sudo gitlab-ctl tail gitaly ``` -1. Save the `/etc/gitlab/gitlab-secrets.json` file from one of the two - application nodes and install it on the other application node, the - [Gitaly node](#configure-gitaly) and the [Sidekiq node](#configure-sidekiq) and - [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). - 1. Verify the GitLab services are running: ```shell @@ -1902,45 +1933,26 @@ running [Prometheus](../monitoring/prometheus/index.md) and 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - external_url 'http://gitlab.example.com' + roles(['monitoring_role', 'consul_role']) - # Disable all other services - alertmanager['enable'] = false - gitaly['enable'] = false - gitlab_exporter['enable'] = false - gitlab_workhorse['enable'] = false - nginx['enable'] = true - postgres_exporter['enable'] = false - postgresql['enable'] = false - redis['enable'] = false - redis_exporter['enable'] = false - sidekiq['enable'] = false - puma['enable'] = false - unicorn['enable'] = false - node_exporter['enable'] = false - gitlab_exporter['enable'] = false + external_url 'http://gitlab.example.com' - # Enable Prometheus - prometheus['enable'] = true + # Prometheus prometheus['listen_address'] = '0.0.0.0:9090' prometheus['monitor_kubernetes'] = false - # Enable Login form - grafana['disable_login_form'] = false - - # Enable Grafana - grafana['enable'] = true + # Grafana grafana['admin_password'] = '<grafana_password>' + grafana['disable_login_form'] = false # Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true consul['configuration'] = { retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } - # Prevent database migrations from running on upgrade automatically - gitlab_rails['auto_migrate'] = false + # Nginx - For Grafana access + nginx['enable'] = true ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -2074,7 +2086,7 @@ but with smaller performance requirements, several modifications can be consider - PostgreSQL: Can be run on reputable Cloud PaaS solutions such as Google Cloud SQL or AWS RDS. In this setup, the PgBouncer and Consul nodes are no longer required: - Consul may still be desired if [Prometheus](../monitoring/prometheus/index.md) auto discovery is a requirement, otherwise you would need to [manually add scrape configurations](../monitoring/prometheus/index.md#adding-custom-scrape-configurations) for all nodes. - As Redis Sentinel runs on the same box as Consul in this architecture, it may need to be run on a separate box if Redis is still being run via Omnibus. - - Redis: Can be run on reputable Cloud PaaS solutions such as Google Memorystore and AWS Elasticache. In this setup, the Redis Sentinel is no longer required. + - Redis: Can be run on reputable Cloud PaaS solutions such as Google Memorystore and AWS ElastiCache. In this setup, the Redis Sentinel is no longer required. <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index 5cc463c953d..b3324cb75fb 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -17,29 +17,34 @@ full list of reference architectures, see | Service | Nodes | Configuration | GCP | AWS | Azure | |------------------------------------------|-------------|-------------------------|------------------|---------------|-----------| -| External load balancing node | 1 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | -| Consul* | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| PostgreSQL* | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | `D32s v3` | -| PgBouncer* | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Internal load balancing node | 1 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | -| Redis - Cache** | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis - Queues / Shared State** | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis Sentinel - Cache** | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | -| Redis Sentinel - Queues / Shared State** | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | +| External load balancing node(3) | 1 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | +| Consul(1) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| PostgreSQL(1) | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | `D32s v3` | +| PgBouncer(1) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Internal load balancing node(3) | 1 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | +| Redis - Cache(2) | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis - Queues / Shared State(2) | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis Sentinel - Cache(2) | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | +| Redis Sentinel - Queues / Shared State(2)| 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | | Gitaly | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` | `m5.16xlarge` | `D64s v3` | | Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | -| Praefect PostgreSQL* | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Praefect PostgreSQL(1) | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | GitLab Rails | 12 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` | | Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | -| Object storage | n/a | n/a | n/a | n/a | n/a | -| NFS server | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| Object storage(4) | n/a | n/a | n/a | n/a | n/a | +| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | + +<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> +<!-- markdownlint-disable MD029 --> +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. +4. Should be run on reputable third party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +<!-- markdownlint-enable MD029 --> NOTE: -Components marked with * can be optionally run on reputable -third party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work. -Components marked with ** can be optionally run on reputable -third party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +For all PaaS solutions that involve configuring instances, it is strongly recommended to implement a minimum of three nodes in three different availability zones to align with resilient cloud architecture practices. ```plantuml @startuml 50k @@ -157,7 +162,7 @@ To set up GitLab and its components to accommodate up to 50,000 users: provides access to the Git repositories. 1. [Configure Sidekiq](#configure-sidekiq). 1. [Configure the main GitLab Rails application](#configure-gitlab-rails) - to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend + to run Puma, Workhorse, GitLab Shell, and to serve all frontend requests (which include UI, API, and Git over HTTP/SSH). 1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab environment. @@ -420,11 +425,6 @@ The following IPs will be used as an example: - `10.6.0.12`: Consul 2 - `10.6.0.13`: Consul 3 -NOTE: -The configuration processes for the other servers in your reference architecture will -use the `/etc/gitlab/gitlab-secrets.json` file from your Consul server to connect -with the other servers. - To configure Consul: 1. SSH in to the server that will host Consul. @@ -435,10 +435,9 @@ To configure Consul: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - roles ['consul_role'] + roles(['consul_role']) ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -455,7 +454,11 @@ To configure Consul: gitlab_rails['auto_migrate'] = false ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Go through the steps again for all the other Consul nodes, and make sure you set up the correct IPs. @@ -547,6 +550,15 @@ in the second step, do not supply the `EXTERNAL_URL` value. sudo gitlab-ctl pg-password-md5 pgbouncer ``` +1. Generate a password hash for the PostgreSQL replication username/password pair. This assumes you will use the default + username of `gitlab_replicator` (recommended). The command will request a password + and a confirmation. Use the value that is output by this command in the next step + as the value of `<postgresql_replication_password_hash>`: + + ```shell + sudo gitlab-ctl pg-password-md5 gitlab_replicator + ``` + 1. Generate a password hash for the Consul database username/password pair. This assumes you will use the default username of `gitlab-consul` (recommended). The command will request a password and confirmation. Use the value that is output by this command in the next @@ -559,19 +571,21 @@ in the second step, do not supply the `EXTERNAL_URL` value. 1. On every database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: ```ruby - # Disable all components except PostgreSQL, Patroni, and Consul - roles ['postgres_role'] + # Disable all components except Patroni and Consul + roles(['patroni_role']) # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' - # Enable Patroni - patroni['enable'] = true - # Set `max_wal_senders` to one more than the number of database nodes in the cluster. + # Sets `max_replication_slots` to double the number of database nodes. + # Patroni uses one extra slot per node when initiating the replication. + patroni['postgresql']['max_replication_slots'] = 8 + + # Set `max_wal_senders` to one more than the number of replication slots in the cluster. # This is used to prevent replication from using up all of the # available database connections. - patroni['postgresql']['max_wal_senders'] = 4 - patroni['postgresql']['max_replication_slots'] = 4 + patroni['postgresql']['max_wal_senders'] = 9 + # Incoming recommended value for max connections is 500. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691. patroni['postgresql']['max_connections'] = 500 @@ -589,6 +603,8 @@ in the second step, do not supply the `EXTERNAL_URL` value. # # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value postgresql['pgbouncer_user_password'] = '<pgbouncer_password_hash>' + # Replace POSTGRESQL_REPLICATION_PASSWORD_HASH with a generated md5 value + postgresql['sql_replication_password'] = '<postgresql_replication_password_hash>' # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value postgresql['sql_user_password'] = '<postgresql_password_hash>' @@ -612,9 +628,8 @@ PostgreSQL, with Patroni managing its failover, will default to use `pg_rewind` Like most failover handling methods, this has a small chance of leading to data loss. Learn more about the various [Patroni replication methods](../postgresql/replication_and_failover.md#selecting-the-appropriate-patroni-replication-method). -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -629,21 +644,7 @@ are supported and can be added if needed. #### PostgreSQL post-configuration -SSH in to the **primary node**: - -1. Open a database prompt: - - ```shell - gitlab-psql -d gitlabhq_production - ``` - -1. Make sure the `pg_trgm` extension is enabled (it might already be): - - ```shell - CREATE EXTENSION pg_trgm; - ``` - -1. Exit the database prompt by typing `\q` and Enter. +SSH in to any of the Patroni nodes on the **primary site**: 1. Check the status of the leader and cluster: @@ -685,7 +686,7 @@ The following IPs will be used as an example: ```ruby # Disable all components except Pgbouncer and Consul agent - roles ['pgbouncer_role'] + roles(['pgbouncer_role']) # Configure PgBouncer pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) @@ -702,7 +703,6 @@ The following IPs will be used as an example: # Configure Consul agent consul['watchers'] = %w(postgresql) - consul['enable'] = true consul['configuration'] = { retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } @@ -714,9 +714,8 @@ The following IPs will be used as an example: node_exporter['listen_address'] = '0.0.0.0:9100' ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -835,8 +834,8 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' - roles ['redis_master_role'] + # Specify server role as 'redis_master_role' and enable Consul agent + roles(['redis_master_role', 'consul_role']) # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -858,7 +857,6 @@ a node and change its status from primary to replica (and vice versa). redis['maxmemory_samples'] = 5 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -870,19 +868,22 @@ a node and change its status from primary to replica (and vice versa). # Set the network addresses that the exporters will listen on node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' - + redis_exporter['flags'] = { + 'redis.addr' => 'redis://10.6.0.51:6379', + 'redis.password' => 'redis-password-goes-here', + } + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. You can specify multiple roles, like sentinel and Redis, as: -`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about [roles](https://docs.gitlab.com/omnibus/roles/). #### Configure the replica Redis Cache nodes @@ -895,8 +896,8 @@ You can specify multiple roles, like sentinel and Redis, as: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_replica_role' - roles ['redis_replica_role'] + # Specify server role as 'redis_replica_role' and enable Consul agent + roles(['redis_replica_role', 'consul_role']) # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -925,7 +926,6 @@ You can specify multiple roles, like sentinel and Redis, as: redis['maxmemory_samples'] = 5 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -937,21 +937,25 @@ You can specify multiple roles, like sentinel and Redis, as: # Set the network addresses that the exporters will listen on node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' + redis_exporter['flags'] = { + 'redis.addr' => 'redis://10.6.0.52:6379', + 'redis.password' => 'redis-password-goes-here', + } # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. You can specify multiple roles, like sentinel and Redis, as: -`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about [roles](https://docs.gitlab.com/omnibus/roles/). These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after @@ -993,7 +997,7 @@ To configure the Sentinel Cache server: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - roles ['redis_sentinel_role'] + roles(['redis_sentinel_role', 'consul_role']) ## Must be the same in every sentinel node redis['master_name'] = 'gitlab-redis-cache' @@ -1057,7 +1061,6 @@ To configure the Sentinel Cache server: #sentinel['failover_timeout'] = 60000 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -1074,11 +1077,11 @@ To configure the Sentinel Cache server: gitlab_rails['auto_migrate'] = false ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Go through the steps again for all the other Consul/Sentinel nodes, and make sure you set up the correct IPs. @@ -1106,8 +1109,8 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' - roles ['redis_master_role'] + # Specify server role as 'redis_master_role' and enable Consul agent + roles(['redis_master_role', 'consul_role']) # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -1123,7 +1126,6 @@ a node and change its status from primary to replica (and vice versa). redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -1140,9 +1142,8 @@ a node and change its status from primary to replica (and vice versa). gitlab_rails['auto_migrate'] = false ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -1160,8 +1161,8 @@ You can specify multiple roles, like sentinel and Redis, as: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_replica_role' - roles ['redis_replica_role'] + # Specify server role as 'redis_replica_role' and enable Consul agent + roles(['redis_replica_role', 'consul_role']) # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -1184,7 +1185,6 @@ You can specify multiple roles, like sentinel and Redis, as: #redis['master_port'] = 6379 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -1201,16 +1201,16 @@ You can specify multiple roles, like sentinel and Redis, as: gitlab_rails['auto_migrate'] = false ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. You can specify multiple roles, like sentinel and Redis, as: -`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about [roles](https://docs.gitlab.com/omnibus/roles/). These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after @@ -1252,7 +1252,7 @@ To configure the Sentinel Queues server: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - roles ['redis_sentinel_role'] + roles(['redis_sentinel_role', 'consul_role']) ## Must be the same in every sentinel node redis['master_name'] = 'gitlab-redis-persistent' @@ -1316,7 +1316,6 @@ To configure the Sentinel Queues server: #sentinel['failover_timeout'] = 60000 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -1333,7 +1332,7 @@ To configure the Sentinel Queues server: gitlab_rails['auto_migrate'] = false ``` -1. To prevent database migrations from running on upgrade, run: +1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: ```shell sudo touch /etc/gitlab/skip-auto-reconfigure @@ -1341,11 +1340,11 @@ To configure the Sentinel Queues server: Only the primary GitLab application server should handle migrations. -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Go through the steps again for all the other Sentinel nodes, and make sure you set up the correct IPs. @@ -1406,9 +1405,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. ```ruby # Disable all components except PostgreSQL and Consul - roles ['postgres_role'] - repmgr['enable'] = false - patroni['enable'] = false + roles(['postgres_role', 'consul_role']) # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' @@ -1418,7 +1415,6 @@ in the second step, do not supply the `EXTERNAL_URL` value. gitlab_rails['auto_migrate'] = false # Configure the Consul agent - consul['enable'] = true ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -1444,7 +1440,11 @@ in the second step, do not supply the `EXTERNAL_URL` value. # END user configuration ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Follow the [post configuration](#praefect-postgresql-post-configuration). <div align="right"> @@ -1542,19 +1542,18 @@ To configure the Praefect nodes, on each one: 1. Edit the `/etc/gitlab/gitlab.rb` file to configure Praefect: ```ruby - # Avoid running unnecessary services on the Gitaly server + # Avoid running unnecessary services on the Praefect server + gitaly['enable'] = false postgresql['enable'] = false redis['enable'] = false - nginx['enable'] = false puma['enable'] = false - unicorn['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false - grafana['enable'] = false - - # If you run a separate monitoring node you can disable these services - alertmanager['enable'] = false prometheus['enable'] = false + alertmanager['enable'] = false + grafana['enable'] = false + gitlab_exporter['enable'] = false + nginx['enable'] = false # Praefect Configuration praefect['enable'] = true @@ -1592,19 +1591,20 @@ To configure the Praefect nodes, on each one: # server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1') praefect['virtual_storages'] = { 'default' => { - 'gitaly-1' => { - 'address' => 'tcp://10.6.0.91:8075', - 'token' => '<praefect_internal_token>', - 'primary' => true - }, - 'gitaly-2' => { - 'address' => 'tcp://10.6.0.92:8075', - 'token' => '<praefect_internal_token>' - }, - 'gitaly-3' => { - 'address' => 'tcp://10.6.0.93:8075', - 'token' => '<praefect_internal_token>' - }, + 'nodes' => { + 'gitaly-1' => { + 'address' => 'tcp://10.6.0.91:8075', + 'token' => '<praefect_internal_token>' + }, + 'gitaly-2' => { + 'address' => 'tcp://10.6.0.92:8075', + 'token' => '<praefect_internal_token>' + }, + 'gitaly-3' => { + 'address' => 'tcp://10.6.0.93:8075', + 'token' => '<praefect_internal_token>' + }, + } } } @@ -1621,11 +1621,25 @@ To configure the Praefect nodes, on each one: # END user configuration ``` - 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and - then replace the file of the same name on this server. If that file isn't on - this server, add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace +the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. - 1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Praefect requires to run some database migrations, much like the main GitLab application. For this + you should select **one Praefect node only to run the migrations**, AKA the _Deploy Node_. This node + must be configured first before the others as follows: + + 1. In the `/etc/gitlab/gitlab.rb` file, change the `praefect['auto_migrate']` setting value from `false` to `true` + + 1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: + + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure + ``` + + 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect and + to run the Praefect database migrations. + +1. On all other Praefect nodes, [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. ### Configure Gitaly @@ -1669,21 +1683,17 @@ On each node: storage paths, enable the network listener, and to configure the token: ```ruby - # /etc/gitlab/gitlab.rb - # Avoid running unnecessary services on the Gitaly server postgresql['enable'] = false redis['enable'] = false - nginx['enable'] = false puma['enable'] = false - unicorn['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false - grafana['enable'] = false - - # If you run a separate monitoring node you can disable these services - alertmanager['enable'] = false prometheus['enable'] = false + alertmanager['enable'] = false + grafana['enable'] = false + gitlab_exporter['enable'] = false + nginx['enable'] = false # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -1691,9 +1701,11 @@ On each node: # Configure the gitlab-shell API callback URL. Without this, `git push` will # fail. This can be your 'front door' GitLab URL or an internal load # balancer. - # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server. gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' + # Gitaly + gitaly['enable'] = true + # Make Gitaly accept connections on all network interfaces. You must use # firewalls to restrict access to this address/port. # Comment out following line if you only want to support TLS connections @@ -1735,9 +1747,8 @@ On each node: }) ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and - then replace the file of the same name on this server. If that file isn't on - this server, add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -1844,28 +1855,19 @@ To configure the Sidekiq nodes, on each one: 1. Open `/etc/gitlab/gitlab.rb` with your editor: ```ruby - ######################################## - ##### Services Disabled ### - ######################################## - - nginx['enable'] = false - grafana['enable'] = false - prometheus['enable'] = false - alertmanager['enable'] = false + # Avoid running unnecessary services on the Sidekiq server gitaly['enable'] = false - gitlab_workhorse['enable'] = false - nginx['enable'] = false - puma['enable'] = false - postgres_exporter['enable'] = false postgresql['enable'] = false redis['enable'] = false - redis_exporter['enable'] = false + puma['enable'] = false + gitlab_workhorse['enable'] = false + prometheus['enable'] = false + alertmanager['enable'] = false + grafana['enable'] = false gitlab_exporter['enable'] = false + nginx['enable'] = false - ######################################## - #### Redis ### - ######################################## - + # Redis ## Redis connection details ## First cluster that will host the cache gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' @@ -1897,10 +1899,7 @@ To configure the Sidekiq nodes, on each one: {host: '10.6.0.83', port: 26379}, ] - ####################################### - ### Gitaly ### - ####################################### - + # Gitaly # git_data_dirs get configured for the Praefect virtual storage # Address is Internal Load Balancer for Praefect # Token is praefect_external_token @@ -1911,31 +1910,26 @@ To configure the Sidekiq nodes, on each one: } }) - ####################################### - ### Postgres ### - ####################################### + # PostgreSQL gitlab_rails['db_host'] = '10.6.0.20' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '<postgresql_user_password>' gitlab_rails['db_adapter'] = 'postgresql' gitlab_rails['db_encoding'] = 'unicode' - # Prevent database migrations from running on upgrade automatically + ## Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false - ####################################### - ### Sidekiq configuration ### - ####################################### + # Sidekiq + sidekiq['enable'] = true sidekiq['listen_address'] = "0.0.0.0" - # Set number of Sidekiq queue processes to the same number as available CPUs + ## Set number of Sidekiq queue processes to the same number as available CPUs sidekiq['queue_groups'] = ['*'] * 4 - # Set number of Sidekiq threads per queue process to the recommend number of 10 + ## Set number of Sidekiq threads per queue process to the recommend number of 10 sidekiq['max_concurrency'] = 10 - ####################################### - ### Monitoring configuration ### - ####################################### + # Monitoring consul['enable'] = true consul['monitoring_service_discovery'] = true @@ -1946,15 +1940,12 @@ To configure the Sidekiq nodes, on each one: # Set the network addresses that the exporters will listen on node_exporter['listen_address'] = '0.0.0.0:9100' - # Rails Status for prometheus + ## Add the monitoring node's IP address to the monitoring whitelist gitlab_rails['monitoring_whitelist'] = ['10.6.0.151/32', '127.0.0.0/8'] - ############################# - ### Object storage ### - ############################# - - # This is an example for configuring Object Storage on GCP - # Replace this config with your chosen Object Storage provider as desired + # Object storage + ## This is an example for configuring Object Storage on GCP + ## Replace this config with your chosen Object Storage provider as desired gitlab_rails['object_store']['connection'] = { 'provider' => 'Google', 'google_project' => '<gcp-project-name>', @@ -1967,11 +1958,26 @@ To configure the Sidekiq nodes, on each one: gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-packages-bucket-name>" gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>" gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>" + + gitlab_rails['backup_upload_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } + gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" + ``` + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + +1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: + + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. + Only a single designated node should handle migrations as detailed in the + [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section. 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -2011,9 +2017,6 @@ On each node perform the following: 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to follow _only_ installation steps 1 and 2 on the page. -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. 1. Edit `/etc/gitlab/gitlab.rb` and use the following configuration. To maintain uniformity of links across nodes, the `external_url` @@ -2035,7 +2038,7 @@ On each node perform the following: }) ## Disable components that will not be on the GitLab application server - roles ['application_role'] + roles(['application_role']) gitaly['enable'] = false nginx['enable'] = true sidekiq['enable'] = false @@ -2108,9 +2111,15 @@ On each node perform the following: gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-packages-bucket-name>" gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>" gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>" + + gitlab_rails['backup_upload_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } + gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). 1. If you're using [Gitaly with TLS support](#gitaly-cluster-tls-support), make sure the `git_data_dirs` entry is configured with `tls` instead of `tcp`: @@ -2129,6 +2138,20 @@ On each node perform the following: sudo cp cert.pem /etc/gitlab/trusted-certs/ ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + +1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: + + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure + ``` + + Only a single designated node should handle migrations as detailed in the + [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section. + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. If you're [using NFS](#configure-nfs-optional): 1. If necessary, install the NFS client utility packages using the following commands: @@ -2168,7 +2191,7 @@ On each node perform the following: registry['gid'] = 9002 ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). 1. Confirm the node can connect to Gitaly: ```shell @@ -2232,52 +2255,30 @@ To configure the Monitoring node: 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to follow _only_ installation steps 1 and 2 on the page. -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - external_url 'http://gitlab.example.com' + roles(['monitoring_role', 'consul_role']) - # Disable all other services - alertmanager['enable'] = false - gitaly['enable'] = false - gitlab_exporter['enable'] = false - gitlab_workhorse['enable'] = false - nginx['enable'] = true - postgres_exporter['enable'] = false - postgresql['enable'] = false - redis['enable'] = false - redis_exporter['enable'] = false - sidekiq['enable'] = false - puma['enable'] = false - unicorn['enable'] = false - node_exporter['enable'] = false - gitlab_exporter['enable'] = false + external_url 'http://gitlab.example.com' - # Enable Prometheus - prometheus['enable'] = true + # Prometheus prometheus['listen_address'] = '0.0.0.0:9090' prometheus['monitor_kubernetes'] = false - # Enable Login form - grafana['disable_login_form'] = false - - # Enable Grafana - grafana['enable'] = true + # Grafana grafana['admin_password'] = '<grafana_password>' + grafana['disable_login_form'] = false # Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true consul['configuration'] = { retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } - # Prevent database migrations from running on upgrade automatically - gitlab_rails['auto_migrate'] = false + # Nginx - For Grafana access + nginx['enable'] = true ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index 792dd7020c7..9952df196c9 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -24,26 +24,31 @@ costly-to-operate environment by using the | Service | Nodes | Configuration | GCP | AWS | Azure | |--------------------------------------------|-------------|-------------------------|-----------------|--------------|----------| -| External load balancing node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Redis** | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | -| Consul* + Sentinel** | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| PostgreSQL* | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| PgBouncer* | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| External load balancing node(3) | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Redis(2) | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | +| Consul(1) + Sentinel(2) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| PostgreSQL(1) | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| PgBouncer(1) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Internal load balancing node(3) | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Gitaly | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | `D8s v3` | | Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Praefect PostgreSQL* | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Praefect PostgreSQL(1) | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Sidekiq | 4 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | | GitLab Rails | 3 | 16 vCPU, 14.4 GB memory | `n1-highcpu-16` | `c5.4xlarge` | `F16s v2`| | Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Object storage | n/a | n/a | n/a | n/a | n/a | +| Object storage(4) | n/a | n/a | n/a | n/a | n/a | | NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> +<!-- markdownlint-disable MD029 --> +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. +4. Should be run on reputable third party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +<!-- markdownlint-enable MD029 --> + NOTE: -Components marked with * can be optionally run on reputable -third party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work. -Components marked with ** can be optionally run on reputable -third party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +For all PaaS solutions that involve configuring instances, it is strongly recommended to implement a minimum of three nodes in three different availability zones to align with resilient cloud architecture practices. ```plantuml @startuml 5k @@ -161,7 +166,7 @@ To set up GitLab and its components to accommodate up to 5,000 users: provides access to the Git repositories. 1. [Configure Sidekiq](#configure-sidekiq). 1. [Configure the main GitLab Rails application](#configure-gitlab-rails) - to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend + to run Puma, Workhorse, GitLab Shell, and to serve all frontend requests (which include UI, API, and Git over HTTP/SSH). 1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab environment. @@ -462,8 +467,8 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' - roles ['redis_master_role'] + # Specify server role as 'redis_master_role' and enable Consul agent + roles(['redis_master_role', 'consul_role']) # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -479,7 +484,6 @@ a node and change its status from primary to replica (and vice versa). redis['password'] = 'redis-password-goes-here' ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -500,10 +504,13 @@ a node and change its status from primary to replica (and vice versa). gitlab_rails['auto_migrate'] = false ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. You can specify multiple roles, like sentinel and Redis, as: -`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about [roles](https://docs.gitlab.com/omnibus/roles/). You can list the current Redis Primary, Replica status via: @@ -538,8 +545,8 @@ run: redis-exporter: (pid 30075) 76861s; run: log: (pid 29674) 76896s 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_replica_role' - roles ['redis_replica_role'] + # Specify server role as 'redis_replica_role' and enable Consul agent + roles(['redis_replica_role', 'consul_role']) # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -562,7 +569,6 @@ run: redis-exporter: (pid 30075) 76861s; run: log: (pid 29674) 76896s #redis['master_port'] = 6379 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -583,12 +589,15 @@ run: redis-exporter: (pid 30075) 76861s; run: log: (pid 29674) 76896s gitlab_rails['auto_migrate'] = false ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. You can specify multiple roles, like sentinel and Redis, as: -`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about [roles](https://docs.gitlab.com/omnibus/roles/). These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after @@ -630,7 +639,7 @@ To configure the Sentinel: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - roles ['redis_sentinel_role', 'consul_role'] + roles(['redis_sentinel_role', 'consul_role']) # Must be the same in every sentinel node redis['master_name'] = 'gitlab-redis' @@ -694,7 +703,6 @@ To configure the Sentinel: # sentinel['failover_timeout'] = 60000 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -712,6 +720,9 @@ To configure the Sentinel: gitlab_rails['auto_migrate'] = false ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Go through the steps again for all the other Consul/Sentinel nodes, and make sure you set up the correct IPs. @@ -805,6 +816,15 @@ in the second step, do not supply the `EXTERNAL_URL` value. sudo gitlab-ctl pg-password-md5 pgbouncer ``` +1. Generate a password hash for the PostgreSQL replication username/password pair. This assumes you will use the default + username of `gitlab_replicator` (recommended). The command will request a password + and a confirmation. Use the value that is output by this command in the next step + as the value of `<postgresql_replication_password_hash>`: + + ```shell + sudo gitlab-ctl pg-password-md5 gitlab_replicator + ``` + 1. Generate a password hash for the Consul database username/password pair. This assumes you will use the default username of `gitlab-consul` (recommended). The command will request a password and confirmation. Use the value that is output by this command in the next @@ -817,19 +837,21 @@ in the second step, do not supply the `EXTERNAL_URL` value. 1. On every database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: ```ruby - # Disable all components except PostgreSQL, Patroni, and Consul - roles ['postgres_role'] + # Disable all components except Patroni and Consul + roles(['patroni_role']) # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' - # Enable Patroni - patroni['enable'] = true - # Set `max_wal_senders` to one more than the number of database nodes in the cluster. + # Sets `max_replication_slots` to double the number of database nodes. + # Patroni uses one extra slot per node when initiating the replication. + patroni['postgresql']['max_replication_slots'] = 8 + + # Set `max_wal_senders` to one more than the number of replication slots in the cluster. # This is used to prevent replication from using up all of the # available database connections. - patroni['postgresql']['max_wal_senders'] = 4 - patroni['postgresql']['max_replication_slots'] = 4 + patroni['postgresql']['max_wal_senders'] = 9 + # Incoming recommended value for max connections is 500. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691. patroni['postgresql']['max_connections'] = 500 @@ -837,7 +859,6 @@ in the second step, do not supply the `EXTERNAL_URL` value. gitlab_rails['auto_migrate'] = false # Configure the Consul agent - consul['enable'] = true consul['services'] = %w(postgresql) ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -847,6 +868,8 @@ in the second step, do not supply the `EXTERNAL_URL` value. # # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value postgresql['pgbouncer_user_password'] = '<pgbouncer_password_hash>' + # Replace POSTGRESQL_REPLICATION_PASSWORD_HASH with a generated md5 value + postgresql['sql_replication_password'] = '<postgresql_replication_password_hash>' # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value postgresql['sql_user_password'] = '<postgresql_password_hash>' @@ -870,9 +893,8 @@ PostgreSQL, with Patroni managing its failover, will default to use `pg_rewind` Like most failover handling methods, this has a small chance of leading to data loss. Learn more about the various [Patroni replication methods](../postgresql/replication_and_failover.md#selecting-the-appropriate-patroni-replication-method). -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -887,21 +909,7 @@ are supported and can be added if needed. #### PostgreSQL post-configuration -SSH in to the **primary node**: - -1. Open a database prompt: - - ```shell - gitlab-psql -d gitlabhq_production - ``` - -1. Enable the `pg_trgm` extension: - - ```shell - CREATE EXTENSION pg_trgm; - ``` - -1. Exit the database prompt by typing `\q` and Enter. +SSH in to any of the Patroni nodes on the **primary site**: 1. Check the status of the leader and cluster: @@ -943,7 +951,7 @@ The following IPs will be used as an example: ```ruby # Disable all components except Pgbouncer and Consul agent - roles ['pgbouncer_role'] + roles(['pgbouncer_role']) # Configure PgBouncer pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) @@ -960,7 +968,6 @@ The following IPs will be used as an example: # Configure Consul agent consul['watchers'] = %w(postgresql) - consul['enable'] = true consul['configuration'] = { retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } @@ -973,6 +980,9 @@ The following IPs will be used as an example: pgbouncer_exporter['listen_address'] = '0.0.0.0:9188' ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Create a `.pgpass` file so Consul is able to @@ -1088,9 +1098,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. ```ruby # Disable all components except PostgreSQL and Consul - roles ['postgres_role'] - repmgr['enable'] = false - patroni['enable'] = false + roles(['postgres_role', 'consul_role']) # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' @@ -1100,7 +1108,6 @@ in the second step, do not supply the `EXTERNAL_URL` value. gitlab_rails['auto_migrate'] = false # Configure the Consul agent - consul['enable'] = true ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -1126,7 +1133,11 @@ in the second step, do not supply the `EXTERNAL_URL` value. # END user configuration ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Follow the [post configuration](#praefect-postgresql-post-configuration). <div align="right"> @@ -1224,19 +1235,18 @@ To configure the Praefect nodes, on each one: 1. Edit the `/etc/gitlab/gitlab.rb` file to configure Praefect: ```ruby - # Avoid running unnecessary services on the Gitaly server + # Avoid running unnecessary services on the Praefect server + gitaly['enable'] = false postgresql['enable'] = false redis['enable'] = false - nginx['enable'] = false puma['enable'] = false - unicorn['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false - grafana['enable'] = false - - # If you run a separate monitoring node you can disable these services - alertmanager['enable'] = false prometheus['enable'] = false + alertmanager['enable'] = false + grafana['enable'] = false + gitlab_exporter['enable'] = false + nginx['enable'] = false # Praefect Configuration praefect['enable'] = true @@ -1274,19 +1284,20 @@ To configure the Praefect nodes, on each one: # server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1') praefect['virtual_storages'] = { 'default' => { - 'gitaly-1' => { - 'address' => 'tcp://10.6.0.91:8075', - 'token' => '<praefect_internal_token>', - 'primary' => true - }, - 'gitaly-2' => { - 'address' => 'tcp://10.6.0.92:8075', - 'token' => '<praefect_internal_token>' - }, - 'gitaly-3' => { - 'address' => 'tcp://10.6.0.93:8075', - 'token' => '<praefect_internal_token>' - }, + 'nodes' => { + 'gitaly-1' => { + 'address' => 'tcp://10.6.0.91:8075', + 'token' => '<praefect_internal_token>' + }, + 'gitaly-2' => { + 'address' => 'tcp://10.6.0.92:8075', + 'token' => '<praefect_internal_token>' + }, + 'gitaly-3' => { + 'address' => 'tcp://10.6.0.93:8075', + 'token' => '<praefect_internal_token>' + }, + } } } @@ -1303,11 +1314,25 @@ To configure the Praefect nodes, on each one: # END user configuration ``` - 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and - then replace the file of the same name on this server. If that file isn't on - this server, add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace +the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + +1. Praefect requires to run some database migrations, much like the main GitLab application. For this + you should select **one Praefect node only to run the migrations**, AKA the _Deploy Node_. This node + must be configured first before the others as follows: + + 1. In the `/etc/gitlab/gitlab.rb` file, change the `praefect['auto_migrate']` setting value from `false` to `true` + + 1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: + + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure + ``` + + 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect and + to run the Praefect database migrations. - 1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. On all other Praefect nodes, [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. ### Configure Gitaly @@ -1351,21 +1376,17 @@ On each node: storage paths, enable the network listener, and to configure the token: ```ruby - # /etc/gitlab/gitlab.rb - # Avoid running unnecessary services on the Gitaly server postgresql['enable'] = false redis['enable'] = false - nginx['enable'] = false puma['enable'] = false - unicorn['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false - grafana['enable'] = false - - # If you run a separate monitoring node you can disable these services - alertmanager['enable'] = false prometheus['enable'] = false + alertmanager['enable'] = false + grafana['enable'] = false + gitlab_exporter['enable'] = false + nginx['enable'] = false # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -1373,9 +1394,11 @@ On each node: # Configure the gitlab-shell API callback URL. Without this, `git push` will # fail. This can be your 'front door' GitLab URL or an internal load # balancer. - # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server. gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' + # Gitaly + gitaly['enable'] = true + # Make Gitaly accept connections on all network interfaces. You must use # firewalls to restrict access to this address/port. # Comment out following line if you only want to support TLS connections @@ -1417,9 +1440,8 @@ On each node: }) ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and - then replace the file of the same name on this server. If that file isn't on - this server, add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -1526,28 +1548,19 @@ To configure the Sidekiq nodes, one each one: 1. Open `/etc/gitlab/gitlab.rb` with your editor: ```ruby - ######################################## - ##### Services Disabled ### - ######################################## - - nginx['enable'] = false - grafana['enable'] = false - prometheus['enable'] = false - alertmanager['enable'] = false + # Avoid running unnecessary services on the Sidekiq server gitaly['enable'] = false - gitlab_workhorse['enable'] = false - nginx['enable'] = false - puma['enable'] = false - postgres_exporter['enable'] = false postgresql['enable'] = false redis['enable'] = false - redis_exporter['enable'] = false + puma['enable'] = false + gitlab_workhorse['enable'] = false + prometheus['enable'] = false + alertmanager['enable'] = false + grafana['enable'] = false gitlab_exporter['enable'] = false + nginx['enable'] = false - ######################################## - #### Redis ### - ######################################## - + # Redis ## Must be the same in every sentinel node redis['master_name'] = 'gitlab-redis' @@ -1561,13 +1574,10 @@ To configure the Sidekiq nodes, one each one: {'host' => '10.6.0.13', 'port' => 26379}, ] - ####################################### - ### Gitaly ### - ####################################### - - # git_data_dirs get configured for the Praefect virtual storage - # Address is Internal Load Balancer for Praefect - # Token is praefect_external_token + # Gitaly Cluster + ## git_data_dirs get configured for the Praefect virtual storage + ## Address is Internal Load Balancer for Praefect + ## Token is praefect_external_token git_data_dirs({ "default" => { "gitaly_address" => "tcp://10.6.0.40:2305", # internal load balancer IP @@ -1575,31 +1585,26 @@ To configure the Sidekiq nodes, one each one: } }) - ####################################### - ### Postgres ### - ####################################### + # PostgreSQL gitlab_rails['db_host'] = '10.6.0.40' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '<postgresql_user_password>' gitlab_rails['db_adapter'] = 'postgresql' gitlab_rails['db_encoding'] = 'unicode' - # Prevent database migrations from running on upgrade automatically + ## Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false - ####################################### - ### Sidekiq configuration ### - ####################################### + # Sidekiq + sidekiq['enable'] = true sidekiq['listen_address'] = "0.0.0.0" - # Set number of Sidekiq queue processes to the same number as available CPUs + ## Set number of Sidekiq queue processes to the same number as available CPUs sidekiq['queue_groups'] = ['*'] * 4 - # Set number of Sidekiq threads per queue process to the recommend number of 10 + ## Set number of Sidekiq threads per queue process to the recommend number of 10 sidekiq['max_concurrency'] = 10 - ####################################### - ### Monitoring configuration ### - ####################################### + # Monitoring consul['enable'] = true consul['monitoring_service_discovery'] = true @@ -1607,19 +1612,16 @@ To configure the Sidekiq nodes, one each one: retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } - # Set the network addresses that the exporters will listen on + ## Set the network addresses that the exporters will listen on node_exporter['listen_address'] = '0.0.0.0:9100' - # Rails Status for prometheus + ## Add the monitoring node's IP address to the monitoring whitelist gitlab_rails['monitoring_whitelist'] = ['10.6.0.81/32', '127.0.0.0/8'] gitlab_rails['prometheus_address'] = '10.6.0.81:9090' - ############################# - ### Object storage ### - ############################# - - # This is an example for configuring Object Storage on GCP - # Replace this config with your chosen Object Storage provider as desired + # Object Storage + ## This is an example for configuring Object Storage on GCP + ## Replace this config with your chosen Object Storage provider as desired gitlab_rails['object_store']['connection'] = { 'provider' => 'Google', 'google_project' => '<gcp-project-name>', @@ -1632,9 +1634,29 @@ To configure the Sidekiq nodes, one each one: gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-packages-bucket-name>" gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>" gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>" + + gitlab_rails['backup_upload_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } + gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + +1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: + + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure + ``` + + Only a single designated node should handle migrations as detailed in the + [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section. + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Verify the GitLab services are running: ```shell @@ -1716,7 +1738,7 @@ On each node perform the following: }) ## Disable components that will not be on the GitLab application server - roles ['application_role'] + roles(['application_role']) gitaly['enable'] = false nginx['enable'] = true sidekiq['enable'] = false @@ -1785,6 +1807,13 @@ On each node perform the following: gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>" gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>" + gitlab_rails['backup_upload_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } + gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" + ## Uncomment and edit the following options if you have set up NFS ## ## Prevent GitLab from starting if NFS data mounts are not available @@ -1819,7 +1848,20 @@ On each node perform the following: sudo cp cert.pem /etc/gitlab/trusted-certs/ ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + +1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: + + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure + ``` + + Only a single designated node should handle migrations as detailed in the + [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section. + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Run `sudo gitlab-rake gitlab:gitaly:check` to confirm the node can connect to Gitaly. 1. Tail the logs to see the requests: @@ -1827,11 +1869,6 @@ On each node perform the following: sudo gitlab-ctl tail gitaly ``` -1. Save the `/etc/gitlab/gitlab-secrets.json` file from one of the two - application nodes and install it on the other application node, the - [Gitaly node](#configure-gitaly) and the [Sidekiq node](#configure-sidekiq) and - [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). - 1. Verify the GitLab services are running: ```shell @@ -1890,45 +1927,26 @@ running [Prometheus](../monitoring/prometheus/index.md) and 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - external_url 'http://gitlab.example.com' + roles(['monitoring_role', 'consul_role']) - # Disable all other services - alertmanager['enable'] = false - gitaly['enable'] = false - gitlab_exporter['enable'] = false - gitlab_workhorse['enable'] = false - nginx['enable'] = true - postgres_exporter['enable'] = false - postgresql['enable'] = false - redis['enable'] = false - redis_exporter['enable'] = false - sidekiq['enable'] = false - puma['enable'] = false - unicorn['enable'] = false - node_exporter['enable'] = false - gitlab_exporter['enable'] = false + external_url 'http://gitlab.example.com' - # Enable Prometheus - prometheus['enable'] = true + # Prometheus prometheus['listen_address'] = '0.0.0.0:9090' prometheus['monitor_kubernetes'] = false - # Enable Login form - grafana['disable_login_form'] = false - - # Enable Grafana - grafana['enable'] = true + # Grafana grafana['admin_password'] = '<grafana_password>' + grafana['disable_login_form'] = false # Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true consul['configuration'] = { retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } - # Prevent database migrations from running on upgrade automatically - gitlab_rails['auto_migrate'] = false + # Nginx - For Grafana access + nginx['enable'] = true ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index 6698737af6a..49024365e30 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -138,12 +138,12 @@ As long as at least one of each component is online and capable of handling the ### Automated database failover **(PREMIUM SELF)** > - Level of complexity: **High** -> - Required domain knowledge: PgBouncer, Repmgr or Patroni, shared storage, distributed systems +> - Required domain knowledge: PgBouncer, Patroni, shared storage, distributed systems By adding automatic failover for database systems, you can enable higher uptime with additional database nodes. This extends the default database with cluster management and failover policies. -[PgBouncer in conjunction with Repmgr or Patroni](../postgresql/replication_and_failover.md) +[PgBouncer in conjunction with Patroni](../postgresql/replication_and_failover.md) is recommended. ### Instance level replication with GitLab Geo **(PREMIUM SELF)** diff --git a/doc/administration/reference_architectures/troubleshooting.md b/doc/administration/reference_architectures/troubleshooting.md index d5f57965a80..4b07cff7de2 100644 --- a/doc/administration/reference_architectures/troubleshooting.md +++ b/doc/administration/reference_architectures/troubleshooting.md @@ -206,211 +206,8 @@ To make sure your configuration is correct: ## Troubleshooting Gitaly -If you have any problems when using standalone Gitaly nodes, first -[check all the versions are up to date](../gitaly/index.md#check-versions-when-using-standalone-gitaly-servers). - -### `gitaly-debug` - -The `gitaly-debug` command provides "production debugging" tools for Gitaly and Git -performance. It is intended to help production engineers and support -engineers investigate Gitaly performance problems. - -If you're using GitLab 11.6 or newer, this tool should be installed on -your GitLab / Gitaly server already at `/opt/gitlab/embedded/bin/gitaly-debug`. -If you're investigating an older GitLab version you can compile this -tool offline and copy the executable to your server: - -```shell -git clone https://gitlab.com/gitlab-org/gitaly.git -cd cmd/gitaly-debug -GOOS=linux GOARCH=amd64 go build -o gitaly-debug -``` - -To see the help page of `gitaly-debug` for a list of supported sub-commands, run: - -```shell -gitaly-debug -h -``` - -### Commits, pushes, and clones return a 401 - -```plaintext -remote: GitLab: 401 Unauthorized -``` - -You will need to sync your `gitlab-secrets.json` file with your GitLab -app nodes. - -### Client side gRPC logs - -Gitaly uses the [gRPC](https://grpc.io/) RPC framework. The Ruby gRPC -client has its own log file which may contain useful information when -you are seeing Gitaly errors. You can control the log level of the -gRPC client with the `GRPC_LOG_LEVEL` environment variable. The -default level is `WARN`. - -You can run a gRPC trace with: - -```shell -sudo GRPC_TRACE=all GRPC_VERBOSITY=DEBUG gitlab-rake gitlab:gitaly:check -``` - -### Observing `gitaly-ruby` traffic - -[`gitaly-ruby`](../gitaly/configure_gitaly.md#gitaly-ruby) is an internal implementation detail of Gitaly, -so, there's not that much visibility into what goes on inside -`gitaly-ruby` processes. - -If you have Prometheus set up to scrape your Gitaly process, you can see -request rates and error codes for individual RPCs in `gitaly-ruby` by -querying `grpc_client_handled_total`. Strictly speaking, this metric does -not differentiate between `gitaly-ruby` and other RPCs, but in practice -(as of GitLab 11.9), all gRPC calls made by Gitaly itself are internal -calls from the main Gitaly process to one of its `gitaly-ruby` sidecars. - -Assuming your `grpc_client_handled_total` counter only observes Gitaly, -the following query shows you RPCs are (most likely) internally -implemented as calls to `gitaly-ruby`: - -```prometheus -sum(rate(grpc_client_handled_total[5m])) by (grpc_method) > 0 -``` - -### Repository changes fail with a `401 Unauthorized` error - -If you're running Gitaly on its own server and notice that users can -successfully clone and fetch repositories (via both SSH and HTTPS), but can't -push to them or make changes to the repository in the web UI without getting a -`401 Unauthorized` message, then it's possible Gitaly is failing to authenticate -with the other nodes due to having the wrong secrets file. - -Confirm the following are all true: - -- When any user performs a `git push` to any repository on this Gitaly node, it - fails with the following error (note the `401 Unauthorized`): - - ```shell - remote: GitLab: 401 Unauthorized - To <REMOTE_URL> - ! [remote rejected] branch-name -> branch-name (pre-receive hook declined) - error: failed to push some refs to '<REMOTE_URL>' - ``` - -- When any user adds or modifies a file from the repository using the GitLab - UI, it immediately fails with a red `401 Unauthorized` banner. -- Creating a new project and [initializing it with a README](../../user/project/working_with_projects.md#blank-projects) - successfully creates the project but doesn't create the README. -- When [tailing the logs](https://docs.gitlab.com/omnibus/settings/logs.html#tail-logs-in-a-console-on-the-server) on an app node and reproducing the error, you get `401` errors - when reaching the [`/api/v4/internal/allowed`](../../development/internal_api.md) endpoint: - - ```shell - # api_json.log - { - "time": "2019-07-18T00:30:14.967Z", - "severity": "INFO", - "duration": 0.57, - "db": 0, - "view": 0.57, - "status": 401, - "method": "POST", - "path": "\/api\/v4\/internal\/allowed", - "params": [ - { - "key": "action", - "value": "git-receive-pack" - }, - { - "key": "changes", - "value": "REDACTED" - }, - { - "key": "gl_repository", - "value": "REDACTED" - }, - { - "key": "project", - "value": "\/path\/to\/project.git" - }, - { - "key": "protocol", - "value": "web" - }, - { - "key": "env", - "value": "{\"GIT_ALTERNATE_OBJECT_DIRECTORIES\":[],\"GIT_ALTERNATE_OBJECT_DIRECTORIES_RELATIVE\":[],\"GIT_OBJECT_DIRECTORY\":null,\"GIT_OBJECT_DIRECTORY_RELATIVE\":null}" - }, - { - "key": "user_id", - "value": "2" - }, - { - "key": "secret_token", - "value": "[FILTERED]" - } - ], - "host": "gitlab.example.com", - "ip": "REDACTED", - "ua": "Ruby", - "route": "\/api\/:version\/internal\/allowed", - "queue_duration": 4.24, - "gitaly_calls": 0, - "gitaly_duration": 0, - "correlation_id": "XPUZqTukaP3" - } - - # nginx_access.log - [IP] - - [18/Jul/2019:00:30:14 +0000] "POST /api/v4/internal/allowed HTTP/1.1" 401 30 "" "Ruby" - ``` - -To fix this problem, confirm that your `gitlab-secrets.json` file -on the Gitaly node matches the one on all other nodes. If it doesn't match, -update the secrets file on the Gitaly node to match the others, then -[reconfigure the node](../restart_gitlab.md#omnibus-gitlab-reconfigure). - -### Command line tools cannot connect to Gitaly - -If you are having trouble connecting to a Gitaly node with command line (CLI) tools, and certain actions result in a `14: Connect Failed` error message, it means that gRPC cannot reach your Gitaly node. - -Verify that you can reach Gitaly via TCP: - -```shell -sudo gitlab-rake gitlab:tcp_check[GITALY_SERVER_IP,GITALY_LISTEN_PORT] -``` - -If the TCP connection fails, check your network settings and your firewall rules. If the TCP connection succeeds, your networking and firewall rules are correct. - -If you use proxy servers in your command line environment, such as Bash, these can interfere with your gRPC traffic. - -If you use Bash or a compatible command line environment, run the following commands to determine whether you have proxy servers configured: - -```shell -echo $http_proxy -echo $https_proxy -``` - -If either of these variables have a value, your Gitaly CLI connections may be getting routed through a proxy which cannot connect to Gitaly. - -To remove the proxy setting, run the following commands (depending on which variables had values): - -```shell -unset http_proxy -unset https_proxy -``` - -### Gitaly not listening on new address after reconfiguring - -When updating the `gitaly['listen_addr']` or `gitaly['prometheus_listen_addr']` values, Gitaly may continue to listen on the old address after a `sudo gitlab-ctl reconfigure`. - -When this occurs, performing a `sudo gitlab-ctl restart` will resolve the issue. This will no longer be necessary after [this issue](https://gitlab.com/gitlab-org/gitaly/-/issues/2521) is resolved. - -### Permission denied errors appearing in Gitaly logs when accessing repositories from a standalone Gitaly node - -If this error occurs even though file permissions are correct, it's likely that -the Gitaly node is experiencing -[clock drift](https://en.wikipedia.org/wiki/Clock_drift). - -Please ensure that the GitLab and Gitaly nodes are synchronized and use an NTP time -server to keep them synchronized if possible. +For troubleshooting information, see Gitaly and Gitaly Cluster +[troubleshooting information](../gitaly/index.md). ## Troubleshooting the GitLab Rails application diff --git a/doc/administration/reply_by_email_postfix_setup.md b/doc/administration/reply_by_email_postfix_setup.md index f0b9daead69..aae5475312f 100644 --- a/doc/administration/reply_by_email_postfix_setup.md +++ b/doc/administration/reply_by_email_postfix_setup.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Set up Postfix for incoming email +# Set up Postfix for incoming email **(FREE SELF)** This document will take you through the steps of setting up a basic Postfix mail server with IMAP authentication on Ubuntu, to be used with [incoming email](incoming_email.md). diff --git a/doc/administration/repository_checks.md b/doc/administration/repository_checks.md index d603e5d8c92..869b1e7068f 100644 --- a/doc/administration/repository_checks.md +++ b/doc/administration/repository_checks.md @@ -41,9 +41,12 @@ in the [`repocheck.log` file](logs.md#repochecklog) on disk: - `/var/log/gitlab/gitlab-rails` for Omnibus GitLab installations - `/home/git/gitlab/log` for installations from source -If the periodic repository check causes false alarms, you can clear all repository check states by -going to **Admin Area > Settings > Repository** -(`/admin/application_settings/repository`) and clicking **Clear all repository checks**. +If the periodic repository check causes false alarms, you can clear all repository check states by: + +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > Repository** (`/admin/application_settings/repository`). +1. Expand the **Repository maintenance** section. +1. Select **Clear all repository checks**. ## Run a check manually diff --git a/doc/administration/repository_storage_paths.md b/doc/administration/repository_storage_paths.md index cea2144122b..a1391f3e0ed 100644 --- a/doc/administration/repository_storage_paths.md +++ b/doc/administration/repository_storage_paths.md @@ -7,13 +7,11 @@ type: reference, howto # Repository storage **(FREE SELF)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/4578) in GitLab 8.10. - GitLab stores [repositories](../user/project/repository/index.md) on repository storage. Repository storage is either: - A `gitaly_address`, which points to a [Gitaly node](gitaly/index.md). -- A `path`, which points directly a directory where the repositories are stored. This method is +- A `path`, which points directly to the directory where the repositories are stored. This method is deprecated and [scheduled to be removed](https://gitlab.com/gitlab-org/gitaly/-/issues/1690) in GitLab 14.0. @@ -142,8 +140,9 @@ Omnibus stores the repositories in a `repositories` subdirectory of the `git-dat After you [configure](#configure-repository-storage-paths) multiple repository storage paths, you can choose where new repositories are stored: -1. Go to the Admin Area (**{admin}**). -1. Go to **Settings > Repository** and expand the **Repository storage** section. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > Repository** and expand the **Repository storage** + section. 1. Enter values in the **Storage nodes for new repositories** fields. 1. Select **Save changes**. @@ -156,5 +155,5 @@ often it is chosen. That is, `(storage weight) / (sum of all weights) * 100 = ch ## Move repositories -To move a repository to a different repository path, use the same process as -[migrating to Gitaly Cluster](gitaly/praefect.md#migrate-to-gitaly-cluster). +To move a repository to a different repository storage (for example, from `default` to `storage2`), use the +same process as [migrating to Gitaly Cluster](gitaly/praefect.md#migrate-to-gitaly-cluster). diff --git a/doc/administration/repository_storage_types.md b/doc/administration/repository_storage_types.md index 21bb11226ce..f55bff1bf34 100644 --- a/doc/administration/repository_storage_types.md +++ b/doc/administration/repository_storage_types.md @@ -80,8 +80,8 @@ Administrators can look up a project's hashed path from its name or ID using: To look up a project's hash path in the Admin Area: -1. Go to the **Admin Area** (**{admin}**). -1. Go to **Overview > Projects** and select the project. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Overview > Projects** and select the project. The **Gitaly relative path** is displayed there and looks similar to: diff --git a/doc/administration/sidekiq.md b/doc/administration/sidekiq.md index a97eff23c2f..95fef1255af 100644 --- a/doc/administration/sidekiq.md +++ b/doc/administration/sidekiq.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Configuring Sidekiq +# Configuring Sidekiq **(FREE SELF)** This section discusses how to configure an external Sidekiq instance using the bundled Sidekiq in the GitLab package. @@ -26,7 +26,7 @@ you want using steps 1 and 2 from the GitLab downloads page. ## Optional: Enable extra Sidekiq processes sidekiq_cluster['enable'] = true sidekiq['queue_groups'] = [ - "elastic_indexer", + "elastic_commit_indexer", "*" ] ``` @@ -187,4 +187,5 @@ gitlab_rails['monitoring_whitelist'] = ['10.10.1.42', '127.0.0.1'] Related Sidekiq configuration: 1. [Extra Sidekiq processes](operations/extra_sidekiq_processes.md) +1. [Extra Sidekiq routing](operations/extra_sidekiq_routing.md) 1. [Using the GitLab-Sidekiq chart](https://docs.gitlab.com/charts/charts/gitlab/sidekiq/) diff --git a/doc/administration/smime_signing_email.md b/doc/administration/smime_signing_email.md index 7492bf4457a..ebc1723076b 100644 --- a/doc/administration/smime_signing_email.md +++ b/doc/administration/smime_signing_email.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Signing outgoing email with S/MIME +# Signing outgoing email with S/MIME **(FREE SELF)** Notification emails sent by GitLab can be signed with S/MIME for improved security. diff --git a/doc/administration/static_objects_external_storage.md b/doc/administration/static_objects_external_storage.md index fcd2bbc035f..48b98156b4f 100644 --- a/doc/administration/static_objects_external_storage.md +++ b/doc/administration/static_objects_external_storage.md @@ -16,7 +16,8 @@ from an external storage, such as a Content Delivery Network (CDN). To configure external storage for static objects: -1. Go to **Admin Area > Settings > Repository**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > Repository**. 1. Expand the **Repository static objects** section. 1. Enter the base URL and an arbitrary token. When you [set up external storage](#set-up-external-storage), use a script that sets these values as `ORIGIN_HOSTNAME` and `STORAGE_TOKEN`. diff --git a/doc/administration/terraform_state.md b/doc/administration/terraform_state.md index 898d98495d9..388ae74f207 100644 --- a/doc/administration/terraform_state.md +++ b/doc/administration/terraform_state.md @@ -23,7 +23,7 @@ Use [external object storage](https://docs.gitlab.com/charts/advanced/external-o ## Disabling Terraform state To disable terraform state site-wide, follow the steps below. -A GitLab administrator may want to disable Terraform state to reduce diskspace or if Terraform is not used in your instance. +A GitLab administrator may want to disable Terraform state to reduce disk space or if Terraform is not used in your instance. To do so, follow the steps below according to your installation's type. **In Omnibus installations:** diff --git a/doc/administration/timezone.md b/doc/administration/timezone.md index 6f460ed0ea8..bc94110c5b6 100644 --- a/doc/administration/timezone.md +++ b/doc/administration/timezone.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Changing your time zone +# Changing your time zone **(FREE SELF)** The global time zone configuration parameter can be changed in `config/gitlab.yml`: diff --git a/doc/administration/troubleshooting/debug.md b/doc/administration/troubleshooting/debug.md index 5a8ee1c5c94..6861cdcde4e 100644 --- a/doc/administration/troubleshooting/debug.md +++ b/doc/administration/troubleshooting/debug.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Debugging Tips +# Debugging tips **(FREE SELF)** Sometimes things don't work the way they should. Here are some tips on debugging issues out in production. @@ -26,7 +26,7 @@ You can enable output of Active Record debug logging in the Rails console session by running: ```ruby -ActiveRecord::Base.logger = Logger.new(STDOUT) +ActiveRecord::Base.logger = Logger.new($stdout) ``` This will show information about database queries triggered by any Ruby code @@ -155,7 +155,7 @@ and more. However, this is not enabled by default. To enable it, define the gitlab_rails['env'] = {"ENABLE_RBTRACE" => "1"} ``` -Then reconfigure the system and restart Unicorn and Sidekiq. To run this +Then reconfigure the system and restart Puma and Sidekiq. To run this in Omnibus, run as root: ```ruby @@ -178,7 +178,7 @@ following tips are only recommended if you do NOT mind users being affected by downtime. Otherwise skip to the next section. 1. Load the problematic URL -1. Run `sudo gdb -p <PID>` to attach to the Unicorn process. +1. Run `sudo gdb -p <PID>` to attach to the Puma process. 1. In the GDB window, type: ```plaintext @@ -186,7 +186,7 @@ downtime. Otherwise skip to the next section. ``` 1. This forces the process to generate a Ruby backtrace. Check - `/var/log/gitlab/unicorn/unicorn_stderr.log` for the backtrace. For example, you may see: + `/var/log/gitlab/puma/puma_stderr.log` for the backtrace. For example, you may see: ```plaintext from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:33:in `block in start' @@ -213,16 +213,19 @@ downtime. Otherwise skip to the next section. exit ``` -Note that if the Unicorn process terminates before you are able to run these +Note that if the Puma process terminates before you are able to run these commands, GDB will report an error. To buy more time, you can always raise the -Unicorn timeout. For omnibus users, you can edit `/etc/gitlab/gitlab.rb` and -increase it from 60 seconds to 300: +Puma worker timeout. For omnibus users, you can edit `/etc/gitlab/gitlab.rb` and +increase it from 60 seconds to 600: ```ruby -unicorn['worker_timeout'] = 300 +gitlab_rails['env'] = { + 'GITLAB_RAILS_RACK_TIMEOUT' => 600 +} ``` -For source installations, edit `config/unicorn.rb`. +For source installations, set the environment variable. +Refer to [Puma Worker timeout](https://docs.gitlab.com/omnibus/settings/puma.html#worker-timeout). [Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab for the changes to take effect. @@ -267,7 +270,7 @@ is a Unicorn worker that is spinning via `top`. Try to use the `gdb` techniques above. In addition, using `strace` may help isolate issues: ```shell -strace -ttTfyyy -s 1024 -p <PID of unicorn worker> -o /tmp/unicorn.txt +strace -ttTfyyy -s 1024 -p <PID of puma worker> -o /tmp/puma.txt ``` If you cannot isolate which Unicorn worker is the issue, try to run `strace` @@ -275,10 +278,10 @@ on all the Unicorn workers to see where the [`/internal/allowed`](../../development/internal_api.md) endpoint gets stuck: ```shell -ps auwx | grep unicorn | awk '{ print " -p " $2}' | xargs strace -ttTfyyy -s 1024 -o /tmp/unicorn.txt +ps auwx | grep puma | awk '{ print " -p " $2}' | xargs strace -ttTfyyy -s 1024 -o /tmp/puma.txt ``` -The output in `/tmp/unicorn.txt` may help diagnose the root cause. +The output in `/tmp/puma.txt` may help diagnose the root cause. ## More information diff --git a/doc/administration/troubleshooting/defcon.md b/doc/administration/troubleshooting/defcon.md index 09e11553d97..7cae6ea1c8f 100644 --- a/doc/administration/troubleshooting/defcon.md +++ b/doc/administration/troubleshooting/defcon.md @@ -5,21 +5,21 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Disaster Recovery +# Disaster recovery **(FREE SELF)** -This document describes a feature that allows to easily disable some important but computationally -expensive parts of the application, in order to relieve stress on the database in an ongoing downtime. +This document describes a feature that allows you to disable some important but computationally +expensive parts of the application to relieve stress on the database during an ongoing downtime. ## `ci_queueing_disaster_recovery` -This feature flag, if enabled temporarily disables fair scheduling on shared runners. -This can help reduce system resource usage on the `jobs/request` endpoint -by significantly reducing computations being performed. +This feature flag, if temporarily enabled, disables fair scheduling on shared runners. +This can help to reduce system resource usage on the `jobs/request` endpoint +by significantly reducing the computations being performed. Side effects: -- In case of a large backlog of jobs, the jobs will be processed in the order -they were put in the system instead of balancing the jobs across many projects +- In case of a large backlog of jobs, the jobs are processed in the order + they were put in the system, instead of balancing the jobs across many projects. - Projects which are out of quota will be run. This affects -only jobs that were created during the last hour, as prior jobs are canceled -by a periodic background worker (`StuckCiJobsWorker`). + only jobs created during the last hour, as prior jobs are canceled + by a periodic background worker (`StuckCiJobsWorker`). diff --git a/doc/administration/troubleshooting/diagnostics_tools.md b/doc/administration/troubleshooting/diagnostics_tools.md index 27a7493b318..fe85b5d5803 100644 --- a/doc/administration/troubleshooting/diagnostics_tools.md +++ b/doc/administration/troubleshooting/diagnostics_tools.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Diagnostics tools +# Diagnostics tools **(FREE SELF)** These are some of the diagnostics tools the GitLab Support team uses during troubleshooting. They are listed here for transparency, and they may be useful for users with experience diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index 588be73e786..92070a86a0d 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -100,7 +100,7 @@ Rails.cache.instance_variable_get(:@data).keys ```ruby # Before 11.6.0 -logger = Logger.new(STDOUT) +logger = Logger.new($stdout) admin_token = User.find_by_username('ADMIN_USERNAME').personal_access_tokens.first.token app.get("URL/?private_token=#{admin_token}") @@ -113,7 +113,7 @@ Gitlab::Profiler.with_user(admin) { app.get(url) } ## Using the GitLab profiler inside console (used as of 10.5) ```ruby -logger = Logger.new(STDOUT) +logger = Logger.new($stdout) admin = User.find_by_username('ADMIN_USERNAME') Gitlab::Profiler.profile('URL', logger: logger, user: admin) ``` @@ -279,6 +279,21 @@ p.each do |project| end ``` +## Bulk update to change all the Jira integrations to Jira instance-level values + +To change all Jira project to use the instance-level integration settings: + +1. In a Rails console: + + ```ruby + jira_service_instance_id = JiraService.find_by(instance: true).id + JiraService.where(active: true, instance: false, template: false, inherit_from_id: nil).find_each do |service| + service.update_attribute(:inherit_from_id, jira_service_instance_id) + end + ``` + +1. Modify and save again the instance-level integration from the UI to propagate the changes to all the group-level and project-level integrations. + ### Bulk update to disable the Slack Notification service To disable notifications for all projects that have Slack service enabled, do: @@ -302,7 +317,18 @@ the displayed size may still show old sizes or commit numbers. To force an updat p = Project.find_by_full_path('<namespace>/<project>') pp p.statistics p.statistics.refresh! -pp p.statistics # compare with earlier values +pp p.statistics +# compare with earlier values + +# check the total artifact storage space separately +builds_with_artifacts = p.builds.with_downloadable_artifacts.all + +artifact_storage = 0 +builds_with_artifacts.find_each do |build| + artifact_storage += build.artifacts_size +end + +puts "#{artifact_storage} bytes" ``` ### Identify deploy keys associated with blocked and non-member users @@ -341,6 +367,16 @@ DeployKeysProject.with_write_access.find_each do |deploy_key_mapping| end ``` +### Find projects using an SQL query + +Find and store an array of projects based on an SQL query: + +```ruby +# Finds projects that end with '%ject' +projects = Project.find_by_sql("SELECT * FROM projects WHERE name LIKE '%ject'") +=> [#<Project id:12 root/my-first-project>>, #<Project id:13 root/my-second-project>>] +``` + ## Wikis ### Recreate @@ -524,7 +560,8 @@ User.billable.count Using cURL and jq (up to a max 100, see the [pagination docs](../../api/README.md#pagination)): ```shell -curl --silent --header "Private-Token: ********************" "https://gitlab.example.com/api/v4/users?per_page=100&active" | jq --compact-output '.[] | [.id,.name,.username]' +curl --silent --header "Private-Token: ********************" \ + "https://gitlab.example.com/api/v4/users?per_page=100&active" | jq --compact-output '.[] | [.id,.name,.username]' ``` ### Block or Delete Users that have no projects or groups @@ -694,6 +731,16 @@ emails.each do |e| end ``` +### Find groups using an SQL query + +Find and store an array of groups based on an SQL query: + +```ruby +# Finds groups and subgroups that end with '%oup' +Group.find_by_sql("SELECT * FROM namespaces WHERE name LIKE '%oup'") +=> [#<Group id:3 @test-group>, #<Group id:4 @template-group/template-subgroup>] +``` + ## Routes ### Remove redirecting routes @@ -839,7 +886,7 @@ License.current.trial? ### Check if a project feature is available on the instance -Features listed in <https://gitlab.com/gitlab-org/gitlab/blob/master/ee/app/models/license.rb>. +Features listed in <https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/models/license.rb>. ```ruby License.current.feature_available?(:jira_dev_panel_integration) @@ -847,7 +894,7 @@ License.current.feature_available?(:jira_dev_panel_integration) ### Check if a project feature is available in a project -Features listed in [`license.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/app/models/license.rb). +Features listed in [`license.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/models/license.rb). ```ruby p = Project.find_by_full_path('<group>/<project>') @@ -863,55 +910,6 @@ license.save License.current # check to make sure it applied ``` -## Unicorn - -From [Zendesk ticket #91083](https://gitlab.zendesk.com/agent/tickets/91083) (internal) - -### Poll Unicorn requests by seconds - -```ruby -require 'rubygems' -require 'unicorn' - -# Usage for this program -def usage - puts "ruby unicorn_status.rb <path to unix socket> <poll interval in seconds>" - puts "Polls the given Unix socket every interval in seconds. Will not allow you to drop below 3 second poll intervals." - puts "Example: /opt/gitlab/embedded/bin/ruby poll_unicorn.rb /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket 10" -end - -# Look for required args. Throw usage and exit if they don't exist. -if ARGV.count < 2 - usage - exit 1 -end - -# Get the socket and threshold values. -socket = ARGV[0] -threshold = (ARGV[1]).to_i - -# Check threshold - is it less than 3? If so, set to 3 seconds. Safety first! -if threshold.to_i < 3 - threshold = 3 -end - -# Check - does that socket exist? -unless File.exist?(socket) - puts "Socket file not found: #{socket}" - exit 1 -end - -# Poll the given socket every THRESHOLD seconds as specified above. -puts "Running infinite loop. Use CTRL+C to exit." -puts "------------------------------------------" -loop do - Raindrops::Linux.unix_listener_stats([socket]).each do |addr, stats| - puts DateTime.now.to_s + " Active: " + stats.active.to_s + " Queued: " + stats.queued.to_s - end - sleep threshold -end -``` - ## Registry ### Registry Disk Space Usage by Project @@ -1189,6 +1187,28 @@ Prints the metrics saved in `conversational_development_index_metrics`. rake gitlab:usage_data:generate_and_send ``` +## Kubernetes integration + +Find cluster: + +```ruby +cluster = Clusters::Cluster.find(1) +cluster = Clusters::Cluster.find_by(name: 'cluster_name') +``` + +Delete cluster without associated resources: + +```ruby +# Find an admin user +user = User.find_by(username: 'admin_user') + +# Find the cluster with the ID +cluster = Clusters::Cluster.find(1) + +# Delete the cluster +Clusters::DestroyService.new(user).execute(cluster) +``` + ## Elasticsearch ### Configuration attributes @@ -1206,11 +1226,11 @@ Among other attributes, in the output you will notice that all the settings avai You can then set anyone of Elasticsearch integration settings by issuing a command similar to: ```ruby -ApplicationSetting.last.update_attributes(elasticsearch_url: '<your ES URL and port>') +ApplicationSetting.last.update(elasticsearch_url: '<your ES URL and port>') #or -ApplicationSetting.last.update_attributes(elasticsearch_indexing: false) +ApplicationSetting.last.update(elasticsearch_indexing: false) ``` #### Getting attributes diff --git a/doc/administration/troubleshooting/group_saml_scim.md b/doc/administration/troubleshooting/group_saml_scim.md index 63e69589b54..9e9ef492ebd 100644 --- a/doc/administration/troubleshooting/group_saml_scim.md +++ b/doc/administration/troubleshooting/group_saml_scim.md @@ -42,6 +42,10 @@ SCIM mapping:  +Group Sync: + + + ## Okta Basic SAML app configuration: diff --git a/doc/administration/troubleshooting/img/azure_configure_group_claim.png b/doc/administration/troubleshooting/img/azure_configure_group_claim.png Binary files differnew file mode 100644 index 00000000000..31df5fff625 --- /dev/null +++ b/doc/administration/troubleshooting/img/azure_configure_group_claim.png diff --git a/doc/administration/troubleshooting/index.md b/doc/administration/troubleshooting/index.md index 1c205cc987a..75c5ce460e9 100644 --- a/doc/administration/troubleshooting/index.md +++ b/doc/administration/troubleshooting/index.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Troubleshooting a GitLab installation +# Troubleshooting a GitLab installation **(FREE SELF)** This page documents a collection of resources to help you troubleshoot a GitLab installation. diff --git a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md index 9766b2210ca..b43825092b7 100644 --- a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md +++ b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Kubernetes, GitLab and You +# Kubernetes, GitLab, and you **(FREE SELF)** This is a list of useful information regarding Kubernetes that the GitLab Support Team sometimes uses while troubleshooting. GitLab is making this public, so that anyone @@ -147,7 +147,7 @@ and they will assist you with any issues you are having. You can also use `gitlab-rake`, instead of `/usr/local/bin/gitlab-rake`. -- Troubleshooting **Operations > Kubernetes** integration: +- Troubleshooting **Infrastructure > Kubernetes** integration: - Check the output of `kubectl get events -w --all-namespaces`. - Check the logs of pods within `gitlab-managed-apps` namespace. diff --git a/doc/administration/troubleshooting/linux_cheat_sheet.md b/doc/administration/troubleshooting/linux_cheat_sheet.md index c4e991ccc1b..9eadbad171e 100644 --- a/doc/administration/troubleshooting/linux_cheat_sheet.md +++ b/doc/administration/troubleshooting/linux_cheat_sheet.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Linux Cheat Sheet +# Linux cheat sheet **(FREE SELF)** This is the GitLab Support Team's collection of information regarding Linux, that they sometimes use while troubleshooting. It is listed here for transparency, @@ -177,8 +177,8 @@ strace -tt -T -f -y -yy -s 1024 -p <pid> # -o output file -# run strace on all unicorn processes -ps auwx | grep unicorn | awk '{ print " -p " $2}' | xargs strace -tt -T -f -y -yy -s 1024 -o /tmp/unicorn.txt +# run strace on all puma processes +ps auwx | grep puma | awk '{ print " -p " $2}' | xargs strace -tt -T -f -y -yy -s 1024 -o /tmp/puma.txt ``` Be aware that strace can have major impacts to system performance when it is running. diff --git a/doc/administration/troubleshooting/log_parsing.md b/doc/administration/troubleshooting/log_parsing.md index a0f71960e14..c16302dd251 100644 --- a/doc/administration/troubleshooting/log_parsing.md +++ b/doc/administration/troubleshooting/log_parsing.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Parsing GitLab logs with `jq` +# Parsing GitLab logs with `jq` **(FREE SELF)** We recommend using log aggregation and search tools like Kibana and Splunk whenever possible, but if they are not available you can still quickly parse diff --git a/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md b/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md index 481c95b925a..e55118d7309 100644 --- a/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md +++ b/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Navigating GitLab via Rails console +# Navigating GitLab via Rails console **(FREE SELF)** At the heart of GitLab is a web application [built using the Ruby on Rails framework](https://about.gitlab.com/blog/2018/10/29/why-we-use-rails-to-build-gitlab/). @@ -46,7 +46,7 @@ Let's enable debug logging for Active Record so we can see the underlying database queries made: ```ruby -ActiveRecord::Base.logger = Logger.new(STDOUT) +ActiveRecord::Base.logger = Logger.new($stdout) ``` Now, let's try retrieving a user from the database: diff --git a/doc/administration/troubleshooting/postgresql.md b/doc/administration/troubleshooting/postgresql.md index 9565b7594d6..341c6bfbc65 100644 --- a/doc/administration/troubleshooting/postgresql.md +++ b/doc/administration/troubleshooting/postgresql.md @@ -53,8 +53,7 @@ This section is for links to information elsewhere in the GitLab documentation. - [PostgreSQL scaling](../postgresql/replication_and_failover.md) - Including [troubleshooting](../postgresql/replication_and_failover.md#troubleshooting) - `gitlab-ctl repmgr-check-master` (or `gitlab-ctl patroni check-leader` if - you're using Patroni) and PgBouncer errors. + `gitlab-ctl patroni check-leader` and PgBouncer errors. - [Developer database documentation](../../development/README.md#database-guides), some of which is absolutely not for production use. Including: diff --git a/doc/administration/troubleshooting/sidekiq.md b/doc/administration/troubleshooting/sidekiq.md index 297a8355036..7a8ac8c3dbe 100644 --- a/doc/administration/troubleshooting/sidekiq.md +++ b/doc/administration/troubleshooting/sidekiq.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Troubleshooting Sidekiq +# Troubleshooting Sidekiq **(FREE SELF)** Sidekiq is the background job processor GitLab uses to asynchronously run tasks. When things go wrong it can be difficult to troubleshoot. These diff --git a/doc/administration/troubleshooting/ssl.md b/doc/administration/troubleshooting/ssl.md index 7c0b745f8c2..5ce4c7f0fb2 100644 --- a/doc/administration/troubleshooting/ssl.md +++ b/doc/administration/troubleshooting/ssl.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Troubleshooting SSL +# Troubleshooting SSL **(FREE SELF)** This page contains a list of common SSL-related errors and scenarios that you may encounter while working with GitLab. It should serve as an addition to the diff --git a/doc/administration/troubleshooting/test_environments.md b/doc/administration/troubleshooting/test_environments.md index 16ef4f83978..5fe493a536c 100644 --- a/doc/administration/troubleshooting/test_environments.md +++ b/doc/administration/troubleshooting/test_environments.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Apps for a Testing Environment +# Apps for a testing environment **(FREE SELF)** This is the GitLab Support Team's collection of information regarding testing environments, for use while troubleshooting. It is listed here for transparency, and it may be useful diff --git a/doc/administration/troubleshooting/tracing_correlation_id.md b/doc/administration/troubleshooting/tracing_correlation_id.md index ad2b8586b8b..7b9ce5c6d7b 100644 --- a/doc/administration/troubleshooting/tracing_correlation_id.md +++ b/doc/administration/troubleshooting/tracing_correlation_id.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Finding relevant log entries with a correlation ID +# Finding relevant log entries with a correlation ID **(FREE SELF)** In GitLab 11.6 and later, a unique request tracking ID, known as the "correlation ID" has been logged by the GitLab instance for most requests. Each individual request to GitLab gets diff --git a/doc/administration/whats-new.md b/doc/administration/whats-new.md index a5e3a232890..ae19e0f0341 100644 --- a/doc/administration/whats-new.md +++ b/doc/administration/whats-new.md @@ -29,7 +29,8 @@ in the first patch release, such as `13.10.1`. You can configure the What's new variant: -1. Navigate to **Admin Area > Settings > Preferences**, then expand **What's new**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > Preferences**, then expand **What's new**. 1. Choose one of the following options: | Option | Description | |
