diff options
| author | Robert Speicher <rspeicher@gmail.com> | 2021-01-20 13:34:23 -0600 |
|---|---|---|
| committer | Robert Speicher <rspeicher@gmail.com> | 2021-01-20 13:34:23 -0600 |
| commit | 6438df3a1e0fb944485cebf07976160184697d72 (patch) | |
| tree | 00b09bfd170e77ae9391b1a2f5a93ef6839f2597 /doc/administration | |
| parent | 42bcd54d971da7ef2854b896a7b34f4ef8601067 (diff) | |
| download | gitlab-ce-6438df3a1e0fb944485cebf07976160184697d72.tar.gz | |
Add latest changes from gitlab-org/gitlab@13-8-stable-eev13.8.0-rc42
Diffstat (limited to 'doc/administration')
48 files changed, 1453 insertions, 1723 deletions
diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md index 7436661920a..27f6bbcd028 100644 --- a/doc/administration/audit_events.md +++ b/doc/administration/audit_events.md @@ -103,6 +103,8 @@ From there, you can see the following actions: Project events can also be accessed via the [Project Audit Events API](../api/audit_events.md#project-audit-events). +Project event queries are limited to a maximum of 30 days. + ### Instance events **(PREMIUM ONLY)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/2336) in [GitLab Premium](https://about.gitlab.com/pricing/) 9.3. diff --git a/doc/administration/auth/jwt.md b/doc/administration/auth/jwt.md index 448922230e7..59d00265a1b 100644 --- a/doc/administration/auth/jwt.md +++ b/doc/administration/auth/jwt.md @@ -38,7 +38,7 @@ JWT will provide you with a secret key for you to use. algorithm: 'HS256', # Supported algorithms: 'RS256', 'RS384', 'RS512', 'ES256', 'ES384', 'ES512', 'HS256', 'HS384', 'HS512' uid_claim: 'email', required_claims: ['name', 'email'], - info_maps: { name: 'name', email: 'email' }, + info_map: { name: 'name', email: 'email' }, auth_url: 'https://example.com/', valid_within: 3600 # 1 hour } diff --git a/doc/administration/auth/ldap/google_secure_ldap.md b/doc/administration/auth/ldap/google_secure_ldap.md index dcfb77f277e..6fecf74d935 100644 --- a/doc/administration/auth/ldap/google_secure_ldap.md +++ b/doc/administration/auth/ldap/google_secure_ldap.md @@ -20,7 +20,7 @@ The steps below cover: ## Configuring Google LDAP client -1. Navigate to <https://admin.google.com/Dashboard> and sign in as a GSuite domain administrator. +1. Navigate to <https://admin.google.com/Dashboard> and sign in as a Google Workspace domain administrator. 1. Go to **Apps > LDAP > Add Client**. diff --git a/doc/administration/auth/ldap/index.md b/doc/administration/auth/ldap/index.md index 4dedaba3754..de0f123acf1 100644 --- a/doc/administration/auth/ldap/index.md +++ b/doc/administration/auth/ldap/index.md @@ -201,9 +201,9 @@ LDAP attributes that GitLab uses to create an account for the LDAP user. The spe | ------- | ----------- | -------- | -------- | | `username` | The username is used in paths for the user's own projects (like `gitlab.example.com/username/project`) and when mentioning them in issues, merge request and comments (like `@username`). If the attribute specified for `username` contains an email address, the GitLab username is part of the email address before the `@`. | no | `['uid', 'userid', 'sAMAccountName']` | | `email` | LDAP attribute for user email. | no | `['mail', 'email', 'userPrincipalName']` | -| `name` | LDAP attribute for user display name. If no full name could be found at the attribute specified for `name`, the full name is determined using the attributes specified for `first_name` and `last_name`. | no | `'cn'` or `'displayName'` | -| `first_name` | LDAP attribute for user first name. | no | `'givenName'` | -| `last_name` | LDAP attribute for user last name. | no | `'sn'` | +| `name` | LDAP attribute for user display name. If `name` is blank, the full name is taken from the `first_name` and `last_name`. | no | Attributes `'cn'`, or `'displayName'` commonly carry full names. Alternatively, you can force the use of `first_name` and `last_name` by specifying an absent attribute such as `'somethingNonExistent'`. | +| `first_name` | LDAP attribute for user first name. Used when the attribute configured for `name` does not exist. | no | `'givenName'` | +| `last_name` | LDAP attribute for user last name. Used when the attribute configured for `name` does not exist. | no | `'sn'` | ### LDAP Sync Configuration Settings **(STARTER ONLY)** diff --git a/doc/administration/compliance.md b/doc/administration/compliance.md index 85b72d7b18f..6079e838ac6 100644 --- a/doc/administration/compliance.md +++ b/doc/administration/compliance.md @@ -13,17 +13,17 @@ documentation. The [security features](../security/README.md) in GitLab may also help you meet relevant compliance standards. -|Feature |GitLab tier |GitLab.com | -| ---------| :--------: | :-------: | -|**[Restrict SSH Keys](../security/ssh_keys_restrictions.md)**<br>Control the technology and key length of SSH keys used to access GitLab|Core+|| -|**[Granular user roles and flexible permissions](../user/permissions.md)**<br>Manage access and permissions with five different user roles and settings for external users. Set permissions according to people's role, rather than either read or write access to a repository. Don't share the source code with people that only need access to the issue tracker.|Core+|✓| -|**[Enforce TOS acceptance](../user/admin_area/settings/terms.md)**<br>Enforce your users accepting new terms of service by blocking GitLab traffic.|Core+|| -|**[Email all users of a project, group, or entire server](../tools/email.md)**<br>An admin can email groups of users based on project or group membership, or email everyone using the GitLab instance. This is great for scheduled maintenance or upgrades.|Starter+|| -|**[Omnibus package supports log forwarding](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-forwarding)**<br>Forward your logs to a central system.|Starter+|| -|**[Lock project membership to group](../user/group/index.md#member-lock)**<br>Group owners can prevent new members from being added to projects within a group.|Starter+|✓| -|**[LDAP group sync](auth/ldap/index.md#group-sync)**<br>GitLab Enterprise Edition gives admins the ability to automatically sync groups and manage SSH keys, permissions, and authentication, so you can focus on building your product, not configuring your tools.|Starter+|| -|**[LDAP group sync filters](auth/ldap/index.md#group-sync)**<br>GitLab Enterprise Edition Premium gives more flexibility to synchronize with LDAP based on filters, meaning you can leverage LDAP attributes to map GitLab permissions.|Premium+|| -|**[Audit events](audit_events.md)**<br>To maintain the integrity of your code, GitLab Enterprise Edition Premium gives admins the ability to view any modifications made within the GitLab server in an advanced audit events system, so you can control, analyze, and track every change.|Premium+|| -|**[Auditor users](auditor_users.md)**<br>Auditor users are users who are given read-only access to all projects, groups, and other resources on the GitLab instance.|Premium+|| -|**[Credentials inventory](../user/admin_area/credentials_inventory.md)**<br>With a credentials inventory, GitLab administrators can keep track of the credentials used by all of the users in their GitLab instance. |Ultimate|| -|**Separation of Duties using [Protected branches](../user/project/protected_branches.md#protected-branches-approval-by-code-owners) and [custom CI Configuration Paths](../ci/pipelines/settings.md#custom-ci-configuration-path)**<br> GitLab Silver and Premium users can leverage the GitLab cross-project YAML configurations to define deployers of code and developers of code. View the [Separation of Duties Deploy Project](https://gitlab.com/guided-explorations/separation-of-duties-deploy/blob/master/README.md) and [Separation of Duties Project](https://gitlab.com/guided-explorations/separation-of-duties/blob/master/README.md) to see how to use this set up to define these roles.|Premium+|| +|Feature |GitLab tier |GitLab.com | Product level | +| ---------| :--------: | :-------: | :-----------: | +|**[Restrict SSH Keys](../security/ssh_keys_restrictions.md)**<br>Control the technology and key length of SSH keys used to access GitLab|Core+||Instance| +|**[Granular user roles and flexible permissions](../user/permissions.md)**<br>Manage access and permissions with five different user roles and settings for external users. Set permissions according to people's role, rather than either read or write access to a repository. Don't share the source code with people that only need access to the issue tracker.|Core+|✓|Instance, Group, Project| +|**[Enforce TOS acceptance](../user/admin_area/settings/terms.md)**<br>Enforce your users accepting new terms of service by blocking GitLab traffic.|Core+||Instance| +|**[Email all users of a project, group, or entire server](../tools/email.md)**<br>An admin can email groups of users based on project or group membership, or email everyone using the GitLab instance. This is great for scheduled maintenance or upgrades.|Starter+|||Instance +|**[Omnibus package supports log forwarding](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-forwarding)**<br>Forward your logs to a central system.|Starter+||Instance| +|**[Lock project membership to group](../user/group/index.md#member-lock)**<br>Group owners can prevent new members from being added to projects within a group.|Starter+|✓|Group| +|**[LDAP group sync](auth/ldap/index.md#group-sync)**<br>GitLab Enterprise Edition gives admins the ability to automatically sync groups and manage SSH keys, permissions, and authentication, so you can focus on building your product, not configuring your tools.|Starter+||Instance| +|**[LDAP group sync filters](auth/ldap/index.md#group-sync)**<br>GitLab Enterprise Edition Premium gives more flexibility to synchronize with LDAP based on filters, meaning you can leverage LDAP attributes to map GitLab permissions.|Premium+||Instance| +|**[Audit events](audit_events.md)**<br>To maintain the integrity of your code, GitLab Enterprise Edition Premium gives admins the ability to view any modifications made within the GitLab server in an advanced audit events system, so you can control, analyze, and track every change.|Premium+|✓|Instance, Group, Project| +|**[Auditor users](auditor_users.md)**<br>Auditor users are users who are given read-only access to all projects, groups, and other resources on the GitLab instance.|Premium+||Instance| +|**[Credentials inventory](../user/admin_area/credentials_inventory.md)**<br>With a credentials inventory, GitLab administrators can keep track of the credentials used by all of the users in their GitLab instance. |Ultimate||Instance| +|**Separation of Duties using [Protected branches](../user/project/protected_branches.md#protected-branches-approval-by-code-owners) and [custom CI Configuration Paths](../ci/pipelines/settings.md#custom-ci-configuration-path)**<br> GitLab Silver and Premium users can leverage the GitLab cross-project YAML configurations to define deployers of code and developers of code. View the [Separation of Duties Deploy Project](https://gitlab.com/guided-explorations/separation-of-duties-deploy/blob/master/README.md) and [Separation of Duties Project](https://gitlab.com/guided-explorations/separation-of-duties/blob/master/README.md) to see how to use this set up to define these roles.|Premium+|✓|Project| diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index be84b260127..2b0a3e2f114 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -15,7 +15,7 @@ See [Geo limitations](../index.md#limitations) for more information. WARNING: Disaster recovery for multi-secondary configurations is in **Alpha**. -For the latest updates, check the [Disaster Recovery epic for complete maturity](https://gitlab.com/groups/gitlab-org/-/epics/590). +For the latest updates, check the [Disaster Recovery epic for complete maturity](https://gitlab.com/groups/gitlab-org/-/epics/3574). Multi-secondary configurations require the complete re-synchronization and re-configuration of all non-promoted secondaries and will cause downtime. @@ -181,7 +181,7 @@ secondary. If the node is paused, be sure to resume before promoting. This issue has been fixed in GitLab 13.4 and later. WARNING: - If the secondary node [has been paused](../../geo/index.md#pausing-and-resuming-replication), this performs +If the secondary node [has been paused](../../geo/index.md#pausing-and-resuming-replication), this performs a point-in-time recovery to the last known state. Data that was created on the primary while the secondary was paused will be lost. @@ -220,6 +220,75 @@ Data that was created on the primary while the secondary was paused will be lost previously for the **secondary**. 1. Success! The **secondary** has now been promoted to **primary**. +#### Promoting a **secondary** node with a Patroni standby cluster + +The `gitlab-ctl promote-to-primary-node` command cannot be used yet in +conjunction with a Patroni standby cluster, as it can only +perform changes on a **secondary** with only a single machine. Instead, you must +do this manually. + +WARNING: +In GitLab 13.2 and 13.3, promoting a secondary node to a primary while the +secondary is paused fails. Do not pause replication before promoting a +secondary. If the node is paused, be sure to resume before promoting. This +issue has been fixed in GitLab 13.4 and later. + +WARNING: +If the secondary node [has been paused](../../geo/index.md#pausing-and-resuming-replication), this performs +a point-in-time recovery to the last known state. +Data that was created on the primary while the secondary was paused will be lost. + +1. SSH in to the Standby Leader database node in the **secondary** and trigger PostgreSQL to + promote to read-write: + + ```shell + sudo gitlab-ctl promote-db + ``` + +1. Disable Patroni auto-failover: + + ```shell + sudo gitlab-ctl patroni pause + ``` + +1. Edit `/etc/gitlab/gitlab.rb` on every application and Sidekiq nodes in the secondary to reflect its new status as primary by removing any lines that enabled the `geo_secondary_role`: + + ```ruby + ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. + geo_secondary_role['enable'] = true + + ## In 11.5+ documentation, the role was enabled as follows. Remove this line. + roles ['geo_secondary_role'] + ``` + +1. Edit `/etc/gitlab/gitlab.rb` on every Patroni node in the secondary to disable the standby cluster: + + ```ruby + patroni['standby_cluster']['enable'] = false + ``` + +1. Reconfigure GitLab on each machine for the changes to take effect: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +1. Resume Patroni auto-failover: + + ```shell + sudo gitlab-ctl patroni resume + ``` + +1. Promote the **secondary** to **primary**. SSH into a single application server and execute: + + ```shell + sudo gitlab-rake geo:set_secondary_as_primary + ``` + +1. Verify you can connect to the newly promoted **primary** using the URL used previously for the **secondary**. + +1. Success! The **secondary** has now been promoted to **primary**. + #### Promoting a **secondary** node with an external PostgreSQL database The `gitlab-ctl promote-to-primary-node` command cannot be used in conjunction with @@ -278,7 +347,7 @@ required: 1. Verify you can connect to the newly promoted **primary** site using the URL used previously for the **secondary** site. -Success! The **secondary** site has now been promoted to **primary**. +1. Success! The **secondary** site has now been promoted to **primary**. ### Step 4. (Optional) Updating the primary domain DNS record diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md index 334f05ef3ce..1985ea2e04c 100644 --- a/doc/administration/geo/index.md +++ b/doc/administration/geo/index.md @@ -292,12 +292,6 @@ This list of limitations only reflects the latest version of GitLab. If you are ### Limitations on replication/verification -You can keep track of the progress to implement the missing items in -these epics/issues: - -- [Unreplicated Data Types](https://gitlab.com/groups/gitlab-org/-/epics/893) -- [Verify all replicated data](https://gitlab.com/groups/gitlab-org/-/epics/1430) - There is a complete list of all GitLab [data types](replication/datatypes.md) and [existing support for replication and verification](replication/datatypes.md#limitations-on-replicationverification). ## Frequently Asked Questions diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md index 42501e16f5f..5adf256f6fa 100644 --- a/doc/administration/geo/replication/configuration.md +++ b/doc/administration/geo/replication/configuration.md @@ -170,6 +170,11 @@ keys must be manually replicated to the **secondary** node. sudo service sshd reload ``` +1. Verify SSH is still functional. + + SSH into your GitLab **secondary** server in a new terminal. If you are unable to connect, + verify the permissions are correct according to the previous steps. + ### Step 3. Add the **secondary** node 1. SSH into your GitLab **secondary** server and login as root: diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md index 9778e08a30b..6e2ddfb812c 100644 --- a/doc/administration/geo/setup/database.md +++ b/doc/administration/geo/setup/database.md @@ -497,54 +497,135 @@ For instructions about how to set up Patroni on the primary node, see the If you are currently using `repmgr` on your Geo primary, see [these instructions](#migrating-from-repmgr-to-patroni) for migrating from `repmgr` to Patroni. A production-ready and secure setup requires at least three Patroni instances on -the primary, and a similar configuration on the secondary nodes. Be sure to use -password credentials and other database best practices. +the primary site, and a similar configuration on the secondary sites. Be sure to +use password credentials and other database best practices. Similar to `repmgr`, using Patroni on a secondary node is optional. -To set up database replication with Patroni on a secondary node, configure a -_permanent replication slot_ on the primary node's Patroni cluster, and ensure -password authentication is used. - -On Patroni instances for the primary node, add the following to the -`/etc/gitlab/gitlab.rb` file: - -```ruby -# You need one entry for each secondary, with a unique name following PostgreSQL slot_name constraints: -# -# Configuration syntax will be: 'unique_slotname' => { 'type' => 'physical' }, -# We don't support setting a permanent replication slot for logical replication type -patroni['replication_slots'] = { - 'geo_secondary' => { 'type' => 'physical' } -} - -postgresql['md5_auth_cidr_addresses'] = [ - 'PATRONI_PRIMARY1_IP/32', 'PATRONI_PRIMARY2_IP/32', 'PATRONI_PRIMARY3_IP/32', 'PATRONI_PRIMARY_PGBOUNCER/32', - 'PATRONI_SECONDARY1_IP/32', 'PATRONI_SECONDARY2_IP/32', 'PATRONI_SECONDARY3_IP/32' # we list all secondary instances as they can all become a Standby Leader - # any other instance that needs access to the database as per documentation -] - -postgresql['pgbouncer_user_password'] = 'PGBOUNCER_PASSWORD_HASH' -postgresql['sql_replication_password'] = 'POSTGRESQL_REPLICATION_PASSWORD_HASH' -postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' -``` - -On Patroni instances for the secondary node, add the following to the -`/etc/gitlab/gitlab.rb` file: - -```ruby -postgresql['md5_auth_cidr_addresses'] = [ - 'PATRONI_SECONDARY1_IP/32', 'PATRONI_SECONDARY2_IP/32', 'PATRONI_SECONDARY3_IP/32', 'PATRONI_SECONDARY_PGBOUNCER/32', - # any other instance that needs access to the database as per documentation -] - -patroni['enable'] = true -patroni['standby_cluster']['enable'] = true -patroni['standby_cluster']['host'] = 'PATRONI_PRIMARY_LEADER_IP' # this needs to be changed anytime the primary Leader changes -patroni['standby_cluster']['port'] = 5432 -patroni['standby_cluster']['primary_slot_name'] = 'geo_secondary' # or the unique replication slot name you setup before -patroni['replication_password'] = 'PLAIN_TEXT_POSTGRESQL_REPLICATION_PASSWORD' -``` +### Step 1. Configure Patroni permanent replication slot on the primary site + +To set up database replication with Patroni on a secondary node, we need to +configure a _permanent replication slot_ on the primary node's Patroni cluster, +and ensure password authentication is used. + +For each Patroni instance on the primary site **starting on the Patroni +Leader instance**: + +1. SSH into your Patroni instance and login as root: + + ```shell + sudo -i + ``` + +1. Edit `/etc/gitlab/gitlab.rb` and add the following: + + ```ruby + consul['enable'] = true + consul['configuration'] = { + retry_join: %w[CONSUL_PRIMARY1_IP CONSULT_PRIMARY2_IP CONSULT_PRIMARY3_IP] + } + + repmgr['enable'] = false + + # You need one entry for each secondary, with a unique name following PostgreSQL slot_name constraints: + # + # Configuration syntax will be: 'unique_slotname' => { 'type' => 'physical' }, + # We don't support setting a permanent replication slot for logical replication type + patroni['replication_slots'] = { + 'geo_secondary' => { 'type' => 'physical' } + } + + patroni['use_pg_rewind'] = true + patroni['postgresql']['max_wal_senders'] = 8 # Use double of the amount of patroni/reserved slots (3 patronis + 1 reserved slot for a Geo secondary). + patroni['postgresql']['max_replication_slots'] = 8 # Use double of the amount of patroni/reserved slots (3 patronis + 1 reserved slot for a Geo secondary). + + postgresql['md5_auth_cidr_addresses'] = [ + 'PATRONI_PRIMARY1_IP/32', 'PATRONI_PRIMARY2_IP/32', 'PATRONI_PRIMARY3_IP/32', 'PATRONI_PRIMARY_PGBOUNCER/32', + 'PATRONI_SECONDARY1_IP/32', 'PATRONI_SECONDARY2_IP/32', 'PATRONI_SECONDARY3_IP/32', 'PATRONI_SECONDARY_PGBOUNCER/32' # We list all secondary instances as they can all become a Standby Leader + ] + + postgresql['pgbouncer_user_password'] = 'PGBOUNCER_PASSWORD_HASH' + postgresql['sql_replication_password'] = 'POSTGRESQL_REPLICATION_PASSWORD_HASH' + postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' + ``` + +1. Reconfigure GitLab for the changes to take effect: + + ```shell + gitlab-ctl reconfigure + ``` + +### Step 2. Configure a Standby cluster on the secondary site + +NOTE: +If you are converting a secondary site to a Patroni Cluster, you must start +on the PostgreSQL instance. It will become the Patroni Standby Leader instance, +and then you can switchover to another replica if you need. + +For each Patroni instance on the secondary site: + +1. SSH into your Patroni node and login as root: + + ```shell + sudo -i + ``` + +1. Edit `/etc/gitlab/gitlab.rb` and add the following: + + ```ruby + roles ['consul_role', 'postgres_role'] + + consul['enable'] = true + consul['configuration'] = { + retry_join: %w[CONSUL_SECONDARY1_IP CONSULT_SECONDARY2_IP CONSULT_SECONDARY3_IP] + } + + repmgr['enable'] = false + + postgresql['md5_auth_cidr_addresses'] = [ + 'PATRONI_SECONDARY1_IP/32', 'PATRONI_SECONDARY2_IP/32', 'PATRONI_SECONDARY3_IP/32', 'PATRONI_SECONDARY_PGBOUNCER/32', + # Any other instance that needs access to the database as per documentation + ] + + patroni['enable'] = false + patroni['standby_cluster']['enable'] = true + patroni['standby_cluster']['host'] = 'PATRONI_PRIMARY_LEADER_IP' # This needs to be changed anytime the primary Leader changes + patroni['standby_cluster']['port'] = 5432 + patroni['standby_cluster']['primary_slot_name'] = 'geo_secondary' # Or the unique replication slot name you setup before + patroni['replication_password'] = 'PLAIN_TEXT_POSTGRESQL_REPLICATION_PASSWORD' + patroni['use_pg_rewind'] = true + patroni['postgresql']['max_wal_senders'] = 5 # A minimum of three for one replica, plus two for each additional replica + patroni['postgresql']['max_replication_slots'] = 5 # A minimum of three for one replica, plus two for each additional replica + ``` + +1. Reconfigure GitLab for the changes to take effect. + This is required to bootstrap PostgreSQL users and settings: + + ```shell + gitlab-ctl reconfigure + ``` + +1. Remove the PostgreSQL data directory: + + WARNING: + If you are converting a secondary site to a Patroni Cluster, you must skip + this step on the PostgreSQL instance. + + ```shell + rm -rf /var/opt/gitlab/postgresql/data + ``` + +1. Edit `/etc/gitlab/gitlab.rb` to enable Patroni: + + ```ruby + patroni['enable'] = true + ``` + +1. Reconfigure GitLab for the changes to take effect: + + ```shell + gitlab-ctl reconfigure + ``` ## Migrating from repmgr to Patroni diff --git a/doc/administration/gitaly/img/gitlab_gitaly_version_mismatch_v12_4.png b/doc/administration/gitaly/img/gitlab_gitaly_version_mismatch_v12_4.png Binary files differdeleted file mode 100644 index 4d2c5cdb00c..00000000000 --- a/doc/administration/gitaly/img/gitlab_gitaly_version_mismatch_v12_4.png +++ /dev/null diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index e0b0d52fe3f..9577fb40abe 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -395,7 +395,7 @@ Gitaly makes the following assumptions: You can't define Gitaly servers with some as a local Gitaly server (without `gitaly_address`) and some as remote -server (with `gitaly_address`) unless you setup with special +server (with `gitaly_address`) unless you use [mixed configuration](#mixed-configuration). **For Omnibus GitLab** @@ -502,8 +502,8 @@ If it's excluded, default Git storage directory is used for that storage shard. ### Disable Gitaly where not required (optional) -If you are running Gitaly [as a remote service](#run-gitaly-on-its-own-server) you may want to -disable the local Gitaly service that runs on your GitLab server by default, leaving it only running +If you are running Gitaly [as a remote service](#run-gitaly-on-its-own-server), you may want to +disable the local Gitaly service that runs on your GitLab server by default and have it only running where required. Disabling Gitaly on the GitLab instance only makes sense when you run GitLab in a custom cluster configuration, where @@ -1007,7 +1007,7 @@ When GitLab calls a function that has a "Rugged patch", it performs two checks: - Is the feature flag for this patch set in the database? If so, the feature flag setting controls the GitLab use of "Rugged patch" code. -- If the feature flag is not set, GitLab tries accessing the filesystem underneath the +- If the feature flag is not set, GitLab tries accessing the file system underneath the Gitaly server directly. If it can, it uses the "Rugged patch": - If using Unicorn. - If using Puma and [thread count](../../install/requirements.md#puma-threads) is set @@ -1015,9 +1015,9 @@ When GitLab calls a function that has a "Rugged patch", it performs two checks: The result of these checks is cached. -To see if GitLab can access the repository filesystem directly, we use the following heuristic: +To see if GitLab can access the repository file system directly, we use the following heuristic: -- Gitaly ensures that the filesystem has a metadata file in its root with a UUID in it. +- Gitaly ensures that the file system has a metadata file in its root with a UUID in it. - Gitaly reports this UUID to GitLab via the `ServerInfo` RPC. - GitLab Rails tries to read the metadata file directly. If it exists, and if the UUID's match, assume we have direct access. @@ -1044,13 +1044,11 @@ The second facet presents the only real solution. For this, we developed Check [Gitaly timeouts](../../user/admin_area/settings/gitaly_timeouts.md) when troubleshooting Gitaly. -### Checking versions when using standalone Gitaly servers +### Check versions when using standalone Gitaly servers When using standalone Gitaly servers, you must make sure they are the same version -as GitLab to ensure full compatibility. Check **Admin Area > Gitaly Servers** on -your GitLab instance and confirm all Gitaly Servers are `Up to date`. - - +as GitLab to ensure full compatibility. Check **Admin Area > Overview > Gitaly Servers** on +your GitLab instance and confirm all Gitaly servers indicate that they are up to date. ### `gitaly-debug` diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index 6eaafae6015..fe8b3e5f566 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -5,17 +5,16 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Gitaly Cluster +# Gitaly Cluster **(CORE ONLY)** [Gitaly](index.md), the service that provides storage for Git repositories, can be run in a clustered configuration to increase fault tolerance. In this configuration, every Git repository is stored on every Gitaly node in the -cluster. Multiple clusters (or shards), can be configured. +cluster. Multiple clusters (or shards) can be configured. NOTE: -Gitaly Clusters can be created using [GitLab Core](https://about.gitlab.com/pricing/#self-managed) -and higher tiers. However, technical support is limited to GitLab Premium and Ultimate customers -only. Not available in GitLab.com. +Technical support for Gitaly clusters is limited to GitLab Premium and Ultimate +customers. Praefect is a router and transaction manager for Gitaly, and a required component for running a Gitaly Cluster. @@ -1364,7 +1363,7 @@ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.t If your GitLab instance already has repositories on single Gitaly nodes, these aren't migrated to Gitaly Cluster automatically. -Repositories may be moved from one storage location using the [Project repository storage moves API](../../api/project_repository_storage_moves.md): +Project repositories may be moved from one storage location using the [Project repository storage moves API](../../api/project_repository_storage_moves.md): NOTE: The Project repository storage moves API [cannot move all repository types](../../api/project_repository_storage_moves.md#limitations). @@ -1388,6 +1387,8 @@ To move repositories to Gitaly Cluster: using the API to confirm that all projects have moved. No projects should be returned with `repository_storage` field set to the old storage. +In a similar way, you can move Snippet repositories using the [Snippet repository storage moves API](../../api/snippet_repository_storage_moves.md): + ## Debugging Praefect If you receive an error, check `/var/log/gitlab/gitlab-rails/production.log`. diff --git a/doc/administration/housekeeping.md b/doc/administration/housekeeping.md index 90d2f0d916d..178fc438df2 100644 --- a/doc/administration/housekeeping.md +++ b/doc/administration/housekeeping.md @@ -40,3 +40,20 @@ from your project on the same schedule as the `git gc` operation, freeing up sto You can find this option under your project's **Settings > General > Advanced**.  + +## How housekeeping handles pool repositories + +Housekeeping for pool repositories is handled differently from standard repositories. +It is ultimately performed by the Gitaly RPC `FetchIntoObjectPool`. + +This is the current call stack by which it is invoked: + +1. `Repositories::HousekeepingService#execute_gitlab_shell_gc` +1. `GitGarbageCollectWorker#perform` +1. `Projects::GitDeduplicationService#fetch_from_source` +1. `ObjectPool#fetch` +1. `ObjectPoolService#fetch` +1. `Gitaly::FetchIntoObjectPoolRequest` + +To manually invoke it from a Rails console, if needed, you can call `project.pool_repository.object_pool.fetch`. +This is a potentially long-running task, though Gitaly will timeout in about 8 hours. diff --git a/doc/administration/incoming_email.md b/doc/administration/incoming_email.md index 7a49542f8bb..2eb5da7d9ab 100644 --- a/doc/administration/incoming_email.md +++ b/doc/administration/incoming_email.md @@ -251,7 +251,7 @@ incoming_email: #### Gmail -Example configuration for Gmail/G Suite. Assumes mailbox `gitlab-incoming@gmail.com`. +Example configuration for Gmail/Google Workspace. Assumes mailbox `gitlab-incoming@gmail.com`. NOTE: `incoming_email_email` cannot be a Gmail alias account. diff --git a/doc/administration/index.md b/doc/administration/index.md index 32c0b1b0712..f071fde2faa 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -33,14 +33,13 @@ Learn how to install, configure, update, and maintain your GitLab instance. ### Installing GitLab - [Install](../install/README.md): Requirements, directory structures, and installation methods. - - [Database load balancing](database_load_balancing.md): Distribute database queries among multiple database servers. **(STARTER ONLY)** - - [Omnibus support for log forwarding](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-shipping-gitlab-enterprise-edition-only) **(STARTER ONLY)** + - [Database load balancing](database_load_balancing.md): Distribute database queries among multiple database servers. + - [Omnibus support for log forwarding](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-shipping-gitlab-enterprise-edition-only). - [Reference architectures](reference_architectures/index.md): Add additional resources to support more users. - [Installing GitLab on Amazon Web Services (AWS)](../install/aws/index.md): Set up GitLab on Amazon AWS. -- [Geo](geo/index.md): Replicate your GitLab instance to other geographic locations as a read-only fully operational version. **(PREMIUM ONLY)** -- [Disaster Recovery](geo/disaster_recovery/index.md): Quickly fail-over to a different site with minimal effort in a disaster situation. **(PREMIUM ONLY)** -- [Pivotal Tile](../install/pivotal/index.md): Deploy GitLab as a preconfigured appliance using Ops Manager (BOSH) for Pivotal Cloud Foundry. **(PREMIUM ONLY)** -- [Add License](../user/admin_area/license.md): Upload a license at install time to unlock features that are in paid tiers of GitLab. **(STARTER ONLY)** +- [Geo](geo/index.md): Replicate your GitLab instance to other geographic locations as a read-only fully operational version. +- [Disaster Recovery](geo/disaster_recovery/index.md): Quickly fail-over to a different site with minimal effort in a disaster situation. +- [Add License](../user/admin_area/license.md): Upload a license at install time to unlock features that are in paid tiers of GitLab. ### Configuring GitLab @@ -72,11 +71,9 @@ Learn how to install, configure, update, and maintain your GitLab instance. to GitLab users through the UI. - [Elasticsearch](../integration/elasticsearch.md): Enable Elasticsearch to empower Advanced Search. Useful when you deal with a huge amount of data. - **(STARTER ONLY)** -- [External Classification Policy Authorization](../user/admin_area/settings/external_authorization.md) - **(PREMIUM ONLY)** +- [External Classification Policy Authorization](../user/admin_area/settings/external_authorization.md). - [Upload a license](../user/admin_area/license.md): Upload a license to unlock - features that are in paid tiers of GitLab. **(STARTER ONLY)** + features that are in paid tiers of GitLab. - [Admin Area](../user/admin_area/index.md): for self-managed instance-wide configuration and maintenance. - [S/MIME Signing](smime_signing_email.md): how to sign all outgoing notification @@ -90,7 +87,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Favicon](../user/admin_area/appearance.md#favicon): Change the default favicon to your own logo. - [Branded login page](../user/admin_area/appearance.md#sign-in--sign-up-pages): Customize the login page with your own logo, title, and description. - ["New Project" page](../user/admin_area/appearance.md#new-project-pages): Customize the text to be displayed on the page that opens whenever your users create a new project. -- [Additional custom email text](../user/admin_area/settings/email.md#custom-additional-text): Add additional custom text to emails sent from GitLab. **(PREMIUM ONLY)** +- [Additional custom email text](../user/admin_area/settings/email.md#custom-additional-text): Add additional custom text to emails sent from GitLab. ### Maintaining GitLab @@ -127,15 +124,15 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Sign-up restrictions](../user/admin_area/settings/sign_up_restrictions.md): block email addresses of specific domains, or whitelist only specific domains. - [Access restrictions](../user/admin_area/settings/visibility_and_access_controls.md#enabled-git-access-protocols): Define which Git access protocols can be used to talk to GitLab (SSH, HTTP, HTTPS). - [Authentication and Authorization](auth/README.md): Configure external authentication with LDAP, SAML, CAS, and additional providers. - - [Sync LDAP](auth/ldap/index.md) **(STARTER ONLY)** - - [Kerberos authentication](../integration/kerberos.md) **(STARTER ONLY)** + - [Sync LDAP](auth/ldap/index.md). + - [Kerberos authentication](../integration/kerberos.md). - See also other [authentication](../topics/authentication/index.md#gitlab-administrators) topics (for example, enforcing 2FA). -- [Email users](../tools/email.md): Email GitLab users from within GitLab. **(STARTER ONLY)** +- [Email users](../tools/email.md): Email GitLab users from within GitLab. - [User Cohorts](../user/admin_area/analytics/user_cohorts.md): Display the monthly cohorts of new users and their activities over time. - [Audit events](audit_events.md): View the changes made within the GitLab server for: - - Groups and projects. **(STARTER)** - - Instances. **(PREMIUM ONLY)** -- [Auditor users](auditor_users.md): Users with read-only access to all projects, groups, and other resources on the GitLab instance. **(PREMIUM ONLY)** + - Groups and projects. + - Instances. +- [Auditor users](auditor_users.md): Users with read-only access to all projects, groups, and other resources on the GitLab instance. - [Incoming email](incoming_email.md): Configure incoming emails to allow users to [reply by email](reply_by_email.md), create [issues by email](../user/project/issues/managing_issues.md#new-issue-via-email) and [merge requests by email](../user/project/merge_requests/creating_merge_requests.md#new-merge-request-by-email), and to enable [Service Desk](../user/project/service_desk.md). @@ -143,7 +140,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. basic Postfix mail server with IMAP authentication on Ubuntu for incoming emails. - [Abuse reports](../user/admin_area/abuse_reports.md): View and resolve abuse reports from your users. -- [Credentials Inventory](../user/admin_area/credentials_inventory.md): With Credentials inventory, GitLab administrators can keep track of the credentials used by their users in their GitLab self-managed instance. **(ULTIMATE ONLY)** +- [Credentials Inventory](../user/admin_area/credentials_inventory.md): With Credentials inventory, GitLab administrators can keep track of the credentials used by their users in their GitLab self-managed instance. ## Project settings @@ -151,7 +148,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Gitaly](gitaly/index.md): Configuring Gitaly, the Git repository storage service for GitLab. - [Default labels](../user/admin_area/labels.md): Create labels that are automatically added to every new project. - [Restrict the use of public or internal projects](../public_access/public_access.md#restricting-the-use-of-public-or-internal-projects): Restrict the use of visibility levels for users when they create a project or a snippet. -- [Custom project templates](../user/admin_area/custom_project_templates.md): Configure a set of projects to be used as custom templates when creating a new project. **(PREMIUM ONLY)** +- [Custom project templates](../user/admin_area/custom_project_templates.md): Configure a set of projects to be used as custom templates when creating a new project. ## Package Registry administration @@ -165,7 +162,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Repository storage paths](repository_storage_paths.md): Manage the paths used to store repositories. - [Repository storage types](repository_storage_types.md): Information about the different repository storage types. - [Repository storage Rake tasks](raketasks/storage.md): A collection of Rake tasks to list and migrate existing projects and attachments associated with it from Legacy storage to Hashed storage. -- [Limit repository size](../user/admin_area/settings/account_and_limit_settings.md): Set a hard limit for your repositories' size. **(STARTER ONLY)** +- [Limit repository size](../user/admin_area/settings/account_and_limit_settings.md): Set a hard limit for your repositories' size. - [Static objects external storage](static_objects_external_storage.md): Set external storage for static objects in a repository. ## Continuous Integration settings @@ -176,7 +173,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Job artifacts](job_artifacts.md): Enable, disable, and configure job artifacts (a set of files and directories which are outputted by a job when it completes successfully). - [Job logs](job_logs.md): Information about the job logs. - [Register runners](../ci/runners/README.md#types-of-runners): Learn how to register and configure runners. -- [Shared runners pipelines quota](../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota): Limit the usage of pipeline minutes for shared runners. **(STARTER ONLY)** +- [Shared runners pipelines quota](../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota): Limit the usage of pipeline minutes for shared runners. - [Enable/disable Auto DevOps](../topics/autodevops/index.md#enablingdisabling-auto-devops): Enable or disable Auto DevOps for your instance. ## Snippet settings @@ -213,7 +210,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. ## Analytics -- [Pseudonymizer](pseudonymizer.md): Export data from a GitLab database to CSV files in a secure way. **(ULTIMATE)** +- [Pseudonymizer](pseudonymizer.md): Export data from a GitLab database to CSV files in a secure way. ## Troubleshooting diff --git a/doc/administration/integration/plantuml.md b/doc/administration/integration/plantuml.md index a7458999aaa..cd61dc9a2bf 100644 --- a/doc/administration/integration/plantuml.md +++ b/doc/administration/integration/plantuml.md @@ -37,7 +37,7 @@ A simple `docker-compose.yml` file would be: version: "3" services: gitlab: - image: 'gitlab/gitlab-ce:12.2.5-ce.0' + image: 'gitlab/gitlab-ee:12.2.5-ee.0' environment: GITLAB_OMNIBUS_CONFIG: | nginx['custom_gitlab_server_config'] = "location /-/plantuml/ { \n proxy_cache off; \n proxy_pass http://plantuml:8080/; \n}\n" @@ -90,8 +90,8 @@ the configuration below accordingly. ### Making local PlantUML accessible using custom GitLab setup The PlantUML server runs locally on your server, so it is not accessible -externally. As such, it is necessary to catch external PlantUML calls and -redirect them to the local server. +externally by default. As such, it is necessary to catch external PlantUML +calls and redirect them to the local server. The idea is to redirect each call to `https://gitlab.example.com/-/plantuml/` to the local PlantUML server `http://plantuml:8080/` or `http://localhost:8080/plantuml/`, depending on your setup. @@ -112,6 +112,12 @@ To activate the changes, run the following command: sudo gitlab-ctl reconfigure ``` +Note that the redirection through GitLab **must** be configured +when running [GitLab with TLS](https://docs.gitlab.com/omnibus/settings/ssl.html) +due to PlantUML's use of the insecure HTTP protocol. Newer browsers such +as [Google Chrome 86+](https://www.chromestatus.com/feature/4926989725073408) +do not load insecure HTTP resources on a page served over HTTPS. + ### Security PlantUML has features that allows fetching network resources. diff --git a/doc/administration/job_logs.md b/doc/administration/job_logs.md index 7af167de4ad..b2c6864e671 100644 --- a/doc/administration/job_logs.md +++ b/doc/administration/job_logs.md @@ -50,7 +50,7 @@ these steps to move the logs to a new location without losing any data. Jobs in progress are not affected, based on how [data flow](#data-flow) works. ```ruby - sidekiq['experimental_queue_selector'] = true + sidekiq['queue_selector'] = true sidekiq['queue_groups'] = [ "feature_category!=continuous_integration" ] @@ -185,7 +185,7 @@ Feature.enable(:ci_enable_live_trace) ``` NOTE: -The transition period is handled gracefully. Upcoming logs are +The transition period is handled gracefully. Upcoming logs are generated with the incremental architecture, and on-going logs stay with the legacy architecture, which means that on-going logs aren't forcibly re-generated with the incremental architecture. diff --git a/doc/administration/monitoring/performance/index.md b/doc/administration/monitoring/performance/index.md index 5b82696ae4b..072baa16e29 100644 --- a/doc/administration/monitoring/performance/index.md +++ b/doc/administration/monitoring/performance/index.md @@ -61,3 +61,14 @@ half above the interval. For example, for a user defined interval of 15 seconds the actual interval can be anywhere between 7.5 and 22.5. The interval is re-generated for every sampling run instead of being generated one time and reused for the duration of the process' lifetime. + +User defined intervals can be specified by means of environment variables. +The following environment variables are recognized: + +- `RUBY_SAMPLER_INTERVAL_SECONDS` +- `DATABASE_SAMPLER_INTERVAL_SECONDS` +- `ACTION_CABLE_SAMPLER_INTERVAL_SECONDS` +- `PUMA_SAMPLER_INTERVAL_SECONDS` +- `UNICORN_SAMPLER_INTERVAL_SECONDS` +- `THREADS_SAMPLER_INTERVAL_SECONDS` +- `GLOBAL_SEARCH_SAMPLER_INTERVAL_SECONDS` diff --git a/doc/administration/monitoring/prometheus/gitlab_exporter.md b/doc/administration/monitoring/prometheus/gitlab_exporter.md index 8e4d87cfa78..5add842bccc 100644 --- a/doc/administration/monitoring/prometheus/gitlab_exporter.md +++ b/doc/administration/monitoring/prometheus/gitlab_exporter.md @@ -30,3 +30,27 @@ To enable the GitLab exporter in an Omnibus GitLab instance: Prometheus automatically begins collecting performance data from the GitLab exporter exposed at `localhost:9168`. + +## Use a different Rack server + +>- Introduced in [Omnibus GitLab 13.8](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4896). +>- WEBrick is now the default Rack server instead of Puma. + +By default, the GitLab exporter runs on [WEBrick](https://github.com/ruby/webrick), a single-threaded Ruby web server. +You can choose a different Rack server that better matches your performance needs. +For instance, in multi-node setups that contain a large number of Prometheus scrapers +but only a few monitoring nodes, you may decide to run a multi-threaded server such as Puma instead. + +To change the Rack server to Puma: + +1. Edit `/etc/gitlab/gitlab.rb`. +1. Add, or find and uncomment, the following line, and set it to `puma`: + + ```ruby + gitlab_exporter['server_name'] = 'puma' + ``` + +1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) + for the changes to take effect. + +The supported Rack servers are `webrick` and `puma`. diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index c5e0570515d..4493c1677bc 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -210,6 +210,7 @@ configuration option in `gitlab.yml`. These metrics are served from the | `limited_capacity_worker_max_running_jobs` | Gauge | 13.5 | Maximum number of running jobs | `worker` | | `limited_capacity_worker_remaining_work_count` | Gauge | 13.5 | Number of jobs waiting to be enqueued | `worker` | | `destroyed_job_artifacts_count_total` | Counter | 13.6 | Number of destroyed expired job artifacts | | +| `destroyed_pipeline_artifacts_count_total` | Counter | 13.8 | Number of destroyed expired pipeline artifacts | | ## Database load balancing metrics **(PREMIUM ONLY)** diff --git a/doc/administration/nfs.md b/doc/administration/nfs.md index 1e2ac531329..bddf770180b 100644 --- a/doc/administration/nfs.md +++ b/doc/administration/nfs.md @@ -15,9 +15,10 @@ For data objects such as LFS, Uploads, Artifacts, etc., an [Object Storage servi is recommended over NFS where possible, due to better performance. WARNING: -From GitLab 13.0, using NFS for Git repositories is deprecated. In GitLab 14.0, -support for NFS for Git repositories is scheduled to be removed. Upgrade to -[Gitaly Cluster](gitaly/praefect.md) as soon as possible. +From GitLab 13.0, using NFS for Git repositories is deprecated. +From GitLab 14.0, technical support for NFS for Git repositories +will no longer be provided. Upgrade to [Gitaly Cluster](gitaly/praefect.md) +as soon as possible. Filesystem performance can impact overall GitLab performance, especially for actions that read or write to Git repositories. For steps you can use to test diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md index a89c50a8412..999d9b87363 100644 --- a/doc/administration/object_storage.md +++ b/doc/administration/object_storage.md @@ -61,10 +61,10 @@ must be enabled, only the following providers can be used: - [Google Cloud Storage](#google-cloud-storage-gcs) - [Azure Blob storage](#azure-blob-storage) -Background upload isn't supported with the consolidated object storage -configuration. We recommend enabling direct upload mode because it doesn't -require a shared folder, and [this setting may become the -default](https://gitlab.com/gitlab-org/gitlab/-/issues/27331). +When consolidated object storage is used, direct upload is enabled +automatically. Background upload is not supported. For storage-specific +configuration, [direct upload may become the default](https://gitlab.com/gitlab-org/gitlab/-/issues/27331) +because it does not require a shared folder. Consolidated object storage configuration can't be used for backups or Mattermost. See the [full table for a complete list](#storage-specific-configuration). @@ -232,7 +232,7 @@ The connection settings match those provided by [fog-aws](https://github.com/fog | `aws_secret_access_key` | AWS credentials, or compatible | | | `aws_signature_version` | AWS signature version to use. `2` or `4` are valid options. Digital Ocean Spaces and other providers may need `2`. | `4` | | `enable_signature_v4_streaming` | Set to `true` to enable HTTP chunked transfers with [AWS v4 signatures](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be `false`. | `true` | -| `region` | AWS region | us-east-1 | +| `region` | AWS region. | | | `host` | S3 compatible host for when not using AWS, e.g. `localhost` or `storage.example.com`. HTTPS and port 443 is assumed. | `s3.amazonaws.com` | | `endpoint` | Can be used when configuring an S3 compatible service such as [MinIO](https://min.io), by entering a URL such as `http://127.0.0.1:9000`. This takes precedence over `host`. | (optional) | | `path_style` | Set to `true` to use `host/bucket_name/object` style paths instead of `bucket_name.host/object`. Leave as `false` for AWS S3. | `false` | diff --git a/doc/administration/operations/extra_sidekiq_processes.md b/doc/administration/operations/extra_sidekiq_processes.md index 1f611a50a53..1b7c0edab04 100644 --- a/doc/administration/operations/extra_sidekiq_processes.md +++ b/doc/administration/operations/extra_sidekiq_processes.md @@ -114,7 +114,7 @@ you list: > - [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/45) in [GitLab Starter](https://about.gitlab.com/pricing/) 12.8. > - [Sidekiq cluster including queue selector moved](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/181) to GitLab [Core](https://about.gitlab.com/pricing/#self-managed) in GitLab 12.10. -> - [Marked as supported](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/147) in GitLab [Core](https://about.gitlab.com/pricing/#self-managed) in GitLab 13.6. Renamed from `experimental_queue_selector` to `queue_selector`. +> - [Renamed from `experimental_queue_selector` to `queue_selector`](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/147) in GitLab 13.6. In addition to selecting queues by name, as above, the `queue_selector` option allows queue groups to be selected in a more general way using diff --git a/doc/administration/operations/moving_repositories.md b/doc/administration/operations/moving_repositories.md index 7cc15f9cea4..029f3bb01ed 100644 --- a/doc/administration/operations/moving_repositories.md +++ b/doc/administration/operations/moving_repositories.md @@ -23,12 +23,14 @@ For more information, see: - [Configuring additional storage for Gitaly](../gitaly/index.md#network-architecture). Within this example, additional storage called `storage1` and `storage2` is configured. - [The API documentation](../../api/project_repository_storage_moves.md) details the endpoints for - querying and scheduling repository moves. + querying and scheduling project repository moves. +- [The API documentation](../../api/snippet_repository_storage_moves.md) details the endpoints for + querying and scheduling snippet repository moves. - [Migrate existing repositories to Gitaly Cluster](../gitaly/praefect.md#migrate-existing-repositories-to-gitaly-cluster). ### Limitations -Read more in the [API documentation](../../api/project_repository_storage_moves.md#limitations). +Read more in the [API documentation for projects](../../api/project_repository_storage_moves.md#limitations) and the [API documentation for snippets](../../api/snippet_repository_storage_moves.md#limitations). ## Migrating to another GitLab instance diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md index 633129e98bd..ab6202fef4c 100644 --- a/doc/administration/packages/container_registry.md +++ b/doc/administration/packages/container_registry.md @@ -600,6 +600,28 @@ on how to achieve that. If you use an external container registry, some features associated with the container registry may be unavailable or have [inherent risks](../../user/packages/container_registry/index.md#use-with-external-container-registries). +For the integration to work, the external registry must be configured to +use a JSON Web Token to authenticate with GitLab. The +[external registry's runtime configuration](https://docs.docker.com/registry/configuration/#token) +**must** have the following entries: + +```yaml +auth: + token: + realm: https://gitlab.example.com/jwt/auth + service: container_registry + issuer: gitlab-issuer + rootcertbundle: /root/certs/certbundle +``` + +Without these entries, the registry logins cannot authenticate with GitLab. +GitLab also remains unaware of +[nested image names](../../user/packages/container_registry/#image-naming-convention) +under the project hierarchy, like +`registry.example.com/group/project/image-name:tag` or +`registry.example.com/group/project/my/image-name:tag`, and only recognizes +`registry.example.com/group/project:tag`. + **Omnibus GitLab** You can use GitLab as an auth endpoint with an external container registry. @@ -609,18 +631,23 @@ You can use GitLab as an auth endpoint with an external container registry. ```ruby gitlab_rails['registry_enabled'] = true gitlab_rails['registry_api_url'] = "http://localhost:5000" - gitlab_rails['registry_issuer'] = "omnibus-gitlab-issuer" + gitlab_rails['registry_issuer'] = "gitlab-issuer" ``` `gitlab_rails['registry_enabled'] = true` is needed to enable GitLab Container Registry features and authentication endpoint. The GitLab bundled Container Registry service does not start, even with this enabled. + `gitlab_rails['registry_api_url'] = "http://localhost:5000"` can + carry a different hostname and port depending on where the external registry + is hosted. It must also specify `https` if the external registry is + configured to use TLS. + 1. A certificate-key pair is required for GitLab and the external container registry to communicate securely. You need to create a certificate-key pair, configuring the external container registry with the public - certificate and configuring GitLab with the private key. To do that, add - the following to `/etc/gitlab/gitlab.rb`: + certificate (`rootcertbundle`) and configuring GitLab with the private key. + To do that, add the following to `/etc/gitlab/gitlab.rb`: ```ruby # registry['internal_key'] should contain the contents of the custom key @@ -664,7 +691,7 @@ You can use GitLab as an auth endpoint with an external container registry. api_url: "http://localhost:5000" path: /var/opt/gitlab/gitlab-rails/shared/registry key: /var/opt/gitlab/gitlab-rails/certificate.key - issuer: omnibus-gitlab-issuer + issuer: gitlab-issuer ``` 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. @@ -840,6 +867,26 @@ understand the implications. WARNING: This is a destructive operation. +When you run `registry-garbage-collect` with the -m flag, garbage collection unlinks manifests that +are part of a multi-arch manifest, unless they're tagged in the same repository. +See [this issue](https://gitlab.com/gitlab-org/container-registry/-/issues/149) for details. + +To work around this issue, instead of: + +```plaintext +myrepo/multiarchmanifest:latest +myrepo/manifest/amd-64:latest +myrepo/manifest/arm:latest +``` + +Use: + +```plaintext +myrepo/multiarchmanifest:latest +myrepo/manifest:amd-64-latest +myrepo/manifest:arm-latest +``` + The GitLab Container Registry follows the same default workflow as Docker Distribution: retain untagged manifests and all layers, even ones that are not referenced directly. All content can be accessed by using context addressable identifiers. diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index af0476b72e9..9e949468805 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -38,20 +38,20 @@ which you can set it up: - Run the Pages daemon in the same server as GitLab, listening on a **secondary IP**. - Run the Pages daemon in a [separate server](#running-gitlab-pages-on-a-separate-server). In that case, the [Pages path](#change-storage-path) must also be present in the server that - the Pages daemon is installed, so you will have to share it via network. + the Pages daemon is installed, so you must share it through the network. - Run the Pages daemon in the same server as GitLab, listening on the same IP - but on different ports. In that case, you will have to proxy the traffic with + but on different ports. In that case, you must proxy the traffic with a load balancer. If you choose that route note that you should use TCP load - balancing for HTTPS. If you use TLS-termination (HTTPS-load balancing) the - pages will not be able to be served with user provided certificates. For + balancing for HTTPS. If you use TLS-termination (HTTPS-load balancing), the + pages can't be served with user-provided certificates. For HTTP it's OK to use HTTP or TCP load balancing. -In this document, we will proceed assuming the first option. If you are not +In this document, we proceed assuming the first option. If you are not supporting custom domains a secondary IP is not needed. ## Prerequisites -Before proceeding with the Pages configuration, you will need to: +Before proceeding with the Pages configuration, you must: 1. Have a domain for Pages that is not a subdomain of your GitLab instance domain. @@ -69,7 +69,7 @@ Before proceeding with the Pages configuration, you will need to: 1. (Only for custom domains) Have a **secondary IP**. NOTE: -If your GitLab instance and the Pages daemon are deployed in a private network or behind a firewall, your GitLab Pages websites will only be accessible to devices/users that have access to the private network. +If your GitLab instance and the Pages daemon are deployed in a private network or behind a firewall, your GitLab Pages websites are only accessible to devices/users that have access to the private network. ### Add the domain to the Public Suffix List @@ -94,13 +94,34 @@ host that GitLab runs. For example, an entry would look like this: ```plaintext *.example.io. 1800 IN A 192.0.2.1 -*.example.io. 1800 IN AAAA 2001::1 +*.example.io. 1800 IN AAAA 2001:db8::1 ``` -where `example.io` is the domain under which GitLab Pages will be served -and `192.0.2.1` is the IPv4 address of your GitLab instance and `2001::1` is the +Where `example.io` is the domain GitLab Pages is served from, +`192.0.2.1` is the IPv4 address of your GitLab instance, and `2001:db8::1` is the IPv6 address. If you don't have IPv6, you can omit the AAAA record. +#### Custom domains + +If support for custom domains is needed, the Pages root domain and its subdomains should point to +the secondary IP (which is dedicated for the Pages daemon). `<namespace>.<pages root domain>` should +point at Pages directly. Without this, users aren't able to use `CNAME` records to point their +custom domains to their GitLab Pages. + +For example, an entry could look like this: + +```plaintext +example.com 1800 IN A 192.0.2.1 +*.example.io. 1800 IN A 192.0.2.2 +``` + +This example contains the following: + +- `example.com`: The GitLab domain. +- `example.io`: The domain GitLab Pages is served from. +- `192.0.2.1`: The primary IP of your GitLab instance. +- `192.0.2.2`: The secondary IP, which is dedicated to GitLab Pages. + NOTE: You should not use the GitLab domain to serve user pages. For more information see the [security section](#security). @@ -123,7 +144,7 @@ since that is needed in all configurations. URL scheme: `http://<namespace>.example.io/<project_slug>` This is the minimum setup that you can use Pages with. It is the base for all -other setups as described below. NGINX will proxy all requests to the daemon. +other setups as described below. NGINX proxies all requests to the daemon. The Pages daemon doesn't listen to the outside world. 1. Set the external URL for GitLab Pages in `/etc/gitlab/gitlab.rb`: @@ -147,7 +168,7 @@ Watch the [video tutorial](https://youtu.be/dD8c7WNcc6s) for this configuration. URL scheme: `https://<namespace>.example.io/<project_slug>` -NGINX will proxy all requests to the daemon. Pages daemon doesn't listen to the +NGINX proxies all requests to the daemon. Pages daemon doesn't listen to the outside world. 1. Place the certificate and key inside `/etc/gitlab/ssl` @@ -168,8 +189,8 @@ outside world. ### Additional configuration for Docker container -The GitLab Pages daemon will not have permissions to bind mounts when it runs -in a Docker container. To overcome this issue you'll need to change the chroot +The GitLab Pages daemon doesn't have permissions to bind mounts when it runs +in a Docker container. To overcome this issue, you must change the chroot behavior: 1. Edit `/etc/gitlab/gitlab.rb`. @@ -189,7 +210,7 @@ The [GitLab Pages README](https://gitlab.com/gitlab-org/gitlab-pages#caveats) ha Below is a table of all configuration settings known to Pages in Omnibus GitLab, and what they do. These options can be adjusted in `/etc/gitlab/gitlab.rb`, -and will take effect after you [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +and take effect after you [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). Most of these settings don't need to be configured manually unless you need more granular control over how the Pages daemon runs and serves content in your environment. @@ -218,7 +239,7 @@ control over how the Pages daemon runs and serves content in your environment. | `inplace_chroot` | On [systems that don't support bind-mounts](index.md#additional-configuration-for-docker-container), this instructs GitLab Pages to chroot into its `pages_path` directory. Some caveats exist when using inplace chroot; refer to the GitLab Pages [README](https://gitlab.com/gitlab-org/gitlab-pages/blob/master/README.md#caveats) for more information. | `insecure_ciphers` | Use default list of cipher suites, may contain insecure ones like 3DES and RC4. | `internal_gitlab_server` | Internal GitLab server address used exclusively for API requests. Useful if you want to send that traffic over an internal load balancer. Defaults to GitLab `external_url`. -| `listen_proxy` | The addresses to listen on for reverse-proxy requests. Pages will bind to these addresses' network socket and receives incoming requests from it. Sets the value of `proxy_pass` in `$nginx-dir/conf/gitlab-pages.conf`. +| `listen_proxy` | The addresses to listen on for reverse-proxy requests. Pages binds to these addresses' network sockets and receives incoming requests from them. Sets the value of `proxy_pass` in `$nginx-dir/conf/gitlab-pages.conf`. | `log_directory` | Absolute path to a log directory. | `log_format` | The log output format: `text` or `json`. | `log_verbose` | Verbose logging, true/false. @@ -250,7 +271,7 @@ control over how the Pages daemon runs and serves content in your environment. In addition to the wildcard domains, you can also have the option to configure GitLab Pages to work with custom domains. Again, there are two options here: support custom domains with and without TLS certificates. The easiest setup is -that without TLS certificates. In either case, you'll need a **secondary IP**. If +that without TLS certificates. In either case, you need a **secondary IP**. If you have IPv6 as well as IPv4 addresses, you can use them both. ### Custom domains @@ -274,11 +295,11 @@ world. Custom domains are supported, but no TLS. pages_external_url "http://example.io" nginx['listen_addresses'] = ['192.0.2.1'] pages_nginx['enable'] = false - gitlab_pages['external_http'] = ['192.0.2.2:80', '[2001::2]:80'] + gitlab_pages['external_http'] = ['192.0.2.2:80', '[2001:db8::2]:80'] ``` where `192.0.2.1` is the primary IP address that GitLab is listening to and - `192.0.2.2` and `2001::2` are the secondary IPs the GitLab Pages daemon + `192.0.2.2` and `2001:db8::2` are the secondary IPs the GitLab Pages daemon listens on. If you don't have IPv6, you can omit the IPv6 address. 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -307,12 +328,12 @@ world. Custom domains and TLS are supported. pages_nginx['enable'] = false gitlab_pages['cert'] = "/etc/gitlab/ssl/example.io.crt" gitlab_pages['cert_key'] = "/etc/gitlab/ssl/example.io.key" - gitlab_pages['external_http'] = ['192.0.2.2:80', '[2001::2]:80'] - gitlab_pages['external_https'] = ['192.0.2.2:443', '[2001::2]:443'] + gitlab_pages['external_http'] = ['192.0.2.2:80', '[2001:db8::2]:80'] + gitlab_pages['external_https'] = ['192.0.2.2:443', '[2001:db8::2]:443'] ``` where `192.0.2.1` is the primary IP address that GitLab is listening to and - `192.0.2.2` and `2001::2` are the secondary IPs where the GitLab Pages daemon + `192.0.2.2` and `2001:db8::2` are the secondary IPs where the GitLab Pages daemon listens on. If you don't have IPv6, you can omit the IPv6 address. 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -321,7 +342,7 @@ world. Custom domains and TLS are supported. To prevent malicious users from hijacking domains that don't belong to them, GitLab supports [custom domain verification](../../user/project/pages/custom_domains_ssl_tls_certification/index.md#steps). -When adding a custom domain, users will be required to prove they own it by +When adding a custom domain, users are required to prove they own it by adding a GitLab-controlled verification code to the DNS records for that domain. If your user base is private or otherwise trusted, you can disable the @@ -337,11 +358,11 @@ This setting is enabled by default. allows users to add Let's Encrypt SSL certificates for GitLab Pages sites served under a custom domain. -To enable it, you'll need to: +To enable it, you must: -1. Choose an email on which you will receive notifications about expiring domains. +1. Choose an email address on which you want to receive notifications about expiring domains. 1. Navigate to your instance's **Admin Area > Settings > Preferences** and expand **Pages** settings. -1. Enter the email for receiving notifications and accept Let's Encrypt's Terms of Service as shown below. +1. Enter the email address for receiving notifications and accept Let's Encrypt's Terms of Service as shown below. 1. Click **Save changes**.  @@ -384,7 +405,7 @@ all the App nodes and Sidekiq nodes. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32095) in GitLab 12.7. You can enforce [Access Control](#access-control) for all GitLab Pages websites hosted -on your GitLab instance. By doing so, only logged-in users will have access to them. +on your GitLab instance. By doing so, only logged-in users have access to them. This setting overrides Access Control set by users in individual projects. This can be useful to preserve information published with Pages websites to the users @@ -417,7 +438,7 @@ internet connectivity is gated by a proxy. To use a proxy for GitLab Pages: When using certificates issued by a custom CA, [Access Control](../../user/project/pages/pages_access_control.md#gitlab-pages-access-control) and the [online view of HTML job artifacts](../../ci/pipelines/job_artifacts.md#browsing-artifacts) -will fail to work if the custom CA is not recognized. +fails to work if the custom CA is not recognized. This usually results in this error: `Post /oauth/token: x509: certificate signed by unknown authority`. @@ -501,7 +522,7 @@ the below steps to do a no downtime transfer to a new storage location. 1. Pause Pages deployments by setting the following in `/etc/gitlab/gitlab.rb`: ```ruby - sidekiq['experimental_queue_selector'] = true + sidekiq['queue_selector'] = true sidekiq['queue_groups'] = [ "feature_category!=pages" ] @@ -602,7 +623,7 @@ database encryption. Proceed with caution. changes to take effect. The `gitlab-secrets.json` file is now updated with the new configuration. -1. Set up a new server. This will become the **Pages server**. +1. Set up a new server. This becomes the **Pages server**. 1. Create an [NFS share](../nfs.md) on the **Pages server** and configure this share to @@ -668,7 +689,7 @@ Pages server. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217912) in GitLab 13.3. GitLab Pages can use different sources to get domain configuration. -The default value is `nil`; however, GitLab Pages will default to `auto`. +The default value is `nil`. However, GitLab Pages defaults to `auto`. ```ruby gitlab_pages['domain_config_source'] = nil @@ -806,10 +827,10 @@ provided at `/etc/ssl/ca-bundle.pem`. It's from `/opt/gitlab/embedded/ssl/certs/cacert.pem` as part of starting up Pages. -If the permissions on the source file are incorrect (they should be `0644`) then -the file inside the chroot jail will also be wrong. +If the permissions on the source file are incorrect (they should be `0644`), then +the file inside the chroot jail is also wrong. -Pages will log errors in `/var/log/gitlab/gitlab-pages/current` like: +Pages logs errors in `/var/log/gitlab/gitlab-pages/current` like: ```plaintext x509: failed to load system roots and no roots provided diff --git a/doc/administration/pages/source.md b/doc/administration/pages/source.md index c7c25f0f3a7..0aebeaf2ebe 100644 --- a/doc/administration/pages/source.md +++ b/doc/administration/pages/source.md @@ -17,8 +17,8 @@ You are encouraged to read the [Omnibus documentation](index.md) as it provides some invaluable information to the configuration of GitLab Pages. Please proceed to read it before going forward with this guide. -We also highly recommend that you use the Omnibus GitLab packages, as we -optimize them specifically for GitLab, and we will take care of upgrading GitLab +We also highly recommend that you use the Omnibus GitLab packages. We +optimize them specifically for GitLab, and we take care of upgrading GitLab Pages to the latest supported version. ## Overview @@ -38,22 +38,22 @@ which you can set it up: 1. Run the Pages daemon in the same server as GitLab, listening on a secondary IP. 1. Run the Pages daemon in a separate server. In that case, the [Pages path](#change-storage-path) must also be present in the server that - the Pages daemon is installed, so you will have to share it via network. + the Pages daemon is installed, so you must share it through the network. 1. Run the Pages daemon in the same server as GitLab, listening on the same IP - but on different ports. In that case, you will have to proxy the traffic with - a load balancer. If you choose that route note that you should use TCP load - balancing for HTTPS. If you use TLS-termination (HTTPS-load balancing) the - pages will not be able to be served with user provided certificates. For - HTTP it's OK to use HTTP or TCP load balancing. + but on different ports. In that case, you must proxy the traffic with + a load balancer. If you choose that route, note that you should use TCP load + balancing for HTTPS. If you use TLS-termination (HTTPS-load balancing), the + pages aren't able to be served with user-provided certificates. For + HTTP, it's OK to use HTTP or TCP load balancing. -In this document, we will proceed assuming the first option. If you are not -supporting custom domains a secondary IP is not needed. +In this document, we proceed assuming the first option. If you aren't +supporting custom domains, a secondary IP isn't needed. ## Prerequisites Before proceeding with the Pages configuration, make sure that: -1. You have a separate domain under which GitLab Pages will be served. In +1. You have a separate domain to serve GitLab Pages from. In this document we assume that to be `example.io`. 1. You have configured a **wildcard DNS record** for that domain. 1. You have installed the `zip` and `unzip` packages in the same server that @@ -74,7 +74,7 @@ host that GitLab runs. For example, an entry would look like this: *.example.io. 1800 IN A 192.0.2.1 ``` -where `example.io` is the domain under which GitLab Pages will be served +Where `example.io` is the domain to serve GitLab Pages from, and `192.0.2.1` is the IP address of your GitLab instance. NOTE: @@ -97,7 +97,7 @@ since that is needed in all configurations. URL scheme: `http://<namespace>.example.io/<project_slug>` This is the minimum setup that you can use Pages with. It is the base for all -other setups as described below. NGINX will proxy all requests to the daemon. +other setups as described below. NGINX proxies all requests to the daemon. The Pages daemon doesn't listen to the outside world. 1. Install the Pages daemon: @@ -117,7 +117,7 @@ The Pages daemon doesn't listen to the outside world. ``` 1. Edit `gitlab.yml` and under the `pages` setting, set `enabled` to `true` and - the `host` to the FQDN under which GitLab Pages will be served: + the `host` to the FQDN to serve GitLab Pages from: ```yaml ## GitLab Pages @@ -159,7 +159,7 @@ The Pages daemon doesn't listen to the outside world. URL scheme: `https://<namespace>.example.io/<project_slug>` -NGINX will proxy all requests to the daemon. Pages daemon doesn't listen to the +NGINX proxies all requests to the daemon. Pages daemon doesn't listen to the outside world. 1. Install the Pages daemon: @@ -238,8 +238,8 @@ world. Custom domains are supported, but no TLS. ``` 1. Edit `gitlab.yml` to look like the example below. You need to change the - `host` to the FQDN under which GitLab Pages will be served. Set - `external_http` to the secondary IP on which the pages daemon will listen + `host` to the FQDN to serve GitLab Pages from. Set + `external_http` to the secondary IP on which the pages daemon listens for connections: ```yaml @@ -303,9 +303,9 @@ world. Custom domains and TLS are supported. ``` 1. Edit `gitlab.yml` to look like the example below. You need to change the - `host` to the FQDN under which GitLab Pages will be served. Set + `host` to the FQDN to serve GitLab Pages from. Set `external_http` and `external_https` to the secondary IP on which the pages - daemon will listen for connections: + daemon listens for connections: ```yaml ## GitLab Pages diff --git a/doc/administration/postgresql/pgbouncer.md b/doc/administration/postgresql/pgbouncer.md index f09ac3052f4..951edfeaec2 100644 --- a/doc/administration/postgresql/pgbouncer.md +++ b/doc/administration/postgresql/pgbouncer.md @@ -66,6 +66,12 @@ This content has been moved to a [new location](replication_and_failover.md#conf 1. At this point, your instance should connect to the database through PgBouncer. If you are having issues, see the [Troubleshooting](#troubleshooting) section +## Backups + +Do not backup or restore GitLab through a PgBouncer connection: this will cause a GitLab outage. + +[Read more about this and how to reconfigure backups](../../raketasks/backup_restore.md#backup-and-restore-for-installations-using-pgbouncer). + ## Enable Monitoring > [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3786) in GitLab 12.0. @@ -150,7 +156,10 @@ ote_pid | tls ## Procedure for bypassing PgBouncer -Some database changes have to be done directly, and not through PgBouncer. This includes database restores and GitLab upgrades (because of the database migrations). +Some database changes have to be done directly, and not through PgBouncer. + +Read more about the affected tasks: [database restores](../../raketasks/backup_restore.md#backup-and-restore-for-installations-using-pgbouncer) +and [GitLab upgrades](https://docs.gitlab.com/omnibus/update/README.html#use-postgresql-ha). 1. To find the primary node, run the following on a database node: diff --git a/doc/administration/postgresql/replication_and_failover.md b/doc/administration/postgresql/replication_and_failover.md index 303246c9c82..75d0c558962 100644 --- a/doc/administration/postgresql/replication_and_failover.md +++ b/doc/administration/postgresql/replication_and_failover.md @@ -46,22 +46,19 @@ Each database node runs three services: `PostgreSQL` - The database itself. -`repmgrd` - Communicates with other repmgrd services in the cluster and handles -failover when issues with the master server occurs. The failover procedure +`Patroni` - Communicates with other patroni services in the cluster and handles +failover when issues with the leader server occurs. The failover procedure consists of: -- Selecting a new master for the cluster. -- Promoting the new node to master. -- Instructing remaining servers to follow the new master node. -- The old master node is automatically evicted from the cluster and should be - rejoined manually once recovered. +- Selecting a new leader for the cluster. +- Promoting the new node to leader. +- Instructing remaining servers to follow the new leader node. -`Consul` agent - Monitors the status of each node in the database cluster and -tracks its health in a service definition on the Consul cluster. +`Consul` agent - To communicate with Consul cluster which stores the current Patroni state. The agent monitors the status of each node in the database cluster and tracks its health in a service definition on the Consul cluster. ### Consul server node -The Consul server node runs the Consul server service. +The Consul server node runs the Consul server service. These nodes must have reached the quorum and elected a leader _before_ Patroni cluster bootstrap otherwise database nodes will wait until such Consul leader is elected. ### PgBouncer node @@ -80,7 +77,7 @@ Each service in the package comes with a set of [default ports](https://docs.git - Application servers connect to either PgBouncer directly via its [default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#pgbouncer) or via a configured Internal Load Balancer (TCP) that serves multiple PgBouncers. - PgBouncer connects to the primary database servers [PostgreSQL default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#postgresql) -- Repmgr connects to the database servers [PostgreSQL default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#postgresql) +- Patroni actively manages the running PostgreSQL processes and configuration. - PostgreSQL secondaries connect to the primary database servers [PostgreSQL default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#postgresql) - Consul servers and agents connect to each others [Consul default ports](https://docs.gitlab.com/omnibus/package-information/defaults.html#consul) @@ -141,7 +138,7 @@ available database connections. In this document we are assuming 3 database nodes, which makes this configuration: ```ruby -postgresql['max_wal_senders'] = 4 +patroni['postgresql']['max_wal_senders'] = 4 ``` As previously mentioned, you'll have to prepare the network subnets that will @@ -186,18 +183,6 @@ Few notes on the service itself: - `/etc/gitlab/gitlab.rb`: hashed, and in plain text - `/var/opt/gitlab/pgbouncer/pg_auth`: hashed -#### Repmgr information - -When using default setup, you will only have to prepare the network subnets that will -be allowed to authenticate with the service. - -Few notes on the service itself: - -- The service runs under the same system account as the database - - In the package, this is by default `gitlab-psql` -- The service will have a superuser database user account generated for it - - This defaults to `gitlab_repmgr` - ### Installing Omnibus GitLab First, make sure to [download/install](https://about.gitlab.com/install/) @@ -212,72 +197,80 @@ When installing the GitLab package, do not supply `EXTERNAL_URL` value. 1. Make sure to [configure the Consul nodes](../consul.md). 1. Make sure you collect [`CONSUL_SERVER_NODES`](#consul-information), [`PGBOUNCER_PASSWORD_HASH`](#pgbouncer-information), [`POSTGRESQL_PASSWORD_HASH`](#postgresql-information), the [number of db nodes](#postgresql-information), and the [network address](#network-information) before executing the next step. -1. On the master database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: +#### Configuring Patroni cluster - ```ruby - # Disable all components except PostgreSQL and Repmgr and Consul - roles ['postgres_role'] +You must enable Patroni explicitly to be able to use it (with `patroni['enable'] = true`). When Patroni is enabled +repmgr will be disabled automatically. - # PostgreSQL configuration - postgresql['listen_address'] = '0.0.0.0' - postgresql['hot_standby'] = 'on' - postgresql['wal_level'] = 'replica' - postgresql['shared_preload_libraries'] = 'repmgr_funcs' +Any PostgreSQL configuration item that controls replication, for example `wal_level`, `max_wal_senders`, etc, are strictly +controlled by Patroni and will override the original settings that you make with the `postgresql[...]` configuration key. +Hence, they are all separated and placed under `patroni['postgresql'][...]`. This behavior is limited to replication. +Patroni honours any other PostgreSQL configuration that was made with the `postgresql[...]` configuration key. For example, +`max_wal_senders` by default is set to `5`. If you wish to change this you must set it with the `patroni['postgresql']['max_wal_senders']` +configuration key. - # Disable automatic database migrations - gitlab_rails['auto_migrate'] = false +NOTE: +The configuration of a Patroni node is very similar to a repmgr but shorter. When Patroni is enabled, first you can ignore +any replication setting of PostgreSQL (it will be overwritten anyway). Then you can remove any `repmgr[...]` or +repmgr-specific configuration as well. Especially, make sure that you remove `postgresql['shared_preload_libraries'] = 'repmgr_funcs'`. - # Configure the Consul agent - consul['services'] = %w(postgresql) +Here is an example similar to [the one that was done with repmgr](#configuring-repmgr-nodes): - # START user configuration - # Please set the real values as explained in Required Information section - # - # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value - postgresql['pgbouncer_user_password'] = 'PGBOUNCER_PASSWORD_HASH' - # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value - postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' - # Replace X with value of number of db nodes + 1 - postgresql['max_wal_senders'] = X - postgresql['max_replication_slots'] = X +```ruby +# Disable all components except PostgreSQL, Patroni (or Repmgr), and Consul +roles['postgres_role'] - # Replace XXX.XXX.XXX.XXX/YY with Network Address - postgresql['trust_auth_cidr_addresses'] = %w(XXX.XXX.XXX.XXX/YY) - repmgr['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 XXX.XXX.XXX.XXX/YY) +# Enable Patroni (which automatically disables Repmgr). +patroni['enable'] = true - # Replace placeholders: - # - # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z - # with the addresses gathered for CONSUL_SERVER_NODES - consul['configuration'] = { - retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z) - } - # - # END user configuration - ``` +# PostgreSQL configuration +postgresql['listen_address'] = '0.0.0.0' - > `postgres_role` was introduced with GitLab 10.3 +# Disable automatic database migrations +gitlab_rails['auto_migrate'] = false -1. On secondary nodes, add all the configuration specified above for primary node - to `/etc/gitlab/gitlab.rb`. In addition, append the following configuration - to inform `gitlab-ctl` that they are standby nodes initially and it need not - attempt to register them as primary node +# Configure the Consul agent +consul['services'] = %w(postgresql) - ```ruby - # Specify if a node should attempt to be master on initialization - repmgr['master_on_initialization'] = false - ``` +# START user configuration +# Please set the real values as explained in Required Information section +# +# Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value +postgresql['pgbouncer_user_password'] = 'PGBOUNCER_PASSWORD_HASH' +# Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value +postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. [Enable Monitoring](#enable-monitoring) +# Replace X with value of number of db nodes + 1 (OPTIONAL the default value is 5) +patroni['postgresql']['max_wal_senders'] = X +patroni['postgresql']['max_replication_slots'] = X -> Please note: -> -> - If you want your database to listen on a specific interface, change the configuration: -> `postgresql['listen_address'] = '0.0.0.0'`. -> - If your PgBouncer service runs under a different user account, -> you also need to specify: `postgresql['pgbouncer_user'] = PGBOUNCER_USERNAME` in -> your configuration. +# Replace XXX.XXX.XXX.XXX/YY with Network Address +postgresql['trust_auth_cidr_addresses'] = %w(XXX.XXX.XXX.XXX/YY) + +# Replace placeholders: +# +# Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z +# with the addresses gathered for CONSUL_SERVER_NODES +consul['configuration'] = { + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z) +} +# +# END user configuration +``` + +You do not need an additional or different configuration for replica nodes. As a matter of fact, you don't have to have +a predetermined primary node. Therefore all database nodes use the same configuration. + +Once the configuration of a node is done, you must [reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) +on each node for the changes to take effect. + +Generally, when Consul cluster is ready, the first node that [reconfigures](../restart_gitlab.md#omnibus-gitlab-reconfigure) +becomes the leader. You do not need to sequence the nodes reconfiguration. You can run them in parallel or in any order. +If you choose an arbitrary order you do not have any predetermined master. + +NOTE: +As opposed to repmgr, once the nodes are reconfigured you do not need any further action or additional command to join +the replicas. #### Enable Monitoring @@ -298,129 +291,6 @@ If you enable Monitoring, it must be enabled on **all** database servers. 1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. -#### Database nodes post-configuration - -##### Primary node - -Select one node as a primary node. - -1. Open a database prompt: - - ```shell - gitlab-psql -d gitlabhq_production - ``` - -1. Enable the `pg_trgm` extension: - - ```shell - CREATE EXTENSION pg_trgm; - ``` - -1. Enable the `btree_gist` extension: - - ```shell - CREATE EXTENSION btree_gist; - ``` - -1. Exit the database prompt by typing `\q` and Enter. - -1. Verify the cluster is initialized with one node: - - ```shell - gitlab-ctl repmgr cluster show - ``` - - The output should be similar to the following: - - ```plaintext - Role | Name | Upstream | Connection String - ----------+----------|----------|---------------------------------------- - * master | HOSTNAME | | host=HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr - ``` - -1. Note down the hostname or IP address in the connection string: `host=HOSTNAME`. We will - refer to the hostname in the next section as `MASTER_NODE_NAME`. If the value - is not an IP address, it will need to be a resolvable name (via DNS or - `/etc/hosts`) - -##### Secondary nodes - -1. Set up the repmgr standby: - - ```shell - gitlab-ctl repmgr standby setup MASTER_NODE_NAME - ``` - - Do note that this will remove the existing data on the node. The command - has a wait time. - - The output should be similar to the following: - - ```console - # gitlab-ctl repmgr standby setup MASTER_NODE_NAME - Doing this will delete the entire contents of /var/opt/gitlab/postgresql/data - If this is not what you want, hit Ctrl-C now to exit - To skip waiting, rerun with the -w option - Sleeping for 30 seconds - Stopping the database - Removing the data - Cloning the data - Starting the database - Registering the node with the cluster - ok: run: repmgrd: (pid 19068) 0s - ``` - -1. Verify the node now appears in the cluster: - - ```shell - gitlab-ctl repmgr cluster show - ``` - - The output should be similar to the following: - - ```plaintext - Role | Name | Upstream | Connection String - ----------+---------|-----------|------------------------------------------------ - * master | MASTER | | host=MASTER_NODE_NAME user=gitlab_repmgr dbname=gitlab_repmgr - standby | STANDBY | MASTER | host=STANDBY_HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr - ``` - -Repeat the above steps on all secondary nodes. - -#### Database checkpoint - -Before moving on, make sure the databases are configured correctly. Run the -following command on the **primary** node to verify that replication is working -properly: - -```shell -gitlab-ctl repmgr cluster show -``` - -The output should be similar to: - -```plaintext -Role | Name | Upstream | Connection String -----------+--------------|--------------|-------------------------------------------------------------------- -* master | MASTER | | host=MASTER port=5432 user=gitlab_repmgr dbname=gitlab_repmgr - standby | STANDBY | MASTER | host=STANDBY port=5432 user=gitlab_repmgr dbname=gitlab_repmgr -``` - -If the 'Role' column for any node says "FAILED", check the -[Troubleshooting section](#troubleshooting) before proceeding. - -Also, check that the check master command works successfully on each node: - -```shell -su - gitlab-consul -gitlab-ctl repmgr-check-master || echo 'This node is a standby repmgr node' -``` - -This command relies on exit codes to tell Consul whether a particular node is a master -or secondary. The most important thing here is that this command does not produce errors. -If there are errors it's most likely due to incorrect `gitlab-consul` database user permissions. -Check the [Troubleshooting section](#troubleshooting) before proceeding. - ### Configuring the PgBouncer node 1. Make sure you collect [`CONSUL_SERVER_NODES`](#consul-information), [`CONSUL_PASSWORD_HASH`](#consul-information), and [`PGBOUNCER_PASSWORD_HASH`](#pgbouncer-information) before executing the next step. @@ -577,6 +447,12 @@ PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` on your database nodes. [PgBouncer error `ERROR: pgbouncer cannot connect to server`](#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) in the Troubleshooting section before proceeding. +### Backups + +Do not backup or restore GitLab through a PgBouncer connection: this will cause a GitLab outage. + +[Read more about this and how to reconfigure backups](../../raketasks/backup_restore.md#backup-and-restore-for-installations-using-pgbouncer). + ### Ensure GitLab is running At this point, your GitLab instance should be up and running. Verify you're able @@ -605,9 +481,9 @@ Here is a list and description of each machine and the assigned IP: - `10.6.0.21`: PgBouncer 1 - `10.6.0.22`: PgBouncer 2 - `10.6.0.23`: PgBouncer 3 -- `10.6.0.31`: PostgreSQL master -- `10.6.0.32`: PostgreSQL secondary -- `10.6.0.33`: PostgreSQL secondary +- `10.6.0.31`: PostgreSQL 1 +- `10.6.0.32`: PostgreSQL 2 +- `10.6.0.33`: PostgreSQL 3 - `10.6.0.41`: GitLab application All passwords are set to `toomanysecrets`, please do not use this password or derived hashes and the `external_url` for GitLab is `http://gitlab.example.com`. @@ -667,29 +543,28 @@ An internal load balancer (TCP) is then required to be setup to serve each PgBou #### Example recommended setup for PostgreSQL servers -##### Primary node - -On primary node edit `/etc/gitlab/gitlab.rb`: +On database nodes edit `/etc/gitlab/gitlab.rb`: ```ruby -# Disable all components except PostgreSQL and Repmgr and Consul +# Disable all components except PostgreSQL, Patroni (or Repmgr), and Consul roles ['postgres_role'] # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' postgresql['hot_standby'] = 'on' postgresql['wal_level'] = 'replica' -postgresql['shared_preload_libraries'] = 'repmgr_funcs' + +# Enable Patroni (which automatically disables Repmgr). +patroni['enable'] = true # Disable automatic database migrations gitlab_rails['auto_migrate'] = false postgresql['pgbouncer_user_password'] = '771a8625958a529132abe6f1a4acb19c' postgresql['sql_user_password'] = '450409b85a0223a214b5fb1484f34d0f' -postgresql['max_wal_senders'] = 4 +patroni['postgresql']['max_wal_senders'] = 4 postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/16) -repmgr['trust_auth_cidr_addresses'] = %w(10.6.0.0/16) # Configure the Consul agent consul['services'] = %w(postgresql) @@ -702,85 +577,27 @@ consul['monitoring_service_discovery'] = true [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -##### Secondary nodes - -On secondary nodes, edit `/etc/gitlab/gitlab.rb` and add all the configuration -added to primary node, noted above. In addition, append the following -configuration: - -```ruby -# Specify if a node should attempt to be master on initialization -repmgr['master_on_initialization'] = false -``` - -[Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -###### Example recommended setup for application server - -On the server edit `/etc/gitlab/gitlab.rb`: - -```ruby -external_url 'http://gitlab.example.com' - -gitlab_rails['db_host'] = '10.6.0.20' # Internal Load Balancer for PgBouncer nodes -gitlab_rails['db_port'] = 6432 -gitlab_rails['db_password'] = 'toomanysecrets' -gitlab_rails['auto_migrate'] = false - -postgresql['enable'] = false -pgbouncer['enable'] = false -consul['enable'] = true - -# Configure Consul agent -consul['watchers'] = %w(postgresql) - -pgbouncer['users'] = { - 'gitlab-consul': { - password: '5e0e3263571e3704ad655076301d6ebe' - }, - 'pgbouncer': { - password: '771a8625958a529132abe6f1a4acb19c' - } -} - -consul['configuration'] = { - retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) -} -``` - -[Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - #### Example recommended setup manual steps After deploying the configuration follow these steps: -1. On `10.6.0.31`, our primary database: - - Enable the `pg_trgm` and `btree_gist` extensions: - - ```shell - gitlab-psql -d gitlabhq_production - ``` +1. Find the primary database node: ```shell - CREATE EXTENSION pg_trgm; - CREATE EXTENSION btree_gist; + gitlab-ctl get-postgresql-primary ``` -1. On `10.6.0.32`, our first standby database: +1. On the primary database node: - Make this node a standby of the primary: + Enable the `pg_trgm` and `btree_gist` extensions: ```shell - gitlab-ctl repmgr standby setup 10.6.0.21 + gitlab-psql -d gitlabhq_production ``` -1. On `10.6.0.33`, our second standby database: - - Make this node a standby of the primary: - ```shell - gitlab-ctl repmgr standby setup 10.6.0.21 + CREATE EXTENSION pg_trgm; + CREATE EXTENSION btree_gist; ``` 1. On `10.6.0.41`, our application server: @@ -802,15 +619,15 @@ After deploying the configuration follow these steps: This example uses 3 PostgreSQL servers, and 1 application node (with PgBouncer setup alongside). It differs from the [recommended setup](#example-recommended-setup) by moving the Consul servers into the same servers we use for PostgreSQL. -The trade-off is between reducing server counts, against the increased operational complexity of needing to deal with PostgreSQL [failover](#failover-procedure) and [restore](#restore-procedure) procedures in addition to [Consul outage recovery](../consul.md#outage-recovery) on the same set of machines. +The trade-off is between reducing server counts, against the increased operational complexity of needing to deal with PostgreSQL [failover](#manual-failover-procedure-for-patroni) procedures in addition to [Consul outage recovery](../consul.md#outage-recovery) on the same set of machines. In this example we start with all servers on the same 10.6.0.0/16 private network range, they can connect to each freely other on those addresses. Here is a list and description of each machine and the assigned IP: -- `10.6.0.21`: PostgreSQL master -- `10.6.0.22`: PostgreSQL secondary -- `10.6.0.23`: PostgreSQL secondary +- `10.6.0.21`: PostgreSQL 1 +- `10.6.0.22`: PostgreSQL 2 +- `10.6.0.23`: PostgreSQL 3 - `10.6.0.31`: GitLab application All passwords are set to `toomanysecrets`, please do not use this password or derived hashes. @@ -821,9 +638,7 @@ Please note that after the initial configuration, if a failover occurs, the Post #### Example minimal configuration for database servers -##### Primary node - -On primary database node edit `/etc/gitlab/gitlab.rb`: +On database nodes edit `/etc/gitlab/gitlab.rb`: ```ruby # Disable all components except PostgreSQL, Repmgr, and Consul @@ -833,7 +648,9 @@ roles ['postgres_role'] postgresql['listen_address'] = '0.0.0.0' postgresql['hot_standby'] = 'on' postgresql['wal_level'] = 'replica' -postgresql['shared_preload_libraries'] = 'repmgr_funcs' + +# Enable Patroni (which automatically disables Repmgr). +patroni['enable'] = true # Disable automatic database migrations gitlab_rails['auto_migrate'] = false @@ -843,10 +660,9 @@ consul['services'] = %w(postgresql) postgresql['pgbouncer_user_password'] = '771a8625958a529132abe6f1a4acb19c' postgresql['sql_user_password'] = '450409b85a0223a214b5fb1484f34d0f' -postgresql['max_wal_senders'] = 4 +patroni['postgresql']['max_wal_senders'] = 4 postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/16) -repmgr['trust_auth_cidr_addresses'] = %w(10.6.0.0/16) consul['configuration'] = { server: true, @@ -856,16 +672,6 @@ consul['configuration'] = { [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -##### Secondary nodes - -On secondary nodes, edit `/etc/gitlab/gitlab.rb` and add all the information added -to primary node, noted above. In addition, append the following configuration - -```ruby -# Specify if a node should attempt to be master on initialization -repmgr['master_on_initialization'] = false -``` - #### Example minimal configuration for application server On the server edit `/etc/gitlab/gitlab.rb`: @@ -908,249 +714,10 @@ consul['configuration'] = { The manual steps for this configuration are the same as for the [example recommended setup](#example-recommended-setup-manual-steps). -### Failover procedure - -By default, if the master database fails, `repmgrd` should promote one of the -standby nodes to master automatically, and Consul will update PgBouncer with -the new master. - -If you need to failover manually, you have two options: - -**Shutdown the current master database** - -Run: - -```shell -gitlab-ctl stop postgresql -``` - -The automated failover process will see this and failover to one of the -standby nodes. - -**Or perform a manual failover** - -1. Ensure the old master node is not still active. -1. Login to the server that should become the new master and run: - - ```shell - gitlab-ctl repmgr standby promote - ``` - -1. If there are any other standby servers in the cluster, have them follow - the new master server: - - ```shell - gitlab-ctl repmgr standby follow NEW_MASTER - ``` - -### Restore procedure - -If a node fails, it can be removed from the cluster, or added back as a standby -after it has been restored to service. - -#### Remove a standby from the cluster - - From any other node in the cluster, run: - - ```shell - gitlab-ctl repmgr standby unregister --node=X - ``` - - where X is the value of node in `repmgr.conf` on the old server. - - To find this, you can use: - - ```shell - awk -F = '$1 == "node" { print $2 }' /var/opt/gitlab/postgresql/repmgr.conf - ``` - - It will output something like: - - ```plaintext - 959789412 - ``` - - Then you will use this ID to unregister the node: - - ```shell - gitlab-ctl repmgr standby unregister --node=959789412 - ``` - -#### Add a node as a standby server - - From the standby node, run: - - ```shell - gitlab-ctl repmgr standby follow NEW_MASTER - gitlab-ctl restart repmgrd - ``` - - WARNING: - When the server is brought back online, and before - you switch it to a standby node, repmgr will report that there are two masters. - If there are any clients that are still attempting to write to the old master, - this will cause a split, and the old master will need to be resynced from - scratch by performing a `gitlab-ctl repmgr standby setup NEW_MASTER`. - -#### Add a failed master back into the cluster as a standby node - - Once `repmgrd` and PostgreSQL are running, the node will need to follow the new - as a standby node. - - ```shell - gitlab-ctl repmgr standby follow NEW_MASTER - ``` - - Once the node is following the new master as a standby, the node needs to be - [unregistered from the cluster on the new master node](#remove-a-standby-from-the-cluster). - - Once the old master node has been unregistered from the cluster, it will need - to be setup as a new standby: - - ```shell - gitlab-ctl repmgr standby setup NEW_MASTER - ``` - - Failure to unregister and read the old master node can lead to subsequent failovers - not working. - -### Alternate configurations - -#### Database authorization - -By default, we give any host on the database network the permission to perform -repmgr operations using PostgreSQL's `trust` method. If you do not want this -level of trust, there are alternatives. - -You can trust only the specific nodes that will be database clusters, or you -can require md5 authentication. - -#### Trust specific addresses - -If you know the IP address, or FQDN of all database and PgBouncer nodes in the -cluster, you can trust only those nodes. - -In `/etc/gitlab/gitlab.rb` on all of the database nodes, set -`repmgr['trust_auth_cidr_addresses']` to an array of strings containing all of -the addresses. - -If setting to a node's FQDN, they must have a corresponding PTR record in DNS. -If setting to a node's IP address, specify it as `XXX.XXX.XXX.XXX/32`. - -For example: - -```ruby -repmgr['trust_auth_cidr_addresses'] = %w(192.168.1.44/32 db2.example.com) -``` - -#### MD5 Authentication - -If you are running on an untrusted network, repmgr can use md5 authentication -with a [`.pgpass` file](https://www.postgresql.org/docs/11/libpq-pgpass.html) -to authenticate. - -You can specify by IP address, FQDN, or by subnet, using the same format as in -the previous section: - -1. On the current master node, create a password for the `gitlab` and - `gitlab_repmgr` user: - - ```shell - gitlab-psql -d template1 - template1=# \password gitlab_repmgr - Enter password: **** - Confirm password: **** - template1=# \password gitlab - ``` - -1. On each database node: - - 1. Edit `/etc/gitlab/gitlab.rb`: - 1. Ensure `repmgr['trust_auth_cidr_addresses']` is **not** set - 1. Set `postgresql['md5_auth_cidr_addresses']` to the desired value - 1. Set `postgresql['sql_replication_user'] = 'gitlab_repmgr'` - 1. Reconfigure with `gitlab-ctl reconfigure` - 1. Restart PostgreSQL with `gitlab-ctl restart postgresql` - - 1. Create a `.pgpass` file. Enter the `gitlab_repmgr` password twice to - when asked: - - ```shell - gitlab-ctl write-pgpass --user gitlab_repmgr --hostuser gitlab-psql --database '*' - ``` - -1. On each PgBouncer node, edit `/etc/gitlab/gitlab.rb`: - 1. Ensure `gitlab_rails['db_password']` is set to the plaintext password for - the `gitlab` database user - 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect - -## Troubleshooting - -### Consul and PostgreSQL changes not taking effect - -Due to the potential impacts, `gitlab-ctl reconfigure` only reloads Consul and PostgreSQL, it will not restart the services. However, not all changes can be activated by reloading. - -To restart either service, run `gitlab-ctl restart SERVICE` - -For PostgreSQL, it is usually safe to restart the master node by default. Automatic failover defaults to a 1 minute timeout. Provided the database returns before then, nothing else needs to be done. To be safe, you can stop `repmgrd` on the standby nodes first with `gitlab-ctl stop repmgrd`, then start afterwards with `gitlab-ctl start repmgrd`. - -On the Consul server nodes, it is important to [restart the Consul service](../consul.md#restart-consul) in a controlled manner. - -### `gitlab-ctl repmgr-check-master` command produces errors - -If this command displays errors about database permissions it is likely that something failed during -install, resulting in the `gitlab-consul` database user getting incorrect permissions. Follow these -steps to fix the problem: - -1. On the master database node, connect to the database prompt - `gitlab-psql -d template1` -1. Delete the `gitlab-consul` user - `DROP USER "gitlab-consul";` -1. Exit the database prompt - `\q` -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) and the user will be re-added with the proper permissions. -1. Change to the `gitlab-consul` user - `su - gitlab-consul` -1. Try the check command again - `gitlab-ctl repmgr-check-master`. - -Now there should not be errors. If errors still occur then there is another problem. - -### PgBouncer error `ERROR: pgbouncer cannot connect to server` - -You may get this error when running `gitlab-rake gitlab:db:configure` or you -may see the error in the PgBouncer log file. - -```plaintext -PG::ConnectionBad: ERROR: pgbouncer cannot connect to server -``` - -The problem may be that your PgBouncer node's IP address is not included in the -`trust_auth_cidr_addresses` setting in `/etc/gitlab/gitlab.rb` on the database nodes. - -You can confirm that this is the issue by checking the PostgreSQL log on the master -database node. If you see the following error then `trust_auth_cidr_addresses` -is the problem. - -```plaintext -2018-03-29_13:59:12.11776 FATAL: no pg_hba.conf entry for host "123.123.123.123", user "pgbouncer", database "gitlabhq_production", SSL off -``` - -To fix the problem, add the IP address to `/etc/gitlab/gitlab.rb`. - -```ruby -postgresql['trust_auth_cidr_addresses'] = %w(123.123.123.123/32 <other_cidrs>) -``` - -[Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -### Issues with other components - -If you're running into an issue with a component not outlined here, be sure to check the troubleshooting section of their specific documentation page: - -- [Consul](../consul.md#troubleshooting-consul) -- [PostgreSQL](https://docs.gitlab.com/omnibus/settings/database.html#troubleshooting) - ## Patroni NOTE: -Starting from GitLab 13.1, Patroni is available for **experimental** use to replace repmgr. Due to its -experimental nature, Patroni support is **subject to change without notice.** +Using Patroni instead of Repmgr is supported for PostgreSQL 11 and required for PostgreSQL 12. Patroni is an opinionated solution for PostgreSQL high-availability. It takes the control of PostgreSQL, overrides its configuration and manages its lifecycle (start, stop, restart). This is a more active approach when compared to repmgr. @@ -1174,80 +741,44 @@ functional or does not have a leader, Patroni and by extension PostgreSQL will n API which can be accessed via its [default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#patroni) on each node. -### Configuring Patroni cluster - -You must enable Patroni explicitly to be able to use it (with `patroni['enable'] = true`). When Patroni is enabled -repmgr will be disabled automatically. +### Selecting the appropriate Patroni replication method -Any PostgreSQL configuration item that controls replication, for example `wal_level`, `max_wal_senders`, etc, are strictly -controlled by Patroni and will override the original settings that you make with the `postgresql[...]` configuration key. -Hence, they are all separated and placed under `patroni['postgresql'][...]`. This behavior is limited to replication. -Patroni honours any other PostgreSQL configuration that was made with the `postgresql[...]` configuration key. For example, -`max_wal_senders` by default is set to `5`. If you wish to change this you must set it with the `patroni['postgresql']['max_wal_senders']` -configuration key. +[Review the Patroni documentation carefully](https://patroni.readthedocs.io/en/latest/SETTINGS.html#postgresql) +before making changes as **_some of the options carry a risk of potential data +loss if not fully understood_**. The [replication mode](https://patroni.readthedocs.io/en/latest/replication_modes.html) +configured determines the amount of tolerable data loss. -The configuration of Patroni node is very similar to a repmgr but shorter. When Patroni is enabled, first you can ignore -any replication setting of PostgreSQL (it will be overwritten anyway). Then you can remove any `repmgr[...]` or -repmgr-specific configuration as well. Especially, make sure that you remove `postgresql['shared_preload_libraries'] = 'repmgr_funcs'`. +WARNING: +Replication is not a backup strategy! There is no replacement for a well-considered and tested backup solution. -Here is an example similar to [the one that was done with repmgr](#configuring-the-database-nodes): +Omnibus GitLab defaults [`synchronous_commit`](https://www.postgresql.org/docs/11/runtime-config-wal.html#GUC-SYNCHRONOUS-COMMIT) to `on`. ```ruby -# Disable all components except PostgreSQL and Repmgr and Consul -roles['postgres_role'] +postgresql['synchronous_commit'] = 'on' +gitlab['geo-postgresql']['synchronous_commit'] = 'on' +``` -# Enable Patroni -patroni['enable'] = true +#### Customizing Patroni failover behavior -# PostgreSQL configuration -postgresql['listen_address'] = '0.0.0.0' +Omnibus GitLab exposes several options allowing more control over the [Patroni restoration process](#recovering-the-patroni-cluster). -# Disable automatic database migrations -gitlab_rails['auto_migrate'] = false +Each option is shown below with its default value in `/etc/gitlab/gitlab.rb`. -# Configure the Consul agent -consul['services'] = %w(postgresql) - -# START user configuration -# Please set the real values as explained in Required Information section -# -# Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value -postgresql['pgbouncer_user_password'] = 'PGBOUNCER_PASSWORD_HASH' -# Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value -postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' - -# Replace X with value of number of db nodes + 1 (OPTIONAL the default value is 5) -patroni['postgresql']['max_wal_senders'] = X -patroni['postgresql']['max_replication_slots'] = X - -# Replace XXX.XXX.XXX.XXX/YY with Network Address -postgresql['trust_auth_cidr_addresses'] = %w(XXX.XXX.XXX.XXX/YY) - -# Replace placeholders: -# -# Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z -# with the addresses gathered for CONSUL_SERVER_NODES -consul['configuration'] = { - retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z) -} -# -# END user configuration +```ruby +patroni['use_pg_rewind'] = true +patroni['remove_data_directory_on_rewind_failure'] = false +patroni['remove_data_directory_on_diverged_timelines'] = false ``` -You do not need an additional or different configuration for replica nodes. As a matter of fact, you don't have to have -a predetermined primary node. Therefore all database nodes use the same configuration. +[The upstream documentation will always be more up to date](https://patroni.readthedocs.io/en/latest/SETTINGS.html#postgresql), but the table below should provide a minimal overview of functionality. -Once the configuration of a node is done, you must [reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) -on each node for the changes to take effect. +|Setting|Overview| +|-|-| +|`use_pg_rewind`|Try running `pg_rewind` on the former cluster leader before it rejoins the database cluster.| +|`remove_data_directory_on_rewind_failure`|If `pg_rewind` fails, remove the local PostgreSQL data directory and re-replicate from the current cluster leader.| +|`remove_data_directory_on_diverged_timelines`|If `pg_rewind` cannot be used and the former leader's timeline has diverged from the current one, then delete the local data directory and re-replicate from the current cluster leader.| -Generally, when Consul cluster is ready, the first node that [reconfigures](../restart_gitlab.md#omnibus-gitlab-reconfigure) -becomes the leader. You do not need to sequence the nodes reconfiguration. You can run them in parallel or in any order. -If you choose an arbitrary order you do not have any predetermined master. - -As opposed to repmgr, once the nodes are reconfigured you do not need any further action or additional command to join -the replicas. - -#### Database authorization for Patroni +### Database authorization for Patroni Patroni uses Unix socket to manage PostgreSQL instance. Therefore, the connection from the `local` socket must be trusted. @@ -1297,6 +828,16 @@ a manual one, where you have two slightly different options: For further details on this subject, see the [Patroni documentation](https://patroni.readthedocs.io/en/latest/rest_api.html#switchover-and-failover-endpoints). +#### Geo secondary site considerations + +Similar to `repmgr`, when a Geo secondary site is replicating from a primary site that uses `Patroni` and `PgBouncer`, [replicating through PgBouncer is not supported](https://github.com/pgbouncer/pgbouncer/issues/382#issuecomment-517911529) and the secondary must replicate directly from the leader node in the `Patroni` cluster. Therefore, when there is an automatic or manual failover in the `Patroni` cluster, you will need to manually re-point your secondary site to replicate from the new leader with: + +```shell +sudo gitlab-ctl replicate-geo-database --host=<new_leader_ip> --replication-slot=<slot_name> +``` + +Otherwise, the replication will not happen anymore, even if the original node gets re-added as a follower node. This will re-sync your secondary site database and may take a long time depending on the amount of data to sync. You may also need to run `gitlab-ctl reconfigure` if replication is still not working after re-syncing. + ### Recovering the Patroni cluster To recover the old primary and rejoin it to the cluster as a replica, you can simply start Patroni with: @@ -1363,9 +904,9 @@ You can switch an exiting database cluster to use Patroni instead of repmgr with ### Upgrading PostgreSQL major version in a Patroni cluster -As of GitLab 13.3, PostgreSQL 11.7 and 12.3 are both shipped with Omnibus GitLab. GitLab still -uses PostgreSQL 11 by default. Therefore `gitlab-ctl pg-upgrade` does not automatically upgrade -to PostgreSQL 12. If you want to upgrade to PostgreSQL 12, you must ask for it explicitly. +As of GitLab 13.3, PostgreSQL 11.7 and 12.3 are both shipped with Omnibus GitLab, and as of GitLab 13.7 +PostgreSQL 12 is used by default. If you want to upgrade to PostgreSQL 12 in versions prior to GitLab 13.7, +you must ask for it explicitly. WARNING: The procedure for upgrading PostgreSQL in a Patroni cluster is different than when upgrading using repmgr. @@ -1450,3 +991,448 @@ NOTE: Reverting PostgreSQL upgrade with `gitlab-ctl revert-pg-upgrade` has the same considerations as `gitlab-ctl pg-upgrade`. You should follow the same procedure by first stopping the replicas, then reverting the leader, and finally reverting the replicas. + +## Repmgr + +NOTE: +Using Patroni instead of Repmgr is supported for PostgreSQL 11 and required for PostgreSQL 12. + +### Configuring Repmgr Nodes + +1. On the master database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: + + ```ruby + # Disable all components except PostgreSQL and Repmgr and Consul + roles ['postgres_role'] + + # PostgreSQL configuration + postgresql['listen_address'] = '0.0.0.0' + postgresql['hot_standby'] = 'on' + postgresql['wal_level'] = 'replica' + postgresql['shared_preload_libraries'] = 'repmgr_funcs' + + # Disable automatic database migrations + gitlab_rails['auto_migrate'] = false + + # Configure the Consul agent + consul['services'] = %w(postgresql) + + # START user configuration + # Please set the real values as explained in Required Information section + # + # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value + postgresql['pgbouncer_user_password'] = 'PGBOUNCER_PASSWORD_HASH' + # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value + postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' + # Replace X with value of number of db nodes + 1 + postgresql['max_wal_senders'] = X + postgresql['max_replication_slots'] = X + + # Replace XXX.XXX.XXX.XXX/YY with Network Address + postgresql['trust_auth_cidr_addresses'] = %w(XXX.XXX.XXX.XXX/YY) + repmgr['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 XXX.XXX.XXX.XXX/YY) + + # Replace placeholders: + # + # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z + # with the addresses gathered for CONSUL_SERVER_NODES + consul['configuration'] = { + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z) + } + # + # END user configuration + ``` + + > `postgres_role` was introduced with GitLab 10.3 + +1. On secondary nodes, add all the configuration specified above for primary node + to `/etc/gitlab/gitlab.rb`. In addition, append the following configuration + to inform `gitlab-ctl` that they are standby nodes initially and it need not + attempt to register them as primary node + + ```ruby + # Specify if a node should attempt to be master on initialization + repmgr['master_on_initialization'] = false + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Enable Monitoring](#enable-monitoring) + +> Please note: +> +> - If you want your database to listen on a specific interface, change the configuration: +> `postgresql['listen_address'] = '0.0.0.0'`. +> - If your PgBouncer service runs under a different user account, +> you also need to specify: `postgresql['pgbouncer_user'] = PGBOUNCER_USERNAME` in +> your configuration. + +#### Database nodes post-configuration + +##### Primary node + +Select one node as a primary node. + +1. Open a database prompt: + + ```shell + gitlab-psql -d gitlabhq_production + ``` + +1. Enable the `pg_trgm` extension: + + ```shell + CREATE EXTENSION pg_trgm; + ``` + +1. Enable the `btree_gist` extension: + + ```shell + CREATE EXTENSION btree_gist; + ``` + +1. Exit the database prompt by typing `\q` and Enter. + +1. Verify the cluster is initialized with one node: + + ```shell + gitlab-ctl repmgr cluster show + ``` + + The output should be similar to the following: + + ```plaintext + Role | Name | Upstream | Connection String + ----------+----------|----------|---------------------------------------- + * master | HOSTNAME | | host=HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr + ``` + +1. Note down the hostname or IP address in the connection string: `host=HOSTNAME`. We will + refer to the hostname in the next section as `MASTER_NODE_NAME`. If the value + is not an IP address, it will need to be a resolvable name (via DNS or + `/etc/hosts`) + +##### Secondary nodes + +1. Set up the repmgr standby: + + ```shell + gitlab-ctl repmgr standby setup MASTER_NODE_NAME + ``` + + Do note that this will remove the existing data on the node. The command + has a wait time. + + The output should be similar to the following: + + ```console + # gitlab-ctl repmgr standby setup MASTER_NODE_NAME + Doing this will delete the entire contents of /var/opt/gitlab/postgresql/data + If this is not what you want, hit Ctrl-C now to exit + To skip waiting, rerun with the -w option + Sleeping for 30 seconds + Stopping the database + Removing the data + Cloning the data + Starting the database + Registering the node with the cluster + ok: run: repmgrd: (pid 19068) 0s + ``` + +1. Verify the node now appears in the cluster: + + ```shell + gitlab-ctl repmgr cluster show + ``` + + The output should be similar to the following: + + ```plaintext + Role | Name | Upstream | Connection String + ----------+---------|-----------|------------------------------------------------ + * master | MASTER | | host=MASTER_NODE_NAME user=gitlab_repmgr dbname=gitlab_repmgr + standby | STANDBY | MASTER | host=STANDBY_HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr + ``` + +Repeat the above steps on all secondary nodes. + +#### Database checkpoint + +Before moving on, make sure the databases are configured correctly. Run the +following command on the **primary** node to verify that replication is working +properly: + +```shell +gitlab-ctl repmgr cluster show +``` + +The output should be similar to: + +```plaintext +Role | Name | Upstream | Connection String +----------+--------------|--------------|-------------------------------------------------------------------- +* master | MASTER | | host=MASTER port=5432 user=gitlab_repmgr dbname=gitlab_repmgr + standby | STANDBY | MASTER | host=STANDBY port=5432 user=gitlab_repmgr dbname=gitlab_repmgr +``` + +If the 'Role' column for any node says "FAILED", check the +[Troubleshooting section](#troubleshooting) before proceeding. + +Also, check that the check master command works successfully on each node: + +```shell +su - gitlab-consul +gitlab-ctl repmgr-check-master || echo 'This node is a standby repmgr node' +``` + +This command relies on exit codes to tell Consul whether a particular node is a master +or secondary. The most important thing here is that this command does not produce errors. +If there are errors it's most likely due to incorrect `gitlab-consul` database user permissions. +Check the [Troubleshooting section](#troubleshooting) before proceeding. + +### Repmgr failover procedure + +By default, if the master database fails, `repmgrd` should promote one of the +standby nodes to master automatically, and Consul will update PgBouncer with +the new master. + +If you need to failover manually, you have two options: + +**Shutdown the current master database** + +Run: + +```shell +gitlab-ctl stop postgresql +``` + +The automated failover process will see this and failover to one of the +standby nodes. + +**Or perform a manual failover** + +1. Ensure the old master node is not still active. +1. Login to the server that should become the new master and run: + + ```shell + gitlab-ctl repmgr standby promote + ``` + +1. If there are any other standby servers in the cluster, have them follow + the new master server: + + ```shell + gitlab-ctl repmgr standby follow NEW_MASTER + ``` + +#### Geo secondary site considerations + +When a Geo secondary site is replicating from a primary site that uses `repmgr` and `PgBouncer`, [replicating through PgBouncer is not supported](https://github.com/pgbouncer/pgbouncer/issues/382#issuecomment-517911529) and the secondary must replicate directly from the leader node in the `repmgr` cluster. Therefore, when there is a failover in the `repmgr` cluster, you will need to manually re-point your secondary site to replicate from the new leader with: + +```shell +sudo gitlab-ctl replicate-geo-database --host=<new_leader_ip> --replication-slot=<slot_name> +``` + +Otherwise, the replication will not happen anymore, even if the original node gets re-added as a follower node. This will re-sync your secondary site database and may take a long time depending on the amount of data to sync. You may also need to run `gitlab-ctl reconfigure` if replication is still not working after re-syncing. + +### Repmgr Restore procedure + +If a node fails, it can be removed from the cluster, or added back as a standby +after it has been restored to service. + +#### Remove a standby from the cluster + + From any other node in the cluster, run: + + ```shell + gitlab-ctl repmgr standby unregister --node=X + ``` + + where X is the value of node in `repmgr.conf` on the old server. + + To find this, you can use: + + ```shell + awk -F = '$1 == "node" { print $2 }' /var/opt/gitlab/postgresql/repmgr.conf + ``` + + It will output something like: + + ```plaintext + 959789412 + ``` + + Then you will use this ID to unregister the node: + + ```shell + gitlab-ctl repmgr standby unregister --node=959789412 + ``` + +#### Add a node as a standby server + + From the standby node, run: + + ```shell + gitlab-ctl repmgr standby follow NEW_MASTER + gitlab-ctl restart repmgrd + ``` + + WARNING: + When the server is brought back online, and before + you switch it to a standby node, repmgr will report that there are two masters. + If there are any clients that are still attempting to write to the old master, + this will cause a split, and the old master will need to be resynced from + scratch by performing a `gitlab-ctl repmgr standby setup NEW_MASTER`. + +#### Add a failed master back into the cluster as a standby node + + Once `repmgrd` and PostgreSQL are running, the node will need to follow the new + as a standby node. + + ```shell + gitlab-ctl repmgr standby follow NEW_MASTER + ``` + + Once the node is following the new master as a standby, the node needs to be + [unregistered from the cluster on the new master node](#remove-a-standby-from-the-cluster). + + Once the old master node has been unregistered from the cluster, it will need + to be setup as a new standby: + + ```shell + gitlab-ctl repmgr standby setup NEW_MASTER + ``` + + Failure to unregister and read the old master node can lead to subsequent failovers + not working. + +### Alternate configurations + +#### Database authorization + +By default, we give any host on the database network the permission to perform +repmgr operations using PostgreSQL's `trust` method. If you do not want this +level of trust, there are alternatives. + +You can trust only the specific nodes that will be database clusters, or you +can require md5 authentication. + +#### Trust specific addresses + +If you know the IP address, or FQDN of all database and PgBouncer nodes in the +cluster, you can trust only those nodes. + +In `/etc/gitlab/gitlab.rb` on all of the database nodes, set +`repmgr['trust_auth_cidr_addresses']` to an array of strings containing all of +the addresses. + +If setting to a node's FQDN, they must have a corresponding PTR record in DNS. +If setting to a node's IP address, specify it as `XXX.XXX.XXX.XXX/32`. + +For example: + +```ruby +repmgr['trust_auth_cidr_addresses'] = %w(192.168.1.44/32 db2.example.com) +``` + +#### MD5 Authentication + +If you are running on an untrusted network, repmgr can use md5 authentication +with a [`.pgpass` file](https://www.postgresql.org/docs/11/libpq-pgpass.html) +to authenticate. + +You can specify by IP address, FQDN, or by subnet, using the same format as in +the previous section: + +1. On the current master node, create a password for the `gitlab` and + `gitlab_repmgr` user: + + ```shell + gitlab-psql -d template1 + template1=# \password gitlab_repmgr + Enter password: **** + Confirm password: **** + template1=# \password gitlab + ``` + +1. On each database node: + + 1. Edit `/etc/gitlab/gitlab.rb`: + 1. Ensure `repmgr['trust_auth_cidr_addresses']` is **not** set + 1. Set `postgresql['md5_auth_cidr_addresses']` to the desired value + 1. Set `postgresql['sql_replication_user'] = 'gitlab_repmgr'` + 1. Reconfigure with `gitlab-ctl reconfigure` + 1. Restart PostgreSQL with `gitlab-ctl restart postgresql` + + 1. Create a `.pgpass` file. Enter the `gitlab_repmgr` password twice to + when asked: + + ```shell + gitlab-ctl write-pgpass --user gitlab_repmgr --hostuser gitlab-psql --database '*' + ``` + +1. On each PgBouncer node, edit `/etc/gitlab/gitlab.rb`: + 1. Ensure `gitlab_rails['db_password']` is set to the plaintext password for + the `gitlab` database user + 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect + +## Troubleshooting + +### Consul and PostgreSQL changes not taking effect + +Due to the potential impacts, `gitlab-ctl reconfigure` only reloads Consul and PostgreSQL, it will not restart the services. However, not all changes can be activated by reloading. + +To restart either service, run `gitlab-ctl restart SERVICE` + +For PostgreSQL, it is usually safe to restart the master node by default. Automatic failover defaults to a 1 minute timeout. Provided the database returns before then, nothing else needs to be done. To be safe, you can stop `repmgrd` on the standby nodes first with `gitlab-ctl stop repmgrd`, then start afterwards with `gitlab-ctl start repmgrd`. + +On the Consul server nodes, it is important to [restart the Consul service](../consul.md#restart-consul) in a controlled manner. + +### `gitlab-ctl repmgr-check-master` command produces errors + +If this command displays errors about database permissions it is likely that something failed during +install, resulting in the `gitlab-consul` database user getting incorrect permissions. Follow these +steps to fix the problem: + +1. On the master database node, connect to the database prompt - `gitlab-psql -d template1` +1. Delete the `gitlab-consul` user - `DROP USER "gitlab-consul";` +1. Exit the database prompt - `\q` +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) and the user will be re-added with the proper permissions. +1. Change to the `gitlab-consul` user - `su - gitlab-consul` +1. Try the check command again - `gitlab-ctl repmgr-check-master`. + +Now there should not be errors. If errors still occur then there is another problem. + +### PgBouncer error `ERROR: pgbouncer cannot connect to server` + +You may get this error when running `gitlab-rake gitlab:db:configure` or you +may see the error in the PgBouncer log file. + +```plaintext +PG::ConnectionBad: ERROR: pgbouncer cannot connect to server +``` + +The problem may be that your PgBouncer node's IP address is not included in the +`trust_auth_cidr_addresses` setting in `/etc/gitlab/gitlab.rb` on the database nodes. + +You can confirm that this is the issue by checking the PostgreSQL log on the master +database node. If you see the following error then `trust_auth_cidr_addresses` +is the problem. + +```plaintext +2018-03-29_13:59:12.11776 FATAL: no pg_hba.conf entry for host "123.123.123.123", user "pgbouncer", database "gitlabhq_production", SSL off +``` + +To fix the problem, add the IP address to `/etc/gitlab/gitlab.rb`. + +```ruby +postgresql['trust_auth_cidr_addresses'] = %w(123.123.123.123/32 <other_cidrs>) +``` + +[Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +### Issues with other components + +If you're running into an issue with a component not outlined here, be sure to check the troubleshooting section of their specific documentation page: + +- [Consul](../consul.md#troubleshooting-consul) +- [PostgreSQL](https://docs.gitlab.com/omnibus/settings/database.html#troubleshooting) diff --git a/doc/administration/raketasks/check.md b/doc/administration/raketasks/check.md index 25869fd425d..41defd89df5 100644 --- a/doc/administration/raketasks/check.md +++ b/doc/administration/raketasks/check.md @@ -51,6 +51,62 @@ sudo gitlab-rake gitlab:git:fsck sudo -u git -H bundle exec rake gitlab:git:fsck RAILS_ENV=production ``` +## Checksum of repository refs + +One Git repository can be compared to another by checksumming all refs of each +repository. If both repositories have the same refs, and if both repositories +pass an integrity check, then we can be confident that both repositories are the +same. + +For example, this can be used to compare a backup of a repository against the +source repository. + +### Check all GitLab repositories + +This task loops through all repositories on the GitLab server and outputs +checksums in the format `<PROJECT ID>,<CHECKSUM>`. + +- If a repository doesn't exist, the project ID will have a blank checksum. +- If a repository exists but is empty, the output checksum is `0000000000000000000000000000000000000000`. +- Projects which don't exist are skipped. + +**Omnibus Installation** + +```shell +sudo gitlab-rake gitlab:git:checksum_projects +``` + +**Source Installation** + +```shell +sudo -u git -H bundle exec rake gitlab:git:checksum_projects RAILS_ENV=production +``` + +For example, if: + +- Project with ID#2 doesn't exist, it will be skipped. +- Project with ID#4 doesn't have a repository, its checksum will be blank. +- Project with ID#5 has an empty repository, its checksum will be `0000000000000000000000000000000000000000`. + +The output would then look something like: + +```plaintext +1,cfa3f06ba235c13df0bb28e079bcea62c5848af2 +3,3f3fb58a8106230e3a6c6b48adc2712fb3b6ef87 +4, +5,0000000000000000000000000000000000000000 +6,6c6b48adc2712fb3b6ef87cfa3f06ba235c13df0 +``` + +### Check specific GitLab repositories + +Optionally, specific project IDs can be checksummed by setting an environment +variable `CHECKSUM_PROJECT_IDS` with a list of comma-separated integers, for example: + +```shell +CHECKSUM_PROJECT_IDS="1,3" sudo gitlab-rake gitlab:git:checksum_projects +``` + ## Uploaded files integrity Various types of files can be uploaded to a GitLab installation by users. @@ -158,7 +214,7 @@ If the issue persists, try triggering `gc` via the ```ruby p = Project.find_by_path("project-name") -Projects::HousekeepingService.new(p, :gc).execute +Repositories::HousekeepingService.new(p, :gc).execute ``` ### Delete references to missing remote uploads diff --git a/doc/administration/raketasks/github_import.md b/doc/administration/raketasks/github_import.md index 5c0bc721a9a..630570cb81f 100644 --- a/doc/administration/raketasks/github_import.md +++ b/doc/administration/raketasks/github_import.md @@ -19,7 +19,7 @@ before/after the brackets. Also, some shells (for example, `zsh`) can interpret ## Caveats -If the GitHub [rate limit](https://developer.github.com/v3/#rate-limiting) is reached while importing, +If the GitHub [rate limit](https://docs.github.com/v3/#rate-limiting) is reached while importing, the importing process waits (`sleep()`) until it can continue importing. ## Importing multiple projects diff --git a/doc/administration/raketasks/storage.md b/doc/administration/raketasks/storage.md index c7afebf15bd..158b541b36b 100644 --- a/doc/administration/raketasks/storage.md +++ b/doc/administration/raketasks/storage.md @@ -82,8 +82,8 @@ Support for legacy storage will be removed in GitLab 14.0. If you're on GitLab The option to choose between hashed and legacy storage in the admin area has been disabled. -This task must be run on any machine that has Rails/Sidekiq configured and will -schedule all your existing projects and attachments associated with it to be +This task must be run on any machine that has Rails/Sidekiq configured and will +schedule all your existing projects and attachments associated with it to be migrated to the **Hashed** storage type: - **Omnibus installation** diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index 216d3867d0a..afa53b5efa8 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -422,9 +422,9 @@ install the necessary dependencies from step 1, and add the GitLab package repository from step 2. When installing GitLab in the second step, do not supply the `EXTERNAL_URL` value. -#### PostgreSQL primary node +#### PostgreSQL nodes -1. SSH in to the PostgreSQL primary node. +1. SSH in to one of the PostgreSQL nodes. 1. Generate a password hash for the PostgreSQL username/password pair. This assumes you will use the default username of `gitlab` (recommended). The command will request a password and confirmation. Use the value that is output by this command in the next @@ -452,23 +452,33 @@ in the second step, do not supply the `EXTERNAL_URL` value. sudo gitlab-ctl pg-password-md5 gitlab-consul ``` -1. On the primary database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: +1. On every database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: ```ruby - # Disable all components except PostgreSQL and Repmgr and Consul + # Disable all components except PostgreSQL, Patroni, and Consul roles ['postgres_role'] # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' - postgresql['hot_standby'] = 'on' - postgresql['wal_level'] = 'replica' - postgresql['shared_preload_libraries'] = 'repmgr_funcs' + + # Enable Patroni + patroni['enable'] = true + # Set `max_wal_senders` to one more than the number of database nodes in the cluster. + # This is used to prevent replication from using up all of the + # available database connections. + patroni['postgresql']['max_wal_senders'] = 4 + patroni['postgresql']['max_replication_slots'] = 4 + # Incoming recommended value for max connections is 500. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691. + patroni['postgresql']['max_connections'] = 500 # Disable automatic database migrations gitlab_rails['auto_migrate'] = false # Configure the Consul agent + consul['enable'] = true consul['services'] = %w(postgresql) + ## Enable service discovery for Prometheus + consul['monitoring_service_discovery'] = true # START user configuration # Please set the real values as explained in Required Information section @@ -477,18 +487,9 @@ in the second step, do not supply the `EXTERNAL_URL` value. postgresql['pgbouncer_user_password'] = '<pgbouncer_password_hash>' # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value postgresql['sql_user_password'] = '<postgresql_password_hash>' - # Set `max_wal_senders` to one more than the number of database nodes in the cluster. - # This is used to prevent replication from using up all of the - # available database connections. - postgresql['max_wal_senders'] = 4 - postgresql['max_replication_slots'] = 4 # Replace XXX.XXX.XXX.XXX/YY with Network Address postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/24) - repmgr['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 10.6.0.0/24) - - ## Enable service discovery for Prometheus - consul['monitoring_service_discovery'] = true # Set the network addresses that the exporters will listen on for monitoring node_exporter['listen_address'] = '0.0.0.0:9100' @@ -503,70 +504,9 @@ in the second step, do not supply the `EXTERNAL_URL` value. # END user configuration ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. - -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -<div align="right"> - <a type="button" class="btn btn-default" href="#setup-components"> - Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> - </a> -</div> - -#### PostgreSQL secondary nodes - -1. On both the secondary nodes, add the same configuration specified above for the primary node - with an additional setting (`repmgr['master_on_initialization'] = false`) that will inform `gitlab-ctl` that they are standby nodes initially - and there's no need to attempt to register them as a primary node: - - ```ruby - # Disable all components except PostgreSQL and Repmgr and Consul - roles ['postgres_role'] - - # PostgreSQL configuration - postgresql['listen_address'] = '0.0.0.0' - postgresql['hot_standby'] = 'on' - postgresql['wal_level'] = 'replica' - postgresql['shared_preload_libraries'] = 'repmgr_funcs' - - # Disable automatic database migrations - gitlab_rails['auto_migrate'] = false - - # Configure the Consul agent - consul['services'] = %w(postgresql) - - # Specify if a node should attempt to be primary on initialization. - repmgr['master_on_initialization'] = false - - # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value - postgresql['pgbouncer_user_password'] = '<pgbouncer_password_hash>' - # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value - postgresql['sql_user_password'] = '<postgresql_password_hash>' - # Set `max_wal_senders` to one more than the number of database nodes in the cluster. - # This is used to prevent replication from using up all of the - # available database connections. - postgresql['max_wal_senders'] = 4 - postgresql['max_replication_slots'] = 4 - - # Replace with your network addresses - postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/24) - repmgr['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 10.6.0.0/24) - - ## Enable service discovery for Prometheus - consul['monitoring_service_discovery'] = true - - # Set the network addresses that the exporters will listen on for monitoring - node_exporter['listen_address'] = '0.0.0.0:9100' - postgres_exporter['listen_address'] = '0.0.0.0:9187' - - ## The IPs of the Consul server nodes - ## You can also use FQDNs and intermix them with IPs - consul['configuration'] = { - retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), - } - ``` +PostgreSQL, with Patroni managing its failover, will default to use `pg_rewind` by default to handle conflicts. +Like most failover handling methods, this has a small chance of leading to data loss. +Learn more about the various [Patroni replication methods](../postgresql/replication_and_failover.md#selecting-the-appropriate-patroni-replication-method). 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace the file of the same name on this server. If that file is not on this server, @@ -601,84 +541,25 @@ SSH in to the **primary node**: 1. Exit the database prompt by typing `\q` and Enter. -1. Verify the cluster is initialized with one node: +1. Check the status of the leader and cluster: ```shell - gitlab-ctl repmgr cluster show + gitlab-ctl patroni members ``` The output should be similar to the following: ```plaintext - Role | Name | Upstream | Connection String - ----------+----------|----------|---------------------------------------- - * master | HOSTNAME | | host=HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr - ``` - -1. Note down the hostname or IP address in the connection string: `host=HOSTNAME`. We will - refer to the hostname in the next section as `<primary_node_name>`. If the value - is not an IP address, it will need to be a resolvable name (via DNS or - `/etc/hosts`) - -SSH in to the **secondary node**: - -1. Set up the repmgr standby: - - ```shell - gitlab-ctl repmgr standby setup <primary_node_name> + | Cluster | Member | Host | Role | State | TL | Lag in MB | Pending restart | + |---------------|-----------------------------------|-----------|--------|---------|-----|-----------|-----------------| + | postgresql-ha | <PostgreSQL primary hostname> | 10.6.0.21 | Leader | running | 175 | | * | + | postgresql-ha | <PostgreSQL secondary 1 hostname> | 10.6.0.22 | | running | 175 | 0 | * | + | postgresql-ha | <PostgreSQL secondary 2 hostname> | 10.6.0.23 | | running | 175 | 0 | * | ``` - Do note that this will remove the existing data on the node. The command - has a wait time. - - The output should be similar to the following: - - ```console - Doing this will delete the entire contents of /var/opt/gitlab/postgresql/data - If this is not what you want, hit Ctrl-C now to exit - To skip waiting, rerun with the -w option - Sleeping for 30 seconds - Stopping the database - Removing the data - Cloning the data - Starting the database - Registering the node with the cluster - ok: run: repmgrd: (pid 19068) 0s - ``` - -Before moving on, make sure the databases are configured correctly. Run the -following command on the **primary** node to verify that replication is working -properly and the secondary nodes appear in the cluster: - -```shell -gitlab-ctl repmgr cluster show -``` - -The output should be similar to the following: - -```plaintext -Role | Name | Upstream | Connection String -----------+---------|-----------|------------------------------------------------ -* master | MASTER | | host=<primary_node_name> user=gitlab_repmgr dbname=gitlab_repmgr - standby | STANDBY | MASTER | host=<secondary_node_name> user=gitlab_repmgr dbname=gitlab_repmgr - standby | STANDBY | MASTER | host=<secondary_node_name> user=gitlab_repmgr dbname=gitlab_repmgr -``` - -If the 'Role' column for any node says "FAILED", check the +If the 'State' column for any node doesn't say "running", check the [Troubleshooting section](troubleshooting.md) before proceeding. -Also, check that the `repmgr-check-master` command works successfully on each node: - -```shell -su - gitlab-consul -gitlab-ctl repmgr-check-master || echo 'This node is a standby repmgr node' -``` - -This command relies on exit codes to tell Consul whether a particular node is a master -or secondary. The most important thing here is that this command does not produce errors. -If there are errors it's most likely due to incorrect `gitlab-consul` database user permissions. -Check the [Troubleshooting section](troubleshooting.md) before proceeding. - <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> @@ -696,7 +577,7 @@ The following IPs will be used as an example: 1. On each PgBouncer node, edit `/etc/gitlab/gitlab.rb`, and replace `<consul_password_hash>` and `<pgbouncer_password_hash>` with the - password hashes you [set up previously](#postgresql-primary-node): + password hashes you [set up previously](#postgresql-nodes): ```ruby # Disable all components except Pgbouncer and Consul agent @@ -704,15 +585,16 @@ The following IPs will be used as an example: # Configure PgBouncer pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) - pgbouncer['users'] = { - 'gitlab-consul': { - password: '<consul_password_hash>' - }, - 'pgbouncer': { - password: '<pgbouncer_password_hash>' - } + 'gitlab-consul': { + password: '<consul_password_hash>' + }, + 'pgbouncer': { + password: '<pgbouncer_password_hash>' + } } + # Incoming recommended value for max db connections is 150. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691. + pgbouncer['max_db_connections'] = 150 # Configure Consul agent consul['watchers'] = %w(postgresql) @@ -2125,6 +2007,12 @@ to use GitLab Pages, this currently [requires NFS](troubleshooting.md#gitlab-pag See how to [configure NFS](../nfs.md). +WARNING: +From GitLab 13.0, using NFS for Git repositories is deprecated. +From GitLab 14.0, technical support for NFS for Git repositories +will no longer be provided. Upgrade to [Gitaly Cluster](../gitaly/praefect.md) +as soon as possible. + <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> diff --git a/doc/administration/reference_architectures/1k_users.md b/doc/administration/reference_architectures/1k_users.md index 5b75c7ac9c4..161964353f5 100644 --- a/doc/administration/reference_architectures/1k_users.md +++ b/doc/administration/reference_architectures/1k_users.md @@ -16,7 +16,8 @@ requirements, a single-node solution with many organizations . > - **Supported users (approximate):** 1,000 -> - **High Availability:** No +> - **High Availability:** No. For a highly-available environment, you can +> follow the [3K reference architecture](3k_users.md). | Users | Configuration | GCP | AWS | Azure | |--------------|-------------------------|----------------|-----------------|----------------| diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index c5a8c3927c0..d96e93d4ab4 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -422,9 +422,9 @@ install the necessary dependencies from step 1, and add the GitLab package repository from step 2. When installing GitLab in the second step, do not supply the `EXTERNAL_URL` value. -#### PostgreSQL primary node +#### PostgreSQL nodes -1. SSH in to the PostgreSQL primary node. +1. SSH in to one of the PostgreSQL nodes. 1. Generate a password hash for the PostgreSQL username/password pair. This assumes you will use the default username of `gitlab` (recommended). The command will request a password and confirmation. Use the value that is output by this command in the next @@ -452,23 +452,33 @@ in the second step, do not supply the `EXTERNAL_URL` value. sudo gitlab-ctl pg-password-md5 gitlab-consul ``` -1. On the primary database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: +1. On every database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: ```ruby - # Disable all components except PostgreSQL and Repmgr and Consul + # Disable all components except PostgreSQL, Patroni, and Consul roles ['postgres_role'] # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' - postgresql['hot_standby'] = 'on' - postgresql['wal_level'] = 'replica' - postgresql['shared_preload_libraries'] = 'repmgr_funcs' + + # Enable Patroni + patroni['enable'] = true + # Set `max_wal_senders` to one more than the number of database nodes in the cluster. + # This is used to prevent replication from using up all of the + # available database connections. + patroni['postgresql']['max_wal_senders'] = 4 + patroni['postgresql']['max_replication_slots'] = 4 + # Incoming recommended value for max connections is 500. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691. + patroni['postgresql']['max_connections'] = 500 # Disable automatic database migrations gitlab_rails['auto_migrate'] = false # Configure the Consul agent + consul['enable'] = true consul['services'] = %w(postgresql) + ## Enable service discovery for Prometheus + consul['monitoring_service_discovery'] = true # START user configuration # Please set the real values as explained in Required Information section @@ -477,18 +487,9 @@ in the second step, do not supply the `EXTERNAL_URL` value. postgresql['pgbouncer_user_password'] = '<pgbouncer_password_hash>' # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value postgresql['sql_user_password'] = '<postgresql_password_hash>' - # Set `max_wal_senders` to one more than the number of database nodes in the cluster. - # This is used to prevent replication from using up all of the - # available database connections. - postgresql['max_wal_senders'] = 4 - postgresql['max_replication_slots'] = 4 # Replace XXX.XXX.XXX.XXX/YY with Network Address postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/24) - repmgr['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 10.6.0.0/24) - - ## Enable service discovery for Prometheus - consul['monitoring_service_discovery'] = true # Set the network addresses that the exporters will listen on for monitoring node_exporter['listen_address'] = '0.0.0.0:9100' @@ -503,70 +504,9 @@ in the second step, do not supply the `EXTERNAL_URL` value. # END user configuration ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. - -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -<div align="right"> - <a type="button" class="btn btn-default" href="#setup-components"> - Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> - </a> -</div> - -#### PostgreSQL secondary nodes - -1. On both the secondary nodes, add the same configuration specified above for the primary node - with an additional setting (`repmgr['master_on_initialization'] = false`) that will inform `gitlab-ctl` that they are standby nodes initially - and there's no need to attempt to register them as a primary node: - - ```ruby - # Disable all components except PostgreSQL and Repmgr and Consul - roles ['postgres_role'] - - # PostgreSQL configuration - postgresql['listen_address'] = '0.0.0.0' - postgresql['hot_standby'] = 'on' - postgresql['wal_level'] = 'replica' - postgresql['shared_preload_libraries'] = 'repmgr_funcs' - - # Disable automatic database migrations - gitlab_rails['auto_migrate'] = false - - # Configure the Consul agent - consul['services'] = %w(postgresql) - - # Specify if a node should attempt to be primary on initialization. - repmgr['master_on_initialization'] = false - - # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value - postgresql['pgbouncer_user_password'] = '<pgbouncer_password_hash>' - # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value - postgresql['sql_user_password'] = '<postgresql_password_hash>' - # Set `max_wal_senders` to one more than the number of database nodes in the cluster. - # This is used to prevent replication from using up all of the - # available database connections. - postgresql['max_wal_senders'] = 4 - postgresql['max_replication_slots'] = 4 - - # Replace with your network addresses - postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/24) - repmgr['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 10.6.0.0/24) - - ## Enable service discovery for Prometheus - consul['monitoring_service_discovery'] = true - - # Set the network addresses that the exporters will listen on for monitoring - node_exporter['listen_address'] = '0.0.0.0:9100' - postgres_exporter['listen_address'] = '0.0.0.0:9187' - - ## The IPs of the Consul server nodes - ## You can also use FQDNs and intermix them with IPs - consul['configuration'] = { - retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), - } - ``` +PostgreSQL, with Patroni managing its failover, will default to use `pg_rewind` by default to handle conflicts. +Like most failover handling methods, this has a small chance of leading to data loss. +Learn more about the various [Patroni replication methods](../postgresql/replication_and_failover.md#selecting-the-appropriate-patroni-replication-method). 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace the file of the same name on this server. If that file is not on this server, @@ -601,84 +541,25 @@ SSH in to the **primary node**: 1. Exit the database prompt by typing `\q` and Enter. -1. Verify the cluster is initialized with one node: +1. Check the status of the leader and cluster: ```shell - gitlab-ctl repmgr cluster show + gitlab-ctl patroni members ``` The output should be similar to the following: ```plaintext - Role | Name | Upstream | Connection String - ----------+----------|----------|---------------------------------------- - * master | HOSTNAME | | host=HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr - ``` - -1. Note down the hostname or IP address in the connection string: `host=HOSTNAME`. We will - refer to the hostname in the next section as `<primary_node_name>`. If the value - is not an IP address, it will need to be a resolvable name (via DNS or - `/etc/hosts`) - -SSH in to the **secondary node**: - -1. Set up the repmgr standby: - - ```shell - gitlab-ctl repmgr standby setup <primary_node_name> + | Cluster | Member | Host | Role | State | TL | Lag in MB | Pending restart | + |---------------|-----------------------------------|-----------|--------|---------|-----|-----------|-----------------| + | postgresql-ha | <PostgreSQL primary hostname> | 10.6.0.21 | Leader | running | 175 | | * | + | postgresql-ha | <PostgreSQL secondary 1 hostname> | 10.6.0.22 | | running | 175 | 0 | * | + | postgresql-ha | <PostgreSQL secondary 2 hostname> | 10.6.0.23 | | running | 175 | 0 | * | ``` - Do note that this will remove the existing data on the node. The command - has a wait time. - - The output should be similar to the following: - - ```console - Doing this will delete the entire contents of /var/opt/gitlab/postgresql/data - If this is not what you want, hit Ctrl-C now to exit - To skip waiting, rerun with the -w option - Sleeping for 30 seconds - Stopping the database - Removing the data - Cloning the data - Starting the database - Registering the node with the cluster - ok: run: repmgrd: (pid 19068) 0s - ``` - -Before moving on, make sure the databases are configured correctly. Run the -following command on the **primary** node to verify that replication is working -properly and the secondary nodes appear in the cluster: - -```shell -gitlab-ctl repmgr cluster show -``` - -The output should be similar to the following: - -```plaintext -Role | Name | Upstream | Connection String -----------+---------|-----------|------------------------------------------------ -* master | MASTER | | host=<primary_node_name> user=gitlab_repmgr dbname=gitlab_repmgr - standby | STANDBY | MASTER | host=<secondary_node_name> user=gitlab_repmgr dbname=gitlab_repmgr - standby | STANDBY | MASTER | host=<secondary_node_name> user=gitlab_repmgr dbname=gitlab_repmgr -``` - -If the 'Role' column for any node says "FAILED", check the +If the 'State' column for any node doesn't say "running", check the [Troubleshooting section](troubleshooting.md) before proceeding. -Also, check that the `repmgr-check-master` command works successfully on each node: - -```shell -su - gitlab-consul -gitlab-ctl repmgr-check-master || echo 'This node is a standby repmgr node' -``` - -This command relies on exit codes to tell Consul whether a particular node is a master -or secondary. The most important thing here is that this command does not produce errors. -If there are errors it's most likely due to incorrect `gitlab-consul` database user permissions. -Check the [Troubleshooting section](troubleshooting.md) before proceeding. - <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> @@ -696,7 +577,7 @@ The following IPs will be used as an example: 1. On each PgBouncer node, edit `/etc/gitlab/gitlab.rb`, and replace `<consul_password_hash>` and `<pgbouncer_password_hash>` with the - password hashes you [set up previously](#postgresql-primary-node): + password hashes you [set up previously](#postgresql-nodes): ```ruby # Disable all components except Pgbouncer and Consul agent @@ -704,15 +585,16 @@ The following IPs will be used as an example: # Configure PgBouncer pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) - pgbouncer['users'] = { - 'gitlab-consul': { - password: '<consul_password_hash>' - }, - 'pgbouncer': { - password: '<pgbouncer_password_hash>' - } + 'gitlab-consul': { + password: '<consul_password_hash>' + }, + 'pgbouncer': { + password: '<pgbouncer_password_hash>' + } } + # Incoming recommended value for max db connections is 150. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691. + pgbouncer['max_db_connections'] = 150 # Configure Consul agent consul['watchers'] = %w(postgresql) @@ -2125,6 +2007,12 @@ to use GitLab Pages, this currently [requires NFS](troubleshooting.md#gitlab-pag See how to [configure NFS](../nfs.md). +WARNING: +From GitLab 13.0, using NFS for Git repositories is deprecated. +From GitLab 14.0, technical support for NFS for Git repositories +will no longer be provided. Upgrade to [Gitaly Cluster](../gitaly/praefect.md) +as soon as possible. + <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index c341ff5baec..3d38586fa62 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -12,7 +12,8 @@ For a full list of reference architectures, see [Available reference architectures](index.md#available-reference-architectures). > - **Supported users (approximate):** 2,000 -> - **High Availability:** No +> - **High Availability:** No. For a highly-available environment, you can +> follow the [3K reference architecture](3k_users.md). > - **Test requests per second (RPS) rates:** API: 40 RPS, Web: 4 RPS, Git: 4 RPS | Service | Nodes | Configuration | GCP | AWS | Azure | @@ -271,7 +272,7 @@ further configuration steps. ```ruby # Disable all components except PostgreSQL roles ['postgres_role'] - repmgr['enable'] = false + patroni['enable'] = false consul['enable'] = false prometheus['enable'] = false alertmanager['enable'] = false @@ -954,7 +955,13 @@ along with [Gitaly](#configure-gitaly), are recommended over using NFS whenever possible. However, if you intend to use GitLab Pages, [you must use NFS](troubleshooting.md#gitlab-pages-requires-nfs). -For information about configuring NFS, see the [NFS documentation page](../nfs.md). +See how to [configure NFS](../nfs.md). + +WARNING: +From GitLab 13.0, using NFS for Git repositories is deprecated. +From GitLab 14.0, technical support for NFS for Git repositories +will no longer be provided. Upgrade to [Gitaly Cluster](../gitaly/praefect.md) +as soon as possible. <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index 370247ed856..648fac8025f 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -7,16 +7,17 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Reference architecture: up to 3,000 users **(PREMIUM ONLY)** -This page describes GitLab reference architecture for up to 3,000 users. For a -full list of reference architectures, see -[Available reference architectures](index.md#available-reference-architectures). - -NOTE: -This reference architecture is designed to help your organization achieve a +This page describes GitLab reference architecture for up to 3,000 users. +It is designed to help your organization achieve a highly-available GitLab deployment. If you do not have the expertise or need to maintain a highly-available environment, you can have a simpler and less costly-to-operate environment by using the [2,000-user reference architecture](2k_users.md). +If you have fewer than 3,000 users and still want a highly-available GitLab deployment, +you should still use this reference architecture but scale down the node sizes. + +For a full list of reference architectures, see +[Available reference architectures](index.md#available-reference-architectures). > - **Supported users (approximate):** 3,000 > - **High Availability:** Yes @@ -670,9 +671,9 @@ install the necessary dependencies from step 1, and add the GitLab package repository from step 2. When installing GitLab in the second step, do not supply the `EXTERNAL_URL` value. -#### PostgreSQL primary node +#### PostgreSQL nodes -1. SSH in to the PostgreSQL primary node. +1. SSH in to one of the PostgreSQL nodes. 1. Generate a password hash for the PostgreSQL username/password pair. This assumes you will use the default username of `gitlab` (recommended). The command will request a password and confirmation. Use the value that is output by this command in the next @@ -700,114 +701,33 @@ in the second step, do not supply the `EXTERNAL_URL` value. sudo gitlab-ctl pg-password-md5 gitlab-consul ``` -1. On the primary database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: +1. On every database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: ```ruby - # Disable all components except PostgreSQL and Repmgr and Consul + # Disable all components except PostgreSQL, Patroni, and Consul roles ['postgres_role'] # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' - postgresql['hot_standby'] = 'on' - postgresql['wal_level'] = 'replica' - postgresql['shared_preload_libraries'] = 'repmgr_funcs' - - # Disable automatic database migrations - gitlab_rails['auto_migrate'] = false - - # Configure the Consul agent - consul['services'] = %w(postgresql) - # START user configuration - # Please set the real values as explained in Required Information section - # - # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value - postgresql['pgbouncer_user_password'] = '<pgbouncer_password_hash>' - # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value - postgresql['sql_user_password'] = '<postgresql_password_hash>' + # Enable Patroni + patroni['enable'] = true # Set `max_wal_senders` to one more than the number of database nodes in the cluster. # This is used to prevent replication from using up all of the # available database connections. - postgresql['max_wal_senders'] = 4 - postgresql['max_replication_slots'] = 4 - - # Replace XXX.XXX.XXX.XXX/YY with Network Address - postgresql['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 10.6.0.0/24) - repmgr['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 10.6.0.0/24) - - ## Enable service discovery for Prometheus - consul['enable'] = true - consul['monitoring_service_discovery'] = true - - # Set the network addresses that the exporters will listen on for monitoring - node_exporter['listen_address'] = '0.0.0.0:9100' - postgres_exporter['listen_address'] = '0.0.0.0:9187' - postgres_exporter['dbname'] = 'gitlabhq_production' - postgres_exporter['password'] = '<postgresql_password_hash>' - - ## The IPs of the Consul server nodes - ## You can also use FQDNs and intermix them with IPs - consul['configuration'] = { - retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), - } - # - # END user configuration - ``` - -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. You can list the current PostgreSQL primary, secondary nodes status via: - - ```shell - sudo /opt/gitlab/bin/gitlab-ctl repmgr cluster show - ``` - -1. Verify the GitLab services are running: - - ```shell - sudo gitlab-ctl status - ``` - - The output should be similar to the following: - - ```plaintext - run: consul: (pid 30593) 77133s; run: log: (pid 29912) 77156s - run: logrotate: (pid 23449) 3341s; run: log: (pid 29794) 77175s - run: node-exporter: (pid 30613) 77133s; run: log: (pid 29824) 77170s - run: postgres-exporter: (pid 30620) 77132s; run: log: (pid 29894) 77163s - run: postgresql: (pid 30630) 77132s; run: log: (pid 29618) 77181s - run: repmgrd: (pid 30639) 77132s; run: log: (pid 29985) 77150s - ``` - -<div align="right"> - <a type="button" class="btn btn-default" href="#setup-components"> - Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> - </a> -</div> - -#### PostgreSQL secondary nodes - -1. On both the secondary nodes, add the same configuration specified above for the primary node - with an additional setting that will inform `gitlab-ctl` that they are standby nodes initially - and there's no need to attempt to register them as a primary node: - - ```ruby - # Disable all components except PostgreSQL and Repmgr and Consul - roles ['postgres_role'] - - # PostgreSQL configuration - postgresql['listen_address'] = '0.0.0.0' - postgresql['hot_standby'] = 'on' - postgresql['wal_level'] = 'replica' - postgresql['shared_preload_libraries'] = 'repmgr_funcs' + patroni['postgresql']['max_wal_senders'] = 4 + patroni['postgresql']['max_replication_slots'] = 4 + # Incoming recommended value for max connections is 500. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691. + patroni['postgresql']['max_connections'] = 500 # Disable automatic database migrations gitlab_rails['auto_migrate'] = false # Configure the Consul agent + consul['enable'] = true consul['services'] = %w(postgresql) - - # Specify if a node should attempt to be primary on initialization. - repmgr['master_on_initialization'] = false + ## Enable service discovery for Prometheus + consul['monitoring_service_discovery'] = true # START user configuration # Please set the real values as explained in Required Information section @@ -816,34 +736,31 @@ in the second step, do not supply the `EXTERNAL_URL` value. postgresql['pgbouncer_user_password'] = '<pgbouncer_password_hash>' # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value postgresql['sql_user_password'] = '<postgresql_password_hash>' - # Set `max_wal_senders` to one more than the number of database nodes in the cluster. - # This is used to prevent replication from using up all of the - # available database connections. - postgresql['max_wal_senders'] = 4 - postgresql['max_replication_slots'] = 4 # Replace XXX.XXX.XXX.XXX/YY with Network Address - postgresql['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 10.6.0.0/24) - repmgr['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 10.6.0.0/24) - - ## Enable service discovery for Prometheus - consul['enable'] = true - consul['monitoring_service_discovery'] = true + postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/24) # Set the network addresses that the exporters will listen on for monitoring node_exporter['listen_address'] = '0.0.0.0:9100' postgres_exporter['listen_address'] = '0.0.0.0:9187' - postgres_exporter['dbname'] = 'gitlabhq_production' - postgres_exporter['password'] = '<postgresql_password_hash>' ## The IPs of the Consul server nodes ## You can also use FQDNs and intermix them with IPs consul['configuration'] = { retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), } + # # END user configuration ``` +PostgreSQL, with Patroni managing its failover, will default to use `pg_rewind` by default to handle conflicts. +Like most failover handling methods, this has a small chance of leading to data loss. +Learn more about the various [Patroni replication methods](../postgresql/replication_and_failover.md#selecting-the-appropriate-patroni-replication-method). + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace + the file of the same name on this server. If that file is not on this server, + add the file from your Consul server to this server. + 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/database.html) @@ -874,84 +791,25 @@ SSH in to the **primary node**: 1. Exit the database prompt by typing `\q` and Enter. -1. Verify the cluster is initialized with one node: +1. Check the status of the leader and cluster: ```shell - gitlab-ctl repmgr cluster show + gitlab-ctl patroni members ``` The output should be similar to the following: ```plaintext - Role | Name | Upstream | Connection String - ----------+----------|----------|---------------------------------------- - * master | HOSTNAME | | host=HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr - ``` - -1. Note down the hostname or IP address in the connection string: `host=HOSTNAME`. We will - refer to the hostname in the next section as `<primary_node_name>`. If the value - is not an IP address, it will need to be a resolvable name (via DNS or - `/etc/hosts`) - -SSH in to the **secondary node**: - -1. Set up the repmgr standby: - - ```shell - gitlab-ctl repmgr standby setup <primary_node_name> - ``` - - Do note that this will remove the existing data on the node. The command - has a wait time. - - The output should be similar to the following: - - ```console - Doing this will delete the entire contents of /var/opt/gitlab/postgresql/data - If this is not what you want, hit Ctrl-C now to exit - To skip waiting, rerun with the -w option - Sleeping for 30 seconds - Stopping the database - Removing the data - Cloning the data - Starting the database - Registering the node with the cluster - ok: run: repmgrd: (pid 19068) 0s + | Cluster | Member | Host | Role | State | TL | Lag in MB | Pending restart | + |---------------|-----------------------------------|-----------|--------|---------|-----|-----------|-----------------| + | postgresql-ha | <PostgreSQL primary hostname> | 10.6.0.31 | Leader | running | 175 | | * | + | postgresql-ha | <PostgreSQL secondary 1 hostname> | 10.6.0.32 | | running | 175 | 0 | * | + | postgresql-ha | <PostgreSQL secondary 2 hostname> | 10.6.0.33 | | running | 175 | 0 | * | ``` -Before moving on, make sure the databases are configured correctly. Run the -following command on the **primary** node to verify that replication is working -properly and the secondary nodes appear in the cluster: - -```shell -gitlab-ctl repmgr cluster show -``` - -The output should be similar to the following: - -```plaintext -Role | Name | Upstream | Connection String -----------+---------|-----------|------------------------------------------------ -* master | MASTER | | host=<primary_node_name> user=gitlab_repmgr dbname=gitlab_repmgr - standby | STANDBY | MASTER | host=<secondary_node_name> user=gitlab_repmgr dbname=gitlab_repmgr - standby | STANDBY | MASTER | host=<secondary_node_name> user=gitlab_repmgr dbname=gitlab_repmgr -``` - -If the 'Role' column for any node says "FAILED", check the +If the 'State' column for any node doesn't say "running", check the [Troubleshooting section](troubleshooting.md) before proceeding. -Also, check that the `repmgr-check-master` command works successfully on each node: - -```shell -su - gitlab-consul -gitlab-ctl repmgr-check-master || echo 'This node is a standby repmgr node' -``` - -This command relies on exit codes to tell Consul whether a particular node is a master -or secondary. The most important thing here is that this command does not produce errors. -If there are errors it's most likely due to incorrect `gitlab-consul` database user permissions. -Check the [Troubleshooting section](troubleshooting.md) before proceeding. - <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> @@ -969,7 +827,7 @@ The following IPs will be used as an example: 1. On each PgBouncer node, edit `/etc/gitlab/gitlab.rb`, and replace `<consul_password_hash>` and `<pgbouncer_password_hash>` with the - password hashes you [set up previously](#postgresql-primary-node): + password hashes you [set up previously](#postgresql-nodes): ```ruby # Disable all components except Pgbouncer and Consul agent @@ -977,15 +835,16 @@ The following IPs will be used as an example: # Configure PgBouncer pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) - pgbouncer['users'] = { - 'gitlab-consul': { - password: '<consul_password_hash>' - }, - 'pgbouncer': { - password: '<pgbouncer_password_hash>' - } + 'gitlab-consul': { + password: '<consul_password_hash>' + }, + 'pgbouncer': { + password: '<pgbouncer_password_hash>' + } } + # Incoming recommended value for max db connections is 150. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691. + pgbouncer['max_db_connections'] = 150 # Configure Consul agent consul['watchers'] = %w(postgresql) @@ -1841,6 +1700,12 @@ to use GitLab Pages, this currently [requires NFS](troubleshooting.md#gitlab-pag See how to [configure NFS](../nfs.md). +WARNING: +From GitLab 13.0, using NFS for Git repositories is deprecated. +From GitLab 14.0, technical support for NFS for Git repositories +will no longer be provided. Upgrade to [Gitaly Cluster](../gitaly/praefect.md) +as soon as possible. + <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index f76c8a8a877..093869d331b 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -422,9 +422,9 @@ install the necessary dependencies from step 1, and add the GitLab package repository from step 2. When installing GitLab in the second step, do not supply the `EXTERNAL_URL` value. -#### PostgreSQL primary node +#### PostgreSQL nodes -1. SSH in to the PostgreSQL primary node. +1. SSH in to one of the PostgreSQL nodes. 1. Generate a password hash for the PostgreSQL username/password pair. This assumes you will use the default username of `gitlab` (recommended). The command will request a password and confirmation. Use the value that is output by this command in the next @@ -452,23 +452,33 @@ in the second step, do not supply the `EXTERNAL_URL` value. sudo gitlab-ctl pg-password-md5 gitlab-consul ``` -1. On the primary database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: +1. On every database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: ```ruby - # Disable all components except PostgreSQL and Repmgr and Consul + # Disable all components except PostgreSQL, Patroni, and Consul roles ['postgres_role'] # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' - postgresql['hot_standby'] = 'on' - postgresql['wal_level'] = 'replica' - postgresql['shared_preload_libraries'] = 'repmgr_funcs' + + # Enable Patroni + patroni['enable'] = true + # Set `max_wal_senders` to one more than the number of database nodes in the cluster. + # This is used to prevent replication from using up all of the + # available database connections. + patroni['postgresql']['max_wal_senders'] = 4 + patroni['postgresql']['max_replication_slots'] = 4 + # Incoming recommended value for max connections is 500. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691. + patroni['postgresql']['max_connections'] = 500 # Disable automatic database migrations gitlab_rails['auto_migrate'] = false # Configure the Consul agent + consul['enable'] = true consul['services'] = %w(postgresql) + ## Enable service discovery for Prometheus + consul['monitoring_service_discovery'] = true # START user configuration # Please set the real values as explained in Required Information section @@ -477,18 +487,9 @@ in the second step, do not supply the `EXTERNAL_URL` value. postgresql['pgbouncer_user_password'] = '<pgbouncer_password_hash>' # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value postgresql['sql_user_password'] = '<postgresql_password_hash>' - # Set `max_wal_senders` to one more than the number of database nodes in the cluster. - # This is used to prevent replication from using up all of the - # available database connections. - postgresql['max_wal_senders'] = 4 - postgresql['max_replication_slots'] = 4 # Replace XXX.XXX.XXX.XXX/YY with Network Address postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/24) - repmgr['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 10.6.0.0/24) - - ## Enable service discovery for Prometheus - consul['monitoring_service_discovery'] = true # Set the network addresses that the exporters will listen on for monitoring node_exporter['listen_address'] = '0.0.0.0:9100' @@ -503,70 +504,9 @@ in the second step, do not supply the `EXTERNAL_URL` value. # END user configuration ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. - -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -<div align="right"> - <a type="button" class="btn btn-default" href="#setup-components"> - Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> - </a> -</div> - -#### PostgreSQL secondary nodes - -1. On both the secondary nodes, add the same configuration specified above for the primary node - with an additional setting (`repmgr['master_on_initialization'] = false`) that will inform `gitlab-ctl` that they are standby nodes initially - and there's no need to attempt to register them as a primary node: - - ```ruby - # Disable all components except PostgreSQL and Repmgr and Consul - roles ['postgres_role'] - - # PostgreSQL configuration - postgresql['listen_address'] = '0.0.0.0' - postgresql['hot_standby'] = 'on' - postgresql['wal_level'] = 'replica' - postgresql['shared_preload_libraries'] = 'repmgr_funcs' - - # Disable automatic database migrations - gitlab_rails['auto_migrate'] = false - - # Configure the Consul agent - consul['services'] = %w(postgresql) - - # Specify if a node should attempt to be primary on initialization. - repmgr['master_on_initialization'] = false - - # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value - postgresql['pgbouncer_user_password'] = '<pgbouncer_password_hash>' - # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value - postgresql['sql_user_password'] = '<postgresql_password_hash>' - # Set `max_wal_senders` to one more than the number of database nodes in the cluster. - # This is used to prevent replication from using up all of the - # available database connections. - postgresql['max_wal_senders'] = 4 - postgresql['max_replication_slots'] = 4 - - # Replace with your network addresses - postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/24) - repmgr['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 10.6.0.0/24) - - ## Enable service discovery for Prometheus - consul['monitoring_service_discovery'] = true - - # Set the network addresses that the exporters will listen on for monitoring - node_exporter['listen_address'] = '0.0.0.0:9100' - postgres_exporter['listen_address'] = '0.0.0.0:9187' - - ## The IPs of the Consul server nodes - ## You can also use FQDNs and intermix them with IPs - consul['configuration'] = { - retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), - } - ``` +PostgreSQL, with Patroni managing its failover, will default to use `pg_rewind` by default to handle conflicts. +Like most failover handling methods, this has a small chance of leading to data loss. +Learn more about the various [Patroni replication methods](../postgresql/replication_and_failover.md#selecting-the-appropriate-patroni-replication-method). 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace the file of the same name on this server. If that file is not on this server, @@ -601,84 +541,25 @@ SSH in to the **primary node**: 1. Exit the database prompt by typing `\q` and Enter. -1. Verify the cluster is initialized with one node: +1. Check the status of the leader and cluster: ```shell - gitlab-ctl repmgr cluster show + gitlab-ctl patroni members ``` The output should be similar to the following: ```plaintext - Role | Name | Upstream | Connection String - ----------+----------|----------|---------------------------------------- - * master | HOSTNAME | | host=HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr - ``` - -1. Note down the hostname or IP address in the connection string: `host=HOSTNAME`. We will - refer to the hostname in the next section as `<primary_node_name>`. If the value - is not an IP address, it will need to be a resolvable name (via DNS or - `/etc/hosts`) - -SSH in to the **secondary node**: - -1. Set up the repmgr standby: - - ```shell - gitlab-ctl repmgr standby setup <primary_node_name> + | Cluster | Member | Host | Role | State | TL | Lag in MB | Pending restart | + |---------------|-----------------------------------|-----------|--------|---------|-----|-----------|-----------------| + | postgresql-ha | <PostgreSQL primary hostname> | 10.6.0.21 | Leader | running | 175 | | * | + | postgresql-ha | <PostgreSQL secondary 1 hostname> | 10.6.0.22 | | running | 175 | 0 | * | + | postgresql-ha | <PostgreSQL secondary 2 hostname> | 10.6.0.23 | | running | 175 | 0 | * | ``` - Do note that this will remove the existing data on the node. The command - has a wait time. - - The output should be similar to the following: - - ```console - Doing this will delete the entire contents of /var/opt/gitlab/postgresql/data - If this is not what you want, hit Ctrl-C now to exit - To skip waiting, rerun with the -w option - Sleeping for 30 seconds - Stopping the database - Removing the data - Cloning the data - Starting the database - Registering the node with the cluster - ok: run: repmgrd: (pid 19068) 0s - ``` - -Before moving on, make sure the databases are configured correctly. Run the -following command on the **primary** node to verify that replication is working -properly and the secondary nodes appear in the cluster: - -```shell -gitlab-ctl repmgr cluster show -``` - -The output should be similar to the following: - -```plaintext -Role | Name | Upstream | Connection String -----------+---------|-----------|------------------------------------------------ -* master | MASTER | | host=<primary_node_name> user=gitlab_repmgr dbname=gitlab_repmgr - standby | STANDBY | MASTER | host=<secondary_node_name> user=gitlab_repmgr dbname=gitlab_repmgr - standby | STANDBY | MASTER | host=<secondary_node_name> user=gitlab_repmgr dbname=gitlab_repmgr -``` - -If the 'Role' column for any node says "FAILED", check the +If the 'State' column for any node doesn't say "running", check the [Troubleshooting section](troubleshooting.md) before proceeding. -Also, check that the `repmgr-check-master` command works successfully on each node: - -```shell -su - gitlab-consul -gitlab-ctl repmgr-check-master || echo 'This node is a standby repmgr node' -``` - -This command relies on exit codes to tell Consul whether a particular node is a master -or secondary. The most important thing here is that this command does not produce errors. -If there are errors it's most likely due to incorrect `gitlab-consul` database user permissions. -Check the [Troubleshooting section](troubleshooting.md) before proceeding. - <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> @@ -696,7 +577,7 @@ The following IPs will be used as an example: 1. On each PgBouncer node, edit `/etc/gitlab/gitlab.rb`, and replace `<consul_password_hash>` and `<pgbouncer_password_hash>` with the - password hashes you [set up previously](#postgresql-primary-node): + password hashes you [set up previously](#postgresql-nodes): ```ruby # Disable all components except Pgbouncer and Consul agent @@ -704,15 +585,16 @@ The following IPs will be used as an example: # Configure PgBouncer pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) - pgbouncer['users'] = { - 'gitlab-consul': { - password: '<consul_password_hash>' - }, - 'pgbouncer': { - password: '<pgbouncer_password_hash>' - } + 'gitlab-consul': { + password: '<consul_password_hash>' + }, + 'pgbouncer': { + password: '<pgbouncer_password_hash>' + } } + # Incoming recommended value for max db connections is 150. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691. + pgbouncer['max_db_connections'] = 150 # Configure Consul agent consul['watchers'] = %w(postgresql) @@ -2125,6 +2007,12 @@ to use GitLab Pages, this currently [requires NFS](troubleshooting.md#gitlab-pag See how to [configure NFS](../nfs.md). +WARNING: +From GitLab 13.0, using NFS for Git repositories is deprecated. +From GitLab 14.0, technical support for NFS for Git repositories +will no longer be provided. Upgrade to [Gitaly Cluster](../gitaly/praefect.md) +as soon as possible. + <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index 6a0547aeaf9..16ad866a108 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -671,9 +671,9 @@ install the necessary dependencies from step 1, and add the GitLab package repository from step 2. When installing GitLab in the second step, do not supply the `EXTERNAL_URL` value. -#### PostgreSQL primary node +#### PostgreSQL nodes -1. SSH in to the PostgreSQL primary node. +1. SSH in to one of the PostgreSQL nodes. 1. Generate a password hash for the PostgreSQL username/password pair. This assumes you will use the default username of `gitlab` (recommended). The command will request a password and confirmation. Use the value that is output by this command in the next @@ -701,114 +701,33 @@ in the second step, do not supply the `EXTERNAL_URL` value. sudo gitlab-ctl pg-password-md5 gitlab-consul ``` -1. On the primary database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: +1. On every database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: ```ruby - # Disable all components except PostgreSQL and Repmgr and Consul + # Disable all components except PostgreSQL, Patroni, and Consul roles ['postgres_role'] # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' - postgresql['hot_standby'] = 'on' - postgresql['wal_level'] = 'replica' - postgresql['shared_preload_libraries'] = 'repmgr_funcs' - # Disable automatic database migrations - gitlab_rails['auto_migrate'] = false - - # Configure the Consul agent - consul['services'] = %w(postgresql) - - # START user configuration - # Please set the real values as explained in Required Information section - # - # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value - postgresql['pgbouncer_user_password'] = '<pgbouncer_password_hash>' - # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value - postgresql['sql_user_password'] = '<postgresql_password_hash>' + # Enable Patroni + patroni['enable'] = true # Set `max_wal_senders` to one more than the number of database nodes in the cluster. # This is used to prevent replication from using up all of the # available database connections. - postgresql['max_wal_senders'] = 4 - postgresql['max_replication_slots'] = 4 - - # Replace XXX.XXX.XXX.XXX/YY with Network Address - postgresql['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 10.6.0.0/24) - repmgr['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 10.6.0.0/24) - - ## Enable service discovery for Prometheus - consul['enable'] = true - consul['monitoring_service_discovery'] = true - - # Set the network addresses that the exporters will listen on for monitoring - node_exporter['listen_address'] = '0.0.0.0:9100' - postgres_exporter['listen_address'] = '0.0.0.0:9187' - postgres_exporter['dbname'] = 'gitlabhq_production' - postgres_exporter['password'] = '<postgresql_password_hash>' - - ## The IPs of the Consul server nodes - ## You can also use FQDNs and intermix them with IPs - consul['configuration'] = { - retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), - } - # - # END user configuration - ``` - -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. You can list the current PostgreSQL primary, secondary nodes status via: - - ```shell - sudo /opt/gitlab/bin/gitlab-ctl repmgr cluster show - ``` - -1. Verify the GitLab services are running: - - ```shell - sudo gitlab-ctl status - ``` - - The output should be similar to the following: - - ```plaintext - run: consul: (pid 30593) 77133s; run: log: (pid 29912) 77156s - run: logrotate: (pid 23449) 3341s; run: log: (pid 29794) 77175s - run: node-exporter: (pid 30613) 77133s; run: log: (pid 29824) 77170s - run: postgres-exporter: (pid 30620) 77132s; run: log: (pid 29894) 77163s - run: postgresql: (pid 30630) 77132s; run: log: (pid 29618) 77181s - run: repmgrd: (pid 30639) 77132s; run: log: (pid 29985) 77150s - ``` - -<div align="right"> - <a type="button" class="btn btn-default" href="#setup-components"> - Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> - </a> -</div> - -#### PostgreSQL secondary nodes - -1. On both the secondary nodes, add the same configuration specified above for the primary node - with an additional setting that will inform `gitlab-ctl` that they are standby nodes initially - and there's no need to attempt to register them as a primary node: - - ```ruby - # Disable all components except PostgreSQL and Repmgr and Consul - roles ['postgres_role'] - - # PostgreSQL configuration - postgresql['listen_address'] = '0.0.0.0' - postgresql['hot_standby'] = 'on' - postgresql['wal_level'] = 'replica' - postgresql['shared_preload_libraries'] = 'repmgr_funcs' + patroni['postgresql']['max_wal_senders'] = 4 + patroni['postgresql']['max_replication_slots'] = 4 + # Incoming recommended value for max connections is 500. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691. + patroni['postgresql']['max_connections'] = 500 # Disable automatic database migrations gitlab_rails['auto_migrate'] = false # Configure the Consul agent + consul['enable'] = true consul['services'] = %w(postgresql) - - # Specify if a node should attempt to be primary on initialization. - repmgr['master_on_initialization'] = false + ## Enable service discovery for Prometheus + consul['monitoring_service_discovery'] = true # START user configuration # Please set the real values as explained in Required Information section @@ -817,34 +736,31 @@ in the second step, do not supply the `EXTERNAL_URL` value. postgresql['pgbouncer_user_password'] = '<pgbouncer_password_hash>' # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value postgresql['sql_user_password'] = '<postgresql_password_hash>' - # Set `max_wal_senders` to one more than the number of database nodes in the cluster. - # This is used to prevent replication from using up all of the - # available database connections. - postgresql['max_wal_senders'] = 4 - postgresql['max_replication_slots'] = 4 # Replace XXX.XXX.XXX.XXX/YY with Network Address - postgresql['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 10.6.0.0/24) - repmgr['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 10.6.0.0/24) - - ## Enable service discovery for Prometheus - consul['enable'] = true - consul['monitoring_service_discovery'] = true + postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/24) # Set the network addresses that the exporters will listen on for monitoring node_exporter['listen_address'] = '0.0.0.0:9100' postgres_exporter['listen_address'] = '0.0.0.0:9187' - postgres_exporter['dbname'] = 'gitlabhq_production' - postgres_exporter['password'] = '<postgresql_password_hash>' ## The IPs of the Consul server nodes ## You can also use FQDNs and intermix them with IPs consul['configuration'] = { retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), } + # # END user configuration ``` +PostgreSQL, with Patroni managing its failover, will default to use `pg_rewind` by default to handle conflicts. +Like most failover handling methods, this has a small chance of leading to data loss. +Learn more about the various [Patroni replication methods](../postgresql/replication_and_failover.md#selecting-the-appropriate-patroni-replication-method). + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace + the file of the same name on this server. If that file is not on this server, + add the file from your Consul server to this server. + 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/database.html) @@ -874,84 +790,25 @@ SSH in to the **primary node**: 1. Exit the database prompt by typing `\q` and Enter. -1. Verify the cluster is initialized with one node: +1. Check the status of the leader and cluster: ```shell - gitlab-ctl repmgr cluster show + gitlab-ctl patroni members ``` The output should be similar to the following: ```plaintext - Role | Name | Upstream | Connection String - ----------+----------|----------|---------------------------------------- - * master | HOSTNAME | | host=HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr - ``` - -1. Note down the hostname or IP address in the connection string: `host=HOSTNAME`. We will - refer to the hostname in the next section as `<primary_node_name>`. If the value - is not an IP address, it will need to be a resolvable name (via DNS or - `/etc/hosts`) - -SSH in to the **secondary node**: - -1. Set up the repmgr standby: - - ```shell - gitlab-ctl repmgr standby setup <primary_node_name> + | Cluster | Member | Host | Role | State | TL | Lag in MB | Pending restart | + |---------------|-----------------------------------|-----------|--------|---------|-----|-----------|-----------------| + | postgresql-ha | <PostgreSQL primary hostname> | 10.6.0.31 | Leader | running | 175 | | * | + | postgresql-ha | <PostgreSQL secondary 1 hostname> | 10.6.0.32 | | running | 175 | 0 | * | + | postgresql-ha | <PostgreSQL secondary 2 hostname> | 10.6.0.33 | | running | 175 | 0 | * | ``` - Do note that this will remove the existing data on the node. The command - has a wait time. - - The output should be similar to the following: - - ```console - Doing this will delete the entire contents of /var/opt/gitlab/postgresql/data - If this is not what you want, hit Ctrl-C now to exit - To skip waiting, rerun with the -w option - Sleeping for 30 seconds - Stopping the database - Removing the data - Cloning the data - Starting the database - Registering the node with the cluster - ok: run: repmgrd: (pid 19068) 0s - ``` - -Before moving on, make sure the databases are configured correctly. Run the -following command on the **primary** node to verify that replication is working -properly and the secondary nodes appear in the cluster: - -```shell -gitlab-ctl repmgr cluster show -``` - -The output should be similar to the following: - -```plaintext -Role | Name | Upstream | Connection String -----------+---------|-----------|------------------------------------------------ -* master | MASTER | | host=<primary_node_name> user=gitlab_repmgr dbname=gitlab_repmgr - standby | STANDBY | MASTER | host=<secondary_node_name> user=gitlab_repmgr dbname=gitlab_repmgr - standby | STANDBY | MASTER | host=<secondary_node_name> user=gitlab_repmgr dbname=gitlab_repmgr -``` - -If the 'Role' column for any node says "FAILED", check the +If the 'State' column for any node doesn't say "running", check the [Troubleshooting section](troubleshooting.md) before proceeding. -Also, check that the `repmgr-check-master` command works successfully on each node: - -```shell -su - gitlab-consul -gitlab-ctl repmgr-check-master || echo 'This node is a standby repmgr node' -``` - -This command relies on exit codes to tell Consul whether a particular node is a master -or secondary. The most important thing here is that this command does not produce errors. -If there are errors it's most likely due to incorrect `gitlab-consul` database user permissions. -Check the [Troubleshooting section](troubleshooting.md) before proceeding. - <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> @@ -969,7 +826,7 @@ The following IPs will be used as an example: 1. On each PgBouncer node, edit `/etc/gitlab/gitlab.rb`, and replace `<consul_password_hash>` and `<pgbouncer_password_hash>` with the - password hashes you [set up previously](#postgresql-primary-node): + password hashes you [set up previously](#postgresql-nodes): ```ruby # Disable all components except Pgbouncer and Consul agent @@ -977,15 +834,16 @@ The following IPs will be used as an example: # Configure PgBouncer pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) - pgbouncer['users'] = { - 'gitlab-consul': { - password: '<consul_password_hash>' - }, - 'pgbouncer': { - password: '<pgbouncer_password_hash>' - } + 'gitlab-consul': { + password: '<consul_password_hash>' + }, + 'pgbouncer': { + password: '<pgbouncer_password_hash>' + } } + # Incoming recommended value for max db connections is 150. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691. + pgbouncer['max_db_connections'] = 150 # Configure Consul agent consul['watchers'] = %w(postgresql) @@ -1841,6 +1699,12 @@ to use GitLab Pages, this currently [requires NFS](troubleshooting.md#gitlab-pag See how to [configure NFS](../nfs.md). +WARNING: +From GitLab 13.0, using NFS for Git repositories is deprecated. +From GitLab 14.0, technical support for NFS for Git repositories +will no longer be provided. Upgrade to [Gitaly Cluster](../gitaly/praefect.md) +as soon as possible. + <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index 6b04dad9baf..b90b8d67b68 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -95,7 +95,6 @@ have more actual perceived uptime for your users. > - Level of complexity: **Low** > - Required domain knowledge: PostgreSQL, GitLab configurations, Git -> - Supported tiers: [GitLab Core, Starter, Premium, and Ultimate](https://about.gitlab.com/pricing/) This solution is appropriate for many teams that have the default GitLab installation. With automatic backups of the GitLab repositories, configuration, and the database, @@ -107,7 +106,6 @@ is the least complex to setup. This provides a point-in-time recovery of a prede > - Level of complexity: **Medium** > - Required domain knowledge: HAProxy, shared storage, distributed systems -> - Supported tiers: [GitLab Starter, Premium, and Ultimate](https://about.gitlab.com/pricing/) This requires separating out GitLab into multiple application nodes with an added [load balancer](../load_balancer.md). The load balancer will distribute traffic @@ -123,11 +121,13 @@ to the default installation: - Enable zero-downtime upgrades. - Increase availability. +For more details on how to configure a traffic load balancer with GitLab, you can refer +to any of the [available reference architectures](#available-reference-architectures) with more than 1,000 users. + ### Zero downtime updates **(STARTER ONLY)** > - Level of complexity: **Medium** > - Required domain knowledge: PostgreSQL, HAProxy, shared storage, distributed systems -> - Supported tiers: [GitLab Starter, Premium, and Ultimate](https://about.gitlab.com/pricing/) GitLab supports [zero-downtime updates](https://docs.gitlab.com/omnibus/update/#zero-downtime-updates). Single GitLab nodes can be updated with only a [few minutes of downtime](https://docs.gitlab.com/omnibus/update/README.html#single-node-deployment). @@ -138,7 +138,6 @@ As long as at least one of each component is online and capable of handling the > - Level of complexity: **High** > - Required domain knowledge: PgBouncer, Repmgr or Patroni, shared storage, distributed systems -> - Supported tiers: [GitLab Premium and Ultimate](https://about.gitlab.com/pricing/) By adding automatic failover for database systems, you can enable higher uptime with additional database nodes. This extends the default database with @@ -150,7 +149,6 @@ is recommended. > - Level of complexity: **Very High** > - Required domain knowledge: Storage replication -> - Supported tiers: [GitLab Premium and Ultimate](https://about.gitlab.com/pricing/) [GitLab Geo](../geo/index.md) allows you to replicate your GitLab instance to other geographical locations as a read-only fully operational instance diff --git a/doc/administration/reference_architectures/troubleshooting.md b/doc/administration/reference_architectures/troubleshooting.md index 406ff57c66d..835231ac584 100644 --- a/doc/administration/reference_architectures/troubleshooting.md +++ b/doc/administration/reference_architectures/troubleshooting.md @@ -23,12 +23,12 @@ with the Fog library that GitLab uses. Symptoms include: ### GitLab Pages requires NFS If you intend to use [GitLab Pages](../../user/project/pages/index.md), this currently requires -[NFS](../nfs.md). There is [work in progress](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/196) -to remove this dependency. In the future, GitLab Pages may use -[object storage](https://gitlab.com/gitlab-org/gitlab/-/issues/208135). +[NFS](../nfs.md). There is [work in progress](https://gitlab.com/groups/gitlab-org/-/epics/3901) +to remove this dependency. In the future, GitLab Pages will use +object storage. The dependency on disk storage also prevents Pages being deployed using the -[GitLab Helm chart](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/37). +[GitLab Helm chart](https://gitlab.com/groups/gitlab-org/-/epics/4283). ### Incremental logging is required for CI to use object storage @@ -206,13 +206,8 @@ To make sure your configuration is correct: ## Troubleshooting Gitaly -### Checking versions when using standalone Gitaly nodes - -When using standalone Gitaly nodes, you must make sure they are the same version -as GitLab to ensure full compatibility. Check **Admin Area > Gitaly Servers** on -your GitLab instance and confirm all Gitaly Servers are `Up to date`. - - +If you have any problems when using standalone Gitaly nodes, first +[check all the versions are up to date](../gitaly/index.md#check-versions-when-using-standalone-gitaly-servers). ### `gitaly-debug` @@ -514,39 +509,24 @@ See the suggested fix [in Geo documentation](../geo/replication/troubleshooting. See the suggested fix [in Geo documentation](../geo/replication/troubleshooting.md#message-log--invalid-ip-mask-md5-name-or-service-not-known). -## Troubleshooting PostgreSQL +## Troubleshooting PostgreSQL with Patroni -In case you are experiencing any issues connecting through PgBouncer, the first place to check is always the logs: +In case you are experiencing any issues connecting through PgBouncer, the first place to check is always the logs for PostgreSQL (which is run through Patroni): ```shell -sudo gitlab-ctl tail postgresql +sudo gitlab-ctl tail patroni ``` -### Consul and PostgreSQL changes not taking effect +### Consul and PostgreSQL with Patroni changes not taking effect Due to the potential impacts, `gitlab-ctl reconfigure` only reloads Consul and PostgreSQL, it will not restart the services. However, not all changes can be activated by reloading. -To restart either service, run `gitlab-ctl restart SERVICE` +To restart either service, run `gitlab-ctl restart consul` or `gitlab-ctl restart patroni` respectively. -For PostgreSQL, it is usually safe to restart the master node by default. Automatic failover defaults to a 1 minute timeout. Provided the database returns before then, nothing else needs to be done. To be safe, you can stop `repmgrd` on the standby nodes first with `gitlab-ctl stop repmgrd`, then start afterwards with `gitlab-ctl start repmgrd`. +For PostgreSQL with Patroni, to prevent the primary node from being failed over automatically, it's safest to stop all secondaries first, then restart the primary and finally restart the secondaries again. On the Consul server nodes, it is important to restart the Consul service in a controlled fashion. Read our [Consul documentation](../consul.md#restart-consul) for instructions on how to restart the service. -### `gitlab-ctl repmgr-check-master` command produces errors - -If this command displays errors about database permissions it is likely that something failed during -install, resulting in the `gitlab-consul` database user getting incorrect permissions. Follow these -steps to fix the problem: - -1. On the master database node, connect to the database prompt - `gitlab-psql -d template1` -1. Delete the `gitlab-consul` user - `DROP USER "gitlab-consul";` -1. Exit the database prompt - `\q` -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) and the user will be re-added with the proper permissions. -1. Change to the `gitlab-consul` user - `su - gitlab-consul` -1. Try the check command again - `gitlab-ctl repmgr-check-master`. - -Now there should not be errors. If errors still occur then there is another problem. - ### PgBouncer error `ERROR: pgbouncer cannot connect to server` You may get this error when running `gitlab-rake gitlab:db:configure` or you diff --git a/doc/administration/repository_storage_paths.md b/doc/administration/repository_storage_paths.md index 090f95eca12..c71d1a5714c 100644 --- a/doc/administration/repository_storage_paths.md +++ b/doc/administration/repository_storage_paths.md @@ -122,3 +122,9 @@ weights are used to determine the storage location the repository is created on. Beginning with GitLab 8.13.4, multiple paths can be chosen. New repositories are randomly placed on one of the selected paths. + +## Move a repository to a different repository path + +To move a repository to a different repository path, use the +[Project repository storage moves](../api/project_repository_storage_moves.md) API. Use +the same process as [migrating existing repositories to Gitaly Cluster](gitaly/praefect.md#migrate-existing-repositories-to-gitaly-cluster). diff --git a/doc/administration/terraform_state.md b/doc/administration/terraform_state.md index edd44380f30..be5647aa133 100644 --- a/doc/administration/terraform_state.md +++ b/doc/administration/terraform_state.md @@ -20,11 +20,8 @@ These locations can be configured using the options described below. ## Using local storage -NOTE: -This is the default configuration - -To change the location where Terraform state files are stored locally, follow the steps -below. +The default configuration uses local storage. To change the location where +Terraform state files are stored locally, follow the steps below. **In Omnibus installations:** diff --git a/doc/administration/troubleshooting/elasticsearch.md b/doc/administration/troubleshooting/elasticsearch.md index 755273eb06e..fb153adfeab 100644 --- a/doc/administration/troubleshooting/elasticsearch.md +++ b/doc/administration/troubleshooting/elasticsearch.md @@ -36,6 +36,7 @@ The type of problem will determine what steps to take. The possible troubleshoot - Indexing. - Integration. - Performance. +- Background Migrations. ### Search Results workflow @@ -147,6 +148,30 @@ graph TD; F7(Escalate to<br>GitLab support.) ``` +### Background Migrations workflow + +```mermaid +graph TD; + D --> |No| D1 + D --> |Yes| D2 + D2 --> |No| D3 + D2 --> |Yes| D4 + D4 --> |No| D5 + D4 --> |Yes| D6 + D6 --> |No| D8 + D6 --> |Yes| D7 + + D{Is there a halted migration?} + D1[Migrations run in the<br>background and will<br>stop when completed.] + D2{Does the elasticsearch.log<br>file contain errors?} + D3[This is likely a bug/issue<br>in GitLab and will require<br>deeper investigation. Escalate<br>to GitLab support.] + D4{Have the errors<br>been addressed?} + D5[Have an Elasticsearch admin<br>review and address<br>the errors.] + D6{Has the migration<br>been retried?} + D7[This is likely a bug/issue<br>in GitLab and will require<br>deeper investigation. Escalate<br>to GitLab support.] + D8[Retry the migration from<br>the Admin > Settings ><br>Advanced Search UI.] +``` + ## Troubleshooting walkthrough Most Elasticsearch troubleshooting can be broken down into 4 categories: @@ -155,6 +180,7 @@ Most Elasticsearch troubleshooting can be broken down into 4 categories: - [Troubleshooting indexing](#troubleshooting-indexing) - [Troubleshooting integration](#troubleshooting-integration) - [Troubleshooting performance](#troubleshooting-performance) +- [Troubleshooting background migrations](#troubleshooting-background-migrations) Generally speaking, if it does not fall into those four categories, it is either: @@ -330,6 +356,18 @@ dig further into these. Feel free to reach out to GitLab support, but this is likely to be something a skilled Elasticsearch admin has more experience with. +### Troubleshooting background migrations + +Troubleshooting background migration failures can be difficult and may require contacting +an Elasticsearch admin or GitLab Support. + +The best place to start while debugging issues with a background migration is the +[`elasticsearch.log` file](../logs.md#elasticsearchlog). Migrations will +print information while a migration is in progress and any errors encountered. +Apply fixes for any errors found in the log and retry the migration. + +If you still encounter issues after retrying the migration, reach out to GitLab support. + ## Common issues All common issues [should be documented](../../integration/elasticsearch.md#troubleshooting). If not, diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index 2482a4fe7ad..4a112733bfa 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -384,7 +384,7 @@ project = Project.find_by_full_path('PROJECT_PATH') Projects::ImportExport::ExportService.new(project, user).execute ``` -If the project you wish to export is available at `https://gitlab.example.com/baltig/pipeline-templates`, the value to use for `PROJECT_PATH` would be `baltig/pipeline-templates`. +If the project you wish to export is available at `https://gitlab.example.com/baltig/pipeline-templates`, the value to use for `PROJECT_PATH` would be `baltig/pipeline-templates`. If this all runs successfully, you will see output like the following before being returned to the Rails console prompt: @@ -392,7 +392,7 @@ If this all runs successfully, you will see output like the following before bei => nil ``` -The exported project will be located within a `.tar.gz` file in `/var/opt/gitlab/gitlab-rails/uploads/-/system/import_export_upload/export_file/`. +The exported project will be located within a `.tar.gz` file in `/var/opt/gitlab/gitlab-rails/uploads/-/system/import_export_upload/export_file/`. ## Repository @@ -598,7 +598,7 @@ group = Group.find_by_path_or_name('group-name') group.project_creation_level=0 ``` -### Modify group - disable 2FA requirement +### Modify group - disable 2FA requirement WARNING: When disabling the 2FA Requirement on a subgroup, the whole parent group (including all subgroups) is affected by this change. @@ -743,7 +743,7 @@ m.project.try(:ci_service) ```ruby project = Project.find_by_full_path 'group/project' content = project.repository.gitlab_ci_yml_for(project.repository.root_ref_sha) -Gitlab::Ci::YamlProcessor.validation_message(content, user: User.first) +Gitlab::Ci::Lint.new(project: project, current_user: User.first).validate(content) ``` ### Disable AutoDevOps on Existing Projects diff --git a/doc/administration/troubleshooting/postgresql.md b/doc/administration/troubleshooting/postgresql.md index 7052b68370c..4ccae10e5b3 100644 --- a/doc/administration/troubleshooting/postgresql.md +++ b/doc/administration/troubleshooting/postgresql.md @@ -157,7 +157,7 @@ See current settings with: ```shell sudo gitlab-rails runner "c = ApplicationRecord.connection ; puts c.execute('SHOW statement_timeout').to_a ; -puts c.execute('SHOW lock_timeout').to_a ; +puts c.execute('SHOW deadlock_timeout').to_a ; puts c.execute('SHOW idle_in_transaction_session_timeout').to_a ;" ``` @@ -165,9 +165,19 @@ It may take a little while to respond. ```ruby {"statement_timeout"=>"1min"} -{"lock_timeout"=>"0"} +{"deadlock_timeout"=>"0"} {"idle_in_transaction_session_timeout"=>"1min"} ``` +These settings can be updated in `/etc/gitlab/gitlab.rb` with: + +```ruby +postgresql['deadlock_timeout'] = '5s' +postgresql['statement_timeout'] = '15s' +postgresql['idle_in_transaction_session_timeout'] = '60s' +``` + +Once saved, [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + NOTE: These are Omnibus GitLab settings. If an external database, such as a customer's PostgreSQL installation or Amazon RDS is being used, these values don't get set, and would have to be set externally. diff --git a/doc/administration/troubleshooting/tracing_correlation_id.md b/doc/administration/troubleshooting/tracing_correlation_id.md index 2981b9e1368..ad2b8586b8b 100644 --- a/doc/administration/troubleshooting/tracing_correlation_id.md +++ b/doc/administration/troubleshooting/tracing_correlation_id.md @@ -29,7 +29,7 @@ documentation for some popular browsers. - [Network Monitor - Firefox Developer Tools](https://developer.mozilla.org/en-US/docs/Tools/Network_Monitor) - [Inspect Network Activity In Chrome DevTools](https://developers.google.com/web/tools/chrome-devtools/network/) - [Safari Web Development Tools](https://developer.apple.com/safari/tools/) -- [Microsoft Edge Network panel](https://docs.microsoft.com/en-us/microsoft-edge/devtools-guide/network#request-details) +- [Microsoft Edge Network panel](https://docs.microsoft.com/en-us/microsoft-edge/devtools-guide-chromium/network/) To locate a relevant request and view its correlation ID: diff --git a/doc/administration/uploads.md b/doc/administration/uploads.md index 94e7bbe2cff..49af8358e29 100644 --- a/doc/administration/uploads.md +++ b/doc/administration/uploads.md @@ -8,46 +8,17 @@ info: To determine the technical writer assigned to the Stage/Group associated w Uploads represent all user data that may be sent to GitLab as a single file. As an example, avatars and notes' attachments are uploads. Uploads are integral to GitLab functionality, and therefore cannot be disabled. -## Upload parameters - -> - [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/214785) in GitLab 13.5. -> - It's [deployed behind a feature flag](../user/feature_flags.md), enabled by default. -> - It's enabled on GitLab.com. -> - It's recommended for production use. -> - For GitLab self-managed instances, GitLab administrators can opt to disable it. **(CORE ONLY)** - -In 13.5 and later, upload parameters are passed [between Workhorse and GitLab Rails](../development/architecture.md#simplified-component-overview) differently than they -were before. - -This change is deployed behind a feature flag that is **enabled by default**. - -If you experience any issues with upload, -[GitLab administrators with access to the GitLab Rails console](feature_flags.md) -can opt to disable it. - -To enable it: - -```ruby -Feature.enable(:upload_middleware_jwt_params_handler) -``` - -To disable it: - -```ruby -Feature.disable(:upload_middleware_jwt_params_handler) -``` - ## Using local storage This is the default configuration. To change the location where the uploads are stored locally, use the steps in this section based on your installation method: -**In Omnibus GitLab installations:** - NOTE: -For historical reasons, uploads are stored into a base directory, which by -default is `uploads/-/system`. It's strongly discouraged to change this -configuration option for an existing GitLab installation. +For historical reasons, instance level uploads (for example the [favicon](../user/admin_area/appearance.md#favicon)) are stored into a base directory, +which by default is `uploads/-/system`. It is strongly discouraged to change the base +directory on an existing GitLab installation. + +**In Omnibus GitLab installations:** _The uploads are stored by default in `/var/opt/gitlab/gitlab-rails/uploads`._ @@ -55,16 +26,17 @@ _The uploads are stored by default in `/var/opt/gitlab/gitlab-rails/uploads`._ `/etc/gitlab/gitlab.rb` and add the following line: ```ruby - gitlab_rails['uploads_storage_path'] = "/mnt/storage/" - gitlab_rails['uploads_base_dir'] = "uploads" + gitlab_rails['uploads_directory'] = "/mnt/storage/uploads" ``` + This setting only applies if you haven't changed the `gitlab_rails['uploads_storage_path']` directory. + 1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. **In installations from source:** _The uploads are stored by default in -`/home/git/gitlab/public/uploads/-/system`._ +`/home/git/gitlab/public/uploads`._ 1. To change the storage path for example to `/mnt/storage/uploads`, edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: @@ -93,7 +65,7 @@ This configuration relies on valid AWS credentials to be configured already. We recommend using the [consolidated object storage settings](object_storage.md#consolidated-object-storage-configuration). The following instructions apply to the original configuration format. -## Object Storage Settings +### Object Storage Settings For source installations the following settings are nested under `uploads:` and then `object_store:`. On Omnibus GitLab installs they are prefixed by `uploads_object_store_`. @@ -106,14 +78,14 @@ For source installations the following settings are nested under `uploads:` and | `proxy_download` | Set to `true` to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` | | `connection` | Various connection options described below | | -### Connection settings +#### Connection settings See [the available connection settings for different providers](object_storage.md#connection-settings). **In Omnibus installations:** _The uploads are stored by default in -`/var/opt/gitlab/gitlab-rails/public/uploads/-/system`._ +`/var/opt/gitlab/gitlab-rails/uploads`._ 1. Edit `/etc/gitlab/gitlab.rb` and add the following lines by replacing with the values you want: @@ -145,7 +117,7 @@ _The uploads are stored by default in **In installations from source:** _The uploads are stored by default in -`/home/git/gitlab/public/uploads/-/system`._ +`/home/git/gitlab/public/uploads`._ 1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: @@ -165,12 +137,12 @@ _The uploads are stored by default in 1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. 1. Migrate any existing local uploads to the object storage using [`gitlab:uploads:migrate:all` Rake task](raketasks/uploads/migrate.md). -### OpenStack example +#### OpenStack example **In Omnibus installations:** _The uploads are stored by default in -`/var/opt/gitlab/gitlab-rails/public/uploads/-/system`._ +`/var/opt/gitlab/gitlab-rails/uploads`._ 1. Edit `/etc/gitlab/gitlab.rb` and add the following lines by replacing with the values you want: @@ -196,7 +168,7 @@ _The uploads are stored by default in **In installations from source:** _The uploads are stored by default in -`/home/git/gitlab/public/uploads/-/system`._ +`/home/git/gitlab/public/uploads`._ 1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines: |
