From 6e4e1050d9dba2b7b2523fdd1768823ab85feef4 Mon Sep 17 00:00:00 2001 From: GitLab Bot Date: Thu, 20 Aug 2020 18:42:06 +0000 Subject: Add latest changes from gitlab-org/gitlab@13-3-stable-ee --- doc/administration/audit_events.md | 37 +- doc/administration/audit_reports.md | 31 + doc/administration/auth/ldap/index.md | 4 +- doc/administration/auth/okta.md | 2 +- doc/administration/auth/smartcard.md | 4 + doc/administration/consul.md | 239 +++ doc/administration/environment_variables.md | 1 + doc/administration/feature_flags.md | 4 +- doc/administration/file_hooks.md | 7 + .../disaster_recovery/background_verification.md | 8 +- .../geo/disaster_recovery/planned_failover.md | 14 +- .../geo/replication/configuration.md | 11 +- doc/administration/geo/replication/database.md | 64 +- doc/administration/geo/replication/datatypes.md | 41 +- doc/administration/geo/replication/disable_geo.md | 2 +- .../geo/replication/docker_registry.md | 2 +- .../geo/replication/external_database.md | 49 - .../geo/replication/geo_validation_tests.md | 22 +- .../replication/img/adding_a_secondary_node.png | Bin 23555 -> 0 bytes .../img/adding_a_secondary_node_v13_3.png | Bin 0 -> 20195 bytes doc/administration/geo/replication/index.md | 3 +- .../geo/replication/multiple_servers.md | 41 +- .../geo/replication/object_storage.md | 2 +- .../geo/replication/remove_geo_node.md | 2 +- .../geo/replication/troubleshooting.md | 347 +--- doc/administration/geo/replication/tuning.md | 17 +- .../geo/replication/version_specific_updates.md | 162 +- doc/administration/git_annex.md | 8 +- doc/administration/git_protocol.md | 4 + .../gitaly/img/cluster_example_v13_3.png | Bin 0 -> 27703 bytes .../gitaly/img/shard_example_v13_3.png | Bin 0 -> 14869 bytes doc/administration/gitaly/index.md | 2 +- doc/administration/gitaly/praefect.md | 308 ++- doc/administration/gitaly/reference.md | 2 +- .../high_availability/alpha_database.md | 2 +- doc/administration/high_availability/consul.md | 197 +- doc/administration/high_availability/gitaly.md | 59 +- doc/administration/high_availability/gitlab.md | 214 +- .../high_availability/load_balancer.md | 134 +- .../high_availability/monitoring_node.md | 103 +- doc/administration/high_availability/nfs.md | 321 +-- .../high_availability/nfs_host_client_setup.md | 156 +- doc/administration/high_availability/pgbouncer.md | 165 +- doc/administration/high_availability/sidekiq.md | 189 +- .../img/repository_storages_admin_ui_v13_1.png | Bin 85160 -> 26234 bytes doc/administration/index.md | 8 +- doc/administration/instance_limits.md | 76 +- doc/administration/integration/plantuml.md | 7 + doc/administration/integration/terminal.md | 4 +- doc/administration/invalidate_markdown_cache.md | 7 + doc/administration/issue_closing_pattern.md | 11 +- doc/administration/job_artifacts.md | 45 +- doc/administration/lfs/index.md | 45 +- doc/administration/load_balancer.md | 126 ++ doc/administration/logs.md | 28 + doc/administration/merge_request_diffs.md | 7 + doc/administration/monitoring/github_imports.md | 3 +- .../img/self_monitoring_default_dashboard.png | Bin 51508 -> 0 bytes .../img/self_monitoring_overview_dashboard.png | Bin 0 -> 51508 bytes .../gitlab_self_monitoring_project/index.md | 8 +- .../performance/grafana_configuration.md | 4 +- .../monitoring/performance/performance_bar.md | 6 +- .../monitoring/performance/request_profiling.md | 6 +- .../monitoring/prometheus/gitlab_metrics.md | 7 +- doc/administration/monitoring/prometheus/index.md | 93 +- doc/administration/nfs.md | 368 ++++ doc/administration/object_storage.md | 84 +- .../operations/cleaning_up_redis_sessions.md | 3 +- .../operations/extra_sidekiq_processes.md | 2 +- .../operations/filesystem_benchmarking.md | 2 +- .../operations/moving_repositories.md | 47 +- doc/administration/operations/puma.md | 2 +- .../operations/sidekiq_memory_killer.md | 2 +- doc/administration/packages/container_registry.md | 75 +- doc/administration/packages/index.md | 2 +- doc/administration/pages/index.md | 77 +- doc/administration/pages/source.md | 14 +- doc/administration/postgresql/external.md | 2 +- doc/administration/postgresql/index.md | 2 +- doc/administration/postgresql/pgbouncer.md | 168 ++ .../postgresql/replication_and_failover.md | 51 +- doc/administration/raketasks/check.md | 19 + doc/administration/raketasks/maintenance.md | 51 +- .../raketasks/project_import_export.md | 2 +- doc/administration/raketasks/storage.md | 4 +- doc/administration/raketasks/uploads/migrate.md | 10 +- .../redis/replication_and_failover.md | 75 +- .../redis/replication_and_failover_external.md | 10 +- doc/administration/redis/standalone.md | 51 +- .../reference_architectures/10k_users.md | 2115 +++++++++++++++++++- .../reference_architectures/1k_users.md | 109 +- .../reference_architectures/25k_users.md | 2115 +++++++++++++++++++- .../reference_architectures/2k_users.md | 31 +- .../reference_architectures/3k_users.md | 84 +- .../reference_architectures/50k_users.md | 2115 +++++++++++++++++++- .../reference_architectures/5k_users.md | 81 +- .../img/reference-architectures.png | Bin 47459 -> 12585 bytes .../reference_architectures/index.md | 151 +- .../reference_architectures/troubleshooting.md | 10 +- doc/administration/reply_by_email.md | 8 + doc/administration/reply_by_email_postfix_setup.md | 2 +- doc/administration/repository_checks.md | 7 + doc/administration/repository_storage_paths.md | 4 +- doc/administration/server_hooks.md | 10 +- doc/administration/sidekiq.md | 188 ++ doc/administration/smime_signing_email.md | 2 +- doc/administration/snippets/index.md | 17 +- .../static_objects_external_storage.md | 11 +- doc/administration/terraform_state.md | 2 - doc/administration/troubleshooting/debug.md | 8 +- .../troubleshooting/gitlab_rails_cheat_sheet.md | 23 +- .../troubleshooting/group_saml_scim.md | 2 +- .../troubleshooting/img/network_monitor_xid.png | Bin 0 -> 131339 bytes doc/administration/troubleshooting/postgresql.md | 4 +- doc/administration/troubleshooting/sidekiq.md | 22 +- .../troubleshooting/tracing_correlation_id.md | 126 ++ doc/administration/wikis/index.md | 75 + 117 files changed, 9163 insertions(+), 2753 deletions(-) create mode 100644 doc/administration/audit_reports.md create mode 100644 doc/administration/consul.md delete mode 100644 doc/administration/geo/replication/img/adding_a_secondary_node.png create mode 100644 doc/administration/geo/replication/img/adding_a_secondary_node_v13_3.png create mode 100644 doc/administration/gitaly/img/cluster_example_v13_3.png create mode 100644 doc/administration/gitaly/img/shard_example_v13_3.png create mode 100644 doc/administration/load_balancer.md delete mode 100644 doc/administration/monitoring/gitlab_self_monitoring_project/img/self_monitoring_default_dashboard.png create mode 100644 doc/administration/monitoring/gitlab_self_monitoring_project/img/self_monitoring_overview_dashboard.png create mode 100644 doc/administration/nfs.md create mode 100644 doc/administration/postgresql/pgbouncer.md create mode 100644 doc/administration/sidekiq.md create mode 100644 doc/administration/troubleshooting/img/network_monitor_xid.png create mode 100644 doc/administration/troubleshooting/tracing_correlation_id.md create mode 100644 doc/administration/wikis/index.md (limited to 'doc/administration') diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md index c85fb2d2e47..e7eab5a291e 100644 --- a/doc/administration/audit_events.md +++ b/doc/administration/audit_events.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Analytics +group: Compliance info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- @@ -48,24 +48,25 @@ You need Owner [permissions](../user/permissions.md) to view the group Audit Eve To view a group's audit events, navigate to **Group > Settings > Audit Events**. From there, you can see the following actions: -- Group name or path changed -- Group repository size limit changed -- Group created or deleted -- Group changed visibility -- User was added to group and with which [permissions](../user/permissions.md) -- User sign-in via [Group SAML](../user/group/saml_sso/index.md) -- Permissions changes of a user assigned to a group -- Removed user from group -- Project repository imported into group +- Group name or path changed. +- Group repository size limit changed. +- Group created or deleted. +- Group changed visibility. +- User was added to group and with which [permissions](../user/permissions.md). +- User sign-in via [Group SAML](../user/group/saml_sso/index.md). +- Permissions changes of a user assigned to a group. +- Removed user from group. +- Project repository imported into group. - [Project shared with group](../user/project/members/share_project_with_groups.md) - and with which [permissions](../user/permissions.md) -- Removal of a previously shared group with a project -- LFS enabled or disabled -- Shared runners minutes limit changed -- Membership lock enabled or disabled -- Request access enabled or disabled -- 2FA enforcement or grace period changed -- Roles allowed to create project changed + and with which [permissions](../user/permissions.md). +- Removal of a previously shared group with a project. +- LFS enabled or disabled. +- Shared runners minutes limit changed. +- Membership lock enabled or disabled. +- Request access enabled or disabled. +- 2FA enforcement or grace period changed. +- Roles allowed to create project changed. +- Group CI/CD variable added, removed, or protected status changed. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30857) in GitLab 13.3. Group events can also be accessed via the [Group Audit Events API](../api/audit_events.md#group-audit-events-starter) diff --git a/doc/administration/audit_reports.md b/doc/administration/audit_reports.md new file mode 100644 index 00000000000..d5a08b711be --- /dev/null +++ b/doc/administration/audit_reports.md @@ -0,0 +1,31 @@ +--- +stage: Manage +group: Compliance +description: 'Learn how to create evidence artifacts typically requested by a 3rd party auditor.' +--- + +# Audit Reports + +GitLab can help owners and administrators respond to auditors by generating +comprehensive reports. These **Audit Reports** vary in scope, depending on the +need: + +## Use cases + +- Generate a report of audit events to provide to an external auditor requesting proof of certain logging capabilities. +- Provide a report of all users showing their group and project memberships for a quarterly access review so the auditor can verify compliance with an organization's access management policy. + +## APIs + +- `https://docs.gitlab.com/ee/api/audit_events.html` +- `https://docs.gitlab.com/ee/api/graphql/reference/#user` +- `https://docs.gitlab.com/ee/api/graphql/reference/#groupmember` +- `https://docs.gitlab.com/ee/api/graphql/reference/#projectmember` + +## Features + +- `https://docs.gitlab.com/ee/administration/audit_events.html` +- `https://docs.gitlab.com/ee/administration/logs.html` + +We plan on making Audit Events [downloadable as a CSV](https://gitlab.com/gitlab-org/gitlab/-/issues/1449) +in the near future. diff --git a/doc/administration/auth/ldap/index.md b/doc/administration/auth/ldap/index.md index aef6c70ff92..548e734c931 100644 --- a/doc/administration/auth/ldap/index.md +++ b/doc/administration/auth/ldap/index.md @@ -548,7 +548,7 @@ or more LDAP group links](#adding-group-links-starter-only). ### Adding group links **(STARTER ONLY)** -For information on adding group links via CNs and filters, refer to [the GitLab groups documentation](../../../user/group/index.md#manage-group-memberships-via-ldap). +For information on adding group links via CNs and filters, refer to [the GitLab groups documentation](../../../user/group/index.md#manage-group-memberships-via-ldap-starter-only). ### Administrator sync **(STARTER ONLY)** @@ -596,6 +596,8 @@ group, as opposed to the full DN. ### Global group memberships lock **(STARTER ONLY)** +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/1793) in GitLab 12.0. + "Lock memberships to LDAP synchronization" setting allows instance administrators to lock down user abilities to invite new members to a group. diff --git a/doc/administration/auth/okta.md b/doc/administration/auth/okta.md index 78b10aedb77..7a55e127733 100644 --- a/doc/administration/auth/okta.md +++ b/doc/administration/auth/okta.md @@ -156,7 +156,7 @@ You might want to try this out on an incognito browser window. ## Configuring groups ->**Note:** +NOTE: **Note:** Make sure the groups exist and are assigned to the Okta app. You can take a look of the [SAML documentation](../../integration/saml.md#saml-groups) on configuring groups. diff --git a/doc/administration/auth/smartcard.md b/doc/administration/auth/smartcard.md index 80d2efbad84..0ecf3ca090d 100644 --- a/doc/administration/auth/smartcard.md +++ b/doc/administration/auth/smartcard.md @@ -310,6 +310,10 @@ attribute. As a prerequisite, you must use an LDAP server that: 1. Save the file and [restart](../restart_gitlab.md#installations-from-source) GitLab for the changes to take effect. +## Passwords for users created via smartcard authentication + +The [Generated passwords for users created through integrated authentication](../../security/passwords_for_integrated_authentication_methods.md) guide provides an overview of how GitLab generates and sets passwords for users created via smartcard authentication. + +This document was moved to [another location](../gitaly/index.md). diff --git a/doc/administration/high_availability/gitlab.md b/doc/administration/high_availability/gitlab.md index dc8c997bab5..748373c8941 100644 --- a/doc/administration/high_availability/gitlab.md +++ b/doc/administration/high_availability/gitlab.md @@ -1,215 +1,5 @@ --- -type: reference +redirect_to: ../reference_architectures/index.md --- -# Configuring GitLab application (Rails) - -This section describes how to configure the GitLab application (Rails) component. - -NOTE: **Note:** -There is some additional configuration near the bottom for -additional GitLab application servers. It's important to read and understand -these additional steps before proceeding with GitLab installation. - -NOTE: **Note:** -[Cloud Object Storage service](object_storage.md) with [Gitaly](gitaly.md) -is recommended over [NFS](nfs.md) wherever possible for improved performance. - -1. If necessary, install the NFS client utility packages using the following - commands: - - ```shell - # Ubuntu/Debian - apt-get install nfs-common - - # CentOS/Red Hat - yum install nfs-utils nfs-utils-lib - ``` - -1. Specify the necessary NFS exports in `/etc/fstab`. - The exact contents of `/etc/fstab` will depend on how you chose - to configure your NFS server. See [NFS documentation](nfs.md#nfs-client-mount-options) - for examples and the various options. - -1. Create the shared directories. These may be different depending on your NFS - mount locations. - - ```shell - mkdir -p /var/opt/gitlab/.ssh /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/git-data - ``` - -1. Download/install Omnibus GitLab using **steps 1 and 2** from - [GitLab downloads](https://about.gitlab.com/install/). Do not complete other - steps on the download page. -1. Create/edit `/etc/gitlab/gitlab.rb` and use the following configuration. - Be sure to change the `external_url` to match your eventual GitLab front-end - URL. Depending your the NFS configuration, you may need to change some GitLab - data locations. See [NFS documentation](nfs.md) for `/etc/gitlab/gitlab.rb` - configuration values for various scenarios. The example below assumes you've - added NFS mounts in the default data locations. Additionally the UID and GIDs - given are just examples and you should configure with your preferred values. - - ```ruby - external_url 'https://gitlab.example.com' - - # Prevent GitLab from starting if NFS data mounts are not available - high_availability['mountpoint'] = '/var/opt/gitlab/git-data' - - # Disable components that will not be on the GitLab application server - roles ['application_role'] - nginx['enable'] = true - - # PostgreSQL connection details - gitlab_rails['db_adapter'] = 'postgresql' - gitlab_rails['db_encoding'] = 'unicode' - gitlab_rails['db_host'] = '10.1.0.5' # IP/hostname of database server - gitlab_rails['db_password'] = 'DB password' - - # Redis connection details - gitlab_rails['redis_port'] = '6379' - gitlab_rails['redis_host'] = '10.1.0.6' # IP/hostname of Redis server - gitlab_rails['redis_password'] = 'Redis Password' - - # Ensure UIDs and GIDs match between servers for permissions via NFS - user['uid'] = 9000 - user['gid'] = 9000 - web_server['uid'] = 9001 - web_server['gid'] = 9001 - registry['uid'] = 9002 - registry['gid'] = 9002 - ``` - -1. [Enable monitoring](#enable-monitoring) - - NOTE: **Note:** - To maintain uniformity of links across HA clusters, the `external_url` - on the first application server as well as the additional application - servers should point to the external URL that users will use to access GitLab. - In a typical HA setup, this will be the URL of the load balancer which will - route traffic to all GitLab application servers in the HA cluster. - - NOTE: **Note:** - When you specify `https` in the `external_url`, as in the example - above, GitLab assumes you have SSL certificates in `/etc/gitlab/ssl/`. If - certificates are not present, NGINX will fail to start. See - [NGINX documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) - for more information. - - NOTE: **Note:** - It is best to set the `uid` and `gid`s prior to the initial reconfigure - of GitLab. Omnibus will not recursively `chown` directories if set after the initial reconfigure. - -## First GitLab application server - -On the first application server, run: - -```shell -sudo gitlab-ctl reconfigure -``` - -This should compile the configuration and initialize the database. Do -not run this on additional application servers until the next step. - -## Extra configuration for additional GitLab application servers - -Additional GitLab servers (servers configured **after** the first GitLab server) -need some extra configuration. - -1. Configure shared secrets. These values can be obtained from the primary - GitLab server in `/etc/gitlab/gitlab-secrets.json`. Copy this file to the - secondary servers **prior to** running the first `reconfigure` in the steps - above. - - ```ruby - gitlab_shell['secret_token'] = 'fbfb19c355066a9afb030992231c4a363357f77345edd0f2e772359e5be59b02538e1fa6cae8f93f7d23355341cea2b93600dab6d6c3edcdced558fc6d739860' - gitlab_rails['otp_key_base'] = 'b719fe119132c7810908bba18315259ed12888d4f5ee5430c42a776d840a396799b0a5ef0a801348c8a357f07aa72bbd58e25a84b8f247a25c72f539c7a6c5fa' - gitlab_rails['secret_key_base'] = '6e657410d57c71b4fc3ed0d694e7842b1895a8b401d812c17fe61caf95b48a6d703cb53c112bc01ebd197a85da81b18e29682040e99b4f26594772a4a2c98c6d' - gitlab_rails['db_key_base'] = 'bf2e47b68d6cafaef1d767e628b619365becf27571e10f196f98dc85e7771042b9203199d39aff91fcb6837c8ed83f2a912b278da50999bb11a2fbc0fba52964' - ``` - -1. Run `touch /etc/gitlab/skip-auto-reconfigure` to prevent database migrations - from running on upgrade. Only the primary GitLab application server should - handle migrations. - -1. **Recommended** Configure host keys. Copy the contents (private and public keys) of `/etc/ssh/` on - the primary application server to `/etc/ssh` on all secondary servers. This - prevents false man-in-the-middle-attack alerts when accessing servers in your - High Availability cluster behind a load balancer. - -1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. - -NOTE: **Note:** -You will need to restart the GitLab applications nodes after an update has occurred and database -migrations performed. - -## Enable Monitoring - -> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3786) in GitLab 12.0. - -If you enable Monitoring, it must be enabled on **all** GitLab servers. - -1. Make sure to collect [`CONSUL_SERVER_NODES`](../postgresql/replication_and_failover.md#consul-information), which are the IP addresses or DNS records of the Consul server nodes, for the next step. Note they are presented as `Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z` - -1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration: - - ```ruby - # Enable service discovery for Prometheus - consul['enable'] = true - consul['monitoring_service_discovery'] = true - - # Replace placeholders - # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z - # with the addresses of the Consul server nodes - consul['configuration'] = { - retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z), - } - - # Set the network addresses that the exporters will listen on - node_exporter['listen_address'] = '0.0.0.0:9100' - gitlab_workhorse['prometheus_listen_addr'] = '0.0.0.0:9229' - sidekiq['listen_address'] = "0.0.0.0" - puma['listen'] = '0.0.0.0' - - # Add the monitoring node's IP address to the monitoring whitelist and allow it to - # scrape the NGINX metrics. Replace placeholder `monitoring.gitlab.example.com` with - # the address and/or subnets gathered from the monitoring node(s). - gitlab_rails['monitoring_whitelist'] = ['monitoring.gitlab.example.com', '127.0.0.0/8'] - nginx['status']['options']['allow'] = ['monitoring.gitlab.example.com', '127.0.0.0/8'] - ``` - -1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. - - CAUTION: **Warning:** - If running Unicorn, after changing `unicorn['listen']` in `gitlab.rb`, and - running `sudo gitlab-ctl reconfigure`, it can take an extended period of time - for Unicorn to complete reloading after receiving a `HUP`. For more - information, see the - [issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4401). - -## Troubleshooting - -- `mount: wrong fs type, bad option, bad superblock on` - -You have not installed the necessary NFS client utilities. See step 1 above. - -- `mount: mount point /var/opt/gitlab/... does not exist` - -This particular directory does not exist on the NFS server. Ensure -the share is exported and exists on the NFS server and try to remount. - ---- - -## Upgrading GitLab HA - -GitLab HA installations can be upgraded with no downtime, but the -upgrade process must be carefully coordinated to avoid failures. See the -[Omnibus GitLab multi-node upgrade -document](https://docs.gitlab.com/omnibus/update/#multi-node--ha-deployment) -for more details. - -Read more on high-availability configuration: - -1. [Configure the database](../postgresql/replication_and_failover.md) -1. [Configure Redis](redis.md) -1. [Configure NFS](nfs.md) -1. [Configure the load balancers](load_balancer.md) +This document was moved to [another location](../reference_architectures/index.md). diff --git a/doc/administration/high_availability/load_balancer.md b/doc/administration/high_availability/load_balancer.md index 75703327140..5cedc4e11ae 100644 --- a/doc/administration/high_availability/load_balancer.md +++ b/doc/administration/high_availability/load_balancer.md @@ -1,135 +1,5 @@ --- -type: reference +redirect_to: ../load_balancer.md --- -# Load Balancer for multi-node GitLab - -In an multi-node GitLab configuration, you will need a load balancer to route -traffic to the application servers. The specifics on which load balancer to use -or the exact configuration is beyond the scope of GitLab documentation. We hope -that if you're managing HA systems like GitLab you have a load balancer of -choice already. Some examples including HAProxy (open-source), F5 Big-IP LTM, -and Citrix Net Scaler. This documentation will outline what ports and protocols -you need to use with GitLab. - -## SSL - -How will you handle SSL in your multi-node environment? There are several different -options: - -- Each application node terminates SSL -- The load balancer(s) terminate SSL and communication is not secure between - the load balancer(s) and the application nodes -- The load balancer(s) terminate SSL and communication is *secure* between the - load balancer(s) and the application nodes - -### Application nodes terminate SSL - -Configure your load balancer(s) to pass connections on port 443 as 'TCP' rather -than 'HTTP(S)' protocol. This will pass the connection to the application nodes -NGINX service untouched. NGINX will have the SSL certificate and listen on port 443. - -See [NGINX HTTPS documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) -for details on managing SSL certificates and configuring NGINX. - -### Load Balancer(s) terminate SSL without backend SSL - -Configure your load balancer(s) to use the 'HTTP(S)' protocol rather than 'TCP'. -The load balancer(s) will then be responsible for managing SSL certificates and -terminating SSL. - -Since communication between the load balancer(s) and GitLab will not be secure, -there is some additional configuration needed. See -[NGINX Proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#supporting-proxied-ssl) -for details. - -### Load Balancer(s) terminate SSL with backend SSL - -Configure your load balancer(s) to use the 'HTTP(S)' protocol rather than 'TCP'. -The load balancer(s) will be responsible for managing SSL certificates that -end users will see. - -Traffic will also be secure between the load balancer(s) and NGINX in this -scenario. There is no need to add configuration for proxied SSL since the -connection will be secure all the way. However, configuration will need to be -added to GitLab to configure SSL certificates. See -[NGINX HTTPS documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) -for details on managing SSL certificates and configuring NGINX. - -## Ports - -### Basic ports - -| LB Port | Backend Port | Protocol | -| ------- | ------------ | ------------------------ | -| 80 | 80 | HTTP (*1*) | -| 443 | 443 | TCP or HTTPS (*1*) (*2*) | -| 22 | 22 | TCP | - -- (*1*): [Web terminal](../../ci/environments/index.md#web-terminals) support requires - your load balancer to correctly handle WebSocket connections. When using - HTTP or HTTPS proxying, this means your load balancer must be configured - to pass through the `Connection` and `Upgrade` hop-by-hop headers. See the - [web terminal](../integration/terminal.md) integration guide for - more details. -- (*2*): When using HTTPS protocol for port 443, you will need to add an SSL - certificate to the load balancers. If you wish to terminate SSL at the - GitLab application server instead, use TCP protocol. - -### GitLab Pages Ports - -If you're using GitLab Pages with custom domain support you will need some -additional port configurations. -GitLab Pages requires a separate virtual IP address. Configure DNS to point the -`pages_external_url` from `/etc/gitlab/gitlab.rb` at the new virtual IP address. See the -[GitLab Pages documentation](../pages/index.md) for more information. - -| LB Port | Backend Port | Protocol | -| ------- | ------------- | --------- | -| 80 | Varies (*1*) | HTTP | -| 443 | Varies (*1*) | TCP (*2*) | - -- (*1*): The backend port for GitLab Pages depends on the - `gitlab_pages['external_http']` and `gitlab_pages['external_https']` - setting. See [GitLab Pages documentation](../pages/index.md) for more details. -- (*2*): Port 443 for GitLab Pages should always use the TCP protocol. Users can - configure custom domains with custom SSL, which would not be possible - if SSL was terminated at the load balancer. - -### Alternate SSH Port - -Some organizations have policies against opening SSH port 22. In this case, -it may be helpful to configure an alternate SSH hostname that allows users -to use SSH on port 443. An alternate SSH hostname will require a new virtual IP address -compared to the other GitLab HTTP configuration above. - -Configure DNS for an alternate SSH hostname such as `altssh.gitlab.example.com`. - -| LB Port | Backend Port | Protocol | -| ------- | ------------ | -------- | -| 443 | 22 | TCP | - -## Readiness check - -It is strongly recommend that multi-node deployments configure load balancers to utilize the [readiness check](../../user/admin_area/monitoring/health_check.md#readiness) to ensure a node is ready to accept traffic, before routing traffic to it. This is especially important when utilizing Puma, as there is a brief period during a restart where Puma will not accept requests. - ---- - -Read more on high-availability configuration: - -1. [Configure the database](../postgresql/replication_and_failover.md) -1. [Configure Redis](redis.md) -1. [Configure NFS](nfs.md) -1. [Configure the GitLab application servers](gitlab.md) - - +This document was moved to [another location](../load_balancer.md). diff --git a/doc/administration/high_availability/monitoring_node.md b/doc/administration/high_availability/monitoring_node.md index 6b6f0ae9ea3..76bcf6d0d40 100644 --- a/doc/administration/high_availability/monitoring_node.md +++ b/doc/administration/high_availability/monitoring_node.md @@ -1,104 +1,5 @@ --- -type: reference +redirect_to: ../monitoring/prometheus/index.md --- -# Configuring a Monitoring node for Scaling and High Availability - -> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3786) in GitLab 12.0. - -You can configure a Prometheus node to monitor GitLab. - -## Standalone Monitoring node using Omnibus GitLab - -The Omnibus GitLab package can be used to configure a standalone Monitoring node running [Prometheus](../monitoring/prometheus/index.md) and [Grafana](../monitoring/performance/grafana_configuration.md). -The monitoring node is not highly available. See [Scaling and High Availability](../reference_architectures/index.md) -for an overview of GitLab scaling and high availability options. - -The steps below are the minimum necessary to configure a Monitoring node running Prometheus and Grafana with -Omnibus: - -1. SSH into the Monitoring node. -1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab - package you want using **steps 1 and 2** from the GitLab downloads page. - - Do not complete any other steps on the download page. - -1. Make sure to collect [`CONSUL_SERVER_NODES`](../postgresql/replication_and_failover.md#consul-information), which are the IP addresses or DNS records of the Consul server nodes, for the next step. Note they are presented as `Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z` - -1. Edit `/etc/gitlab/gitlab.rb` and add the contents: - - ```ruby - external_url 'http://gitlab.example.com' - - # Enable Prometheus - prometheus['enable'] = true - prometheus['listen_address'] = '0.0.0.0:9090' - prometheus['monitor_kubernetes'] = false - - # Enable Login form - grafana['disable_login_form'] = false - - # Enable Grafana - grafana['enable'] = true - grafana['admin_password'] = 'toomanysecrets' - - # Enable service discovery for Prometheus - consul['enable'] = true - consul['monitoring_service_discovery'] = true - - # Replace placeholders - # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z - # with the addresses of the Consul server nodes - consul['configuration'] = { - retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z), - } - - # Disable all other services - gitlab_rails['auto_migrate'] = false - alertmanager['enable'] = false - gitaly['enable'] = false - gitlab_exporter['enable'] = false - gitlab_workhorse['enable'] = false - nginx['enable'] = true - postgres_exporter['enable'] = false - postgresql['enable'] = false - redis['enable'] = false - redis_exporter['enable'] = false - sidekiq['enable'] = false - puma['enable'] = false - node_exporter['enable'] = false - gitlab_exporter['enable'] = false - ``` - -1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. - -The next step is to tell all the other nodes where the monitoring node is: - -1. Edit `/etc/gitlab/gitlab.rb`, and add, or find and uncomment the following line: - - ```ruby - gitlab_rails['prometheus_address'] = '10.0.0.1:9090' - ``` - - Where `10.0.0.1:9090` is the IP address and port of the Prometheus node. - -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to - take effect. - -## Migrating to Service Discovery - -Once monitoring using Service Discovery is enabled with `consul['monitoring_service_discovery'] = true`, -ensure that `prometheus['scrape_configs']` is not set in `/etc/gitlab/gitlab.rb`. Setting both -`consul['monitoring_service_discovery'] = true` and `prometheus['scrape_configs']` in `/etc/gitlab/gitlab.rb` -will result in errors. - - +This document was moved to [another location](../monitoring/prometheus/index.md). diff --git a/doc/administration/high_availability/nfs.md b/doc/administration/high_availability/nfs.md index 6e8dc2c6c57..e3342fa0813 100644 --- a/doc/administration/high_availability/nfs.md +++ b/doc/administration/high_availability/nfs.md @@ -1,322 +1,5 @@ --- -type: reference +redirect_to: ../nfs.md --- -# NFS - -You can view information and options set for each of the mounted NFS file -systems by running `nfsstat -m` and `cat /etc/fstab`. - -CAUTION: **Caution:** -From GitLab 13.0, using NFS for Git repositories is deprecated. In GitLab 14.0, -support for NFS for Git repositories is scheduled to be removed. Upgrade to -[Gitaly Cluster](../gitaly/praefect.md) as soon as possible. - -NOTE: **Note:** -Filesystem performance has a big impact on overall GitLab -performance, especially for actions that read or write to Git repositories. See -[Filesystem Performance Benchmarking](../operations/filesystem_benchmarking.md) -for steps to test filesystem performance. - -## Known kernel version incompatibilities - -RedHat Enterprise Linux (RHEL) and CentOS v7.7 and v7.8 ship with kernel -version `3.10.0-1127`, which [contains a -bug](https://bugzilla.redhat.com/show_bug.cgi?id=1783554) that causes -[uploads to fail to copy over NFS](https://gitlab.com/gitlab-org/gitlab/-/issues/218999). The -following GitLab versions include a fix to work properly with that -kernel version: - -1. [12.10.12](https://about.gitlab.com/releases/2020/06/25/gitlab-12-10-12-released/) -1. [13.0.7](https://about.gitlab.com/releases/2020/06/25/gitlab-13-0-7-released/) -1. [13.1.1](https://about.gitlab.com/releases/2020/06/24/gitlab-13-1-1-released/) -1. 13.2 and up - -If you are using that kernel version, be sure to upgrade GitLab to avoid -errors. - -## NFS Server features - -### Required features - -**File locking**: GitLab **requires** advisory file locking, which is only -supported natively in NFS version 4. NFSv3 also supports locking as long as -Linux Kernel 2.6.5+ is used. We recommend using version 4 and do not -specifically test NFSv3. - -### Recommended options - -When you define your NFS exports, we recommend you also add the following -options: - -- `no_root_squash` - NFS normally changes the `root` user to `nobody`. This is - a good security measure when NFS shares will be accessed by many different - users. However, in this case only GitLab will use the NFS share so it - is safe. GitLab recommends the `no_root_squash` setting because we need to - manage file permissions automatically. Without the setting you may receive - errors when the Omnibus package tries to alter permissions. Note that GitLab - and other bundled components do **not** run as `root` but as non-privileged - users. The recommendation for `no_root_squash` is to allow the Omnibus package - to set ownership and permissions on files, as needed. In some cases where the - `no_root_squash` option is not available, the `root` flag can achieve the same - result. -- `sync` - Force synchronous behavior. Default is asynchronous and under certain - circumstances it could lead to data loss if a failure occurs before data has - synced. - -Due to the complexities of running Omnibus with LDAP and the complexities of -maintaining ID mapping without LDAP, in most cases you should enable numeric UIDs -and GIDs (which is off by default in some cases) for simplified permission -management between systems: - -- [NetApp instructions](https://library.netapp.com/ecmdocs/ECMP1401220/html/GUID-24367A9F-E17B-4725-ADC1-02D86F56F78E.html) -- For non-NetApp devices, disable NFSv4 `idmapping` by performing opposite of [enable NFSv4 idmapper](https://wiki.archlinux.org/index.php/NFS#Enabling_NFSv4_idmapping) - -### Disable NFS server delegation - -We recommend that all NFS users disable the NFS server delegation feature. This -is to avoid a [Linux kernel bug](https://bugzilla.redhat.com/show_bug.cgi?id=1552203) -which causes NFS clients to slow precipitously due to -[excessive network traffic from numerous `TEST_STATEID` NFS messages](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/52017). - -To disable NFS server delegation, do the following: - -1. On the NFS server, run: - - ```shell - echo 0 > /proc/sys/fs/leases-enable - sysctl -w fs.leases-enable=0 - ``` - -1. Restart the NFS server process. For example, on CentOS run `service nfs restart`. - -#### Important notes - -The kernel bug may be fixed in -[more recent kernels with this commit](https://github.com/torvalds/linux/commit/95da1b3a5aded124dd1bda1e3cdb876184813140). - -Red Hat Enterprise 7 [shipped a kernel update](https://access.redhat.com/errata/RHSA-2019:2029) -on August 6, 2019 that may also have resolved this problem. - -You may not need to disable NFS server delegation if you know you are using a version of -the Linux kernel that has been fixed. That said, GitLab still encourages instance -administrators to keep NFS server delegation disabled. - -### Improving NFS performance with GitLab - -#### Improving NFS performance with Unicorn - -NOTE: **Note:** -From GitLab 12.1, it will automatically be detected if Rugged can and should be used per storage. - -If you previously enabled Rugged using the feature flag, you will need to unset the feature flag by using: - -```shell -sudo gitlab-rake gitlab:features:unset_rugged -``` - -If the Rugged feature flag is explicitly set to either true or false, GitLab will use the value explicitly set. - -#### Improving NFS performance with Puma - -NOTE: **Note:** -From GitLab 12.7, Rugged auto-detection is disabled if Puma thread count is greater than 1. - -If you want to use Rugged with Puma, it is recommended to [set Puma thread count to 1](https://docs.gitlab.com/omnibus/settings/puma.html#puma-settings). - -If you want to use Rugged with Puma thread count more than 1, Rugged can be enabled using the [feature flag](../../development/gitaly.md#legacy-rugged-code) - -If the Rugged feature flag is explicitly set to either true or false, GitLab will use the value explicitly set. - -### Known issues - -#### Avoid using AWS's Elastic File System (EFS) - -GitLab strongly recommends against using AWS Elastic File System (EFS). -Our support team will not be able to assist on performance issues related to -file system access. - -Customers and users have reported that AWS EFS does not perform well for GitLab's -use-case. Workloads where many small files are written in a serialized manner, like `git`, -are not well-suited for EFS. EBS with an NFS server on top will perform much better. - -If you do choose to use EFS, avoid storing GitLab log files (e.g. those in `/var/log/gitlab`) -there because this will also affect performance. We recommend that the log files be -stored on a local volume. - -For more details on another person's experience with EFS, see this [Commit Brooklyn 2019 video](https://youtu.be/K6OS8WodRBQ?t=313). - -#### Avoid using CephFS and GlusterFS - -GitLab strongly recommends against using CephFS and GlusterFS. -These distributed file systems are not well-suited for GitLab's input/output access patterns because Git uses many small files and access times and file locking times to propagate will make Git activity very slow. - -#### Avoid using PostgreSQL with NFS - -GitLab strongly recommends against running your PostgreSQL database -across NFS. The GitLab support team will not be able to assist on performance issues related to -this configuration. - -Additionally, this configuration is specifically warned against in the -[PostgreSQL Documentation](https://www.postgresql.org/docs/current/creating-cluster.html#CREATING-CLUSTER-NFS): - ->PostgreSQL does nothing special for NFS file systems, meaning it assumes NFS behaves exactly like ->locally-connected drives. If the client or server NFS implementation does not provide standard file ->system semantics, this can cause reliability problems. Specifically, delayed (asynchronous) writes ->to the NFS server can cause data corruption problems. - -For supported database architecture, please see our documentation on -[Configuring a Database for GitLab HA](../postgresql/replication_and_failover.md). - -## NFS Client mount options - -Here is an example snippet to add to `/etc/fstab`: - - ```plaintext - 10.1.0.1:/var/opt/gitlab/.ssh /var/opt/gitlab/.ssh nfs4 defaults,vers=4.1,hard,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 - 10.1.0.1:/var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads nfs4 defaults,vers=4.1,hard,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 - 10.1.0.1:/var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared nfs4 defaults,vers=4.1,hard,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 - 10.1.0.1:/var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/gitlab-ci/builds nfs4 defaults,vers=4.1,hard,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 - 10.1.0.1:/var/opt/gitlab/git-data /var/opt/gitlab/git-data nfs4 defaults,vers=4.1,hard,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 - ``` - -Note there are several options that you should consider using: - -| Setting | Description | -| ------- | ----------- | -| `vers=4.1` |NFS v4.1 should be used instead of v4.0 because there is a Linux [NFS client bug in v4.0](https://gitlab.com/gitlab-org/gitaly/-/issues/1339) that can cause significant problems due to stale data. -| `nofail` | Don't halt boot process waiting for this mount to become available -| `lookupcache=positive` | Tells the NFS client to honor `positive` cache results but invalidates any `negative` cache results. Negative cache results cause problems with Git. Specifically, a `git push` can fail to register uniformly across all NFS clients. The negative cache causes the clients to 'remember' that the files did not exist previously. -| `hard` | Instead of `soft`. [Further details](#soft-mount-option). - -### soft mount option - -We recommend that you use `hard` in your mount options, unless you have a specific -reason to use `soft`. - -On GitLab.com, we use `soft` because there were times when we had NFS servers -reboot and `soft` improved availability, but everyone's infrastructure is different. -If your NFS is provided by on-premise storage arrays with redundant controllers, -for example, you shouldn't need to worry about NFS server availability. - -The NFS man page states: - -> "soft" timeout can cause silent data corruption in certain cases - -Read the [Linux man page](https://linux.die.net/man/5/nfs) to understand the difference, -and if you do use `soft`, ensure that you've taken steps to mitigate the risks. - -If you experience behavior that might have been caused by -writes to disk on the NFS server not occurring, such as commits going missing, -use the `hard` option, because (from the man page): - -> use the soft option only when client responsiveness is more important than data integrity - -Other vendors make similar recommendations, including -[SAP](http://wiki.scn.sap.com/wiki/x/PARnFQ) and NetApp's -[knowledge base](https://kb.netapp.com/Advice_and_Troubleshooting/Data_Storage_Software/ONTAP_OS/What_are_the_differences_between_hard_mount_and_soft_mount), -they highlight that if the NFS client driver caches data, `soft` means there is no certainty if -writes by GitLab are actually on disk. - -Mount points set with the option `hard` may not perform as well, and if the -NFS server goes down, `hard` will cause processes to hang when interacting with -the mount point. Use `SIGKILL` (`kill -9`) to deal with hung processes. -The `intr` option -[stopped working in the 2.6 kernel](https://access.redhat.com/solutions/157873). - -## A single NFS mount - -It's recommended to nest all GitLab data directories within a mount, that allows automatic -restore of backups without manually moving existing data. - -```plaintext -mountpoint -└── gitlab-data - ├── builds - ├── git-data - ├── shared - └── uploads -``` - -To do so, we'll need to configure Omnibus with the paths to each directory nested -in the mount point as follows: - -Mount `/gitlab-nfs` then use the following Omnibus -configuration to move each data location to a subdirectory: - -```ruby -git_data_dirs({"default" => { "path" => "/gitlab-nfs/gitlab-data/git-data"} }) -gitlab_rails['uploads_directory'] = '/gitlab-nfs/gitlab-data/uploads' -gitlab_rails['shared_path'] = '/gitlab-nfs/gitlab-data/shared' -gitlab_ci['builds_directory'] = '/gitlab-nfs/gitlab-data/builds' -``` - -Run `sudo gitlab-ctl reconfigure` to start using the central location. Please -be aware that if you had existing data you will need to manually copy/rsync it -to these new locations and then restart GitLab. - -## Bind mounts - -Alternatively to changing the configuration in Omnibus, bind mounts can be used -to store the data on an NFS mount. - -Bind mounts provide a way to specify just one NFS mount and then -bind the default GitLab data locations to the NFS mount. Start by defining your -single NFS mount point as you normally would in `/etc/fstab`. Let's assume your -NFS mount point is `/gitlab-nfs`. Then, add the following bind mounts in -`/etc/fstab`: - -```shell -/gitlab-nfs/gitlab-data/git-data /var/opt/gitlab/git-data none bind 0 0 -/gitlab-nfs/gitlab-data/.ssh /var/opt/gitlab/.ssh none bind 0 0 -/gitlab-nfs/gitlab-data/uploads /var/opt/gitlab/gitlab-rails/uploads none bind 0 0 -/gitlab-nfs/gitlab-data/shared /var/opt/gitlab/gitlab-rails/shared none bind 0 0 -/gitlab-nfs/gitlab-data/builds /var/opt/gitlab/gitlab-ci/builds none bind 0 0 -``` - -Using bind mounts will require manually making sure the data directories -are empty before attempting a restore. Read more about the -[restore prerequisites](../../raketasks/backup_restore.md). - -## Multiple NFS mounts - -When using default Omnibus configuration you will need to share 4 data locations -between all GitLab cluster nodes. No other locations should be shared. The -following are the 4 locations need to be shared: - -| Location | Description | Default configuration | -| -------- | ----------- | --------------------- | -| `/var/opt/gitlab/git-data` | Git repository data. This will account for a large portion of your data | `git_data_dirs({"default" => { "path" => "/var/opt/gitlab/git-data"} })` -| `/var/opt/gitlab/gitlab-rails/uploads` | User uploaded attachments | `gitlab_rails['uploads_directory'] = '/var/opt/gitlab/gitlab-rails/uploads'` -| `/var/opt/gitlab/gitlab-rails/shared` | Build artifacts, GitLab Pages, LFS objects, temp files, etc. If you're using LFS this may also account for a large portion of your data | `gitlab_rails['shared_path'] = '/var/opt/gitlab/gitlab-rails/shared'` -| `/var/opt/gitlab/gitlab-ci/builds` | GitLab CI/CD build traces | `gitlab_ci['builds_directory'] = '/var/opt/gitlab/gitlab-ci/builds'` - -Other GitLab directories should not be shared between nodes. They contain -node-specific files and GitLab code that does not need to be shared. To ship -logs to a central location consider using remote syslog. Omnibus GitLab packages -provide configuration for [UDP log shipping](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-shipping-gitlab-enterprise-edition-only). - -Having multiple NFS mounts will require manually making sure the data directories -are empty before attempting a restore. Read more about the -[restore prerequisites](../../raketasks/backup_restore.md). - ---- - -Read more on high-availability configuration: - -1. [Configure the database](../postgresql/replication_and_failover.md) -1. [Configure Redis](redis.md) -1. [Configure the GitLab application servers](gitlab.md) -1. [Configure the load balancers](load_balancer.md) - - +This document was moved to [another location](../nfs.md). diff --git a/doc/administration/high_availability/nfs_host_client_setup.md b/doc/administration/high_availability/nfs_host_client_setup.md index 213680e2f64..e3342fa0813 100644 --- a/doc/administration/high_availability/nfs_host_client_setup.md +++ b/doc/administration/high_availability/nfs_host_client_setup.md @@ -1,157 +1,5 @@ --- -type: reference +redirect_to: ../nfs.md --- -# Configuring NFS for GitLab HA - -Setting up NFS for a GitLab HA setup allows all applications nodes in a cluster -to share the same files and maintain data consistency. Application nodes in an HA -setup act as clients while the NFS server plays host. - -> Note: The instructions provided in this documentation allow for setting a quick -proof of concept but will leave NFS as potential single point of failure and -therefore not recommended for use in production. Explore options such as [Pacemaker -and Corosync](https://clusterlabs.org) for highly available NFS in production. - -Below are instructions for setting up an application node(client) in an HA cluster -to read from and write to a central NFS server(host). - -NOTE: **Note:** -Using EFS may negatively impact performance. Please review the [relevant documentation](nfs.md#avoid-using-awss-elastic-file-system-efs) for additional details. - -## NFS Server Setup - -> Follow the instructions below to set up and configure your NFS server. - -### Step 1 - Install NFS Server on Host - -Installing the `nfs-kernel-server` package allows you to share directories with the clients running the GitLab application. - -```shell -apt-get update -apt-get install nfs-kernel-server -``` - -### Step 2 - Export Host's Home Directory to Client - -In this setup we will share the home directory on the host with the client. Edit the exports file as below to share the host's home directory with the client. If you have multiple clients running GitLab you must enter the client IP addresses in line in the `/etc/exports` file. - -```plaintext -#/etc/exports for one client -/home (rw,sync,no_root_squash,no_subtree_check) - -#/etc/exports for three clients -/home (rw,sync,no_root_squash,no_subtree_check) (rw,sync,no_root_squash,no_subtree_check) (rw,sync,no_root_squash,no_subtree_check) -``` - -Restart the NFS server after making changes to the `exports` file for the changes -to take effect. - -```shell -systemctl restart nfs-kernel-server -``` - -NOTE: **Note:** -You may need to update your server's firewall. See the [firewall section](#nfs-in-a-firewalled-environment) at the end of this guide. - -## Client / GitLab application node Setup - -> Follow the instructions below to connect any GitLab Rails application node running -inside your HA environment to the NFS server configured above. - -### Step 1 - Install NFS Common on Client - -The `nfs-common` provides NFS functionality without installing server components which -we don't need running on the application nodes. - -```shell -apt-get update -apt-get install nfs-common -``` - -### Step 2 - Create Mount Points on Client - -Create a directory on the client that we can mount the shared directory from the host. -Please note that if your mount point directory contains any files they will be hidden -once the remote shares are mounted. An empty/new directory on the client is recommended -for this purpose. - -```shell -mkdir -p /nfs/home -``` - -Confirm that the mount point works by mounting it on the client and checking that -it is mounted with the command below: - -```shell -mount :/home -df -h -``` - -### Step 3 - Set up Automatic Mounts on Boot - -Edit `/etc/fstab` on the client as below to mount the remote shares automatically at boot. -Note that GitLab requires advisory file locking, which is only supported natively in -NFS version 4. NFSv3 also supports locking as long as Linux Kernel 2.6.5+ is used. -We recommend using version 4 and do not specifically test NFSv3. -See [NFS documentation](nfs.md#nfs-client-mount-options) for guidance on mount options. - -```plaintext -#/etc/fstab -:/home /nfs/home nfs4 defaults,hard,vers=4.1,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 -``` - -Reboot the client and confirm that the mount point is mounted automatically. - -NOTE: **Note:** -If you followed our guide to [GitLab Pages on a separate server](../pages/index.md#running-gitlab-pages-on-a-separate-server) -here, please continue there with the pages-specific NFS mounts. -The step below is for broader use-cases than only sharing pages data. - -### Step 4 - Set up GitLab to Use NFS mounts - -When using the default Omnibus configuration you will need to share 4 data locations -between all GitLab cluster nodes. No other locations should be shared. Changing the -default file locations in `gitlab.rb` on the client allows you to have one main mount -point and have all the required locations as subdirectories to use the NFS mount for -`git-data`. - -```plaintext -git_data_dirs({"default" => {"path" => "/nfs/home/var/opt/gitlab-data/git-data"}}) -gitlab_rails['uploads_directory'] = '/nfs/home/var/opt/gitlab-data/uploads' -gitlab_rails['shared_path'] = '/nfs/home/var/opt/gitlab-data/shared' -gitlab_ci['builds_directory'] = '/nfs/home/var/opt/gitlab-data/builds' -``` - -Save the changes in `gitlab.rb` and run `gitlab-ctl reconfigure`. - -## NFS in a Firewalled Environment - -If the traffic between your NFS server and NFS client(s) is subject to port filtering -by a firewall, then you will need to reconfigure that firewall to allow NFS communication. - -[This guide from TDLP](http://tldp.org/HOWTO/NFS-HOWTO/security.html#FIREWALLS) -covers the basics of using NFS in a firewalled environment. Additionally, we encourage you to -search for and review the specific documentation for your operating system or distribution and your firewall software. - -Example for Ubuntu: - -Check that NFS traffic from the client is allowed by the firewall on the host by running -the command: `sudo ufw status`. If it's being blocked, then you can allow traffic from a specific -client with the command below. - -```shell -sudo ufw allow from to any port nfs -``` - - +This document was moved to [another location](../nfs.md). diff --git a/doc/administration/high_availability/pgbouncer.md b/doc/administration/high_availability/pgbouncer.md index 15e4da5b1f7..44f4aa37651 100644 --- a/doc/administration/high_availability/pgbouncer.md +++ b/doc/administration/high_availability/pgbouncer.md @@ -1,166 +1,5 @@ --- -type: reference +redirect_to: ../postgresql/pgbouncer.md --- -# Working with the bundled PgBouncer service **(PREMIUM ONLY)** - -As part of its High Availability stack, GitLab Premium includes a bundled version of [PgBouncer](http://www.pgbouncer.org/) that can be managed through `/etc/gitlab/gitlab.rb`. PgBouncer is used to seamlessly migrate database connections between servers in a failover scenario. Additionally, it can be used in a non-HA setup to pool connections, speeding up response time while reducing resource usage. - -In a HA setup, it's recommended to run a PgBouncer node separately for each database node with an internal load balancer (TCP) serving each accordingly. - -## Operations - -### Running PgBouncer as part of an HA GitLab installation - -This content has been moved to a [new location](../postgresql/replication_and_failover.md#configuring-the-pgbouncer-node). - -### Running PgBouncer as part of a non-HA GitLab installation - -1. Generate PGBOUNCER_USER_PASSWORD_HASH with the command `gitlab-ctl pg-password-md5 pgbouncer` - -1. Generate SQL_USER_PASSWORD_HASH with the command `gitlab-ctl pg-password-md5 gitlab`. We'll also need to enter the plaintext SQL_USER_PASSWORD later - -1. On your database node, ensure the following is set in your `/etc/gitlab/gitlab.rb` - - ```ruby - postgresql['pgbouncer_user_password'] = 'PGBOUNCER_USER_PASSWORD_HASH' - postgresql['sql_user_password'] = 'SQL_USER_PASSWORD_HASH' - postgresql['listen_address'] = 'XX.XX.XX.Y' # Where XX.XX.XX.Y is the ip address on the node postgresql should listen on - postgresql['md5_auth_cidr_addresses'] = %w(AA.AA.AA.B/32) # Where AA.AA.AA.B is the IP address of the pgbouncer node - ``` - -1. Run `gitlab-ctl reconfigure` - - **Note:** If the database was already running, it will need to be restarted after reconfigure by running `gitlab-ctl restart postgresql`. - -1. On the node you are running PgBouncer on, make sure the following is set in `/etc/gitlab/gitlab.rb` - - ```ruby - pgbouncer['enable'] = true - pgbouncer['databases'] = { - gitlabhq_production: { - host: 'DATABASE_HOST', - user: 'pgbouncer', - password: 'PGBOUNCER_USER_PASSWORD_HASH' - } - } - ``` - -1. Run `gitlab-ctl reconfigure` - -1. On the node running Puma, make sure the following is set in `/etc/gitlab/gitlab.rb` - - ```ruby - gitlab_rails['db_host'] = 'PGBOUNCER_HOST' - gitlab_rails['db_port'] = '6432' - gitlab_rails['db_password'] = 'SQL_USER_PASSWORD' - ``` - -1. Run `gitlab-ctl reconfigure` - -1. At this point, your instance should connect to the database through PgBouncer. If you are having issues, see the [Troubleshooting](#troubleshooting) section - -## Enable Monitoring - -> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3786) in GitLab 12.0. - -If you enable Monitoring, it must be enabled on **all** PgBouncer servers. - -1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration: - - ```ruby - # Enable service discovery for Prometheus - consul['enable'] = true - consul['monitoring_service_discovery'] = true - - # Replace placeholders - # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z - # with the addresses of the Consul server nodes - consul['configuration'] = { - retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z), - } - - # Set the network addresses that the exporters will listen on - node_exporter['listen_address'] = '0.0.0.0:9100' - pgbouncer_exporter['listen_address'] = '0.0.0.0:9188' - ``` - -1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. - -### Interacting with PgBouncer - -#### Administrative console - -As part of Omnibus GitLab, we provide a command `gitlab-ctl pgb-console` to automatically connect to the PgBouncer administrative console. Please see the [PgBouncer documentation](https://www.pgbouncer.org/usage.html#admin-console) for detailed instructions on how to interact with the console. - -To start a session, run - -```shell -# gitlab-ctl pgb-console -Password for user pgbouncer: -psql (11.7, server 1.7.2/bouncer) -Type "help" for help. - -pgbouncer=# -``` - -The password you will be prompted for is the PGBOUNCER_USER_PASSWORD - -To get some basic information about the instance, run - -```shell -pgbouncer=# show databases; show clients; show servers; - name | host | port | database | force_user | pool_size | reserve_pool | pool_mode | max_connections | current_connections ----------------------+-----------+------+---------------------+------------+-----------+--------------+-----------+-----------------+--------------------- - gitlabhq_production | 127.0.0.1 | 5432 | gitlabhq_production | | 100 | 5 | | 0 | 1 - pgbouncer | | 6432 | pgbouncer | pgbouncer | 2 | 0 | statement | 0 | 0 -(2 rows) - - type | user | database | state | addr | port | local_addr | local_port | connect_time | request_time | ptr | link -| remote_pid | tls -------+-----------+---------------------+--------+-----------+-------+------------+------------+---------------------+---------------------+-----------+------ -+------------+----- - C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44590 | 127.0.0.1 | 6432 | 2018-04-24 22:13:10 | 2018-04-24 22:17:10 | 0x12444c0 | -| 0 | - C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44592 | 127.0.0.1 | 6432 | 2018-04-24 22:13:10 | 2018-04-24 22:17:10 | 0x12447c0 | -| 0 | - C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44594 | 127.0.0.1 | 6432 | 2018-04-24 22:13:10 | 2018-04-24 22:17:10 | 0x1244940 | -| 0 | - C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44706 | 127.0.0.1 | 6432 | 2018-04-24 22:14:22 | 2018-04-24 22:16:31 | 0x1244ac0 | -| 0 | - C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44708 | 127.0.0.1 | 6432 | 2018-04-24 22:14:22 | 2018-04-24 22:15:15 | 0x1244c40 | -| 0 | - C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44794 | 127.0.0.1 | 6432 | 2018-04-24 22:15:15 | 2018-04-24 22:15:15 | 0x1244dc0 | -| 0 | - C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44798 | 127.0.0.1 | 6432 | 2018-04-24 22:15:15 | 2018-04-24 22:16:31 | 0x1244f40 | -| 0 | - C | pgbouncer | pgbouncer | active | 127.0.0.1 | 44660 | 127.0.0.1 | 6432 | 2018-04-24 22:13:51 | 2018-04-24 22:17:12 | 0x1244640 | -| 0 | -(8 rows) - - type | user | database | state | addr | port | local_addr | local_port | connect_time | request_time | ptr | link | rem -ote_pid | tls -------+--------+---------------------+-------+-----------+------+------------+------------+---------------------+---------------------+-----------+------+---- ---------+----- - S | gitlab | gitlabhq_production | idle | 127.0.0.1 | 5432 | 127.0.0.1 | 35646 | 2018-04-24 22:15:15 | 2018-04-24 22:17:10 | 0x124dca0 | | - 19980 | -(1 row) -``` - -## Troubleshooting - -In case you are experiencing any issues connecting through PgBouncer, the first place to check is always the logs: - -```shell -# gitlab-ctl tail pgbouncer -``` - -Additionally, you can check the output from `show databases` in the [Administrative console](#administrative-console). In the output, you would expect to see values in the `host` field for the `gitlabhq_production` database. Additionally, `current_connections` should be greater than 1. - -### Message: `LOG: invalid CIDR mask in address` - -See the suggested fix [in Geo documentation](../geo/replication/troubleshooting.md#message-log--invalid-cidr-mask-in-address). - -### Message: `LOG: invalid IP mask "md5": Name or service not known` - -See the suggested fix [in Geo documentation](../geo/replication/troubleshooting.md#message-log--invalid-ip-mask-md5-name-or-service-not-known). +This document was moved to [another location](../postgresql/pgbouncer.md). diff --git a/doc/administration/high_availability/sidekiq.md b/doc/administration/high_availability/sidekiq.md index 98a9af64e5e..ac92ae2eaaa 100644 --- a/doc/administration/high_availability/sidekiq.md +++ b/doc/administration/high_availability/sidekiq.md @@ -1,190 +1,5 @@ --- -type: reference +redirect_to: ../sidekiq.md --- -# Configuring Sidekiq - -This section discusses how to configure an external Sidekiq instance. - -Sidekiq requires connection to the Redis, PostgreSQL and Gitaly instance. -To configure the Sidekiq node: - -1. SSH into the Sidekiq server. - -1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab package -you want using steps 1 and 2 from the GitLab downloads page. -**Do not complete any other steps on the download page.** - -1. Open `/etc/gitlab/gitlab.rb` with your editor. - -1. Generate the Sidekiq configuration: - - ```ruby - sidekiq['listen_address'] = "10.10.1.48" - - ## Optional: Enable extra Sidekiq processes - sidekiq_cluster['enable'] = true - sidekiq_cluster['enable'] = true - "elastic_indexer" - ] - ``` - -1. Setup Sidekiq's connection to Redis: - - ```ruby - ## Must be the same in every sentinel node - redis['master_name'] = 'gitlab-redis' - - ## The same password for Redis authentication you set up for the master node. - redis['master_password'] = 'YOUR_PASSOWORD' - - ## A list of sentinels with `host` and `port` - gitlab_rails['redis_sentinels'] = [ - {'host' => '10.10.1.34', 'port' => 26379}, - {'host' => '10.10.1.35', 'port' => 26379}, - {'host' => '10.10.1.36', 'port' => 26379}, - ] - ``` - -1. Setup Sidekiq's connection to Gitaly: - - ```ruby - git_data_dirs({ - 'default' => { 'gitaly_address' => 'tcp://gitaly:8075' }, - }) - gitlab_rails['gitaly_token'] = 'YOUR_TOKEN' - ``` - -1. Setup Sidekiq's connection to PostgreSQL: - - ```ruby - gitlab_rails['db_host'] = '10.10.1.30' - gitlab_rails['db_password'] = 'YOUR_PASSOWORD' - gitlab_rails['db_port'] = '5432' - gitlab_rails['db_adapter'] = 'postgresql' - gitlab_rails['db_encoding'] = 'unicode' - gitlab_rails['auto_migrate'] = false - ``` - - Remember to add the Sidekiq nodes to the PostgreSQL whitelist: - - ```ruby - postgresql['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 10.10.1.30/32 10.10.1.31/32 10.10.1.32/32 10.10.1.33/32 10.10.1.38/32) - ``` - -1. Disable other services: - - ```ruby - nginx['enable'] = false - grafana['enable'] = false - prometheus['enable'] = false - gitlab_rails['auto_migrate'] = false - alertmanager['enable'] = false - gitaly['enable'] = false - gitlab_monitor['enable'] = false - gitlab_workhorse['enable'] = false - nginx['enable'] = false - postgres_exporter['enable'] = false - postgresql['enable'] = false - redis['enable'] = false - redis_exporter['enable'] = false - puma['enable'] = false - gitlab_exporter['enable'] = false - ``` - -1. Run `gitlab-ctl reconfigure`. - -NOTE: **Note:** -You will need to restart the Sidekiq nodes after an update has occurred and database -migrations performed. - -## Example configuration - -Here's what the ending `/etc/gitlab/gitlab.rb` would look like: - -```ruby -######################################## -##### Services Disabled ### -######################################## - -nginx['enable'] = false -grafana['enable'] = false -prometheus['enable'] = false -gitlab_rails['auto_migrate'] = false -alertmanager['enable'] = false -gitaly['enable'] = false -gitlab_monitor['enable'] = false -gitlab_workhorse['enable'] = false -nginx['enable'] = false -postgres_exporter['enable'] = false -postgresql['enable'] = false -redis['enable'] = false -redis_exporter['enable'] = false -puma['enable'] = false -gitlab_exporter['enable'] = false - -######################################## -#### Redis ### -######################################## - -## Must be the same in every sentinel node -redis['master_name'] = 'gitlab-redis' - -## The same password for Redis authentication you set up for the master node. -redis['master_password'] = 'YOUR_PASSOWORD' - -## A list of sentinels with `host` and `port` -gitlab_rails['redis_sentinels'] = [ - {'host' => '10.10.1.34', 'port' => 26379}, - {'host' => '10.10.1.35', 'port' => 26379}, - {'host' => '10.10.1.36', 'port' => 26379}, - ] - -####################################### -### Gitaly ### -####################################### - -git_data_dirs({ - 'default' => { 'gitaly_address' => 'tcp://gitaly:8075' }, -}) -gitlab_rails['gitaly_token'] = 'YOUR_TOKEN' - -####################################### -### Postgres ### -####################################### -gitlab_rails['db_host'] = '10.10.1.30' -gitlab_rails['db_password'] = 'YOUR_PASSOWORD' -gitlab_rails['db_port'] = '5432' -gitlab_rails['db_adapter'] = 'postgresql' -gitlab_rails['db_encoding'] = 'unicode' -gitlab_rails['auto_migrate'] = false - -####################################### -### Sidekiq configuration ### -####################################### -sidekiq['listen_address'] = "10.10.1.48" - -####################################### -### Monitoring configuration ### -####################################### -consul['enable'] = true -consul['monitoring_service_discovery'] = true - -consul['configuration'] = { - bind_addr: '10.10.1.48', - retry_join: %w(10.10.1.34 10.10.1.35 10.10.1.36) -} - -# Set the network addresses that the exporters will listen on -node_exporter['listen_address'] = '10.10.1.48:9100' - -# Rails Status for prometheus -gitlab_rails['monitoring_whitelist'] = ['10.10.1.42', '127.0.0.1'] -``` - -## Further reading - -Related Sidekiq configuration: - -1. [Extra Sidekiq processes](../operations/extra_sidekiq_processes.md) -1. [Using the GitLab-Sidekiq chart](https://docs.gitlab.com/charts/charts/gitlab/sidekiq/) +This document was moved to [another location](../sidekiq.md). diff --git a/doc/administration/img/repository_storages_admin_ui_v13_1.png b/doc/administration/img/repository_storages_admin_ui_v13_1.png index f8c13a35369..a2b88d14a36 100644 Binary files a/doc/administration/img/repository_storages_admin_ui_v13_1.png and b/doc/administration/img/repository_storages_admin_ui_v13_1.png differ diff --git a/doc/administration/index.md b/doc/administration/index.md index fa415e5f78d..ed079abf708 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -138,7 +138,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. ## Package Registry administration - [Container Registry](packages/container_registry.md): Configure Container Registry with GitLab. -- [Package Registry](packages/index.md): Enable GitLab to act as an NPM Registry and a Maven Repository. **(PREMIUM ONLY)** +- [Package Registry](packages/index.md): Enable GitLab to act as an NPM Registry and a Maven Repository. - [Dependency Proxy](packages/dependency_proxy.md): Configure the Dependency Proxy, a local proxy for frequently used upstream images/packages. **(PREMIUM ONLY)** ### Repository settings @@ -163,7 +163,11 @@ Learn how to install, configure, update, and maintain your GitLab instance. ## Snippet settings -- [Setting snippet content size limit](snippets/index.md): Set a maximum size limit for snippets' content. +- [Setting snippet content size limit](snippets/index.md): Set a maximum content size limit for snippets. + +## Wiki settings + +- [Setting wiki page content size limit](wikis/index.md): Set a maximum content size limit for wiki pages. ## Git configuration options diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md index 6d2fbd95d5e..f30dba331b8 100644 --- a/doc/administration/instance_limits.md +++ b/doc/administration/instance_limits.md @@ -160,7 +160,7 @@ There is a limit when embedding metrics in GFM for performance reasons. ## Number of webhooks -On GitLab.com, the [maximum number of webhooks](../user/gitlab_com/index.md#maximum-number-of-webhooks) per project, and per group, is limited. +On GitLab.com, the [maximum number of webhooks and their size](../user/gitlab_com/index.md#webhooks) per project, and per group, is limited. To set this limit on a self-managed installation, run the following in the [GitLab Rails console](troubleshooting/debug.md#starting-a-rails-console-session): @@ -314,6 +314,58 @@ To update this limit to a new value on a self-managed installation, run the foll Plan.default.actual_limits.update!(ci_instance_level_variables: 30) ``` +### Maximum file size per type of artifact + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37226) in GitLab 13.3. + +Job artifacts defined with [`artifacts:reports`](../ci/pipelines/job_artifacts.md#artifactsreports) +that are uploaded by the Runner are rejected if the file size exceeds the maximum +file size limit. The limit is determined by comparing the project's +[maximum artifact size setting](../user/admin_area/settings/continuous_integration.md#maximum-artifacts-size-core-only) +with the instance limit for the given artifact type, and choosing the smaller value. + +Limits are set in megabytes, so the smallest possible value that can be defined is `1 MB`. + +Each type of artifact has a size limit that can be set. A default of `0` means there +is no limit for that specific artifact type, and the project's maximum artifact size +setting is used: + +| Artifact limit name | Default value | +|---------------------------------------------|---------------| +| `ci_max_artifact_size_accessibility` | 0 | +| `ci_max_artifact_size_archive` | 0 | +| `ci_max_artifact_size_browser_performance` | 0 | +| `ci_max_artifact_size_cluster_applications` | 0 | +| `ci_max_artifact_size_cobertura` | 0 | +| `ci_max_artifact_size_codequality` | 0 | +| `ci_max_artifact_size_container_scanning` | 0 | +| `ci_max_artifact_size_coverage_fuzzing` | 0 | +| `ci_max_artifact_size_dast` | 0 | +| `ci_max_artifact_size_dependency_scanning` | 0 | +| `ci_max_artifact_size_dotenv` | 0 | +| `ci_max_artifact_size_junit` | 0 | +| `ci_max_artifact_size_license_management` | 0 | +| `ci_max_artifact_size_license_scanning` | 0 | +| `ci_max_artifact_size_load_performance` | 0 | +| `ci_max_artifact_size_lsif` | 20 MB ([introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37226) in GitLab 13.3) | +| `ci_max_artifact_size_metadata` | 0 | +| `ci_max_artifact_size_metrics_referee` | 0 | +| `ci_max_artifact_size_metrics` | 0 | +| `ci_max_artifact_size_network_referee` | 0 | +| `ci_max_artifact_size_performance` | 0 | +| `ci_max_artifact_size_requirements` | 0 | +| `ci_max_artifact_size_sast` | 0 | +| `ci_max_artifact_size_secret_detection` | 0 | +| `ci_max_artifact_size_terraform` | 5 MB ([introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37018) in GitLab 13.3) | +| `ci_max_artifact_size_trace` | 0 | + +For example, to set the `ci_max_artifact_size_junit` limit to 10MB on a self-managed +installation, run the following in the [GitLab Rails console](troubleshooting/debug.md#starting-a-rails-console-session): + +```ruby +Plan.default.actual_limits.update!(ci_max_artifact_size_junit: 10) +``` + ## Instance monitoring and metrics ### Incident Management inbound alert limits @@ -388,6 +440,24 @@ Reports that go over the 20 MB limit won't be loaded. Affected reports: ## Advanced Global Search limits +### Maximum file size indexed + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/8638) in GitLab 13.3. + +You can set a limit on the content of repository files that are indexed in +Elasticsearch. Any files larger than this limit will not be indexed, and thus +will not be searchable. + +Setting a limit helps reduce the memory usage of the indexing processes as well +as the overall index size. This value defaults to `1024 KiB` (1 MiB) as any +text files larger than this likely aren't meant to be read by humans. + +NOTE: **Note:** +You must set a limit, as an unlimited file size is not supported. Setting this +value to be greater than the amount of memory on GitLab's Sidekiq nodes will +lead to GitLab's Sidekiq nodes running out of memory as they will pre-allocate +this amount of memory during indexing. + ### Maximum field length > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/201826) in GitLab 12.8. @@ -396,6 +466,9 @@ You can set a limit on the content of text fields indexed for Global Search. Setting a maximum helps to reduce the load of the indexing processes. If any text field exceeds this limit then the text will be truncated to this number of characters and the rest will not be indexed and hence will not be searchable. +This is applicable to all indexed data except repository files that get +indexed, which have a separate limit (see [Maximum file size +indexed](#maximum-file-size-indexed)). - On GitLab.com this is limited to 20000 characters - For self-managed installations it is unlimited by default @@ -408,6 +481,7 @@ Set the limit to `0` to disable it. ## Wiki limits +- [Wiki page content size limit](wikis/index.md#wiki-page-content-size-limit). - [Length restrictions for file and directory names](../user/project/wiki/index.md#length-restrictions-for-file-and-directory-names). ## Snippets limits diff --git a/doc/administration/integration/plantuml.md b/doc/administration/integration/plantuml.md index 2a30eced7c4..49ea59d239c 100644 --- a/doc/administration/integration/plantuml.md +++ b/doc/administration/integration/plantuml.md @@ -1,3 +1,10 @@ +--- +stage: Create +group: Source Code +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers" +type: reference, howto +--- + # PlantUML & GitLab > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/8537) in GitLab 8.16. diff --git a/doc/administration/integration/terminal.md b/doc/administration/integration/terminal.md index fd9e09dc17a..c363bd30543 100644 --- a/doc/administration/integration/terminal.md +++ b/doc/administration/integration/terminal.md @@ -57,7 +57,7 @@ through to the next one in the chain. If you installed GitLab using Omnibus, or from source, starting with GitLab 8.15, this should be done by the default configuration, so there's no need for you to do anything. -However, if you run a [load balancer](../high_availability/load_balancer.md) in +However, if you run a [load balancer](../load_balancer.md) in front of GitLab, you may need to make some changes to your configuration. These guides document the necessary steps for a selection of popular reverse proxies: @@ -98,4 +98,4 @@ they will receive a `Connection failed` message. > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/8413) in GitLab 8.17. Terminal sessions, by default, do not expire. -You can limit terminal session lifetime in your GitLab instance. To do so, navigate to **{admin}** [**Admin Area > Settings > Web terminal**](../../user/admin_area/settings/index.md#general), and set a `max session time`. +You can limit terminal session lifetime in your GitLab instance. To do so, navigate to [**Admin Area > Settings > Web terminal**](../../user/admin_area/settings/index.md#general), and set a `max session time`. diff --git a/doc/administration/invalidate_markdown_cache.md b/doc/administration/invalidate_markdown_cache.md index 5fd804e11dc..ac43d0eae63 100644 --- a/doc/administration/invalidate_markdown_cache.md +++ b/doc/administration/invalidate_markdown_cache.md @@ -1,3 +1,10 @@ +--- +stage: Create +group: Source Code +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers" +type: reference +--- + # Invalidate Markdown Cache For performance reasons, GitLab caches the HTML version of Markdown text diff --git a/doc/administration/issue_closing_pattern.md b/doc/administration/issue_closing_pattern.md index 579b957eb47..7abe0f725f2 100644 --- a/doc/administration/issue_closing_pattern.md +++ b/doc/administration/issue_closing_pattern.md @@ -1,6 +1,13 @@ +--- +stage: Create +group: Source Code +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers" +type: reference +--- + # Issue closing pattern **(CORE ONLY)** ->**Note:** +NOTE: **Note:** This is the administration documentation. There is a separate [user documentation](../user/project/issues/managing_issues.md#closing-issues-automatically) on issue closing pattern. @@ -16,7 +23,7 @@ is installed on. The default pattern can be located in [`gitlab.yml.example`](https://gitlab.com/gitlab-org/gitlab/blob/master/config/gitlab.yml.example) under the "Automatic issue closing" section. -> **Tip:** +TIP: **Tip:** You are advised to use to test the issue closing pattern. Because Rubular doesn't understand `%{issue_ref}`, you can replace this by `#\d+` when testing your patterns, which matches only local issue references like `#123`. diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index cb31a8934b1..66a7bcb90f6 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -164,9 +164,30 @@ _The artifacts are stored by default in gitlab-rake gitlab:artifacts:migrate ``` +1. Optional: Verify all files migrated properly. + From [PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database) + (`sudo gitlab-psql -d gitlabhq_production`) verify `objectstg` below (where `file_store=2`) has count of all artifacts: + + ```shell + gitlabhq_production=# SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM ci_job_artifacts; + + total | filesystem | objectstg + ------+------------+----------- + 2409 | 0 | 2409 + ``` + + Verify no files on disk in `artifacts` folder: + + ```shell + sudo find /var/opt/gitlab/gitlab-rails/shared/artifacts -type f | grep -v tmp/cache | wc -l + ``` + + In some cases, you may need to run the [orphan artifact file cleanup Rake task](../raketasks/cleanup.md#remove-orphan-artifact-files) + to clean up orphaned artifacts. + CAUTION: **Caution:** JUnit test report artifact (`junit.xml.gz`) migration -[is not supported](https://gitlab.com/gitlab-org/gitlab/-/issues/27698) +[was not supported until GitLab 12.8](https://gitlab.com/gitlab-org/gitlab/-/issues/27698#note_317190991) by the `gitlab:artifacts:migrate` script. **In installations from source:** @@ -197,9 +218,29 @@ _The artifacts are stored by default in sudo -u git -H bundle exec rake gitlab:artifacts:migrate RAILS_ENV=production ``` +1. Optional: Verify all files migrated properly. + From PostgreSQL console (`sudo -u git -H psql -d gitlabhq_production`) verify `objectstg` below (where `file_store=2`) has count of all artifacts: + + ```shell + gitlabhq_production=# SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM ci_job_artifacts; + + total | filesystem | objectstg + ------+------------+----------- + 2409 | 0 | 2409 + ``` + + Verify no files on disk in `artifacts` folder: + + ```shell + sudo find /var/opt/gitlab/gitlab-rails/shared/artifacts -type f | grep -v tmp/cache | wc -l + ``` + + In some cases, you may need to run the [orphan artifact file cleanup Rake task](../raketasks/cleanup.md#remove-orphan-artifact-files) + to clean up orphaned artifacts. + CAUTION: **Caution:** JUnit test report artifact (`junit.xml.gz`) migration -[is not supported](https://gitlab.com/gitlab-org/gitlab/-/issues/27698) +[was not supported until GitLab 12.8](https://gitlab.com/gitlab-org/gitlab/-/issues/27698#note_317190991) by the `gitlab:artifacts:migrate` script. ### OpenStack example diff --git a/doc/administration/lfs/index.md b/doc/administration/lfs/index.md index 4a8151bd091..5c1a9519a35 100644 --- a/doc/administration/lfs/index.md +++ b/doc/administration/lfs/index.md @@ -1,4 +1,8 @@ --- +stage: Create +group: Source Code +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers" +type: reference, howto disqus_identifier: 'https://docs.gitlab.com/ee/workflow/lfs/lfs_administration.html' --- @@ -152,7 +156,24 @@ On Omnibus installations, the settings are prefixed by `lfs_object_store_`: This will migrate existing LFS objects to object storage. New LFS objects will be forwarded to object storage unless - `gitlab_rails['lfs_object_store_background_upload']` is set to false. + `gitlab_rails['lfs_object_store_background_upload']` and `gitlab_rails['lfs_object_store_direct_upload']` is set to `false`. +1. Optional: Verify all files migrated properly. + From [PostgreSQL console](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database) + (`sudo gitlab-psql -d gitlabhq_production`) verify `objectstg` below (where `file_store=2`) has count of all artifacts: + + ```shell + gitlabhq_production=# SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM lfs_objects; + + total | filesystem | objectstg + ------+------------+----------- + 2409 | 0 | 2409 + ``` + + Verify no files on disk in `artifacts` folder: + + ```shell + sudo find /var/opt/gitlab/gitlab-rails/shared/lfs-objects -type f | grep -v tmp/cache | wc -l + ``` ### S3 for installations from source @@ -187,14 +208,30 @@ For source installations the settings are nested under `lfs:` and then ``` This will migrate existing LFS objects to object storage. New LFS objects - will be forwarded to object storage unless `background_upload` is set to - false. + will be forwarded to object storage unless `background_upload` and `direct_upload` is set to + `false`. +1. Optional: Verify all files migrated properly. + From PostgreSQL console (`sudo -u git -H psql -d gitlabhq_production`) verify `objectstg` below (where `file_store=2`) has count of all artifacts: + + ```shell + gitlabhq_production=# SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM lfs_objects; + + total | filesystem | objectstg + ------+------------+----------- + 2409 | 0 | 2409 + ``` + + Verify no files on disk in `artifacts` folder: + + ```shell + sudo find /var/opt/gitlab/gitlab-rails/shared/lfs-objects -type f | grep -v tmp/cache | wc -l + ``` ### Migrating back to local storage In order to migrate back to local storage: -1. Set both `direct_upload` and `background_upload` to false under the LFS object storage settings. Don't forget to restart GitLab. +1. Set both `direct_upload` and `background_upload` to `false` under the LFS object storage settings. Don't forget to restart GitLab. 1. Run `rake gitlab:lfs:migrate_to_local` on your console. 1. Disable `object_storage` for LFS objects in `gitlab.rb`. Remember to restart GitLab afterwards. diff --git a/doc/administration/load_balancer.md b/doc/administration/load_balancer.md new file mode 100644 index 00000000000..fe534f30f66 --- /dev/null +++ b/doc/administration/load_balancer.md @@ -0,0 +1,126 @@ +--- +type: reference +--- + +# Load Balancer for multi-node GitLab + +In an multi-node GitLab configuration, you will need a load balancer to route +traffic to the application servers. The specifics on which load balancer to use +or the exact configuration is beyond the scope of GitLab documentation. We hope +that if you're managing HA systems like GitLab you have a load balancer of +choice already. Some examples including HAProxy (open-source), F5 Big-IP LTM, +and Citrix Net Scaler. This documentation will outline what ports and protocols +you need to use with GitLab. + +## SSL + +How will you handle SSL in your multi-node environment? There are several different +options: + +- Each application node terminates SSL +- The load balancer(s) terminate SSL and communication is not secure between + the load balancer(s) and the application nodes +- The load balancer(s) terminate SSL and communication is *secure* between the + load balancer(s) and the application nodes + +### Application nodes terminate SSL + +Configure your load balancer(s) to pass connections on port 443 as 'TCP' rather +than 'HTTP(S)' protocol. This will pass the connection to the application nodes +NGINX service untouched. NGINX will have the SSL certificate and listen on port 443. + +See [NGINX HTTPS documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) +for details on managing SSL certificates and configuring NGINX. + +### Load Balancer(s) terminate SSL without backend SSL + +Configure your load balancer(s) to use the 'HTTP(S)' protocol rather than 'TCP'. +The load balancer(s) will then be responsible for managing SSL certificates and +terminating SSL. + +Since communication between the load balancer(s) and GitLab will not be secure, +there is some additional configuration needed. See +[NGINX Proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#supporting-proxied-ssl) +for details. + +### Load Balancer(s) terminate SSL with backend SSL + +Configure your load balancer(s) to use the 'HTTP(S)' protocol rather than 'TCP'. +The load balancer(s) will be responsible for managing SSL certificates that +end users will see. + +Traffic will also be secure between the load balancer(s) and NGINX in this +scenario. There is no need to add configuration for proxied SSL since the +connection will be secure all the way. However, configuration will need to be +added to GitLab to configure SSL certificates. See +[NGINX HTTPS documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) +for details on managing SSL certificates and configuring NGINX. + +## Ports + +### Basic ports + +| LB Port | Backend Port | Protocol | +| ------- | ------------ | ------------------------ | +| 80 | 80 | HTTP (*1*) | +| 443 | 443 | TCP or HTTPS (*1*) (*2*) | +| 22 | 22 | TCP | + +- (*1*): [Web terminal](../ci/environments/index.md#web-terminals) support requires + your load balancer to correctly handle WebSocket connections. When using + HTTP or HTTPS proxying, this means your load balancer must be configured + to pass through the `Connection` and `Upgrade` hop-by-hop headers. See the + [web terminal](integration/terminal.md) integration guide for + more details. +- (*2*): When using HTTPS protocol for port 443, you will need to add an SSL + certificate to the load balancers. If you wish to terminate SSL at the + GitLab application server instead, use TCP protocol. + +### GitLab Pages Ports + +If you're using GitLab Pages with custom domain support you will need some +additional port configurations. +GitLab Pages requires a separate virtual IP address. Configure DNS to point the +`pages_external_url` from `/etc/gitlab/gitlab.rb` at the new virtual IP address. See the +[GitLab Pages documentation](pages/index.md) for more information. + +| LB Port | Backend Port | Protocol | +| ------- | ------------- | --------- | +| 80 | Varies (*1*) | HTTP | +| 443 | Varies (*1*) | TCP (*2*) | + +- (*1*): The backend port for GitLab Pages depends on the + `gitlab_pages['external_http']` and `gitlab_pages['external_https']` + setting. See [GitLab Pages documentation](pages/index.md) for more details. +- (*2*): Port 443 for GitLab Pages should always use the TCP protocol. Users can + configure custom domains with custom SSL, which would not be possible + if SSL was terminated at the load balancer. + +### Alternate SSH Port + +Some organizations have policies against opening SSH port 22. In this case, +it may be helpful to configure an alternate SSH hostname that allows users +to use SSH on port 443. An alternate SSH hostname will require a new virtual IP address +compared to the other GitLab HTTP configuration above. + +Configure DNS for an alternate SSH hostname such as `altssh.gitlab.example.com`. + +| LB Port | Backend Port | Protocol | +| ------- | ------------ | -------- | +| 443 | 22 | TCP | + +## Readiness check + +It is strongly recommend that multi-node deployments configure load balancers to utilize the [readiness check](../user/admin_area/monitoring/health_check.md#readiness) to ensure a node is ready to accept traffic, before routing traffic to it. This is especially important when utilizing Puma, as there is a brief period during a restart where Puma will not accept requests. + + diff --git a/doc/administration/logs.md b/doc/administration/logs.md index 3db9d32563e..2e8d0bf7461 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -14,6 +14,11 @@ Find more about them [in Audit Events documentation](audit_events.md). System log files are typically plain text in a standard log file format. This guide talks about how to read and use these system log files. +[Read more about how to customise logging on Omnibus GitLab +installations](https://docs.gitlab.com/omnibus/settings/logs.html) +including adjusting log retention, log forwarding, +switching logs from JSON to plain text logging, and more. + ## `production_json.log` This file lives in `/var/log/gitlab/gitlab-rails/production_json.log` for @@ -832,6 +837,29 @@ For example: This message shows that Geo detected that a repository update was needed for project `1`. +## `update_mirror_service_json.log` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/commit/7f637e2af7006dc2b1b2649d9affc0b86cfb33c4) in GitLab 11.12. + +This file is stored in: + +- `/var/log/gitlab/gitlab-rails/update_mirror_service_json.log` for Omnibus GitLab installations. +- `/home/git/gitlab/log/update_mirror_service_json.log` for installations from source. + +This file contains information about any errors that occurred during project mirroring. + +```json +{ + "severity":"ERROR", + "time":"2020-07-28T23:29:29.473Z", + "correlation_id":"5HgIkCJsO53", + "user_id":"x", + "project_id":"x", + "import_url":"https://mirror-source/group/project.git", + "error_message":"The LFS objects download list couldn't be imported. Error: Unauthorized" +} +``` + ## Registry Logs For Omnibus installations, Container Registry logs reside in `/var/log/gitlab/registry/current`. diff --git a/doc/administration/merge_request_diffs.md b/doc/administration/merge_request_diffs.md index 93cdbff6621..3f4cd6e2751 100644 --- a/doc/administration/merge_request_diffs.md +++ b/doc/administration/merge_request_diffs.md @@ -1,3 +1,10 @@ +--- +stage: Create +group: Editor +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers" +type: reference +--- + # Merge request diffs storage **(CORE ONLY)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/52568) in GitLab 11.8. diff --git a/doc/administration/monitoring/github_imports.md b/doc/administration/monitoring/github_imports.md index 21cc4c708a8..0d79684c951 100644 --- a/doc/administration/monitoring/github_imports.md +++ b/doc/administration/monitoring/github_imports.md @@ -6,8 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Monitoring GitHub imports ->**Note:** -Available since [GitLab 10.2](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/14731). +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/14731) in GitLab 10.2. The GitHub importer exposes various Prometheus metrics that you can use to monitor the health and progress of the importer. diff --git a/doc/administration/monitoring/gitlab_self_monitoring_project/img/self_monitoring_default_dashboard.png b/doc/administration/monitoring/gitlab_self_monitoring_project/img/self_monitoring_default_dashboard.png deleted file mode 100644 index 1d61823ce41..00000000000 Binary files a/doc/administration/monitoring/gitlab_self_monitoring_project/img/self_monitoring_default_dashboard.png and /dev/null differ diff --git a/doc/administration/monitoring/gitlab_self_monitoring_project/img/self_monitoring_overview_dashboard.png b/doc/administration/monitoring/gitlab_self_monitoring_project/img/self_monitoring_overview_dashboard.png new file mode 100644 index 00000000000..1d61823ce41 Binary files /dev/null and b/doc/administration/monitoring/gitlab_self_monitoring_project/img/self_monitoring_overview_dashboard.png differ diff --git a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md index 44ba26296b9..e272cccb7ce 100644 --- a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md +++ b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md @@ -55,10 +55,10 @@ panels, provide a regular expression in the **Instance label regex** field. The dashboard uses metrics available in [Omnibus GitLab](https://docs.gitlab.com/omnibus/) installations. -![GitLab self monitoring default dashboard](img/self_monitoring_default_dashboard.png) +![GitLab self monitoring overview dashboard](img/self_monitoring_overview_dashboard.png) You can also -[create your own dashboards](../../../operations/metrics/dashboards/index.md#defining-custom-dashboards-per-project). +[create your own dashboards](../../../operations/metrics/dashboards/index.md). ## Connection to Prometheus @@ -83,8 +83,8 @@ Once the webhook is setup, you can You can add custom metrics in the self monitoring project by: -1. [Duplicating](../../../operations/metrics/dashboards/index.md#duplicating-a-gitlab-defined-dashboard) the default dashboard. -1. [Editing](../../../operations/metrics/dashboards/index.md#view-and-edit-the-source-file-of-a-custom-dashboard) the newly created dashboard file and configuring it with [dashboard YAML properties](../../../operations/metrics/dashboards/yaml.md). +1. [Duplicating](../../../operations/metrics/dashboards/index.md#duplicate-a-gitlab-defined-dashboard) the overview dashboard. +1. [Editing](../../../operations/metrics/index.md) the newly created dashboard file and configuring it with [dashboard YAML properties](../../../operations/metrics/dashboards/yaml.md). ## Troubleshooting diff --git a/doc/administration/monitoring/performance/grafana_configuration.md b/doc/administration/monitoring/performance/grafana_configuration.md index 96f1377fb73..136a2749e80 100644 --- a/doc/administration/monitoring/performance/grafana_configuration.md +++ b/doc/administration/monitoring/performance/grafana_configuration.md @@ -68,7 +68,7 @@ repository. After setting up Grafana, you can enable a link to access it easily from the GitLab sidebar: -1. Navigate to the **{admin}** **Admin Area > Settings > Metrics and profiling**. +1. Navigate to the **Admin Area > Settings > Metrics and profiling**. 1. Expand **Metrics - Grafana**. 1. Check the **Enable access to Grafana** checkbox. 1. Configure the **Grafana URL**: @@ -77,7 +77,7 @@ GitLab sidebar: - *Otherwise,* enter the full URL of the Grafana instance. 1. Click **Save changes**. -GitLab displays your link in the **{admin}** **Admin Area > Monitoring > Metrics Dashboard**. +GitLab displays your link in the **Admin Area > Monitoring > Metrics Dashboard**. ## Security Update diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md index 8002a9ab296..e247ec3708c 100644 --- a/doc/administration/monitoring/performance/performance_bar.md +++ b/doc/administration/monitoring/performance/performance_bar.md @@ -23,7 +23,7 @@ From left to right, it displays: details: ![Gitaly profiling using the Performance Bar](img/performance_bar_gitaly_calls.png) - **Rugged calls**: the time taken (in milliseconds) and the total number of - [Rugged](../../high_availability/nfs.md#improving-nfs-performance-with-gitlab) calls. + [Rugged](../../nfs.md#improving-nfs-performance-with-gitlab) calls. Click to display a modal window with more details: ![Rugged profiling using the Performance Bar](img/performance_bar_rugged_calls.png) - **Redis calls**: the time taken (in milliseconds) and the total number of @@ -72,8 +72,8 @@ Requests with warnings display `(!)` after their path in the **Request selector* The GitLab Performance Bar is disabled by default. To enable it for a given group: 1. Sign in as a user with Administrator [permissions](../../../user/permissions.md). -1. In the menu bar, click the **{admin}** **Admin Area** icon. -1. Navigate to **{settings}** **Settings > Metrics and profiling** +1. In the menu bar, click **Admin Area**. +1. Navigate to **Settings > Metrics and profiling** (`admin/application_settings/metrics_and_profiling`), and expand the section **Profiling - Performance bar**. 1. Click **Enable access to the Performance Bar**. diff --git a/doc/administration/monitoring/performance/request_profiling.md b/doc/administration/monitoring/performance/request_profiling.md index a3b29493d84..5746b95eb44 100644 --- a/doc/administration/monitoring/performance/request_profiling.md +++ b/doc/administration/monitoring/performance/request_profiling.md @@ -9,8 +9,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w To profile a request: 1. Sign in to GitLab as a user with Administrator or Maintainer [permissions](../../../user/permissions.md). -1. In the navigation bar, click **{admin}** **Admin area**. -1. Navigate to **{monitor}** **Monitoring > Requests Profiles**. +1. In the navigation bar, click **Admin area**. +1. Navigate to **Monitoring > Requests Profiles**. 1. In the **Requests Profiles** section, copy the token. 1. Pass the headers `X-Profile-Token: ` and `X-Profile-Mode: `(where `` can be `execution` or `memory`) to the request you want to profile. When @@ -29,7 +29,7 @@ To profile a request: Profiled requests can take longer than usual. After the request completes, you can view the profiling output from the -**{monitor}** **Monitoring > Requests Profiles** administration page: +**Monitoring > Requests Profiles** administration page: ![Profiling output](img/request_profile_result.png) diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index fa5e5da80c3..bff689c0c0c 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w To enable the GitLab Prometheus metrics: 1. Log into GitLab as a user with [administrator permissions](../../../user/permissions.md). -1. Navigate to **{admin}** **Admin Area > Settings > Metrics and profiling**. +1. Navigate to **Admin Area > Settings > Metrics and profiling**. 1. Find the **Metrics - Prometheus** section, and click **Enable Prometheus Metrics**. 1. [Restart GitLab](../../restart_gitlab.md#omnibus-gitlab-restart) for the changes to take effect. @@ -50,7 +50,8 @@ The following metrics are available: | `gitlab_page_out_of_bounds` | Counter | 12.8 | Counter for the PageLimiter pagination limit being hit | `controller`, `action`, `bot` | | `gitlab_rails_queue_duration_seconds` | Histogram | 9.4 | Measures latency between GitLab Workhorse forwarding a request to Rails | | | `gitlab_sql_duration_seconds` | Histogram | 10.2 | SQL execution time, excluding `SCHEMA` operations and `BEGIN` / `COMMIT` | | -| `gitlab_transaction_allocated_memory_bytes` | Histogram | 10.2 | Allocated memory for all transactions (`gitlab_transaction_*` metrics) | | +| `gitlab_ruby_threads_max_expected_threads` | Gauge | 13.3 | Maximum number of threads expected to be running and performing application work | +| `gitlab_ruby_threads_running_threads` | Gauge | 13.3 | Number of running Ruby threads by name | | `gitlab_transaction_cache__count_total` | Counter | 10.2 | Counter for total Rails cache calls (per key) | | | `gitlab_transaction_cache__duration_total` | Counter | 10.2 | Counter for total time (seconds) spent in Rails cache calls (per key) | | | `gitlab_transaction_cache_count_total` | Counter | 10.2 | Counter for total Rails cache calls (aggregate) | | @@ -95,8 +96,6 @@ The following metrics are available: | `gitlab_transaction_db_count_total` | Counter | 13.1 | Counter for total number of SQL calls | `controller`, `action` | | `gitlab_transaction_db_write_count_total` | Counter | 13.1 | Counter for total number of write SQL calls | `controller`, `action` | | `gitlab_transaction_db_cached_count_total` | Counter | 13.1 | Counter for total number of cached SQL calls | `controller`, `action` | -| `http_redis_requests_duration_seconds` | Histogram | 13.1 | Redis requests duration during web transactions | `controller`, `action` | -| `http_redis_requests_total` | Counter | 13.1 | Redis requests count during web transactions | `controller`, `action` | | `http_elasticsearch_requests_duration_seconds` **(STARTER)** | Histogram | 13.1 | Elasticsearch requests duration during web transactions | `controller`, `action` | | `http_elasticsearch_requests_total` **(STARTER)** | Counter | 13.1 | Elasticsearch requests count during web transactions | `controller`, `action` | | `pipelines_created_total` | Counter | 9.4 | Counter of pipelines created | | diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index f0ad0a1a2e6..7d93e9797be 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -109,6 +109,81 @@ prometheus['scrape_configs'] = [ ] ``` +### Standalone Prometheus using Omnibus GitLab + +The Omnibus GitLab package can be used to configure a standalone Monitoring node running Prometheus and [Grafana](../performance/grafana_configuration.md). + +The steps below are the minimum necessary to configure a Monitoring node running Prometheus and Grafana with Omnibus GitLab: + +1. SSH into the Monitoring node. +1. [Install](https://about.gitlab.com/install/) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page, but + do not follow the remaining steps. +1. Make sure to collect the IP addresses or DNS records of the Consul server nodes, for the next step. +1. Edit `/etc/gitlab/gitlab.rb` and add the contents: + + ```ruby + external_url 'http://gitlab.example.com' + + # Enable Prometheus + prometheus['enable'] = true + prometheus['listen_address'] = '0.0.0.0:9090' + prometheus['monitor_kubernetes'] = false + + # Enable Login form + grafana['disable_login_form'] = false + + # Enable Grafana + grafana['enable'] = true + grafana['admin_password'] = 'toomanysecrets' + + # Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + # The addresses can be IPs or FQDNs + consul['configuration'] = { + retry_join: %w(10.0.0.1 10.0.0.2 10.0.0.3), + } + + # Disable all other services + gitlab_rails['auto_migrate'] = false + alertmanager['enable'] = false + gitaly['enable'] = false + gitlab_exporter['enable'] = false + gitlab_workhorse['enable'] = false + nginx['enable'] = true + postgres_exporter['enable'] = false + postgresql['enable'] = false + redis['enable'] = false + redis_exporter['enable'] = false + sidekiq['enable'] = false + puma['enable'] = false + node_exporter['enable'] = false + gitlab_exporter['enable'] = false + ``` + +1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. + +The next step is to tell all the other nodes where the monitoring node is: + +1. Edit `/etc/gitlab/gitlab.rb`, and add, or find and uncomment the following line: + + ```ruby + gitlab_rails['prometheus_address'] = '10.0.0.1:9090' + ``` + + Where `10.0.0.1:9090` is the IP address and port of the Prometheus node. + +1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to + take effect. + +NOTE: **Note:** +Once monitoring using Service Discovery is enabled with `consul['monitoring_service_discovery'] = true`, +ensure that `prometheus['scrape_configs']` is not set in `/etc/gitlab/gitlab.rb`. Setting both +`consul['monitoring_service_discovery'] = true` and `prometheus['scrape_configs']` in `/etc/gitlab/gitlab.rb` +will result in errors. + ### Using an external Prometheus server NOTE: **Note:** @@ -128,14 +203,24 @@ To use an external Prometheus server: 1. Set each bundled service's [exporter](#bundled-software-metrics) to listen on a network address, for example: ```ruby + node_exporter['listen_address'] = '0.0.0.0:9100' + gitlab_workhorse['prometheus_listen_addr'] = "0.0.0.0:9229" + + # Rails nodes gitlab_exporter['listen_address'] = '0.0.0.0' - sidekiq['listen_address'] = '0.0.0.0' gitlab_exporter['listen_port'] = '9168' - node_exporter['listen_address'] = '0.0.0.0:9100' + + # Sidekiq nodes + sidekiq['listen_address'] = '0.0.0.0' + + # Redis nodes redis_exporter['listen_address'] = '0.0.0.0:9121' + + # PostgreSQL nodes postgres_exporter['listen_address'] = '0.0.0.0:9187' + + # Gitaly nodes gitaly['prometheus_listen_addr'] = "0.0.0.0:9236" - gitlab_workhorse['prometheus_listen_addr'] = "0.0.0.0:9229" ``` 1. Install and set up a dedicated Prometheus instance, if necessary, using the [official installation instructions](https://prometheus.io/docs/prometheus/latest/installation/). @@ -227,7 +312,7 @@ To use an external Prometheus server: You can visit `http://localhost:9090` for the dashboard that Prometheus offers by default. ->**Note:** +NOTE: **Note:** If SSL has been enabled on your GitLab instance, you may not be able to access Prometheus on the same browser as GitLab if using the same FQDN due to [HSTS](https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security). We plan to [provide access via GitLab](https://gitlab.com/gitlab-org/multi-user-prometheus), but in the interim there are diff --git a/doc/administration/nfs.md b/doc/administration/nfs.md new file mode 100644 index 00000000000..bae6bd0dd6c --- /dev/null +++ b/doc/administration/nfs.md @@ -0,0 +1,368 @@ +--- +type: reference +--- + +# Using NFS with GitLab + +NFS can be used as an alternative for object storage but this isn't typically +recommended for performance reasons. Note however it is required for [GitLab +Pages](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/196). + +For data objects such as LFS, Uploads, Artifacts, etc., an [Object Storage service](object_storage.md) +is recommended over NFS where possible, due to better performance. + +CAUTION: **Caution:** +From GitLab 13.0, using NFS for Git repositories is deprecated. In GitLab 14.0, +support for NFS for Git repositories is scheduled to be removed. Upgrade to +[Gitaly Cluster](gitaly/praefect.md) as soon as possible. + +NOTE: **Note:** +Filesystem performance has a big impact on overall GitLab +performance, especially for actions that read or write to Git repositories. See +[Filesystem Performance Benchmarking](operations/filesystem_benchmarking.md) +for steps to test filesystem performance. + +## Known kernel version incompatibilities + +RedHat Enterprise Linux (RHEL) and CentOS v7.7 and v7.8 ship with kernel +version `3.10.0-1127`, which [contains a +bug](https://bugzilla.redhat.com/show_bug.cgi?id=1783554) that causes +[uploads to fail to copy over NFS](https://gitlab.com/gitlab-org/gitlab/-/issues/218999). The +following GitLab versions include a fix to work properly with that +kernel version: + +1. [12.10.12](https://about.gitlab.com/releases/2020/06/25/gitlab-12-10-12-released/) +1. [13.0.7](https://about.gitlab.com/releases/2020/06/25/gitlab-13-0-7-released/) +1. [13.1.1](https://about.gitlab.com/releases/2020/06/24/gitlab-13-1-1-released/) +1. 13.2 and up + +If you are using that kernel version, be sure to upgrade GitLab to avoid +errors. + +## Fast lookup of authorized SSH keys + +The [fast SSH key lookup](operations/fast_ssh_key_lookup.md) feature can improve +performance of GitLab instances even if they're using block storage. + +[Fast SSH key lookup](operations/fast_ssh_key_lookup.md) is a replacement for +`authorized_keys` (in `/var/opt/gitlab/.ssh`) using the GitLab database. + +NFS increases latency, so fast lookup is recommended if `/var/opt/gitlab` +is moved to NFS. + +We are investigating the use of +[fast lookup as the default](https://gitlab.com/groups/gitlab-org/-/epics/3104). + +## NFS server + +Installing the `nfs-kernel-server` package allows you to share directories with +the clients running the GitLab application: + +```shell +sudo apt-get update +sudo apt-get install nfs-kernel-server +``` + +### Required features + +**File locking**: GitLab **requires** advisory file locking, which is only +supported natively in NFS version 4. NFSv3 also supports locking as long as +Linux Kernel 2.6.5+ is used. We recommend using version 4 and do not +specifically test NFSv3. + +### Recommended options + +When you define your NFS exports, we recommend you also add the following +options: + +- `no_root_squash` - NFS normally changes the `root` user to `nobody`. This is + a good security measure when NFS shares will be accessed by many different + users. However, in this case only GitLab will use the NFS share so it + is safe. GitLab recommends the `no_root_squash` setting because we need to + manage file permissions automatically. Without the setting you may receive + errors when the Omnibus package tries to alter permissions. Note that GitLab + and other bundled components do **not** run as `root` but as non-privileged + users. The recommendation for `no_root_squash` is to allow the Omnibus package + to set ownership and permissions on files, as needed. In some cases where the + `no_root_squash` option is not available, the `root` flag can achieve the same + result. +- `sync` - Force synchronous behavior. Default is asynchronous and under certain + circumstances it could lead to data loss if a failure occurs before data has + synced. + +Due to the complexities of running Omnibus with LDAP and the complexities of +maintaining ID mapping without LDAP, in most cases you should enable numeric UIDs +and GIDs (which is off by default in some cases) for simplified permission +management between systems: + +- [NetApp instructions](https://library.netapp.com/ecmdocs/ECMP1401220/html/GUID-24367A9F-E17B-4725-ADC1-02D86F56F78E.html) +- For non-NetApp devices, disable NFSv4 `idmapping` by performing opposite of [enable NFSv4 idmapper](https://wiki.archlinux.org/index.php/NFS#Enabling_NFSv4_idmapping) + +### Disable NFS server delegation + +We recommend that all NFS users disable the NFS server delegation feature. This +is to avoid a [Linux kernel bug](https://bugzilla.redhat.com/show_bug.cgi?id=1552203) +which causes NFS clients to slow precipitously due to +[excessive network traffic from numerous `TEST_STATEID` NFS messages](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/52017). + +To disable NFS server delegation, do the following: + +1. On the NFS server, run: + + ```shell + echo 0 > /proc/sys/fs/leases-enable + sysctl -w fs.leases-enable=0 + ``` + +1. Restart the NFS server process. For example, on CentOS run `service nfs restart`. + +NOTE: **Important note:** +The kernel bug may be fixed in +[more recent kernels with this commit](https://github.com/torvalds/linux/commit/95da1b3a5aded124dd1bda1e3cdb876184813140). +Red Hat Enterprise 7 [shipped a kernel update](https://access.redhat.com/errata/RHSA-2019:2029) +on August 6, 2019 that may also have resolved this problem. +You may not need to disable NFS server delegation if you know you are using a version of +the Linux kernel that has been fixed. That said, GitLab still encourages instance +administrators to keep NFS server delegation disabled. + +### Improving NFS performance with GitLab + +#### Improving NFS performance with Unicorn + +NOTE: **Note:** +From GitLab 12.1, it will automatically be detected if Rugged can and should be used per storage. + +If you previously enabled Rugged using the feature flag, you will need to unset the feature flag by using: + +```shell +sudo gitlab-rake gitlab:features:unset_rugged +``` + +If the Rugged feature flag is explicitly set to either true or false, GitLab will use the value explicitly set. + +#### Improving NFS performance with Puma + +NOTE: **Note:** +From GitLab 12.7, Rugged auto-detection is disabled if Puma thread count is greater than 1. + +If you want to use Rugged with Puma, it is recommended to [set Puma thread count to 1](https://docs.gitlab.com/omnibus/settings/puma.html#puma-settings). + +If you want to use Rugged with Puma thread count more than 1, Rugged can be enabled using the [feature flag](../development/gitaly.md#legacy-rugged-code) + +If the Rugged feature flag is explicitly set to either true or false, GitLab will use the value explicitly set. + +## NFS client + +The `nfs-common` provides NFS functionality without installing server components which +we don't need running on the application nodes. + +```shell +apt-get update +apt-get install nfs-common +``` + +### Mount options + +Here is an example snippet to add to `/etc/fstab`: + +```plaintext +10.1.0.1:/var/opt/gitlab/.ssh /var/opt/gitlab/.ssh nfs4 defaults,vers=4.1,hard,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 +10.1.0.1:/var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads nfs4 defaults,vers=4.1,hard,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 +10.1.0.1:/var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared nfs4 defaults,vers=4.1,hard,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 +10.1.0.1:/var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/gitlab-ci/builds nfs4 defaults,vers=4.1,hard,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 +10.1.0.1:/var/opt/gitlab/git-data /var/opt/gitlab/git-data nfs4 defaults,vers=4.1,hard,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 +``` + +Note there are several options that you should consider using: + +| Setting | Description | +| ------- | ----------- | +| `vers=4.1` |NFS v4.1 should be used instead of v4.0 because there is a Linux [NFS client bug in v4.0](https://gitlab.com/gitlab-org/gitaly/-/issues/1339) that can cause significant problems due to stale data. +| `nofail` | Don't halt boot process waiting for this mount to become available +| `lookupcache=positive` | Tells the NFS client to honor `positive` cache results but invalidates any `negative` cache results. Negative cache results cause problems with Git. Specifically, a `git push` can fail to register uniformly across all NFS clients. The negative cache causes the clients to 'remember' that the files did not exist previously. +| `hard` | Instead of `soft`. [Further details](#soft-mount-option). + +#### `soft` mount option + +It's recommended that you use `hard` in your mount options, unless you have a specific +reason to use `soft`. + +On GitLab.com, we use `soft` because there were times when we had NFS servers +reboot and `soft` improved availability, but everyone's infrastructure is different. +If your NFS is provided by on-premise storage arrays with redundant controllers, +for example, you shouldn't need to worry about NFS server availability. + +The NFS man page states: + +> "soft" timeout can cause silent data corruption in certain cases + +Read the [Linux man page](https://linux.die.net/man/5/nfs) to understand the difference, +and if you do use `soft`, ensure that you've taken steps to mitigate the risks. + +If you experience behavior that might have been caused by +writes to disk on the NFS server not occurring, such as commits going missing, +use the `hard` option, because (from the man page): + +> use the soft option only when client responsiveness is more important than data integrity + +Other vendors make similar recommendations, including +[SAP](http://wiki.scn.sap.com/wiki/x/PARnFQ) and NetApp's +[knowledge base](https://kb.netapp.com/Advice_and_Troubleshooting/Data_Storage_Software/ONTAP_OS/What_are_the_differences_between_hard_mount_and_soft_mount), +they highlight that if the NFS client driver caches data, `soft` means there is no certainty if +writes by GitLab are actually on disk. + +Mount points set with the option `hard` may not perform as well, and if the +NFS server goes down, `hard` will cause processes to hang when interacting with +the mount point. Use `SIGKILL` (`kill -9`) to deal with hung processes. +The `intr` option +[stopped working in the 2.6 kernel](https://access.redhat.com/solutions/157873). + +### A single NFS mount + +It's recommended to nest all GitLab data directories within a mount, that allows automatic +restore of backups without manually moving existing data. + +```plaintext +mountpoint +└── gitlab-data + ├── builds + ├── git-data + ├── shared + └── uploads +``` + +To do so, we'll need to configure Omnibus with the paths to each directory nested +in the mount point as follows: + +Mount `/gitlab-nfs` then use the following Omnibus +configuration to move each data location to a subdirectory: + +```ruby +git_data_dirs({"default" => { "path" => "/gitlab-nfs/gitlab-data/git-data"} }) +gitlab_rails['uploads_directory'] = '/gitlab-nfs/gitlab-data/uploads' +gitlab_rails['shared_path'] = '/gitlab-nfs/gitlab-data/shared' +gitlab_ci['builds_directory'] = '/gitlab-nfs/gitlab-data/builds' +``` + +Run `sudo gitlab-ctl reconfigure` to start using the central location. Please +be aware that if you had existing data you will need to manually copy/rsync it +to these new locations and then restart GitLab. + +### Bind mounts + +Alternatively to changing the configuration in Omnibus, bind mounts can be used +to store the data on an NFS mount. + +Bind mounts provide a way to specify just one NFS mount and then +bind the default GitLab data locations to the NFS mount. Start by defining your +single NFS mount point as you normally would in `/etc/fstab`. Let's assume your +NFS mount point is `/gitlab-nfs`. Then, add the following bind mounts in +`/etc/fstab`: + +```shell +/gitlab-nfs/gitlab-data/git-data /var/opt/gitlab/git-data none bind 0 0 +/gitlab-nfs/gitlab-data/.ssh /var/opt/gitlab/.ssh none bind 0 0 +/gitlab-nfs/gitlab-data/uploads /var/opt/gitlab/gitlab-rails/uploads none bind 0 0 +/gitlab-nfs/gitlab-data/shared /var/opt/gitlab/gitlab-rails/shared none bind 0 0 +/gitlab-nfs/gitlab-data/builds /var/opt/gitlab/gitlab-ci/builds none bind 0 0 +``` + +Using bind mounts will require manually making sure the data directories +are empty before attempting a restore. Read more about the +[restore prerequisites](../raketasks/backup_restore.md). + +You can view information and options set for each of the mounted NFS file +systems by running `nfsstat -m` and `cat /etc/fstab`. + +### Multiple NFS mounts + +When using default Omnibus configuration you will need to share 4 data locations +between all GitLab cluster nodes. No other locations should be shared. The +following are the 4 locations need to be shared: + +| Location | Description | Default configuration | +| -------- | ----------- | --------------------- | +| `/var/opt/gitlab/git-data` | Git repository data. This will account for a large portion of your data | `git_data_dirs({"default" => { "path" => "/var/opt/gitlab/git-data"} })` +| `/var/opt/gitlab/gitlab-rails/uploads` | User uploaded attachments | `gitlab_rails['uploads_directory'] = '/var/opt/gitlab/gitlab-rails/uploads'` +| `/var/opt/gitlab/gitlab-rails/shared` | Build artifacts, GitLab Pages, LFS objects, temp files, etc. If you're using LFS this may also account for a large portion of your data | `gitlab_rails['shared_path'] = '/var/opt/gitlab/gitlab-rails/shared'` +| `/var/opt/gitlab/gitlab-ci/builds` | GitLab CI/CD build traces | `gitlab_ci['builds_directory'] = '/var/opt/gitlab/gitlab-ci/builds'` + +Other GitLab directories should not be shared between nodes. They contain +node-specific files and GitLab code that does not need to be shared. To ship +logs to a central location consider using remote syslog. Omnibus GitLab packages +provide configuration for [UDP log shipping](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-shipping-gitlab-enterprise-edition-only). + +Having multiple NFS mounts will require manually making sure the data directories +are empty before attempting a restore. Read more about the +[restore prerequisites](../raketasks/backup_restore.md). + +## NFS in a Firewalled Environment + +If the traffic between your NFS server and NFS client(s) is subject to port filtering +by a firewall, then you will need to reconfigure that firewall to allow NFS communication. + +[This guide from TDLP](http://tldp.org/HOWTO/NFS-HOWTO/security.html#FIREWALLS) +covers the basics of using NFS in a firewalled environment. Additionally, we encourage you to +search for and review the specific documentation for your operating system or distribution and your firewall software. + +Example for Ubuntu: + +Check that NFS traffic from the client is allowed by the firewall on the host by running +the command: `sudo ufw status`. If it's being blocked, then you can allow traffic from a specific +client with the command below. + +```shell +sudo ufw allow from to any port nfs +``` + +## Known issues + +### Avoid using AWS's Elastic File System (EFS) + +GitLab strongly recommends against using AWS Elastic File System (EFS). +Our support team will not be able to assist on performance issues related to +file system access. + +Customers and users have reported that AWS EFS does not perform well for GitLab's +use-case. Workloads where many small files are written in a serialized manner, like `git`, +are not well-suited for EFS. EBS with an NFS server on top will perform much better. + +If you do choose to use EFS, avoid storing GitLab log files (e.g. those in `/var/log/gitlab`) +there because this will also affect performance. We recommend that the log files be +stored on a local volume. + +For more details on another person's experience with EFS, see this [Commit Brooklyn 2019 video](https://youtu.be/K6OS8WodRBQ?t=313). + +### Avoid using CephFS and GlusterFS + +GitLab strongly recommends against using CephFS and GlusterFS. +These distributed file systems are not well-suited for GitLab's input/output access patterns because Git uses many small files and access times and file locking times to propagate will make Git activity very slow. + +### Avoid using PostgreSQL with NFS + +GitLab strongly recommends against running your PostgreSQL database +across NFS. The GitLab support team will not be able to assist on performance issues related to +this configuration. + +Additionally, this configuration is specifically warned against in the +[PostgreSQL Documentation](https://www.postgresql.org/docs/current/creating-cluster.html#CREATING-CLUSTER-NFS): + +>PostgreSQL does nothing special for NFS file systems, meaning it assumes NFS behaves exactly like +>locally-connected drives. If the client or server NFS implementation does not provide standard file +>system semantics, this can cause reliability problems. Specifically, delayed (asynchronous) writes +>to the NFS server can cause data corruption problems. + +For supported database architecture, please see our documentation on +[Configuring a Database for GitLab HA](postgresql/replication_and_failover.md). + + diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md index b51b722fbd7..49716883310 100644 --- a/doc/administration/object_storage.md +++ b/doc/administration/object_storage.md @@ -15,12 +15,18 @@ GitLab has been tested on a number of object storage providers: - [Amazon S3](https://aws.amazon.com/s3/) - [Google Cloud Storage](https://cloud.google.com/storage) -- [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces) +- [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces/) - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) - [Openstack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) - On-premises hardware and appliances from various storage vendors. - MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. +### Known compatibility issues + +- Dell EMC ECS: Prior to GitLab 13.3, there is a [known bug in GitLab Workhorse that prevents + HTTP Range Requests from working with CI job artifacts](https://gitlab.com/gitlab-org/gitlab/-/issues/223806). + Be sure to upgrade to GitLab v13.3.0 or above if you use S3 storage with this hardware. + ## Configuration guides There are two ways of specifying object storage configuration in GitLab: @@ -55,6 +61,10 @@ NOTE: **Note:** Consolidated object storage configuration cannot be used for backups or Mattermost. See [the full table for a complete list](#storage-specific-configuration). +NOTE: **Note:** +Enabling consolidated object storage will enable object storage for all object types. +If you wish to use local storage for specific object types, you can [selectively disable object storages](#selectively-disabling-object-storage). + Most types of objects, such as CI artifacts, LFS files, upload attachments, and so on can be saved in object storage by specifying a single credential for object storage with multiple buckets. A [different bucket @@ -84,6 +94,11 @@ See the section on [ETag mismatch errors](#etag-mismatch) for more details. 'aws_access_key_id' => '', 'aws_secret_access_key' => '' } + # OPTIONAL: The following lines are only needed if server side encryption is required + gitlab_rails['object_store']['storage_options'] = { + 'server_side_encryption' => '', + 'server_side_encryption_kms_key_id' => '' + } gitlab_rails['object_store']['objects']['artifacts']['bucket'] = '' gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = '' gitlab_rails['object_store']['objects']['lfs']['bucket'] = '' @@ -119,6 +134,9 @@ See the section on [ETag mismatch errors](#etag-mismatch) for more details. aws_access_key_id: aws_secret_access_key: region: + storage_options: + server_side_encryption: + server_side_encryption_key_kms_id: objects: artifacts: bucket: @@ -184,7 +202,8 @@ gitlab_rails['object_store']['connection'] = { |---------|-------------| | `enabled` | Enable/disable object storage | | `proxy_download` | Set to `true` to [enable proxying all files served](#proxy-download). Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | -| `connection` | Various connection options described below | +| `connection` | Various [connection options](#connection-settings) described below | +| `storage_options` | Options to use when saving new objects, such as [server side encryption](#server-side-encryption-headers). Introduced in GitLab 13.3 | | `objects` | [Object-specific configuration](#object-specific-configuration) ### Connection settings @@ -494,16 +513,18 @@ If you configure GitLab to use object storage for CI logs and artifacts, ### Proxy Download -A number of the use cases for object storage allow client traffic to be redirected to the -object storage back end, like when Git clients request large files via LFS or when -downloading CI artifacts and logs. +Clients can download files in object storage by receiving a pre-signed, time-limited URL, +or by GitLab proxying the data from object storage to the client. +Downloading files from object storage directly +helps reduce the amount of egress traffic GitLab +needs to process. When the files are stored on local block storage or NFS, GitLab has to act as a proxy. This is not the default behavior with object storage. The `proxy_download` setting controls this behavior: the default is generally `false`. -Verify this in the documentation for each use case. Set it to `true` so that GitLab proxies -the files. +Verify this in the documentation for each use case. Set it to `true` if you want +GitLab to proxy the files. When not proxying files, GitLab returns an [HTTP 302 redirect with a pre-signed, time-limited object storage URL](https://gitlab.com/gitlab-org/gitlab/-/issues/32117#note_218532298). @@ -524,7 +545,9 @@ certificate, or may return common TLS errors such as: x509: certificate signed by unknown authority ``` -- Clients will need network access to the object storage. Errors that might result +- Clients will need network access to the object storage. +Network firewalls could block access. +Errors that might result if this access is not in place include: ```plaintext @@ -535,6 +558,10 @@ Getting a `403 Forbidden` response is specifically called out on the [package repository documentation](packages/index.md#using-object-storage) as a side effect of how some build tools work. +Additionally for a short time period users could share pre-signed, time-limited object storage URLs +with others without authentication. Also bandwidth charges may be incurred +between the object storage provider and the client. + ### ETag mismatch Using the default GitLab settings, some object storage back-ends such as @@ -576,21 +603,46 @@ configuration. #### Encrypted S3 buckets -> - Introduced in [GitLab 13.1](https://gitlab.com/gitlab-org/gitlab-workhorse/-/merge_requests/466) for instance profiles only. -> - Introduced in [GitLab 13.2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/34460) for static credentials when [consolidated object storage configuration](#consolidated-object-storage-configuration) is used. +> - Introduced in [GitLab 13.1](https://gitlab.com/gitlab-org/gitlab-workhorse/-/merge_requests/466) for instance profiles only and [S3 default encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/bucket-encryption.html). +> - Introduced in [GitLab 13.2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/34460) for static credentials when [consolidated object storage configuration](#consolidated-object-storage-configuration) and [S3 default encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/bucket-encryption.html) are used. When configured either with an instance profile or with the consolidated -object configuration, GitLab Workhorse properly uploads files to S3 buckets -that have [SSE-S3 or SSE-KMS encryption enabled by +object configuration, GitLab Workhorse properly uploads files to S3 +buckets that have [SSE-S3 or SSE-KMS encryption enabled by default](https://docs.aws.amazon.com/kms/latest/developerguide/services-s3.html). -Note that customer master keys (CMKs) and -SSE-C encryption are [not yet supported since this requires supplying -keys to the GitLab configuration](https://gitlab.com/gitlab-org/gitlab/-/issues/226006). +Note that customer master keys (CMKs) and SSE-C encryption are [not +supported since this requires sending the encryption keys in every request](https://gitlab.com/gitlab-org/gitlab/-/issues/226006). + +##### Server-side encryption headers + +> Introduced in [GitLab 13.3](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38240). + +Setting a default encryption on an S3 bucket is the easiest way to +enable encryption, but you may want to [set a bucket policy to ensure +only encrypted objects are uploaded](https://aws.amazon.com/premiumsupport/knowledge-center/s3-bucket-store-kms-encrypted-objects/). +To do this, you must configure GitLab to send the proper encryption headers +in the `storage_options` configuration section: + +| Setting | Description | +|-------------------------------------|-------------| +| `server_side_encryption` | Encryption mode (AES256 or aws:kms) | +| `server_side_encryption_kms_key_id` | Amazon Resource Name. Only needed when `aws:kms` is used in `server_side_encryption`. See the [Amazon documentation on using KMS encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html) | + +As with the case for default encryption, these options only work when +the Workhorse S3 client is enabled. One of the following two conditions +must be fulfilled: + +- `use_iam_profile` is `true` in the connection settings. +- Consolidated object storage settings are in use. + +[ETag mismatch errors](#etag-mismatch) will occur if server side +encryption headers are used without enabling the Workhorse S3 client. ##### Disabling the feature The Workhorse S3 client is enabled by default when the -[`use_iam_profile` configuration option](#iam-permissions) is set to `true`. +[`use_iam_profile` configuration option](#iam-permissions) is set to `true` or consolidated +object storage settings are configured. The feature can be disabled using the `:use_workhorse_s3_client` feature flag. To disable the feature, ask a GitLab administrator with diff --git a/doc/administration/operations/cleaning_up_redis_sessions.md b/doc/administration/operations/cleaning_up_redis_sessions.md index 38fac8a0eca..d2aec2f7c47 100644 --- a/doc/administration/operations/cleaning_up_redis_sessions.md +++ b/doc/administration/operations/cleaning_up_redis_sessions.md @@ -15,7 +15,8 @@ prefixed with `session:gitlab:`, so they would look like `session:gitlab:976aa289e2189b17d7ef525a6702ace9`. Below we describe how to remove the keys in the old format. -**Note:** the instructions below must be modified in accordance with your +NOTE: **Note:** +The instructions below must be modified in accordance with your configuration settings if you have used the advanced Redis settings outlined in [Configuration Files Documentation](https://gitlab.com/gitlab-org/gitlab/blob/master/config/README.md). diff --git a/doc/administration/operations/extra_sidekiq_processes.md b/doc/administration/operations/extra_sidekiq_processes.md index 155680354da..e589ecc4216 100644 --- a/doc/administration/operations/extra_sidekiq_processes.md +++ b/doc/administration/operations/extra_sidekiq_processes.md @@ -82,7 +82,7 @@ To start multiple processes: ``` After the extra Sidekiq processes are added, navigate to -**{admin}** **Admin Area > Monitoring > Background Jobs** (`/admin/background_jobs`) in GitLab. +**Admin Area > Monitoring > Background Jobs** (`/admin/background_jobs`) in GitLab. ![Multiple Sidekiq processes](img/sidekiq-cluster.png) diff --git a/doc/administration/operations/filesystem_benchmarking.md b/doc/administration/operations/filesystem_benchmarking.md index 856061348ed..64afd1b44f3 100644 --- a/doc/administration/operations/filesystem_benchmarking.md +++ b/doc/administration/operations/filesystem_benchmarking.md @@ -26,7 +26,7 @@ To install: Then run the following: ```shell -fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=/path/to/git-data/testfile --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75 +fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --bs=4k --iodepth=64 --readwrite=randrw --rwmixread=75 --size=4G --filename=/path/to/git-data/testfile ``` This will create a 4GB file in `/path/to/git-data/testfile`. It performs diff --git a/doc/administration/operations/moving_repositories.md b/doc/administration/operations/moving_repositories.md index 960005fe25d..4763c012538 100644 --- a/doc/administration/operations/moving_repositories.md +++ b/doc/administration/operations/moving_repositories.md @@ -9,16 +9,17 @@ We will look at three scenarios: the target directory is empty, the target directory contains an outdated copy of the repositories, and how to deal with thousands of repositories. -**Each of the approaches we list can/will overwrite data in the +DANGER: **Danger:** +Each of the approaches we list can/will overwrite data in the target directory `/mnt/gitlab/repositories`. Do not mix up the -source and the target.** +source and the target. -## Target directory is empty: use a tar pipe +## Target directory is empty: use a `tar` pipe If the target directory `/mnt/gitlab/repositories` is empty the -simplest thing to do is to use a tar pipe. This method has low -overhead and tar is almost always already installed on your system. -However, it is not possible to resume an interrupted tar pipe: if +simplest thing to do is to use a `tar` pipe. This method has low +overhead and `tar` is almost always already installed on your system. +However, it is not possible to resume an interrupted `tar` pipe: if that happens then all data must be copied again. ```shell @@ -28,9 +29,9 @@ sudo -u git sh -c 'tar -C /var/opt/gitlab/git-data/repositories -cf - -- . |\ If you want to see progress, replace `-xf` with `-xvf`. -### Tar pipe to another server +### `tar` pipe to another server -You can also use a tar pipe to copy data to another server. If your +You can also use a `tar` pipe to copy data to another server. If your `git` user has SSH access to the new server as `git@newserver`, you can pipe the data through SSH. @@ -42,13 +43,13 @@ sudo -u git sh -c 'tar -C /var/opt/gitlab/git-data/repositories -cf - -- . |\ If you want to compress the data before it goes over the network (which will cost you CPU cycles) you can replace `ssh` with `ssh -C`. -## The target directory contains an outdated copy of the repositories: use rsync +## The target directory contains an outdated copy of the repositories: use `rsync` If the target directory already contains a partial / outdated copy of the repositories it may be wasteful to copy all the data again -with tar. In this scenario it is better to use rsync. This utility +with `tar`. In this scenario it is better to use `rsync`. This utility is either already installed on your system or easily installable -via apt, yum etc. +via `apt`, `yum`, etc. ```shell sudo -u git sh -c 'rsync -a --delete /var/opt/gitlab/git-data/repositories/. \ @@ -59,30 +60,30 @@ The `/.` in the command above is very important, without it you can easily get the wrong directory structure in the target directory. If you want to see progress, replace `-a` with `-av`. -### Single rsync to another server +### Single `rsync` to another server If the `git` user on your source system has SSH access to the target -server you can send the repositories over the network with rsync. +server you can send the repositories over the network with `rsync`. ```shell sudo -u git sh -c 'rsync -a --delete /var/opt/gitlab/git-data/repositories/. \ git@newserver:/mnt/gitlab/repositories' ``` -## Thousands of Git repositories: use one rsync per repository +## Thousands of Git repositories: use one `rsync` per repository -Every time you start an rsync job it has to inspect all files in +Every time you start an `rsync` job it has to inspect all files in the source directory, all files in the target directory, and then decide what files to copy or not. If the source or target directory -has many contents this startup phase of rsync can become a burden -for your GitLab server. In cases like this you can make rsync's +has many contents this startup phase of `rsync` can become a burden +for your GitLab server. In cases like this you can make `rsync`'s life easier by dividing its work in smaller pieces, and sync one repository at a time. -In addition to rsync we will use [GNU +In addition to `rsync` we will use [GNU Parallel](http://www.gnu.org/software/parallel/). This utility is -not included in GitLab so you need to install it yourself with apt -or yum. Also note that the GitLab scripts we used below were added +not included in GitLab so you need to install it yourself with `apt` +or `yum`. Also note that the GitLab scripts we used below were added in GitLab 8.1. **This process does not clean up repositories at the target location that no @@ -90,9 +91,9 @@ longer exist at the source.** If you start using your GitLab instance with `/mnt/gitlab/repositories`, you need to run `gitlab-rake gitlab:cleanup:repos` after switching to the new repository storage directory. -### Parallel rsync for all repositories known to GitLab +### Parallel `rsync` for all repositories known to GitLab -This will sync repositories with 10 rsync processes at a time. We keep +This will sync repositories with 10 `rsync` processes at a time. We keep track of progress so that the transfer can be restarted if necessary. First we create a new directory, owned by `git`, to hold transfer @@ -147,7 +148,7 @@ cat /home/git/transfer-logs/* | sort | uniq -u |\ ` ``` -### Parallel rsync only for repositories with recent activity +### Parallel `rsync` only for repositories with recent activity Suppose you have already done one sync that started after 2015-10-1 12:00 UTC. Then you might only want to sync repositories that were changed via GitLab diff --git a/doc/administration/operations/puma.md b/doc/administration/operations/puma.md index 62b93d40a6b..e7b4bb88faf 100644 --- a/doc/administration/operations/puma.md +++ b/doc/administration/operations/puma.md @@ -27,7 +27,7 @@ will _not_ carry over automatically, due to differences between the two applicat deployments, see [Configuring Puma Settings](https://docs.gitlab.com/omnibus/settings/puma.html#configuring-puma-settings). For Helm based deployments, see the [Webservice Chart documentation](https://docs.gitlab.com/charts/charts/gitlab/webservice/index.html). -Additionally we strongly recommend that multi-node deployments [configure their load balancers to utilize the readiness check](../high_availability/load_balancer.md#readiness-check) due to a difference between Unicorn and Puma in how they handle connections during a restart of the service. +Additionally we strongly recommend that multi-node deployments [configure their load balancers to utilize the readiness check](../load_balancer.md#readiness-check) due to a difference between Unicorn and Puma in how they handle connections during a restart of the service. ## Performance caveat when using Puma with Rugged diff --git a/doc/administration/operations/sidekiq_memory_killer.md b/doc/administration/operations/sidekiq_memory_killer.md index fdccfacc8a9..e829d735c4f 100644 --- a/doc/administration/operations/sidekiq_memory_killer.md +++ b/doc/administration/operations/sidekiq_memory_killer.md @@ -71,5 +71,5 @@ The MemoryKiller is controlled using environment variables. If the process hard shutdown/restart is not performed by Sidekiq, the Sidekiq process will be forcefully terminated after - `Sidekiq.options[:timeout] * 2` seconds. An external supervision mechanism + `Sidekiq.options[:timeout] + 2` seconds. An external supervision mechanism (e.g. runit) must restart Sidekiq afterwards. diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md index 44f1d075a5e..1883f6659e6 100644 --- a/doc/administration/packages/container_registry.md +++ b/doc/administration/packages/container_registry.md @@ -331,6 +331,9 @@ The different supported drivers are: | swift | OpenStack Swift Object Storage | | oss | Aliyun OSS | +NOTE: **Note:** +Although most S3 compatible services (like [MinIO](https://min.io/)) should work with the registry, we only guarantee support for AWS S3. Because we cannot assert the correctness of third-party S3 implementations, we can debug issues, but we cannot patch the registry unless an issue is reproducible against an AWS S3 bucket. + Read more about the individual driver's configuration options in the [Docker Registry docs](https://docs.docker.com/registry/configuration/#storage). @@ -446,27 +449,62 @@ to be in read-only mode for a while. During this time, you can pull from the Container Registry, but you cannot push. 1. Optional: To reduce the amount of data to be migrated, run the [garbage collection tool without downtime](#performing-garbage-collection-without-downtime). -1. Copy initial data to your S3 bucket, for example with the AWS CLI [`cp`](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/cp.html) +1. This example uses the `aws` CLI. If you haven't configured the + CLI before, you have to configure your credentials by running `sudo aws configure`. + Because a non-admin user likely can't access the Container Registry folder, + ensure you use `sudo`. To check your credential configuration, run + [`ls`](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/ls.html) to list + all buckets. + + ```shell + sudo aws --endpoint-url https://your-object-storage-backend.com s3 ls + ``` + + If you are using AWS as your back end, you do not need the [`--endpoint-url`](https://docs.aws.amazon.com/cli/latest/reference/#options). +1. Copy initial data to your S3 bucket, for example with the `aws` CLI + [`cp`](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/cp.html) or [`sync`](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/sync.html) command. Make sure to keep the `docker` folder as the top-level folder inside the bucket. ```shell - aws s3 sync registry s3://mybucket + sudo aws --endpoint-url https://your-object-storage-backend.com s3 sync registry s3://mybucket ``` + TIP: **Tip:** + If you have a lot of data, you may be able to improve performance by + [running parallel sync operations](https://aws.amazon.com/premiumsupport/knowledge-center/s3-improve-transfer-sync-command/). + 1. To perform the final data sync, [put the Container Registry in `read-only` mode](#performing-garbage-collection-without-downtime) and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). 1. Sync any changes since the initial data load to your S3 bucket and delete files that exist in the destination bucket but not in the source: ```shell - aws s3 sync registry s3://mybucket --delete + sudo aws --endpoint-url https://your-object-storage-backend.com s3 sync registry s3://mybucket --delete --dryrun ``` + After verifying the command is going to perform as expected, remove the + [`--dryrun`](https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html) + flag and run the command. + DANGER: **Danger:** - The `--delete` flag will delete files that exist in the destination but not in the source. + The [`--delete`](https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html) + flag will delete files that exist in the destination but not in the source. Make sure not to swap the source and destination, or you will delete all data in the Registry. +1. Verify all Container Registry files have been uploaded to object storage + by looking at the file count returned by these two commands: + + ```shell + sudo find registry -type f | wc -l + ``` + + ```shell + sudo aws --endpoint-url https://your-object-storage-backend.com s3 ls s3://mybucket --recursive | wc -l + ``` + + The output of these commands should match, except for the content in the + `_uploads` directories and sub-directories. 1. Configure your registry to [use the S3 bucket for storage](#use-object-storage). 1. For the changes to take effect, set the Registry back to `read-write` mode and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -693,6 +731,35 @@ notifications: backoff: 1000 ``` +## Run the Cleanup policy now + +To reduce the amount of [Container Registry disk space used by a given project](../troubleshooting/gitlab_rails_cheat_sheet.md#registry-disk-space-usage-by-project), +administrators can clean up image tags +and [run garbage collection](#container-registry-garbage-collection). + +To remove image tags by running the cleanup policy, run the following commands in the +[GitLab Rails console](../troubleshooting/navigating_gitlab_via_rails_console.md): + +```ruby +# Numeric ID of the project whose container registry should be cleaned up +P = + +# Numeric ID of a developer, maintainer or owner in that project +U = + +# Get required details / objects +user = User.find_by_id(U) +project = Project.find_by_id(P) +repo = ContainerRepository.find_by(project_id: P) +policy = ContainerExpirationPolicy.find_by(project_id: P) + +# Start the tag cleanup +Projects::ContainerRepository::CleanupTagsService.new(project, user, policy.attributes.except("created_at", "updated_at")).execute(repo) +``` + +NOTE: **Note:** +You can also [run cleanup on a schedule](../../user/packages/container_registry/index.md#cleanup-policy). + ## Container Registry garbage collection NOTE: **Note:** diff --git a/doc/administration/packages/index.md b/doc/administration/packages/index.md index 5d07136ef40..3af1f0c933b 100644 --- a/doc/administration/packages/index.md +++ b/doc/administration/packages/index.md @@ -4,7 +4,7 @@ group: Package info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# GitLab Package Registry administration **(PREMIUM ONLY)** +# GitLab Package Registry administration GitLab Packages allows organizations to utilize GitLab as a private repository for a variety of common package managers. Users are able to build and publish diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index 4efd92eaa07..9e2aa602767 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -114,7 +114,7 @@ since that is needed in all configurations. --- -URL scheme: `http://page.example.io` +URL scheme: `http://.example.io/` This is the minimum setup that you can use Pages with. It is the base for all other setups as described below. NGINX will proxy all requests to the daemon. @@ -139,7 +139,7 @@ Watch the [video tutorial](https://youtu.be/dD8c7WNcc6s) for this configuration. --- -URL scheme: `https://page.example.io` +URL scheme: `https://.example.io/` NGINX will proxy all requests to the daemon. Pages daemon doesn't listen to the outside world. @@ -204,6 +204,7 @@ control over how the Pages daemon runs and serves content in your environment. | `external_https` | Configure Pages to bind to one or more secondary IP addresses, serving HTTPS requests. Multiple addresses can be given as an array, along with exact ports, for example `['1.2.3.4', '1.2.3.5:8063']`. Sets value for `listen_https`. | `gitlab_client_http_timeout` | GitLab API HTTP client connection timeout in seconds (default: 10s). | `gitlab_client_jwt_expiry` | JWT Token expiry time in seconds (default: 30s). +| `domain_config_source` | Domain configuration source (default: `disk`) | `gitlab_id` | The OAuth application public ID. Leave blank to automatically fill when Pages authenticates with GitLab. | `gitlab_secret` | The OAuth application secret. Leave blank to automatically fill when Pages authenticates with GitLab. | `gitlab_server` | Server to use for authentication when access control is enabled; defaults to GitLab `external_url`. @@ -254,7 +255,7 @@ you have IPv6 as well as IPv4 addresses, you can use them both. --- -URL scheme: `http://page.example.io` and `http://domain.com` +URL scheme: `http://.example.io/` and `http://custom-domain.com` In that case, the Pages daemon is running, NGINX still proxies requests to the daemon but the daemon is also able to receive requests from the outside @@ -285,7 +286,7 @@ world. Custom domains are supported, but no TLS. --- -URL scheme: `https://page.example.io` and `https://domain.com` +URL scheme: `https://.example.io/` and `https://custom-domain.com` In that case, the Pages daemon is running, NGINX still proxies requests to the daemon but the daemon is also able to receive requests from the outside @@ -388,9 +389,9 @@ To do that: 1. Click **Save changes**. CAUTION: **Warning:** -This action will not make all currently public web-sites private until they redeployed. -This issue among others will be resolved by -[changing GitLab Pages configuration mechanism](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/282). +For self-managed installations, all public websites remain private until they are +redeployed. This issue will be resolved by +[sourcing domain configuration from the GitLab API](https://gitlab.com/gitlab-org/gitlab/-/issues/218357). ### Running behind a proxy @@ -409,7 +410,7 @@ pages: ### Using a custom Certificate Authority (CA) NOTE: **Note:** -[Before 13.2](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4289), when using Omnibus, a [workaround was required](https://docs.gitlab.com/13.1/ee/administration/pages/index.html#using-a-custom-certificate-authority-ca). +[Before 13.3](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4411), when using Omnibus, a [workaround was required](https://docs.gitlab.com/13.1/ee/administration/pages/index.html#using-a-custom-certificate-authority-ca). When using certificates issued by a custom CA, [Access Control](../../user/project/pages/pages_access_control.md#gitlab-pages-access-control) and the [online view of HTML job artifacts](../../ci/pipelines/job_artifacts.md#browsing-artifacts) @@ -536,7 +537,7 @@ database encryption. Proceed with caution. 1. Set up a new server. This will become the **Pages server**. -1. Create an [NFS share](../high_availability/nfs_host_client_setup.md) +1. Create an [NFS share](../nfs.md) on the **Pages server** and configure this share to allow access from your main **GitLab server**. Note that the example there is more general and @@ -601,6 +602,43 @@ configuring a load balancer to work at the IP level, and so on. If you wish to set up GitLab Pages on multiple servers, perform the above procedure for each Pages server. +## Domain source configuration + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217912) in GitLab 13.3. + +GitLab Pages can use different sources to get domain configuration. +The default value is `nil`; however, GitLab Pages will default to `disk`. + + ```ruby + gitlab_pages['domain_config_source'] = nil + ``` + +You can specify `gitlab` to enable [API-based configuration](#gitlab-api-based-configuration). + +For more details see this [blog post](https://about.gitlab.com/blog/2020/08/03/how-gitlab-pages-uses-the-gitlab-api-to-serve-content/). + +### GitLab API-based configuration + +GitLab Pages can use an API-based configuration. This replaces disk source configuration, which +was used prior to GitLab 13.0. Follow these steps to enable it: + +1. Add the following to your `/etc/gitlab/gitlab.erb` file: + + ```ruby + gitlab_pages['domain_config_source'] = "gitlab" + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +If you encounter an issue, you can disable it by choosing `disk` or `nil`: + +```ruby +gitlab_pages['domain_config_source'] = nil +``` + +For other common issues, see the [troubleshooting section](#failed-to-connect-to-the-internal-gitlab-api) +or report an issue. + ## Backup GitLab Pages are part of the [regular backup](../../raketasks/backup_restore.md), so there is no separate backup to configure. @@ -696,3 +734,24 @@ date > /var/opt/gitlab/gitlab-rails/shared/pages/.update ``` If you've customized the Pages storage path, adjust the command above to use your custom path. + +### Failed to connect to the internal GitLab API + +If you have enabled [API-based configuration](#gitlab-api-based-configuration) and see the following error: + +```plaintext +ERRO[0010] Failed to connect to the internal GitLab API after 0.50s error="failed to connect to internal Pages API: HTTP status: 401" +``` + +If you are [Running GitLab Pages on a separate server](#running-gitlab-pages-on-a-separate-server) +you must copy the `/etc/gitlab/gitlab-secrets.json` file +from the **GitLab server** to the **Pages server** after upgrading to GitLab 13.3, +as described in that section. + +Other reasons may include network connectivity issues between your +**GitLab server** and your **Pages server** such as firewall configurations or closed ports. +For example, if there is a connection timeout: + +```plaintext +error="failed to connect to internal Pages API: Get \"https://gitlab.example.com:3000/api/v4/internal/pages/status\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" +``` diff --git a/doc/administration/pages/source.md b/doc/administration/pages/source.md index d5b49bdf839..486bc7a8777 100644 --- a/doc/administration/pages/source.md +++ b/doc/administration/pages/source.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # GitLab Pages administration for source installations ->**Note:** +NOTE: **Note:** Before attempting to enable GitLab Pages, first make sure you have [installed GitLab](../../install/installation.md) successfully. @@ -77,7 +77,7 @@ host that GitLab runs. For example, an entry would look like this: where `example.io` is the domain under which GitLab Pages will be served and `192.0.2.1` is the IP address of your GitLab instance. -> **Note:** +NOTE: **Note:** You should not use the GitLab domain to serve user pages. For more information see the [security section](#security). @@ -94,7 +94,7 @@ since that is needed in all configurations. - [Wildcard DNS setup](#dns-configuration) -URL scheme: `http://page.example.io` +URL scheme: `http://.example.io/` This is the minimum setup that you can use Pages with. It is the base for all other setups as described below. NGINX will proxy all requests to the daemon. @@ -157,7 +157,7 @@ The Pages daemon doesn't listen to the outside world. - [Wildcard DNS setup](#dns-configuration) - Wildcard TLS certificate -URL scheme: `https://page.example.io` +URL scheme: `https://.example.io/` NGINX will proxy all requests to the daemon. Pages daemon doesn't listen to the outside world. @@ -221,7 +221,7 @@ that without TLS certificates. - [Wildcard DNS setup](#dns-configuration) - Secondary IP -URL scheme: `http://page.example.io` and `http://domain.com` +URL scheme: `http://.example.io/` and `http://custom-domain.com` In that case, the pages daemon is running, NGINX still proxies requests to the daemon but the daemon is also able to receive requests from the outside @@ -286,7 +286,7 @@ world. Custom domains are supported, but no TLS. - Wildcard TLS certificate - Secondary IP -URL scheme: `https://page.example.io` and `https://domain.com` +URL scheme: `https://.example.io/` and `https://custom-domain.com` In that case, the pages daemon is running, NGINX still proxies requests to the daemon but the daemon is also able to receive requests from the outside @@ -349,7 +349,7 @@ world. Custom domains and TLS are supported. ## NGINX caveats ->**Note:** +NOTE: **Note:** The following information applies only for installations from source. Be extra careful when setting up the domain name in the NGINX configuration. You must diff --git a/doc/administration/postgresql/external.md b/doc/administration/postgresql/external.md index 6e2bbc0aae1..e2cfb95ec48 100644 --- a/doc/administration/postgresql/external.md +++ b/doc/administration/postgresql/external.md @@ -32,7 +32,7 @@ If you use a cloud-managed service, or provide your own PostgreSQL instance: gitlab_rails['db_password'] = 'DB password' ``` - For more information on GitLab HA setups, refer to [configuring GitLab for HA](../high_availability/gitlab.md). + For more information on GitLab multi-node setups, refer to the [reference architectures](../reference_architectures/index.md). 1. Reconfigure for the changes to take effect: diff --git a/doc/administration/postgresql/index.md b/doc/administration/postgresql/index.md index 7e0a2f3cae1..2720d8e696b 100644 --- a/doc/administration/postgresql/index.md +++ b/doc/administration/postgresql/index.md @@ -13,7 +13,7 @@ There are essentially three setups to choose from. This setup is for when you have installed GitLab using the [Omnibus GitLab **Enterprise Edition** (EE) package](https://about.gitlab.com/install/?version=ee). -All the tools that are needed like PostgreSQL, PgBouncer, Repmgr are bundled in +All the tools that are needed like PostgreSQL, PgBouncer, Patroni, and repmgr are bundled in the package, so you can it to set up the whole PostgreSQL infrastructure (primary, replica). [> Read how to set up PostgreSQL replication and failover using Omnibus GitLab](replication_and_failover.md) diff --git a/doc/administration/postgresql/pgbouncer.md b/doc/administration/postgresql/pgbouncer.md new file mode 100644 index 00000000000..9db3e017359 --- /dev/null +++ b/doc/administration/postgresql/pgbouncer.md @@ -0,0 +1,168 @@ +--- +type: reference +--- + +# Working with the bundled PgBouncer service **(PREMIUM ONLY)** + +[PgBouncer](http://www.pgbouncer.org/) is used to seamlessly migrate database +connections between servers in a failover scenario. Additionally, it can be used +in a non-fault-tolerant setup to pool connections, speeding up response time +while reducing resource usage. + +GitLab Premium includes a bundled version of PgBouncer that can be managed +through `/etc/gitlab/gitlab.rb`. + +## PgBouncer as part of a fault-tolerant GitLab installation + +This content has been moved to a [new location](replication_and_failover.md#configuring-the-pgbouncer-node). + +## PgBouncer as part of a non-fault-tolerant GitLab installation + +1. Generate PGBOUNCER_USER_PASSWORD_HASH with the command `gitlab-ctl pg-password-md5 pgbouncer` + +1. Generate SQL_USER_PASSWORD_HASH with the command `gitlab-ctl pg-password-md5 gitlab`. We'll also need to enter the plaintext SQL_USER_PASSWORD later + +1. On your database node, ensure the following is set in your `/etc/gitlab/gitlab.rb` + + ```ruby + postgresql['pgbouncer_user_password'] = 'PGBOUNCER_USER_PASSWORD_HASH' + postgresql['sql_user_password'] = 'SQL_USER_PASSWORD_HASH' + postgresql['listen_address'] = 'XX.XX.XX.Y' # Where XX.XX.XX.Y is the ip address on the node postgresql should listen on + postgresql['md5_auth_cidr_addresses'] = %w(AA.AA.AA.B/32) # Where AA.AA.AA.B is the IP address of the pgbouncer node + ``` + +1. Run `gitlab-ctl reconfigure` + + NOTE: **Note:** + If the database was already running, it will need to be restarted after reconfigure by running `gitlab-ctl restart postgresql`. + +1. On the node you are running PgBouncer on, make sure the following is set in `/etc/gitlab/gitlab.rb` + + ```ruby + pgbouncer['enable'] = true + pgbouncer['databases'] = { + gitlabhq_production: { + host: 'DATABASE_HOST', + user: 'pgbouncer', + password: 'PGBOUNCER_USER_PASSWORD_HASH' + } + } + ``` + +1. Run `gitlab-ctl reconfigure` + +1. On the node running Puma, make sure the following is set in `/etc/gitlab/gitlab.rb` + + ```ruby + gitlab_rails['db_host'] = 'PGBOUNCER_HOST' + gitlab_rails['db_port'] = '6432' + gitlab_rails['db_password'] = 'SQL_USER_PASSWORD' + ``` + +1. Run `gitlab-ctl reconfigure` + +1. At this point, your instance should connect to the database through PgBouncer. If you are having issues, see the [Troubleshooting](#troubleshooting) section + +## Enable Monitoring + +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3786) in GitLab 12.0. + +If you enable Monitoring, it must be enabled on **all** PgBouncer servers. + +1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration: + + ```ruby + # Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + # Replace placeholders + # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z + # with the addresses of the Consul server nodes + consul['configuration'] = { + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z), + } + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + pgbouncer_exporter['listen_address'] = '0.0.0.0:9188' + ``` + +1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. + +## Administrative console + +As part of Omnibus GitLab, a command is provided to automatically connect to the +PgBouncer administrative console. See the +[PgBouncer documentation](https://www.pgbouncer.org/usage.html#admin-console) +for detailed instructions on how to interact with the console. + +To start a session run the following and provide the password for the `pgbouncer` +user: + +```shell +sudo gitlab-ctl pgb-console +``` + +To get some basic information about the instance: + +```shell +pgbouncer=# show databases; show clients; show servers; + name | host | port | database | force_user | pool_size | reserve_pool | pool_mode | max_connections | current_connections +---------------------+-----------+------+---------------------+------------+-----------+--------------+-----------+-----------------+--------------------- + gitlabhq_production | 127.0.0.1 | 5432 | gitlabhq_production | | 100 | 5 | | 0 | 1 + pgbouncer | | 6432 | pgbouncer | pgbouncer | 2 | 0 | statement | 0 | 0 +(2 rows) + + type | user | database | state | addr | port | local_addr | local_port | connect_time | request_time | ptr | link +| remote_pid | tls +------+-----------+---------------------+--------+-----------+-------+------------+------------+---------------------+---------------------+-----------+------ ++------------+----- + C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44590 | 127.0.0.1 | 6432 | 2018-04-24 22:13:10 | 2018-04-24 22:17:10 | 0x12444c0 | +| 0 | + C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44592 | 127.0.0.1 | 6432 | 2018-04-24 22:13:10 | 2018-04-24 22:17:10 | 0x12447c0 | +| 0 | + C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44594 | 127.0.0.1 | 6432 | 2018-04-24 22:13:10 | 2018-04-24 22:17:10 | 0x1244940 | +| 0 | + C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44706 | 127.0.0.1 | 6432 | 2018-04-24 22:14:22 | 2018-04-24 22:16:31 | 0x1244ac0 | +| 0 | + C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44708 | 127.0.0.1 | 6432 | 2018-04-24 22:14:22 | 2018-04-24 22:15:15 | 0x1244c40 | +| 0 | + C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44794 | 127.0.0.1 | 6432 | 2018-04-24 22:15:15 | 2018-04-24 22:15:15 | 0x1244dc0 | +| 0 | + C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44798 | 127.0.0.1 | 6432 | 2018-04-24 22:15:15 | 2018-04-24 22:16:31 | 0x1244f40 | +| 0 | + C | pgbouncer | pgbouncer | active | 127.0.0.1 | 44660 | 127.0.0.1 | 6432 | 2018-04-24 22:13:51 | 2018-04-24 22:17:12 | 0x1244640 | +| 0 | +(8 rows) + + type | user | database | state | addr | port | local_addr | local_port | connect_time | request_time | ptr | link | rem +ote_pid | tls +------+--------+---------------------+-------+-----------+------+------------+------------+---------------------+---------------------+-----------+------+---- +--------+----- + S | gitlab | gitlabhq_production | idle | 127.0.0.1 | 5432 | 127.0.0.1 | 35646 | 2018-04-24 22:15:15 | 2018-04-24 22:17:10 | 0x124dca0 | | + 19980 | +(1 row) +``` + +## Troubleshooting + +In case you are experiencing any issues connecting through PgBouncer, the first +place to check is always the logs: + +```shell +sudo gitlab-ctl tail pgbouncer +``` + +Additionally, you can check the output from `show databases` in the +[administrative console](#administrative-console). In the output, you would expect +to see values in the `host` field for the `gitlabhq_production` database. +Additionally, `current_connections` should be greater than 1. + +### Message: `LOG: invalid CIDR mask in address` + +See the suggested fix [in Geo documentation](../geo/replication/troubleshooting.md#message-log--invalid-cidr-mask-in-address). + +### Message: `LOG: invalid IP mask "md5": Name or service not known` + +See the suggested fix [in Geo documentation](../geo/replication/troubleshooting.md#message-log--invalid-ip-mask-md5-name-or-service-not-known). diff --git a/doc/administration/postgresql/replication_and_failover.md b/doc/administration/postgresql/replication_and_failover.md index 5f550f09e5b..bc2af167e6c 100644 --- a/doc/administration/postgresql/replication_and_failover.md +++ b/doc/administration/postgresql/replication_and_failover.md @@ -29,6 +29,11 @@ You also need to take into consideration the underlying network topology, making sure you have redundant connectivity between all Database and GitLab instances to avoid the network becoming a single point of failure. +NOTE: **Note:** +As of GitLab 13.3, PostgreSQL 12 is shipped with Omnibus GitLab. Clustering for PostgreSQL 12 is only supported with +Patroni. See the [Patroni](#patroni) section for further details. The support for repmgr will not be extended beyond +PostgreSQL 11. + ### Database node Each database node runs three services: @@ -97,7 +102,7 @@ This is why you will need: When using default setup, minimum configuration requires: -- `CONSUL_USERNAME`. Defaults to `gitlab-consul` +- `CONSUL_USERNAME`. The default user for Omnibus GitLab is `gitlab-consul` - `CONSUL_DATABASE_PASSWORD`. Password for the database user. - `CONSUL_PASSWORD_HASH`. This is a hash generated out of Consul username/password pair. Can be generated with: @@ -140,7 +145,7 @@ server nodes. We will need the following password information for the application's database user: -- `POSTGRESQL_USERNAME`. Defaults to `gitlab` +- `POSTGRESQL_USERNAME`. The default user for Omnibus GitLab is `gitlab` - `POSTGRESQL_USER_PASSWORD`. The password for the database user - `POSTGRESQL_PASSWORD_HASH`. This is a hash generated out of the username/password pair. Can be generated with: @@ -153,7 +158,7 @@ We will need the following password information for the application's database u When using default setup, minimum configuration requires: -- `PGBOUNCER_USERNAME`. Defaults to `pgbouncer` +- `PGBOUNCER_USERNAME`. The default user for Omnibus GitLab is `pgbouncer` - `PGBOUNCER_PASSWORD`. This is a password for PgBouncer service. - `PGBOUNCER_PASSWORD_HASH`. This is a hash generated out of PgBouncer username/password pair. Can be generated with: @@ -198,7 +203,7 @@ When installing the GitLab package, do not supply `EXTERNAL_URL` value. ### Configuring the Database nodes -1. Make sure to [configure the Consul nodes](../high_availability/consul.md). +1. Make sure to [configure the Consul nodes](../consul.md). 1. Make sure you collect [`CONSUL_SERVER_NODES`](#consul-information), [`PGBOUNCER_PASSWORD_HASH`](#pgbouncer-information), [`POSTGRESQL_PASSWORD_HASH`](#postgresql-information), the [number of db nodes](#postgresql-information), and the [network address](#network-information) before executing the next step. 1. On the master database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: @@ -305,6 +310,12 @@ Select one node as a primary node. CREATE EXTENSION pg_trgm; ``` +1. Enable the `btree_gist` extension: + + ```shell + CREATE EXTENSION btree_gist; + ``` + 1. Exit the database prompt by typing `\q` and Enter. 1. Verify the cluster is initialized with one node: @@ -455,7 +466,7 @@ Check the [Troubleshooting section](#troubleshooting) before proceeding. gitlab-ctl write-pgpass --host 127.0.0.1 --database pgbouncer --user pgbouncer --hostuser gitlab-consul ``` -1. [Enable monitoring](../high_availability/pgbouncer.md#enable-monitoring) +1. [Enable monitoring](../postgresql/pgbouncer.md#enable-monitoring) #### PgBouncer Checkpoint @@ -736,9 +747,9 @@ consul['configuration'] = { After deploying the configuration follow these steps: -1. On `10.6.0.31`, our primary database +1. On `10.6.0.31`, our primary database: - Enable the `pg_trgm` extension + Enable the `pg_trgm` and `btree_gist` extensions: ```shell gitlab-psql -d gitlabhq_production @@ -746,33 +757,34 @@ After deploying the configuration follow these steps: ```shell CREATE EXTENSION pg_trgm; + CREATE EXTENSION btree_gist; ``` -1. On `10.6.0.32`, our first standby database +1. On `10.6.0.32`, our first standby database: - Make this node a standby of the primary + Make this node a standby of the primary: ```shell gitlab-ctl repmgr standby setup 10.6.0.21 ``` -1. On `10.6.0.33`, our second standby database +1. On `10.6.0.33`, our second standby database: - Make this node a standby of the primary + Make this node a standby of the primary: ```shell gitlab-ctl repmgr standby setup 10.6.0.21 ``` -1. On `10.6.0.41`, our application server +1. On `10.6.0.41`, our application server: - Set `gitlab-consul` user's PgBouncer password to `toomanysecrets` + Set `gitlab-consul` user's PgBouncer password to `toomanysecrets`: ```shell gitlab-ctl write-pgpass --host 127.0.0.1 --database pgbouncer --user pgbouncer --hostuser gitlab-consul ``` - Run database migrations + Run database migrations: ```shell gitlab-rake gitlab:db:configure @@ -783,7 +795,7 @@ After deploying the configuration follow these steps: This example uses 3 PostgreSQL servers, and 1 application node (with PgBouncer setup alongside). It differs from the [recommended setup](#example-recommended-setup) by moving the Consul servers into the same servers we use for PostgreSQL. -The trade-off is between reducing server counts, against the increased operational complexity of needing to deal with PostgreSQL [failover](#failover-procedure) and [restore](#restore-procedure) procedures in addition to [Consul outage recovery](../high_availability/consul.md#outage-recovery) on the same set of machines. +The trade-off is between reducing server counts, against the increased operational complexity of needing to deal with PostgreSQL [failover](#failover-procedure) and [restore](#restore-procedure) procedures in addition to [Consul outage recovery](../consul.md#outage-recovery) on the same set of machines. In this example we start with all servers on the same 10.6.0.0/16 private network range, they can connect to each freely other on those addresses. @@ -1075,7 +1087,7 @@ To restart either service, run `gitlab-ctl restart SERVICE` For PostgreSQL, it is usually safe to restart the master node by default. Automatic failover defaults to a 1 minute timeout. Provided the database returns before then, nothing else needs to be done. To be safe, you can stop `repmgrd` on the standby nodes first with `gitlab-ctl stop repmgrd`, then start afterwards with `gitlab-ctl start repmgrd`. -On the Consul server nodes, it is important to restart the Consul service in a controlled fashion. Read our [Consul documentation](../high_availability/consul.md#restarting-the-server-cluster) for instructions on how to restart the service. +On the Consul server nodes, it is important to [restart the Consul service](../consul.md#restart-consul) in a controlled manner. ### `gitlab-ctl repmgr-check-master` command produces errors @@ -1122,11 +1134,10 @@ postgresql['trust_auth_cidr_addresses'] = %w(123.123.123.123/32 ) ### Issues with other components -If you're running into an issue with a component not outlined here, be sure to check the troubleshooting section of their specific documentation page. +If you're running into an issue with a component not outlined here, be sure to check the troubleshooting section of their specific documentation page: -- [Consul](../high_availability/consul.md#troubleshooting) +- [Consul](../consul.md#troubleshooting-consul) - [PostgreSQL](https://docs.gitlab.com/omnibus/settings/database.html#troubleshooting) -- [GitLab application](../high_availability/gitlab.md#troubleshooting) ## Patroni @@ -1324,7 +1335,7 @@ You can switch an exiting database cluster to use Patroni instead of repmgr with NOTE: **Note:** Ensure that there is no `walsender` process running on the primary node. - `ps aux | grep walsender` must not show any running process. + `ps aux | grep walsender` must not show any running process. 1. On the primary node, [configure Patroni](#configuring-patroni-cluster). Remove `repmgr` and any other repmgr-specific configuration. Also remove any configuration that is related to PostgreSQL replication. diff --git a/doc/administration/raketasks/check.md b/doc/administration/raketasks/check.md index da5caf3966f..15014fffd01 100644 --- a/doc/administration/raketasks/check.md +++ b/doc/administration/raketasks/check.md @@ -135,3 +135,22 @@ The LDAP check Rake task will test the bind DN and password credentials (if configured) and will list a sample of LDAP users. This task is also executed as part of the `gitlab:check` task, but can run independently. See [LDAP Rake Tasks - LDAP Check](ldap.md#check) for details. + +## Troubleshooting + +The following are solutions to problems you might discover using the Rake tasks documented +above. + +### Dangling commits + +`gitlab:git:fsck` can find dangling commits. To fix them, try +[manually triggering housekeeping](../housekeeping.md#manual-housekeeping) +for the affected project(s). + +If the issue persists, try triggering `gc` via the +[Rails Console](../troubleshooting/navigating_gitlab_via_rails_console.md#starting-a-rails-console-session): + +```ruby +p = Project.find_by_path("project-name") +Projects::HousekeepingService.new(p, :gc).execute +``` diff --git a/doc/administration/raketasks/maintenance.md b/doc/administration/raketasks/maintenance.md index 19781b6a5db..553afba78e3 100644 --- a/doc/administration/raketasks/maintenance.md +++ b/doc/administration/raketasks/maintenance.md @@ -23,32 +23,39 @@ Example output: ```plaintext System information -System: Debian 7.8 -Current User: git -Using RVM: no -Ruby Version: 2.1.5p273 -Gem Version: 2.4.3 -Bundler Version: 1.7.6 -Rake Version: 10.3.2 -Redis Version: 3.2.5 -Sidekiq Version: 2.17.8 +System: Ubuntu 20.04 +Proxy: no +Current User: git +Using RVM: no +Ruby Version: 2.6.6p146 +Gem Version: 2.7.10 +Bundler Version:1.17.3 +Rake Version: 12.3.3 +Redis Version: 5.0.9 +Git Version: 2.27.0 +Sidekiq Version:5.2.9 +Go Version: unknown GitLab information -Version: 7.7.1 -Revision: 41ab9e1 -Directory: /home/git/gitlab -DB Adapter: postgresql -URL: https://gitlab.example.com -HTTP Clone URL: https://gitlab.example.com/some-project.git -SSH Clone URL: git@gitlab.example.com:some-project.git -Using LDAP: no -Using Omniauth: no +Version: 13.2.2-ee +Revision: 618883a1f9d +Directory: /opt/gitlab/embedded/service/gitlab-rails +DB Adapter: PostgreSQL +DB Version: 11.7 +URL: http://gitlab.example.com +HTTP Clone URL: http://gitlab.example.com/some-group/some-project.git +SSH Clone URL: git@gitlab.example.com:some-group/some-project.git +Elasticsearch: no +Geo: no +Using LDAP: no +Using Omniauth: yes +Omniauth Providers: GitLab Shell -Version: 2.4.1 -Repositories: /home/git/repositories/ -Hooks: /home/git/gitlab-shell/hooks/ -Git: /usr/bin/git +Version: 13.3.0 +Repository storage paths: +- default: /var/opt/gitlab/git-data/repositories +GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell ``` ## Show GitLab license information **(STARTER ONLY)** diff --git a/doc/administration/raketasks/project_import_export.md b/doc/administration/raketasks/project_import_export.md index b807e03b01f..51e0e2e46b6 100644 --- a/doc/administration/raketasks/project_import_export.md +++ b/doc/administration/raketasks/project_import_export.md @@ -46,6 +46,6 @@ Note the following: compatible as described in the [Version history](../../user/project/settings/import_export.md#version-history). - The project import option must be enabled in application settings (`/admin/application_settings/general`) under **Import sources**, which is available - under **{admin}** **Admin Area >** **{settings}** **Settings > Visibility and access controls**. + under **Admin Area > Settings > Visibility and access controls**. - The exports are stored in a temporary [shared directory](../../development/shared_files.md) and are deleted every 24 hours by a specific worker. diff --git a/doc/administration/raketasks/storage.md b/doc/administration/raketasks/storage.md index 74fd2c2ebb6..a97bff83290 100644 --- a/doc/administration/raketasks/storage.md +++ b/doc/administration/raketasks/storage.md @@ -100,7 +100,7 @@ to project IDs 50 to 100 in an Omnibus GitLab installation: sudo gitlab-rake gitlab:storage:migrate_to_hashed ID_FROM=50 ID_TO=100 ``` -You can monitor the progress in the **{admin}** **Admin Area > Monitoring > Background Jobs** page. +You can monitor the progress in the **Admin Area > Monitoring > Background Jobs** page. There is a specific queue you can watch to see how long it will take to finish: `hashed_storage:hashed_storage_project_migrate`. @@ -150,7 +150,7 @@ to project IDs 50 to 100 in an Omnibus GitLab installation: sudo gitlab-rake gitlab:storage:rollback_to_legacy ID_FROM=50 ID_TO=100 ``` -You can monitor the progress in the **{admin}** **Admin Area > Monitoring > Background Jobs** page. +You can monitor the progress in the **Admin Area > Monitoring > Background Jobs** page. On the **Queues** tab, you can watch the `hashed_storage:hashed_storage_project_rollback` queue to see how long the process will take to finish. After it reaches zero, you can confirm every project has been rolled back by running the commands bellow. diff --git a/doc/administration/raketasks/uploads/migrate.md b/doc/administration/raketasks/uploads/migrate.md index b164c4e744d..8c020e91a15 100644 --- a/doc/administration/raketasks/uploads/migrate.md +++ b/doc/administration/raketasks/uploads/migrate.md @@ -141,7 +141,15 @@ keeping in mind the task name in this case is `gitlab:uploads:migrate_to_local`. To migrate uploads from object storage to local storage: -1. Disable both `direct_upload` and `background_upload` under `uploads` settings in `gitlab.rb`. +1. Disable both `direct_upload` and `background_upload` under `uploads` settings in `gitlab.rb`: + + ```ruby + gitlab_rails['uploads_object_store_direct_upload'] = false + gitlab_rails['uploads_object_store_background_upload'] = false + ``` + + Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure). + 1. Run the Rake task: **Omnibus Installation** diff --git a/doc/administration/redis/replication_and_failover.md b/doc/administration/redis/replication_and_failover.md index ac31b909c89..ca041adb1d8 100644 --- a/doc/administration/redis/replication_and_failover.md +++ b/doc/administration/redis/replication_and_failover.md @@ -466,7 +466,7 @@ While it doesn't require a list of all Sentinel nodes, in case of a failure, it needs to access at least one of the listed. NOTE: **Note:** -The following steps should be performed in the [GitLab application server](../high_availability/gitlab.md) +The following steps should be performed in the GitLab application server which ideally should not have Redis or Sentinels on it for a HA setup. 1. SSH into the server where the GitLab application is installed. @@ -637,47 +637,63 @@ lives easier. If you want to know what happens underneath keep reading. ### Running multiple Redis clusters -GitLab supports running [separate Redis clusters for different persistent -classes](https://docs.gitlab.com/omnibus/settings/redis.html#running-with-multiple-redis-instances): -cache, queues, and shared_state. To make this work with Sentinel: +Omnibus GitLab supports running separate Redis and Sentinel instances for different +persistence classes. -1. Set the appropriate variable in `/etc/gitlab/gitlab.rb` for each instance you are using: +| Class | Purpose | +| -------------- | ------------------------------------------------ | +| `cache` | Store cached data. | +| `queues` | Store Sidekiq background jobs. | +| `shared_state` | Store session-related and other persistent data. | +| `actioncable` | Pub/Sub queue backend for ActionCable. | + +To make this work with Sentinel: + +1. [Configure the different Redis/Sentinels](#configuring-redis) instances based on your needs. +1. For each Rails application instance, edit its `/etc/gitlab/gitlab.rb` file: ```ruby gitlab_rails['redis_cache_instance'] = REDIS_CACHE_URL gitlab_rails['redis_queues_instance'] = REDIS_QUEUES_URL gitlab_rails['redis_shared_state_instance'] = REDIS_SHARED_STATE_URL - ``` - - **Note**: Redis URLs should be in the format: `redis://:PASSWORD@SENTINEL_PRIMARY_NAME` + gitlab_rails['redis_actioncable_instance'] = REDIS_ACTIONCABLE_URL - 1. PASSWORD is the plaintext password for the Redis instance - 1. SENTINEL_PRIMARY_NAME is the Sentinel primary name (e.g. `gitlab-redis-cache`) - -1. Include an array of hashes with host/port combinations, such as the following: - - ```ruby + # Configure the Sentinels gitlab_rails['redis_cache_sentinels'] = [ - { host: REDIS_CACHE_SENTINEL_HOST, port: PORT1 }, - { host: REDIS_CACHE_SENTINEL_HOST2, port: PORT2 } + { host: REDIS_CACHE_SENTINEL_HOST, port: 26379 }, + { host: REDIS_CACHE_SENTINEL_HOST2, port: 26379 } ] gitlab_rails['redis_queues_sentinels'] = [ - { host: REDIS_QUEUES_SENTINEL_HOST, port: PORT1 }, - { host: REDIS_QUEUES_SENTINEL_HOST2, port: PORT2 } + { host: REDIS_QUEUES_SENTINEL_HOST, port: 26379 }, + { host: REDIS_QUEUES_SENTINEL_HOST2, port: 26379 } ] gitlab_rails['redis_shared_state_sentinels'] = [ - { host: SHARED_STATE_SENTINEL_HOST, port: PORT1 }, - { host: SHARED_STATE_SENTINEL_HOST2, port: PORT2 } + { host: SHARED_STATE_SENTINEL_HOST, port: 26379 }, + { host: SHARED_STATE_SENTINEL_HOST2, port: 26379 } + ] + gitlab_rails['redis_actioncable_sentinels'] = [ + { host: ACTIONCABLE_SENTINEL_HOST, port: 26379 }, + { host: ACTIONCABLE_SENTINEL_HOST2, port: 26379 } ] ``` -1. Note that for each persistence class, GitLab will default to using the - configuration specified in `gitlab_rails['redis_sentinels']` unless - overridden by the settings above. -1. Be sure to include BOTH configuration options for each persistent classes. For example, - if you choose to configure a cache instance, you must specify both `gitlab_rails['redis_cache_instance']` - and `gitlab_rails['redis_cache_sentinels']` for GitLab to generate the proper configuration files. -1. Run `gitlab-ctl reconfigure` + Note that: + + - Redis URLs should be in the format: `redis://:PASSWORD@SENTINEL_PRIMARY_NAME`, where: + - `PASSWORD` is the plaintext password for the Redis instance. + - `SENTINEL_PRIMARY_NAME` is the Sentinel primary name set with `redis['master_name']`, + for example `gitlab-redis-cache`. + +1. Save the file and reconfigure GitLab for the change to take effect: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +NOTE: **Note:** +For each persistence class, GitLab will default to using the +configuration specified in `gitlab_rails['redis_sentinels']` unless +overridden by the previously described settings. ### Control running services @@ -736,6 +752,5 @@ Read more: 1. [Reference architectures](../reference_architectures/index.md) 1. [Configure the database](../postgresql/replication_and_failover.md) -1. [Configure NFS](../high_availability/nfs.md) -1. [Configure the GitLab application servers](../high_availability/gitlab.md) -1. [Configure the load balancers](../high_availability/load_balancer.md) +1. [Configure NFS](../nfs.md) +1. [Configure the load balancers](../load_balancer.md) diff --git a/doc/administration/redis/replication_and_failover_external.md b/doc/administration/redis/replication_and_failover_external.md index 244b44dd76a..d530e6a8fd7 100644 --- a/doc/administration/redis/replication_and_failover_external.md +++ b/doc/administration/redis/replication_and_failover_external.md @@ -19,13 +19,7 @@ The following are the requirements for providing your own Redis instance: - Redis version 5.0 or higher is recommended, as this is what ships with Omnibus GitLab packages starting with GitLab 12.7. -- Support for Redis 3.2 is deprecated with GitLab 12.10 and will be completely - removed in GitLab 13.0. -- GitLab 12.0 and later requires Redis version 3.2 or higher. Older Redis - versions do not support an optional count argument to SPOP which is now - required for [Merge Trains](../../ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md). -- In addition, if Redis 4 or later is available, GitLab makes use of certain - commands like `UNLINK` and `USAGE` which were introduced only in Redis 4. +- GitLab 13.0 and later requires Redis version 4.0 or higher. - Standalone Redis or Redis high availability with Sentinel are supported. Redis Cluster is not supported. - Managed Redis from cloud providers such as AWS ElastiCache will work. If these @@ -221,7 +215,7 @@ the correct credentials for the Sentinel nodes. While it doesn't require a list of all Sentinel nodes, in case of a failure, it needs to access at least one of listed ones. -The following steps should be performed in the [GitLab application server](../high_availability/gitlab.md) +The following steps should be performed in the GitLab application server which ideally should not have Redis or Sentinels in the same machine: 1. Edit `/home/git/gitlab/config/resque.yml` following the example in diff --git a/doc/administration/redis/standalone.md b/doc/administration/redis/standalone.md index 12e932dbc5e..ea5a7850244 100644 --- a/doc/administration/redis/standalone.md +++ b/doc/administration/redis/standalone.md @@ -15,7 +15,7 @@ is generally stable and can handle many requests, so it is an acceptable trade off to have only a single instance. See the [reference architectures](../reference_architectures/index.md) page for an overview of GitLab scaling options. -## Set up a standalone Redis instance +## Set up the standalone Redis instance The steps below are the minimum necessary to configure a Redis server with Omnibus GitLab: @@ -28,36 +28,49 @@ Omnibus GitLab: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - ## Enable Redis - redis['enable'] = true - - ## Disable all other services - sidekiq['enable'] = false - gitlab_workhorse['enable'] = false - puma['enable'] = false - postgresql['enable'] = false - nginx['enable'] = false - prometheus['enable'] = false - alertmanager['enable'] = false - pgbouncer_exporter['enable'] = false - gitlab_exporter['enable'] = false - gitaly['enable'] = false + ## Enable Redis and disable all other services + ## https://docs.gitlab.com/omnibus/roles/ + roles ['redis_master_role'] + ## Redis configuration redis['bind'] = '0.0.0.0' redis['port'] = 6379 - redis['password'] = 'SECRET_PASSWORD_HERE' + redis['password'] = '' - gitlab_rails['enable'] = false + ## Disable automatic database migrations + ## Only the primary GitLab application server should handle migrations + gitlab_rails['auto_migrate'] = false ``` 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Note the Redis node's IP address or hostname, port, and - Redis password. These will be necessary when configuring the GitLab - application servers later. + Redis password. These will be necessary when [configuring the GitLab + application servers](#set-up-the-gitlab-rails-application-instance). [Advanced configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed. +## Set up the GitLab Rails application instance + +On the instance where GitLab is installed: + +1. Edit the `/etc/gitlab/gitlab.rb` file and add the following contents: + + ```ruby + ## Disable Redis + redis['enable'] = false + + gitlab_rails['redis_host'] = 'redis.example.com' + gitlab_rails['redis_port'] = 6379 + + ## Required if Redis authentication is configured on the Redis node + gitlab_rails['redis_password'] = '' + ``` + +1. Save your changes to `/etc/gitlab/gitlab.rb`. + +1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + ## Troubleshooting See the [Redis troubleshooting guide](troubleshooting.md). diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index 5367021af4e..fe2dad41066 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -1,76 +1,2047 @@ -# Reference architecture: up to 10,000 users +--- +reading_time: true +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- -This page describes GitLab reference architecture for up to 10,000 users. -For a full list of reference architectures, see +# Reference architecture: up to 10,000 users **(PREMIUM ONLY)** + +This page describes GitLab reference architecture for up to 10,000 users. For a +full list of reference architectures, see [Available reference architectures](index.md#available-reference-architectures). > - **Supported users (approximate):** 10,000 -> - **High Availability:** True -> - **Test RPS rates:** API: 200 RPS, Web: 20 RPS, Git: 20 RPS - -| Service | Nodes | Configuration ([8](#footnotes)) | GCP | AWS | Azure | -|--------------------------------------------------------------|-------|---------------------------------|----------------|-----------------------|----------------| -| GitLab Rails ([1](#footnotes)) | 3 | 32 vCPU, 28.8GB Memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | -| PostgreSQL | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | D4s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | -| Gitaly ([2](#footnotes)) ([5](#footnotes)) ([7](#footnotes)) | X | 16 vCPU, 60GB Memory | n1-standard-16 | m5.4xlarge | D16s v3 | -| Redis ([3](#footnotes)) - Cache | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis ([3](#footnotes)) - Queues / Shared State | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis Sentinel ([3](#footnotes)) - Cache | 3 | 1 vCPU, 1.7GB Memory | g1-small | t2.small | B1MS | -| Redis Sentinel ([3](#footnotes)) - Queues / Shared State | 3 | 1 vCPU, 1.7GB Memory | g1-small | t2.small | B1MS | -| Consul | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | -| Sidekiq | 4 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Object Storage ([4](#footnotes)) | - | - | - | - | - | -| NFS Server ([5](#footnotes)) ([7](#footnotes)) | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | F4s v2 | -| Monitoring node | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | F4s v2 | -| External load balancing node ([6](#footnotes)) | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | -| Internal load balancing node ([6](#footnotes)) | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | - -## Footnotes - -1. In our architectures we run each GitLab Rails node using the Puma webserver - and have its number of workers set to 90% of available CPUs along with four threads. For - nodes that are running Rails with other components the worker value should be reduced - accordingly where we've found 50% achieves a good balance but this is dependent - on workload. - -1. Gitaly node requirements are dependent on customer data, specifically the number of - projects and their sizes. We recommend two nodes as an absolute minimum for HA environments - and at least four nodes should be used when supporting 50,000 or more users. - We also recommend that each Gitaly node should store no more than 5TB of data - and have the number of [`gitaly-ruby` workers](../gitaly/index.md#gitaly-ruby) - set to 20% of available CPUs. Additional nodes should be considered in conjunction - with a review of expected data size and spread based on the recommendations above. - -1. Recommended Redis setup differs depending on the size of the architecture. - For smaller architectures (less than 3,000 users) a single instance should suffice. - For medium sized installs (3,000 - 5,000) we suggest one Redis cluster for all - classes and that Redis Sentinel is hosted alongside Consul. - For larger architectures (10,000 users or more) we suggest running a separate - [Redis Cluster](../redis/replication_and_failover.md#running-multiple-redis-clusters) for the Cache class - and another for the Queues and Shared State classes respectively. We also recommend - that you run the Redis Sentinel clusters separately for each Redis Cluster. - -1. For data objects such as LFS, Uploads, Artifacts, etc. We recommend an [Object Storage service](../object_storage.md) - over NFS where possible, due to better performance and availability. - -1. NFS can be used as an alternative for both repository data (replacing Gitaly) and - object storage but this isn't typically recommended for performance reasons. Note however it is required for - [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/196). - -1. Our architectures have been tested and validated with [HAProxy](https://www.haproxy.org/) - as the load balancer. Although other load balancers with similar feature sets - could also be used, those load balancers have not been validated. - -1. We strongly recommend that any Gitaly or NFS nodes be set up with SSD disks over - HDD with a throughput of at least 8,000 IOPS for read operations and 2,000 IOPS for write - as these components have heavy I/O. These IOPS values are recommended only as a starter - as with time they may be adjusted higher or lower depending on the scale of your - environment's workload. If you're running the environment on a Cloud provider - you may need to refer to their documentation on how configure IOPS correctly. - -1. The architectures were built and tested with the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) - CPU platform on GCP. On different hardware you may find that adjustments, either lower - or higher, are required for your CPU or Node counts accordingly. For more information, a - [Sysbench](https://github.com/akopytov/sysbench) benchmark of the CPU can be found - [here](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). +> - **High Availability:** Yes +> - **Test requests per second (RPS) rates:** API: 200 RPS, Web: 20 RPS, Git: 20 RPS + +| Service | Nodes | Configuration | GCP | AWS | Azure | +|--------------------------------------------|-------------|-------------------------|-----------------|-------------|----------| +| External load balancing node | 1 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Consul | 3 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| PostgreSQL | 3 | 4 vCPU, 15GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Internal load balancing node | 1 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Redis - Cache | 3 | 4 vCPU, 15GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| Redis - Queues / Shared State | 3 | 4 vCPU, 15GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7GB memory | g1-small | t2.small | B1MS | +| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7GB memory | g1-small | t2.small | B1MS | +| Gitaly | 2 (minimum) | 16 vCPU, 60GB memory | n1-standard-16 | m5.4xlarge | D16s v3 | +| Sidekiq | 4 | 4 vCPU, 15GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| GitLab Rails | 3 | 32 vCPU, 28.8GB memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | +| Monitoring node | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| Object Storage | n/a | n/a | n/a | n/a | n/a | +| NFS Server | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | + +The Google Cloud Platform (GCP) architectures were built and tested using the +[Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) +CPU platform. On different hardware you may find that adjustments, either lower +or higher, are required for your CPU or node counts. For more information, see +our [Sysbench](https://github.com/akopytov/sysbench)-based +[CPU benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). + +For data objects (such as LFS, Uploads, or Artifacts), an +[object storage service](#configure-the-object-storage) is recommended instead +of NFS where possible, due to better performance and availability. Since this +doesn't require a node to be set up, *Object Storage* is noted as not +applicable (n/a) in the previous table. + +## Setup components + +To set up GitLab and its components to accommodate up to 10,000 users: + +1. [Configure the external load balancing node](#configure-the-external-load-balancer) + that will handle the load balancing of the three GitLab application services nodes. +1. [Configure Consul](#configure-consul). +1. [Configure PostgreSQL](#configure-postgresql), the database for GitLab. +1. [Configure PgBouncer](#configure-pgbouncer). +1. [Configure the internal load balancing node](#configure-the-internal-load-balancer) +1. [Configure Redis](#configure-redis). +1. [Configure Gitaly](#configure-gitaly), + which provides access to the Git repositories. +1. [Configure Sidekiq](#configure-sidekiq). +1. [Configure the main GitLab Rails application](#configure-gitlab-rails) + to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend requests (UI, API, Git + over HTTP/SSH). +1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab environment. +1. [Configure the Object Storage](#configure-the-object-storage) + used for shared data objects. +1. [Configure NFS (Optional)](#configure-nfs-optional) + to have shared disk storage service as an alternative to Gitaly and/or Object Storage (although + not recommended). NFS is required for GitLab Pages, you can skip this step if you're not using + that feature. + +We start with all servers on the same 10.6.0.0/24 private network range, they +can connect to each other freely on those addresses. + +Here is a list and description of each machine and the assigned IP: + +- `10.6.0.10`: External Load Balancer +- `10.6.0.11`: Consul 1 +- `10.6.0.12`: Consul 2 +- `10.6.0.13`: Consul 3 +- `10.6.0.21`: PostgreSQL primary +- `10.6.0.22`: PostgreSQL secondary 1 +- `10.6.0.23`: PostgreSQL secondary 2 +- `10.6.0.31`: PgBouncer 1 +- `10.6.0.32`: PgBouncer 2 +- `10.6.0.33`: PgBouncer 3 +- `10.6.0.40`: Internal Load Balancer +- `10.6.0.51`: Redis - Cache Primary +- `10.6.0.52`: Redis - Cache Replica 1 +- `10.6.0.53`: Redis - Cache Replica 2 +- `10.6.0.71`: Sentinel - Cache 1 +- `10.6.0.72`: Sentinel - Cache 2 +- `10.6.0.73`: Sentinel - Cache 3 +- `10.6.0.61`: Redis - Queues Primary +- `10.6.0.62`: Redis - Queues Replica 1 +- `10.6.0.63`: Redis - Queues Replica 2 +- `10.6.0.81`: Sentinel - Queues 1 +- `10.6.0.82`: Sentinel - Queues 2 +- `10.6.0.83`: Sentinel - Queues 3 +- `10.6.0.91`: Gitaly 1 +- `10.6.0.92`: Gitaly 2 +- `10.6.0.101`: Sidekiq 1 +- `10.6.0.102`: Sidekiq 2 +- `10.6.0.103`: Sidekiq 3 +- `10.6.0.104`: Sidekiq 4 +- `10.6.0.111`: GitLab application 1 +- `10.6.0.112`: GitLab application 2 +- `10.6.0.113`: GitLab application 3 +- `10.6.0.121`: Prometheus + +## Configure the external load balancer + +NOTE: **Note:** +This architecture has been tested and validated with [HAProxy](https://www.haproxy.org/) +as the load balancer. Although other load balancers with similar feature sets +could also be used, those load balancers have not been validated. + +In an active/active GitLab configuration, you will need a load balancer to route +traffic to the application servers. The specifics on which load balancer to use +or the exact configuration is beyond the scope of GitLab documentation. We hope +that if you're managing multi-node systems like GitLab you have a load balancer of +choice already. Some examples including HAProxy (open-source), F5 Big-IP LTM, +and Citrix Net Scaler. This documentation will outline what ports and protocols +you need to use with GitLab. + +The next question is how you will handle SSL in your environment. +There are several different options: + +- [The application node terminates SSL](#application-node-terminates-ssl). +- [The load balancer terminates SSL without backend SSL](#load-balancer-terminates-ssl-without-backend-ssl) + and communication is not secure between the load balancer and the application node. +- [The load balancer terminates SSL with backend SSL](#load-balancer-terminates-ssl-with-backend-ssl) + and communication is *secure* between the load balancer and the application node. + +### Application node terminates SSL + +Configure your load balancer to pass connections on port 443 as `TCP` rather +than `HTTP(S)` protocol. This will pass the connection to the application node's +NGINX service untouched. NGINX will have the SSL certificate and listen on port 443. + +See the [NGINX HTTPS documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) +for details on managing SSL certificates and configuring NGINX. + +### Load balancer terminates SSL without backend SSL + +Configure your load balancer to use the `HTTP(S)` protocol rather than `TCP`. +The load balancer will then be responsible for managing SSL certificates and +terminating SSL. + +Since communication between the load balancer and GitLab will not be secure, +there is some additional configuration needed. See the +[NGINX proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#supporting-proxied-ssl) +for details. + +### Load balancer terminates SSL with backend SSL + +Configure your load balancer(s) to use the 'HTTP(S)' protocol rather than 'TCP'. +The load balancer(s) will be responsible for managing SSL certificates that +end users will see. + +Traffic will also be secure between the load balancer(s) and NGINX in this +scenario. There is no need to add configuration for proxied SSL since the +connection will be secure all the way. However, configuration will need to be +added to GitLab to configure SSL certificates. See +[NGINX HTTPS documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) +for details on managing SSL certificates and configuring NGINX. + +### Ports + +The basic ports to be used are shown in the table below. + +| LB Port | Backend Port | Protocol | +| ------- | ------------ | ------------------------ | +| 80 | 80 | HTTP (*1*) | +| 443 | 443 | TCP or HTTPS (*1*) (*2*) | +| 22 | 22 | TCP | + +- (*1*): [Web terminal](../../ci/environments/index.md#web-terminals) support requires + your load balancer to correctly handle WebSocket connections. When using + HTTP or HTTPS proxying, this means your load balancer must be configured + to pass through the `Connection` and `Upgrade` hop-by-hop headers. See the + [web terminal](../integration/terminal.md) integration guide for + more details. +- (*2*): When using HTTPS protocol for port 443, you will need to add an SSL + certificate to the load balancers. If you wish to terminate SSL at the + GitLab application server instead, use TCP protocol. + +If you're using GitLab Pages with custom domain support you will need some +additional port configurations. +GitLab Pages requires a separate virtual IP address. Configure DNS to point the +`pages_external_url` from `/etc/gitlab/gitlab.rb` at the new virtual IP address. See the +[GitLab Pages documentation](../pages/index.md) for more information. + +| LB Port | Backend Port | Protocol | +| ------- | ------------- | --------- | +| 80 | Varies (*1*) | HTTP | +| 443 | Varies (*1*) | TCP (*2*) | + +- (*1*): The backend port for GitLab Pages depends on the + `gitlab_pages['external_http']` and `gitlab_pages['external_https']` + setting. See [GitLab Pages documentation](../pages/index.md) for more details. +- (*2*): Port 443 for GitLab Pages should always use the TCP protocol. Users can + configure custom domains with custom SSL, which would not be possible + if SSL was terminated at the load balancer. + +#### Alternate SSH Port + +Some organizations have policies against opening SSH port 22. In this case, +it may be helpful to configure an alternate SSH hostname that allows users +to use SSH on port 443. An alternate SSH hostname will require a new virtual IP address +compared to the other GitLab HTTP configuration above. + +Configure DNS for an alternate SSH hostname such as `altssh.gitlab.example.com`. + +| LB Port | Backend Port | Protocol | +| ------- | ------------ | -------- | +| 443 | 22 | TCP | + + + +## Configure Consul + +The following IPs will be used as an example: + +- `10.6.0.11`: Consul 1 +- `10.6.0.12`: Consul 2 +- `10.6.0.13`: Consul 3 + +NOTE: **Note:** +The configuration processes for the other servers in your reference architecture will +use the `/etc/gitlab/gitlab-secrets.json` file from your Consul server to connect +with the other servers. + +To configure Consul: + +1. SSH into the server that will host Consul. +1. [Download/install](https://about.gitlab.com/install/) the + Omnibus GitLab Enterprise Edition package using **steps 1 and 2** from the + GitLab downloads page. + - Make sure you select the correct Omnibus package, with the same version + the GitLab application is running. + - Do not complete any other steps on the download page. + +1. Edit `/etc/gitlab/gitlab.rb` and add the contents: + + ```ruby + roles ['consul_role'] + + ## Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + server: true, + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + + # Disable auto migrations + gitlab_rails['auto_migrate'] = false + ``` + +1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Go through the steps again for all the other Consul nodes, and + make sure you set up the correct IPs. + +NOTE: **Note:** +A Consul leader will be elected when the provisioning of the third Consul server is completed. +Viewing the Consul logs `sudo gitlab-ctl tail consul` will display +`...[INFO] consul: New leader elected: ...` + +You can list the current Consul members (server, client): + +```shell +sudo /opt/gitlab/embedded/bin/consul members +``` + +You can verify the GitLab services are running: + +```shell +sudo gitlab-ctl status +``` + +The output should be similar to the following: + +```plaintext +run: consul: (pid 30074) 76834s; run: log: (pid 29740) 76844s +run: logrotate: (pid 30925) 3041s; run: log: (pid 29649) 76861s +run: node-exporter: (pid 30093) 76833s; run: log: (pid 29663) 76855s +``` + + + +## Configure PostgreSQL + +In this section, you'll be guided through configuring an external PostgreSQL database +to be used with GitLab. + +### Provide your own PostgreSQL instance + +If you're hosting GitLab on a cloud provider, you can optionally use a +managed service for PostgreSQL. For example, AWS offers a managed Relational +Database Service (RDS) that runs PostgreSQL. + +If you use a cloud-managed service, or provide your own PostgreSQL: + +1. Set up PostgreSQL according to the + [database requirements document](../../install/requirements.md#database). +1. Set up a `gitlab` username with a password of your choice. The `gitlab` user + needs privileges to create the `gitlabhq_production` database. +1. Configure the GitLab application servers with the appropriate details. + This step is covered in [Configuring the GitLab Rails application](#configure-gitlab-rails). + +### Standalone PostgreSQL using Omnibus GitLab + +The following IPs will be used as an example: + +- `10.6.0.21`: PostgreSQL primary +- `10.6.0.22`: PostgreSQL secondary 1 +- `10.6.0.23`: PostgreSQL secondary 2 + +First, make sure to [install](https://about.gitlab.com/install/) +the Linux GitLab package **on each node**. Following the steps, +install the necessary dependencies from step 1, and add the +GitLab package repository from step 2. When installing GitLab +in the second step, do not supply the `EXTERNAL_URL` value. + +#### PostgreSQL primary node + +1. SSH into the PostgreSQL primary node. +1. Generate a password hash for the PostgreSQL username/password pair. This assumes you will use the default + username of `gitlab` (recommended). The command will request a password + and confirmation. Use the value that is output by this command in the next + step as the value of ``: + + ```shell + sudo gitlab-ctl pg-password-md5 gitlab + ``` + +1. Generate a password hash for the PgBouncer username/password pair. This assumes you will use the default + username of `pgbouncer` (recommended). The command will request a password + and confirmation. Use the value that is output by this command in the next + step as the value of ``: + + ```shell + sudo gitlab-ctl pg-password-md5 pgbouncer + ``` + +1. Generate a password hash for the Consul database username/password pair. This assumes you will use the default + username of `gitlab-consul` (recommended). The command will request a password + and confirmation. Use the value that is output by this command in the next + step as the value of ``: + + ```shell + sudo gitlab-ctl pg-password-md5 gitlab-consul + ``` + +1. On the primary database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: + + ```ruby + # Disable all components except PostgreSQL and Repmgr and Consul + roles ['postgres_role'] + + # PostgreSQL configuration + postgresql['listen_address'] = '0.0.0.0' + postgresql['hot_standby'] = 'on' + postgresql['wal_level'] = 'replica' + postgresql['shared_preload_libraries'] = 'repmgr_funcs' + + # Disable automatic database migrations + gitlab_rails['auto_migrate'] = false + + # Configure the Consul agent + consul['services'] = %w(postgresql) + + # START user configuration + # Please set the real values as explained in Required Information section + # + # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value + postgresql['pgbouncer_user_password'] = '' + # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value + postgresql['sql_user_password'] = '' + # Set `max_wal_senders` to one more than the number of database nodes in the cluster. + # This is used to prevent replication from using up all of the + # available database connections. + postgresql['max_wal_senders'] = 4 + postgresql['max_replication_slots'] = 4 + + # Replace XXX.XXX.XXX.XXX/YY with Network Address + postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/24) + repmgr['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 10.6.0.0/24) + + ## Enable service discovery for Prometheus + consul['monitoring_service_discovery'] = true + + # Set the network addresses that the exporters will listen on for monitoring + node_exporter['listen_address'] = '0.0.0.0:9100' + postgres_exporter['listen_address'] = '0.0.0.0:9187' + + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + # + # END user configuration + ``` + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace + the file of the same name on this server. If that file is not on this server, + add the file from your Consul server to this server. + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + + + +#### PostgreSQL secondary nodes + +1. On both the secondary nodes, add the same configuration specified above for the primary node + with an additional setting (`repmgr['master_on_initialization'] = false`) that will inform `gitlab-ctl` that they are standby nodes initially + and there's no need to attempt to register them as a primary node: + + ```ruby + # Disable all components except PostgreSQL and Repmgr and Consul + roles ['postgres_role'] + + # PostgreSQL configuration + postgresql['listen_address'] = '0.0.0.0' + postgresql['hot_standby'] = 'on' + postgresql['wal_level'] = 'replica' + postgresql['shared_preload_libraries'] = 'repmgr_funcs' + + # Disable automatic database migrations + gitlab_rails['auto_migrate'] = false + + # Configure the Consul agent + consul['services'] = %w(postgresql) + + # Specify if a node should attempt to be primary on initialization. + repmgr['master_on_initialization'] = false + + # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value + postgresql['pgbouncer_user_password'] = '' + # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value + postgresql['sql_user_password'] = '' + # Set `max_wal_senders` to one more than the number of database nodes in the cluster. + # This is used to prevent replication from using up all of the + # available database connections. + postgresql['max_wal_senders'] = 4 + postgresql['max_replication_slots'] = 4 + + # Replace with your network addresses + postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/24) + repmgr['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 10.6.0.0/24) + + ## Enable service discovery for Prometheus + consul['monitoring_service_discovery'] = true + + # Set the network addresses that the exporters will listen on for monitoring + node_exporter['listen_address'] = '0.0.0.0:9100' + postgres_exporter['listen_address'] = '0.0.0.0:9187' + + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + ``` + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace + the file of the same name on this server. If that file is not on this server, + add the file from your Consul server to this server. + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/database.html) +are supported and can be added if needed. + + + +#### PostgreSQL post-configuration + +SSH into the **primary node**: + +1. Open a database prompt: + + ```shell + gitlab-psql -d gitlabhq_production + ``` + +1. Make sure the `pg_trgm` extension is enabled (it might already be): + + ```shell + CREATE EXTENSION pg_trgm; + ``` + +1. Exit the database prompt by typing `\q` and Enter. + +1. Verify the cluster is initialized with one node: + + ```shell + gitlab-ctl repmgr cluster show + ``` + + The output should be similar to the following: + + ```plaintext + Role | Name | Upstream | Connection String + ----------+----------|----------|---------------------------------------- + * master | HOSTNAME | | host=HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr + ``` + +1. Note down the hostname or IP address in the connection string: `host=HOSTNAME`. We will + refer to the hostname in the next section as ``. If the value + is not an IP address, it will need to be a resolvable name (via DNS or + `/etc/hosts`) + +SSH into the **secondary node**: + +1. Set up the repmgr standby: + + ```shell + gitlab-ctl repmgr standby setup + ``` + + Do note that this will remove the existing data on the node. The command + has a wait time. + + The output should be similar to the following: + + ```console + Doing this will delete the entire contents of /var/opt/gitlab/postgresql/data + If this is not what you want, hit Ctrl-C now to exit + To skip waiting, rerun with the -w option + Sleeping for 30 seconds + Stopping the database + Removing the data + Cloning the data + Starting the database + Registering the node with the cluster + ok: run: repmgrd: (pid 19068) 0s + ``` + +Before moving on, make sure the databases are configured correctly. Run the +following command on the **primary** node to verify that replication is working +properly and the secondary nodes appear in the cluster: + +```shell +gitlab-ctl repmgr cluster show +``` + +The output should be similar to the following: + +```plaintext +Role | Name | Upstream | Connection String +----------+---------|-----------|------------------------------------------------ +* master | MASTER | | host= user=gitlab_repmgr dbname=gitlab_repmgr + standby | STANDBY | MASTER | host= user=gitlab_repmgr dbname=gitlab_repmgr + standby | STANDBY | MASTER | host= user=gitlab_repmgr dbname=gitlab_repmgr +``` + +If the 'Role' column for any node says "FAILED", check the +[Troubleshooting section](troubleshooting.md) before proceeding. + +Also, check that the `repmgr-check-master` command works successfully on each node: + +```shell +su - gitlab-consul +gitlab-ctl repmgr-check-master || echo 'This node is a standby repmgr node' +``` + +This command relies on exit codes to tell Consul whether a particular node is a master +or secondary. The most important thing here is that this command does not produce errors. +If there are errors it's most likely due to incorrect `gitlab-consul` database user permissions. +Check the [Troubleshooting section](troubleshooting.md) before proceeding. + + + +## Configure PgBouncer + +Now that the PostgreSQL servers are all set up, let's configure PgBouncer. +The following IPs will be used as an example: + +- `10.6.0.31`: PgBouncer 1 +- `10.6.0.32`: PgBouncer 2 +- `10.6.0.33`: PgBouncer 3 + +1. On each PgBouncer node, edit `/etc/gitlab/gitlab.rb`, and replace + `` and `` with the + password hashes you [set up previously](#postgresql-primary-node): + + ```ruby + # Disable all components except Pgbouncer and Consul agent + roles ['pgbouncer_role'] + + # Configure PgBouncer + pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) + + pgbouncer['users'] = { + 'gitlab-consul': { + password: '' + }, + 'pgbouncer': { + password: '' + } + } + + # Configure Consul agent + consul['watchers'] = %w(postgresql) + consul['enable'] = true + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) + } + + # Enable service discovery for Prometheus + consul['monitoring_service_discovery'] = true + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + ``` + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace + the file of the same name on this server. If that file is not on this server, + add the file from your Consul server to this server. + +1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + + NOTE: **Note:** + If an error `execute[generate databases.ini]` occurs, this is due to an existing + [known issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4713). + It will be resolved when you run a second `reconfigure` after the next step. + +1. Create a `.pgpass` file so Consul is able to + reload PgBouncer. Enter the PgBouncer password twice when asked: + + ```shell + gitlab-ctl write-pgpass --host 127.0.0.1 --database pgbouncer --user pgbouncer --hostuser gitlab-consul + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) once again + to resolve any potential errors from the previous steps. +1. Ensure each node is talking to the current primary: + + ```shell + gitlab-ctl pgb-console # You will be prompted for PGBOUNCER_PASSWORD + ``` + +1. Once the console prompt is available, run the following queries: + + ```shell + show databases ; show clients ; + ``` + + The output should be similar to the following: + + ```plaintext + name | host | port | database | force_user | pool_size | reserve_pool | pool_mode | max_connections | current_connections + ---------------------+-------------+------+---------------------+------------+-----------+--------------+-----------+-----------------+--------------------- + gitlabhq_production | MASTER_HOST | 5432 | gitlabhq_production | | 20 | 0 | | 0 | 0 + pgbouncer | | 6432 | pgbouncer | pgbouncer | 2 | 0 | statement | 0 | 0 + (2 rows) + + type | user | database | state | addr | port | local_addr | local_port | connect_time | request_time | ptr | link | remote_pid | tls + ------+-----------+---------------------+---------+----------------+-------+------------+------------+---------------------+---------------------+-----------+------+------------+----- + C | pgbouncer | pgbouncer | active | 127.0.0.1 | 56846 | 127.0.0.1 | 6432 | 2017-08-21 18:09:59 | 2017-08-21 18:10:48 | 0x22b3880 | | 0 | + (2 rows) + ``` + + + +### Configure the internal load balancer + +If you're running more than one PgBouncer node as recommended, then at this time you'll need to set +up a TCP internal load balancer to serve each correctly. + +The following IP will be used as an example: + +- `10.6.0.40`: Internal Load Balancer + +Here's how you could do it with [HAProxy](https://www.haproxy.org/): + +```plaintext +global + log /dev/log local0 + log localhost local1 notice + log stdout format raw local0 + +defaults + log global + default-server inter 10s fall 3 rise 2 + balance leastconn + +frontend internal-pgbouncer-tcp-in + bind *:6432 + mode tcp + option tcplog + + default_backend pgbouncer + +backend pgbouncer + mode tcp + option tcp-check + + server pgbouncer1 10.6.0.21:6432 check + server pgbouncer2 10.6.0.22:6432 check + server pgbouncer3 10.6.0.23:6432 check +``` + +Refer to your preferred Load Balancer's documentation for further guidance. + + + +## Configure Redis + +Using [Redis](https://redis.io/) in scalable environment is possible using a **Primary** x **Replica** +topology with a [Redis Sentinel](https://redis.io/topics/sentinel) service to watch and automatically +start the failover procedure. + +Redis requires authentication if used with Sentinel. See +[Redis Security](https://redis.io/topics/security) documentation for more +information. We recommend using a combination of a Redis password and tight +firewall rules to secure your Redis service. +You are highly encouraged to read the [Redis Sentinel](https://redis.io/topics/sentinel) documentation +before configuring Redis with GitLab to fully understand the topology and +architecture. + +The requirements for a Redis setup are the following: + +1. All Redis nodes must be able to talk to each other and accept incoming + connections over Redis (`6379`) and Sentinel (`26379`) ports (unless you + change the default ones). +1. The server that hosts the GitLab application must be able to access the + Redis nodes. +1. Protect the nodes from access from external networks + ([Internet](https://gitlab.com/gitlab-org/gitlab-foss/uploads/c4cc8cd353604bd80315f9384035ff9e/The_Internet_IT_Crowd.png)), + using a firewall. + +In this section, you'll be guided through configuring two external Redis clusters +to be used with GitLab. The following IPs will be used as an example: + +- `10.6.0.51`: Redis - Cache Primary +- `10.6.0.52`: Redis - Cache Replica 1 +- `10.6.0.53`: Redis - Cache Replica 2 +- `10.6.0.71`: Sentinel - Cache 1 +- `10.6.0.72`: Sentinel - Cache 2 +- `10.6.0.73`: Sentinel - Cache 3 +- `10.6.0.61`: Redis - Queues Primary +- `10.6.0.62`: Redis - Queues Replica 1 +- `10.6.0.63`: Redis - Queues Replica 2 +- `10.6.0.81`: Sentinel - Queues 1 +- `10.6.0.82`: Sentinel - Queues 2 +- `10.6.0.83`: Sentinel - Queues 3 + +NOTE: **Providing your own Redis instance:** +Managed Redis from cloud providers such as AWS ElastiCache will work. If these +services support high availability, be sure it is **not** the Redis Cluster type. +Redis version 5.0 or higher is required, as this is what ships with +Omnibus GitLab packages starting with GitLab 13.0. Older Redis versions +do not support an optional count argument to SPOP which is now required for +[Merge Trains](../../ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md). +Note the Redis node's IP address or hostname, port, and password (if required). +These will be necessary when configuring the +[GitLab application servers](#configure-gitlab-rails) later. + +### Configure the Redis and Sentinel Cache cluster + +This is the section where we install and set up the new Redis Cache instances. + +NOTE: **Note:** +Redis nodes (both primary and replica) will need the same password defined in +`redis['password']`. At any time during a failover the Sentinels can +reconfigure a node and change its status from primary to replica and vice versa. + +#### Configure the primary Redis Cache node + +1. SSH into the **Primary** Redis server. +1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page. + - Make sure you select the correct Omnibus package, with the same version + and type (Community, Enterprise editions) of your current install. + - Do not complete any other steps on the download page. + +1. Edit `/etc/gitlab/gitlab.rb` and add the contents: + + ```ruby + # Specify server role as 'redis_master_role' + roles ['redis_master_role'] + + # IP address pointing to a local IP that the other machines can reach to. + # You can also set bind to '0.0.0.0' which listen in all interfaces. + # If you really need to bind to an external accessible IP, make + # sure you add extra firewall rules to prevent unauthorized access. + redis['bind'] = '10.6.0.51' + + # Define a port so Redis can listen for TCP requests which will allow other + # machines to connect to it. + redis['port'] = 6379 + + # Set up password authentication for Redis (use the same password in all nodes). + redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + + ## Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + redis_exporter['listen_address'] = '0.0.0.0:9121' + + # Prevent database migrations from running on upgrade + gitlab_rails['auto_migrate'] = false + ``` + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace + the file of the same name on this server. If that file is not on this server, + add the file from your Consul server to this server. + +1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +NOTE: **Note:** +You can specify multiple roles like sentinel and Redis as: +`roles ['redis_sentinel_role', 'redis_master_role']`. +Read more about [roles](https://docs.gitlab.com/omnibus/roles/). + +#### Configure the replica Redis Cache nodes + +1. SSH into the **replica** Redis server. +1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page. + - Make sure you select the correct Omnibus package, with the same version + and type (Community, Enterprise editions) of your current install. + - Do not complete any other steps on the download page. + +1. Edit `/etc/gitlab/gitlab.rb` and add the contents: + + ```ruby + # Specify server role as 'redis_replica_role' + roles ['redis_replica_role'] + + # IP address pointing to a local IP that the other machines can reach to. + # You can also set bind to '0.0.0.0' which listen in all interfaces. + # If you really need to bind to an external accessible IP, make + # sure you add extra firewall rules to prevent unauthorized access. + redis['bind'] = '10.6.0.52' + + # Define a port so Redis can listen for TCP requests which will allow other + # machines to connect to it. + redis['port'] = 6379 + + # The same password for Redis authentication you set up for the primary node. + redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + + # The IP of the primary Redis node. + redis['master_ip'] = '10.6.0.51' + + # Port of primary Redis server, uncomment to change to non default. Defaults + # to `6379`. + #redis['master_port'] = 6379 + + ## Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + redis_exporter['listen_address'] = '0.0.0.0:9121' + + # Prevent database migrations from running on upgrade + gitlab_rails['auto_migrate'] = false + ``` + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace + the file of the same name on this server. If that file is not on this server, + add the file from your Consul server to this server. + +1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Go through the steps again for all the other replica nodes, and + make sure to set up the IPs correctly. + +NOTE: **Note:** +You can specify multiple roles like sentinel and Redis as: +`roles ['redis_sentinel_role', 'redis_master_role']`. +Read more about [roles](https://docs.gitlab.com/omnibus/roles/). + +These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after +a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-cache-nodes), and even after a +`gitlab-ctl reconfigure`, they will get their configuration restored by +the same Sentinels. + +Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) +are supported and can be added if needed. + + + +#### Configure the Sentinel Cache nodes + +NOTE: **Note:** +If you are using an external Redis Sentinel instance, be sure +to exclude the `requirepass` parameter from the Sentinel +configuration. This parameter will cause clients to report `NOAUTH +Authentication required.`. [Redis Sentinel 3.2.x does not support +password authentication](https://github.com/antirez/redis/issues/3279). + +Now that the Redis servers are all set up, let's configure the Sentinel +servers. The following IPs will be used as an example: + +- `10.6.0.71`: Sentinel - Cache 1 +- `10.6.0.72`: Sentinel - Cache 2 +- `10.6.0.73`: Sentinel - Cache 3 + +To configure the Sentinel Cache server: + +1. SSH into the server that will host Consul/Sentinel. +1. [Download/install](https://about.gitlab.com/install/) the + Omnibus GitLab Enterprise Edition package using **steps 1 and 2** from the + GitLab downloads page. + - Make sure you select the correct Omnibus package, with the same version + the GitLab application is running. + - Do not complete any other steps on the download page. + +1. Edit `/etc/gitlab/gitlab.rb` and add the contents: + + ```ruby + roles ['redis_sentinel_role'] + + ## Must be the same in every sentinel node + redis['master_name'] = 'gitlab-redis-cache' + + ## The same password for Redis authentication you set up for the primary node. + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + + ## The IP of the primary Redis node. + redis['master_ip'] = '10.6.0.51' + + ## Define a port so Redis can listen for TCP requests which will allow other + ## machines to connect to it. + redis['port'] = 6379 + + ## Port of primary Redis server, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + + ## Configure Sentinel's IP + sentinel['bind'] = '10.6.0.71' + + ## Port that Sentinel listens on, uncomment to change to non default. Defaults + ## to `26379`. + #sentinel['port'] = 26379 + + ## Quorum must reflect the amount of voting sentinels it take to start a failover. + ## Value must NOT be greater then the amount of sentinels. + ## + ## The quorum can be used to tune Sentinel in two ways: + ## 1. If a the quorum is set to a value smaller than the majority of Sentinels + ## we deploy, we are basically making Sentinel more sensible to primary failures, + ## triggering a failover as soon as even just a minority of Sentinels is no longer + ## able to talk with the primary. + ## 1. If a quorum is set to a value greater than the majority of Sentinels, we are + ## making Sentinel able to failover only when there are a very large number (larger + ## than majority) of well connected Sentinels which agree about the primary being down.s + sentinel['quorum'] = 2 + + ## Consider unresponsive server down after x amount of ms. + #sentinel['down_after_milliseconds'] = 10000 + + ## Specifies the failover timeout in milliseconds. It is used in many ways: + ## + ## - The time needed to re-start a failover after a previous failover was + ## already tried against the same primary by a given Sentinel, is two + ## times the failover timeout. + ## + ## - The time needed for a replica replicating to a wrong primary according + ## to a Sentinel current configuration, to be forced to replicate + ## with the right primary, is exactly the failover timeout (counting since + ## the moment a Sentinel detected the misconfiguration). + ## + ## - The time needed to cancel a failover that is already in progress but + ## did not produced any configuration change (REPLICAOF NO ONE yet not + ## acknowledged by the promoted replica). + ## + ## - The maximum time a failover in progress waits for all the replica to be + ## reconfigured as replicas of the new primary. However even after this time + ## the replicas will be reconfigured by the Sentinels anyway, but not with + ## the exact parallel-syncs progression as specified. + #sentinel['failover_timeout'] = 60000 + + ## Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + redis_exporter['listen_address'] = '0.0.0.0:9121' + + # Disable auto migrations + gitlab_rails['auto_migrate'] = false + ``` + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace + the file of the same name on this server. If that file is not on this server, + add the file from your Consul server to this server. + +1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Go through the steps again for all the other Consul/Sentinel nodes, and + make sure you set up the correct IPs. + + + +### Configure the Redis and Sentinel Queues cluster + +This is the section where we install and set up the new Redis Queues instances. + +NOTE: **Note:** +Redis nodes (both primary and replica) will need the same password defined in +`redis['password']`. At any time during a failover the Sentinels can +reconfigure a node and change its status from primary to replica and vice versa. + +#### Configure the primary Redis Queues node + +1. SSH into the **Primary** Redis server. +1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page. + - Make sure you select the correct Omnibus package, with the same version + and type (Community, Enterprise editions) of your current install. + - Do not complete any other steps on the download page. + +1. Edit `/etc/gitlab/gitlab.rb` and add the contents: + + ```ruby + # Specify server role as 'redis_master_role' + roles ['redis_master_role'] + + # IP address pointing to a local IP that the other machines can reach to. + # You can also set bind to '0.0.0.0' which listen in all interfaces. + # If you really need to bind to an external accessible IP, make + # sure you add extra firewall rules to prevent unauthorized access. + redis['bind'] = '10.6.0.61' + + # Define a port so Redis can listen for TCP requests which will allow other + # machines to connect to it. + redis['port'] = 6379 + + # Set up password authentication for Redis (use the same password in all nodes). + redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + + ## Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + redis_exporter['listen_address'] = '0.0.0.0:9121' + ``` + +1. Only the primary GitLab application server should handle migrations. To + prevent database migrations from running on upgrade, add the following + configuration to your `/etc/gitlab/gitlab.rb` file: + + ```ruby + gitlab_rails['auto_migrate'] = false + ``` + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace + the file of the same name on this server. If that file is not on this server, + add the file from your Consul server to this server. + +1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +NOTE: **Note:** +You can specify multiple roles like sentinel and Redis as: +`roles ['redis_sentinel_role', 'redis_master_role']`. +Read more about [roles](https://docs.gitlab.com/omnibus/roles/). + +#### Configure the replica Redis Queues nodes + +1. SSH into the **replica** Redis Queue server. +1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page. + - Make sure you select the correct Omnibus package, with the same version + and type (Community, Enterprise editions) of your current install. + - Do not complete any other steps on the download page. + +1. Edit `/etc/gitlab/gitlab.rb` and add the contents: + + ```ruby + # Specify server role as 'redis_replica_role' + roles ['redis_replica_role'] + + # IP address pointing to a local IP that the other machines can reach to. + # You can also set bind to '0.0.0.0' which listen in all interfaces. + # If you really need to bind to an external accessible IP, make + # sure you add extra firewall rules to prevent unauthorized access. + redis['bind'] = '10.6.0.62' + + # Define a port so Redis can listen for TCP requests which will allow other + # machines to connect to it. + redis['port'] = 6379 + + # The same password for Redis authentication you set up for the primary node. + redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + + # The IP of the primary Redis node. + redis['master_ip'] = '10.6.0.61' + + # Port of primary Redis server, uncomment to change to non default. Defaults + # to `6379`. + #redis['master_port'] = 6379 + + ## Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + redis_exporter['listen_address'] = '0.0.0.0:9121' + + # Disable auto migrations + gitlab_rails['auto_migrate'] = false + ``` + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace + the file of the same name on this server. If that file is not on this server, + add the file from your Consul server to this server. + +1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Go through the steps again for all the other replica nodes, and + make sure to set up the IPs correctly. + +NOTE: **Note:** +You can specify multiple roles like sentinel and Redis as: +`roles ['redis_sentinel_role', 'redis_master_role']`. +Read more about [roles](https://docs.gitlab.com/omnibus/roles/). + +These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after +a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-queues-nodes), and even after a +`gitlab-ctl reconfigure`, they will get their configuration restored by +the same Sentinels. + +Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) +are supported and can be added if needed. + + + +#### Configure the Sentinel Queues nodes + +NOTE: **Note:** +If you are using an external Redis Sentinel instance, be sure +to exclude the `requirepass` parameter from the Sentinel +configuration. This parameter will cause clients to report `NOAUTH +Authentication required.`. [Redis Sentinel 3.2.x does not support +password authentication](https://github.com/antirez/redis/issues/3279). + +Now that the Redis servers are all set up, let's configure the Sentinel +servers. The following IPs will be used as an example: + +- `10.6.0.81`: Sentinel - Queues 1 +- `10.6.0.82`: Sentinel - Queues 2 +- `10.6.0.83`: Sentinel - Queues 3 + +To configure the Sentinel Queues server: + +1. SSH into the server that will host Sentinel. +1. [Download/install](https://about.gitlab.com/install/) the + Omnibus GitLab Enterprise Edition package using **steps 1 and 2** from the + GitLab downloads page. + - Make sure you select the correct Omnibus package, with the same version + the GitLab application is running. + - Do not complete any other steps on the download page. + +1. Edit `/etc/gitlab/gitlab.rb` and add the contents: + + ```ruby + roles ['redis_sentinel_role'] + + ## Must be the same in every sentinel node + redis['master_name'] = 'gitlab-redis-persistent' + + ## The same password for Redis authentication you set up for the primary node. + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + + ## The IP of the primary Redis node. + redis['master_ip'] = '10.6.0.61' + + ## Define a port so Redis can listen for TCP requests which will allow other + ## machines to connect to it. + redis['port'] = 6379 + + ## Port of primary Redis server, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + + ## Configure Sentinel's IP + sentinel['bind'] = '10.6.0.81' + + ## Port that Sentinel listens on, uncomment to change to non default. Defaults + ## to `26379`. + #sentinel['port'] = 26379 + + ## Quorum must reflect the amount of voting sentinels it take to start a failover. + ## Value must NOT be greater then the amount of sentinels. + ## + ## The quorum can be used to tune Sentinel in two ways: + ## 1. If a the quorum is set to a value smaller than the majority of Sentinels + ## we deploy, we are basically making Sentinel more sensible to primary failures, + ## triggering a failover as soon as even just a minority of Sentinels is no longer + ## able to talk with the primary. + ## 1. If a quorum is set to a value greater than the majority of Sentinels, we are + ## making Sentinel able to failover only when there are a very large number (larger + ## than majority) of well connected Sentinels which agree about the primary being down.s + sentinel['quorum'] = 2 + + ## Consider unresponsive server down after x amount of ms. + #sentinel['down_after_milliseconds'] = 10000 + + ## Specifies the failover timeout in milliseconds. It is used in many ways: + ## + ## - The time needed to re-start a failover after a previous failover was + ## already tried against the same primary by a given Sentinel, is two + ## times the failover timeout. + ## + ## - The time needed for a replica replicating to a wrong primary according + ## to a Sentinel current configuration, to be forced to replicate + ## with the right primary, is exactly the failover timeout (counting since + ## the moment a Sentinel detected the misconfiguration). + ## + ## - The time needed to cancel a failover that is already in progress but + ## did not produced any configuration change (REPLICAOF NO ONE yet not + ## acknowledged by the promoted replica). + ## + ## - The maximum time a failover in progress waits for all the replica to be + ## reconfigured as replicas of the new primary. However even after this time + ## the replicas will be reconfigured by the Sentinels anyway, but not with + ## the exact parallel-syncs progression as specified. + #sentinel['failover_timeout'] = 60000 + + ## Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + redis_exporter['listen_address'] = '0.0.0.0:9121' + + # Disable auto migrations + gitlab_rails['auto_migrate'] = false + ``` + +1. To prevent database migrations from running on upgrade, run: + + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure + ``` + + Only the primary GitLab application server should handle migrations. + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace + the file of the same name on this server. If that file is not on this server, + add the file from your Consul server to this server. + +1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Go through the steps again for all the other Sentinel nodes, and + make sure you set up the correct IPs. + + + +## Configure Gitaly + +Deploying Gitaly in its own server can benefit GitLab installations that are +larger than a single machine. + +The Gitaly node requirements are dependent on customer data, specifically the number of +projects and their repository sizes. Two nodes are recommended as an absolute minimum. +Each Gitaly node should store no more than 5TB of data and have the number of +[`gitaly-ruby` workers](../gitaly/index.md#gitaly-ruby) set to 20% of available CPUs. +Additional nodes should be considered in conjunction with a review of expected +data size and spread based on the recommendations above. + +It is also strongly recommended that all Gitaly nodes be set up with SSD disks with +a throughput of at least 8,000 IOPS for read operations and 2,000 IOPS for write, +as Gitaly has heavy I/O. These IOPS values are recommended only as a starter as with +time they may be adjusted higher or lower depending on the scale of your environment's workload. +If you're running the environment on a Cloud provider, you may need to refer to +their documentation on how to configure IOPS correctly. + +Some things to note: + +- The GitLab Rails application shards repositories into [repository storages](../repository_storage_paths.md). +- A Gitaly server can host one or more storages. +- A GitLab server can use one or more Gitaly servers. +- Gitaly addresses must be specified in such a way that they resolve + correctly for ALL Gitaly clients. +- Gitaly servers must not be exposed to the public internet, as Gitaly's network + traffic is unencrypted by default. The use of a firewall is highly recommended + to restrict access to the Gitaly server. Another option is to + [use TLS](#gitaly-tls-support). + +TIP: **Tip:** +For more information about Gitaly's history and network architecture see the +[standalone Gitaly documentation](../gitaly/index.md). + +Note: **Note:** +The token referred to throughout the Gitaly documentation is +just an arbitrary password selected by the administrator. It is unrelated to +tokens created for the GitLab API or other similar web API tokens. + +Below we describe how to configure two Gitaly servers, with IPs and +domain names: + +- `10.6.0.91`: Gitaly 1 (`gitaly1.internal`) +- `10.6.0.92`: Gitaly 2 (`gitaly2.internal`) + +The secret token is assumed to be `gitalysecret` and that +your GitLab installation has three repository storages: + +- `default` on Gitaly 1 +- `storage1` on Gitaly 1 +- `storage2` on Gitaly 2 + +On each node: + +1. [Download/Install](https://about.gitlab.com/install/) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page but + **without** providing the `EXTERNAL_URL` value. +1. Edit `/etc/gitlab/gitlab.rb` to configure storage paths, enable + the network listener and configure the token: + + + + ```ruby + # /etc/gitlab/gitlab.rb + + # Gitaly and GitLab use two shared secrets for authentication, one to authenticate gRPC requests + # to Gitaly, and a second for authentication callbacks from GitLab-Shell to the GitLab internal API. + # The following two values must be the same as their respective values + # of the GitLab Rails application setup + gitaly['auth_token'] = 'gitalysecret' + gitlab_shell['secret_token'] = 'shellsecret' + + # Avoid running unnecessary services on the Gitaly server + postgresql['enable'] = false + redis['enable'] = false + nginx['enable'] = false + puma['enable'] = false + unicorn['enable'] = false + sidekiq['enable'] = false + gitlab_workhorse['enable'] = false + grafana['enable'] = false + + # If you run a seperate monitoring node you can disable these services + alertmanager['enable'] = false + prometheus['enable'] = false + + # Prevent database connections during 'gitlab-ctl reconfigure' + gitlab_rails['rake_cache_clear'] = false + gitlab_rails['auto_migrate'] = false + + # Configure the gitlab-shell API callback URL. Without this, `git push` will + # fail. This can be your 'front door' GitLab URL or an internal load + # balancer. + # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server. + gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' + + # Make Gitaly accept connections on all network interfaces. You must use + # firewalls to restrict access to this address/port. + # Comment out following line if you only want to support TLS connections + gitaly['listen_addr'] = "0.0.0.0:8075" + ``` + +1. Append the following to `/etc/gitlab/gitlab.rb` for each respective server: + 1. On `gitaly1.internal`: + + ```ruby + git_data_dirs({ + 'default' => { + 'path' => '/var/opt/gitlab/git-data' + }, + 'storage1' => { + 'path' => '/mnt/gitlab/git-data' + }, + }) + ``` + + 1. On `gitaly2.internal`: + + ```ruby + git_data_dirs({ + 'storage2' => { + 'path' => '/mnt/gitlab/git-data' + }, + }) + ``` + + + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace + the file of the same name on this server. If that file is not on this server, + add the file from your Consul server to this server. + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +### Gitaly TLS support + +Gitaly supports TLS encryption. To be able to communicate +with a Gitaly instance that listens for secure connections you will need to use `tls://` URL +scheme in the `gitaly_address` of the corresponding storage entry in the GitLab configuration. + +You will need to bring your own certificates as this isn't provided automatically. +The certificate, or its certificate authority, must be installed on all Gitaly +nodes (including the Gitaly node using the certificate) and on all client nodes +that communicate with it following the procedure described in +[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). + +NOTE: **Note:** +The self-signed certificate must specify the address you use to access the +Gitaly server. If you are addressing the Gitaly server by a hostname, you can +either use the Common Name field for this, or add it as a Subject Alternative +Name. If you are addressing the Gitaly server by its IP address, you must add it +as a Subject Alternative Name to the certificate. +[gRPC does not support using an IP address as Common Name in a certificate](https://github.com/grpc/grpc/issues/2691). + +NOTE: **Note:** +It is possible to configure Gitaly servers with both an +unencrypted listening address `listen_addr` and an encrypted listening +address `tls_listen_addr` at the same time. This allows you to do a +gradual transition from unencrypted to encrypted traffic, if necessary. + +To configure Gitaly with TLS: + +1. Create the `/etc/gitlab/ssl` directory and copy your key and certificate there: + + ```shell + sudo mkdir -p /etc/gitlab/ssl + sudo chmod 755 /etc/gitlab/ssl + sudo cp key.pem cert.pem /etc/gitlab/ssl/ + sudo chmod 644 key.pem cert.pem + ``` + +1. Copy the cert to `/etc/gitlab/trusted-certs` so Gitaly will trust the cert when + calling into itself: + + ```shell + sudo cp /etc/gitlab/ssl/cert.pem /etc/gitlab/trusted-certs/ + ``` + +1. Edit `/etc/gitlab/gitlab.rb` and add: + + + + ```ruby + gitaly['tls_listen_addr'] = "0.0.0.0:9999" + gitaly['certificate_path'] = "/etc/gitlab/ssl/cert.pem" + gitaly['key_path'] = "/etc/gitlab/ssl/key.pem" + ``` + +1. Delete `gitaly['listen_addr']` to allow only encrypted connections. + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + + + +## Configure Sidekiq + +Sidekiq requires connections to the Redis, PostgreSQL and Gitaly instances. +The following IPs will be used as an example: + +- `10.6.0.101`: Sidekiq 1 +- `10.6.0.102`: Sidekiq 2 +- `10.6.0.103`: Sidekiq 3 +- `10.6.0.104`: Sidekiq 4 + +To configure the Sidekiq nodes, on each one: + +1. SSH into the Sidekiq server. +1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab package +you want using steps 1 and 2 from the GitLab downloads page. +**Do not complete any other steps on the download page.** +1. Open `/etc/gitlab/gitlab.rb` with your editor: + + ```ruby + ######################################## + ##### Services Disabled ### + ######################################## + + nginx['enable'] = false + grafana['enable'] = false + prometheus['enable'] = false + alertmanager['enable'] = false + gitaly['enable'] = false + gitlab_workhorse['enable'] = false + nginx['enable'] = false + puma['enable'] = false + postgres_exporter['enable'] = false + postgresql['enable'] = false + redis['enable'] = false + redis_exporter['enable'] = false + gitlab_exporter['enable'] = false + + ######################################## + #### Redis ### + ######################################## + + ## Redis connection details + ## First cluster that will host the cache + gitlab_rails['redis_cache_instance'] = 'redis://:@gitlab-redis-cache' + + gitlab_rails['redis_cache_sentinels'] = [ + {host: '10.6.0.71', port: 26379}, + {host: '10.6.0.72', port: 26379}, + {host: '10.6.0.73', port: 26379}, + ] + + ## Second cluster that will host the queues, shared state, and actioncable + gitlab_rails['redis_queues_instance'] = 'redis://:@gitlab-redis-persistent' + gitlab_rails['redis_shared_state_instance'] = 'redis://:@gitlab-redis-persistent' + gitlab_rails['redis_actioncable_instance'] = 'redis://:@gitlab-redis-persistent' + + gitlab_rails['redis_queues_sentinels'] = [ + {host: '10.6.0.81', port: 26379}, + {host: '10.6.0.82', port: 26379}, + {host: '10.6.0.83', port: 26379}, + ] + gitlab_rails['redis_shared_state_sentinels'] = [ + {host: '10.6.0.81', port: 26379}, + {host: '10.6.0.82', port: 26379}, + {host: '10.6.0.83', port: 26379}, + ] + gitlab_rails['redis_actioncable_sentinels'] = [ + {host: '10.6.0.81', port: 26379}, + {host: '10.6.0.82', port: 26379}, + {host: '10.6.0.83', port: 26379}, + ] + + ####################################### + ### Gitaly ### + ####################################### + + git_data_dirs({ + 'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, + 'storage1' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, + 'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' }, + }) + gitlab_rails['gitaly_token'] = 'YOUR_TOKEN' + + ####################################### + ### Postgres ### + ####################################### + gitlab_rails['db_host'] = '10.6.0.20' # internal load balancer IP + gitlab_rails['db_port'] = 6432 + gitlab_rails['db_password'] = '' + gitlab_rails['db_adapter'] = 'postgresql' + gitlab_rails['db_encoding'] = 'unicode' + gitlab_rails['auto_migrate'] = false + + ####################################### + ### Sidekiq configuration ### + ####################################### + sidekiq['listen_address'] = "0.0.0.0" + sidekiq['cluster'] = true # no need to set this after GitLab 13.0 + + ####################################### + ### Monitoring configuration ### + ####################################### + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) + } + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + + # Rails Status for prometheus + gitlab_rails['monitoring_whitelist'] = ['10.6.0.121/32', '127.0.0.0/8'] + ``` + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace + the file of the same name on this server. If that file is not on this server, + add the file from your Consul server to this server. + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +TIP: **Tip:** +You can also run [multiple Sidekiq processes](../operations/extra_sidekiq_processes.md). + + + +## Configure GitLab Rails + +NOTE: **Note:** +In our architectures we run each GitLab Rails node using the Puma webserver +and have its number of workers set to 90% of available CPUs along with four threads. For +nodes that are running Rails with other components the worker value should be reduced +accordingly where we've found 50% achieves a good balance but this is dependent +on workload. + +This section describes how to configure the GitLab application (Rails) component. + +The following IPs will be used as an example: + +- `10.6.0.111`: GitLab application 1 +- `10.6.0.112`: GitLab application 2 +- `10.6.0.113`: GitLab application 3 + +On each node perform the following: + +1. Download/install Omnibus GitLab using **steps 1 and 2** from + [GitLab downloads](https://about.gitlab.com/install/). Do not complete other + steps on the download page. + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace + the file of the same name on this server. If that file is not on this server, + add the file from your Consul server to this server. + +1. Edit `/etc/gitlab/gitlab.rb` and use the following configuration. + To maintain uniformity of links across nodes, the `external_url` + on the application server should point to the external URL that users will use + to access GitLab. This would be the URL of the [external load balancer](#configure-the-external-load-balancer) + which will route traffic to the GitLab application server: + + ```ruby + external_url 'https://gitlab.example.com' + + # Gitaly and GitLab use two shared secrets for authentication, one to authenticate gRPC requests + # to Gitaly, and a second for authentication callbacks from GitLab-Shell to the GitLab internal API. + # The following two values must be the same as their respective values + # of the Gitaly setup + gitlab_rails['gitaly_token'] = 'gitalysecret' + gitlab_shell['secret_token'] = 'shellsecret' + + git_data_dirs({ + 'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, + 'storage1' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, + 'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' }, + }) + + ## Disable components that will not be on the GitLab application server + roles ['application_role'] + gitaly['enable'] = false + nginx['enable'] = true + + ## PostgreSQL connection details + # Disable PostgreSQL on the application node + postgresql['enable'] = false + gitlab_rails['db_host'] = '10.6.0.20' # internal load balancer IP + gitlab_rails['db_port'] = 6432 + gitlab_rails['db_password'] = '' + gitlab_rails['auto_migrate'] = false + + ## Redis connection details + ## First cluster that will host the cache + gitlab_rails['redis_cache_instance'] = 'redis://:@gitlab-redis-cache' + + gitlab_rails['redis_cache_sentinels'] = [ + {host: '10.6.0.71', port: 26379}, + {host: '10.6.0.72', port: 26379}, + {host: '10.6.0.73', port: 26379}, + ] + + ## Second cluster that will host the queues, shared state, and actionable + gitlab_rails['redis_queues_instance'] = 'redis://:@gitlab-redis-persistent' + gitlab_rails['redis_shared_state_instance'] = 'redis://:@gitlab-redis-persistent' + gitlab_rails['redis_actioncable_instance'] = 'redis://:@gitlab-redis-persistent' + + gitlab_rails['redis_queues_sentinels'] = [ + {host: '10.6.0.81', port: 26379}, + {host: '10.6.0.82', port: 26379}, + {host: '10.6.0.83', port: 26379}, + ] + gitlab_rails['redis_shared_state_sentinels'] = [ + {host: '10.6.0.81', port: 26379}, + {host: '10.6.0.82', port: 26379}, + {host: '10.6.0.83', port: 26379}, + ] + gitlab_rails['redis_actioncable_sentinels'] = [ + {host: '10.6.0.81', port: 26379}, + {host: '10.6.0.82', port: 26379}, + {host: '10.6.0.83', port: 26379}, + ] + + # Set the network addresses that the exporters used for monitoring will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + gitlab_workhorse['prometheus_listen_addr'] = '0.0.0.0:9229' + sidekiq['listen_address'] = "0.0.0.0" + puma['listen'] = '0.0.0.0' + + # Add the monitoring node's IP address to the monitoring whitelist and allow it to + # scrape the NGINX metrics + gitlab_rails['monitoring_whitelist'] = ['10.6.0.121/32', '127.0.0.0/8'] + nginx['status']['options']['allow'] = ['10.6.0.121/32', '127.0.0.0/8'] + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. If you're using [Gitaly with TLS support](#gitaly-tls-support), make sure the + `git_data_dirs` entry is configured with `tls` instead of `tcp`: + + ```ruby + git_data_dirs({ + 'default' => { 'gitaly_address' => 'tls://gitaly1.internal:9999' }, + 'storage1' => { 'gitaly_address' => 'tls://gitaly1.internal:9999' }, + 'storage2' => { 'gitaly_address' => 'tls://gitaly2.internal:9999' }, + }) + ``` + + 1. Copy the cert into `/etc/gitlab/trusted-certs`: + + ```shell + sudo cp cert.pem /etc/gitlab/trusted-certs/ + ``` + +1. If you're [using NFS](#configure-nfs-optional): + 1. If necessary, install the NFS client utility packages using the following + commands: + + ```shell + # Ubuntu/Debian + apt-get install nfs-common + + # CentOS/Red Hat + yum install nfs-utils nfs-utils-lib + ``` + + 1. Specify the necessary NFS mounts in `/etc/fstab`. + The exact contents of `/etc/fstab` will depend on how you chose + to configure your NFS server. See the [NFS documentation](../high_availability/nfs.md) + for examples and the various options. + + 1. Create the shared directories. These may be different depending on your NFS + mount locations. + + ```shell + mkdir -p /var/opt/gitlab/.ssh /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/git-data + ``` + + 1. Edit `/etc/gitlab/gitlab.rb` and use the following configuration: + + ```ruby + ## Prevent GitLab from starting if NFS data mounts are not available + high_availability['mountpoint'] = '/var/opt/gitlab/git-data' + + ## Ensure UIDs and GIDs match between servers for permissions via NFS + user['uid'] = 9000 + user['gid'] = 9000 + web_server['uid'] = 9001 + web_server['gid'] = 9001 + registry['uid'] = 9002 + registry['gid'] = 9002 + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Confirm the node can connect to Gitaly: + + ```shell + sudo gitlab-rake gitlab:gitaly:check + ``` + + Then, tail the logs to see the requests: + + ```shell + sudo gitlab-ctl tail gitaly + ``` + +1. Optionally, from the Gitaly servers, confirm that Gitaly can perform callbacks to the internal API: + + ```shell + sudo /opt/gitlab/embedded/service/gitlab-shell/bin/check -config /opt/gitlab/embedded/service/gitlab-shell/config.yml + ``` + +NOTE: **Note:** +When you specify `https` in the `external_url`, as in the example +above, GitLab assumes you have SSL certificates in `/etc/gitlab/ssl/`. If +certificates are not present, NGINX will fail to start. See the +[NGINX documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) +for more information. + +### GitLab Rails post-configuration + +Initialize the GitLab database, by running the following in one of the Rails nodes: + +```shell +sudo gitlab-rake gitlab:db:configure +``` + +NOTE: **Note:** +If you encounter a `rake aborted!` error stating that PgBouncer is failing to connect to +PostgreSQL it may be that your PgBouncer node's IP address is missing from +PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` on your database nodes. See +[PgBouncer error `ERROR: pgbouncer cannot connect to server`](troubleshooting.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) +in the Troubleshooting section before proceeding. + + + +## Configure Prometheus + +The Omnibus GitLab package can be used to configure a standalone Monitoring node +running [Prometheus](../monitoring/prometheus/index.md) and +[Grafana](../monitoring/performance/grafana_configuration.md). + +The following IP will be used as an example: + +- `10.6.0.121`: Prometheus + +To configure the Monitoring node: + +1. SSH into the Monitoring node. +1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page. + Do not complete any other steps on the download page. + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace + the file of the same name on this server. If that file is not on this server, + add the file from your Consul server to this server. + +1. Edit `/etc/gitlab/gitlab.rb` and add the contents: + + ```ruby + external_url 'http://gitlab.example.com' + + # Disable all other services + gitlab_rails['auto_migrate'] = false + alertmanager['enable'] = false + gitaly['enable'] = false + gitlab_exporter['enable'] = false + gitlab_workhorse['enable'] = false + nginx['enable'] = true + postgres_exporter['enable'] = false + postgresql['enable'] = false + redis['enable'] = false + redis_exporter['enable'] = false + sidekiq['enable'] = false + puma['enable'] = false + unicorn['enable'] = false + node_exporter['enable'] = false + gitlab_exporter['enable'] = false + + # Enable Prometheus + prometheus['enable'] = true + prometheus['listen_address'] = '0.0.0.0:9090' + prometheus['monitor_kubernetes'] = false + + # Enable Login form + grafana['disable_login_form'] = false + + # Enable Grafana + grafana['enable'] = true + grafana['admin_password'] = '' + + # Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) + } + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. In the GitLab UI, set `admin/application_settings/metrics_and_profiling` > Metrics - Grafana to `/-/grafana` to +`http[s]:///-/grafana` + + + +## Configure the object storage + +GitLab supports using an object storage service for holding numerous types of data. +It's recommended over [NFS](#configure-nfs-optional) and in general it's better +in larger setups as object storage is typically much more performant, reliable, +and scalable. + +Object storage options that GitLab has tested, or is aware of customers using include: + +- SaaS/Cloud solutions such as [Amazon S3](https://aws.amazon.com/s3/), [Google cloud storage](https://cloud.google.com/storage). +- On-premises hardware and appliances from various storage vendors. +- MinIO. There is [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. + +For configuring GitLab to use Object Storage refer to the following guides +based on what features you intend to use: + +1. Configure [object storage for backups](../../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage). +1. Configure [object storage for job artifacts](../job_artifacts.md#using-object-storage) + including [incremental logging](../job_logs.md#new-incremental-logging-architecture). +1. Configure [object storage for LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage). +1. Configure [object storage for uploads](../uploads.md#using-object-storage-core-only). +1. Configure [object storage for merge request diffs](../merge_request_diffs.md#using-object-storage). +1. Configure [object storage for Container Registry](../packages/container_registry.md#use-object-storage) (optional feature). +1. Configure [object storage for Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage) (optional feature). +1. Configure [object storage for packages](../packages/index.md#using-object-storage) (optional feature). **(PREMIUM ONLY)** +1. Configure [object storage for Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature). **(PREMIUM ONLY)** +1. Configure [object storage for Pseudonymizer](../pseudonymizer.md#configuration) (optional feature). **(ULTIMATE ONLY)** +1. Configure [object storage for autoscale Runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional - for improved performance). +1. Configure [object storage for Terraform state files](../terraform_state.md#using-object-storage-core-only). + +Using separate buckets for each data type is the recommended approach for GitLab. + +A limitation of our configuration is that each use of object storage is separately configured. +[We have an issue for improving this](https://gitlab.com/gitlab-org/gitlab/-/issues/23345) +and easily using one bucket with separate folders is one improvement that this might bring. + +There is at least one specific issue with using the same bucket: +when GitLab is deployed with the Helm chart restore from backup +[will not properly function](https://docs.gitlab.com/charts/advanced/external-object-storage/#lfs-artifacts-uploads-packages-external-diffs-pseudonymizer) +unless separate buckets are used. + +One risk of using a single bucket would be if your organization decided to +migrate GitLab to the Helm deployment in the future. GitLab would run, but the situation with +backups might not be realized until the organization had a critical requirement for the backups to +work. + + + +## Configure NFS (optional) + +[Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly) +are recommended over NFS wherever possible for improved performance. If you intend +to use GitLab Pages, this currently [requires NFS](troubleshooting.md#gitlab-pages-requires-nfs). + +See how to [configure NFS](../high_availability/nfs.md). + + + +## Troubleshooting + +See the [troubleshooting documentation](troubleshooting.md). + + diff --git a/doc/administration/reference_architectures/1k_users.md b/doc/administration/reference_architectures/1k_users.md index def23619a5c..d3cf5f49413 100644 --- a/doc/administration/reference_architectures/1k_users.md +++ b/doc/administration/reference_architectures/1k_users.md @@ -1,86 +1,49 @@ -# Reference architecture: up to 1,000 users +--- +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- -This page describes GitLab reference architecture for up to 1,000 users. -For a full list of reference architectures, see -[Available reference architectures](index.md#available-reference-architectures). - -> - **Supported users (approximate):** 1,000 -> - **High Availability:** False +# Reference architecture: up to 1,000 users **(CORE ONLY)** -| Users | Configuration([8](#footnotes)) | GCP | AWS | Azure | -|-------|------------------------------------|----------------|---------------------|------------------------| -| 500 | 4 vCPU, 3.6GB Memory | `n1-highcpu-4` | `c5.xlarge` | F4s v2 | -| 1000 | 8 vCPU, 7.2GB Memory | `n1-highcpu-8` | `c5.2xlarge` | F8s v2 | +This page describes GitLab reference architecture for up to 1,000 users. For a +full list of reference architectures, see +[Available reference architectures](index.md#available-reference-architectures). -In addition to the above, we recommend having at least -2GB of swap on your server, even if you currently have -enough available RAM. Having swap will help reduce the chance of errors occurring -if your available memory changes. We also recommend -configuring the kernel's swappiness setting -to a low value like `10` to make the most of your RAM while still having the swap -available when needed. +If you need to serve up to 1,000 users and you don't have strict availability +requirements, a single-node solution with +[frequent backups](index.md#automated-backups-core-only) is appropriate for +many organizations . -For situations where you need to serve up to 1,000 users, a single-node -solution with [frequent backups](index.md#automated-backups-core-only) is appropriate -for many organizations. With automatic backup of the GitLab repositories, -configuration, and the database, if you don't have strict availability -requirements, this is the ideal solution. +> - **Supported users (approximate):** 1,000 +> - **High Availability:** No + +| Users | Configuration | GCP | AWS | Azure | +|--------------|-------------------------|----------------|-----------------|----------------| +| Up to 500 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| Up to 1,000 | 8 vCPU, 7.2GB memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | + +The Google Cloud Platform (GCP) architectures were built and tested using the +[Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) +CPU platform. On different hardware you may find that adjustments, either lower +or higher, are required for your CPU or node counts. For more information, see +our [Sysbench](https://github.com/akopytov/sysbench)-based +[CPU benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). + +In addition to the stated configurations, we recommend having at least 2GB of +swap on your server, even if you currently have enough available memory. Having +swap will help reduce the chance of errors occurring if your available memory +changes. We also recommend configuring the kernel's swappiness setting to a +lower value (such as `10`) to make the most of your memory, while still having +the swap available when needed. ## Setup instructions -- For this default reference architecture, use the standard [installation instructions](../../install/README.md) to install GitLab. +For this default reference architecture, to install GitLab use the standard +[installation instructions](../../install/README.md). NOTE: **Note:** You can also optionally configure GitLab to use an [external PostgreSQL service](../postgresql/external.md) or an [external object storage service](../high_availability/object_storage.md) for added performance and reliability at a reduced complexity cost. - -## Footnotes - -1. In our architectures we run each GitLab Rails node using the Puma webserver - and have its number of workers set to 90% of available CPUs along with four threads. For - nodes that are running Rails with other components the worker value should be reduced - accordingly where we've found 50% achieves a good balance but this is dependent - on workload. - -1. Gitaly node requirements are dependent on customer data, specifically the number of - projects and their sizes. We recommend two nodes as an absolute minimum for HA environments - and at least four nodes should be used when supporting 50,000 or more users. - We also recommend that each Gitaly node should store no more than 5TB of data - and have the number of [`gitaly-ruby` workers](../gitaly/index.md#gitaly-ruby) - set to 20% of available CPUs. Additional nodes should be considered in conjunction - with a review of expected data size and spread based on the recommendations above. - -1. Recommended Redis setup differs depending on the size of the architecture. - For smaller architectures (less than 3,000 users) a single instance should suffice. - For medium sized installs (3,000 - 5,000) we suggest one Redis cluster for all - classes and that Redis Sentinel is hosted alongside Consul. - For larger architectures (10,000 users or more) we suggest running a separate - [Redis Cluster](../redis/replication_and_failover.md#running-multiple-redis-clusters) for the Cache class - and another for the Queues and Shared State classes respectively. We also recommend - that you run the Redis Sentinel clusters separately for each Redis Cluster. - -1. For data objects such as LFS, Uploads, Artifacts, etc. We recommend an [Object Storage service](../object_storage.md) - over NFS where possible, due to better performance and availability. - -1. NFS can be used as an alternative for both repository data (replacing Gitaly) and - object storage but this isn't typically recommended for performance reasons. Note however it is required for - [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/196). - -1. Our architectures have been tested and validated with [HAProxy](https://www.haproxy.org/) - as the load balancer. Although other load balancers with similar feature sets - could also be used, those load balancers have not been validated. - -1. We strongly recommend that any Gitaly or NFS nodes be set up with SSD disks over - HDD with a throughput of at least 8,000 IOPS for read operations and 2,000 IOPS for write - as these components have heavy I/O. These IOPS values are recommended only as a starter - as with time they may be adjusted higher or lower depending on the scale of your - environment's workload. If you're running the environment on a Cloud provider - you may need to refer to their documentation on how configure IOPS correctly. - -1. The architectures were built and tested with the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) - CPU platform on GCP. On different hardware you may find that adjustments, either lower - or higher, are required for your CPU or Node counts accordingly. For more information, a - [Sysbench](https://github.com/akopytov/sysbench) benchmark of the CPU can be found - [here](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index 17f4300eb03..1cfa2565893 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -1,76 +1,2047 @@ -# Reference architecture: up to 25,000 users +--- +reading_time: true +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- -This page describes GitLab reference architecture for up to 25,000 users. -For a full list of reference architectures, see +# Reference architecture: up to 25,000 users **(PREMIUM ONLY)** + +This page describes GitLab reference architecture for up to 25,000 users. For a +full list of reference architectures, see [Available reference architectures](index.md#available-reference-architectures). > - **Supported users (approximate):** 25,000 -> - **High Availability:** True -> - **Test RPS rates:** API: 500 RPS, Web: 50 RPS, Git: 50 RPS - -| Service | Nodes | Configuration ([8](#footnotes)) | GCP | AWS | Azure | -|--------------------------------------------------------------|-------|---------------------------------|------------------|-----------------------|----------------| -| GitLab Rails ([1](#footnotes)) | 5 | 32 vCPU, 28.8GB Memory | `n1-highcpu-32` | `c5.9xlarge` | F32s v2 | -| PostgreSQL | 3 | 8 vCPU, 30GB Memory | `n1-standard-8` | `m5.2xlarge` | D8s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | F2s v2 | -| Gitaly ([2](#footnotes)) ([5](#footnotes)) ([7](#footnotes)) | X | 32 vCPU, 120GB Memory | `n1-standard-32` | `m5.8xlarge` | D32s v3 | -| Redis ([3](#footnotes)) - Cache | 3 | 4 vCPU, 15GB Memory | `n1-standard-4` | `m5.xlarge` | D4s v3 | -| Redis ([3](#footnotes)) - Queues / Shared State | 3 | 4 vCPU, 15GB Memory | `n1-standard-4` | `m5.xlarge` | D4s v3 | -| Redis Sentinel ([3](#footnotes)) - Cache | 3 | 1 vCPU, 1.7GB Memory | `g1-small` | `t2.small` | B1MS | -| Redis Sentinel ([3](#footnotes)) - Queues / Shared State | 3 | 1 vCPU, 1.7GB Memory | `g1-small` | `t2.small` | B1MS | -| Consul | 3 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | F2s v2 | -| Sidekiq | 4 | 4 vCPU, 15GB Memory | `n1-standard-4` | `m5.xlarge` | D4s v3 | -| Object Storage ([4](#footnotes)) | - | - | - | - | - | -| NFS Server ([5](#footnotes)) ([7](#footnotes)) | 1 | 4 vCPU, 3.6GB Memory | `n1-highcpu-4` | `c5.xlarge` | F4s v2 | -| Monitoring node | 1 | 4 vCPU, 3.6GB Memory | `n1-highcpu-4` | `c5.xlarge` | F4s v2 | -| External load balancing node ([6](#footnotes)) | 1 | 4 vCPU, 3.6GB Memory | `n1-highcpu-4` | `c5.xlarge` | F4s v2 | -| Internal load balancing node ([6](#footnotes)) | 1 | 4 vCPU, 3.6GB Memory | `n1-highcpu-4` | `c5.xlarge` | F4s v2 | - -## Footnotes - -1. In our architectures we run each GitLab Rails node using the Puma webserver - and have its number of workers set to 90% of available CPUs along with four threads. For - nodes that are running Rails with other components the worker value should be reduced - accordingly where we've found 50% achieves a good balance but this is dependent - on workload. - -1. Gitaly node requirements are dependent on customer data, specifically the number of - projects and their sizes. We recommend two nodes as an absolute minimum for HA environments - and at least four nodes should be used when supporting 50,000 or more users. - We also recommend that each Gitaly node should store no more than 5TB of data - and have the number of [`gitaly-ruby` workers](../gitaly/index.md#gitaly-ruby) - set to 20% of available CPUs. Additional nodes should be considered in conjunction - with a review of expected data size and spread based on the recommendations above. - -1. Recommended Redis setup differs depending on the size of the architecture. - For smaller architectures (less than 3,000 users) a single instance should suffice. - For medium sized installs (3,000 - 5,000) we suggest one Redis cluster for all - classes and that Redis Sentinel is hosted alongside Consul. - For larger architectures (10,000 users or more) we suggest running a separate - [Redis Cluster](../redis/replication_and_failover.md#running-multiple-redis-clusters) for the Cache class - and another for the Queues and Shared State classes respectively. We also recommend - that you run the Redis Sentinel clusters separately for each Redis Cluster. - -1. For data objects such as LFS, Uploads, Artifacts, etc. We recommend an [Object Storage service](../object_storage.md) - over NFS where possible, due to better performance and availability. - -1. NFS can be used as an alternative for both repository data (replacing Gitaly) and - object storage but this isn't typically recommended for performance reasons. Note however it is required for - [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/196). - -1. Our architectures have been tested and validated with [HAProxy](https://www.haproxy.org/) - as the load balancer. Although other load balancers with similar feature sets - could also be used, those load balancers have not been validated. - -1. We strongly recommend that any Gitaly or NFS nodes be set up with SSD disks over - HDD with a throughput of at least 8,000 IOPS for read operations and 2,000 IOPS for write - as these components have heavy I/O. These IOPS values are recommended only as a starter - as with time they may be adjusted higher or lower depending on the scale of your - environment's workload. If you're running the environment on a Cloud provider - you may need to refer to their documentation on how configure IOPS correctly. - -1. The architectures were built and tested with the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) - CPU platform on GCP. On different hardware you may find that adjustments, either lower - or higher, are required for your CPU or Node counts accordingly. For more information, a - [Sysbench](https://github.com/akopytov/sysbench) benchmark of the CPU can be found - [here](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). +> - **High Availability:** Yes +> - **Test requests per second (RPS) rates:** API: 500 RPS, Web: 50 RPS, Git: 50 RPS + +| Service | Nodes | Configuration | GCP | AWS | Azure | +|-----------------------------------------|-------------|-------------------------|-----------------|-------------|----------| +| External load balancing node | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| Consul | 3 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| PostgreSQL | 3 | 8 vCPU, 30GB memory | n1-standard-8 | m5.2xlarge | D8s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Internal load balancing node | 1 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Redis - Cache | 3 | 4 vCPU, 15GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| Redis - Queues / Shared State | 3 | 4 vCPU, 15GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7GB memory | g1-small | t2.small | B1MS | +| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7GB memory | g1-small | t2.small | B1MS | +| Gitaly | 2 (minimum) | 32 vCPU, 120GB memory | n1-standard-32 | m5.8xlarge | D32s v3 | +| Sidekiq | 4 | 4 vCPU, 15GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| GitLab Rails | 5 | 32 vCPU, 28.8GB memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | +| Monitoring node | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| Object Storage | n/a | n/a | n/a | n/a | n/a | +| NFS Server | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | + +The Google Cloud Platform (GCP) architectures were built and tested using the +[Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) +CPU platform. On different hardware you may find that adjustments, either lower +or higher, are required for your CPU or node counts. For more information, see +our [Sysbench](https://github.com/akopytov/sysbench)-based +[CPU benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). + +For data objects (such as LFS, Uploads, or Artifacts), an +[object storage service](#configure-the-object-storage) is recommended instead +of NFS where possible, due to better performance and availability. Since this +doesn't require a node to be set up, *Object Storage* is noted as not +applicable (n/a) in the previous table. + +## Setup components + +To set up GitLab and its components to accommodate up to 25,000 users: + +1. [Configure the external load balancing node](#configure-the-external-load-balancer) + that will handle the load balancing of the three GitLab application services nodes. +1. [Configure Consul](#configure-consul). +1. [Configure PostgreSQL](#configure-postgresql), the database for GitLab. +1. [Configure PgBouncer](#configure-pgbouncer). +1. [Configure the internal load balancing node](#configure-the-internal-load-balancer) +1. [Configure Redis](#configure-redis). +1. [Configure Gitaly](#configure-gitaly), + which provides access to the Git repositories. +1. [Configure Sidekiq](#configure-sidekiq). +1. [Configure the main GitLab Rails application](#configure-gitlab-rails) + to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend requests (UI, API, Git + over HTTP/SSH). +1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab environment. +1. [Configure the Object Storage](#configure-the-object-storage) + used for shared data objects. +1. [Configure NFS (Optional)](#configure-nfs-optional) + to have shared disk storage service as an alternative to Gitaly and/or Object Storage (although + not recommended). NFS is required for GitLab Pages, you can skip this step if you're not using + that feature. + +We start with all servers on the same 10.6.0.0/24 private network range, they +can connect to each other freely on those addresses. + +Here is a list and description of each machine and the assigned IP: + +- `10.6.0.10`: External Load Balancer +- `10.6.0.11`: Consul 1 +- `10.6.0.12`: Consul 2 +- `10.6.0.13`: Consul 3 +- `10.6.0.21`: PostgreSQL primary +- `10.6.0.22`: PostgreSQL secondary 1 +- `10.6.0.23`: PostgreSQL secondary 2 +- `10.6.0.31`: PgBouncer 1 +- `10.6.0.32`: PgBouncer 2 +- `10.6.0.33`: PgBouncer 3 +- `10.6.0.40`: Internal Load Balancer +- `10.6.0.51`: Redis - Cache Primary +- `10.6.0.52`: Redis - Cache Replica 1 +- `10.6.0.53`: Redis - Cache Replica 2 +- `10.6.0.71`: Sentinel - Cache 1 +- `10.6.0.72`: Sentinel - Cache 2 +- `10.6.0.73`: Sentinel - Cache 3 +- `10.6.0.61`: Redis - Queues Primary +- `10.6.0.62`: Redis - Queues Replica 1 +- `10.6.0.63`: Redis - Queues Replica 2 +- `10.6.0.81`: Sentinel - Queues 1 +- `10.6.0.82`: Sentinel - Queues 2 +- `10.6.0.83`: Sentinel - Queues 3 +- `10.6.0.91`: Gitaly 1 +- `10.6.0.92`: Gitaly 2 +- `10.6.0.101`: Sidekiq 1 +- `10.6.0.102`: Sidekiq 2 +- `10.6.0.103`: Sidekiq 3 +- `10.6.0.104`: Sidekiq 4 +- `10.6.0.111`: GitLab application 1 +- `10.6.0.112`: GitLab application 2 +- `10.6.0.113`: GitLab application 3 +- `10.6.0.121`: Prometheus + +## Configure the external load balancer + +NOTE: **Note:** +This architecture has been tested and validated with [HAProxy](https://www.haproxy.org/) +as the load balancer. Although other load balancers with similar feature sets +could also be used, those load balancers have not been validated. + +In an active/active GitLab configuration, you will need a load balancer to route +traffic to the application servers. The specifics on which load balancer to use +or the exact configuration is beyond the scope of GitLab documentation. We hope +that if you're managing multi-node systems like GitLab you have a load balancer of +choice already. Some examples including HAProxy (open-source), F5 Big-IP LTM, +and Citrix Net Scaler. This documentation will outline what ports and protocols +you need to use with GitLab. + +The next question is how you will handle SSL in your environment. +There are several different options: + +- [The application node terminates SSL](#application-node-terminates-ssl). +- [The load balancer terminates SSL without backend SSL](#load-balancer-terminates-ssl-without-backend-ssl) + and communication is not secure between the load balancer and the application node. +- [The load balancer terminates SSL with backend SSL](#load-balancer-terminates-ssl-with-backend-ssl) + and communication is *secure* between the load balancer and the application node. + +### Application node terminates SSL + +Configure your load balancer to pass connections on port 443 as `TCP` rather +than `HTTP(S)` protocol. This will pass the connection to the application node's +NGINX service untouched. NGINX will have the SSL certificate and listen on port 443. + +See the [NGINX HTTPS documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) +for details on managing SSL certificates and configuring NGINX. + +### Load balancer terminates SSL without backend SSL + +Configure your load balancer to use the `HTTP(S)` protocol rather than `TCP`. +The load balancer will then be responsible for managing SSL certificates and +terminating SSL. + +Since communication between the load balancer and GitLab will not be secure, +there is some additional configuration needed. See the +[NGINX proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#supporting-proxied-ssl) +for details. + +### Load balancer terminates SSL with backend SSL + +Configure your load balancer(s) to use the 'HTTP(S)' protocol rather than 'TCP'. +The load balancer(s) will be responsible for managing SSL certificates that +end users will see. + +Traffic will also be secure between the load balancer(s) and NGINX in this +scenario. There is no need to add configuration for proxied SSL since the +connection will be secure all the way. However, configuration will need to be +added to GitLab to configure SSL certificates. See +[NGINX HTTPS documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) +for details on managing SSL certificates and configuring NGINX. + +### Ports + +The basic ports to be used are shown in the table below. + +| LB Port | Backend Port | Protocol | +| ------- | ------------ | ------------------------ | +| 80 | 80 | HTTP (*1*) | +| 443 | 443 | TCP or HTTPS (*1*) (*2*) | +| 22 | 22 | TCP | + +- (*1*): [Web terminal](../../ci/environments/index.md#web-terminals) support requires + your load balancer to correctly handle WebSocket connections. When using + HTTP or HTTPS proxying, this means your load balancer must be configured + to pass through the `Connection` and `Upgrade` hop-by-hop headers. See the + [web terminal](../integration/terminal.md) integration guide for + more details. +- (*2*): When using HTTPS protocol for port 443, you will need to add an SSL + certificate to the load balancers. If you wish to terminate SSL at the + GitLab application server instead, use TCP protocol. + +If you're using GitLab Pages with custom domain support you will need some +additional port configurations. +GitLab Pages requires a separate virtual IP address. Configure DNS to point the +`pages_external_url` from `/etc/gitlab/gitlab.rb` at the new virtual IP address. See the +[GitLab Pages documentation](../pages/index.md) for more information. + +| LB Port | Backend Port | Protocol | +| ------- | ------------- | --------- | +| 80 | Varies (*1*) | HTTP | +| 443 | Varies (*1*) | TCP (*2*) | + +- (*1*): The backend port for GitLab Pages depends on the + `gitlab_pages['external_http']` and `gitlab_pages['external_https']` + setting. See [GitLab Pages documentation](../pages/index.md) for more details. +- (*2*): Port 443 for GitLab Pages should always use the TCP protocol. Users can + configure custom domains with custom SSL, which would not be possible + if SSL was terminated at the load balancer. + +#### Alternate SSH Port + +Some organizations have policies against opening SSH port 22. In this case, +it may be helpful to configure an alternate SSH hostname that allows users +to use SSH on port 443. An alternate SSH hostname will require a new virtual IP address +compared to the other GitLab HTTP configuration above. + +Configure DNS for an alternate SSH hostname such as `altssh.gitlab.example.com`. + +| LB Port | Backend Port | Protocol | +| ------- | ------------ | -------- | +| 443 | 22 | TCP | + + + +## Configure Consul + +The following IPs will be used as an example: + +- `10.6.0.11`: Consul 1 +- `10.6.0.12`: Consul 2 +- `10.6.0.13`: Consul 3 + +NOTE: **Note:** +The configuration processes for the other servers in your reference architecture will +use the `/etc/gitlab/gitlab-secrets.json` file from your Consul server to connect +with the other servers. + +To configure Consul: + +1. SSH into the server that will host Consul. +1. [Download/install](https://about.gitlab.com/install/) the + Omnibus GitLab Enterprise Edition package using **steps 1 and 2** from the + GitLab downloads page. + - Make sure you select the correct Omnibus package, with the same version + the GitLab application is running. + - Do not complete any other steps on the download page. + +1. Edit `/etc/gitlab/gitlab.rb` and add the contents: + + ```ruby + roles ['consul_role'] + + ## Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + server: true, + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + + # Disable auto migrations + gitlab_rails['auto_migrate'] = false + ``` + +1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Go through the steps again for all the other Consul nodes, and + make sure you set up the correct IPs. + +NOTE: **Note:** +A Consul leader will be elected when the provisioning of the third Consul server is completed. +Viewing the Consul logs `sudo gitlab-ctl tail consul` will display +`...[INFO] consul: New leader elected: ...` + +You can list the current Consul members (server, client): + +```shell +sudo /opt/gitlab/embedded/bin/consul members +``` + +You can verify the GitLab services are running: + +```shell +sudo gitlab-ctl status +``` + +The output should be similar to the following: + +```plaintext +run: consul: (pid 30074) 76834s; run: log: (pid 29740) 76844s +run: logrotate: (pid 30925) 3041s; run: log: (pid 29649) 76861s +run: node-exporter: (pid 30093) 76833s; run: log: (pid 29663) 76855s +``` + + + +## Configure PostgreSQL + +In this section, you'll be guided through configuring an external PostgreSQL database +to be used with GitLab. + +### Provide your own PostgreSQL instance + +If you're hosting GitLab on a cloud provider, you can optionally use a +managed service for PostgreSQL. For example, AWS offers a managed Relational +Database Service (RDS) that runs PostgreSQL. + +If you use a cloud-managed service, or provide your own PostgreSQL: + +1. Set up PostgreSQL according to the + [database requirements document](../../install/requirements.md#database). +1. Set up a `gitlab` username with a password of your choice. The `gitlab` user + needs privileges to create the `gitlabhq_production` database. +1. Configure the GitLab application servers with the appropriate details. + This step is covered in [Configuring the GitLab Rails application](#configure-gitlab-rails). + +### Standalone PostgreSQL using Omnibus GitLab + +The following IPs will be used as an example: + +- `10.6.0.21`: PostgreSQL primary +- `10.6.0.22`: PostgreSQL secondary 1 +- `10.6.0.23`: PostgreSQL secondary 2 + +First, make sure to [install](https://about.gitlab.com/install/) +the Linux GitLab package **on each node**. Following the steps, +install the necessary dependencies from step 1, and add the +GitLab package repository from step 2. When installing GitLab +in the second step, do not supply the `EXTERNAL_URL` value. + +#### PostgreSQL primary node + +1. SSH into the PostgreSQL primary node. +1. Generate a password hash for the PostgreSQL username/password pair. This assumes you will use the default + username of `gitlab` (recommended). The command will request a password + and confirmation. Use the value that is output by this command in the next + step as the value of ``: + + ```shell + sudo gitlab-ctl pg-password-md5 gitlab + ``` + +1. Generate a password hash for the PgBouncer username/password pair. This assumes you will use the default + username of `pgbouncer` (recommended). The command will request a password + and confirmation. Use the value that is output by this command in the next + step as the value of ``: + + ```shell + sudo gitlab-ctl pg-password-md5 pgbouncer + ``` + +1. Generate a password hash for the Consul database username/password pair. This assumes you will use the default + username of `gitlab-consul` (recommended). The command will request a password + and confirmation. Use the value that is output by this command in the next + step as the value of ``: + + ```shell + sudo gitlab-ctl pg-password-md5 gitlab-consul + ``` + +1. On the primary database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: + + ```ruby + # Disable all components except PostgreSQL and Repmgr and Consul + roles ['postgres_role'] + + # PostgreSQL configuration + postgresql['listen_address'] = '0.0.0.0' + postgresql['hot_standby'] = 'on' + postgresql['wal_level'] = 'replica' + postgresql['shared_preload_libraries'] = 'repmgr_funcs' + + # Disable automatic database migrations + gitlab_rails['auto_migrate'] = false + + # Configure the Consul agent + consul['services'] = %w(postgresql) + + # START user configuration + # Please set the real values as explained in Required Information section + # + # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value + postgresql['pgbouncer_user_password'] = '' + # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value + postgresql['sql_user_password'] = '' + # Set `max_wal_senders` to one more than the number of database nodes in the cluster. + # This is used to prevent replication from using up all of the + # available database connections. + postgresql['max_wal_senders'] = 4 + postgresql['max_replication_slots'] = 4 + + # Replace XXX.XXX.XXX.XXX/YY with Network Address + postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/24) + repmgr['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 10.6.0.0/24) + + ## Enable service discovery for Prometheus + consul['monitoring_service_discovery'] = true + + # Set the network addresses that the exporters will listen on for monitoring + node_exporter['listen_address'] = '0.0.0.0:9100' + postgres_exporter['listen_address'] = '0.0.0.0:9187' + + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + # + # END user configuration + ``` + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace + the file of the same name on this server. If that file is not on this server, + add the file from your Consul server to this server. + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + + + +#### PostgreSQL secondary nodes + +1. On both the secondary nodes, add the same configuration specified above for the primary node + with an additional setting (`repmgr['master_on_initialization'] = false`) that will inform `gitlab-ctl` that they are standby nodes initially + and there's no need to attempt to register them as a primary node: + + ```ruby + # Disable all components except PostgreSQL and Repmgr and Consul + roles ['postgres_role'] + + # PostgreSQL configuration + postgresql['listen_address'] = '0.0.0.0' + postgresql['hot_standby'] = 'on' + postgresql['wal_level'] = 'replica' + postgresql['shared_preload_libraries'] = 'repmgr_funcs' + + # Disable automatic database migrations + gitlab_rails['auto_migrate'] = false + + # Configure the Consul agent + consul['services'] = %w(postgresql) + + # Specify if a node should attempt to be primary on initialization. + repmgr['master_on_initialization'] = false + + # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value + postgresql['pgbouncer_user_password'] = '' + # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value + postgresql['sql_user_password'] = '' + # Set `max_wal_senders` to one more than the number of database nodes in the cluster. + # This is used to prevent replication from using up all of the + # available database connections. + postgresql['max_wal_senders'] = 4 + postgresql['max_replication_slots'] = 4 + + # Replace with your network addresses + postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/24) + repmgr['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 10.6.0.0/24) + + ## Enable service discovery for Prometheus + consul['monitoring_service_discovery'] = true + + # Set the network addresses that the exporters will listen on for monitoring + node_exporter['listen_address'] = '0.0.0.0:9100' + postgres_exporter['listen_address'] = '0.0.0.0:9187' + + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + ``` + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace + the file of the same name on this server. If that file is not on this server, + add the file from your Consul server to this server. + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/database.html) +are supported and can be added if needed. + + + +#### PostgreSQL post-configuration + +SSH into the **primary node**: + +1. Open a database prompt: + + ```shell + gitlab-psql -d gitlabhq_production + ``` + +1. Make sure the `pg_trgm` extension is enabled (it might already be): + + ```shell + CREATE EXTENSION pg_trgm; + ``` + +1. Exit the database prompt by typing `\q` and Enter. + +1. Verify the cluster is initialized with one node: + + ```shell + gitlab-ctl repmgr cluster show + ``` + + The output should be similar to the following: + + ```plaintext + Role | Name | Upstream | Connection String + ----------+----------|----------|---------------------------------------- + * master | HOSTNAME | | host=HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr + ``` + +1. Note down the hostname or IP address in the connection string: `host=HOSTNAME`. We will + refer to the hostname in the next section as ``. If the value + is not an IP address, it will need to be a resolvable name (via DNS or + `/etc/hosts`) + +SSH into the **secondary node**: + +1. Set up the repmgr standby: + + ```shell + gitlab-ctl repmgr standby setup + ``` + + Do note that this will remove the existing data on the node. The command + has a wait time. + + The output should be similar to the following: + + ```console + Doing this will delete the entire contents of /var/opt/gitlab/postgresql/data + If this is not what you want, hit Ctrl-C now to exit + To skip waiting, rerun with the -w option + Sleeping for 30 seconds + Stopping the database + Removing the data + Cloning the data + Starting the database + Registering the node with the cluster + ok: run: repmgrd: (pid 19068) 0s + ``` + +Before moving on, make sure the databases are configured correctly. Run the +following command on the **primary** node to verify that replication is working +properly and the secondary nodes appear in the cluster: + +```shell +gitlab-ctl repmgr cluster show +``` + +The output should be similar to the following: + +```plaintext +Role | Name | Upstream | Connection String +----------+---------|-----------|------------------------------------------------ +* master | MASTER | | host= user=gitlab_repmgr dbname=gitlab_repmgr + standby | STANDBY | MASTER | host= user=gitlab_repmgr dbname=gitlab_repmgr + standby | STANDBY | MASTER | host= user=gitlab_repmgr dbname=gitlab_repmgr +``` + +If the 'Role' column for any node says "FAILED", check the +[Troubleshooting section](troubleshooting.md) before proceeding. + +Also, check that the `repmgr-check-master` command works successfully on each node: + +```shell +su - gitlab-consul +gitlab-ctl repmgr-check-master || echo 'This node is a standby repmgr node' +``` + +This command relies on exit codes to tell Consul whether a particular node is a master +or secondary. The most important thing here is that this command does not produce errors. +If there are errors it's most likely due to incorrect `gitlab-consul` database user permissions. +Check the [Troubleshooting section](troubleshooting.md) before proceeding. + + + +## Configure PgBouncer + +Now that the PostgreSQL servers are all set up, let's configure PgBouncer. +The following IPs will be used as an example: + +- `10.6.0.31`: PgBouncer 1 +- `10.6.0.32`: PgBouncer 2 +- `10.6.0.33`: PgBouncer 3 + +1. On each PgBouncer node, edit `/etc/gitlab/gitlab.rb`, and replace + `` and `` with the + password hashes you [set up previously](#postgresql-primary-node): + + ```ruby + # Disable all components except Pgbouncer and Consul agent + roles ['pgbouncer_role'] + + # Configure PgBouncer + pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) + + pgbouncer['users'] = { + 'gitlab-consul': { + password: '' + }, + 'pgbouncer': { + password: '' + } + } + + # Configure Consul agent + consul['watchers'] = %w(postgresql) + consul['enable'] = true + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) + } + + # Enable service discovery for Prometheus + consul['monitoring_service_discovery'] = true + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + ``` + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace + the file of the same name on this server. If that file is not on this server, + add the file from your Consul server to this server. + +1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + + NOTE: **Note:** + If an error `execute[generate databases.ini]` occurs, this is due to an existing + [known issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4713). + It will be resolved when you run a second `reconfigure` after the next step. + +1. Create a `.pgpass` file so Consul is able to + reload PgBouncer. Enter the PgBouncer password twice when asked: + + ```shell + gitlab-ctl write-pgpass --host 127.0.0.1 --database pgbouncer --user pgbouncer --hostuser gitlab-consul + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) once again + to resolve any potential errors from the previous steps. +1. Ensure each node is talking to the current primary: + + ```shell + gitlab-ctl pgb-console # You will be prompted for PGBOUNCER_PASSWORD + ``` + +1. Once the console prompt is available, run the following queries: + + ```shell + show databases ; show clients ; + ``` + + The output should be similar to the following: + + ```plaintext + name | host | port | database | force_user | pool_size | reserve_pool | pool_mode | max_connections | current_connections + ---------------------+-------------+------+---------------------+------------+-----------+--------------+-----------+-----------------+--------------------- + gitlabhq_production | MASTER_HOST | 5432 | gitlabhq_production | | 20 | 0 | | 0 | 0 + pgbouncer | | 6432 | pgbouncer | pgbouncer | 2 | 0 | statement | 0 | 0 + (2 rows) + + type | user | database | state | addr | port | local_addr | local_port | connect_time | request_time | ptr | link | remote_pid | tls + ------+-----------+---------------------+---------+----------------+-------+------------+------------+---------------------+---------------------+-----------+------+------------+----- + C | pgbouncer | pgbouncer | active | 127.0.0.1 | 56846 | 127.0.0.1 | 6432 | 2017-08-21 18:09:59 | 2017-08-21 18:10:48 | 0x22b3880 | | 0 | + (2 rows) + ``` + + + +### Configure the internal load balancer + +If you're running more than one PgBouncer node as recommended, then at this time you'll need to set +up a TCP internal load balancer to serve each correctly. + +The following IP will be used as an example: + +- `10.6.0.40`: Internal Load Balancer + +Here's how you could do it with [HAProxy](https://www.haproxy.org/): + +```plaintext +global + log /dev/log local0 + log localhost local1 notice + log stdout format raw local0 + +defaults + log global + default-server inter 10s fall 3 rise 2 + balance leastconn + +frontend internal-pgbouncer-tcp-in + bind *:6432 + mode tcp + option tcplog + + default_backend pgbouncer + +backend pgbouncer + mode tcp + option tcp-check + + server pgbouncer1 10.6.0.21:6432 check + server pgbouncer2 10.6.0.22:6432 check + server pgbouncer3 10.6.0.23:6432 check +``` + +Refer to your preferred Load Balancer's documentation for further guidance. + + + +## Configure Redis + +Using [Redis](https://redis.io/) in scalable environment is possible using a **Primary** x **Replica** +topology with a [Redis Sentinel](https://redis.io/topics/sentinel) service to watch and automatically +start the failover procedure. + +Redis requires authentication if used with Sentinel. See +[Redis Security](https://redis.io/topics/security) documentation for more +information. We recommend using a combination of a Redis password and tight +firewall rules to secure your Redis service. +You are highly encouraged to read the [Redis Sentinel](https://redis.io/topics/sentinel) documentation +before configuring Redis with GitLab to fully understand the topology and +architecture. + +The requirements for a Redis setup are the following: + +1. All Redis nodes must be able to talk to each other and accept incoming + connections over Redis (`6379`) and Sentinel (`26379`) ports (unless you + change the default ones). +1. The server that hosts the GitLab application must be able to access the + Redis nodes. +1. Protect the nodes from access from external networks + ([Internet](https://gitlab.com/gitlab-org/gitlab-foss/uploads/c4cc8cd353604bd80315f9384035ff9e/The_Internet_IT_Crowd.png)), + using a firewall. + +In this section, you'll be guided through configuring two external Redis clusters +to be used with GitLab. The following IPs will be used as an example: + +- `10.6.0.51`: Redis - Cache Primary +- `10.6.0.52`: Redis - Cache Replica 1 +- `10.6.0.53`: Redis - Cache Replica 2 +- `10.6.0.71`: Sentinel - Cache 1 +- `10.6.0.72`: Sentinel - Cache 2 +- `10.6.0.73`: Sentinel - Cache 3 +- `10.6.0.61`: Redis - Queues Primary +- `10.6.0.62`: Redis - Queues Replica 1 +- `10.6.0.63`: Redis - Queues Replica 2 +- `10.6.0.81`: Sentinel - Queues 1 +- `10.6.0.82`: Sentinel - Queues 2 +- `10.6.0.83`: Sentinel - Queues 3 + +NOTE: **Providing your own Redis instance:** +Managed Redis from cloud providers such as AWS ElastiCache will work. If these +services support high availability, be sure it is **not** the Redis Cluster type. +Redis version 5.0 or higher is required, as this is what ships with +Omnibus GitLab packages starting with GitLab 13.0. Older Redis versions +do not support an optional count argument to SPOP which is now required for +[Merge Trains](../../ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md). +Note the Redis node's IP address or hostname, port, and password (if required). +These will be necessary when configuring the +[GitLab application servers](#configure-gitlab-rails) later. + +### Configure the Redis and Sentinel Cache cluster + +This is the section where we install and set up the new Redis Cache instances. + +NOTE: **Note:** +Redis nodes (both primary and replica) will need the same password defined in +`redis['password']`. At any time during a failover the Sentinels can +reconfigure a node and change its status from primary to replica and vice versa. + +#### Configure the primary Redis Cache node + +1. SSH into the **Primary** Redis server. +1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page. + - Make sure you select the correct Omnibus package, with the same version + and type (Community, Enterprise editions) of your current install. + - Do not complete any other steps on the download page. + +1. Edit `/etc/gitlab/gitlab.rb` and add the contents: + + ```ruby + # Specify server role as 'redis_master_role' + roles ['redis_master_role'] + + # IP address pointing to a local IP that the other machines can reach to. + # You can also set bind to '0.0.0.0' which listen in all interfaces. + # If you really need to bind to an external accessible IP, make + # sure you add extra firewall rules to prevent unauthorized access. + redis['bind'] = '10.6.0.51' + + # Define a port so Redis can listen for TCP requests which will allow other + # machines to connect to it. + redis['port'] = 6379 + + # Set up password authentication for Redis (use the same password in all nodes). + redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + + ## Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + redis_exporter['listen_address'] = '0.0.0.0:9121' + + # Prevent database migrations from running on upgrade + gitlab_rails['auto_migrate'] = false + ``` + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace + the file of the same name on this server. If that file is not on this server, + add the file from your Consul server to this server. + +1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +NOTE: **Note:** +You can specify multiple roles like sentinel and Redis as: +`roles ['redis_sentinel_role', 'redis_master_role']`. +Read more about [roles](https://docs.gitlab.com/omnibus/roles/). + +#### Configure the replica Redis Cache nodes + +1. SSH into the **replica** Redis server. +1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page. + - Make sure you select the correct Omnibus package, with the same version + and type (Community, Enterprise editions) of your current install. + - Do not complete any other steps on the download page. + +1. Edit `/etc/gitlab/gitlab.rb` and add the contents: + + ```ruby + # Specify server role as 'redis_replica_role' + roles ['redis_replica_role'] + + # IP address pointing to a local IP that the other machines can reach to. + # You can also set bind to '0.0.0.0' which listen in all interfaces. + # If you really need to bind to an external accessible IP, make + # sure you add extra firewall rules to prevent unauthorized access. + redis['bind'] = '10.6.0.52' + + # Define a port so Redis can listen for TCP requests which will allow other + # machines to connect to it. + redis['port'] = 6379 + + # The same password for Redis authentication you set up for the primary node. + redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + + # The IP of the primary Redis node. + redis['master_ip'] = '10.6.0.51' + + # Port of primary Redis server, uncomment to change to non default. Defaults + # to `6379`. + #redis['master_port'] = 6379 + + ## Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + redis_exporter['listen_address'] = '0.0.0.0:9121' + + # Prevent database migrations from running on upgrade + gitlab_rails['auto_migrate'] = false + ``` + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace + the file of the same name on this server. If that file is not on this server, + add the file from your Consul server to this server. + +1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Go through the steps again for all the other replica nodes, and + make sure to set up the IPs correctly. + +NOTE: **Note:** +You can specify multiple roles like sentinel and Redis as: +`roles ['redis_sentinel_role', 'redis_master_role']`. +Read more about [roles](https://docs.gitlab.com/omnibus/roles/). + +These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after +a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-cache-nodes), and even after a +`gitlab-ctl reconfigure`, they will get their configuration restored by +the same Sentinels. + +Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) +are supported and can be added if needed. + + + +#### Configure the Sentinel Cache nodes + +NOTE: **Note:** +If you are using an external Redis Sentinel instance, be sure +to exclude the `requirepass` parameter from the Sentinel +configuration. This parameter will cause clients to report `NOAUTH +Authentication required.`. [Redis Sentinel 3.2.x does not support +password authentication](https://github.com/antirez/redis/issues/3279). + +Now that the Redis servers are all set up, let's configure the Sentinel +servers. The following IPs will be used as an example: + +- `10.6.0.71`: Sentinel - Cache 1 +- `10.6.0.72`: Sentinel - Cache 2 +- `10.6.0.73`: Sentinel - Cache 3 + +To configure the Sentinel Cache server: + +1. SSH into the server that will host Consul/Sentinel. +1. [Download/install](https://about.gitlab.com/install/) the + Omnibus GitLab Enterprise Edition package using **steps 1 and 2** from the + GitLab downloads page. + - Make sure you select the correct Omnibus package, with the same version + the GitLab application is running. + - Do not complete any other steps on the download page. + +1. Edit `/etc/gitlab/gitlab.rb` and add the contents: + + ```ruby + roles ['redis_sentinel_role'] + + ## Must be the same in every sentinel node + redis['master_name'] = 'gitlab-redis-cache' + + ## The same password for Redis authentication you set up for the primary node. + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + + ## The IP of the primary Redis node. + redis['master_ip'] = '10.6.0.51' + + ## Define a port so Redis can listen for TCP requests which will allow other + ## machines to connect to it. + redis['port'] = 6379 + + ## Port of primary Redis server, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + + ## Configure Sentinel's IP + sentinel['bind'] = '10.6.0.71' + + ## Port that Sentinel listens on, uncomment to change to non default. Defaults + ## to `26379`. + #sentinel['port'] = 26379 + + ## Quorum must reflect the amount of voting sentinels it take to start a failover. + ## Value must NOT be greater then the amount of sentinels. + ## + ## The quorum can be used to tune Sentinel in two ways: + ## 1. If a the quorum is set to a value smaller than the majority of Sentinels + ## we deploy, we are basically making Sentinel more sensible to primary failures, + ## triggering a failover as soon as even just a minority of Sentinels is no longer + ## able to talk with the primary. + ## 1. If a quorum is set to a value greater than the majority of Sentinels, we are + ## making Sentinel able to failover only when there are a very large number (larger + ## than majority) of well connected Sentinels which agree about the primary being down.s + sentinel['quorum'] = 2 + + ## Consider unresponsive server down after x amount of ms. + #sentinel['down_after_milliseconds'] = 10000 + + ## Specifies the failover timeout in milliseconds. It is used in many ways: + ## + ## - The time needed to re-start a failover after a previous failover was + ## already tried against the same primary by a given Sentinel, is two + ## times the failover timeout. + ## + ## - The time needed for a replica replicating to a wrong primary according + ## to a Sentinel current configuration, to be forced to replicate + ## with the right primary, is exactly the failover timeout (counting since + ## the moment a Sentinel detected the misconfiguration). + ## + ## - The time needed to cancel a failover that is already in progress but + ## did not produced any configuration change (REPLICAOF NO ONE yet not + ## acknowledged by the promoted replica). + ## + ## - The maximum time a failover in progress waits for all the replica to be + ## reconfigured as replicas of the new primary. However even after this time + ## the replicas will be reconfigured by the Sentinels anyway, but not with + ## the exact parallel-syncs progression as specified. + #sentinel['failover_timeout'] = 60000 + + ## Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + redis_exporter['listen_address'] = '0.0.0.0:9121' + + # Disable auto migrations + gitlab_rails['auto_migrate'] = false + ``` + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace + the file of the same name on this server. If that file is not on this server, + add the file from your Consul server to this server. + +1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Go through the steps again for all the other Consul/Sentinel nodes, and + make sure you set up the correct IPs. + + + +### Configure the Redis and Sentinel Queues cluster + +This is the section where we install and set up the new Redis Queues instances. + +NOTE: **Note:** +Redis nodes (both primary and replica) will need the same password defined in +`redis['password']`. At any time during a failover the Sentinels can +reconfigure a node and change its status from primary to replica and vice versa. + +#### Configure the primary Redis Queues node + +1. SSH into the **Primary** Redis server. +1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page. + - Make sure you select the correct Omnibus package, with the same version + and type (Community, Enterprise editions) of your current install. + - Do not complete any other steps on the download page. + +1. Edit `/etc/gitlab/gitlab.rb` and add the contents: + + ```ruby + # Specify server role as 'redis_master_role' + roles ['redis_master_role'] + + # IP address pointing to a local IP that the other machines can reach to. + # You can also set bind to '0.0.0.0' which listen in all interfaces. + # If you really need to bind to an external accessible IP, make + # sure you add extra firewall rules to prevent unauthorized access. + redis['bind'] = '10.6.0.61' + + # Define a port so Redis can listen for TCP requests which will allow other + # machines to connect to it. + redis['port'] = 6379 + + # Set up password authentication for Redis (use the same password in all nodes). + redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + + ## Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + redis_exporter['listen_address'] = '0.0.0.0:9121' + ``` + +1. Only the primary GitLab application server should handle migrations. To + prevent database migrations from running on upgrade, add the following + configuration to your `/etc/gitlab/gitlab.rb` file: + + ```ruby + gitlab_rails['auto_migrate'] = false + ``` + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace + the file of the same name on this server. If that file is not on this server, + add the file from your Consul server to this server. + +1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +NOTE: **Note:** +You can specify multiple roles like sentinel and Redis as: +`roles ['redis_sentinel_role', 'redis_master_role']`. +Read more about [roles](https://docs.gitlab.com/omnibus/roles/). + +#### Configure the replica Redis Queues nodes + +1. SSH into the **replica** Redis Queue server. +1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page. + - Make sure you select the correct Omnibus package, with the same version + and type (Community, Enterprise editions) of your current install. + - Do not complete any other steps on the download page. + +1. Edit `/etc/gitlab/gitlab.rb` and add the contents: + + ```ruby + # Specify server role as 'redis_replica_role' + roles ['redis_replica_role'] + + # IP address pointing to a local IP that the other machines can reach to. + # You can also set bind to '0.0.0.0' which listen in all interfaces. + # If you really need to bind to an external accessible IP, make + # sure you add extra firewall rules to prevent unauthorized access. + redis['bind'] = '10.6.0.62' + + # Define a port so Redis can listen for TCP requests which will allow other + # machines to connect to it. + redis['port'] = 6379 + + # The same password for Redis authentication you set up for the primary node. + redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + + # The IP of the primary Redis node. + redis['master_ip'] = '10.6.0.61' + + # Port of primary Redis server, uncomment to change to non default. Defaults + # to `6379`. + #redis['master_port'] = 6379 + + ## Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + redis_exporter['listen_address'] = '0.0.0.0:9121' + + # Disable auto migrations + gitlab_rails['auto_migrate'] = false + ``` + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace + the file of the same name on this server. If that file is not on this server, + add the file from your Consul server to this server. + +1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Go through the steps again for all the other replica nodes, and + make sure to set up the IPs correctly. + +NOTE: **Note:** +You can specify multiple roles like sentinel and Redis as: +`roles ['redis_sentinel_role', 'redis_master_role']`. +Read more about [roles](https://docs.gitlab.com/omnibus/roles/). + +These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after +a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-queues-nodes), and even after a +`gitlab-ctl reconfigure`, they will get their configuration restored by +the same Sentinels. + +Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) +are supported and can be added if needed. + + + +#### Configure the Sentinel Queues nodes + +NOTE: **Note:** +If you are using an external Redis Sentinel instance, be sure +to exclude the `requirepass` parameter from the Sentinel +configuration. This parameter will cause clients to report `NOAUTH +Authentication required.`. [Redis Sentinel 3.2.x does not support +password authentication](https://github.com/antirez/redis/issues/3279). + +Now that the Redis servers are all set up, let's configure the Sentinel +servers. The following IPs will be used as an example: + +- `10.6.0.81`: Sentinel - Queues 1 +- `10.6.0.82`: Sentinel - Queues 2 +- `10.6.0.83`: Sentinel - Queues 3 + +To configure the Sentinel Queues server: + +1. SSH into the server that will host Sentinel. +1. [Download/install](https://about.gitlab.com/install/) the + Omnibus GitLab Enterprise Edition package using **steps 1 and 2** from the + GitLab downloads page. + - Make sure you select the correct Omnibus package, with the same version + the GitLab application is running. + - Do not complete any other steps on the download page. + +1. Edit `/etc/gitlab/gitlab.rb` and add the contents: + + ```ruby + roles ['redis_sentinel_role'] + + ## Must be the same in every sentinel node + redis['master_name'] = 'gitlab-redis-persistent' + + ## The same password for Redis authentication you set up for the primary node. + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + + ## The IP of the primary Redis node. + redis['master_ip'] = '10.6.0.61' + + ## Define a port so Redis can listen for TCP requests which will allow other + ## machines to connect to it. + redis['port'] = 6379 + + ## Port of primary Redis server, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + + ## Configure Sentinel's IP + sentinel['bind'] = '10.6.0.81' + + ## Port that Sentinel listens on, uncomment to change to non default. Defaults + ## to `26379`. + #sentinel['port'] = 26379 + + ## Quorum must reflect the amount of voting sentinels it take to start a failover. + ## Value must NOT be greater then the amount of sentinels. + ## + ## The quorum can be used to tune Sentinel in two ways: + ## 1. If a the quorum is set to a value smaller than the majority of Sentinels + ## we deploy, we are basically making Sentinel more sensible to primary failures, + ## triggering a failover as soon as even just a minority of Sentinels is no longer + ## able to talk with the primary. + ## 1. If a quorum is set to a value greater than the majority of Sentinels, we are + ## making Sentinel able to failover only when there are a very large number (larger + ## than majority) of well connected Sentinels which agree about the primary being down.s + sentinel['quorum'] = 2 + + ## Consider unresponsive server down after x amount of ms. + #sentinel['down_after_milliseconds'] = 10000 + + ## Specifies the failover timeout in milliseconds. It is used in many ways: + ## + ## - The time needed to re-start a failover after a previous failover was + ## already tried against the same primary by a given Sentinel, is two + ## times the failover timeout. + ## + ## - The time needed for a replica replicating to a wrong primary according + ## to a Sentinel current configuration, to be forced to replicate + ## with the right primary, is exactly the failover timeout (counting since + ## the moment a Sentinel detected the misconfiguration). + ## + ## - The time needed to cancel a failover that is already in progress but + ## did not produced any configuration change (REPLICAOF NO ONE yet not + ## acknowledged by the promoted replica). + ## + ## - The maximum time a failover in progress waits for all the replica to be + ## reconfigured as replicas of the new primary. However even after this time + ## the replicas will be reconfigured by the Sentinels anyway, but not with + ## the exact parallel-syncs progression as specified. + #sentinel['failover_timeout'] = 60000 + + ## Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + redis_exporter['listen_address'] = '0.0.0.0:9121' + + # Disable auto migrations + gitlab_rails['auto_migrate'] = false + ``` + +1. To prevent database migrations from running on upgrade, run: + + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure + ``` + + Only the primary GitLab application server should handle migrations. + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace + the file of the same name on this server. If that file is not on this server, + add the file from your Consul server to this server. + +1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Go through the steps again for all the other Sentinel nodes, and + make sure you set up the correct IPs. + + + +## Configure Gitaly + +Deploying Gitaly in its own server can benefit GitLab installations that are +larger than a single machine. + +The Gitaly node requirements are dependent on customer data, specifically the number of +projects and their repository sizes. Two nodes are recommended as an absolute minimum. +Each Gitaly node should store no more than 5TB of data and have the number of +[`gitaly-ruby` workers](../gitaly/index.md#gitaly-ruby) set to 20% of available CPUs. +Additional nodes should be considered in conjunction with a review of expected +data size and spread based on the recommendations above. + +It is also strongly recommended that all Gitaly nodes be set up with SSD disks with +a throughput of at least 8,000 IOPS for read operations and 2,000 IOPS for write, +as Gitaly has heavy I/O. These IOPS values are recommended only as a starter as with +time they may be adjusted higher or lower depending on the scale of your environment's workload. +If you're running the environment on a Cloud provider, you may need to refer to +their documentation on how to configure IOPS correctly. + +Some things to note: + +- The GitLab Rails application shards repositories into [repository storages](../repository_storage_paths.md). +- A Gitaly server can host one or more storages. +- A GitLab server can use one or more Gitaly servers. +- Gitaly addresses must be specified in such a way that they resolve + correctly for ALL Gitaly clients. +- Gitaly servers must not be exposed to the public internet, as Gitaly's network + traffic is unencrypted by default. The use of a firewall is highly recommended + to restrict access to the Gitaly server. Another option is to + [use TLS](#gitaly-tls-support). + +TIP: **Tip:** +For more information about Gitaly's history and network architecture see the +[standalone Gitaly documentation](../gitaly/index.md). + +Note: **Note:** +The token referred to throughout the Gitaly documentation is +just an arbitrary password selected by the administrator. It is unrelated to +tokens created for the GitLab API or other similar web API tokens. + +Below we describe how to configure two Gitaly servers, with IPs and +domain names: + +- `10.6.0.91`: Gitaly 1 (`gitaly1.internal`) +- `10.6.0.92`: Gitaly 2 (`gitaly2.internal`) + +The secret token is assumed to be `gitalysecret` and that +your GitLab installation has three repository storages: + +- `default` on Gitaly 1 +- `storage1` on Gitaly 1 +- `storage2` on Gitaly 2 + +On each node: + +1. [Download/Install](https://about.gitlab.com/install/) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page but + **without** providing the `EXTERNAL_URL` value. +1. Edit `/etc/gitlab/gitlab.rb` to configure storage paths, enable + the network listener and configure the token: + + + + ```ruby + # /etc/gitlab/gitlab.rb + + # Gitaly and GitLab use two shared secrets for authentication, one to authenticate gRPC requests + # to Gitaly, and a second for authentication callbacks from GitLab-Shell to the GitLab internal API. + # The following two values must be the same as their respective values + # of the GitLab Rails application setup + gitaly['auth_token'] = 'gitalysecret' + gitlab_shell['secret_token'] = 'shellsecret' + + # Avoid running unnecessary services on the Gitaly server + postgresql['enable'] = false + redis['enable'] = false + nginx['enable'] = false + puma['enable'] = false + unicorn['enable'] = false + sidekiq['enable'] = false + gitlab_workhorse['enable'] = false + grafana['enable'] = false + + # If you run a seperate monitoring node you can disable these services + alertmanager['enable'] = false + prometheus['enable'] = false + + # Prevent database connections during 'gitlab-ctl reconfigure' + gitlab_rails['rake_cache_clear'] = false + gitlab_rails['auto_migrate'] = false + + # Configure the gitlab-shell API callback URL. Without this, `git push` will + # fail. This can be your 'front door' GitLab URL or an internal load + # balancer. + # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server. + gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' + + # Make Gitaly accept connections on all network interfaces. You must use + # firewalls to restrict access to this address/port. + # Comment out following line if you only want to support TLS connections + gitaly['listen_addr'] = "0.0.0.0:8075" + ``` + +1. Append the following to `/etc/gitlab/gitlab.rb` for each respective server: + 1. On `gitaly1.internal`: + + ```ruby + git_data_dirs({ + 'default' => { + 'path' => '/var/opt/gitlab/git-data' + }, + 'storage1' => { + 'path' => '/mnt/gitlab/git-data' + }, + }) + ``` + + 1. On `gitaly2.internal`: + + ```ruby + git_data_dirs({ + 'storage2' => { + 'path' => '/mnt/gitlab/git-data' + }, + }) + ``` + + + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace + the file of the same name on this server. If that file is not on this server, + add the file from your Consul server to this server. + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +### Gitaly TLS support + +Gitaly supports TLS encryption. To be able to communicate +with a Gitaly instance that listens for secure connections you will need to use `tls://` URL +scheme in the `gitaly_address` of the corresponding storage entry in the GitLab configuration. + +You will need to bring your own certificates as this isn't provided automatically. +The certificate, or its certificate authority, must be installed on all Gitaly +nodes (including the Gitaly node using the certificate) and on all client nodes +that communicate with it following the procedure described in +[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). + +NOTE: **Note:** +The self-signed certificate must specify the address you use to access the +Gitaly server. If you are addressing the Gitaly server by a hostname, you can +either use the Common Name field for this, or add it as a Subject Alternative +Name. If you are addressing the Gitaly server by its IP address, you must add it +as a Subject Alternative Name to the certificate. +[gRPC does not support using an IP address as Common Name in a certificate](https://github.com/grpc/grpc/issues/2691). + +NOTE: **Note:** +It is possible to configure Gitaly servers with both an +unencrypted listening address `listen_addr` and an encrypted listening +address `tls_listen_addr` at the same time. This allows you to do a +gradual transition from unencrypted to encrypted traffic, if necessary. + +To configure Gitaly with TLS: + +1. Create the `/etc/gitlab/ssl` directory and copy your key and certificate there: + + ```shell + sudo mkdir -p /etc/gitlab/ssl + sudo chmod 755 /etc/gitlab/ssl + sudo cp key.pem cert.pem /etc/gitlab/ssl/ + sudo chmod 644 key.pem cert.pem + ``` + +1. Copy the cert to `/etc/gitlab/trusted-certs` so Gitaly will trust the cert when + calling into itself: + + ```shell + sudo cp /etc/gitlab/ssl/cert.pem /etc/gitlab/trusted-certs/ + ``` + +1. Edit `/etc/gitlab/gitlab.rb` and add: + + + + ```ruby + gitaly['tls_listen_addr'] = "0.0.0.0:9999" + gitaly['certificate_path'] = "/etc/gitlab/ssl/cert.pem" + gitaly['key_path'] = "/etc/gitlab/ssl/key.pem" + ``` + +1. Delete `gitaly['listen_addr']` to allow only encrypted connections. + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + + + +## Configure Sidekiq + +Sidekiq requires connections to the Redis, PostgreSQL and Gitaly instances. +The following IPs will be used as an example: + +- `10.6.0.101`: Sidekiq 1 +- `10.6.0.102`: Sidekiq 2 +- `10.6.0.103`: Sidekiq 3 +- `10.6.0.104`: Sidekiq 4 + +To configure the Sidekiq nodes, on each one: + +1. SSH into the Sidekiq server. +1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab package +you want using steps 1 and 2 from the GitLab downloads page. +**Do not complete any other steps on the download page.** +1. Open `/etc/gitlab/gitlab.rb` with your editor: + + ```ruby + ######################################## + ##### Services Disabled ### + ######################################## + + nginx['enable'] = false + grafana['enable'] = false + prometheus['enable'] = false + alertmanager['enable'] = false + gitaly['enable'] = false + gitlab_workhorse['enable'] = false + nginx['enable'] = false + puma['enable'] = false + postgres_exporter['enable'] = false + postgresql['enable'] = false + redis['enable'] = false + redis_exporter['enable'] = false + gitlab_exporter['enable'] = false + + ######################################## + #### Redis ### + ######################################## + + ## Redis connection details + ## First cluster that will host the cache + gitlab_rails['redis_cache_instance'] = 'redis://:@gitlab-redis-cache' + + gitlab_rails['redis_cache_sentinels'] = [ + {host: '10.6.0.71', port: 26379}, + {host: '10.6.0.72', port: 26379}, + {host: '10.6.0.73', port: 26379}, + ] + + ## Second cluster that will host the queues, shared state, and actioncable + gitlab_rails['redis_queues_instance'] = 'redis://:@gitlab-redis-persistent' + gitlab_rails['redis_shared_state_instance'] = 'redis://:@gitlab-redis-persistent' + gitlab_rails['redis_actioncable_instance'] = 'redis://:@gitlab-redis-persistent' + + gitlab_rails['redis_queues_sentinels'] = [ + {host: '10.6.0.81', port: 26379}, + {host: '10.6.0.82', port: 26379}, + {host: '10.6.0.83', port: 26379}, + ] + gitlab_rails['redis_shared_state_sentinels'] = [ + {host: '10.6.0.81', port: 26379}, + {host: '10.6.0.82', port: 26379}, + {host: '10.6.0.83', port: 26379}, + ] + gitlab_rails['redis_actioncable_sentinels'] = [ + {host: '10.6.0.81', port: 26379}, + {host: '10.6.0.82', port: 26379}, + {host: '10.6.0.83', port: 26379}, + ] + + ####################################### + ### Gitaly ### + ####################################### + + git_data_dirs({ + 'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, + 'storage1' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, + 'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' }, + }) + gitlab_rails['gitaly_token'] = 'YOUR_TOKEN' + + ####################################### + ### Postgres ### + ####################################### + gitlab_rails['db_host'] = '10.6.0.20' # internal load balancer IP + gitlab_rails['db_port'] = 6432 + gitlab_rails['db_password'] = '' + gitlab_rails['db_adapter'] = 'postgresql' + gitlab_rails['db_encoding'] = 'unicode' + gitlab_rails['auto_migrate'] = false + + ####################################### + ### Sidekiq configuration ### + ####################################### + sidekiq['listen_address'] = "0.0.0.0" + sidekiq['cluster'] = true # no need to set this after GitLab 13.0 + + ####################################### + ### Monitoring configuration ### + ####################################### + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) + } + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + + # Rails Status for prometheus + gitlab_rails['monitoring_whitelist'] = ['10.6.0.121/32', '127.0.0.0/8'] + ``` + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace + the file of the same name on this server. If that file is not on this server, + add the file from your Consul server to this server. + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +TIP: **Tip:** +You can also run [multiple Sidekiq processes](../operations/extra_sidekiq_processes.md). + + + +## Configure GitLab Rails + +NOTE: **Note:** +In our architectures we run each GitLab Rails node using the Puma webserver +and have its number of workers set to 90% of available CPUs along with four threads. For +nodes that are running Rails with other components the worker value should be reduced +accordingly where we've found 50% achieves a good balance but this is dependent +on workload. + +This section describes how to configure the GitLab application (Rails) component. + +The following IPs will be used as an example: + +- `10.6.0.111`: GitLab application 1 +- `10.6.0.112`: GitLab application 2 +- `10.6.0.113`: GitLab application 3 + +On each node perform the following: + +1. Download/install Omnibus GitLab using **steps 1 and 2** from + [GitLab downloads](https://about.gitlab.com/install/). Do not complete other + steps on the download page. + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace + the file of the same name on this server. If that file is not on this server, + add the file from your Consul server to this server. + +1. Edit `/etc/gitlab/gitlab.rb` and use the following configuration. + To maintain uniformity of links across nodes, the `external_url` + on the application server should point to the external URL that users will use + to access GitLab. This would be the URL of the [external load balancer](#configure-the-external-load-balancer) + which will route traffic to the GitLab application server: + + ```ruby + external_url 'https://gitlab.example.com' + + # Gitaly and GitLab use two shared secrets for authentication, one to authenticate gRPC requests + # to Gitaly, and a second for authentication callbacks from GitLab-Shell to the GitLab internal API. + # The following two values must be the same as their respective values + # of the Gitaly setup + gitlab_rails['gitaly_token'] = 'gitalysecret' + gitlab_shell['secret_token'] = 'shellsecret' + + git_data_dirs({ + 'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, + 'storage1' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, + 'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' }, + }) + + ## Disable components that will not be on the GitLab application server + roles ['application_role'] + gitaly['enable'] = false + nginx['enable'] = true + + ## PostgreSQL connection details + # Disable PostgreSQL on the application node + postgresql['enable'] = false + gitlab_rails['db_host'] = '10.6.0.20' # internal load balancer IP + gitlab_rails['db_port'] = 6432 + gitlab_rails['db_password'] = '' + gitlab_rails['auto_migrate'] = false + + ## Redis connection details + ## First cluster that will host the cache + gitlab_rails['redis_cache_instance'] = 'redis://:@gitlab-redis-cache' + + gitlab_rails['redis_cache_sentinels'] = [ + {host: '10.6.0.71', port: 26379}, + {host: '10.6.0.72', port: 26379}, + {host: '10.6.0.73', port: 26379}, + ] + + ## Second cluster that will host the queues, shared state, and actionable + gitlab_rails['redis_queues_instance'] = 'redis://:@gitlab-redis-persistent' + gitlab_rails['redis_shared_state_instance'] = 'redis://:@gitlab-redis-persistent' + gitlab_rails['redis_actioncable_instance'] = 'redis://:@gitlab-redis-persistent' + + gitlab_rails['redis_queues_sentinels'] = [ + {host: '10.6.0.81', port: 26379}, + {host: '10.6.0.82', port: 26379}, + {host: '10.6.0.83', port: 26379}, + ] + gitlab_rails['redis_shared_state_sentinels'] = [ + {host: '10.6.0.81', port: 26379}, + {host: '10.6.0.82', port: 26379}, + {host: '10.6.0.83', port: 26379}, + ] + gitlab_rails['redis_actioncable_sentinels'] = [ + {host: '10.6.0.81', port: 26379}, + {host: '10.6.0.82', port: 26379}, + {host: '10.6.0.83', port: 26379}, + ] + + # Set the network addresses that the exporters used for monitoring will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + gitlab_workhorse['prometheus_listen_addr'] = '0.0.0.0:9229' + sidekiq['listen_address'] = "0.0.0.0" + puma['listen'] = '0.0.0.0' + + # Add the monitoring node's IP address to the monitoring whitelist and allow it to + # scrape the NGINX metrics + gitlab_rails['monitoring_whitelist'] = ['10.6.0.121/32', '127.0.0.0/8'] + nginx['status']['options']['allow'] = ['10.6.0.121/32', '127.0.0.0/8'] + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. If you're using [Gitaly with TLS support](#gitaly-tls-support), make sure the + `git_data_dirs` entry is configured with `tls` instead of `tcp`: + + ```ruby + git_data_dirs({ + 'default' => { 'gitaly_address' => 'tls://gitaly1.internal:9999' }, + 'storage1' => { 'gitaly_address' => 'tls://gitaly1.internal:9999' }, + 'storage2' => { 'gitaly_address' => 'tls://gitaly2.internal:9999' }, + }) + ``` + + 1. Copy the cert into `/etc/gitlab/trusted-certs`: + + ```shell + sudo cp cert.pem /etc/gitlab/trusted-certs/ + ``` + +1. If you're [using NFS](#configure-nfs-optional): + 1. If necessary, install the NFS client utility packages using the following + commands: + + ```shell + # Ubuntu/Debian + apt-get install nfs-common + + # CentOS/Red Hat + yum install nfs-utils nfs-utils-lib + ``` + + 1. Specify the necessary NFS mounts in `/etc/fstab`. + The exact contents of `/etc/fstab` will depend on how you chose + to configure your NFS server. See the [NFS documentation](../high_availability/nfs.md) + for examples and the various options. + + 1. Create the shared directories. These may be different depending on your NFS + mount locations. + + ```shell + mkdir -p /var/opt/gitlab/.ssh /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/git-data + ``` + + 1. Edit `/etc/gitlab/gitlab.rb` and use the following configuration: + + ```ruby + ## Prevent GitLab from starting if NFS data mounts are not available + high_availability['mountpoint'] = '/var/opt/gitlab/git-data' + + ## Ensure UIDs and GIDs match between servers for permissions via NFS + user['uid'] = 9000 + user['gid'] = 9000 + web_server['uid'] = 9001 + web_server['gid'] = 9001 + registry['uid'] = 9002 + registry['gid'] = 9002 + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Confirm the node can connect to Gitaly: + + ```shell + sudo gitlab-rake gitlab:gitaly:check + ``` + + Then, tail the logs to see the requests: + + ```shell + sudo gitlab-ctl tail gitaly + ``` + +1. Optionally, from the Gitaly servers, confirm that Gitaly can perform callbacks to the internal API: + + ```shell + sudo /opt/gitlab/embedded/service/gitlab-shell/bin/check -config /opt/gitlab/embedded/service/gitlab-shell/config.yml + ``` + +NOTE: **Note:** +When you specify `https` in the `external_url`, as in the example +above, GitLab assumes you have SSL certificates in `/etc/gitlab/ssl/`. If +certificates are not present, NGINX will fail to start. See the +[NGINX documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) +for more information. + +### GitLab Rails post-configuration + +Initialize the GitLab database, by running the following in one of the Rails nodes: + +```shell +sudo gitlab-rake gitlab:db:configure +``` + +NOTE: **Note:** +If you encounter a `rake aborted!` error stating that PgBouncer is failing to connect to +PostgreSQL it may be that your PgBouncer node's IP address is missing from +PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` on your database nodes. See +[PgBouncer error `ERROR: pgbouncer cannot connect to server`](troubleshooting.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) +in the Troubleshooting section before proceeding. + + + +## Configure Prometheus + +The Omnibus GitLab package can be used to configure a standalone Monitoring node +running [Prometheus](../monitoring/prometheus/index.md) and +[Grafana](../monitoring/performance/grafana_configuration.md). + +The following IP will be used as an example: + +- `10.6.0.121`: Prometheus + +To configure the Monitoring node: + +1. SSH into the Monitoring node. +1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page. + Do not complete any other steps on the download page. + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace + the file of the same name on this server. If that file is not on this server, + add the file from your Consul server to this server. + +1. Edit `/etc/gitlab/gitlab.rb` and add the contents: + + ```ruby + external_url 'http://gitlab.example.com' + + # Disable all other services + gitlab_rails['auto_migrate'] = false + alertmanager['enable'] = false + gitaly['enable'] = false + gitlab_exporter['enable'] = false + gitlab_workhorse['enable'] = false + nginx['enable'] = true + postgres_exporter['enable'] = false + postgresql['enable'] = false + redis['enable'] = false + redis_exporter['enable'] = false + sidekiq['enable'] = false + puma['enable'] = false + unicorn['enable'] = false + node_exporter['enable'] = false + gitlab_exporter['enable'] = false + + # Enable Prometheus + prometheus['enable'] = true + prometheus['listen_address'] = '0.0.0.0:9090' + prometheus['monitor_kubernetes'] = false + + # Enable Login form + grafana['disable_login_form'] = false + + # Enable Grafana + grafana['enable'] = true + grafana['admin_password'] = '' + + # Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) + } + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. In the GitLab UI, set `admin/application_settings/metrics_and_profiling` > Metrics - Grafana to `/-/grafana` to +`http[s]:///-/grafana` + + + +## Configure the object storage + +GitLab supports using an object storage service for holding numerous types of data. +It's recommended over [NFS](#configure-nfs-optional) and in general it's better +in larger setups as object storage is typically much more performant, reliable, +and scalable. + +Object storage options that GitLab has tested, or is aware of customers using include: + +- SaaS/Cloud solutions such as [Amazon S3](https://aws.amazon.com/s3/), [Google cloud storage](https://cloud.google.com/storage). +- On-premises hardware and appliances from various storage vendors. +- MinIO. There is [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. + +For configuring GitLab to use Object Storage refer to the following guides +based on what features you intend to use: + +1. Configure [object storage for backups](../../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage). +1. Configure [object storage for job artifacts](../job_artifacts.md#using-object-storage) + including [incremental logging](../job_logs.md#new-incremental-logging-architecture). +1. Configure [object storage for LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage). +1. Configure [object storage for uploads](../uploads.md#using-object-storage-core-only). +1. Configure [object storage for merge request diffs](../merge_request_diffs.md#using-object-storage). +1. Configure [object storage for Container Registry](../packages/container_registry.md#use-object-storage) (optional feature). +1. Configure [object storage for Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage) (optional feature). +1. Configure [object storage for packages](../packages/index.md#using-object-storage) (optional feature). **(PREMIUM ONLY)** +1. Configure [object storage for Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature). **(PREMIUM ONLY)** +1. Configure [object storage for Pseudonymizer](../pseudonymizer.md#configuration) (optional feature). **(ULTIMATE ONLY)** +1. Configure [object storage for autoscale Runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional - for improved performance). +1. Configure [object storage for Terraform state files](../terraform_state.md#using-object-storage-core-only). + +Using separate buckets for each data type is the recommended approach for GitLab. + +A limitation of our configuration is that each use of object storage is separately configured. +[We have an issue for improving this](https://gitlab.com/gitlab-org/gitlab/-/issues/23345) +and easily using one bucket with separate folders is one improvement that this might bring. + +There is at least one specific issue with using the same bucket: +when GitLab is deployed with the Helm chart restore from backup +[will not properly function](https://docs.gitlab.com/charts/advanced/external-object-storage/#lfs-artifacts-uploads-packages-external-diffs-pseudonymizer) +unless separate buckets are used. + +One risk of using a single bucket would be if your organization decided to +migrate GitLab to the Helm deployment in the future. GitLab would run, but the situation with +backups might not be realized until the organization had a critical requirement for the backups to +work. + + + +## Configure NFS (optional) + +[Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly) +are recommended over NFS wherever possible for improved performance. If you intend +to use GitLab Pages, this currently [requires NFS](troubleshooting.md#gitlab-pages-requires-nfs). + +See how to [configure NFS](../high_availability/nfs.md). + + + +## Troubleshooting + +See the [troubleshooting documentation](troubleshooting.md). + + diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index d182daf45b3..a7feb78a365 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -1,27 +1,30 @@ --- reading_time: true +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Reference architecture: up to 2,000 users +# Reference architecture: up to 2,000 users **(CORE ONLY)** This page describes GitLab reference architecture for up to 2,000 users. For a full list of reference architectures, see [Available reference architectures](index.md#available-reference-architectures). > - **Supported users (approximate):** 2,000 -> - **High Availability:** False +> - **High Availability:** No > - **Test requests per second (RPS) rates:** API: 40 RPS, Web: 4 RPS, Git: 4 RPS -| Service | Nodes | Configuration | GCP | AWS | Azure | -|------------------------------------------|--------|-------------------------|-----------------|----------------|-----------| -| Load balancer | 1 | 2 vCPU, 1.8GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| PostgreSQL | 1 | 2 vCPU, 7.5GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | -| Redis | 1 | 1 vCPU, 3.75GB memory | `n1-standard-1` | `m5.large` | `D2s v3` | -| Gitaly | 1 | 4 vCPU, 15GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| GitLab Rails | 2 | 8 vCPU, 7.2GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | -| Monitoring node | 1 | 2 vCPU, 1.8GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Object storage | n/a | n/a | n/a | n/a | n/a | -| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| Service | Nodes | Configuration | GCP | AWS | Azure | +|------------------------------------------|--------|-------------------------|----------------|--------------|---------| +| Load balancer | 1 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| PostgreSQL | 1 | 2 vCPU, 7.5GB memory | n1-standard-2 | m5.large | D2s v3 | +| Redis | 1 | 1 vCPU, 3.75GB memory | n1-standard-1 | m5.large | D2s v3 | +| Gitaly | 1 | 4 vCPU, 15GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| GitLab Rails | 2 | 8 vCPU, 7.2GB memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | +| Monitoring node | 1 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Object storage | n/a | n/a | n/a | n/a | n/a | +| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | The Google Cloud Platform (GCP) architectures were built and tested using the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) @@ -554,7 +557,7 @@ On each node perform the following: 1. Specify the necessary NFS mounts in `/etc/fstab`. The exact contents of `/etc/fstab` will depend on how you chose - to configure your NFS server. See the [NFS documentation](../high_availability/nfs.md) + to configure your NFS server. See the [NFS documentation](../nfs.md) for examples and the various options. 1. Create the shared directories. These may be different depending on your NFS @@ -852,7 +855,7 @@ along with [Gitaly](#configure-gitaly), are recommended over using NFS whenever possible. However, if you intend to use GitLab Pages, [you must use NFS](troubleshooting.md#gitlab-pages-requires-nfs). -For information about configuring NFS, see the [NFS documentation page](../high_availability/nfs.md). +For information about configuring NFS, see the [NFS documentation page](../nfs.md).
diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index 04cb9fa4769..2f88413de6f 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -1,49 +1,54 @@ --- reading_time: true +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Reference architecture: up to 3,000 users +# Reference architecture: up to 3,000 users **(PREMIUM ONLY)** -This page describes GitLab reference architecture for up to 3,000 users. -For a full list of reference architectures, see +This page describes GitLab reference architecture for up to 3,000 users. For a +full list of reference architectures, see [Available reference architectures](index.md#available-reference-architectures). NOTE: **Note:** -The 3,000-user reference architecture documented below is -designed to help your organization achieve a highly-available GitLab deployment. -If you do not have the expertise or need to maintain a highly-available -environment, you can have a simpler and less costly-to-operate environment by -following the [2,000-user reference architecture](2k_users.md). +This reference architecture is designed to help your organization achieve a +highly-available GitLab deployment. If you do not have the expertise or need to +maintain a highly-available environment, you can have a simpler and less +costly-to-operate environment by using the +[2,000-user reference architecture](2k_users.md). > - **Supported users (approximate):** 3,000 -> - **High Availability:** True -> - **Test RPS rates:** API: 60 RPS, Web: 6 RPS, Git: 6 RPS - -| Service | Nodes | Configuration | GCP | AWS | Azure | -|--------------------------------------------------------------|-------|---------------------------------|-----------------|-------------------------|----------------| -| External load balancing node | 1 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Redis | 3 | 2 vCPU, 7.5GB Memory | `n1-standard-2` | `m5.large` | `D2s v3` | -| Consul + Sentinel | 3 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| PostgreSQL | 3 | 2 vCPU, 7.5GB Memory | `n1-standard-2` | `m5.large` | `D2s v3` | -| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Internal load balancing node | 1 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Gitaly | 2 minimum | 4 vCPU, 15GB Memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Sidekiq | 4 | 2 vCPU, 7.5GB Memory | `n1-standard-2` | `m5.large` | `D2s v3` | -| GitLab Rails | 3 | 8 vCPU, 7.2GB Memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | -| Monitoring node | 1 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Object Storage | n/a | n/a | n/a | n/a | n/a | -| NFS Server (optional, not recommended) | 1 | 4 vCPU, 3.6GB Memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | - -The architectures were built and tested with the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) -CPU platform on GCP. On different hardware you may find that adjustments, either lower -or higher, are required for your CPU or Node counts accordingly. For more information, a -[Sysbench](https://github.com/akopytov/sysbench) benchmark of the CPU can be found -[here](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). - -For data objects such as LFS, Uploads, Artifacts, etc, an [object storage service](#configure-the-object-storage) -is recommended over NFS where possible, due to better performance and availability. -Since this doesn't require a node to be set up, it's marked as not applicable (n/a) -in the table above. +> - **High Availability:** Yes +> - **Test requests per second (RPS) rates:** API: 60 RPS, Web: 6 RPS, Git: 6 RPS + +| Service | Nodes | Configuration | GCP | AWS | Azure | +|--------------------------------------------|-------------|-----------------------|----------------|-------------|---------| +| External load balancing node | 1 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Redis | 3 | 2 vCPU, 7.5GB memory | n1-standard-2 | m5.large | D2s v3 | +| Consul + Sentinel | 3 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| PostgreSQL | 3 | 2 vCPU, 7.5GB memory | n1-standard-2 | m5.large | D2s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Internal load balancing node | 1 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Gitaly | 2 (minimum) | 4 vCPU, 15GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| Sidekiq | 4 | 2 vCPU, 7.5GB memory | n1-standard-2 | m5.large | D2s v3 | +| GitLab Rails | 3 | 8 vCPU, 7.2GB memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | +| Monitoring node | 1 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Object Storage | n/a | n/a | n/a | n/a | n/a | +| NFS Server (optional, not recommended) | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | + +The Google Cloud Platform (GCP) architectures were built and tested using the +[Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) +CPU platform. On different hardware you may find that adjustments, either lower +or higher, are required for your CPU or node counts. For more information, see +our [Sysbench](https://github.com/akopytov/sysbench)-based +[CPU benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). + +For data objects (such as LFS, Uploads, or Artifacts), an +[object storage service](#configure-the-object-storage) is recommended instead +of NFS where possible, due to better performance and availability. Since this +doesn't require a node to be set up, *Object Storage* is noted as not +applicable (n/a) in the previous table. ## Setup components @@ -804,10 +809,11 @@ SSH into the **primary node**: gitlab-psql -d gitlabhq_production ``` -1. Enable the `pg_trgm` extension: +1. Enable the `pg_trgm` and `btree_gist` extensions: ```shell CREATE EXTENSION pg_trgm; + CREATE EXTENSION btree_gist; ``` 1. Exit the database prompt by typing `\q` and Enter. @@ -1439,7 +1445,7 @@ On each node perform the following: 1. Specify the necessary NFS mounts in `/etc/fstab`. The exact contents of `/etc/fstab` will depend on how you chose - to configure your NFS server. See the [NFS documentation](../high_availability/nfs.md) + to configure your NFS server. See the [NFS documentation](../nfs.md) for examples and the various options. 1. Create the shared directories. These may be different depending on your NFS @@ -1748,7 +1754,7 @@ work. are recommended over NFS wherever possible for improved performance. If you intend to use GitLab Pages, this currently [requires NFS](troubleshooting.md#gitlab-pages-requires-nfs). -See how to [configure NFS](../high_availability/nfs.md). +See how to [configure NFS](../nfs.md).
diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index 2540fe68dac..565845b4bf5 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -1,76 +1,2047 @@ -# Reference architecture: up to 50,000 users +--- +reading_time: true +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- -This page describes GitLab reference architecture for up to 50,000 users. -For a full list of reference architectures, see +# Reference architecture: up to 50,000 users **(PREMIUM ONLY)** + +This page describes GitLab reference architecture for up to 50,000 users. For a +full list of reference architectures, see [Available reference architectures](index.md#available-reference-architectures). > - **Supported users (approximate):** 50,000 -> - **High Availability:** True -> - **Test RPS rates:** API: 1000 RPS, Web: 100 RPS, Git: 100 RPS - -| Service | Nodes | Configuration ([8](#footnotes)) | GCP | AWS | Azure | -|--------------------------------------------------------------|-------|---------------------------------|----------------|-----------------------|----------------| -| GitLab Rails ([1](#footnotes)) | 12 | 32 vCPU, 28.8GB Memory | `n1-highcpu-32` | `c5.9xlarge` | F32s v2 | -| PostgreSQL | 3 | 16 vCPU, 60GB Memory | `n1-standard-16` | `m5.4xlarge` | D16s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | F2s v2 | -| Gitaly ([2](#footnotes)) ([5](#footnotes)) ([7](#footnotes)) | X | 64 vCPU, 240GB Memory | `n1-standard-64` | `m5.16xlarge` | D64s v3 | -| Redis ([3](#footnotes)) - Cache | 3 | 4 vCPU, 15GB Memory | `n1-standard-4` | `m5.xlarge` | D4s v3 | -| Redis ([3](#footnotes)) - Queues / Shared State | 3 | 4 vCPU, 15GB Memory | `n1-standard-4` | `m5.xlarge` | D4s v3 | -| Redis Sentinel ([3](#footnotes)) - Cache | 3 | 1 vCPU, 1.7GB Memory | `g1-small` | `t2.small` | B1MS | -| Redis Sentinel ([3](#footnotes)) - Queues / Shared State | 3 | 1 vCPU, 1.7GB Memory | `g1-small` | `t2.small` | B1MS | -| Consul | 3 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | F2s v2 | -| Sidekiq | 4 | 4 vCPU, 15GB Memory | `n1-standard-4` | `m5.xlarge` | D4s v3 | -| NFS Server ([5](#footnotes)) ([7](#footnotes)) | 1 | 4 vCPU, 3.6GB Memory | `n1-highcpu-4` | `c5.xlarge` | F4s v2 | -| Object Storage ([4](#footnotes)) | - | - | - | - | - | -| Monitoring node | 1 | 4 vCPU, 3.6GB Memory | `n1-highcpu-4` | `c5.xlarge` | F4s v2 | -| External load balancing node ([6](#footnotes)) | 1 | 8 vCPU, 7.2GB Memory | `n1-highcpu-8` | `c5.2xlarge` | F8s v2 | -| Internal load balancing node ([6](#footnotes)) | 1 | 8 vCPU, 7.2GB Memory | `n1-highcpu-8` | `c5.2xlarge` | F8s v2 | - -## Footnotes - -1. In our architectures we run each GitLab Rails node using the Puma webserver - and have its number of workers set to 90% of available CPUs along with four threads. For - nodes that are running Rails with other components the worker value should be reduced - accordingly where we've found 50% achieves a good balance but this is dependent - on workload. - -1. Gitaly node requirements are dependent on customer data, specifically the number of - projects and their sizes. We recommend two nodes as an absolute minimum for HA environments - and at least four nodes should be used when supporting 50,000 or more users. - We also recommend that each Gitaly node should store no more than 5TB of data - and have the number of [`gitaly-ruby` workers](../gitaly/index.md#gitaly-ruby) - set to 20% of available CPUs. Additional nodes should be considered in conjunction - with a review of expected data size and spread based on the recommendations above. - -1. Recommended Redis setup differs depending on the size of the architecture. - For smaller architectures (less than 3,000 users) a single instance should suffice. - For medium sized installs (3,000 - 5,000) we suggest one Redis cluster for all - classes and that Redis Sentinel is hosted alongside Consul. - For larger architectures (10,000 users or more) we suggest running a separate - [Redis Cluster](../redis/replication_and_failover.md#running-multiple-redis-clusters) for the Cache class - and another for the Queues and Shared State classes respectively. We also recommend - that you run the Redis Sentinel clusters separately for each Redis Cluster. - -1. For data objects such as LFS, Uploads, Artifacts, etc. We recommend an [Object Storage service](../object_storage.md) - over NFS where possible, due to better performance and availability. - -1. NFS can be used as an alternative for both repository data (replacing Gitaly) and - object storage but this isn't typically recommended for performance reasons. Note however it is required for - [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/196). - -1. Our architectures have been tested and validated with [HAProxy](https://www.haproxy.org/) - as the load balancer. Although other load balancers with similar feature sets - could also be used, those load balancers have not been validated. - -1. We strongly recommend that any Gitaly or NFS nodes be set up with SSD disks over - HDD with a throughput of at least 8,000 IOPS for read operations and 2,000 IOPS for write - as these components have heavy I/O. These IOPS values are recommended only as a starter - as with time they may be adjusted higher or lower depending on the scale of your - environment's workload. If you're running the environment on a Cloud provider - you may need to refer to their documentation on how configure IOPS correctly. - -1. The architectures were built and tested with the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) - CPU platform on GCP. On different hardware you may find that adjustments, either lower - or higher, are required for your CPU or Node counts accordingly. For more information, a - [Sysbench](https://github.com/akopytov/sysbench) benchmark of the CPU can be found - [here](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). +> - **High Availability:** Yes +> - **Test requests per second (RPS) rates:** API: 1000 RPS, Web: 100 RPS, Git: 100 RPS + +| Service | Nodes | Configuration | GCP | AWS | Azure | +|-----------------------------------------|-------------|-------------------------|-----------------|--------------|----------| +| External load balancing node | 1 | 8 vCPU, 7.2GB memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | +| Consul | 3 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| PostgreSQL | 3 | 16 vCPU, 60GB memory | n1-standard-16 | m5.4xlarge | D16s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Internal load balancing node | 1 | 8 vCPU, 7.2GB memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | +| Redis - Cache | 3 | 4 vCPU, 15GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| Redis - Queues / Shared State | 3 | 4 vCPU, 15GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7GB memory | g1-small | t2.small | B1MS | +| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7GB memory | g1-small | t2.small | B1MS | +| Gitaly | 2 (minimum) | 64 vCPU, 240GB memory | n1-standard-64 | m5.16xlarge | D64s v3 | +| Sidekiq | 4 | 4 vCPU, 15GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| GitLab Rails | 12 | 32 vCPU, 28.8GB memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | +| Monitoring node | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| Object Storage | n/a | n/a | n/a | n/a | n/a | +| NFS Server | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | + +The Google Cloud Platform (GCP) architectures were built and tested using the +[Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) +CPU platform. On different hardware you may find that adjustments, either lower +or higher, are required for your CPU or node counts. For more information, see +our [Sysbench](https://github.com/akopytov/sysbench)-based +[CPU benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). + +For data objects (such as LFS, Uploads, or Artifacts), an +[object storage service](#configure-the-object-storage) is recommended instead +of NFS where possible, due to better performance and availability. Since this +doesn't require a node to be set up, *Object Storage* is noted as not +applicable (n/a) in the previous table. + +## Setup components + +To set up GitLab and its components to accommodate up to 50,000 users: + +1. [Configure the external load balancing node](#configure-the-external-load-balancer) + that will handle the load balancing of the three GitLab application services nodes. +1. [Configure Consul](#configure-consul). +1. [Configure PostgreSQL](#configure-postgresql), the database for GitLab. +1. [Configure PgBouncer](#configure-pgbouncer). +1. [Configure the internal load balancing node](#configure-the-internal-load-balancer) +1. [Configure Redis](#configure-redis). +1. [Configure Gitaly](#configure-gitaly), + which provides access to the Git repositories. +1. [Configure Sidekiq](#configure-sidekiq). +1. [Configure the main GitLab Rails application](#configure-gitlab-rails) + to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend requests (UI, API, Git + over HTTP/SSH). +1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab environment. +1. [Configure the Object Storage](#configure-the-object-storage) + used for shared data objects. +1. [Configure NFS (Optional)](#configure-nfs-optional) + to have shared disk storage service as an alternative to Gitaly and/or Object Storage (although + not recommended). NFS is required for GitLab Pages, you can skip this step if you're not using + that feature. + +We start with all servers on the same 10.6.0.0/24 private network range, they +can connect to each other freely on those addresses. + +Here is a list and description of each machine and the assigned IP: + +- `10.6.0.10`: External Load Balancer +- `10.6.0.11`: Consul 1 +- `10.6.0.12`: Consul 2 +- `10.6.0.13`: Consul 3 +- `10.6.0.21`: PostgreSQL primary +- `10.6.0.22`: PostgreSQL secondary 1 +- `10.6.0.23`: PostgreSQL secondary 2 +- `10.6.0.31`: PgBouncer 1 +- `10.6.0.32`: PgBouncer 2 +- `10.6.0.33`: PgBouncer 3 +- `10.6.0.40`: Internal Load Balancer +- `10.6.0.51`: Redis - Cache Primary +- `10.6.0.52`: Redis - Cache Replica 1 +- `10.6.0.53`: Redis - Cache Replica 2 +- `10.6.0.71`: Sentinel - Cache 1 +- `10.6.0.72`: Sentinel - Cache 2 +- `10.6.0.73`: Sentinel - Cache 3 +- `10.6.0.61`: Redis - Queues Primary +- `10.6.0.62`: Redis - Queues Replica 1 +- `10.6.0.63`: Redis - Queues Replica 2 +- `10.6.0.81`: Sentinel - Queues 1 +- `10.6.0.82`: Sentinel - Queues 2 +- `10.6.0.83`: Sentinel - Queues 3 +- `10.6.0.91`: Gitaly 1 +- `10.6.0.92`: Gitaly 2 +- `10.6.0.101`: Sidekiq 1 +- `10.6.0.102`: Sidekiq 2 +- `10.6.0.103`: Sidekiq 3 +- `10.6.0.104`: Sidekiq 4 +- `10.6.0.111`: GitLab application 1 +- `10.6.0.112`: GitLab application 2 +- `10.6.0.113`: GitLab application 3 +- `10.6.0.121`: Prometheus + +## Configure the external load balancer + +NOTE: **Note:** +This architecture has been tested and validated with [HAProxy](https://www.haproxy.org/) +as the load balancer. Although other load balancers with similar feature sets +could also be used, those load balancers have not been validated. + +In an active/active GitLab configuration, you will need a load balancer to route +traffic to the application servers. The specifics on which load balancer to use +or the exact configuration is beyond the scope of GitLab documentation. We hope +that if you're managing multi-node systems like GitLab you have a load balancer of +choice already. Some examples including HAProxy (open-source), F5 Big-IP LTM, +and Citrix Net Scaler. This documentation will outline what ports and protocols +you need to use with GitLab. + +The next question is how you will handle SSL in your environment. +There are several different options: + +- [The application node terminates SSL](#application-node-terminates-ssl). +- [The load balancer terminates SSL without backend SSL](#load-balancer-terminates-ssl-without-backend-ssl) + and communication is not secure between the load balancer and the application node. +- [The load balancer terminates SSL with backend SSL](#load-balancer-terminates-ssl-with-backend-ssl) + and communication is *secure* between the load balancer and the application node. + +### Application node terminates SSL + +Configure your load balancer to pass connections on port 443 as `TCP` rather +than `HTTP(S)` protocol. This will pass the connection to the application node's +NGINX service untouched. NGINX will have the SSL certificate and listen on port 443. + +See the [NGINX HTTPS documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) +for details on managing SSL certificates and configuring NGINX. + +### Load balancer terminates SSL without backend SSL + +Configure your load balancer to use the `HTTP(S)` protocol rather than `TCP`. +The load balancer will then be responsible for managing SSL certificates and +terminating SSL. + +Since communication between the load balancer and GitLab will not be secure, +there is some additional configuration needed. See the +[NGINX proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#supporting-proxied-ssl) +for details. + +### Load balancer terminates SSL with backend SSL + +Configure your load balancer(s) to use the 'HTTP(S)' protocol rather than 'TCP'. +The load balancer(s) will be responsible for managing SSL certificates that +end users will see. + +Traffic will also be secure between the load balancer(s) and NGINX in this +scenario. There is no need to add configuration for proxied SSL since the +connection will be secure all the way. However, configuration will need to be +added to GitLab to configure SSL certificates. See +[NGINX HTTPS documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) +for details on managing SSL certificates and configuring NGINX. + +### Ports + +The basic ports to be used are shown in the table below. + +| LB Port | Backend Port | Protocol | +| ------- | ------------ | ------------------------ | +| 80 | 80 | HTTP (*1*) | +| 443 | 443 | TCP or HTTPS (*1*) (*2*) | +| 22 | 22 | TCP | + +- (*1*): [Web terminal](../../ci/environments/index.md#web-terminals) support requires + your load balancer to correctly handle WebSocket connections. When using + HTTP or HTTPS proxying, this means your load balancer must be configured + to pass through the `Connection` and `Upgrade` hop-by-hop headers. See the + [web terminal](../integration/terminal.md) integration guide for + more details. +- (*2*): When using HTTPS protocol for port 443, you will need to add an SSL + certificate to the load balancers. If you wish to terminate SSL at the + GitLab application server instead, use TCP protocol. + +If you're using GitLab Pages with custom domain support you will need some +additional port configurations. +GitLab Pages requires a separate virtual IP address. Configure DNS to point the +`pages_external_url` from `/etc/gitlab/gitlab.rb` at the new virtual IP address. See the +[GitLab Pages documentation](../pages/index.md) for more information. + +| LB Port | Backend Port | Protocol | +| ------- | ------------- | --------- | +| 80 | Varies (*1*) | HTTP | +| 443 | Varies (*1*) | TCP (*2*) | + +- (*1*): The backend port for GitLab Pages depends on the + `gitlab_pages['external_http']` and `gitlab_pages['external_https']` + setting. See [GitLab Pages documentation](../pages/index.md) for more details. +- (*2*): Port 443 for GitLab Pages should always use the TCP protocol. Users can + configure custom domains with custom SSL, which would not be possible + if SSL was terminated at the load balancer. + +#### Alternate SSH Port + +Some organizations have policies against opening SSH port 22. In this case, +it may be helpful to configure an alternate SSH hostname that allows users +to use SSH on port 443. An alternate SSH hostname will require a new virtual IP address +compared to the other GitLab HTTP configuration above. + +Configure DNS for an alternate SSH hostname such as `altssh.gitlab.example.com`. + +| LB Port | Backend Port | Protocol | +| ------- | ------------ | -------- | +| 443 | 22 | TCP | + + + +## Configure Consul + +The following IPs will be used as an example: + +- `10.6.0.11`: Consul 1 +- `10.6.0.12`: Consul 2 +- `10.6.0.13`: Consul 3 + +NOTE: **Note:** +The configuration processes for the other servers in your reference architecture will +use the `/etc/gitlab/gitlab-secrets.json` file from your Consul server to connect +with the other servers. + +To configure Consul: + +1. SSH into the server that will host Consul. +1. [Download/install](https://about.gitlab.com/install/) the + Omnibus GitLab Enterprise Edition package using **steps 1 and 2** from the + GitLab downloads page. + - Make sure you select the correct Omnibus package, with the same version + the GitLab application is running. + - Do not complete any other steps on the download page. + +1. Edit `/etc/gitlab/gitlab.rb` and add the contents: + + ```ruby + roles ['consul_role'] + + ## Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + server: true, + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + + # Disable auto migrations + gitlab_rails['auto_migrate'] = false + ``` + +1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Go through the steps again for all the other Consul nodes, and + make sure you set up the correct IPs. + +NOTE: **Note:** +A Consul leader will be elected when the provisioning of the third Consul server is completed. +Viewing the Consul logs `sudo gitlab-ctl tail consul` will display +`...[INFO] consul: New leader elected: ...` + +You can list the current Consul members (server, client): + +```shell +sudo /opt/gitlab/embedded/bin/consul members +``` + +You can verify the GitLab services are running: + +```shell +sudo gitlab-ctl status +``` + +The output should be similar to the following: + +```plaintext +run: consul: (pid 30074) 76834s; run: log: (pid 29740) 76844s +run: logrotate: (pid 30925) 3041s; run: log: (pid 29649) 76861s +run: node-exporter: (pid 30093) 76833s; run: log: (pid 29663) 76855s +``` + + + +## Configure PostgreSQL + +In this section, you'll be guided through configuring an external PostgreSQL database +to be used with GitLab. + +### Provide your own PostgreSQL instance + +If you're hosting GitLab on a cloud provider, you can optionally use a +managed service for PostgreSQL. For example, AWS offers a managed Relational +Database Service (RDS) that runs PostgreSQL. + +If you use a cloud-managed service, or provide your own PostgreSQL: + +1. Set up PostgreSQL according to the + [database requirements document](../../install/requirements.md#database). +1. Set up a `gitlab` username with a password of your choice. The `gitlab` user + needs privileges to create the `gitlabhq_production` database. +1. Configure the GitLab application servers with the appropriate details. + This step is covered in [Configuring the GitLab Rails application](#configure-gitlab-rails). + +### Standalone PostgreSQL using Omnibus GitLab + +The following IPs will be used as an example: + +- `10.6.0.21`: PostgreSQL primary +- `10.6.0.22`: PostgreSQL secondary 1 +- `10.6.0.23`: PostgreSQL secondary 2 + +First, make sure to [install](https://about.gitlab.com/install/) +the Linux GitLab package **on each node**. Following the steps, +install the necessary dependencies from step 1, and add the +GitLab package repository from step 2. When installing GitLab +in the second step, do not supply the `EXTERNAL_URL` value. + +#### PostgreSQL primary node + +1. SSH into the PostgreSQL primary node. +1. Generate a password hash for the PostgreSQL username/password pair. This assumes you will use the default + username of `gitlab` (recommended). The command will request a password + and confirmation. Use the value that is output by this command in the next + step as the value of ``: + + ```shell + sudo gitlab-ctl pg-password-md5 gitlab + ``` + +1. Generate a password hash for the PgBouncer username/password pair. This assumes you will use the default + username of `pgbouncer` (recommended). The command will request a password + and confirmation. Use the value that is output by this command in the next + step as the value of ``: + + ```shell + sudo gitlab-ctl pg-password-md5 pgbouncer + ``` + +1. Generate a password hash for the Consul database username/password pair. This assumes you will use the default + username of `gitlab-consul` (recommended). The command will request a password + and confirmation. Use the value that is output by this command in the next + step as the value of ``: + + ```shell + sudo gitlab-ctl pg-password-md5 gitlab-consul + ``` + +1. On the primary database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: + + ```ruby + # Disable all components except PostgreSQL and Repmgr and Consul + roles ['postgres_role'] + + # PostgreSQL configuration + postgresql['listen_address'] = '0.0.0.0' + postgresql['hot_standby'] = 'on' + postgresql['wal_level'] = 'replica' + postgresql['shared_preload_libraries'] = 'repmgr_funcs' + + # Disable automatic database migrations + gitlab_rails['auto_migrate'] = false + + # Configure the Consul agent + consul['services'] = %w(postgresql) + + # START user configuration + # Please set the real values as explained in Required Information section + # + # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value + postgresql['pgbouncer_user_password'] = '' + # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value + postgresql['sql_user_password'] = '' + # Set `max_wal_senders` to one more than the number of database nodes in the cluster. + # This is used to prevent replication from using up all of the + # available database connections. + postgresql['max_wal_senders'] = 4 + postgresql['max_replication_slots'] = 4 + + # Replace XXX.XXX.XXX.XXX/YY with Network Address + postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/24) + repmgr['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 10.6.0.0/24) + + ## Enable service discovery for Prometheus + consul['monitoring_service_discovery'] = true + + # Set the network addresses that the exporters will listen on for monitoring + node_exporter['listen_address'] = '0.0.0.0:9100' + postgres_exporter['listen_address'] = '0.0.0.0:9187' + + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + # + # END user configuration + ``` + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace + the file of the same name on this server. If that file is not on this server, + add the file from your Consul server to this server. + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + + + +#### PostgreSQL secondary nodes + +1. On both the secondary nodes, add the same configuration specified above for the primary node + with an additional setting (`repmgr['master_on_initialization'] = false`) that will inform `gitlab-ctl` that they are standby nodes initially + and there's no need to attempt to register them as a primary node: + + ```ruby + # Disable all components except PostgreSQL and Repmgr and Consul + roles ['postgres_role'] + + # PostgreSQL configuration + postgresql['listen_address'] = '0.0.0.0' + postgresql['hot_standby'] = 'on' + postgresql['wal_level'] = 'replica' + postgresql['shared_preload_libraries'] = 'repmgr_funcs' + + # Disable automatic database migrations + gitlab_rails['auto_migrate'] = false + + # Configure the Consul agent + consul['services'] = %w(postgresql) + + # Specify if a node should attempt to be primary on initialization. + repmgr['master_on_initialization'] = false + + # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value + postgresql['pgbouncer_user_password'] = '' + # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value + postgresql['sql_user_password'] = '' + # Set `max_wal_senders` to one more than the number of database nodes in the cluster. + # This is used to prevent replication from using up all of the + # available database connections. + postgresql['max_wal_senders'] = 4 + postgresql['max_replication_slots'] = 4 + + # Replace with your network addresses + postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/24) + repmgr['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 10.6.0.0/24) + + ## Enable service discovery for Prometheus + consul['monitoring_service_discovery'] = true + + # Set the network addresses that the exporters will listen on for monitoring + node_exporter['listen_address'] = '0.0.0.0:9100' + postgres_exporter['listen_address'] = '0.0.0.0:9187' + + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + ``` + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace + the file of the same name on this server. If that file is not on this server, + add the file from your Consul server to this server. + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/database.html) +are supported and can be added if needed. + + + +#### PostgreSQL post-configuration + +SSH into the **primary node**: + +1. Open a database prompt: + + ```shell + gitlab-psql -d gitlabhq_production + ``` + +1. Make sure the `pg_trgm` extension is enabled (it might already be): + + ```shell + CREATE EXTENSION pg_trgm; + ``` + +1. Exit the database prompt by typing `\q` and Enter. + +1. Verify the cluster is initialized with one node: + + ```shell + gitlab-ctl repmgr cluster show + ``` + + The output should be similar to the following: + + ```plaintext + Role | Name | Upstream | Connection String + ----------+----------|----------|---------------------------------------- + * master | HOSTNAME | | host=HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr + ``` + +1. Note down the hostname or IP address in the connection string: `host=HOSTNAME`. We will + refer to the hostname in the next section as ``. If the value + is not an IP address, it will need to be a resolvable name (via DNS or + `/etc/hosts`) + +SSH into the **secondary node**: + +1. Set up the repmgr standby: + + ```shell + gitlab-ctl repmgr standby setup + ``` + + Do note that this will remove the existing data on the node. The command + has a wait time. + + The output should be similar to the following: + + ```console + Doing this will delete the entire contents of /var/opt/gitlab/postgresql/data + If this is not what you want, hit Ctrl-C now to exit + To skip waiting, rerun with the -w option + Sleeping for 30 seconds + Stopping the database + Removing the data + Cloning the data + Starting the database + Registering the node with the cluster + ok: run: repmgrd: (pid 19068) 0s + ``` + +Before moving on, make sure the databases are configured correctly. Run the +following command on the **primary** node to verify that replication is working +properly and the secondary nodes appear in the cluster: + +```shell +gitlab-ctl repmgr cluster show +``` + +The output should be similar to the following: + +```plaintext +Role | Name | Upstream | Connection String +----------+---------|-----------|------------------------------------------------ +* master | MASTER | | host= user=gitlab_repmgr dbname=gitlab_repmgr + standby | STANDBY | MASTER | host= user=gitlab_repmgr dbname=gitlab_repmgr + standby | STANDBY | MASTER | host= user=gitlab_repmgr dbname=gitlab_repmgr +``` + +If the 'Role' column for any node says "FAILED", check the +[Troubleshooting section](troubleshooting.md) before proceeding. + +Also, check that the `repmgr-check-master` command works successfully on each node: + +```shell +su - gitlab-consul +gitlab-ctl repmgr-check-master || echo 'This node is a standby repmgr node' +``` + +This command relies on exit codes to tell Consul whether a particular node is a master +or secondary. The most important thing here is that this command does not produce errors. +If there are errors it's most likely due to incorrect `gitlab-consul` database user permissions. +Check the [Troubleshooting section](troubleshooting.md) before proceeding. + + + +## Configure PgBouncer + +Now that the PostgreSQL servers are all set up, let's configure PgBouncer. +The following IPs will be used as an example: + +- `10.6.0.31`: PgBouncer 1 +- `10.6.0.32`: PgBouncer 2 +- `10.6.0.33`: PgBouncer 3 + +1. On each PgBouncer node, edit `/etc/gitlab/gitlab.rb`, and replace + `` and `` with the + password hashes you [set up previously](#postgresql-primary-node): + + ```ruby + # Disable all components except Pgbouncer and Consul agent + roles ['pgbouncer_role'] + + # Configure PgBouncer + pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) + + pgbouncer['users'] = { + 'gitlab-consul': { + password: '' + }, + 'pgbouncer': { + password: '' + } + } + + # Configure Consul agent + consul['watchers'] = %w(postgresql) + consul['enable'] = true + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) + } + + # Enable service discovery for Prometheus + consul['monitoring_service_discovery'] = true + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + ``` + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace + the file of the same name on this server. If that file is not on this server, + add the file from your Consul server to this server. + +1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + + NOTE: **Note:** + If an error `execute[generate databases.ini]` occurs, this is due to an existing + [known issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4713). + It will be resolved when you run a second `reconfigure` after the next step. + +1. Create a `.pgpass` file so Consul is able to + reload PgBouncer. Enter the PgBouncer password twice when asked: + + ```shell + gitlab-ctl write-pgpass --host 127.0.0.1 --database pgbouncer --user pgbouncer --hostuser gitlab-consul + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) once again + to resolve any potential errors from the previous steps. +1. Ensure each node is talking to the current primary: + + ```shell + gitlab-ctl pgb-console # You will be prompted for PGBOUNCER_PASSWORD + ``` + +1. Once the console prompt is available, run the following queries: + + ```shell + show databases ; show clients ; + ``` + + The output should be similar to the following: + + ```plaintext + name | host | port | database | force_user | pool_size | reserve_pool | pool_mode | max_connections | current_connections + ---------------------+-------------+------+---------------------+------------+-----------+--------------+-----------+-----------------+--------------------- + gitlabhq_production | MASTER_HOST | 5432 | gitlabhq_production | | 20 | 0 | | 0 | 0 + pgbouncer | | 6432 | pgbouncer | pgbouncer | 2 | 0 | statement | 0 | 0 + (2 rows) + + type | user | database | state | addr | port | local_addr | local_port | connect_time | request_time | ptr | link | remote_pid | tls + ------+-----------+---------------------+---------+----------------+-------+------------+------------+---------------------+---------------------+-----------+------+------------+----- + C | pgbouncer | pgbouncer | active | 127.0.0.1 | 56846 | 127.0.0.1 | 6432 | 2017-08-21 18:09:59 | 2017-08-21 18:10:48 | 0x22b3880 | | 0 | + (2 rows) + ``` + + + +### Configure the internal load balancer + +If you're running more than one PgBouncer node as recommended, then at this time you'll need to set +up a TCP internal load balancer to serve each correctly. + +The following IP will be used as an example: + +- `10.6.0.40`: Internal Load Balancer + +Here's how you could do it with [HAProxy](https://www.haproxy.org/): + +```plaintext +global + log /dev/log local0 + log localhost local1 notice + log stdout format raw local0 + +defaults + log global + default-server inter 10s fall 3 rise 2 + balance leastconn + +frontend internal-pgbouncer-tcp-in + bind *:6432 + mode tcp + option tcplog + + default_backend pgbouncer + +backend pgbouncer + mode tcp + option tcp-check + + server pgbouncer1 10.6.0.21:6432 check + server pgbouncer2 10.6.0.22:6432 check + server pgbouncer3 10.6.0.23:6432 check +``` + +Refer to your preferred Load Balancer's documentation for further guidance. + + + +## Configure Redis + +Using [Redis](https://redis.io/) in scalable environment is possible using a **Primary** x **Replica** +topology with a [Redis Sentinel](https://redis.io/topics/sentinel) service to watch and automatically +start the failover procedure. + +Redis requires authentication if used with Sentinel. See +[Redis Security](https://redis.io/topics/security) documentation for more +information. We recommend using a combination of a Redis password and tight +firewall rules to secure your Redis service. +You are highly encouraged to read the [Redis Sentinel](https://redis.io/topics/sentinel) documentation +before configuring Redis with GitLab to fully understand the topology and +architecture. + +The requirements for a Redis setup are the following: + +1. All Redis nodes must be able to talk to each other and accept incoming + connections over Redis (`6379`) and Sentinel (`26379`) ports (unless you + change the default ones). +1. The server that hosts the GitLab application must be able to access the + Redis nodes. +1. Protect the nodes from access from external networks + ([Internet](https://gitlab.com/gitlab-org/gitlab-foss/uploads/c4cc8cd353604bd80315f9384035ff9e/The_Internet_IT_Crowd.png)), + using a firewall. + +In this section, you'll be guided through configuring two external Redis clusters +to be used with GitLab. The following IPs will be used as an example: + +- `10.6.0.51`: Redis - Cache Primary +- `10.6.0.52`: Redis - Cache Replica 1 +- `10.6.0.53`: Redis - Cache Replica 2 +- `10.6.0.71`: Sentinel - Cache 1 +- `10.6.0.72`: Sentinel - Cache 2 +- `10.6.0.73`: Sentinel - Cache 3 +- `10.6.0.61`: Redis - Queues Primary +- `10.6.0.62`: Redis - Queues Replica 1 +- `10.6.0.63`: Redis - Queues Replica 2 +- `10.6.0.81`: Sentinel - Queues 1 +- `10.6.0.82`: Sentinel - Queues 2 +- `10.6.0.83`: Sentinel - Queues 3 + +NOTE: **Providing your own Redis instance:** +Managed Redis from cloud providers such as AWS ElastiCache will work. If these +services support high availability, be sure it is **not** the Redis Cluster type. +Redis version 5.0 or higher is required, as this is what ships with +Omnibus GitLab packages starting with GitLab 13.0. Older Redis versions +do not support an optional count argument to SPOP which is now required for +[Merge Trains](../../ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md). +Note the Redis node's IP address or hostname, port, and password (if required). +These will be necessary when configuring the +[GitLab application servers](#configure-gitlab-rails) later. + +### Configure the Redis and Sentinel Cache cluster + +This is the section where we install and set up the new Redis Cache instances. + +NOTE: **Note:** +Redis nodes (both primary and replica) will need the same password defined in +`redis['password']`. At any time during a failover the Sentinels can +reconfigure a node and change its status from primary to replica and vice versa. + +#### Configure the primary Redis Cache node + +1. SSH into the **Primary** Redis server. +1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page. + - Make sure you select the correct Omnibus package, with the same version + and type (Community, Enterprise editions) of your current install. + - Do not complete any other steps on the download page. + +1. Edit `/etc/gitlab/gitlab.rb` and add the contents: + + ```ruby + # Specify server role as 'redis_master_role' + roles ['redis_master_role'] + + # IP address pointing to a local IP that the other machines can reach to. + # You can also set bind to '0.0.0.0' which listen in all interfaces. + # If you really need to bind to an external accessible IP, make + # sure you add extra firewall rules to prevent unauthorized access. + redis['bind'] = '10.6.0.51' + + # Define a port so Redis can listen for TCP requests which will allow other + # machines to connect to it. + redis['port'] = 6379 + + # Set up password authentication for Redis (use the same password in all nodes). + redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + + ## Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + redis_exporter['listen_address'] = '0.0.0.0:9121' + + # Prevent database migrations from running on upgrade + gitlab_rails['auto_migrate'] = false + ``` + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace + the file of the same name on this server. If that file is not on this server, + add the file from your Consul server to this server. + +1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +NOTE: **Note:** +You can specify multiple roles like sentinel and Redis as: +`roles ['redis_sentinel_role', 'redis_master_role']`. +Read more about [roles](https://docs.gitlab.com/omnibus/roles/). + +#### Configure the replica Redis Cache nodes + +1. SSH into the **replica** Redis server. +1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page. + - Make sure you select the correct Omnibus package, with the same version + and type (Community, Enterprise editions) of your current install. + - Do not complete any other steps on the download page. + +1. Edit `/etc/gitlab/gitlab.rb` and add the contents: + + ```ruby + # Specify server role as 'redis_replica_role' + roles ['redis_replica_role'] + + # IP address pointing to a local IP that the other machines can reach to. + # You can also set bind to '0.0.0.0' which listen in all interfaces. + # If you really need to bind to an external accessible IP, make + # sure you add extra firewall rules to prevent unauthorized access. + redis['bind'] = '10.6.0.52' + + # Define a port so Redis can listen for TCP requests which will allow other + # machines to connect to it. + redis['port'] = 6379 + + # The same password for Redis authentication you set up for the primary node. + redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + + # The IP of the primary Redis node. + redis['master_ip'] = '10.6.0.51' + + # Port of primary Redis server, uncomment to change to non default. Defaults + # to `6379`. + #redis['master_port'] = 6379 + + ## Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + redis_exporter['listen_address'] = '0.0.0.0:9121' + + # Prevent database migrations from running on upgrade + gitlab_rails['auto_migrate'] = false + ``` + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace + the file of the same name on this server. If that file is not on this server, + add the file from your Consul server to this server. + +1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Go through the steps again for all the other replica nodes, and + make sure to set up the IPs correctly. + +NOTE: **Note:** +You can specify multiple roles like sentinel and Redis as: +`roles ['redis_sentinel_role', 'redis_master_role']`. +Read more about [roles](https://docs.gitlab.com/omnibus/roles/). + +These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after +a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-cache-nodes), and even after a +`gitlab-ctl reconfigure`, they will get their configuration restored by +the same Sentinels. + +Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) +are supported and can be added if needed. + + + +#### Configure the Sentinel Cache nodes + +NOTE: **Note:** +If you are using an external Redis Sentinel instance, be sure +to exclude the `requirepass` parameter from the Sentinel +configuration. This parameter will cause clients to report `NOAUTH +Authentication required.`. [Redis Sentinel 3.2.x does not support +password authentication](https://github.com/antirez/redis/issues/3279). + +Now that the Redis servers are all set up, let's configure the Sentinel +servers. The following IPs will be used as an example: + +- `10.6.0.71`: Sentinel - Cache 1 +- `10.6.0.72`: Sentinel - Cache 2 +- `10.6.0.73`: Sentinel - Cache 3 + +To configure the Sentinel Cache server: + +1. SSH into the server that will host Consul/Sentinel. +1. [Download/install](https://about.gitlab.com/install/) the + Omnibus GitLab Enterprise Edition package using **steps 1 and 2** from the + GitLab downloads page. + - Make sure you select the correct Omnibus package, with the same version + the GitLab application is running. + - Do not complete any other steps on the download page. + +1. Edit `/etc/gitlab/gitlab.rb` and add the contents: + + ```ruby + roles ['redis_sentinel_role'] + + ## Must be the same in every sentinel node + redis['master_name'] = 'gitlab-redis-cache' + + ## The same password for Redis authentication you set up for the primary node. + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + + ## The IP of the primary Redis node. + redis['master_ip'] = '10.6.0.51' + + ## Define a port so Redis can listen for TCP requests which will allow other + ## machines to connect to it. + redis['port'] = 6379 + + ## Port of primary Redis server, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + + ## Configure Sentinel's IP + sentinel['bind'] = '10.6.0.71' + + ## Port that Sentinel listens on, uncomment to change to non default. Defaults + ## to `26379`. + #sentinel['port'] = 26379 + + ## Quorum must reflect the amount of voting sentinels it take to start a failover. + ## Value must NOT be greater then the amount of sentinels. + ## + ## The quorum can be used to tune Sentinel in two ways: + ## 1. If a the quorum is set to a value smaller than the majority of Sentinels + ## we deploy, we are basically making Sentinel more sensible to primary failures, + ## triggering a failover as soon as even just a minority of Sentinels is no longer + ## able to talk with the primary. + ## 1. If a quorum is set to a value greater than the majority of Sentinels, we are + ## making Sentinel able to failover only when there are a very large number (larger + ## than majority) of well connected Sentinels which agree about the primary being down.s + sentinel['quorum'] = 2 + + ## Consider unresponsive server down after x amount of ms. + #sentinel['down_after_milliseconds'] = 10000 + + ## Specifies the failover timeout in milliseconds. It is used in many ways: + ## + ## - The time needed to re-start a failover after a previous failover was + ## already tried against the same primary by a given Sentinel, is two + ## times the failover timeout. + ## + ## - The time needed for a replica replicating to a wrong primary according + ## to a Sentinel current configuration, to be forced to replicate + ## with the right primary, is exactly the failover timeout (counting since + ## the moment a Sentinel detected the misconfiguration). + ## + ## - The time needed to cancel a failover that is already in progress but + ## did not produced any configuration change (REPLICAOF NO ONE yet not + ## acknowledged by the promoted replica). + ## + ## - The maximum time a failover in progress waits for all the replica to be + ## reconfigured as replicas of the new primary. However even after this time + ## the replicas will be reconfigured by the Sentinels anyway, but not with + ## the exact parallel-syncs progression as specified. + #sentinel['failover_timeout'] = 60000 + + ## Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + redis_exporter['listen_address'] = '0.0.0.0:9121' + + # Disable auto migrations + gitlab_rails['auto_migrate'] = false + ``` + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace + the file of the same name on this server. If that file is not on this server, + add the file from your Consul server to this server. + +1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Go through the steps again for all the other Consul/Sentinel nodes, and + make sure you set up the correct IPs. + + + +### Configure the Redis and Sentinel Queues cluster + +This is the section where we install and set up the new Redis Queues instances. + +NOTE: **Note:** +Redis nodes (both primary and replica) will need the same password defined in +`redis['password']`. At any time during a failover the Sentinels can +reconfigure a node and change its status from primary to replica and vice versa. + +#### Configure the primary Redis Queues node + +1. SSH into the **Primary** Redis server. +1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page. + - Make sure you select the correct Omnibus package, with the same version + and type (Community, Enterprise editions) of your current install. + - Do not complete any other steps on the download page. + +1. Edit `/etc/gitlab/gitlab.rb` and add the contents: + + ```ruby + # Specify server role as 'redis_master_role' + roles ['redis_master_role'] + + # IP address pointing to a local IP that the other machines can reach to. + # You can also set bind to '0.0.0.0' which listen in all interfaces. + # If you really need to bind to an external accessible IP, make + # sure you add extra firewall rules to prevent unauthorized access. + redis['bind'] = '10.6.0.61' + + # Define a port so Redis can listen for TCP requests which will allow other + # machines to connect to it. + redis['port'] = 6379 + + # Set up password authentication for Redis (use the same password in all nodes). + redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + + ## Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + redis_exporter['listen_address'] = '0.0.0.0:9121' + ``` + +1. Only the primary GitLab application server should handle migrations. To + prevent database migrations from running on upgrade, add the following + configuration to your `/etc/gitlab/gitlab.rb` file: + + ```ruby + gitlab_rails['auto_migrate'] = false + ``` + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace + the file of the same name on this server. If that file is not on this server, + add the file from your Consul server to this server. + +1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +NOTE: **Note:** +You can specify multiple roles like sentinel and Redis as: +`roles ['redis_sentinel_role', 'redis_master_role']`. +Read more about [roles](https://docs.gitlab.com/omnibus/roles/). + +#### Configure the replica Redis Queues nodes + +1. SSH into the **replica** Redis Queue server. +1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page. + - Make sure you select the correct Omnibus package, with the same version + and type (Community, Enterprise editions) of your current install. + - Do not complete any other steps on the download page. + +1. Edit `/etc/gitlab/gitlab.rb` and add the contents: + + ```ruby + # Specify server role as 'redis_replica_role' + roles ['redis_replica_role'] + + # IP address pointing to a local IP that the other machines can reach to. + # You can also set bind to '0.0.0.0' which listen in all interfaces. + # If you really need to bind to an external accessible IP, make + # sure you add extra firewall rules to prevent unauthorized access. + redis['bind'] = '10.6.0.62' + + # Define a port so Redis can listen for TCP requests which will allow other + # machines to connect to it. + redis['port'] = 6379 + + # The same password for Redis authentication you set up for the primary node. + redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + + # The IP of the primary Redis node. + redis['master_ip'] = '10.6.0.61' + + # Port of primary Redis server, uncomment to change to non default. Defaults + # to `6379`. + #redis['master_port'] = 6379 + + ## Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + redis_exporter['listen_address'] = '0.0.0.0:9121' + + # Disable auto migrations + gitlab_rails['auto_migrate'] = false + ``` + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace + the file of the same name on this server. If that file is not on this server, + add the file from your Consul server to this server. + +1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Go through the steps again for all the other replica nodes, and + make sure to set up the IPs correctly. + +NOTE: **Note:** +You can specify multiple roles like sentinel and Redis as: +`roles ['redis_sentinel_role', 'redis_master_role']`. +Read more about [roles](https://docs.gitlab.com/omnibus/roles/). + +These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after +a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-queues-nodes), and even after a +`gitlab-ctl reconfigure`, they will get their configuration restored by +the same Sentinels. + +Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) +are supported and can be added if needed. + + + +#### Configure the Sentinel Queues nodes + +NOTE: **Note:** +If you are using an external Redis Sentinel instance, be sure +to exclude the `requirepass` parameter from the Sentinel +configuration. This parameter will cause clients to report `NOAUTH +Authentication required.`. [Redis Sentinel 3.2.x does not support +password authentication](https://github.com/antirez/redis/issues/3279). + +Now that the Redis servers are all set up, let's configure the Sentinel +servers. The following IPs will be used as an example: + +- `10.6.0.81`: Sentinel - Queues 1 +- `10.6.0.82`: Sentinel - Queues 2 +- `10.6.0.83`: Sentinel - Queues 3 + +To configure the Sentinel Queues server: + +1. SSH into the server that will host Sentinel. +1. [Download/install](https://about.gitlab.com/install/) the + Omnibus GitLab Enterprise Edition package using **steps 1 and 2** from the + GitLab downloads page. + - Make sure you select the correct Omnibus package, with the same version + the GitLab application is running. + - Do not complete any other steps on the download page. + +1. Edit `/etc/gitlab/gitlab.rb` and add the contents: + + ```ruby + roles ['redis_sentinel_role'] + + ## Must be the same in every sentinel node + redis['master_name'] = 'gitlab-redis-persistent' + + ## The same password for Redis authentication you set up for the primary node. + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + + ## The IP of the primary Redis node. + redis['master_ip'] = '10.6.0.61' + + ## Define a port so Redis can listen for TCP requests which will allow other + ## machines to connect to it. + redis['port'] = 6379 + + ## Port of primary Redis server, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + + ## Configure Sentinel's IP + sentinel['bind'] = '10.6.0.81' + + ## Port that Sentinel listens on, uncomment to change to non default. Defaults + ## to `26379`. + #sentinel['port'] = 26379 + + ## Quorum must reflect the amount of voting sentinels it take to start a failover. + ## Value must NOT be greater then the amount of sentinels. + ## + ## The quorum can be used to tune Sentinel in two ways: + ## 1. If a the quorum is set to a value smaller than the majority of Sentinels + ## we deploy, we are basically making Sentinel more sensible to primary failures, + ## triggering a failover as soon as even just a minority of Sentinels is no longer + ## able to talk with the primary. + ## 1. If a quorum is set to a value greater than the majority of Sentinels, we are + ## making Sentinel able to failover only when there are a very large number (larger + ## than majority) of well connected Sentinels which agree about the primary being down.s + sentinel['quorum'] = 2 + + ## Consider unresponsive server down after x amount of ms. + #sentinel['down_after_milliseconds'] = 10000 + + ## Specifies the failover timeout in milliseconds. It is used in many ways: + ## + ## - The time needed to re-start a failover after a previous failover was + ## already tried against the same primary by a given Sentinel, is two + ## times the failover timeout. + ## + ## - The time needed for a replica replicating to a wrong primary according + ## to a Sentinel current configuration, to be forced to replicate + ## with the right primary, is exactly the failover timeout (counting since + ## the moment a Sentinel detected the misconfiguration). + ## + ## - The time needed to cancel a failover that is already in progress but + ## did not produced any configuration change (REPLICAOF NO ONE yet not + ## acknowledged by the promoted replica). + ## + ## - The maximum time a failover in progress waits for all the replica to be + ## reconfigured as replicas of the new primary. However even after this time + ## the replicas will be reconfigured by the Sentinels anyway, but not with + ## the exact parallel-syncs progression as specified. + #sentinel['failover_timeout'] = 60000 + + ## Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + redis_exporter['listen_address'] = '0.0.0.0:9121' + + # Disable auto migrations + gitlab_rails['auto_migrate'] = false + ``` + +1. To prevent database migrations from running on upgrade, run: + + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure + ``` + + Only the primary GitLab application server should handle migrations. + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace + the file of the same name on this server. If that file is not on this server, + add the file from your Consul server to this server. + +1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Go through the steps again for all the other Sentinel nodes, and + make sure you set up the correct IPs. + + + +## Configure Gitaly + +Deploying Gitaly in its own server can benefit GitLab installations that are +larger than a single machine. + +The Gitaly node requirements are dependent on customer data, specifically the number of +projects and their repository sizes. Two nodes are recommended as an absolute minimum. +Each Gitaly node should store no more than 5TB of data and have the number of +[`gitaly-ruby` workers](../gitaly/index.md#gitaly-ruby) set to 20% of available CPUs. +Additional nodes should be considered in conjunction with a review of expected +data size and spread based on the recommendations above. + +It is also strongly recommended that all Gitaly nodes be set up with SSD disks with +a throughput of at least 8,000 IOPS for read operations and 2,000 IOPS for write, +as Gitaly has heavy I/O. These IOPS values are recommended only as a starter as with +time they may be adjusted higher or lower depending on the scale of your environment's workload. +If you're running the environment on a Cloud provider, you may need to refer to +their documentation on how to configure IOPS correctly. + +Some things to note: + +- The GitLab Rails application shards repositories into [repository storages](../repository_storage_paths.md). +- A Gitaly server can host one or more storages. +- A GitLab server can use one or more Gitaly servers. +- Gitaly addresses must be specified in such a way that they resolve + correctly for ALL Gitaly clients. +- Gitaly servers must not be exposed to the public internet, as Gitaly's network + traffic is unencrypted by default. The use of a firewall is highly recommended + to restrict access to the Gitaly server. Another option is to + [use TLS](#gitaly-tls-support). + +TIP: **Tip:** +For more information about Gitaly's history and network architecture see the +[standalone Gitaly documentation](../gitaly/index.md). + +Note: **Note:** +The token referred to throughout the Gitaly documentation is +just an arbitrary password selected by the administrator. It is unrelated to +tokens created for the GitLab API or other similar web API tokens. + +Below we describe how to configure two Gitaly servers, with IPs and +domain names: + +- `10.6.0.91`: Gitaly 1 (`gitaly1.internal`) +- `10.6.0.92`: Gitaly 2 (`gitaly2.internal`) + +The secret token is assumed to be `gitalysecret` and that +your GitLab installation has three repository storages: + +- `default` on Gitaly 1 +- `storage1` on Gitaly 1 +- `storage2` on Gitaly 2 + +On each node: + +1. [Download/Install](https://about.gitlab.com/install/) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page but + **without** providing the `EXTERNAL_URL` value. +1. Edit `/etc/gitlab/gitlab.rb` to configure storage paths, enable + the network listener and configure the token: + + + + ```ruby + # /etc/gitlab/gitlab.rb + + # Gitaly and GitLab use two shared secrets for authentication, one to authenticate gRPC requests + # to Gitaly, and a second for authentication callbacks from GitLab-Shell to the GitLab internal API. + # The following two values must be the same as their respective values + # of the GitLab Rails application setup + gitaly['auth_token'] = 'gitalysecret' + gitlab_shell['secret_token'] = 'shellsecret' + + # Avoid running unnecessary services on the Gitaly server + postgresql['enable'] = false + redis['enable'] = false + nginx['enable'] = false + puma['enable'] = false + unicorn['enable'] = false + sidekiq['enable'] = false + gitlab_workhorse['enable'] = false + grafana['enable'] = false + + # If you run a seperate monitoring node you can disable these services + alertmanager['enable'] = false + prometheus['enable'] = false + + # Prevent database connections during 'gitlab-ctl reconfigure' + gitlab_rails['rake_cache_clear'] = false + gitlab_rails['auto_migrate'] = false + + # Configure the gitlab-shell API callback URL. Without this, `git push` will + # fail. This can be your 'front door' GitLab URL or an internal load + # balancer. + # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server. + gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' + + # Make Gitaly accept connections on all network interfaces. You must use + # firewalls to restrict access to this address/port. + # Comment out following line if you only want to support TLS connections + gitaly['listen_addr'] = "0.0.0.0:8075" + ``` + +1. Append the following to `/etc/gitlab/gitlab.rb` for each respective server: + 1. On `gitaly1.internal`: + + ```ruby + git_data_dirs({ + 'default' => { + 'path' => '/var/opt/gitlab/git-data' + }, + 'storage1' => { + 'path' => '/mnt/gitlab/git-data' + }, + }) + ``` + + 1. On `gitaly2.internal`: + + ```ruby + git_data_dirs({ + 'storage2' => { + 'path' => '/mnt/gitlab/git-data' + }, + }) + ``` + + + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace + the file of the same name on this server. If that file is not on this server, + add the file from your Consul server to this server. + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +### Gitaly TLS support + +Gitaly supports TLS encryption. To be able to communicate +with a Gitaly instance that listens for secure connections you will need to use `tls://` URL +scheme in the `gitaly_address` of the corresponding storage entry in the GitLab configuration. + +You will need to bring your own certificates as this isn't provided automatically. +The certificate, or its certificate authority, must be installed on all Gitaly +nodes (including the Gitaly node using the certificate) and on all client nodes +that communicate with it following the procedure described in +[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). + +NOTE: **Note:** +The self-signed certificate must specify the address you use to access the +Gitaly server. If you are addressing the Gitaly server by a hostname, you can +either use the Common Name field for this, or add it as a Subject Alternative +Name. If you are addressing the Gitaly server by its IP address, you must add it +as a Subject Alternative Name to the certificate. +[gRPC does not support using an IP address as Common Name in a certificate](https://github.com/grpc/grpc/issues/2691). + +NOTE: **Note:** +It is possible to configure Gitaly servers with both an +unencrypted listening address `listen_addr` and an encrypted listening +address `tls_listen_addr` at the same time. This allows you to do a +gradual transition from unencrypted to encrypted traffic, if necessary. + +To configure Gitaly with TLS: + +1. Create the `/etc/gitlab/ssl` directory and copy your key and certificate there: + + ```shell + sudo mkdir -p /etc/gitlab/ssl + sudo chmod 755 /etc/gitlab/ssl + sudo cp key.pem cert.pem /etc/gitlab/ssl/ + sudo chmod 644 key.pem cert.pem + ``` + +1. Copy the cert to `/etc/gitlab/trusted-certs` so Gitaly will trust the cert when + calling into itself: + + ```shell + sudo cp /etc/gitlab/ssl/cert.pem /etc/gitlab/trusted-certs/ + ``` + +1. Edit `/etc/gitlab/gitlab.rb` and add: + + + + ```ruby + gitaly['tls_listen_addr'] = "0.0.0.0:9999" + gitaly['certificate_path'] = "/etc/gitlab/ssl/cert.pem" + gitaly['key_path'] = "/etc/gitlab/ssl/key.pem" + ``` + +1. Delete `gitaly['listen_addr']` to allow only encrypted connections. + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + + + +## Configure Sidekiq + +Sidekiq requires connections to the Redis, PostgreSQL and Gitaly instances. +The following IPs will be used as an example: + +- `10.6.0.101`: Sidekiq 1 +- `10.6.0.102`: Sidekiq 2 +- `10.6.0.103`: Sidekiq 3 +- `10.6.0.104`: Sidekiq 4 + +To configure the Sidekiq nodes, on each one: + +1. SSH into the Sidekiq server. +1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab package +you want using steps 1 and 2 from the GitLab downloads page. +**Do not complete any other steps on the download page.** +1. Open `/etc/gitlab/gitlab.rb` with your editor: + + ```ruby + ######################################## + ##### Services Disabled ### + ######################################## + + nginx['enable'] = false + grafana['enable'] = false + prometheus['enable'] = false + alertmanager['enable'] = false + gitaly['enable'] = false + gitlab_workhorse['enable'] = false + nginx['enable'] = false + puma['enable'] = false + postgres_exporter['enable'] = false + postgresql['enable'] = false + redis['enable'] = false + redis_exporter['enable'] = false + gitlab_exporter['enable'] = false + + ######################################## + #### Redis ### + ######################################## + + ## Redis connection details + ## First cluster that will host the cache + gitlab_rails['redis_cache_instance'] = 'redis://:@gitlab-redis-cache' + + gitlab_rails['redis_cache_sentinels'] = [ + {host: '10.6.0.71', port: 26379}, + {host: '10.6.0.72', port: 26379}, + {host: '10.6.0.73', port: 26379}, + ] + + ## Second cluster that will host the queues, shared state, and actioncable + gitlab_rails['redis_queues_instance'] = 'redis://:@gitlab-redis-persistent' + gitlab_rails['redis_shared_state_instance'] = 'redis://:@gitlab-redis-persistent' + gitlab_rails['redis_actioncable_instance'] = 'redis://:@gitlab-redis-persistent' + + gitlab_rails['redis_queues_sentinels'] = [ + {host: '10.6.0.81', port: 26379}, + {host: '10.6.0.82', port: 26379}, + {host: '10.6.0.83', port: 26379}, + ] + gitlab_rails['redis_shared_state_sentinels'] = [ + {host: '10.6.0.81', port: 26379}, + {host: '10.6.0.82', port: 26379}, + {host: '10.6.0.83', port: 26379}, + ] + gitlab_rails['redis_actioncable_sentinels'] = [ + {host: '10.6.0.81', port: 26379}, + {host: '10.6.0.82', port: 26379}, + {host: '10.6.0.83', port: 26379}, + ] + + ####################################### + ### Gitaly ### + ####################################### + + git_data_dirs({ + 'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, + 'storage1' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, + 'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' }, + }) + gitlab_rails['gitaly_token'] = 'YOUR_TOKEN' + + ####################################### + ### Postgres ### + ####################################### + gitlab_rails['db_host'] = '10.6.0.20' # internal load balancer IP + gitlab_rails['db_port'] = 6432 + gitlab_rails['db_password'] = '' + gitlab_rails['db_adapter'] = 'postgresql' + gitlab_rails['db_encoding'] = 'unicode' + gitlab_rails['auto_migrate'] = false + + ####################################### + ### Sidekiq configuration ### + ####################################### + sidekiq['listen_address'] = "0.0.0.0" + sidekiq['cluster'] = true # no need to set this after GitLab 13.0 + + ####################################### + ### Monitoring configuration ### + ####################################### + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) + } + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + + # Rails Status for prometheus + gitlab_rails['monitoring_whitelist'] = ['10.6.0.121/32', '127.0.0.0/8'] + ``` + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace + the file of the same name on this server. If that file is not on this server, + add the file from your Consul server to this server. + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +TIP: **Tip:** +You can also run [multiple Sidekiq processes](../operations/extra_sidekiq_processes.md). + + + +## Configure GitLab Rails + +NOTE: **Note:** +In our architectures we run each GitLab Rails node using the Puma webserver +and have its number of workers set to 90% of available CPUs along with four threads. For +nodes that are running Rails with other components the worker value should be reduced +accordingly where we've found 50% achieves a good balance but this is dependent +on workload. + +This section describes how to configure the GitLab application (Rails) component. + +The following IPs will be used as an example: + +- `10.6.0.111`: GitLab application 1 +- `10.6.0.112`: GitLab application 2 +- `10.6.0.113`: GitLab application 3 + +On each node perform the following: + +1. Download/install Omnibus GitLab using **steps 1 and 2** from + [GitLab downloads](https://about.gitlab.com/install/). Do not complete other + steps on the download page. + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace + the file of the same name on this server. If that file is not on this server, + add the file from your Consul server to this server. + +1. Edit `/etc/gitlab/gitlab.rb` and use the following configuration. + To maintain uniformity of links across nodes, the `external_url` + on the application server should point to the external URL that users will use + to access GitLab. This would be the URL of the [external load balancer](#configure-the-external-load-balancer) + which will route traffic to the GitLab application server: + + ```ruby + external_url 'https://gitlab.example.com' + + # Gitaly and GitLab use two shared secrets for authentication, one to authenticate gRPC requests + # to Gitaly, and a second for authentication callbacks from GitLab-Shell to the GitLab internal API. + # The following two values must be the same as their respective values + # of the Gitaly setup + gitlab_rails['gitaly_token'] = 'gitalysecret' + gitlab_shell['secret_token'] = 'shellsecret' + + git_data_dirs({ + 'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, + 'storage1' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, + 'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' }, + }) + + ## Disable components that will not be on the GitLab application server + roles ['application_role'] + gitaly['enable'] = false + nginx['enable'] = true + + ## PostgreSQL connection details + # Disable PostgreSQL on the application node + postgresql['enable'] = false + gitlab_rails['db_host'] = '10.6.0.20' # internal load balancer IP + gitlab_rails['db_port'] = 6432 + gitlab_rails['db_password'] = '' + gitlab_rails['auto_migrate'] = false + + ## Redis connection details + ## First cluster that will host the cache + gitlab_rails['redis_cache_instance'] = 'redis://:@gitlab-redis-cache' + + gitlab_rails['redis_cache_sentinels'] = [ + {host: '10.6.0.71', port: 26379}, + {host: '10.6.0.72', port: 26379}, + {host: '10.6.0.73', port: 26379}, + ] + + ## Second cluster that will host the queues, shared state, and actionable + gitlab_rails['redis_queues_instance'] = 'redis://:@gitlab-redis-persistent' + gitlab_rails['redis_shared_state_instance'] = 'redis://:@gitlab-redis-persistent' + gitlab_rails['redis_actioncable_instance'] = 'redis://:@gitlab-redis-persistent' + + gitlab_rails['redis_queues_sentinels'] = [ + {host: '10.6.0.81', port: 26379}, + {host: '10.6.0.82', port: 26379}, + {host: '10.6.0.83', port: 26379}, + ] + gitlab_rails['redis_shared_state_sentinels'] = [ + {host: '10.6.0.81', port: 26379}, + {host: '10.6.0.82', port: 26379}, + {host: '10.6.0.83', port: 26379}, + ] + gitlab_rails['redis_actioncable_sentinels'] = [ + {host: '10.6.0.81', port: 26379}, + {host: '10.6.0.82', port: 26379}, + {host: '10.6.0.83', port: 26379}, + ] + + # Set the network addresses that the exporters used for monitoring will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + gitlab_workhorse['prometheus_listen_addr'] = '0.0.0.0:9229' + sidekiq['listen_address'] = "0.0.0.0" + puma['listen'] = '0.0.0.0' + + # Add the monitoring node's IP address to the monitoring whitelist and allow it to + # scrape the NGINX metrics + gitlab_rails['monitoring_whitelist'] = ['10.6.0.121/32', '127.0.0.0/8'] + nginx['status']['options']['allow'] = ['10.6.0.121/32', '127.0.0.0/8'] + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. If you're using [Gitaly with TLS support](#gitaly-tls-support), make sure the + `git_data_dirs` entry is configured with `tls` instead of `tcp`: + + ```ruby + git_data_dirs({ + 'default' => { 'gitaly_address' => 'tls://gitaly1.internal:9999' }, + 'storage1' => { 'gitaly_address' => 'tls://gitaly1.internal:9999' }, + 'storage2' => { 'gitaly_address' => 'tls://gitaly2.internal:9999' }, + }) + ``` + + 1. Copy the cert into `/etc/gitlab/trusted-certs`: + + ```shell + sudo cp cert.pem /etc/gitlab/trusted-certs/ + ``` + +1. If you're [using NFS](#configure-nfs-optional): + 1. If necessary, install the NFS client utility packages using the following + commands: + + ```shell + # Ubuntu/Debian + apt-get install nfs-common + + # CentOS/Red Hat + yum install nfs-utils nfs-utils-lib + ``` + + 1. Specify the necessary NFS mounts in `/etc/fstab`. + The exact contents of `/etc/fstab` will depend on how you chose + to configure your NFS server. See the [NFS documentation](../high_availability/nfs.md) + for examples and the various options. + + 1. Create the shared directories. These may be different depending on your NFS + mount locations. + + ```shell + mkdir -p /var/opt/gitlab/.ssh /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/git-data + ``` + + 1. Edit `/etc/gitlab/gitlab.rb` and use the following configuration: + + ```ruby + ## Prevent GitLab from starting if NFS data mounts are not available + high_availability['mountpoint'] = '/var/opt/gitlab/git-data' + + ## Ensure UIDs and GIDs match between servers for permissions via NFS + user['uid'] = 9000 + user['gid'] = 9000 + web_server['uid'] = 9001 + web_server['gid'] = 9001 + registry['uid'] = 9002 + registry['gid'] = 9002 + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Confirm the node can connect to Gitaly: + + ```shell + sudo gitlab-rake gitlab:gitaly:check + ``` + + Then, tail the logs to see the requests: + + ```shell + sudo gitlab-ctl tail gitaly + ``` + +1. Optionally, from the Gitaly servers, confirm that Gitaly can perform callbacks to the internal API: + + ```shell + sudo /opt/gitlab/embedded/service/gitlab-shell/bin/check -config /opt/gitlab/embedded/service/gitlab-shell/config.yml + ``` + +NOTE: **Note:** +When you specify `https` in the `external_url`, as in the example +above, GitLab assumes you have SSL certificates in `/etc/gitlab/ssl/`. If +certificates are not present, NGINX will fail to start. See the +[NGINX documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) +for more information. + +### GitLab Rails post-configuration + +Initialize the GitLab database, by running the following in one of the Rails nodes: + +```shell +sudo gitlab-rake gitlab:db:configure +``` + +NOTE: **Note:** +If you encounter a `rake aborted!` error stating that PgBouncer is failing to connect to +PostgreSQL it may be that your PgBouncer node's IP address is missing from +PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` on your database nodes. See +[PgBouncer error `ERROR: pgbouncer cannot connect to server`](troubleshooting.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) +in the Troubleshooting section before proceeding. + + + +## Configure Prometheus + +The Omnibus GitLab package can be used to configure a standalone Monitoring node +running [Prometheus](../monitoring/prometheus/index.md) and +[Grafana](../monitoring/performance/grafana_configuration.md). + +The following IP will be used as an example: + +- `10.6.0.121`: Prometheus + +To configure the Monitoring node: + +1. SSH into the Monitoring node. +1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page. + Do not complete any other steps on the download page. + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace + the file of the same name on this server. If that file is not on this server, + add the file from your Consul server to this server. + +1. Edit `/etc/gitlab/gitlab.rb` and add the contents: + + ```ruby + external_url 'http://gitlab.example.com' + + # Disable all other services + gitlab_rails['auto_migrate'] = false + alertmanager['enable'] = false + gitaly['enable'] = false + gitlab_exporter['enable'] = false + gitlab_workhorse['enable'] = false + nginx['enable'] = true + postgres_exporter['enable'] = false + postgresql['enable'] = false + redis['enable'] = false + redis_exporter['enable'] = false + sidekiq['enable'] = false + puma['enable'] = false + unicorn['enable'] = false + node_exporter['enable'] = false + gitlab_exporter['enable'] = false + + # Enable Prometheus + prometheus['enable'] = true + prometheus['listen_address'] = '0.0.0.0:9090' + prometheus['monitor_kubernetes'] = false + + # Enable Login form + grafana['disable_login_form'] = false + + # Enable Grafana + grafana['enable'] = true + grafana['admin_password'] = '' + + # Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) + } + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. In the GitLab UI, set `admin/application_settings/metrics_and_profiling` > Metrics - Grafana to `/-/grafana` to +`http[s]:///-/grafana` + + + +## Configure the object storage + +GitLab supports using an object storage service for holding numerous types of data. +It's recommended over [NFS](#configure-nfs-optional) and in general it's better +in larger setups as object storage is typically much more performant, reliable, +and scalable. + +Object storage options that GitLab has tested, or is aware of customers using include: + +- SaaS/Cloud solutions such as [Amazon S3](https://aws.amazon.com/s3/), [Google cloud storage](https://cloud.google.com/storage). +- On-premises hardware and appliances from various storage vendors. +- MinIO. There is [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. + +For configuring GitLab to use Object Storage refer to the following guides +based on what features you intend to use: + +1. Configure [object storage for backups](../../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage). +1. Configure [object storage for job artifacts](../job_artifacts.md#using-object-storage) + including [incremental logging](../job_logs.md#new-incremental-logging-architecture). +1. Configure [object storage for LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage). +1. Configure [object storage for uploads](../uploads.md#using-object-storage-core-only). +1. Configure [object storage for merge request diffs](../merge_request_diffs.md#using-object-storage). +1. Configure [object storage for Container Registry](../packages/container_registry.md#use-object-storage) (optional feature). +1. Configure [object storage for Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage) (optional feature). +1. Configure [object storage for packages](../packages/index.md#using-object-storage) (optional feature). **(PREMIUM ONLY)** +1. Configure [object storage for Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature). **(PREMIUM ONLY)** +1. Configure [object storage for Pseudonymizer](../pseudonymizer.md#configuration) (optional feature). **(ULTIMATE ONLY)** +1. Configure [object storage for autoscale Runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional - for improved performance). +1. Configure [object storage for Terraform state files](../terraform_state.md#using-object-storage-core-only). + +Using separate buckets for each data type is the recommended approach for GitLab. + +A limitation of our configuration is that each use of object storage is separately configured. +[We have an issue for improving this](https://gitlab.com/gitlab-org/gitlab/-/issues/23345) +and easily using one bucket with separate folders is one improvement that this might bring. + +There is at least one specific issue with using the same bucket: +when GitLab is deployed with the Helm chart restore from backup +[will not properly function](https://docs.gitlab.com/charts/advanced/external-object-storage/#lfs-artifacts-uploads-packages-external-diffs-pseudonymizer) +unless separate buckets are used. + +One risk of using a single bucket would be if your organization decided to +migrate GitLab to the Helm deployment in the future. GitLab would run, but the situation with +backups might not be realized until the organization had a critical requirement for the backups to +work. + + + +## Configure NFS (optional) + +[Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly) +are recommended over NFS wherever possible for improved performance. If you intend +to use GitLab Pages, this currently [requires NFS](troubleshooting.md#gitlab-pages-requires-nfs). + +See how to [configure NFS](../high_availability/nfs.md). + + + +## Troubleshooting + +See the [troubleshooting documentation](troubleshooting.md). + + diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index 0b4114bca6e..14685ffa53d 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -1,49 +1,54 @@ --- reading_time: true +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Reference architecture: up to 5,000 users +# Reference architecture: up to 5,000 users **(PREMIUM ONLY)** -This page describes GitLab reference architecture for up to 5,000 users. -For a full list of reference architectures, see +This page describes GitLab reference architecture for up to 5,000 users. For a +full list of reference architectures, see [Available reference architectures](index.md#available-reference-architectures). NOTE: **Note:** -The 5,000-user reference architecture documented below is -designed to help your organization achieve a highly-available GitLab deployment. -If you do not have the expertise or need to maintain a highly-available -environment, you can have a simpler and less costly-to-operate environment by -following the [2,000-user reference architecture](2k_users.md). +This reference architecture is designed to help your organization achieve a +highly-available GitLab deployment. If you do not have the expertise or need to +maintain a highly-available environment, you can have a simpler and less +costly-to-operate environment by using the +[2,000-user reference architecture](2k_users.md). > - **Supported users (approximate):** 5,000 -> - **High Availability:** True -> - **Test RPS rates:** API: 100 RPS, Web: 10 RPS, Git: 10 RPS - -| Service | Nodes | Configuration | GCP | AWS | Azure | -|--------------------------------------------------------------|-------|---------------------------------|-----------------|-----------------------|----------------| -| External load balancing node | 1 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Redis | 3 | 2 vCPU, 7.5GB Memory | `n1-standard-2` | `m5.large` | `D2s v3` | -| Consul + Sentinel | 3 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| PostgreSQL | 3 | 2 vCPU, 7.5GB Memory | `n1-standard-2` | `m5.large` | `D2s v3` | -| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Internal load balancing node | 1 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Gitaly | 2 minimum | 8 vCPU, 30GB Memory | `n1-standard-8` | `m5.2xlarge` | `D8s v3` | -| Sidekiq | 4 | 2 vCPU, 7.5GB Memory | `n1-standard-2` | `m5.large` | `D2s v3` | -| GitLab Rails | 3 | 16 vCPU, 14.4GB Memory | `n1-highcpu-16` | `c5.4xlarge` | `F16s v2` | -| Monitoring node | 1 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Object Storage | n/a | n/a | n/a | n/a | n/a | -| NFS Server (optional, not recommended) | 1 | 4 vCPU, 3.6GB Memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | - -The architectures were built and tested with the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) -CPU platform on GCP. On different hardware you may find that adjustments, either lower -or higher, are required for your CPU or Node counts accordingly. For more information, a -[Sysbench](https://github.com/akopytov/sysbench) benchmark of the CPU can be found -[here](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). - -For data objects such as LFS, Uploads, Artifacts, etc, an [object storage service](#configure-the-object-storage) -is recommended over NFS where possible, due to better performance and availability. -Since this doesn't require a node to be set up, it's marked as not applicable (n/a) -in the table above. +> - **High Availability:** Yes +> - **Test requests per second (RPS) rates:** API: 100 RPS, Web: 10 RPS, Git: 10 RPS + +| Service | Nodes | Configuration | GCP | AWS | Azure | +|--------------------------------------------|-------------|-------------------------|----------------|-------------|----------| +| External load balancing node | 1 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Redis | 3 | 2 vCPU, 7.5GB memory | n1-standard-2 | m5.large | D2s v3 | +| Consul + Sentinel | 3 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| PostgreSQL | 3 | 2 vCPU, 7.5GB memory | n1-standard-2 | m5.large | D2s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Internal load balancing node | 1 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Gitaly | 2 (minimum) | 8 vCPU, 30GB memory | n1-standard-8 | m5.2xlarge | D8s v3 | +| Sidekiq | 4 | 2 vCPU, 7.5GB memory | n1-standard-2 | m5.large | D2s v3 | +| GitLab Rails | 3 | 16 vCPU, 14.4GB memory | n1-highcpu-16 | c5.4xlarge | F16s v2 | +| Monitoring node | 1 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Object Storage | n/a | n/a | n/a | n/a | n/a | +| NFS Server (optional, not recommended) | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | + +The Google Cloud Platform (GCP) architectures were built and tested using the +[Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) +CPU platform. On different hardware you may find that adjustments, either lower +or higher, are required for your CPU or node counts. For more information, see +our [Sysbench](https://github.com/akopytov/sysbench)-based +[CPU benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). + +For data objects (such as LFS, Uploads, or Artifacts), an +[object storage service](#configure-the-object-storage) is recommended instead +of NFS where possible, due to better performance and availability. Since this +doesn't require a node to be set up, *Object Storage* is noted as not +applicable (n/a) in the previous table. ## Setup components @@ -1439,7 +1444,7 @@ On each node perform the following: 1. Specify the necessary NFS mounts in `/etc/fstab`. The exact contents of `/etc/fstab` will depend on how you chose - to configure your NFS server. See the [NFS documentation](../high_availability/nfs.md) + to configure your NFS server. See the [NFS documentation](../nfs.md) for examples and the various options. 1. Create the shared directories. These may be different depending on your NFS @@ -1748,7 +1753,7 @@ work. are recommended over NFS wherever possible for improved performance. If you intend to use GitLab Pages, this currently [requires NFS](troubleshooting.md#gitlab-pages-requires-nfs). -See how to [configure NFS](../high_availability/nfs.md). +See how to [configure NFS](../nfs.md).
diff --git a/doc/administration/reference_architectures/img/reference-architectures.png b/doc/administration/reference_architectures/img/reference-architectures.png index e15609e78e1..0f8e663b57b 100644 Binary files a/doc/administration/reference_architectures/img/reference-architectures.png and b/doc/administration/reference_architectures/img/reference-architectures.png differ diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index 8fde71a66bf..4f7be2413dd 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -1,36 +1,45 @@ --- type: reference, concepts +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Reference architectures - +# Reference architectures You can set up GitLab on a single server or scale it up to serve many users. -This page details the recommended Reference Architectures that were built and verified by GitLab's Quality and Support teams. +This page details the recommended Reference Architectures that were built and +verified by GitLab's Quality and Support teams. -Below is a chart representing each architecture tier and the number of users they can handle. As your number of users grow with time, it’s recommended that you scale GitLab accordingly. +Below is a chart representing each architecture tier and the number of users +they can handle. As your number of users grow with time, it’s recommended that +you scale GitLab accordingly. ![Reference Architectures](img/reference-architectures.png) -Testing on these reference architectures were performed with [GitLab's Performance Tool](https://gitlab.com/gitlab-org/quality/performance) -at specific coded workloads, and the throughputs used for testing were calculated based on sample customer data. -After selecting the reference architecture that matches your scale, refer to -[Configure GitLab to Scale](#configure-gitlab-to-scale) to see the components -involved, and how to configure them. +Testing on these reference architectures were performed with +[GitLab's Performance Tool](https://gitlab.com/gitlab-org/quality/performance) +at specific coded workloads, and the throughputs used for testing were +calculated based on sample customer data. Select the +[reference architecture](#available-reference-architectures) that matches your scale. -Each endpoint type is tested with the following number of requests per second (RPS) per 1000 users: +Each endpoint type is tested with the following number of requests per second (RPS) +per 1,000 users: - API: 20 RPS - Web: 2 RPS - Git: 2 RPS -For GitLab instances with less than 2,000 users, it's recommended that you use the [default setup](#automated-backups-core-only) -by [installing GitLab](../../install/README.md) on a single machine to minimize maintenance and resource costs. +For GitLab instances with less than 2,000 users, it's recommended that you use +the [default setup](#automated-backups-core-only) by +[installing GitLab](../../install/README.md) on a single machine to minimize +maintenance and resource costs. -If your organization has more than 2,000 users, the recommendation is to scale GitLab's components to multiple -machine nodes. The machine nodes are grouped by component(s). The addition of these -nodes increases the performance and scalability of to your GitLab instance. +If your organization has more than 2,000 users, the recommendation is to scale +GitLab's components to multiple machine nodes. The machine nodes are grouped by +components. The addition of these nodes increases the performance and +scalability of to your GitLab instance. When scaling GitLab, there are several factors to consider: @@ -39,12 +48,13 @@ When scaling GitLab, there are several factors to consider: - The application nodes connects to a shared file server and PostgreSQL and Redis services on the backend. NOTE: **Note:** -Depending on your workflow, the following recommended -reference architectures may need to be adapted accordingly. Your workload -is influenced by factors including how active your users are, -how much automation you use, mirroring, and repository/change size. Additionally the -displayed memory values are provided by [GCP machine types](https://cloud.google.com/compute/docs/machine-types). -For different cloud vendors, attempt to select options that best match the provided architecture. +Depending on your workflow, the following recommended reference architectures +may need to be adapted accordingly. Your workload is influenced by factors +including how active your users are, how much automation you use, mirroring, +and repository/change size. Additionally the displayed memory values are +provided by [GCP machine types](https://cloud.google.com/compute/docs/machine-types). +For different cloud vendors, attempt to select options that best match the +provided architecture. ## Available reference architectures @@ -60,14 +70,14 @@ The following reference architectures are available: ## Availability Components -GitLab comes with the following components for your use, listed from -least to most complex: +GitLab comes with the following components for your use, listed from least to +most complex: -1. [Automated backups](#automated-backups-core-only) -1. [Traffic load balancer](#traffic-load-balancer-starter-only) -1. [Zero downtime updates](#zero-downtime-updates-starter-only) -1. [Automated database failover](#automated-database-failover-premium-only) -1. [Instance level replication with GitLab Geo](#instance-level-replication-with-gitlab-geo-premium-only) +- [Automated backups](#automated-backups-core-only) +- [Traffic load balancer](#traffic-load-balancer-starter-only) +- [Zero downtime updates](#zero-downtime-updates-starter-only) +- [Automated database failover](#automated-database-failover-premium-only) +- [Instance level replication with GitLab Geo](#instance-level-replication-with-gitlab-geo-premium-only) As you implement these components, begin with a single server and then do backups. Only after completing the first server should you proceed to the next. @@ -115,7 +125,8 @@ to the default installation: > - Supported tiers: [GitLab Starter, Premium, and Ultimate](https://about.gitlab.com/pricing/) GitLab supports [zero-downtime updates](https://docs.gitlab.com/omnibus/update/#zero-downtime-updates). -Although you can perform zero-downtime updates with a single GitLab node, the recommendation is to separate GitLab into several application nodes. +Single GitLab nodes can be updated with only a [few minutes of downtime](https://docs.gitlab.com/omnibus/update/README.html#single-node-deployment). +To avoid this, we recommend to separate GitLab into several application nodes. As long as at least one of each component is online and capable of handling the instance's usage load, your team's productivity will not be interrupted during the update. ### Automated database failover **(PREMIUM ONLY)** @@ -140,40 +151,7 @@ is recommended. instance to other geographical locations as a read-only fully operational instance that can also be promoted in case of disaster. -## Configure GitLab to scale - -NOTE: **Note:** -From GitLab 13.0, using NFS for Git repositories is deprecated. In GitLab 14.0, support for NFS for Git repositories is scheduled to be removed. Upgrade to [Gitaly Cluster](../gitaly/praefect.md) as soon as possible. - -The following components are the ones you need to configure in order to scale -GitLab. They are listed in the order you'll typically configure them if they are -required by your [reference architecture](#reference-architectures) of choice. - -Most of them are bundled in the GitLab deb/rpm package (called Omnibus GitLab), -but depending on your system architecture, you may require some components which are -not included in it. If required, those should be configured before -setting up components provided by GitLab. Advice on how to select the right -solution for your organization is provided in the configuration instructions -column. - -| Component | Description | Configuration instructions | Bundled with Omnibus GitLab | -|-----------|-------------|----------------------------| -| Load balancer(s) ([6](#footnotes)) | Handles load balancing, typically when you have multiple GitLab application services nodes | [Load balancer configuration](../high_availability/load_balancer.md) ([6](#footnotes)) | No | -| Object storage service ([4](#footnotes)) | Recommended store for shared data objects | [Object Storage configuration](../object_storage.md) | No | -| NFS ([5](#footnotes)) ([7](#footnotes)) | Shared disk storage service. Can be used as an alternative Object Storage. Required for GitLab Pages | [NFS configuration](../high_availability/nfs.md) | No | -| [Consul](../../development/architecture.md#consul) ([3](#footnotes)) | Service discovery and health checks/failover | [Consul configuration](../high_availability/consul.md) **(PREMIUM ONLY)** | Yes | -| [PostgreSQL](../../development/architecture.md#postgresql) | Database | [PostgreSQL configuration](https://docs.gitlab.com/omnibus/settings/database.html) | Yes | -| [PgBouncer](../../development/architecture.md#pgbouncer) | Database connection pooler | [PgBouncer configuration](../high_availability/pgbouncer.md#running-pgbouncer-as-part-of-a-non-ha-gitlab-installation) **(PREMIUM ONLY)** | Yes | -| Repmgr | PostgreSQL cluster management and failover | [PostgreSQL and Repmgr configuration](../postgresql/replication_and_failover.md) | Yes | -| Patroni | An alternative PostgreSQL cluster management and failover | [PostgreSQL and Patroni configuration](../postgresql/replication_and_failover.md#patroni) | Yes | -| [Redis](../../development/architecture.md#redis) ([3](#footnotes)) | Key/value store for fast data lookup and caching | [Redis configuration](../high_availability/redis.md) | Yes | -| Redis Sentinel | Redis | [Redis Sentinel configuration](../high_availability/redis.md) | Yes | -| [Gitaly](../../development/architecture.md#gitaly) ([2](#footnotes)) ([7](#footnotes)) | Provides access to Git repositories | [Gitaly configuration](../gitaly/index.md#run-gitaly-on-its-own-server) | Yes | -| [Sidekiq](../../development/architecture.md#sidekiq) | Asynchronous/background jobs | [Sidekiq configuration](../high_availability/sidekiq.md) | Yes | -| [GitLab application services](../../development/architecture.md#unicorn)([1](#footnotes)) | Puma/Unicorn, Workhorse, GitLab Shell - serves front-end requests (UI, API, Git over HTTP/SSH) | [GitLab app scaling configuration](../high_availability/gitlab.md) | Yes | -| [Prometheus](../../development/architecture.md#prometheus) and [Grafana](../../development/architecture.md#grafana) | GitLab environment monitoring | [Monitoring node for scaling](../high_availability/monitoring_node.md) | Yes | - -### Configuring select components with Cloud Native Helm +## Configuring select components with Cloud Native Helm We also provide [Helm charts](https://docs.gitlab.com/charts/) as a Cloud Native installation method for GitLab. For the reference architectures, select components can be set up in this @@ -191,50 +169,3 @@ specs, only translated into Kubernetes resources. For example, if you were to set up a 50k installation with the Rails nodes being run in Helm, then the same amount of resources as given for Omnibus should be given to the Kubernetes cluster with the Rails nodes broken down into a number of smaller Pods across that cluster. - -## Footnotes - -1. In our architectures we run each GitLab Rails node using the Puma webserver - and have its number of workers set to 90% of available CPUs along with four threads. For - nodes that are running Rails with other components the worker value should be reduced - accordingly where we've found 50% achieves a good balance but this is dependent - on workload. - -1. Gitaly node requirements are dependent on customer data, specifically the number of - projects and their sizes. We recommend that each Gitaly node should store no more than 5TB of data - and have the number of [`gitaly-ruby` workers](../gitaly/index.md#gitaly-ruby) - set to 20% of available CPUs. Additional nodes should be considered in conjunction - with a review of expected data size and spread based on the recommendations above. - -1. Recommended Redis setup differs depending on the size of the architecture. - For smaller architectures (less than 3,000 users) a single instance should suffice. - For medium sized installs (3,000 - 5,000) we suggest one Redis cluster for all - classes and that Redis Sentinel is hosted alongside Consul. - For larger architectures (10,000 users or more) we suggest running a separate - [Redis Cluster](../redis/replication_and_failover.md#running-multiple-redis-clusters) for the Cache class - and another for the Queues and Shared State classes respectively. We also recommend - that you run the Redis Sentinel clusters separately for each Redis Cluster. - -1. For data objects such as LFS, Uploads, Artifacts, etc. We recommend an [Object Storage service](../object_storage.md) - over NFS where possible, due to better performance. - -1. NFS can be used as an alternative for object storage but this isn't typically - recommended for performance reasons. Note however it is required for [GitLab - Pages](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/196). - -1. Our architectures have been tested and validated with [HAProxy](https://www.haproxy.org/) - as the load balancer. Although other load balancers with similar feature sets - could also be used, those load balancers have not been validated. - -1. We strongly recommend that any Gitaly or NFS nodes be set up with SSD disks over - HDD with a throughput of at least 8,000 IOPS for read operations and 2,000 IOPS for write - as these components have heavy I/O. These IOPS values are recommended only as a starter - as with time they may be adjusted higher or lower depending on the scale of your - environment's workload. If you're running the environment on a Cloud provider - you may need to refer to their documentation on how configure IOPS correctly. - -1. The architectures were built and tested with the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) - CPU platform on GCP. On different hardware you may find that adjustments, either lower - or higher, are required for your CPU or Node counts accordingly. For more information, a - [Sysbench](https://github.com/akopytov/sysbench) benchmark of the CPU can be found - [here](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). diff --git a/doc/administration/reference_architectures/troubleshooting.md b/doc/administration/reference_architectures/troubleshooting.md index 35bdb65c810..db2e9b89ba2 100644 --- a/doc/administration/reference_architectures/troubleshooting.md +++ b/doc/administration/reference_architectures/troubleshooting.md @@ -1,3 +1,9 @@ +--- +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Troubleshooting a reference architecture setup This page serves as the troubleshooting documentation if you followed one of @@ -17,7 +23,7 @@ with the Fog library that GitLab uses. Symptoms include: ### GitLab Pages requires NFS If you intend to use [GitLab Pages](../../user/project/pages/index.md), this currently requires -[NFS](../high_availability/nfs.md). There is [work in progress](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/196) +[NFS](../nfs.md). There is [work in progress](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/196) to remove this dependency. In the future, GitLab Pages may use [object storage](https://gitlab.com/gitlab-org/gitlab/-/issues/208135). @@ -524,7 +530,7 @@ To restart either service, run `gitlab-ctl restart SERVICE` For PostgreSQL, it is usually safe to restart the master node by default. Automatic failover defaults to a 1 minute timeout. Provided the database returns before then, nothing else needs to be done. To be safe, you can stop `repmgrd` on the standby nodes first with `gitlab-ctl stop repmgrd`, then start afterwards with `gitlab-ctl start repmgrd`. -On the Consul server nodes, it is important to restart the Consul service in a controlled fashion. Read our [Consul documentation](../high_availability/consul.md#restarting-the-server-cluster) for instructions on how to restart the service. +On the Consul server nodes, it is important to restart the Consul service in a controlled fashion. Read our [Consul documentation](../consul.md#restart-consul) for instructions on how to restart the service. ### `gitlab-ctl repmgr-check-master` command produces errors diff --git a/doc/administration/reply_by_email.md b/doc/administration/reply_by_email.md index 6d7069dd461..62645ad17a1 100644 --- a/doc/administration/reply_by_email.md +++ b/doc/administration/reply_by_email.md @@ -1,5 +1,13 @@ +--- +stage: Plan +group: Certify +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Reply by email +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/1173) in GitLab 8.0. + GitLab can be set up to allow users to comment on issues and merge requests by replying to notification emails. diff --git a/doc/administration/reply_by_email_postfix_setup.md b/doc/administration/reply_by_email_postfix_setup.md index 86854a2a7b6..f950134889d 100644 --- a/doc/administration/reply_by_email_postfix_setup.md +++ b/doc/administration/reply_by_email_postfix_setup.md @@ -306,7 +306,7 @@ Courier, which we will install later to add IMAP authentication, requires mailbo ```shell Trying 123.123.123.123... - Connected to mail.example.gitlab.com. + Connected to mail.gitlab.example.com. Escape character is '^]'. - OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE ACL ACL2=UNION] Courier-IMAP ready. Copyright 1998-2011 Double Precision, Inc. See COPYING for distribution information. ``` diff --git a/doc/administration/repository_checks.md b/doc/administration/repository_checks.md index 6d9ab723d2f..7d79840b56d 100644 --- a/doc/administration/repository_checks.md +++ b/doc/administration/repository_checks.md @@ -1,3 +1,10 @@ +--- +stage: Create +group: Editor +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers" +type: reference +--- + # Repository checks > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/3232) in GitLab 8.7. diff --git a/doc/administration/repository_storage_paths.md b/doc/administration/repository_storage_paths.md index 5520eeace0a..16e17458e10 100644 --- a/doc/administration/repository_storage_paths.md +++ b/doc/administration/repository_storage_paths.md @@ -68,7 +68,7 @@ the example below, we add two more mount points that are named `nfs_1` and `nfs_ respectively. NOTE: **Note:** -This example uses NFS. We do not recommend using EFS for storage as it may impact GitLab's performance. See the [relevant documentation](high_availability/nfs.md#avoid-using-awss-elastic-file-system-efs) for more details. +This example uses NFS. We do not recommend using EFS for storage as it may impact GitLab's performance. See the [relevant documentation](nfs.md#avoid-using-awss-elastic-file-system-efs) for more details. **For installations from source** @@ -89,7 +89,7 @@ This example uses NFS. We do not recommend using EFS for storage as it may impac 1. [Restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. ->**Note:** +NOTE: **Note:** The [`gitlab_shell: repos_path` entry](https://gitlab.com/gitlab-org/gitlab-foss/-/blob/8-9-stable/config/gitlab.yml.example#L457) in `gitlab.yml` will be deprecated and replaced by `repositories: storages` in the future, so if you are upgrading from a version prior to 8.10, make sure to add the configuration diff --git a/doc/administration/server_hooks.md b/doc/administration/server_hooks.md index 2fea000b799..ab808fc28d8 100644 --- a/doc/administration/server_hooks.md +++ b/doc/administration/server_hooks.md @@ -163,13 +163,13 @@ them as they can change. The following server hooks have been re-implemented in Go: - `pre-receive`, with the Go implementation used by default. To use the Ruby implementation instead, - [disable](../operations/feature_flags.md#enable-or-disable-feature-flag-strategies) the - `:gitaly_go_preceive_hook` feature flag. + [disable](feature_flags.md#enable-or-disable-the-feature) the `:gitaly_go_preceive_hook` feature + flag. - `update`, with the Go implementation used by default. To use the Ruby implementation instead, - [disable](../operations/feature_flags.md#enable-or-disable-feature-flag-strategies) the - `:gitaly_go_update_hook` feature flag. + [disable](feature_flags.md#enable-or-disable-the-feature) the `:gitaly_go_update_hook` feature + flag. - `post-receive`, however the Ruby implementation is used by default. To use the Go implementation - instead, [enable](../operations/feature_flags.md#enable-or-disable-feature-flag-strategies) the + instead, [enable](feature_flags.md#enable-or-disable-the-feature) the `:gitaly_go_postreceive_hook` feature flag. ## Custom error messages diff --git a/doc/administration/sidekiq.md b/doc/administration/sidekiq.md new file mode 100644 index 00000000000..23e870dbb82 --- /dev/null +++ b/doc/administration/sidekiq.md @@ -0,0 +1,188 @@ +--- +type: reference +--- + +# Configuring Sidekiq + +This section discusses how to configure an external Sidekiq instance using the +bundled Sidekiq in the GitLab package. + +Sidekiq requires connection to the Redis, PostgreSQL and Gitaly instance. +To configure the Sidekiq node: + +1. SSH into the Sidekiq server. +1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab package +you want using steps 1 and 2 from the GitLab downloads page. +**Do not complete any other steps on the download page.** +1. Open `/etc/gitlab/gitlab.rb` with your editor. +1. Generate the Sidekiq configuration: + + ```ruby + sidekiq['listen_address'] = "10.10.1.48" + + ## Optional: Enable extra Sidekiq processes + sidekiq_cluster['enable'] = true + sidekiq_cluster['enable'] = true + "elastic_indexer" + ] + ``` + +1. Setup Sidekiq's connection to Redis: + + ```ruby + ## Must be the same in every sentinel node + redis['master_name'] = 'gitlab-redis' + + ## The same password for Redis authentication you set up for the master node. + redis['master_password'] = 'YOUR_PASSOWORD' + + ## A list of sentinels with `host` and `port` + gitlab_rails['redis_sentinels'] = [ + {'host' => '10.10.1.34', 'port' => 26379}, + {'host' => '10.10.1.35', 'port' => 26379}, + {'host' => '10.10.1.36', 'port' => 26379}, + ] + ``` + +1. Set up Sidekiq's connection to Gitaly: + + ```ruby + git_data_dirs({ + 'default' => { 'gitaly_address' => 'tcp://gitaly:8075' }, + }) + gitlab_rails['gitaly_token'] = 'YOUR_TOKEN' + ``` + +1. Set up Sidekiq's connection to PostgreSQL: + + ```ruby + gitlab_rails['db_host'] = '10.10.1.30' + gitlab_rails['db_password'] = 'YOUR_PASSOWORD' + gitlab_rails['db_port'] = '5432' + gitlab_rails['db_adapter'] = 'postgresql' + gitlab_rails['db_encoding'] = 'unicode' + gitlab_rails['auto_migrate'] = false + ``` + + Remember to add the Sidekiq nodes to PostgreSQL's trusted addresses: + + ```ruby + postgresql['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 10.10.1.30/32 10.10.1.31/32 10.10.1.32/32 10.10.1.33/32 10.10.1.38/32) + ``` + +1. Disable other services: + + ```ruby + nginx['enable'] = false + grafana['enable'] = false + prometheus['enable'] = false + gitlab_rails['auto_migrate'] = false + alertmanager['enable'] = false + gitaly['enable'] = false + gitlab_monitor['enable'] = false + gitlab_workhorse['enable'] = false + nginx['enable'] = false + postgres_exporter['enable'] = false + postgresql['enable'] = false + redis['enable'] = false + redis_exporter['enable'] = false + puma['enable'] = false + gitlab_exporter['enable'] = false + ``` + +1. Run `gitlab-ctl reconfigure`. + +NOTE: **Note:** +You will need to restart the Sidekiq nodes after an update has occurred and database +migrations performed. + +## Example configuration + +Here's what the ending `/etc/gitlab/gitlab.rb` would look like: + +```ruby +######################################## +##### Services Disabled ### +######################################## + +nginx['enable'] = false +grafana['enable'] = false +prometheus['enable'] = false +gitlab_rails['auto_migrate'] = false +alertmanager['enable'] = false +gitaly['enable'] = false +gitlab_monitor['enable'] = false +gitlab_workhorse['enable'] = false +nginx['enable'] = false +postgres_exporter['enable'] = false +postgresql['enable'] = false +redis['enable'] = false +redis_exporter['enable'] = false +puma['enable'] = false +gitlab_exporter['enable'] = false + +######################################## +#### Redis ### +######################################## + +## Must be the same in every sentinel node +redis['master_name'] = 'gitlab-redis' + +## The same password for Redis authentication you set up for the master node. +redis['master_password'] = 'YOUR_PASSOWORD' + +## A list of sentinels with `host` and `port` +gitlab_rails['redis_sentinels'] = [ + {'host' => '10.10.1.34', 'port' => 26379}, + {'host' => '10.10.1.35', 'port' => 26379}, + {'host' => '10.10.1.36', 'port' => 26379}, + ] + +####################################### +### Gitaly ### +####################################### + +git_data_dirs({ + 'default' => { 'gitaly_address' => 'tcp://gitaly:8075' }, +}) +gitlab_rails['gitaly_token'] = 'YOUR_TOKEN' + +####################################### +### Postgres ### +####################################### +gitlab_rails['db_host'] = '10.10.1.30' +gitlab_rails['db_password'] = 'YOUR_PASSOWORD' +gitlab_rails['db_port'] = '5432' +gitlab_rails['db_adapter'] = 'postgresql' +gitlab_rails['db_encoding'] = 'unicode' +gitlab_rails['auto_migrate'] = false + +####################################### +### Sidekiq configuration ### +####################################### +sidekiq['listen_address'] = "10.10.1.48" + +####################################### +### Monitoring configuration ### +####################################### +consul['enable'] = true +consul['monitoring_service_discovery'] = true + +consul['configuration'] = { + bind_addr: '10.10.1.48', + retry_join: %w(10.10.1.34 10.10.1.35 10.10.1.36) +} + +# Set the network addresses that the exporters will listen on +node_exporter['listen_address'] = '10.10.1.48:9100' + +# Rails Status for prometheus +gitlab_rails['monitoring_whitelist'] = ['10.10.1.42', '127.0.0.1'] +``` + +## Further reading + +Related Sidekiq configuration: + +1. [Extra Sidekiq processes](operations/extra_sidekiq_processes.md) +1. [Using the GitLab-Sidekiq chart](https://docs.gitlab.com/charts/charts/gitlab/sidekiq/) diff --git a/doc/administration/smime_signing_email.md b/doc/administration/smime_signing_email.md index 304b65917c1..b1e7e349978 100644 --- a/doc/administration/smime_signing_email.md +++ b/doc/administration/smime_signing_email.md @@ -3,7 +3,7 @@ Notification emails sent by GitLab can be signed with S/MIME for improved security. -> **Note:** +NOTE: **Note:** Please be aware that S/MIME certificates and TLS/SSL certificates are not the same and are used for different purposes: TLS creates a secure channel, whereas S/MIME signs and/or encrypts the message itself diff --git a/doc/administration/snippets/index.md b/doc/administration/snippets/index.md index cf3d8bec1a6..95de3b8c183 100644 --- a/doc/administration/snippets/index.md +++ b/doc/administration/snippets/index.md @@ -1,5 +1,8 @@ --- type: reference, howto +stage: Create +group: Editor +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers" --- # Snippets settings **(CORE ONLY)** @@ -10,15 +13,15 @@ Adjust the snippets' settings of your GitLab instance. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/31133) in GitLab 12.6. -You can set a content size max limit in snippets. This limit can prevent -abuses of the feature. The default content size limit is **52428800 Bytes** (50MB). +You can set a maximum content size limit for snippets. This limit can prevent +abuse of the feature. The default value is **52428800 Bytes** (50 MB). ### How does it work? -The content size limit will be applied when a snippet is created or -updated. Nevertheless, in order not to break any existing snippet, -the limit will only be enforced in stored snippets when the content -is updated. +The content size limit will be applied when a snippet is created or updated. + +In order not to break any existing snippets, the limit doesn't have any +effect on them until a snippet is edited again and the content changes. ### Snippets size limit configuration @@ -27,7 +30,7 @@ In order to configure this setting, use either the Rails console or the [Application settings API](../../api/settings.md). NOTE: **IMPORTANT:** -The value of the limit **must** be in Bytes. +The value of the limit **must** be in bytes. #### Through the Rails console diff --git a/doc/administration/static_objects_external_storage.md b/doc/administration/static_objects_external_storage.md index 973f4304115..34c7c8947fc 100644 --- a/doc/administration/static_objects_external_storage.md +++ b/doc/administration/static_objects_external_storage.md @@ -1,3 +1,10 @@ +--- +stage: Create +group: Editor +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers" +type: reference +--- + # Static objects external storage > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/31025) in GitLab 12.3. @@ -52,10 +59,10 @@ sequenceDiagram ## Set up external storage -While this procedure uses [CloudFlare Workers](https://workers.cloudflare.com) for external storage, +While this procedure uses [Cloudflare Workers](https://workers.cloudflare.com) for external storage, other CDNs or Function as a Service (FaaS) systems should work using the same principles. -1. Choose a CloudFlare Worker domain if you haven't done so already. +1. Choose a Cloudflare Worker domain if you haven't done so already. 1. In the following script, set the following values for the first two constants: - `ORIGIN_HOSTNAME`: the hostname of your GitLab installation. diff --git a/doc/administration/terraform_state.md b/doc/administration/terraform_state.md index 8d3ddc4e306..76e54acce16 100644 --- a/doc/administration/terraform_state.md +++ b/doc/administration/terraform_state.md @@ -26,7 +26,6 @@ below. `/etc/gitlab/gitlab.rb` and add the following line: ```ruby - gitlab_rails['terraform_state_enabled'] = true gitlab_rails['terraform_state_storage_path'] = "/mnt/storage/terraform_state" ``` @@ -76,7 +75,6 @@ See [the available connection settings for different providers](object_storage.m the values you want: ```ruby - gitlab_rails['terraform_state_enabled'] = true gitlab_rails['terraform_state_object_store_enabled'] = true gitlab_rails['terraform_state_object_store_remote_directory'] = "terraform" gitlab_rails['terraform_state_object_store_connection'] = { diff --git a/doc/administration/troubleshooting/debug.md b/doc/administration/troubleshooting/debug.md index 55fd6462bc3..5daf34c1011 100644 --- a/doc/administration/troubleshooting/debug.md +++ b/doc/administration/troubleshooting/debug.md @@ -7,6 +7,10 @@ in production. Troubleshooting and debugging your GitLab instance often requires a [Rails console](https://guides.rubyonrails.org/command_line.html#rails-console). +See also: + +- [GitLab Rails Console Cheat Sheet](gitlab_rails_cheat_sheet.md). +- [Navigating GitLab via Rails console](navigating_gitlab_via_rails_console.md). **For Omnibus installations** @@ -73,7 +77,7 @@ sudo gitlab-rails runner "RAILS_COMMAND" # Example with a two-line Ruby script sudo gitlab-rails runner "user = User.first; puts user.username" -# Example with a ruby script file +# Example with a ruby script file (make sure to use the full path) sudo gitlab-rails runner /path/to/script.rb ``` @@ -85,7 +89,7 @@ sudo -u git -H bundle exec rails runner -e production "RAILS_COMMAND" # Example with a two-line Ruby script sudo -u git -H bundle exec rails runner -e production "user = User.first; puts user.username" -# Example with a ruby script file +# Example with a ruby script file (make sure to use the full path) sudo -u git -H bundle exec rails runner -e production /path/to/script.rb ``` diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index 8e9749fb239..22d699b424b 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -167,7 +167,7 @@ user = User.find_by_username('root') # Find the project, update the xxx-changeme values from above project = Project.find_by_full_path('group-changeme/project-changeme') -# Delete the project +# Immediately delete the project ::Projects::DestroyService.new(project, user, {}).execute ``` @@ -248,7 +248,8 @@ end A Projects Wiki can be recreated by -**Note:** This is a destructive operation, the Wiki will be empty +NOTE: **Note:** +This is a destructive operation, the Wiki will be empty. ```ruby p = Project.find_by_full_path('/') ### enter your projects path @@ -689,10 +690,10 @@ end As a GitLab administrator, you may need to reduce disk space consumption. A common culprit is Docker Registry images that are no longer in use. To find the storage broken down by each project, run the following in the -GitLab Rails console: +[GitLab Rails console](../troubleshooting/navigating_gitlab_via_rails_console.md): ```ruby -projects_and_size = [] +projects_and_size = [["project_id", "creator_id", "registry_size_bytes", "project path"]] # You need to specify the projects that you want to look through. You can get these in any manner. projects = Project.last(100) @@ -707,17 +708,21 @@ projects.each do |p| end if project_total_size > 0 - projects_and_size << [p.full_path,project_total_size] + projects_and_size << [p.project_id, p.creator.id, project_total_size, p.full_path] end end # projects_and_size is filled out now # maybe print it as comma separated output? projects_and_size.each do |ps| - puts "%s,%s" % ps + puts "%s,%s,%s,%s" % ps end ``` +### Run the Cleanup policy now + +Find this content in the [Container Registry troubleshooting docs](../packages/container_registry.md#run-the-cleanup-policy-now). + ## Sidekiq This content has been moved to the [Troubleshooting Sidekiq docs](./sidekiq.md). @@ -831,19 +836,19 @@ Geo::JobArtifactRegistry.synced.missing_on_primary.pluck(:artifact_id) #### Get the number of verification failed repositories ```ruby -Geo::ProjectRegistryFinder.new.count_verification_failed_repositories +Geo::ProjectRegistry.verification_failed('repository').count ``` #### Find the verification failed repositories ```ruby -Geo::ProjectRegistry.verification_failed_repos +Geo::ProjectRegistry.verification_failed('repository') ``` ### Find repositories that failed to sync ```ruby -Geo::ProjectRegistryFinder.new.find_failed_project_registries('repository') +Geo::ProjectRegistry.sync_failed('repository') ``` ### Resync repositories diff --git a/doc/administration/troubleshooting/group_saml_scim.md b/doc/administration/troubleshooting/group_saml_scim.md index e2ce72d5a16..7492688ded0 100644 --- a/doc/administration/troubleshooting/group_saml_scim.md +++ b/doc/administration/troubleshooting/group_saml_scim.md @@ -2,7 +2,7 @@ type: reference --- -# Group SAML and SCIM troubleshooting **(SILVER ONLY)** +# Troubleshooting Group SAML and SCIM **(SILVER ONLY)** These are notes and screenshots regarding Group SAML and SCIM that the GitLab Support Team sometimes uses while troubleshooting, but which do not fit into the official documentation. GitLab is making this public, so that anyone can make use of the Support team’s collected knowledge. diff --git a/doc/administration/troubleshooting/img/network_monitor_xid.png b/doc/administration/troubleshooting/img/network_monitor_xid.png new file mode 100644 index 00000000000..5392f77327f Binary files /dev/null and b/doc/administration/troubleshooting/img/network_monitor_xid.png differ diff --git a/doc/administration/troubleshooting/postgresql.md b/doc/administration/troubleshooting/postgresql.md index 36c1f20818a..b7e33e4501d 100644 --- a/doc/administration/troubleshooting/postgresql.md +++ b/doc/administration/troubleshooting/postgresql.md @@ -57,7 +57,7 @@ This section is for links to information elsewhere in the GitLab documentation. - [GitLab database requirements](../../install/requirements.md#database) including - Support for MySQL was removed in GitLab 12.1; [migrate to PostgreSQL](../../update/mysql_to_postgresql.md) - required extension `pg_trgm` - - required extension `postgres_fdw` for Geo + - required extension `btree_gist` - Errors like this in the `production/sidekiq` log; see: [Set default_transaction_isolation into read committed](https://docs.gitlab.com/omnibus/settings/database.html#set-default_transaction_isolation-into-read-committed): @@ -84,7 +84,7 @@ This section is for links to information elsewhere in the GitLab documentation. PANIC: could not write to file ‘pg_xlog/xlogtemp.123’: No space left on device ``` -- [Checking Geo configuration](../geo/replication/troubleshooting.md#checking-configuration) including +- [Checking Geo configuration](../geo/replication/troubleshooting.md) including - reconfiguring hosts/ports - checking and fixing user/password mappings diff --git a/doc/administration/troubleshooting/sidekiq.md b/doc/administration/troubleshooting/sidekiq.md index 566e2686735..9125ddf545f 100644 --- a/doc/administration/troubleshooting/sidekiq.md +++ b/doc/administration/troubleshooting/sidekiq.md @@ -7,16 +7,18 @@ may be filling up. Users will notice when this happens because new branches may not show up and merge requests may not be updated. The following are some troubleshooting steps that will help you diagnose the bottleneck. -> **Note:** GitLab administrators/users should consider working through these -> debug steps with GitLab Support so the backtraces can be analyzed by our team. -> It may reveal a bug or necessary improvement in GitLab. -> -> **Note:** In any of the backtraces, be wary of suspecting cases where every -> thread appears to be waiting in the database, Redis, or waiting to acquire -> a mutex. This **may** mean there's contention in the database, for example, -> but look for one thread that is different than the rest. This other thread -> may be using all available CPU, or have a Ruby Global Interpreter Lock, -> preventing other threads from continuing. +NOTE **Note:** +GitLab administrators/users should consider working through these +debug steps with GitLab Support so the backtraces can be analyzed by our team. +It may reveal a bug or necessary improvement in GitLab. + +NOTE: **Note:** +In any of the backtraces, be wary of suspecting cases where every +thread appears to be waiting in the database, Redis, or waiting to acquire +a mutex. This **may** mean there's contention in the database, for example, +but look for one thread that is different than the rest. This other thread +may be using all available CPU, or have a Ruby Global Interpreter Lock, +preventing other threads from continuing. ## Log arguments to Sidekiq jobs diff --git a/doc/administration/troubleshooting/tracing_correlation_id.md b/doc/administration/troubleshooting/tracing_correlation_id.md new file mode 100644 index 00000000000..31f537beae5 --- /dev/null +++ b/doc/administration/troubleshooting/tracing_correlation_id.md @@ -0,0 +1,126 @@ +--- +type: reference +--- + +# Finding relevant log entries with a correlation ID + +Since GitLab 11.6, a unique request tracking ID, known as the "correlation ID" has been +logged by the GitLab instance for most requests. Each individual request to GitLab gets +its own correlation ID, which then gets logged in each GitLab component's logs for that +request. This makes it easier to trace behavior in a +distributed system. Without this ID it can be difficult or +impossible to match correlating log entries. + +## Identify the correlation ID for a request + +The correlation ID is logged in structured logs under the key `correlation_id` +and in all response headers GitLab sends under the header `x-request-id`. +You can find your correlation ID by searching in either place. + +### Getting the correlation ID in your browser + +You can use your browser's developer tools to monitor and inspect network +activity with the site that you're visiting. See the links below for network monitoring +documenation for some popular browsers. + +- [Network Monitor - Firefox Developer Tools](https://developer.mozilla.org/en-US/docs/Tools/Network_Monitor) +- [Inspect Network Activity In Chrome DevTools](https://developers.google.com/web/tools/chrome-devtools/network/) +- [Safari Web Development Tools](https://developer.apple.com/safari/tools/) +- [Microsoft Edge Network panel](https://docs.microsoft.com/en-us/microsoft-edge/devtools-guide/network#request-details) + +To locate a relevant request and view its correlation ID: + +1. Enable persistent logging in your network monitor. Some actions in GitLab will redirect you quickly after you submit a form, so this will help capture all relevant activity. +1. To help isolate the requests you are looking for, you can filter for `document` requests. +1. Click the request of interest to view further detail. +1. Go to the **Headers** section and look for **Response Headers**. There you should find an `x-request-id` header with a +value that was randomly generated by GitLab for the request. + +See the following example: + +![Firefox's network monitor showing an request id header](img/network_monitor_xid.png) + +### Getting the correlation ID from your logs + +Another approach to finding the correct correlation ID is to search or watch +your logs and find the `correlation_id` value for the log entry that you're +watching for. + +For example, let's say that you want learn what's happening or breaking when +you reproduce an action in GitLab. You could tail the GitLab logs, filtering +to requests by your user, and then watch the requests until you see what you're +interested in. + +### Getting the correlation ID from curl + +If you're using `curl` then you can use the verbose option to show request and response headers, as well as other debug info. + +```shell +➜ ~ curl --verbose https://gitlab.example.com/api/v4/projects +# look for a line that looks like this +< x-request-id: 4rAMkV3gof4 +``` + +#### Using jq + +This example uses [jq](https://stedolan.github.io/jq/) to filter results and +display values we most likely care about. + +```shell +sudo gitlab-ctl tail gitlab-rails/production_json.log | jq 'select(.username == "bob") | "User: \(.username), \(.method) \(.path), \(.controller)#\(.action), ID: \(.correlation_id)"' +``` + +```plaintext +"User: bob, GET /root/linux, ProjectsController#show, ID: U7k7fh6NpW3" +"User: bob, GET /root/linux/commits/master/signatures, Projects::CommitsController#signatures, ID: XPIHpctzEg1" +"User: bob, GET /root/linux/blob/master/README, Projects::BlobController#show, ID: LOt9hgi1TV4" +``` + +#### Using grep + +This example uses only `grep` and `tr`, which are more likely to be installed than `jq`. + +```shell +sudo gitlab-ctl tail gitlab-rails/production_json.log | grep '"username":"bob"' | tr ',' '\n' | egrep 'method|path|correlation_id' +``` + +```plaintext +{"method":"GET" +"path":"/root/linux" +"username":"bob" +"correlation_id":"U7k7fh6NpW3"} +{"method":"GET" +"path":"/root/linux/commits/master/signatures" +"username":"bob" +"correlation_id":"XPIHpctzEg1"} +{"method":"GET" +"path":"/root/linux/blob/master/README" +"username":"bob" +"correlation_id":"LOt9hgi1TV4"} +``` + +## Searching your logs for the correlation ID + +Once you have the correlation ID you can start searching for relevant log +entries. You can filter the lines by the correlation ID itself. +Combining a `find` and `grep` should be sufficient to find the entries you are looking for. + +```shell +# find -type f -mtime -0 exec grep '' '{}' '+' +find /var/log/gitlab -type f -mtime 0 -exec grep 'LOt9hgi1TV4' '{}' '+' +``` + +```plaintext +/var/log/gitlab/gitlab-workhorse/current:{"correlation_id":"LOt9hgi1TV4","duration_ms":2478,"host":"gitlab.domain.tld","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"https://gitlab.domain.tld/root/linux","remote_addr":"68.0.116.160:0","remote_ip":"[filtered]","status":200,"system":"http","time":"2019-09-17T22:17:19Z","uri":"/root/linux/blob/master/README?format=json\u0026viewer=rich","user_agent":"Mozilla/5.0 (Mac) Gecko Firefox/69.0","written_bytes":1743} +/var/log/gitlab/gitaly/current:{"correlation_id":"LOt9hgi1TV4","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.method":"FindCommits","grpc.request.deadline":"2019-09-17T22:17:47Z","grpc.request.fullMethod":"/gitaly.CommitService/FindCommits","grpc.request.glProjectPath":"root/linux","grpc.request.glRepository":"project-1","grpc.request.repoPath":"@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"@hashed","grpc.service":"gitaly.CommitService","grpc.start_time":"2019-09-17T22:17:17Z","grpc.time_ms":2319.161,"level":"info","msg":"finished streaming call with code OK","peer.address":"@","span.kind":"server","system":"grpc","time":"2019-09-17T22:17:19Z"} +/var/log/gitlab/gitlab-rails/production_json.log:{"method":"GET","path":"/root/linux/blob/master/README","format":"json","controller":"Projects::BlobController","action":"show","status":200,"duration":2448.77,"view":0.49,"db":21.63,"time":"2019-09-17T22:17:19.800Z","params":[{"key":"viewer","value":"rich"},{"key":"namespace_id","value":"root"},{"key":"project_id","value":"linux"},{"key":"id","value":"master/README"}],"remote_ip":"[filtered]","user_id":2,"username":"bob","ua":"Mozilla/5.0 (Mac) Gecko Firefox/69.0","queue_duration":3.38,"gitaly_calls":1,"gitaly_duration":0.77,"rugged_calls":4,"rugged_duration_ms":28.74,"correlation_id":"LOt9hgi1TV4"} +``` + +### Searching in distributed architectures + +If you have done some horizontal scaling in your GitLab infrastructure, then +you will need to search across _all_ of your GitLab nodes. You can do this with +some sort of log aggregation software like Loki, ELK, Splunk, or others. + +You can use a tool like Ansible or PSSH (parellel SSH) that can execute identical commands across your servers in +parallel, or craft your own solution. diff --git a/doc/administration/wikis/index.md b/doc/administration/wikis/index.md new file mode 100644 index 00000000000..077b4f064dc --- /dev/null +++ b/doc/administration/wikis/index.md @@ -0,0 +1,75 @@ +--- +type: reference, howto +stage: Create +group: Knowledge +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + +# Wiki settings **(CORE ONLY)** + +Adjust the wiki settings of your GitLab instance. + +## Wiki page content size limit + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/31176) in GitLab 13.2. + +You can set a maximum content size limit for wiki pages. This limit can prevent +abuse of the feature. The default value is **52428800 Bytes** (50 MB). + +### How does it work? + +The content size limit will be applied when a wiki page is created or updated +through the GitLab UI or API. Local changes pushed via Git will not be validated. + +In order not to break any existing wiki pages, the limit doesn't have any +effect on them until a wiki page is edited again and the content changes. + +### Wiki page content size limit configuration + +This setting is not available through the [Admin Area settings](../../user/admin_area/settings/index.md). +In order to configure this setting, use either the Rails console +or the [Application settings API](../../api/settings.md). + +NOTE: **Note:** +The value of the limit **must** be in bytes. The minimum value is 1024 bytes. + +#### Through the Rails console + +The steps to configure this setting through the Rails console are: + +1. Start the Rails console: + + ```shell + # For Omnibus installations + sudo gitlab-rails console + + # For installations from source + sudo -u git -H bundle exec rails console -e production + ``` + +1. Update the wiki page maximum content size: + + ```ruby + ApplicationSetting.first.update!(wiki_page_max_content_bytes: 50.megabytes) + ``` + +To retrieve the current value, start the Rails console and run: + + ```ruby + Gitlab::CurrentSettings.wiki_page_max_content_bytes + ``` + +#### Through the API + +The process to set the wiki page size limit through the Application Settings API is +exactly the same as you would do to [update any other setting](../../api/settings.md#change-application-settings). + +```shell +curl --request PUT --header "PRIVATE-TOKEN: " https://gitlab.example.com/api/v4/application/settings?wiki_page_max_content_bytes=52428800 +``` + +You can also use the API to [retrieve the current value](../../api/settings.md#get-current-application-settings). + +```shell +curl --header "PRIVATE-TOKEN: " https://gitlab.example.com/api/v4/application/settings +``` -- cgit v1.2.1