diff options
Diffstat (limited to 'doc/administration/reference_architectures')
7 files changed, 488 insertions, 1159 deletions
diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index 0fd597e6a2d..e731085b0ce 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -12,6 +12,7 @@ full list of reference architectures, see > - **Supported users (approximate):** 10,000 > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) +> - **Cloud Native Hybrid:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) > - **Test requests per second (RPS) rates:** API: 200 RPS, Web: 20 RPS, Git (Pull): 20 RPS, Git (Push): 4 RPS > - **[Latest 10k weekly performance testing results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/10k)** @@ -22,18 +23,16 @@ full list of reference architectures, see | PostgreSQL<sup>1</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | `D8s v3` | | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Redis - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis - Queues / Shared State<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis Sentinel - Cache<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | -| Redis Sentinel - Queues / Shared State<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | -| Gitaly | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | `D16s v3` | -| Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Gitaly<sup>5</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | `D16s v3` | +| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | GitLab Rails | 3 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` | | Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | | Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | n/a | -| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| NFS server (non-Gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> @@ -41,6 +40,7 @@ full list of reference architectures, see 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. <!-- markdownlint-enable MD029 --> NOTE: @@ -82,11 +82,6 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 - collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347 - collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347 - - redis_persistent <.[#FF6347]- redis_persistent_sentinel - redis_cache <.[#FF6347]- redis_cache_sentinel } cloud "**Object Storage**" as object_storage #white @@ -137,8 +132,7 @@ our [Sysbench](https://github.com/akopytov/sysbench)-based Due to better performance and availability, for data objects (such as LFS, uploads, or artifacts), using an [object storage service](#configure-the-object-storage) -is recommended instead of using NFS. Using an object storage service also -doesn't require you to provision and maintain a node. +is recommended. It's also worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and that to achieve full High Availability a third-party PostgreSQL database solution will be required. @@ -169,10 +163,8 @@ To set up GitLab and its components to accommodate up to 10,000 users: used for shared data objects. 1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, more advanced code search across your entire GitLab instance. -1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended) - to have shared disk storage service as an alternative to Gitaly or object - storage. You can skip this step if you're not using GitLab Pages (which - requires NFS). +1. [Configure NFS](#configure-nfs) + to have shared disk storage service for certain GitLab operations (non Gitaly or Object Storage). The servers start on the same 10.6.0.0/24 private network range, and can connect to each other freely on these addresses. @@ -193,15 +185,9 @@ The following list includes descriptions of each server and its assigned IP: - `10.6.0.51`: Redis - Cache Primary - `10.6.0.52`: Redis - Cache Replica 1 - `10.6.0.53`: Redis - Cache Replica 2 -- `10.6.0.71`: Sentinel - Cache 1 -- `10.6.0.72`: Sentinel - Cache 2 -- `10.6.0.73`: Sentinel - Cache 3 -- `10.6.0.61`: Redis - Queues Primary -- `10.6.0.62`: Redis - Queues Replica 1 -- `10.6.0.63`: Redis - Queues Replica 2 -- `10.6.0.81`: Sentinel - Queues 1 -- `10.6.0.82`: Sentinel - Queues 2 -- `10.6.0.83`: Sentinel - Queues 3 +- `10.6.0.61`: Redis - Persistent Primary +- `10.6.0.62`: Redis - Persistent Replica 1 +- `10.6.0.63`: Redis - Persistent Replica 2 - `10.6.0.91`: Gitaly 1 - `10.6.0.92`: Gitaly 2 - `10.6.0.93`: Gitaly 3 @@ -792,15 +778,9 @@ to be used with GitLab. The following IPs will be used as an example: - `10.6.0.51`: Redis - Cache Primary - `10.6.0.52`: Redis - Cache Replica 1 - `10.6.0.53`: Redis - Cache Replica 2 -- `10.6.0.71`: Sentinel - Cache 1 -- `10.6.0.72`: Sentinel - Cache 2 -- `10.6.0.73`: Sentinel - Cache 3 -- `10.6.0.61`: Redis - Queues Primary -- `10.6.0.62`: Redis - Queues Replica 1 -- `10.6.0.63`: Redis - Queues Replica 2 -- `10.6.0.81`: Sentinel - Queues 1 -- `10.6.0.82`: Sentinel - Queues 2 -- `10.6.0.83`: Sentinel - Queues 3 +- `10.6.0.61`: Redis - Persistent Primary +- `10.6.0.62`: Redis - Persistent Replica 1 +- `10.6.0.63`: Redis - Persistent Replica 2 ### Providing your own Redis instance @@ -812,7 +792,7 @@ optional count argument to SPOP, which is required for [Merge Trains](../../ci/p Note the Redis node's IP address or hostname, port, and password (if required). These will be necessary later when configuring the [GitLab application servers](#configure-gitlab-rails). -### Configure the Redis and Sentinel Cache cluster +### Configure the Redis Cache cluster This is the section where we install and set up the new Redis Cache instances. @@ -830,8 +810,12 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' and enable Consul agent - roles(['redis_master_role', 'consul_role']) + # Specify server roles as 'redis_master_role' with sentinel and the Consul agent + roles ['redis_sentinel_role', 'redis_master_role', 'consul_role'] + + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -843,8 +827,19 @@ a node and change its status from primary to replica (and vice versa). # machines to connect to it. redis['port'] = 6379 - # Set up password authentication for Redis (use the same password in all nodes). + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + + # Set up password authentication for Redis and replicas (use the same password in all nodes). redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + + ## Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-cache' + + ## The IP of this primary Redis node. + redis['master_ip'] = '10.6.0.51' # Set the Redis Cache instance as an LRU # 90% of available RAM in MB @@ -878,10 +873,6 @@ a node and change its status from primary to replica (and vice versa). 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - #### Configure the replica Redis Cache nodes 1. SSH in to the **replica** Redis server. @@ -889,11 +880,15 @@ You can specify multiple roles, like sentinel and Redis, as: package of your choice. Be sure to both follow _only_ installation steps 1 and 2 on the page, and to select the correct Omnibus GitLab package, with the same version and type (Community or Enterprise editions) as your current install. -1. Edit `/etc/gitlab/gitlab.rb` and add the contents: +1. Edit `/etc/gitlab/gitlab.rb` and add same contents as the primary node in the previous section replacing `redis_master_node` with `redis_replica_node`: ```ruby - # Specify server role as 'redis_replica_role' and enable Consul agent - roles(['redis_replica_role', 'consul_role']) + # Specify server roles as 'redis_sentinel_role' and 'redis_replica_role' + roles ['redis_sentinel_role', 'redis_replica_role', 'consul_role'] + + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -905,15 +900,19 @@ You can specify multiple roles, like sentinel and Redis, as: # machines to connect to it. redis['port'] = 6379 - # The same password for Redis authentication you set up for the primary node. + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + + # Set up password authentication for Redis and replicas (use the same password in all nodes). redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' - # The IP of the primary Redis node. - redis['master_ip'] = '10.6.0.51' + ## Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-cache' - # Port of primary Redis server, uncomment to change to non default. Defaults - # to `6379`. - #redis['master_port'] = 6379 + ## The IP of the primary Redis node. + redis['master_ip'] = '10.6.0.51' # Set the Redis Cache instance as an LRU # 90% of available RAM in MB @@ -949,15 +948,6 @@ You can specify multiple roles, like sentinel and Redis, as: 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - -These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after -a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-cache-nodes), and even after a -`gitlab-ctl reconfigure`, they will get their configuration restored by -the same Sentinels. - Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed. @@ -967,133 +957,15 @@ are supported and can be added if needed. </a> </div> -#### Configure the Sentinel Cache nodes - -Now that the Redis servers are all set up, let's configure the Sentinel -servers. The following IPs will be used as an example: - -- `10.6.0.71`: Sentinel - Cache 1 -- `10.6.0.72`: Sentinel - Cache 2 -- `10.6.0.73`: Sentinel - Cache 3 - -NOTE: -If you're using an external Redis Sentinel instance, be sure to exclude the -`requirepass` parameter from the Sentinel configuration. This parameter causes -clients to report `NOAUTH Authentication required.`. -[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279). - -To configure the Sentinel Cache server: - -1. SSH in to the server that will host Consul/Sentinel. -1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab - package of your choice. Be sure to both follow _only_ installation steps 1 and 2 - on the page, and to select the correct Omnibus GitLab package, with the same version - and type (Community or Enterprise editions) as your current install. -1. Edit `/etc/gitlab/gitlab.rb` and add the contents: - - ```ruby - roles(['redis_sentinel_role', 'consul_role']) - - ## Must be the same in every sentinel node - redis['master_name'] = 'gitlab-redis-cache' - - ## The same password for Redis authentication you set up for the primary node. - redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' - - ## The IP of the primary Redis node. - redis['master_ip'] = '10.6.0.51' - - ## Define a port so Redis can listen for TCP requests which will allow other - ## machines to connect to it. - redis['port'] = 6379 - - ## Port of primary Redis server, uncomment to change to non default. Defaults - ## to `6379`. - #redis['master_port'] = 6379 - - ## Configure Sentinel's IP - sentinel['bind'] = '10.6.0.71' - - ## Port that Sentinel listens on, uncomment to change to non default. Defaults - ## to `26379`. - #sentinel['port'] = 26379 - - ## Quorum must reflect the amount of voting sentinels it take to start a failover. - ## Value must NOT be greater then the amount of sentinels. - ## - ## The quorum can be used to tune Sentinel in two ways: - ## 1. If a the quorum is set to a value smaller than the majority of Sentinels - ## we deploy, we are basically making Sentinel more sensible to primary failures, - ## triggering a failover as soon as even just a minority of Sentinels is no longer - ## able to talk with the primary. - ## 1. If a quorum is set to a value greater than the majority of Sentinels, we are - ## making Sentinel able to failover only when there are a very large number (larger - ## than majority) of well connected Sentinels which agree about the primary being down.s - sentinel['quorum'] = 2 - - ## Consider unresponsive server down after x amount of ms. - #sentinel['down_after_milliseconds'] = 10000 - - ## Specifies the failover timeout in milliseconds. It is used in many ways: - ## - ## - The time needed to re-start a failover after a previous failover was - ## already tried against the same primary by a given Sentinel, is two - ## times the failover timeout. - ## - ## - The time needed for a replica replicating to a wrong primary according - ## to a Sentinel current configuration, to be forced to replicate - ## with the right primary, is exactly the failover timeout (counting since - ## the moment a Sentinel detected the misconfiguration). - ## - ## - The time needed to cancel a failover that is already in progress but - ## did not produced any configuration change (REPLICAOF NO ONE yet not - ## acknowledged by the promoted replica). - ## - ## - The maximum time a failover in progress waits for all the replica to be - ## reconfigured as replicas of the new primary. However even after this time - ## the replicas will be reconfigured by the Sentinels anyway, but not with - ## the exact parallel-syncs progression as specified. - #sentinel['failover_timeout'] = 60000 - - ## Enable service discovery for Prometheus - consul['monitoring_service_discovery'] = true - - ## The IPs of the Consul server nodes - ## You can also use FQDNs and intermix them with IPs - consul['configuration'] = { - retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), - } - - # Set the network addresses that the exporters will listen on - node_exporter['listen_address'] = '0.0.0.0:9100' - redis_exporter['listen_address'] = '0.0.0.0:9121' - - # Prevent database migrations from running on upgrade automatically - gitlab_rails['auto_migrate'] = false - ``` - -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace - the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. - -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. Go through the steps again for all the other Consul/Sentinel nodes, and - make sure you set up the correct IPs. - -<div align="right"> - <a type="button" class="btn btn-default" href="#setup-components"> - Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> - </a> -</div> - -### Configure the Redis and Sentinel Queues cluster +### Configure the Redis Persistent cluster -This is the section where we install and set up the new Redis Queues instances. +This is the section where we install and set up the new Redis Persistent instances. Both the primary and replica Redis nodes need the same password defined in `redis['password']`. At any time during a failover, the Sentinels can reconfigure a node and change its status from primary to replica (and vice versa). -#### Configure the primary Redis Queues node +#### Configure the primary Redis Persistent node 1. SSH in to the **Primary** Redis server. 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab @@ -1103,8 +975,12 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' and enable Consul agent - roles(['redis_master_role', 'consul_role']) + # Specify server roles as 'redis_master_role' with Sentinel and the Consul agent + roles ['redis_sentinel_role', 'redis_master_role', 'consul_role'] + + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -1116,8 +992,19 @@ a node and change its status from primary to replica (and vice versa). # machines to connect to it. redis['port'] = 6379 - # Set up password authentication for Redis (use the same password in all nodes). - redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + + # Set up password authentication for Redis and replicas (use the same password in all nodes). + redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + + ## Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-persistent' + + ## The IP of this primary Redis node. + redis['master_ip'] = '10.6.0.61' ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -1141,13 +1028,9 @@ a node and change its status from primary to replica (and vice versa). 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - -#### Configure the replica Redis Queues nodes +#### Configure the replica Redis Persistent nodes -1. SSH in to the **replica** Redis Queue server. +1. SSH in to the **replica** Redis Persistent server. 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to both follow _only_ installation steps 1 and 2 on the page, and to select the correct Omnibus GitLab package, with the same version @@ -1155,8 +1038,12 @@ You can specify multiple roles, like sentinel and Redis, as: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_replica_role' and enable Consul agent - roles(['redis_replica_role', 'consul_role']) + # Specify server roles as 'redis_sentinel_role' and 'redis_replica_role' + roles ['redis_sentinel_role', 'redis_replica_role', 'consul_role'] + + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -1168,16 +1055,20 @@ You can specify multiple roles, like sentinel and Redis, as: # machines to connect to it. redis['port'] = 6379 + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + # The same password for Redis authentication you set up for the primary node. redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + + ## Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-persistent' # The IP of the primary Redis node. redis['master_ip'] = '10.6.0.61' - # Port of primary Redis server, uncomment to change to non default. Defaults - # to `6379`. - #redis['master_port'] = 6379 - ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -1202,15 +1093,6 @@ You can specify multiple roles, like sentinel and Redis, as: 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - -These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after -a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-queues-nodes), and even after a -`gitlab-ctl reconfigure`, they will get their configuration restored by -the same Sentinels. - Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed. @@ -1220,129 +1102,15 @@ are supported and can be added if needed. </a> </div> -#### Configure the Sentinel Queues nodes - -Now that the Redis servers are all set up, let's configure the Sentinel -servers. The following IPs will be used as an example: - -- `10.6.0.81`: Sentinel - Queues 1 -- `10.6.0.82`: Sentinel - Queues 2 -- `10.6.0.83`: Sentinel - Queues 3 - -NOTE: -If you're using an external Redis Sentinel instance, be sure to exclude the -`requirepass` parameter from the Sentinel configuration. This parameter causes -clients to report `NOAUTH Authentication required.`. -[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279). - -To configure the Sentinel Queues server: - -1. SSH in to the server that will host Sentinel. -1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab - package of your choice. Be sure to both follow _only_ installation steps 1 and 2 - on the page, and to select the correct Omnibus GitLab package, with the same version - and type (Community or Enterprise editions) as your current install. -1. Edit `/etc/gitlab/gitlab.rb` and add the contents: - - ```ruby - roles(['redis_sentinel_role', 'consul_role']) - - ## Must be the same in every sentinel node - redis['master_name'] = 'gitlab-redis-persistent' - - ## The same password for Redis authentication you set up for the primary node. - redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' - - ## The IP of the primary Redis node. - redis['master_ip'] = '10.6.0.61' - - ## Define a port so Redis can listen for TCP requests which will allow other - ## machines to connect to it. - redis['port'] = 6379 - - ## Port of primary Redis server, uncomment to change to non default. Defaults - ## to `6379`. - #redis['master_port'] = 6379 - - ## Configure Sentinel's IP - sentinel['bind'] = '10.6.0.81' - - ## Port that Sentinel listens on, uncomment to change to non default. Defaults - ## to `26379`. - #sentinel['port'] = 26379 - - ## Quorum must reflect the amount of voting sentinels it take to start a failover. - ## Value must NOT be greater then the amount of sentinels. - ## - ## The quorum can be used to tune Sentinel in two ways: - ## 1. If a the quorum is set to a value smaller than the majority of Sentinels - ## we deploy, we are basically making Sentinel more sensible to primary failures, - ## triggering a failover as soon as even just a minority of Sentinels is no longer - ## able to talk with the primary. - ## 1. If a quorum is set to a value greater than the majority of Sentinels, we are - ## making Sentinel able to failover only when there are a very large number (larger - ## than majority) of well connected Sentinels which agree about the primary being down.s - sentinel['quorum'] = 2 - - ## Consider unresponsive server down after x amount of ms. - #sentinel['down_after_milliseconds'] = 10000 - - ## Specifies the failover timeout in milliseconds. It is used in many ways: - ## - ## - The time needed to re-start a failover after a previous failover was - ## already tried against the same primary by a given Sentinel, is two - ## times the failover timeout. - ## - ## - The time needed for a replica replicating to a wrong primary according - ## to a Sentinel current configuration, to be forced to replicate - ## with the right primary, is exactly the failover timeout (counting since - ## the moment a Sentinel detected the misconfiguration). - ## - ## - The time needed to cancel a failover that is already in progress but - ## did not produced any configuration change (REPLICAOF NO ONE yet not - ## acknowledged by the promoted replica). - ## - ## - The maximum time a failover in progress waits for all the replica to be - ## reconfigured as replicas of the new primary. However even after this time - ## the replicas will be reconfigured by the Sentinels anyway, but not with - ## the exact parallel-syncs progression as specified. - #sentinel['failover_timeout'] = 60000 - - ## Enable service discovery for Prometheus - consul['monitoring_service_discovery'] = true - - ## The IPs of the Consul server nodes - ## You can also use FQDNs and intermix them with IPs - consul['configuration'] = { - retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), - } - - # Set the network addresses that the exporters will listen on - node_exporter['listen_address'] = '0.0.0.0:9100' - redis_exporter['listen_address'] = '0.0.0.0:9121' - - # Prevent database migrations from running on upgrade automatically - gitlab_rails['auto_migrate'] = false - ``` - -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace - the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. - -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. Go through the steps again for all the other Sentinel nodes, and - make sure you set up the correct IPs. - -<div align="right"> - <a type="button" class="btn btn-default" href="#setup-components"> - Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> - </a> -</div> - ## Configure Gitaly Cluster [Gitaly Cluster](../gitaly/praefect.md) is a GitLab provided and recommended fault tolerant solution for storing Git repositories. In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down. +NOTE: +Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). +For implementations with Gitaly Sharded, the same Gitaly specs should be used. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section. + The recommended cluster setup includes the following components: - 3 Gitaly nodes: Replicated storage of Git repositories. @@ -1509,7 +1277,7 @@ the details of each Gitaly node that makes up the cluster. Each storage is also and this name is used in several areas of the configuration. In this guide, the name of the storage will be `default`. Also, this guide is geared towards new installs, if upgrading an existing environment to use Gitaly Cluster, you may need to use a different name. -Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more info. +Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more information. The following IPs will be used as an example: @@ -1836,7 +1604,7 @@ To configure the Sidekiq nodes, on each one: 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to follow _only_ installation steps 1 and 2 on the page. -1. Open `/etc/gitlab/gitlab.rb` with your editor: +1. Create or edit `/etc/gitlab/gitlab.rb` and use the following configuration: ```ruby # Avoid running unnecessary services on the Sidekiq server @@ -1851,36 +1619,40 @@ To configure the Sidekiq nodes, on each one: gitlab_exporter['enable'] = false nginx['enable'] = false + # External URL + ## This should match the URL of the external load balancer + external_url 'https://gitlab.example.com' + # Redis ## Redis connection details ## First cluster that will host the cache gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' gitlab_rails['redis_cache_sentinels'] = [ - {host: '10.6.0.71', port: 26379}, - {host: '10.6.0.72', port: 26379}, - {host: '10.6.0.73', port: 26379}, + {host: '10.6.0.51', port: 26379}, + {host: '10.6.0.52', port: 26379}, + {host: '10.6.0.53', port: 26379}, ] - ## Second cluster that will host the queues, shared state, and actioncable + ## Second cluster that will host the persistent queues, shared state, and actioncable gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_queues_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_shared_state_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_actioncable_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] # Gitaly Cluster @@ -2032,30 +1804,30 @@ On each node perform the following: gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' gitlab_rails['redis_cache_sentinels'] = [ - {host: '10.6.0.71', port: 26379}, - {host: '10.6.0.72', port: 26379}, - {host: '10.6.0.73', port: 26379}, + {host: '10.6.0.51', port: 26379}, + {host: '10.6.0.52', port: 26379}, + {host: '10.6.0.53', port: 26379}, ] - ## Second cluster that will host the queues, shared state, and actionable + ## Second cluster that will host the persistent queues, shared state, and actionable gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_queues_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_shared_state_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_actioncable_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] # Set the network addresses that the exporters used for monitoring will listen on @@ -2127,7 +1899,7 @@ On each node perform the following: 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. If you're [using NFS](#configure-nfs-optional): +1. If you're [using NFS](#configure-nfs): 1. If necessary, install the NFS client utility packages using the following commands: @@ -2271,7 +2043,7 @@ To configure the Monitoring node: ## Configure the object storage GitLab supports using an object storage service for holding numerous types of data. -It's recommended over [NFS](#configure-nfs-optional) and in general it's better +It's recommended over [NFS](#configure-nfs) and in general it's better in larger setups as object storage is typically much more performant, reliable, and scalable. @@ -2295,6 +2067,9 @@ There are two ways of specifying object storage configuration in GitLab: Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. +GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared via NFS on any GitLab Rails and Sidekiq nodes. +In GitLab 13.6 and later, it's recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. + For configuring object storage in GitLab 13.1 and earlier, or for storage types not supported by consolidated configuration form, refer to the following guides based on what features you intend to use: @@ -2341,7 +2116,7 @@ cluster alongside your instance, read how to </a> </div> -## Configure NFS (optional) +## Configure NFS [Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly) are recommended over NFS wherever possible for improved performance. If you intend @@ -2355,7 +2130,7 @@ unavailable from GitLab 15.0. No further enhancements are planned for this featu Read: -- The [Gitaly and NFS deprecation notice](../gitaly/index.md#nfs-deprecation-notice). +- [Gitaly and NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation). - About the [correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). <div align="right"> @@ -2415,12 +2190,10 @@ services where applicable): | PostgreSQL<sup>1</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Redis - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Redis - Queues / Shared State<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Redis Sentinel - Cache<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | -| Redis Sentinel - Queues / Shared State<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | -| Gitaly | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | -| Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | +| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | +| Gitaly<sup>5</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | +| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Object storage<sup>4</sup> | n/a | n/a | n/a | @@ -2430,6 +2203,7 @@ services where applicable): 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. <!-- markdownlint-enable MD029 --> NOTE: @@ -2474,11 +2248,6 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 - collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347 - collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347 - - redis_persistent <.[#FF6347]- redis_persistent_sentinel - redis_cache <.[#FF6347]- redis_cache_sentinel } cloud "**Object Storage**" as object_storage #white diff --git a/doc/administration/reference_architectures/1k_users.md b/doc/administration/reference_architectures/1k_users.md index ea40e150e58..ae832c2226f 100644 --- a/doc/administration/reference_architectures/1k_users.md +++ b/doc/administration/reference_architectures/1k_users.md @@ -18,6 +18,8 @@ many organizations. > - **Supported users (approximate):** 1,000 > - **High Availability:** No. For a highly-available environment, you can > follow a modified [3K reference architecture](3k_users.md#supported-modifications-for-lower-user-counts-ha). +> - **Cloud Native Hybrid:** No. For a cloud native hybrid environment, you +> can follow a [modified hybrid reference architecture](#cloud-native-hybrid-reference-architecture-with-helm-charts). > - **Test requests per second (RPS) rates:** API: 20 RPS, Web: 2 RPS, Git (Pull): 2 RPS, Git (Push): 1 RPS > - **[Latest 1k weekly performance testing results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/1k)** @@ -58,3 +60,12 @@ Elasticsearch cluster design and requirements are dependent on your specific data. For recommended best practices about how to set up your Elasticsearch cluster alongside your instance, read how to [choose the optimal cluster configuration](../../integration/elasticsearch.md#guidance-on-choosing-optimal-cluster-configuration). + +## Cloud Native Hybrid reference architecture with Helm Charts + +Cloud Native Hybrid Reference Architecture is an alternative approach where select _stateless_ +components are deployed in Kubernetes via our official [Helm Charts](https://docs.gitlab.com/charts/), +and _stateful_ components are deployed in compute VMs with Omnibus. + +The [2k GitLab Cloud Native Hybrid](2k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) (non HA) and [3k GitLab Cloud Native Hybrid](3k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) (HA) reference architectures are the smallest we recommend in Kubernetes. +For environments that need to serve less users, you can lower the node specs. Depending on your user count, you can lower all suggested node specs as desired. However, it's recommended that you don't go lower than the [general requirements](../../install/requirements.md). diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index f500434d75b..267f81efd91 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -12,6 +12,7 @@ full list of reference architectures, see > - **Supported users (approximate):** 25,000 > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) +> - **Cloud Native Hybrid:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) > - **Test requests per second (RPS) rates:** API: 500 RPS, Web: 50 RPS, Git (Pull): 50 RPS, Git (Push): 10 RPS > - **[Latest 25k weekly performance testing results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/25k)** @@ -22,18 +23,16 @@ full list of reference architectures, see | PostgreSQL<sup>1</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | `D16s v3` | | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Internal load balancing node<sup>3</sup> | 1 | 4 vCPU, 3.6GB memory | `n1-highcpu-4` | `c5.large` | `F2s v2` | -| Redis - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis - Queues / Shared State<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis Sentinel - Cache<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | -| Redis Sentinel - Queues / Shared State<sup>2</sup>| 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | -| Gitaly | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | `D32s v3` | -| Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.large` | `D4s v3` | +| Gitaly<sup>5</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | `D32s v3` | +| Praefect<sup>5</sup> | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | GitLab Rails | 5 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` | | Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | | Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | n/a | -| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| NFS server (non-Gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> @@ -41,6 +40,7 @@ full list of reference architectures, see 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. <!-- markdownlint-enable MD029 --> NOTE: @@ -82,11 +82,6 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 - collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347 - collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347 - - redis_persistent <.[#FF6347]- redis_persistent_sentinel - redis_cache <.[#FF6347]- redis_cache_sentinel } cloud "**Object Storage**" as object_storage #white @@ -137,8 +132,7 @@ our [Sysbench](https://github.com/akopytov/sysbench)-based Due to better performance and availability, for data objects (such as LFS, uploads, or artifacts), using an [object storage service](#configure-the-object-storage) -is recommended instead of using NFS. Using an object storage service also -doesn't require you to provision and maintain a node. +is recommended. It's also worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and that to achieve full High Availability a third-party PostgreSQL database solution will be required. @@ -169,10 +163,9 @@ To set up GitLab and its components to accommodate up to 25,000 users: used for shared data objects. 1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, more advanced code search across your entire GitLab instance. -1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended) - to have shared disk storage service as an alternative to Gitaly or object - storage. You can skip this step if you're not using GitLab Pages (which - requires NFS). +1. [Configure NFS](#configure-nfs) + to have shared disk storage service for certain GitLab operations (non + Gitaly or Object Storage). The servers start on the same 10.6.0.0/24 private network range, and can connect to each other freely on these addresses. @@ -193,15 +186,9 @@ The following list includes descriptions of each server and its assigned IP: - `10.6.0.51`: Redis - Cache Primary - `10.6.0.52`: Redis - Cache Replica 1 - `10.6.0.53`: Redis - Cache Replica 2 -- `10.6.0.71`: Sentinel - Cache 1 -- `10.6.0.72`: Sentinel - Cache 2 -- `10.6.0.73`: Sentinel - Cache 3 -- `10.6.0.61`: Redis - Queues Primary -- `10.6.0.62`: Redis - Queues Replica 1 -- `10.6.0.63`: Redis - Queues Replica 2 -- `10.6.0.81`: Sentinel - Queues 1 -- `10.6.0.82`: Sentinel - Queues 2 -- `10.6.0.83`: Sentinel - Queues 3 +- `10.6.0.61`: Redis - Persistent Primary +- `10.6.0.62`: Redis - Persistent Replica 1 +- `10.6.0.63`: Redis - Persistent Replica 2 - `10.6.0.91`: Gitaly 1 - `10.6.0.92`: Gitaly 2 - `10.6.0.93`: Gitaly 3 @@ -794,15 +781,9 @@ to be used with GitLab. The following IPs will be used as an example: - `10.6.0.51`: Redis - Cache Primary - `10.6.0.52`: Redis - Cache Replica 1 - `10.6.0.53`: Redis - Cache Replica 2 -- `10.6.0.71`: Sentinel - Cache 1 -- `10.6.0.72`: Sentinel - Cache 2 -- `10.6.0.73`: Sentinel - Cache 3 -- `10.6.0.61`: Redis - Queues Primary -- `10.6.0.62`: Redis - Queues Replica 1 -- `10.6.0.63`: Redis - Queues Replica 2 -- `10.6.0.81`: Sentinel - Queues 1 -- `10.6.0.82`: Sentinel - Queues 2 -- `10.6.0.83`: Sentinel - Queues 3 +- `10.6.0.61`: Redis - Persistent Primary +- `10.6.0.62`: Redis - Persistent Replica 1 +- `10.6.0.63`: Redis - Persistent Replica 2 ### Providing your own Redis instance @@ -814,7 +795,7 @@ optional count argument to SPOP, which is required for [Merge Trains](../../ci/p Note the Redis node's IP address or hostname, port, and password (if required). These will be necessary later when configuring the [GitLab application servers](#configure-gitlab-rails). -### Configure the Redis and Sentinel Cache cluster +### Configure the Redis Cache cluster This is the section where we install and set up the new Redis Cache instances. @@ -832,10 +813,14 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' and enable Consul agent - roles(['redis_master_role', 'consul_role'] + # Specify server role as 'redis_sentinel_role' 'redis_master_role' + roles ['redis_sentinel_role', 'redis_master_role', 'consul_role'] - # IP address pointing to a local IP that the other machines can reach to. + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 + + # IP address pointing to a local IP that the other machines can reach. # You can also set bind to '0.0.0.0' which listen in all interfaces. # If you really need to bind to an external accessible IP, make # sure you add extra firewall rules to prevent unauthorized access. @@ -845,8 +830,18 @@ a node and change its status from primary to replica (and vice versa). # machines to connect to it. redis['port'] = 6379 + # Port of primary Redis server for Sentinel, uncomment to change to non default. + # Defaults to `6379`. + # redis['master_port'] = 6379 + # Set up password authentication for Redis (use the same password in all nodes). - redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + + # Must be the same in every Redis node. + redis['master_name'] = 'gitlab-redis-cache' + + # The IP of this primary Redis node. + redis['master_ip'] = '10.6.0.51' # Set the Redis Cache instance as an LRU # 90% of available RAM in MB @@ -880,10 +875,6 @@ a node and change its status from primary to replica (and vice versa). 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - #### Configure the replica Redis Cache nodes 1. SSH in to the **replica** Redis server. @@ -891,14 +882,18 @@ You can specify multiple roles, like sentinel and Redis, as: package of your choice. Be sure to both follow _only_ installation steps 1 and 2 on the page, and to select the correct Omnibus GitLab package, with the same version and type (Community or Enterprise editions) as your current install. -1. Edit `/etc/gitlab/gitlab.rb` and add the contents: +1. Edit `/etc/gitlab/gitlab.rb` and add the same contents as the primary node in the previous section replacing `redis_master_node` with `redis_replica_node`: ```ruby # Specify server role as 'redis_replica_role' and enable Consul agent - roles(['redis_replica_role', 'consul_role'] + roles ['redis_sentinel_role', 'redis_replica_role', 'consul_role'] - # IP address pointing to a local IP that the other machines can reach to. - # You can also set bind to '0.0.0.0' which listen in all interfaces. + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = `0.0.0.0` + sentinel['quorum'] = 2 + + # IP address pointing to a local IP that the other machines can reach. + # Set bind to '0.0.0.0' to listen on all interfaces. # If you really need to bind to an external accessible IP, make # sure you add extra firewall rules to prevent unauthorized access. redis['bind'] = '10.6.0.52' @@ -907,16 +902,19 @@ You can specify multiple roles, like sentinel and Redis, as: # machines to connect to it. redis['port'] = 6379 - # The same password for Redis authentication you set up for the primary node. - redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + + # Set up password authentication for Redis and replicas (use the same password in all nodes). + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + + # Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-cache' # The IP of the primary Redis node. redis['master_ip'] = '10.6.0.51' - # Port of primary Redis server, uncomment to change to non default. Defaults - # to `6379`. - #redis['master_port'] = 6379 - # Set the Redis Cache instance as an LRU # 90% of available RAM in MB redis['maxmemory'] = '13500mb' @@ -952,17 +950,7 @@ You can specify multiple roles, like sentinel and Redis, as: 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - -These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after -a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-cache-nodes), and even after a -`gitlab-ctl reconfigure`, they will get their configuration restored by -the same Sentinels. - -Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) -are supported and can be added if needed. + Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed. <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -970,134 +958,15 @@ are supported and can be added if needed. </a> </div> -#### Configure the Sentinel Cache nodes - -Now that the Redis servers are all set up, let's configure the Sentinel -servers. The following IPs will be used as an example: - -- `10.6.0.71`: Sentinel - Cache 1 -- `10.6.0.72`: Sentinel - Cache 2 -- `10.6.0.73`: Sentinel - Cache 3 - -NOTE: -If you're using an external Redis Sentinel instance, be sure to exclude the -`requirepass` parameter from the Sentinel configuration. This parameter causes -clients to report `NOAUTH Authentication required.`. -[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279). - -To configure the Sentinel Cache server: +### Configure the Redis Persistent cluster -1. SSH in to the server that will host Consul/Sentinel. -1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab - package of your choice. Be sure to both follow _only_ installation steps 1 and 2 - on the page, and to select the correct Omnibus GitLab package, with the same version - and type (Community or Enterprise editions) as your current install. -1. Edit `/etc/gitlab/gitlab.rb` and add the contents: - - ```ruby - roles(['redis_sentinel_role', 'consul_role']) - - ## Must be the same in every sentinel node - redis['master_name'] = 'gitlab-redis-cache' - - ## The same password for Redis authentication you set up for the primary node. - redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' - - ## The IP of the primary Redis node. - redis['master_ip'] = '10.6.0.51' - - ## Define a port so Redis can listen for TCP requests which will allow other - ## machines to connect to it. - redis['port'] = 6379 - - ## Port of primary Redis server, uncomment to change to non default. Defaults - ## to `6379`. - #redis['master_port'] = 6379 - - ## Configure Sentinel's IP - sentinel['bind'] = '10.6.0.71' - - ## Port that Sentinel listens on, uncomment to change to non default. Defaults - ## to `26379`. - #sentinel['port'] = 26379 - - ## Quorum must reflect the amount of voting sentinels it take to start a failover. - ## Value must NOT be greater then the amount of sentinels. - ## - ## The quorum can be used to tune Sentinel in two ways: - ## 1. If a the quorum is set to a value smaller than the majority of Sentinels - ## we deploy, we are basically making Sentinel more sensible to primary failures, - ## triggering a failover as soon as even just a minority of Sentinels is no longer - ## able to talk with the primary. - ## 1. If a quorum is set to a value greater than the majority of Sentinels, we are - ## making Sentinel able to failover only when there are a very large number (larger - ## than majority) of well connected Sentinels which agree about the primary being down.s - sentinel['quorum'] = 2 - - ## Consider unresponsive server down after x amount of ms. - #sentinel['down_after_milliseconds'] = 10000 - - ## Specifies the failover timeout in milliseconds. It is used in many ways: - ## - ## - The time needed to re-start a failover after a previous failover was - ## already tried against the same primary by a given Sentinel, is two - ## times the failover timeout. - ## - ## - The time needed for a replica replicating to a wrong primary according - ## to a Sentinel current configuration, to be forced to replicate - ## with the right primary, is exactly the failover timeout (counting since - ## the moment a Sentinel detected the misconfiguration). - ## - ## - The time needed to cancel a failover that is already in progress but - ## did not produced any configuration change (REPLICAOF NO ONE yet not - ## acknowledged by the promoted replica). - ## - ## - The maximum time a failover in progress waits for all the replica to be - ## reconfigured as replicas of the new primary. However even after this time - ## the replicas will be reconfigured by the Sentinels anyway, but not with - ## the exact parallel-syncs progression as specified. - #sentinel['failover_timeout'] = 60000 - - ## Enable service discovery for Prometheus - consul['monitoring_service_discovery'] = true - - ## The IPs of the Consul server nodes - ## You can also use FQDNs and intermix them with IPs - consul['configuration'] = { - retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), - } - - # Set the network addresses that the exporters will listen on - node_exporter['listen_address'] = '0.0.0.0:9100' - redis_exporter['listen_address'] = '0.0.0.0:9121' - - # Prevent database migrations from running on upgrade automatically - gitlab_rails['auto_migrate'] = false - ``` - -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace - the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. - -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -1. Go through the steps again for all the other Consul/Sentinel nodes, and - make sure you set up the correct IPs. - -<div align="right"> - <a type="button" class="btn btn-default" href="#setup-components"> - Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> - </a> -</div> - -### Configure the Redis and Sentinel Queues cluster - -This is the section where we install and set up the new Redis Queues instances. +This is the section where we install and set up the new Redis Persistent instances. Both the primary and replica Redis nodes need the same password defined in `redis['password']`. At any time during a failover, the Sentinels can reconfigure a node and change its status from primary to replica (and vice versa). -#### Configure the primary Redis Queues node +#### Configure the primary Redis Persistent node 1. SSH in to the **Primary** Redis server. 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab @@ -1107,8 +976,12 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' and enable Consul agent - roles(['redis_master_role', 'consul_role']) + # Specify server roles as 'redis_sentinel_role' and 'redis_master_role' + roles ['redis_sentinel_role', 'redis_master_role', 'consul_role'] + + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -1120,8 +993,19 @@ a node and change its status from primary to replica (and vice versa). # machines to connect to it. redis['port'] = 6379 - # Set up password authentication for Redis (use the same password in all nodes). - redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + + # Set up password authentication for Redis and replicas (use the same password in all nodes). + redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + + ## Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-persistent' + + ## The IP of this primary Redis node. + redis['master_ip'] = '10.6.0.61' ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -1145,13 +1029,9 @@ a node and change its status from primary to replica (and vice versa). 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - -#### Configure the replica Redis Queues nodes +#### Configure the replica Redis Persistent nodes -1. SSH in to the **replica** Redis Queue server. +1. SSH in to the **replica** Redis Persistent server. 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to both follow _only_ installation steps 1 and 2 on the page, and to select the correct Omnibus GitLab package, with the same version @@ -1160,7 +1040,11 @@ You can specify multiple roles, like sentinel and Redis, as: ```ruby # Specify server role as 'redis_replica_role' and enable Consul agent - roles(['redis_replica_role', 'consul_role']) + roles ['redis_sentinel_role', 'redis_replica_role', 'consul_role'] + + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -1172,16 +1056,19 @@ You can specify multiple roles, like sentinel and Redis, as: # machines to connect to it. redis['port'] = 6379 + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + # The same password for Redis authentication you set up for the primary node. - redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + + ## Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-persistent' # The IP of the primary Redis node. redis['master_ip'] = '10.6.0.61' - # Port of primary Redis server, uncomment to change to non default. Defaults - # to `6379`. - #redis['master_port'] = 6379 - ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -1211,15 +1098,6 @@ You can specify multiple roles, like sentinel and Redis, as: 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - -These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after -a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-queues-nodes), and even after a -`gitlab-ctl reconfigure`, they will get their configuration restored by -the same Sentinels. - Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed. @@ -1229,138 +1107,16 @@ are supported and can be added if needed. </a> </div> -#### Configure the Sentinel Queues nodes - -Now that the Redis servers are all set up, let's configure the Sentinel -servers. The following IPs will be used as an example: - -- `10.6.0.81`: Sentinel - Queues 1 -- `10.6.0.82`: Sentinel - Queues 2 -- `10.6.0.83`: Sentinel - Queues 3 - -NOTE: -If you're using an external Redis Sentinel instance, be sure to exclude the -`requirepass` parameter from the Sentinel configuration. This parameter causes -clients to report `NOAUTH Authentication required.`. -[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279). - -To configure the Sentinel Queues server: - -1. SSH in to the server that will host Sentinel. -1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab - package of your choice. Be sure to both follow _only_ installation steps 1 and 2 - on the page, and to select the correct Omnibus GitLab package, with the same version - and type (Community or Enterprise editions) as your current install. -1. Edit `/etc/gitlab/gitlab.rb` and add the contents: - - ```ruby - roles(['redis_sentinel_role', 'consul_role']) - - ## Must be the same in every sentinel node - redis['master_name'] = 'gitlab-redis-persistent' - - ## The same password for Redis authentication you set up for the primary node. - redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' - - ## The IP of the primary Redis node. - redis['master_ip'] = '10.6.0.61' - - ## Define a port so Redis can listen for TCP requests which will allow other - ## machines to connect to it. - redis['port'] = 6379 - - ## Port of primary Redis server, uncomment to change to non default. Defaults - ## to `6379`. - #redis['master_port'] = 6379 - - ## Configure Sentinel's IP - sentinel['bind'] = '10.6.0.81' - - ## Port that Sentinel listens on, uncomment to change to non default. Defaults - ## to `26379`. - #sentinel['port'] = 26379 - - ## Quorum must reflect the amount of voting sentinels it take to start a failover. - ## Value must NOT be greater then the amount of sentinels. - ## - ## The quorum can be used to tune Sentinel in two ways: - ## 1. If a the quorum is set to a value smaller than the majority of Sentinels - ## we deploy, we are basically making Sentinel more sensible to primary failures, - ## triggering a failover as soon as even just a minority of Sentinels is no longer - ## able to talk with the primary. - ## 1. If a quorum is set to a value greater than the majority of Sentinels, we are - ## making Sentinel able to failover only when there are a very large number (larger - ## than majority) of well connected Sentinels which agree about the primary being down.s - sentinel['quorum'] = 2 - - ## Consider unresponsive server down after x amount of ms. - #sentinel['down_after_milliseconds'] = 10000 - - ## Specifies the failover timeout in milliseconds. It is used in many ways: - ## - ## - The time needed to re-start a failover after a previous failover was - ## already tried against the same primary by a given Sentinel, is two - ## times the failover timeout. - ## - ## - The time needed for a replica replicating to a wrong primary according - ## to a Sentinel current configuration, to be forced to replicate - ## with the right primary, is exactly the failover timeout (counting since - ## the moment a Sentinel detected the misconfiguration). - ## - ## - The time needed to cancel a failover that is already in progress but - ## did not produced any configuration change (REPLICAOF NO ONE yet not - ## acknowledged by the promoted replica). - ## - ## - The maximum time a failover in progress waits for all the replica to be - ## reconfigured as replicas of the new primary. However even after this time - ## the replicas will be reconfigured by the Sentinels anyway, but not with - ## the exact parallel-syncs progression as specified. - #sentinel['failover_timeout'] = 60000 - - ## Enable service discovery for Prometheus - consul['monitoring_service_discovery'] = true - - ## The IPs of the Consul server nodes - ## You can also use FQDNs and intermix them with IPs - consul['configuration'] = { - retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), - } - - # Set the network addresses that the exporters will listen on - node_exporter['listen_address'] = '0.0.0.0:9100' - redis_exporter['listen_address'] = '0.0.0.0:9121' - - # Prevent database migrations from running on upgrade automatically - gitlab_rails['auto_migrate'] = false - ``` - -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace - the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. - -1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: - - ```shell - sudo touch /etc/gitlab/skip-auto-reconfigure - ``` - - Only the primary GitLab application server should handle migrations. - -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -1. Go through the steps again for all the other Sentinel nodes, and - make sure you set up the correct IPs. - -<div align="right"> - <a type="button" class="btn btn-default" href="#setup-components"> - Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> - </a> -</div> - ## Configure Gitaly Cluster -[Gitaly Cluster](../gitaly/praefect.md) is a GitLab provided and recommended fault tolerant solution for storing Git repositories. +[Gitaly Cluster](../gitaly/praefect.md) is a GitLab-provided and recommended +fault tolerant solution for storing Git repositories. In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down. +NOTE: +Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). +For implementations with Gitaly Sharded, the same Gitaly specs should be used. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section. + The recommended cluster setup includes the following components: - 3 Gitaly nodes: Replicated storage of Git repositories. @@ -1527,7 +1283,7 @@ the details of each Gitaly node that makes up the cluster. Each storage is also and this name is used in several areas of the configuration. In this guide, the name of the storage will be `default`. Also, this guide is geared towards new installs, if upgrading an existing environment to use Gitaly Cluster, you may need to use a different name. -Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more info. +Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more information. The following IPs will be used as an example: @@ -1854,7 +1610,7 @@ To configure the Sidekiq nodes, on each one: 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to follow _only_ installation steps 1 and 2 on the page. -1. Open `/etc/gitlab/gitlab.rb` with your editor: +1. Create or edit `/etc/gitlab/gitlab.rb` and use the following configuration: ```ruby # Avoid running unnecessary services on the Sidekiq server @@ -1869,36 +1625,40 @@ To configure the Sidekiq nodes, on each one: gitlab_exporter['enable'] = false nginx['enable'] = false + # External URL + ## This should match the URL of the external load balancer + external_url 'https://gitlab.example.com' + # Redis ## Redis connection details ## First cluster that will host the cache gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' gitlab_rails['redis_cache_sentinels'] = [ - {host: '10.6.0.71', port: 26379}, - {host: '10.6.0.72', port: 26379}, - {host: '10.6.0.73', port: 26379}, + {host: '10.6.0.51', port: 26379}, + {host: '10.6.0.52', port: 26379}, + {host: '10.6.0.53', port: 26379}, ] - ## Second cluster that will host the queues, shared state, and actioncable + ## Second cluster that will host the persistent queues, shared state, and actioncable gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_queues_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_shared_state_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_actioncable_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] # Gitaly Cluster @@ -2052,9 +1812,9 @@ On each node perform the following: gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' gitlab_rails['redis_cache_sentinels'] = [ - {host: '10.6.0.71', port: 26379}, - {host: '10.6.0.72', port: 26379}, - {host: '10.6.0.73', port: 26379}, + {host: '10.6.0.51', port: 26379}, + {host: '10.6.0.52', port: 26379}, + {host: '10.6.0.53', port: 26379}, ] ## Second cluster that will host the queues, shared state, and actionable @@ -2063,19 +1823,19 @@ On each node perform the following: gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_queues_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_shared_state_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_actioncable_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] # Set the network addresses that the exporters used for monitoring will listen on @@ -2147,7 +1907,7 @@ On each node perform the following: 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. If you're [using NFS](#configure-nfs-optional): +1. If you're [using NFS](#configure-nfs): 1. If necessary, install the NFS client utility packages using the following commands: @@ -2289,9 +2049,9 @@ To configure the Monitoring node: ## Configure the object storage GitLab supports using an object storage service for holding numerous types of data. -It's recommended over [NFS](#configure-nfs-optional) and in general it's better -in larger setups as object storage is typically much more performant, reliable, -and scalable. +Object storage is also recommended over [NFS](#configure-nfs) and in general +it's better in larger setups as object storage is typically much more performant, +reliable, and scalable. GitLab has been tested on a number of object storage providers: @@ -2313,6 +2073,9 @@ There are two ways of specifying object storage configuration in GitLab: Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. +GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared via NFS on any GitLab Rails and Sidekiq nodes. +In GitLab 13.6 and later, it's recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. + For configuring object storage in GitLab 13.1 and earlier, or for storage types not supported by consolidated configuration form, refer to the following guides based on what features you intend to use: @@ -2359,7 +2122,7 @@ cluster alongside your instance, read how to </a> </div> -## Configure NFS (optional) +## Configure NFS [Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly) are recommended over NFS wherever possible for improved performance. If you intend @@ -2373,7 +2136,7 @@ unavailable from GitLab 15.0. No further enhancements are planned for this featu Read: -- The [Gitaly and NFS deprecation notice](../gitaly/index.md#nfs-deprecation-notice). +- [Gitaly and NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation). - About the [correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). ## Cloud Native Hybrid reference architecture with Helm Charts (alternative) @@ -2427,12 +2190,10 @@ services where applicable): | PostgreSQL<sup>1</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Internal load balancing node<sup>3</sup> | 1 | 4 vCPU, 3.6GB memory | `n1-highcpu-4` | -| Redis - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Redis - Queues / Shared State<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Redis Sentinel - Cache<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | -| Redis Sentinel - Queues / Shared State<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | -| Gitaly | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | -| Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | +| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | +| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | +| Gitaly<sup>5</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | +| Praefect<sup>5</sup> | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Object storage<sup>4</sup> | n/a | n/a | n/a | @@ -2442,6 +2203,7 @@ services where applicable): 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. <!-- markdownlint-enable MD029 --> NOTE: @@ -2486,11 +2248,6 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 - collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347 - collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347 - - redis_persistent <.[#FF6347]- redis_persistent_sentinel - redis_cache <.[#FF6347]- redis_cache_sentinel } cloud "**Object Storage**" as object_storage #white diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index 99dd29c3a83..23b15b563f7 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -13,6 +13,7 @@ For a full list of reference architectures, see > - **Supported users (approximate):** 2,000 > - **High Availability:** No. For a highly-available environment, you can > follow a modified [3K reference architecture](3k_users.md#supported-modifications-for-lower-user-counts-ha). +> - **Cloud Native Hybrid:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) > - **Test requests per second (RPS) rates:** API: 40 RPS, Web: 4 RPS, Git (Pull): 4 RPS, Git (Push): 1 RPS > - **[Latest 2k weekly performance testing results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/2k)** @@ -302,8 +303,8 @@ further configuration steps. 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Note the PostgreSQL node's IP address or hostname, port, and - plain text password. These will be necessary when configuring the [GitLab - application server](#configure-gitlab-rails) later. + plain text password. These will be necessary when configuring the + [GitLab application server](#configure-gitlab-rails) later. Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/database.html) are supported and can be added if needed. @@ -385,8 +386,8 @@ Omnibus: 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Note the Redis node's IP address or hostname, port, and - Redis password. These will be necessary when [configuring the GitLab - application servers](#configure-gitlab-rails) later. + Redis password. These will be necessary when + [configuring the GitLab application servers](#configure-gitlab-rails) later. Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed. @@ -903,6 +904,9 @@ There are two ways of specifying object storage configuration in GitLab: Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. +GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared via NFS on any GitLab Rails and Sidekiq nodes. +In GitLab 13.6 and later, it's recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. + For configuring object storage in GitLab 13.1 and earlier, or for storage types not supported by consolidated configuration form, refer to the following guides based on what features you intend to use: @@ -964,7 +968,7 @@ unavailable from GitLab 15.0. No further enhancements are planned for this featu Read: -- The [Gitaly and NFS deprecation notice](../gitaly/index.md#nfs-deprecation-notice). +- [Gitaly and NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation). - About the [correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). ## Cloud Native Hybrid reference architecture with Helm Charts (alternative) diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index da36968f053..575dd22b729 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -22,6 +22,7 @@ For a full list of reference architectures, see > - **Supported users (approximate):** 3,000 > - **High Availability:** Yes, although [Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution +> - **Cloud Native Hybrid:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) > - **Test requests per second (RPS) rates:** API: 60 RPS, Web: 6 RPS, Git (Pull): 6 RPS, Git (Push): 1 RPS > - **[Latest 3k weekly performance testing results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/3k)** @@ -33,8 +34,8 @@ For a full list of reference architectures, see | PostgreSQL<sup>1</sup> | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Gitaly | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Gitaly<sup>5</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Sidekiq | 4 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | | GitLab Rails | 3 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | @@ -48,6 +49,7 @@ For a full list of reference architectures, see 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. <!-- markdownlint-enable MD029 --> NOTE: @@ -1065,6 +1067,10 @@ The following IPs will be used as an example: [Gitaly Cluster](../gitaly/praefect.md) is a GitLab provided and recommended fault tolerant solution for storing Git repositories. In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down. +NOTE: +Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). +For implementations with Gitaly Sharded, the same Gitaly specs should be used. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section. + The recommended cluster setup includes the following components: - 3 Gitaly nodes: Replicated storage of Git repositories. @@ -1230,7 +1236,7 @@ the details of each Gitaly node that makes up the cluster. Each storage is also and this name is used in several areas of the configuration. In this guide, the name of the storage will be `default`. Also, this guide is geared towards new installs, if upgrading an existing environment to use Gitaly Cluster, you may need to use a different name. -Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more info. +Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more information. The following IPs will be used as an example: @@ -1559,7 +1565,7 @@ To configure the Sidekiq nodes, one each one: 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to follow _only_ installation steps 1 and 2 on the page. -1. Open `/etc/gitlab/gitlab.rb` with your editor: +1. Create or edit `/etc/gitlab/gitlab.rb` and use the following configuration: ```ruby # Avoid running unnecessary services on the Sidekiq server @@ -1574,6 +1580,10 @@ To configure the Sidekiq nodes, one each one: gitlab_exporter['enable'] = false nginx['enable'] = false + # External URL + ## This should match the URL of the external load balancer + external_url 'https://gitlab.example.com' + # Redis redis['master_name'] = 'gitlab-redis' @@ -2011,6 +2021,9 @@ There are two ways of specifying object storage configuration in GitLab: Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. +GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared via NFS on any GitLab Rails and Sidekiq nodes. +In GitLab 13.6 and later, it's recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. + For configuring object storage in GitLab 13.1 and earlier, or for storage types not supported by consolidated configuration form, refer to the following guides based on what features you intend to use: @@ -2071,7 +2084,7 @@ unavailable from GitLab 15.0. No further enhancements are planned for this featu Read: -- The [Gitaly and NFS deprecation notice](../gitaly/index.md#nfs-deprecation-notice). +- [Gitaly and NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation). - About the [correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). ## Supported modifications for lower user counts (HA) @@ -2150,8 +2163,8 @@ services where applicable): | PostgreSQL<sup>1</sup> | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Gitaly | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| Gitaly<sup>5</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | +| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Object storage<sup>4</sup> | n/a | n/a | n/a | @@ -2161,6 +2174,7 @@ services where applicable): 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. <!-- markdownlint-enable MD029 --> NOTE: diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index b071b48cbd9..be44f464e7e 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -12,6 +12,7 @@ full list of reference architectures, see > - **Supported users (approximate):** 50,000 > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) +> - **Cloud Native Hybrid:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) > - **Test requests per second (RPS) rates:** API: 1000 RPS, Web: 100 RPS, Git (Pull): 100 RPS, Git (Push): 20 RPS > - **[Latest 50k weekly performance testing results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/50k)** @@ -22,18 +23,16 @@ full list of reference architectures, see | PostgreSQL<sup>1</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | `D32s v3` | | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Internal load balancing node<sup>3</sup> | 1 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | -| Redis - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis - Queues / Shared State<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis Sentinel - Cache<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | -| Redis Sentinel - Queues / Shared State<sup>2</sup>| 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | -| Gitaly | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` | `m5.16xlarge` | `D64s v3` | -| Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Gitaly<sup>5</sup> | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` | `m5.16xlarge` | `D64s v3` | +| Praefect<sup>5</sup> | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | GitLab Rails | 12 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` | | Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | | Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | n/a | -| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| NFS server (non-Gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> @@ -41,6 +40,7 @@ full list of reference architectures, see 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. <!-- markdownlint-enable MD029 --> NOTE: @@ -82,11 +82,6 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 - collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347 - collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347 - - redis_persistent <.[#FF6347]- redis_persistent_sentinel - redis_cache <.[#FF6347]- redis_cache_sentinel } cloud "**Object Storage**" as object_storage #white @@ -137,8 +132,7 @@ our [Sysbench](https://github.com/akopytov/sysbench)-based Due to better performance and availability, for data objects (such as LFS, uploads, or artifacts), using an [object storage service](#configure-the-object-storage) -is recommended instead of using NFS. Using an object storage service also -doesn't require you to provision and maintain a node. +is recommended. It's also worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and that to achieve full High Availability a third-party PostgreSQL database solution will be required. @@ -169,10 +163,8 @@ To set up GitLab and its components to accommodate up to 50,000 users: used for shared data objects. 1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, more advanced code search across your entire GitLab instance. -1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended) - to have shared disk storage service as an alternative to Gitaly or object - storage. You can skip this step if you're not using GitLab Pages (which - requires NFS). +1. [Configure NFS](#configure-nfs) + to have shared disk storage service for certain GitLab operations (non Gitaly or Object Storage). The servers start on the same 10.6.0.0/24 private network range, and can connect to each other freely on these addresses. @@ -193,15 +185,9 @@ The following list includes descriptions of each server and its assigned IP: - `10.6.0.51`: Redis - Cache Primary - `10.6.0.52`: Redis - Cache Replica 1 - `10.6.0.53`: Redis - Cache Replica 2 -- `10.6.0.71`: Sentinel - Cache 1 -- `10.6.0.72`: Sentinel - Cache 2 -- `10.6.0.73`: Sentinel - Cache 3 -- `10.6.0.61`: Redis - Queues Primary -- `10.6.0.62`: Redis - Queues Replica 1 -- `10.6.0.63`: Redis - Queues Replica 2 -- `10.6.0.81`: Sentinel - Queues 1 -- `10.6.0.82`: Sentinel - Queues 2 -- `10.6.0.83`: Sentinel - Queues 3 +- `10.6.0.61`: Redis - Persistent Primary +- `10.6.0.62`: Redis - Persistent Replica 1 +- `10.6.0.63`: Redis - Persistent Replica 2 - `10.6.0.91`: Gitaly 1 - `10.6.0.92`: Gitaly 2 - `10.6.0.93`: Gitaly 3 @@ -802,15 +788,9 @@ to be used with GitLab. The following IPs will be used as an example: - `10.6.0.51`: Redis - Cache Primary - `10.6.0.52`: Redis - Cache Replica 1 - `10.6.0.53`: Redis - Cache Replica 2 -- `10.6.0.71`: Sentinel - Cache 1 -- `10.6.0.72`: Sentinel - Cache 2 -- `10.6.0.73`: Sentinel - Cache 3 -- `10.6.0.61`: Redis - Queues Primary -- `10.6.0.62`: Redis - Queues Replica 1 -- `10.6.0.63`: Redis - Queues Replica 2 -- `10.6.0.81`: Sentinel - Queues 1 -- `10.6.0.82`: Sentinel - Queues 2 -- `10.6.0.83`: Sentinel - Queues 3 +- `10.6.0.61`: Redis - Persistent Primary +- `10.6.0.62`: Redis - Persistent Replica 1 +- `10.6.0.63`: Redis - Persistent Replica 2 ### Providing your own Redis instance @@ -822,7 +802,7 @@ optional count argument to SPOP, which is required for [Merge Trains](../../ci/p Note the Redis node's IP address or hostname, port, and password (if required). These will be necessary later when configuring the [GitLab application servers](#configure-gitlab-rails). -### Configure the Redis and Sentinel Cache cluster +### Configure the Redis Cache cluster This is the section where we install and set up the new Redis Cache instances. @@ -840,8 +820,12 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' and enable Consul agent - roles(['redis_master_role', 'consul_role']) + # Specify server role as 'redis_master_role' with Sentinel and enable Consul agent + roles(['redis_sentinel_role', 'redis_master_role', 'consul_role']) + + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -853,8 +837,19 @@ a node and change its status from primary to replica (and vice versa). # machines to connect to it. redis['port'] = 6379 - # Set up password authentication for Redis (use the same password in all nodes). + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + + # Set up password authentication for Redis and replicas (use the same password in all nodes). redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + + ## Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-cache' + + ## The IP of this primary Redis node. + redis['master_ip'] = '10.6.0.51' # Set the Redis Cache instance as an LRU # 90% of available RAM in MB @@ -888,10 +883,6 @@ a node and change its status from primary to replica (and vice versa). 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - #### Configure the replica Redis Cache nodes 1. SSH in to the **replica** Redis server. @@ -899,11 +890,15 @@ You can specify multiple roles, like sentinel and Redis, as: package of your choice. Be sure to both follow _only_ installation steps 1 and 2 on the page, and to select the correct Omnibus GitLab package, with the same version and type (Community or Enterprise editions) as your current install. -1. Edit `/etc/gitlab/gitlab.rb` and add the contents: +1. Edit `/etc/gitlab/gitlab.rb` and add the same contents as the priimary node in the previous section by replacing `redis_master_node` with `redis_replica_node`: ```ruby - # Specify server role as 'redis_replica_role' and enable Consul agent - roles(['redis_replica_role', 'consul_role']) + # Specify server role as 'redis_replica_role' with Sentinel and enable Consul agent + roles(['roles_sentinel_role', 'redis_replica_role', 'consul_role']) + + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -915,15 +910,19 @@ You can specify multiple roles, like sentinel and Redis, as: # machines to connect to it. redis['port'] = 6379 - # The same password for Redis authentication you set up for the primary node. + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + + # Set up password authentication for Redis and replicas (use the same password in all nodes). redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' - # The IP of the primary Redis node. - redis['master_ip'] = '10.6.0.51' + ## Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-cache' - # Port of primary Redis server, uncomment to change to non default. Defaults - # to `6379`. - #redis['master_port'] = 6379 + ## The IP of the primary Redis node. + redis['master_ip'] = '10.6.0.51' # Set the Redis Cache instance as an LRU # 90% of available RAM in MB @@ -952,25 +951,18 @@ You can specify multiple roles, like sentinel and Redis, as: gitlab_rails['auto_migrate'] = false ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace - the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus + node you configured and add or replace the file of the same name on this + server. If this is the first Omnibus node you are configuring then you + can skip this step. -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes + to take effect. -1. Go through the steps again for all the other replica nodes, and + 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - -These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after -a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-cache-nodes), and even after a -`gitlab-ctl reconfigure`, they will get their configuration restored by -the same Sentinels. - -Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) -are supported and can be added if needed. + Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed. <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -978,126 +970,7 @@ are supported and can be added if needed. </a> </div> -#### Configure the Sentinel Cache nodes - -Now that the Redis servers are all set up, let's configure the Sentinel -servers. The following IPs will be used as an example: - -- `10.6.0.71`: Sentinel - Cache 1 -- `10.6.0.72`: Sentinel - Cache 2 -- `10.6.0.73`: Sentinel - Cache 3 - -NOTE: -If you're using an external Redis Sentinel instance, be sure to exclude the -`requirepass` parameter from the Sentinel configuration. This parameter causes -clients to report `NOAUTH Authentication required.`. -[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279). - -To configure the Sentinel Cache server: - -1. SSH in to the server that will host Consul/Sentinel. -1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab - package of your choice. Be sure to both follow _only_ installation steps 1 and 2 - on the page, and to select the correct Omnibus GitLab package, with the same version - and type (Community or Enterprise editions) as your current install. -1. Edit `/etc/gitlab/gitlab.rb` and add the contents: - - ```ruby - roles(['redis_sentinel_role', 'consul_role']) - - ## Must be the same in every sentinel node - redis['master_name'] = 'gitlab-redis-cache' - - ## The same password for Redis authentication you set up for the primary node. - redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' - - ## The IP of the primary Redis node. - redis['master_ip'] = '10.6.0.51' - - ## Define a port so Redis can listen for TCP requests which will allow other - ## machines to connect to it. - redis['port'] = 6379 - - ## Port of primary Redis server, uncomment to change to non default. Defaults - ## to `6379`. - #redis['master_port'] = 6379 - - ## Configure Sentinel's IP - sentinel['bind'] = '10.6.0.71' - - ## Port that Sentinel listens on, uncomment to change to non default. Defaults - ## to `26379`. - #sentinel['port'] = 26379 - - ## Quorum must reflect the amount of voting sentinels it take to start a failover. - ## Value must NOT be greater then the amount of sentinels. - ## - ## The quorum can be used to tune Sentinel in two ways: - ## 1. If a the quorum is set to a value smaller than the majority of Sentinels - ## we deploy, we are basically making Sentinel more sensible to primary failures, - ## triggering a failover as soon as even just a minority of Sentinels is no longer - ## able to talk with the primary. - ## 1. If a quorum is set to a value greater than the majority of Sentinels, we are - ## making Sentinel able to failover only when there are a very large number (larger - ## than majority) of well connected Sentinels which agree about the primary being down.s - sentinel['quorum'] = 2 - - ## Consider unresponsive server down after x amount of ms. - #sentinel['down_after_milliseconds'] = 10000 - - ## Specifies the failover timeout in milliseconds. It is used in many ways: - ## - ## - The time needed to re-start a failover after a previous failover was - ## already tried against the same primary by a given Sentinel, is two - ## times the failover timeout. - ## - ## - The time needed for a replica replicating to a wrong primary according - ## to a Sentinel current configuration, to be forced to replicate - ## with the right primary, is exactly the failover timeout (counting since - ## the moment a Sentinel detected the misconfiguration). - ## - ## - The time needed to cancel a failover that is already in progress but - ## did not produced any configuration change (REPLICAOF NO ONE yet not - ## acknowledged by the promoted replica). - ## - ## - The maximum time a failover in progress waits for all the replica to be - ## reconfigured as replicas of the new primary. However even after this time - ## the replicas will be reconfigured by the Sentinels anyway, but not with - ## the exact parallel-syncs progression as specified. - #sentinel['failover_timeout'] = 60000 - - ## Enable service discovery for Prometheus - consul['monitoring_service_discovery'] = true - - ## The IPs of the Consul server nodes - ## You can also use FQDNs and intermix them with IPs - consul['configuration'] = { - retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), - } - - # Set the network addresses that the exporters will listen on - node_exporter['listen_address'] = '0.0.0.0:9100' - redis_exporter['listen_address'] = '0.0.0.0:9121' - - # Prevent database migrations from running on upgrade automatically - gitlab_rails['auto_migrate'] = false - ``` - -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace - the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. - -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -1. Go through the steps again for all the other Consul/Sentinel nodes, and - make sure you set up the correct IPs. - -<div align="right"> - <a type="button" class="btn btn-default" href="#setup-components"> - Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> - </a> -</div> - -### Configure the Redis and Sentinel Queues cluster +### Configure the Redis Persistent cluster This is the section where we install and set up the new Redis Queues instances. @@ -1105,7 +978,7 @@ Both the primary and replica Redis nodes need the same password defined in `redis['password']`. At any time during a failover, the Sentinels can reconfigure a node and change its status from primary to replica (and vice versa). -#### Configure the primary Redis Queues node +#### Configure the primary Redis Persistent node 1. SSH in to the **Primary** Redis server. 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab @@ -1115,8 +988,12 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' and enable Consul agent - roles(['redis_master_role', 'consul_role']) + # Specify server roles as 'redis_master_role' with Sentinel and enable the Consul agent + roles ['redis_sentinel_role', 'redis_master_role', 'consul_role'] + + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -1128,8 +1005,19 @@ a node and change its status from primary to replica (and vice versa). # machines to connect to it. redis['port'] = 6379 - # Set up password authentication for Redis (use the same password in all nodes). - redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + + # Set up password authentication for Redis and replicas (use the same password in all nodes). + redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + + ## Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-persistent' + + ## The IP of this primary Redis node. + redis['master_ip'] = '10.6.0.61' ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -1153,13 +1041,9 @@ a node and change its status from primary to replica (and vice versa). 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -You can specify multiple roles, like sentinel and Redis, as: -`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). +#### Configure the replica Redis Persistent nodes -#### Configure the replica Redis Queues nodes - -1. SSH in to the **replica** Redis Queue server. +1. SSH in to the **replica** Redis Persistent server. 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to both follow _only_ installation steps 1 and 2 on the page, and to select the correct Omnibus GitLab package, with the same version @@ -1167,8 +1051,12 @@ You can specify multiple roles, like sentinel and Redis, as: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_replica_role' and enable Consul agent - roles(['redis_replica_role', 'consul_role']) + # Specify server roles as 'redis_replica_role' with Sentinel and enable Consul agent + roles ['redis_sentinel_role', 'redis_replica_role', 'consul_role'] + + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -1180,16 +1068,20 @@ You can specify multiple roles, like sentinel and Redis, as: # machines to connect to it. redis['port'] = 6379 + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + # The same password for Redis authentication you set up for the primary node. redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + + ## Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-persistent' # The IP of the primary Redis node. redis['master_ip'] = '10.6.0.61' - # Port of primary Redis server, uncomment to change to non default. Defaults - # to `6379`. - #redis['master_port'] = 6379 - ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -1215,144 +1107,7 @@ You can specify multiple roles, like sentinel and Redis, as: 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - -These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after -a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-queues-nodes), and even after a -`gitlab-ctl reconfigure`, they will get their configuration restored by -the same Sentinels. - -Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) -are supported and can be added if needed. - -<div align="right"> - <a type="button" class="btn btn-default" href="#setup-components"> - Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> - </a> -</div> - -#### Configure the Sentinel Queues nodes - -Now that the Redis servers are all set up, let's configure the Sentinel -servers. The following IPs will be used as an example: - -- `10.6.0.81`: Sentinel - Queues 1 -- `10.6.0.82`: Sentinel - Queues 2 -- `10.6.0.83`: Sentinel - Queues 3 - -NOTE: -If you're using an external Redis Sentinel instance, be sure to exclude the -`requirepass` parameter from the Sentinel configuration. This parameter causes -clients to report `NOAUTH Authentication required.`. -[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279). - -To configure the Sentinel Queues server: - -1. SSH in to the server that will host Sentinel. -1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab - package of your choice. Be sure to both follow _only_ installation steps 1 and 2 - on the page, and to select the correct Omnibus GitLab package, with the same version - and type (Community or Enterprise editions) as your current install. -1. Edit `/etc/gitlab/gitlab.rb` and add the contents: - - ```ruby - roles(['redis_sentinel_role', 'consul_role']) - - ## Must be the same in every sentinel node - redis['master_name'] = 'gitlab-redis-persistent' - - ## The same password for Redis authentication you set up for the primary node. - redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' - - ## The IP of the primary Redis node. - redis['master_ip'] = '10.6.0.61' - - ## Define a port so Redis can listen for TCP requests which will allow other - ## machines to connect to it. - redis['port'] = 6379 - - ## Port of primary Redis server, uncomment to change to non default. Defaults - ## to `6379`. - #redis['master_port'] = 6379 - - ## Configure Sentinel's IP - sentinel['bind'] = '10.6.0.81' - - ## Port that Sentinel listens on, uncomment to change to non default. Defaults - ## to `26379`. - #sentinel['port'] = 26379 - - ## Quorum must reflect the amount of voting sentinels it take to start a failover. - ## Value must NOT be greater then the amount of sentinels. - ## - ## The quorum can be used to tune Sentinel in two ways: - ## 1. If a the quorum is set to a value smaller than the majority of Sentinels - ## we deploy, we are basically making Sentinel more sensible to primary failures, - ## triggering a failover as soon as even just a minority of Sentinels is no longer - ## able to talk with the primary. - ## 1. If a quorum is set to a value greater than the majority of Sentinels, we are - ## making Sentinel able to failover only when there are a very large number (larger - ## than majority) of well connected Sentinels which agree about the primary being down.s - sentinel['quorum'] = 2 - - ## Consider unresponsive server down after x amount of ms. - #sentinel['down_after_milliseconds'] = 10000 - - ## Specifies the failover timeout in milliseconds. It is used in many ways: - ## - ## - The time needed to re-start a failover after a previous failover was - ## already tried against the same primary by a given Sentinel, is two - ## times the failover timeout. - ## - ## - The time needed for a replica replicating to a wrong primary according - ## to a Sentinel current configuration, to be forced to replicate - ## with the right primary, is exactly the failover timeout (counting since - ## the moment a Sentinel detected the misconfiguration). - ## - ## - The time needed to cancel a failover that is already in progress but - ## did not produced any configuration change (REPLICAOF NO ONE yet not - ## acknowledged by the promoted replica). - ## - ## - The maximum time a failover in progress waits for all the replica to be - ## reconfigured as replicas of the new primary. However even after this time - ## the replicas will be reconfigured by the Sentinels anyway, but not with - ## the exact parallel-syncs progression as specified. - #sentinel['failover_timeout'] = 60000 - - ## Enable service discovery for Prometheus - consul['monitoring_service_discovery'] = true - - ## The IPs of the Consul server nodes - ## You can also use FQDNs and intermix them with IPs - consul['configuration'] = { - retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), - } - - # Set the network addresses that the exporters will listen on - node_exporter['listen_address'] = '0.0.0.0:9100' - redis_exporter['listen_address'] = '0.0.0.0:9121' - - # Prevent database migrations from running on upgrade automatically - gitlab_rails['auto_migrate'] = false - ``` - -1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: - - ```shell - sudo touch /etc/gitlab/skip-auto-reconfigure - ``` - - Only the primary GitLab application server should handle migrations. - -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace - the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. - -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -1. Go through the steps again for all the other Sentinel nodes, and - make sure you set up the correct IPs. +Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed. <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -1365,6 +1120,10 @@ To configure the Sentinel Queues server: [Gitaly Cluster](../gitaly/praefect.md) is a GitLab provided and recommended fault tolerant solution for storing Git repositories. In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down. +NOTE: +Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). +For implementations with Gitaly Sharded, the same Gitaly specs should be used. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section. + The recommended cluster setup includes the following components: - 3 Gitaly nodes: Replicated storage of Git repositories. @@ -1531,7 +1290,7 @@ the details of each Gitaly node that makes up the cluster. Each storage is also and this name is used in several areas of the configuration. In this guide, the name of the storage will be `default`. Also, this guide is geared towards new installs, if upgrading an existing environment to use Gitaly Cluster, you may need to use a different name. -Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more info. +Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more information. The following IPs will be used as an example: @@ -1858,7 +1617,7 @@ To configure the Sidekiq nodes, on each one: 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to follow _only_ installation steps 1 and 2 on the page. -1. Open `/etc/gitlab/gitlab.rb` with your editor: +1. Create or edit `/etc/gitlab/gitlab.rb` and use the following configuration: ```ruby # Avoid running unnecessary services on the Sidekiq server @@ -1873,36 +1632,40 @@ To configure the Sidekiq nodes, on each one: gitlab_exporter['enable'] = false nginx['enable'] = false + # External URL + ## This should match the URL of the external load balancer + external_url 'https://gitlab.example.com' + # Redis ## Redis connection details ## First cluster that will host the cache gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' gitlab_rails['redis_cache_sentinels'] = [ - {host: '10.6.0.71', port: 26379}, - {host: '10.6.0.72', port: 26379}, - {host: '10.6.0.73', port: 26379}, + {host: '10.6.0.51', port: 26379}, + {host: '10.6.0.52', port: 26379}, + {host: '10.6.0.53', port: 26379}, ] - ## Second cluster that will host the queues, shared state, and actioncable + ## Second cluster that will host the persistent queues, shared state, and actioncable gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_queues_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_shared_state_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_actioncable_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] # Gitaly @@ -2063,30 +1826,30 @@ On each node perform the following: gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' gitlab_rails['redis_cache_sentinels'] = [ - {host: '10.6.0.71', port: 26379}, - {host: '10.6.0.72', port: 26379}, - {host: '10.6.0.73', port: 26379}, + {host: '10.6.0.51', port: 26379}, + {host: '10.6.0.52', port: 26379}, + {host: '10.6.0.53', port: 26379}, ] - ## Second cluster that will host the queues, shared state, and actionable + ## Second cluster that will host the persistent queues, shared state, and actionable gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_queues_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_shared_state_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_actioncable_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] # Set the network addresses that the exporters used for monitoring will listen on @@ -2158,7 +1921,7 @@ On each node perform the following: 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. If you're [using NFS](#configure-nfs-optional): +1. If you're [using NFS](#configure-nfs): 1. If necessary, install the NFS client utility packages using the following commands: @@ -2300,7 +2063,7 @@ To configure the Monitoring node: ## Configure the object storage GitLab supports using an object storage service for holding numerous types of data. -It's recommended over [NFS](#configure-nfs-optional) and in general it's better +It's recommended over [NFS](#configure-nfs) and in general it's better in larger setups as object storage is typically much more performant, reliable, and scalable. @@ -2324,6 +2087,9 @@ There are two ways of specifying object storage configuration in GitLab: Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. +GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared via NFS on any GitLab Rails and Sidekiq nodes. +In GitLab 13.6 and later, it's recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. + For configuring object storage in GitLab 13.1 and earlier, or for storage types not supported by consolidated configuration form, refer to the following guides based on what features you intend to use: @@ -2370,7 +2136,7 @@ cluster alongside your instance, read how to </a> </div> -## Configure NFS (optional) +## Configure NFS [Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly) are recommended over NFS wherever possible for improved performance. If you intend @@ -2384,7 +2150,7 @@ unavailable from GitLab 15.0. No further enhancements are planned for this featu Read: -- The [Gitaly and NFS deprecation notice](../gitaly/index.md#nfs-deprecation-notice). +- [Gitaly and NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation). - About the [correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). ## Cloud Native Hybrid reference architecture with Helm Charts (alternative) @@ -2438,12 +2204,10 @@ services where applicable): | PostgreSQL<sup>1</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Internal load balancing node<sup>3</sup> | 1 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | -| Redis - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Redis - Queues / Shared State<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Redis Sentinel - Cache<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | -| Redis Sentinel - Queues / Shared State<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | -| Gitaly | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` | -| Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | +| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | +| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | +| Gitaly<sup>5</sup> | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` | +| Praefect<sup>5</sup> | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Object storage<sup>4</sup> | n/a | n/a | n/a | @@ -2453,6 +2217,7 @@ services where applicable): 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. <!-- markdownlint-enable MD029 --> NOTE: @@ -2497,11 +2262,6 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 - collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347 - collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347 - - redis_persistent <.[#FF6347]- redis_persistent_sentinel - redis_cache <.[#FF6347]- redis_cache_sentinel } cloud "**Object Storage**" as object_storage #white diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index 4dfe628039a..a5526986be1 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -19,6 +19,7 @@ costly-to-operate environment by using the > - **Supported users (approximate):** 5,000 > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) +> - **Cloud Native Hybrid:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) > - **Test requests per second (RPS) rates:** API: 100 RPS, Web: 10 RPS, Git (Pull): 10 RPS, Git (Push): 2 RPS > - **[Latest 5k weekly performance testing results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/5k)** @@ -30,8 +31,8 @@ costly-to-operate environment by using the | PostgreSQL<sup>1</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Gitaly | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | `D8s v3` | -| Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Gitaly<sup>5</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | `D8s v3` | +| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Sidekiq | 4 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | | GitLab Rails | 3 | 16 vCPU, 14.4 GB memory | `n1-highcpu-16` | `c5.4xlarge` | `F16s v2`| @@ -45,6 +46,7 @@ costly-to-operate environment by using the 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. <!-- markdownlint-enable MD029 --> NOTE: @@ -1056,6 +1058,10 @@ The following IPs will be used as an example: [Gitaly Cluster](../gitaly/praefect.md) is a GitLab provided and recommended fault tolerant solution for storing Git repositories. In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down. +NOTE: +Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). +For implementations with Gitaly Sharded, the same Gitaly specs should be used. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section. + The recommended cluster setup includes the following components: - 3 Gitaly nodes: Replicated storage of Git repositories. @@ -1222,7 +1228,7 @@ the details of each Gitaly node that makes up the cluster. Each storage is also and this name is used in several areas of the configuration. In this guide, the name of the storage will be `default`. Also, this guide is geared towards new installs, if upgrading an existing environment to use Gitaly Cluster, you may need to use a different name. -Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more info. +Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more information. The following IPs will be used as an example: @@ -1549,7 +1555,7 @@ To configure the Sidekiq nodes, one each one: 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to follow _only_ installation steps 1 and 2 on the page. -1. Open `/etc/gitlab/gitlab.rb` with your editor: +1. Create or edit `/etc/gitlab/gitlab.rb` and use the following configuration: ```ruby # Avoid running unnecessary services on the Sidekiq server @@ -1564,6 +1570,10 @@ To configure the Sidekiq nodes, one each one: gitlab_exporter['enable'] = false nginx['enable'] = false + # External URL + ## This should match the URL of the external load balancer + external_url 'https://gitlab.example.com' + # Redis ## Must be the same in every sentinel node redis['master_name'] = 'gitlab-redis' @@ -2005,6 +2015,9 @@ There are two ways of specifying object storage configuration in GitLab: Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. +GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared via NFS on any GitLab Rails and Sidekiq nodes. +In GitLab 13.6 and later, it's recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. + For configuring object storage in GitLab 13.1 and earlier, or for storage types not supported by consolidated configuration form, refer to the following guides based on what features you intend to use: @@ -2065,7 +2078,7 @@ unavailable from GitLab 15.0. No further enhancements are planned for this featu Read: -- The [Gitaly and NFS deprecation notice](../gitaly/index.md#nfs-deprecation-notice). +- [Gitaly and NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation). - About the [correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). ## Cloud Native Hybrid reference architecture with Helm Charts (alternative) @@ -2120,8 +2133,8 @@ services where applicable): | PostgreSQL<sup>1</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Gitaly | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | -| Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| Gitaly<sup>5</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | +| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Object storage<sup>4</sup> | n/a | n/a | n/a | @@ -2131,6 +2144,7 @@ services where applicable): 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. <!-- markdownlint-enable MD029 --> NOTE: |