diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2021-05-19 15:44:42 +0000 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2021-05-19 15:44:42 +0000 |
| commit | 4555e1b21c365ed8303ffb7a3325d773c9b8bf31 (patch) | |
| tree | 5423a1c7516cffe36384133ade12572cf709398d /doc/administration/reference_architectures | |
| parent | e570267f2f6b326480d284e0164a6464ba4081bc (diff) | |
| download | gitlab-ce-4555e1b21c365ed8303ffb7a3325d773c9b8bf31.tar.gz | |
Add latest changes from gitlab-org/gitlab@13-12-stable-eev13.12.0-rc42
Diffstat (limited to 'doc/administration/reference_architectures')
8 files changed, 201 insertions, 110 deletions
diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index 97af1fe8d3c..e4a699b962c 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -24,8 +24,8 @@ full list of reference architectures, see | Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Redis - Cache** | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | Redis - Queues / Shared State** | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis Sentinel - Cache** | 3 | 1 vCPU, 1.7 GB memory | `g1-small` | `t3.small` | `B1MS` | -| Redis Sentinel - Queues / Shared State** | 3 | 1 vCPU, 1.7 GB memory | `g1-small` | `t3.small` | `B1MS` | +| Redis Sentinel - Cache** | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | +| Redis Sentinel - Queues / Shared State** | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | | Gitaly | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | `D16s v3` | | Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Praefect PostgreSQL* | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | @@ -129,7 +129,7 @@ The Google Cloud Platform (GCP) architectures were built and tested using the CPU platform. On different hardware you may find that adjustments, either lower or higher, are required for your CPU or node counts. For more information, see our [Sysbench](https://github.com/akopytov/sysbench)-based -[CPU benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). +[CPU benchmarks](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). Due to better performance and availability, for data objects (such as LFS, uploads, or artifacts), using an [object storage service](#configure-the-object-storage) @@ -1917,7 +1917,12 @@ To configure the Sidekiq nodes, on each one: ### Sidekiq configuration ### ####################################### sidekiq['listen_address'] = "0.0.0.0" - sidekiq['cluster'] = true # no need to set this after GitLab 13.0 + + # Set number of Sidekiq queue processes to the same number as available CPUs + sidekiq['queue_groups'] = ['*'] * 4 + + # Set number of Sidekiq threads per queue process to the recommend number of 10 + sidekiq['max_concurrency'] = 10 ####################################### ### Monitoring configuration ### @@ -1962,7 +1967,9 @@ To configure the Sidekiq nodes, on each one: 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. NOTE: -You can also run [multiple Sidekiq processes](../operations/extra_sidekiq_processes.md). +If you find that the environment's Sidekiq job processing is slow with long queues, +more nodes can be added as required. You can also tune your Sidekiq nodes to +run [multiple Sidekiq processes](../operations/extra_sidekiq_processes.md). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -2363,7 +2370,7 @@ considered and customer technical support will be considered out of scope. As an alternative approach, you can also run select components of GitLab as Cloud Native in Kubernetes via our official [Helm Charts](https://docs.gitlab.com/charts/). In this setup, we support running the equivalent of GitLab Rails and Sidekiq nodes -in a Kubernetes cluster, named Webservice and Sidekiq respectively. In addition, +in a Kubernetes cluster, named Webservice and Sidekiq respectively. In addition, the following other supporting services are supported: NGINX, Task Runner, Migrations, Prometheus and Grafana. @@ -2405,8 +2412,8 @@ services where applicable): | Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Redis - Cache** | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | | Redis - Queues / Shared State** | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Redis Sentinel - Cache** | 3 | 1 vCPU, 1.7 GB memory | `g1-small` | -| Redis Sentinel - Queues / Shared State** | 3 | 1 vCPU, 1.7 GB memory | `g1-small` | +| Redis Sentinel - Cache** | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | +| Redis Sentinel - Queues / Shared State** | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | | Gitaly | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | | Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Praefect PostgreSQL* | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | diff --git a/doc/administration/reference_architectures/1k_users.md b/doc/administration/reference_architectures/1k_users.md index bed584e94bd..6ad4e0f05e3 100644 --- a/doc/administration/reference_architectures/1k_users.md +++ b/doc/administration/reference_architectures/1k_users.md @@ -17,20 +17,20 @@ many organizations . > - **Supported users (approximate):** 1,000 > - **High Availability:** No. For a highly-available environment, you can -> follow the [3K reference architecture](3k_users.md). +> follow a modified [3K reference architecture](3k_users.md#supported-modifications-for-lower-user-counts-ha). > - **Test requests per second (RPS) rates:** API: 20 RPS, Web: 2 RPS, Git (Pull): 2 RPS, Git (Push): 1 RPS -| Users | Configuration | GCP | AWS | Azure | -|--------------|-------------------------|----------------|-----------------|----------------| -| Up to 500 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | -| Up to 1,000 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | `c5.2xlarge` | F8s v2 | +| Users | Configuration | GCP | AWS | Azure | +|--------------|-------------------------|----------------|--------------|----------| +| Up to 500 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| Up to 1,000 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | The Google Cloud Platform (GCP) architectures were built and tested using the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) CPU platform. On different hardware you may find that adjustments, either lower or higher, are required for your CPU or node counts. For more information, see our [Sysbench](https://github.com/akopytov/sysbench)-based -[CPU benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). +[CPU benchmarks](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). In addition to the stated configurations, we recommend having at least 2 GB of swap on your server, even if you currently have enough available memory. Having diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index 7f9f284d085..129386d6ce5 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -15,25 +15,31 @@ full list of reference architectures, see > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) > - **Test requests per second (RPS) rates:** API: 500 RPS, Web: 50 RPS, Git (Pull): 50 RPS, Git (Push): 10 RPS -| Service | Nodes | Configuration | GCP | AWS | Azure | -|-----------------------------------------|-------------|-------------------------|-----------------|-------------|----------| -| External load balancing node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | -| Consul | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | -| PostgreSQL | 3 | 16 vCPU, 60 GB memory | n1-standard-16 | `m5.4xlarge` | D16s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | -| Internal load balancing node | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | `c5.large` | F2s v2 | -| Redis - Cache | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | -| Redis - Queues / Shared State | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | -| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7 GB memory | g1-small | `t3.small` | B1MS | -| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7 GB memory | g1-small | `t3.small` | B1MS | -| Gitaly | 3 | 32 vCPU, 120 GB memory | n1-standard-32 | `m5.8xlarge` | D32s v3 | -| Praefect | 3 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | -| Praefect PostgreSQL | 1+* | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | -| Sidekiq | 4 | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | -| GitLab Rails | 5 | 32 vCPU, 28.8 GB memory | n1-highcpu-32 | `c5.9xlarge` | F32s v2 | -| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | -| Object storage | n/a | n/a | n/a | n/a | n/a | -| NFS server | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | +| Service | Nodes | Configuration | GCP | AWS | Azure | +|------------------------------------------|-------------|-------------------------|------------------|--------------|-----------| +| External load balancing node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| Consul* | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| PostgreSQL* | 3 | 16 vCPU, 60 GB memory | `n1-standard-1` | `m5.4xlarge` | `D16s v3` | +| PgBouncer* | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Internal load balancing node | 1 | 4 vCPU, 3.6GB memory | `n1-highcpu-4` | `c5.large` | `F2s v2` | +| Redis - Cache** | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis - Queues / Shared State** | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis Sentinel - Cache** | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | +| Redis Sentinel - Queues / Shared State** | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | +| Gitaly | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | `D32s v3` | +| Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| Praefect PostgreSQL* | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| GitLab Rails | 5 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` | +| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| Object storage | n/a | n/a | n/a | n/a | n/a | +| NFS server | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | + +NOTE: +Components marked with * can be optionally run on reputable +third party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work. +Components marked with ** can be optionally run on reputable +third party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. ```plantuml @startuml 25k @@ -123,7 +129,7 @@ The Google Cloud Platform (GCP) architectures were built and tested using the CPU platform. On different hardware you may find that adjustments, either lower or higher, are required for your CPU or node counts. For more information, see our [Sysbench](https://github.com/akopytov/sysbench)-based -[CPU benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). +[CPU benchmarks](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). Due to better performance and availability, for data objects (such as LFS, uploads, or artifacts), using an [object storage service](#configure-the-object-storage) @@ -1913,7 +1919,12 @@ To configure the Sidekiq nodes, on each one: ### Sidekiq configuration ### ####################################### sidekiq['listen_address'] = "0.0.0.0" - sidekiq['cluster'] = true # no need to set this after GitLab 13.0 + + # Set number of Sidekiq queue processes to the same number as available CPUs + sidekiq['queue_groups'] = ['*'] * 4 + + # Set number of Sidekiq threads per queue process to the recommend number of 10 + sidekiq['max_concurrency'] = 10 ####################################### ### Monitoring configuration ### @@ -1958,7 +1969,9 @@ To configure the Sidekiq nodes, on each one: 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. NOTE: -You can also run [multiple Sidekiq processes](../operations/extra_sidekiq_processes.md). +If you find that the environment's Sidekiq job processing is slow with long queues, +more nodes can be added as required. You can also tune your Sidekiq nodes to +run [multiple Sidekiq processes](../operations/extra_sidekiq_processes.md). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index e5418e47ca2..69e261cfbe6 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -13,19 +13,25 @@ For a full list of reference architectures, see > - **Supported users (approximate):** 2,000 > - **High Availability:** No. For a highly-available environment, you can -> follow the [3K reference architecture](3k_users.md). +> follow a modified [3K reference architecture](3k_users.md#supported-modifications-for-lower-user-counts-ha). > - **Test requests per second (RPS) rates:** API: 40 RPS, Web: 4 RPS, Git (Pull): 4 RPS, Git (Push): 1 RPS -| Service | Nodes | Configuration | GCP | AWS | Azure | -|------------------------------------------|--------|-------------------------|----------------|--------------|---------| -| Load balancer | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | -| PostgreSQL | 1 | 2 vCPU, 7.5 GB memory | n1-standard-2 | `m5.large` | D2s v3 | -| Redis | 1 | 1 vCPU, 3.75 GB memory | n1-standard-1 | `m5.large` | D2s v3 | -| Gitaly | 1 | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | -| GitLab Rails | 2 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | `c5.2xlarge` | F8s v2 | -| Monitoring node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | -| Object storage | n/a | n/a | n/a | n/a | n/a | -| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | +| Service | Nodes | Configuration | GCP | AWS | Azure | +|------------------------------------------|--------|-------------------------|-----------------|--------------|----------| +| Load balancer | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| PostgreSQL* | 1 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | +| Redis** | 1 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `m5.large` | `D2s v3` | +| Gitaly | 1 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| GitLab Rails | 2 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | +| Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Object storage | n/a | n/a | n/a | n/a | n/a | +| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | + +NOTE: +Components marked with * can be optionally run on reputable +third party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work. +Components marked with ** can be optionally run on reputable +third party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. ```plantuml @startuml 2k @@ -60,7 +66,7 @@ The Google Cloud Platform (GCP) architectures were built and tested using the CPU platform. On different hardware you may find that adjustments, either lower or higher, are required for your CPU or node counts. For more information, see our [Sysbench](https://github.com/akopytov/sysbench)-based -[CPU benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). +[CPU benchmarks](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). Due to better performance and availability, for data objects (such as LFS, uploads, or artifacts), using an [object storage service](#configure-the-object-storage) @@ -642,6 +648,8 @@ On each node perform the following: node_exporter['listen_address'] = '0.0.0.0:9100' gitlab_workhorse['prometheus_listen_addr'] = '0.0.0.0:9229' sidekiq['listen_address'] = "0.0.0.0" + # Set number of Sidekiq threads per queue process to the recommend number of 10 + sidekiq['max_concurrency'] = 10 puma['listen'] = '0.0.0.0' # Add the monitoring node's IP address to the monitoring whitelist and allow it to diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index b8d0a98a1f1..d81f98a35d4 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -14,8 +14,9 @@ than 3,000 users. For fewer users, reduce the stated node sizes as needed. If maintaining a high level of uptime for your GitLab environment isn't a requirement, or if you don't have the expertise to maintain this sort of -environment, we recommend using the [2,000-user reference architecture](2k_users.md) -for your GitLab installation. +environment, we recommend using the non-HA [2,000-user reference architecture](2k_users.md) +for your GitLab installation. If HA is still a requirement, there's several supported +tweaks you can make to this architecture to reduce complexity as detailed here. For a full list of reference architectures, see [Available reference architectures](index.md#available-reference-architectures). @@ -24,22 +25,28 @@ For a full list of reference architectures, see > - **High Availability:** Yes, although [Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution > - **Test requests per second (RPS) rates:** API: 60 RPS, Web: 6 RPS, Git (Pull): 6 RPS, Git (Push): 1 RPS -| Service | Nodes | Configuration | GCP | AWS | Azure | -|--------------------------------------------|-------------|-----------------------|----------------|-------------|---------| -| External load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | -| Redis | 3 | 2 vCPU, 7.5 GB memory | n1-standard-2 | `m5.large` | D2s v3 | -| Consul + Sentinel | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | -| PostgreSQL | 3 | 2 vCPU, 7.5 GB memory | n1-standard-2 | `m5.large` | D2s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | -| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | -| Gitaly | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | -| Praefect | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | -| Praefect PostgreSQL | 1+* | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | -| Sidekiq | 4 | 2 vCPU, 7.5 GB memory | n1-standard-2 | `m5.large` | D2s v3 | -| GitLab Rails | 3 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | `c5.2xlarge` | F8s v2 | -| Monitoring node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | -| Object storage | n/a | n/a | n/a | n/a | n/a | -| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | +| Service | Nodes | Configuration | GCP | AWS | Azure | +|--------------------------------------------|-------------|-----------------------|-----------------|--------------|----------| +| External load balancing node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Redis** | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | +| Consul* + Sentinel** | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| PostgreSQL* | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | +| PgBouncer* | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Gitaly | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Praefect PostgreSQL* | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Sidekiq | 4 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | +| GitLab Rails | 3 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | +| Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Object storage | n/a | n/a | n/a | n/a | n/a | +| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | + +NOTE: +Components marked with * can be optionally run on reputable +third party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work. +Components marked with ** can be optionally run on reputable +third party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. ```plantuml @startuml 3k @@ -129,7 +136,7 @@ The Google Cloud Platform (GCP) architectures were built and tested using the CPU platform. On different hardware you may find that adjustments, either lower or higher, are required for your CPU or node counts. For more information, see our [Sysbench](https://github.com/akopytov/sysbench)-based -[CPU benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). +[CPU benchmarks](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). Due to better performance and availability, for data objects (such as LFS, uploads, or artifacts), using an [object storage service](#configure-the-object-storage) @@ -1595,6 +1602,12 @@ To configure the Sidekiq nodes, one each one: ####################################### sidekiq['listen_address'] = "0.0.0.0" + # Set number of Sidekiq queue processes to the same number as available CPUs + sidekiq['queue_groups'] = ['*'] * 2 + + # Set number of Sidekiq threads per queue process to the recommend number of 10 + sidekiq['max_concurrency'] = 10 + ####################################### ### Monitoring configuration ### ####################################### @@ -1650,7 +1663,9 @@ To configure the Sidekiq nodes, one each one: ``` NOTE: -You can also run [multiple Sidekiq processes](../operations/extra_sidekiq_processes.md). +If you find that the environment's Sidekiq job processing is slow with long queues, +more nodes can be added as required. You can also tune your Sidekiq nodes to +run [multiple Sidekiq processes](../operations/extra_sidekiq_processes.md). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -2040,6 +2055,27 @@ considered and customer technical support will be considered out of scope. [Read more about Gitaly and NFS](../gitaly/index.md#nfs-deprecation-notice) and [the correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). +## Supported modifications for lower user counts (HA) + +The 3k GitLab reference architecture is the smallest we recommend that achieves High Availability (HA). +However, for environments that need to serve less users but maintain HA, there's several +supported modifications you can make to this architecture to reduce complexity and cost. + +It should be noted that to achieve HA with GitLab, this architecture's makeup is ultimately what is +required. Each component has various considerations and rules to follow and this architecture +meets all of these. Smaller versions of this architecture will be fundamentally the same, +but with smaller performance requirements, several modifications can be considered as follows: + +- Lowering node specs: Depending on your user count, you can lower all suggested node specs as desired. However, it's recommended that you don't go lower than the [general requirements](../../install/requirements.md). +- Combining select nodes: Some nodes can be combined to reduce complexity at the cost of some performance: + - GitLab Rails and Sidekiq: Sidekiq nodes can be removed and the component instead enabled on the GitLab Rails nodes. + - PostgreSQL and PgBouncer: PgBouncer nodes can be removed and the component instead enabled on PostgreSQL with the Internal Load Balancer pointing to them instead. +- Running select components in reputable Cloud PaaS solutions: Select components of the GitLab setup can instead be run on Cloud Provider PaaS solutions. By doing this, additional dependent components can also be removed: + - PostgreSQL: Can be run on reputable Cloud PaaS solutions such as Google Cloud SQL or AWS RDS. In this setup, the PgBouncer and Consul nodes are no longer required: + - Consul may still be desired if [Prometheus](../monitoring/prometheus/index.md) auto discovery is a requirement, otherwise you would need to [manually add scrape configurations](../monitoring/prometheus/index.md#adding-custom-scrape-configurations) for all nodes. + - As Redis Sentinel runs on the same box as Consul in this architecture, it may need to be run on a separate box if Redis is still being run via Omnibus. + - Redis: Can be run on reputable Cloud PaaS solutions such as Google Memorystore and AWS Elasticache. In this setup, the Redis Sentinel is no longer required. + <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index 183a998e89a..5cc463c953d 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -15,25 +15,31 @@ full list of reference architectures, see > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) > - **Test requests per second (RPS) rates:** API: 1000 RPS, Web: 100 RPS, Git (Pull): 100 RPS, Git (Push): 20 RPS -| Service | Nodes | Configuration | GCP | AWS | Azure | -|-----------------------------------------|-------------|-------------------------|-----------------|--------------|----------| -| External load balancing node | 1 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | `c5.2xlarge` | F8s v2 | -| Consul | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | -| PostgreSQL | 3 | 32 vCPU, 120 GB memory | n1-standard-32 | `m5.8xlarge` | D32s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | -| Internal load balancing node | 1 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | `c5.2xlarge` | F8s v2 | -| Redis - Cache | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | -| Redis - Queues / Shared State | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | -| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7 GB memory | g1-small | `t3.small` | B1MS | -| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7 GB memory | g1-small | `t3.small` | B1MS | -| Gitaly | 3 | 64 vCPU, 240 GB memory | n1-standard-64 | `m5.16xlarge` | D64s v3 | -| Praefect | 3 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | -| Praefect PostgreSQL | 1+* | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | -| Sidekiq | 4 | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | -| GitLab Rails | 12 | 32 vCPU, 28.8 GB memory | n1-highcpu-32 | `c5.9xlarge` | F32s v2 | -| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | -| Object storage | n/a | n/a | n/a | n/a | n/a | -| NFS server | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | +| Service | Nodes | Configuration | GCP | AWS | Azure | +|------------------------------------------|-------------|-------------------------|------------------|---------------|-----------| +| External load balancing node | 1 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | +| Consul* | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| PostgreSQL* | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | `D32s v3` | +| PgBouncer* | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Internal load balancing node | 1 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | +| Redis - Cache** | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis - Queues / Shared State** | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis Sentinel - Cache** | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | +| Redis Sentinel - Queues / Shared State** | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | +| Gitaly | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` | `m5.16xlarge` | `D64s v3` | +| Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| Praefect PostgreSQL* | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| GitLab Rails | 12 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` | +| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| Object storage | n/a | n/a | n/a | n/a | n/a | +| NFS server | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | + +NOTE: +Components marked with * can be optionally run on reputable +third party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work. +Components marked with ** can be optionally run on reputable +third party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. ```plantuml @startuml 50k @@ -123,7 +129,7 @@ The Google Cloud Platform (GCP) architectures were built and tested using the CPU platform. On different hardware you may find that adjustments, either lower or higher, are required for your CPU or node counts. For more information, see our [Sysbench](https://github.com/akopytov/sysbench)-based -[CPU benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). +[CPU benchmarks](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). Due to better performance and availability, for data objects (such as LFS, uploads, or artifacts), using an [object storage service](#configure-the-object-storage) @@ -1920,7 +1926,12 @@ To configure the Sidekiq nodes, on each one: ### Sidekiq configuration ### ####################################### sidekiq['listen_address'] = "0.0.0.0" - sidekiq['cluster'] = true # no need to set this after GitLab 13.0 + + # Set number of Sidekiq queue processes to the same number as available CPUs + sidekiq['queue_groups'] = ['*'] * 4 + + # Set number of Sidekiq threads per queue process to the recommend number of 10 + sidekiq['max_concurrency'] = 10 ####################################### ### Monitoring configuration ### @@ -1965,7 +1976,9 @@ To configure the Sidekiq nodes, on each one: 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. NOTE: -You can also run [multiple Sidekiq processes](../operations/extra_sidekiq_processes.md). +If you find that the environment's Sidekiq job processing is slow with long queues, +more nodes can be added as required. You can also tune your Sidekiq nodes to +run [multiple Sidekiq processes](../operations/extra_sidekiq_processes.md). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index 70d02bba855..792dd7020c7 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -22,22 +22,28 @@ costly-to-operate environment by using the > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) > - **Test requests per second (RPS) rates:** API: 100 RPS, Web: 10 RPS, Git (Pull): 10 RPS, Git (Push): 2 RPS -| Service | Nodes | Configuration | GCP | AWS | Azure | -|--------------------------------------------|-------------|-------------------------|----------------|-------------|----------| -| External load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | -| Redis | 3 | 2 vCPU, 7.5 GB memory | n1-standard-2 | `m5.large` | D2s v3 | -| Consul + Sentinel | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | -| PostgreSQL | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | -| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | -| Gitaly | 3 | 8 vCPU, 30 GB memory | n1-standard-8 | `m5.2xlarge` | D8s v3 | -| Praefect | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | -| Praefect PostgreSQL | 1+* | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | -| Sidekiq | 4 | 2 vCPU, 7.5 GB memory | n1-standard-2 | `m5.large` | D2s v3 | -| GitLab Rails | 3 | 16 vCPU, 14.4 GB memory | n1-highcpu-16 | `c5.4xlarge` | F16s v2 | -| Monitoring node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | -| Object storage | n/a | n/a | n/a | n/a | n/a | -| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | +| Service | Nodes | Configuration | GCP | AWS | Azure | +|--------------------------------------------|-------------|-------------------------|-----------------|--------------|----------| +| External load balancing node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Redis** | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | +| Consul* + Sentinel** | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| PostgreSQL* | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| PgBouncer* | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Gitaly | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | `D8s v3` | +| Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Praefect PostgreSQL* | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Sidekiq | 4 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | +| GitLab Rails | 3 | 16 vCPU, 14.4 GB memory | `n1-highcpu-16` | `c5.4xlarge` | `F16s v2`| +| Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Object storage | n/a | n/a | n/a | n/a | n/a | +| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | + +NOTE: +Components marked with * can be optionally run on reputable +third party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work. +Components marked with ** can be optionally run on reputable +third party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. ```plantuml @startuml 5k @@ -127,7 +133,7 @@ The Google Cloud Platform (GCP) architectures were built and tested using the CPU platform. On different hardware you may find that adjustments, either lower or higher, are required for your CPU or node counts. For more information, see our [Sysbench](https://github.com/akopytov/sysbench)-based -[CPU benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). +[CPU benchmarks](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). Due to better performance and availability, for data objects (such as LFS, uploads, or artifacts), using an [object storage service](#configure-the-object-storage) @@ -1585,6 +1591,12 @@ To configure the Sidekiq nodes, one each one: ####################################### sidekiq['listen_address'] = "0.0.0.0" + # Set number of Sidekiq queue processes to the same number as available CPUs + sidekiq['queue_groups'] = ['*'] * 4 + + # Set number of Sidekiq threads per queue process to the recommend number of 10 + sidekiq['max_concurrency'] = 10 + ####################################### ### Monitoring configuration ### ####################################### @@ -1639,7 +1651,9 @@ To configure the Sidekiq nodes, one each one: ``` NOTE: -You can also run [multiple Sidekiq processes](../operations/extra_sidekiq_processes.md). +If you find that the environment's Sidekiq job processing is slow with long queues, +more nodes can be added as required. You can also tune your Sidekiq nodes to +run [multiple Sidekiq processes](../operations/extra_sidekiq_processes.md). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> diff --git a/doc/administration/reference_architectures/troubleshooting.md b/doc/administration/reference_architectures/troubleshooting.md index 60a6ee0132f..d5f57965a80 100644 --- a/doc/administration/reference_architectures/troubleshooting.md +++ b/doc/administration/reference_architectures/troubleshooting.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Troubleshooting a reference architecture setup +# Troubleshooting a reference architecture setup **(FREE SELF)** This page serves as the troubleshooting documentation if you followed one of the [reference architectures](index.md#reference-architectures). @@ -33,7 +33,7 @@ The dependency on disk storage also prevents Pages being deployed using the ### Incremental logging is required for CI to use object storage If you configure GitLab to use object storage for CI logs and artifacts, -[you must also enable incremental logging](../job_logs.md#new-incremental-logging-architecture). +[you must also enable incremental logging](../job_logs.md#incremental-logging-architecture). ### Proxy Download |