diff options
Diffstat (limited to 'doc/administration/reference_architectures/25k_users.md')
| -rw-r--r-- | doc/administration/reference_architectures/25k_users.md | 430 |
1 files changed, 217 insertions, 213 deletions
diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index 129386d6ce5..1f72c45c2b7 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -17,29 +17,34 @@ full list of reference architectures, see | Service | Nodes | Configuration | GCP | AWS | Azure | |------------------------------------------|-------------|-------------------------|------------------|--------------|-----------| -| External load balancing node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | -| Consul* | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| PostgreSQL* | 3 | 16 vCPU, 60 GB memory | `n1-standard-1` | `m5.4xlarge` | `D16s v3` | -| PgBouncer* | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Internal load balancing node | 1 | 4 vCPU, 3.6GB memory | `n1-highcpu-4` | `c5.large` | `F2s v2` | -| Redis - Cache** | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis - Queues / Shared State** | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis Sentinel - Cache** | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | -| Redis Sentinel - Queues / Shared State** | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | +| External load balancing node(3) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| Consul(1) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| PostgreSQL(1) | 3 | 16 vCPU, 60 GB memory | `n1-standard-1` | `m5.4xlarge` | `D16s v3` | +| PgBouncer(1) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Internal load balancing node(3) | 1 | 4 vCPU, 3.6GB memory | `n1-highcpu-4` | `c5.large` | `F2s v2` | +| Redis - Cache(2) | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis - Queues / Shared State(2) | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis Sentinel - Cache(2) | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | +| Redis Sentinel - Queues / Shared State(2)| 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | | Gitaly | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | `D32s v3` | | Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | -| Praefect PostgreSQL* | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Praefect PostgreSQL(1) | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | GitLab Rails | 5 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` | | Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | -| Object storage | n/a | n/a | n/a | n/a | n/a | -| NFS server | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| Object storage(4) | n/a | n/a | n/a | n/a | n/a | +| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | + +<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> +<!-- markdownlint-disable MD029 --> +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. +4. Should be run on reputable third party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +<!-- markdownlint-enable MD029 --> NOTE: -Components marked with * can be optionally run on reputable -third party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work. -Components marked with ** can be optionally run on reputable -third party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +For all PaaS solutions that involve configuring instances, it is strongly recommended to implement a minimum of three nodes in three different availability zones to align with resilient cloud architecture practices. ```plantuml @startuml 25k @@ -157,7 +162,7 @@ To set up GitLab and its components to accommodate up to 25,000 users: provides access to the Git repositories. 1. [Configure Sidekiq](#configure-sidekiq). 1. [Configure the main GitLab Rails application](#configure-gitlab-rails) - to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend + to run Puma, Workhorse, GitLab Shell, and to serve all frontend requests (which include UI, API, and Git over HTTP/SSH). 1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab environment. @@ -413,11 +418,6 @@ The following IPs will be used as an example: - `10.6.0.12`: Consul 2 - `10.6.0.13`: Consul 3 -NOTE: -The configuration processes for the other servers in your reference architecture will -use the `/etc/gitlab/gitlab-secrets.json` file from your Consul server to connect -with the other servers. - To configure Consul: 1. SSH in to the server that will host Consul. @@ -428,10 +428,9 @@ To configure Consul: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - roles ['consul_role'] + roles(['consul_role']) ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -448,7 +447,11 @@ To configure Consul: gitlab_rails['auto_migrate'] = false ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Go through the steps again for all the other Consul nodes, and make sure you set up the correct IPs. @@ -540,6 +543,15 @@ in the second step, do not supply the `EXTERNAL_URL` value. sudo gitlab-ctl pg-password-md5 pgbouncer ``` +1. Generate a password hash for the PostgreSQL replication username/password pair. This assumes you will use the default + username of `gitlab_replicator` (recommended). The command will request a password + and a confirmation. Use the value that is output by this command in the next step + as the value of `<postgresql_replication_password_hash>`: + + ```shell + sudo gitlab-ctl pg-password-md5 gitlab_replicator + ``` + 1. Generate a password hash for the Consul database username/password pair. This assumes you will use the default username of `gitlab-consul` (recommended). The command will request a password and confirmation. Use the value that is output by this command in the next @@ -552,19 +564,21 @@ in the second step, do not supply the `EXTERNAL_URL` value. 1. On every database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: ```ruby - # Disable all components except PostgreSQL, Patroni, and Consul - roles ['postgres_role'] + # Disable all components except Patroni and Consul + roles(['patroni_role']) # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' - # Enable Patroni - patroni['enable'] = true - # Set `max_wal_senders` to one more than the number of database nodes in the cluster. + # Sets `max_replication_slots` to double the number of database nodes. + # Patroni uses one extra slot per node when initiating the replication. + patroni['postgresql']['max_replication_slots'] = 8 + + # Set `max_wal_senders` to one more than the number of replication slots in the cluster. # This is used to prevent replication from using up all of the # available database connections. - patroni['postgresql']['max_wal_senders'] = 4 - patroni['postgresql']['max_replication_slots'] = 4 + patroni['postgresql']['max_wal_senders'] = 9 + # Incoming recommended value for max connections is 500. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691. patroni['postgresql']['max_connections'] = 500 @@ -572,7 +586,6 @@ in the second step, do not supply the `EXTERNAL_URL` value. gitlab_rails['auto_migrate'] = false # Configure the Consul agent - consul['enable'] = true consul['services'] = %w(postgresql) ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -582,6 +595,8 @@ in the second step, do not supply the `EXTERNAL_URL` value. # # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value postgresql['pgbouncer_user_password'] = '<pgbouncer_password_hash>' + # Replace POSTGRESQL_REPLICATION_PASSWORD_HASH with a generated md5 value + postgresql['sql_replication_password'] = '<postgresql_replication_password_hash>' # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value postgresql['sql_user_password'] = '<postgresql_password_hash>' @@ -605,9 +620,8 @@ PostgreSQL, with Patroni managing its failover, will default to use `pg_rewind` Like most failover handling methods, this has a small chance of leading to data loss. Learn more about the various [Patroni replication methods](../postgresql/replication_and_failover.md#selecting-the-appropriate-patroni-replication-method). -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -622,21 +636,7 @@ are supported and can be added if needed. #### PostgreSQL post-configuration -SSH in to the **primary node**: - -1. Open a database prompt: - - ```shell - gitlab-psql -d gitlabhq_production - ``` - -1. Make sure the `pg_trgm` extension is enabled (it might already be): - - ```shell - CREATE EXTENSION pg_trgm; - ``` - -1. Exit the database prompt by typing `\q` and Enter. +SSH in to any of the Patroni nodes on the **primary site**: 1. Check the status of the leader and cluster: @@ -678,7 +678,7 @@ The following IPs will be used as an example: ```ruby # Disable all components except Pgbouncer and Consul agent - roles ['pgbouncer_role'] + roles(['pgbouncer_role']) # Configure PgBouncer pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) @@ -695,7 +695,6 @@ The following IPs will be used as an example: # Configure Consul agent consul['watchers'] = %w(postgresql) - consul['enable'] = true consul['configuration'] = { retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } @@ -707,9 +706,8 @@ The following IPs will be used as an example: node_exporter['listen_address'] = '0.0.0.0:9100' ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -828,8 +826,8 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' - roles ['redis_master_role'] + # Specify server role as 'redis_master_role' and enable Consul agent + roles(['redis_master_role', 'consul_role'] # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -851,7 +849,6 @@ a node and change its status from primary to replica (and vice versa). redis['maxmemory_samples'] = 5 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -863,19 +860,22 @@ a node and change its status from primary to replica (and vice versa). # Set the network addresses that the exporters will listen on node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' + redis_exporter['flags'] = { + 'redis.addr' => 'redis://10.6.0.51:6379', + 'redis.password' => 'redis-password-goes-here', + } # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. You can specify multiple roles, like sentinel and Redis, as: -`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about [roles](https://docs.gitlab.com/omnibus/roles/). #### Configure the replica Redis Cache nodes @@ -888,8 +888,8 @@ You can specify multiple roles, like sentinel and Redis, as: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_replica_role' - roles ['redis_replica_role'] + # Specify server role as 'redis_replica_role' and enable Consul agent + roles(['redis_replica_role', 'consul_role'] # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -918,7 +918,6 @@ You can specify multiple roles, like sentinel and Redis, as: redis['maxmemory_samples'] = 5 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -930,21 +929,25 @@ You can specify multiple roles, like sentinel and Redis, as: # Set the network addresses that the exporters will listen on node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' + redis_exporter['flags'] = { + 'redis.addr' => 'redis://10.6.0.52:6379', + 'redis.password' => 'redis-password-goes-here', + } # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. You can specify multiple roles, like sentinel and Redis, as: -`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about [roles](https://docs.gitlab.com/omnibus/roles/). These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after @@ -986,7 +989,7 @@ To configure the Sentinel Cache server: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - roles ['redis_sentinel_role'] + roles(['redis_sentinel_role', 'consul_role']) ## Must be the same in every sentinel node redis['master_name'] = 'gitlab-redis-cache' @@ -1050,7 +1053,6 @@ To configure the Sentinel Cache server: #sentinel['failover_timeout'] = 60000 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -1067,11 +1069,11 @@ To configure the Sentinel Cache server: gitlab_rails['auto_migrate'] = false ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Go through the steps again for all the other Consul/Sentinel nodes, and make sure you set up the correct IPs. @@ -1099,8 +1101,8 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' - roles ['redis_master_role'] + # Specify server role as 'redis_master_role' and enable Consul agent + roles(['redis_master_role', 'consul_role']) # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -1116,7 +1118,6 @@ a node and change its status from primary to replica (and vice versa). redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -1133,14 +1134,13 @@ a node and change its status from primary to replica (and vice versa). gitlab_rails['auto_migrate'] = false ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. You can specify multiple roles, like sentinel and Redis, as: -`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about [roles](https://docs.gitlab.com/omnibus/roles/). #### Configure the replica Redis Queues nodes @@ -1153,8 +1153,8 @@ You can specify multiple roles, like sentinel and Redis, as: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_replica_role' - roles ['redis_replica_role'] + # Specify server role as 'redis_replica_role' and enable Consul agent + roles(['redis_replica_role', 'consul_role']) # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -1177,7 +1177,6 @@ You can specify multiple roles, like sentinel and Redis, as: #redis['master_port'] = 6379 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -1189,21 +1188,25 @@ You can specify multiple roles, like sentinel and Redis, as: # Set the network addresses that the exporters will listen on node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' + redis_exporter['flags'] = { + 'redis.addr' => 'redis://10.6.0.62:6379', + 'redis.password' => 'redis-password-goes-here', + } # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. You can specify multiple roles, like sentinel and Redis, as: -`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about [roles](https://docs.gitlab.com/omnibus/roles/). These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after @@ -1245,7 +1248,7 @@ To configure the Sentinel Queues server: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - roles ['redis_sentinel_role'] + roles(['redis_sentinel_role', 'consul_role']) ## Must be the same in every sentinel node redis['master_name'] = 'gitlab-redis-persistent' @@ -1309,7 +1312,6 @@ To configure the Sentinel Queues server: #sentinel['failover_timeout'] = 60000 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -1326,7 +1328,10 @@ To configure the Sentinel Queues server: gitlab_rails['auto_migrate'] = false ``` -1. To prevent database migrations from running on upgrade, run: +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + +1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: ```shell sudo touch /etc/gitlab/skip-auto-reconfigure @@ -1334,11 +1339,8 @@ To configure the Sentinel Queues server: Only the primary GitLab application server should handle migrations. -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. - 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Go through the steps again for all the other Sentinel nodes, and make sure you set up the correct IPs. @@ -1399,9 +1401,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. ```ruby # Disable all components except PostgreSQL and Consul - roles ['postgres_role'] - repmgr['enable'] = false - patroni['enable'] = false + roles(['postgres_role', 'consul_role']) # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' @@ -1411,7 +1411,6 @@ in the second step, do not supply the `EXTERNAL_URL` value. gitlab_rails['auto_migrate'] = false # Configure the Consul agent - consul['enable'] = true ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -1437,7 +1436,11 @@ in the second step, do not supply the `EXTERNAL_URL` value. # END user configuration ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Follow the [post configuration](#praefect-postgresql-post-configuration). <div align="right"> @@ -1535,19 +1538,18 @@ To configure the Praefect nodes, on each one: 1. Edit the `/etc/gitlab/gitlab.rb` file to configure Praefect: ```ruby - # Avoid running unnecessary services on the Gitaly server + # Avoid running unnecessary services on the Praefect server + gitaly['enable'] = false postgresql['enable'] = false redis['enable'] = false - nginx['enable'] = false puma['enable'] = false - unicorn['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false - grafana['enable'] = false - - # If you run a separate monitoring node you can disable these services - alertmanager['enable'] = false prometheus['enable'] = false + alertmanager['enable'] = false + grafana['enable'] = false + gitlab_exporter['enable'] = false + nginx['enable'] = false # Praefect Configuration praefect['enable'] = true @@ -1585,19 +1587,20 @@ To configure the Praefect nodes, on each one: # server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1') praefect['virtual_storages'] = { 'default' => { - 'gitaly-1' => { - 'address' => 'tcp://10.6.0.91:8075', - 'token' => '<praefect_internal_token>', - 'primary' => true - }, - 'gitaly-2' => { - 'address' => 'tcp://10.6.0.92:8075', - 'token' => '<praefect_internal_token>' - }, - 'gitaly-3' => { - 'address' => 'tcp://10.6.0.93:8075', - 'token' => '<praefect_internal_token>' - }, + 'nodes' => { + 'gitaly-1' => { + 'address' => 'tcp://10.6.0.91:8075', + 'token' => '<praefect_internal_token>' + }, + 'gitaly-2' => { + 'address' => 'tcp://10.6.0.92:8075', + 'token' => '<praefect_internal_token>' + }, + 'gitaly-3' => { + 'address' => 'tcp://10.6.0.93:8075', + 'token' => '<praefect_internal_token>' + }, + } } } @@ -1614,11 +1617,25 @@ To configure the Praefect nodes, on each one: # END user configuration ``` - 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and - then replace the file of the same name on this server. If that file isn't on - this server, add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace +the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + +1. Praefect requires to run some database migrations, much like the main GitLab application. For this + you should select **one Praefect node only to run the migrations**, AKA the _Deploy Node_. This node + must be configured first before the others as follows: + + 1. In the `/etc/gitlab/gitlab.rb` file, change the `praefect['auto_migrate']` setting value from `false` to `true` + + 1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: - 1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure + ``` + + 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect and + to run the Praefect database migrations. + +1. On all other Praefect nodes, [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. ### Configure Gitaly @@ -1662,21 +1679,17 @@ On each node: storage paths, enable the network listener, and to configure the token: ```ruby - # /etc/gitlab/gitlab.rb - # Avoid running unnecessary services on the Gitaly server postgresql['enable'] = false redis['enable'] = false - nginx['enable'] = false puma['enable'] = false - unicorn['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false - grafana['enable'] = false - - # If you run a separate monitoring node you can disable these services - alertmanager['enable'] = false prometheus['enable'] = false + alertmanager['enable'] = false + grafana['enable'] = false + gitlab_exporter['enable'] = false + nginx['enable'] = false # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -1684,9 +1697,11 @@ On each node: # Configure the gitlab-shell API callback URL. Without this, `git push` will # fail. This can be your 'front door' GitLab URL or an internal load # balancer. - # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server. gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' + # Gitaly + gitaly['enable'] = true + # Make Gitaly accept connections on all network interfaces. You must use # firewalls to restrict access to this address/port. # Comment out following line if you only want to support TLS connections @@ -1728,9 +1743,8 @@ On each node: }) ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and - then replace the file of the same name on this server. If that file isn't on - this server, add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -1837,28 +1851,19 @@ To configure the Sidekiq nodes, on each one: 1. Open `/etc/gitlab/gitlab.rb` with your editor: ```ruby - ######################################## - ##### Services Disabled ### - ######################################## - - nginx['enable'] = false - grafana['enable'] = false - prometheus['enable'] = false - alertmanager['enable'] = false + # Avoid running unnecessary services on the Sidekiq server gitaly['enable'] = false - gitlab_workhorse['enable'] = false - nginx['enable'] = false - puma['enable'] = false - postgres_exporter['enable'] = false postgresql['enable'] = false redis['enable'] = false - redis_exporter['enable'] = false + puma['enable'] = false + gitlab_workhorse['enable'] = false + prometheus['enable'] = false + alertmanager['enable'] = false + grafana['enable'] = false gitlab_exporter['enable'] = false + nginx['enable'] = false - ######################################## - #### Redis ### - ######################################## - + # Redis ## Redis connection details ## First cluster that will host the cache gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' @@ -1890,13 +1895,10 @@ To configure the Sidekiq nodes, on each one: {host: '10.6.0.83', port: 26379}, ] - ####################################### - ### Gitaly ### - ####################################### - - # git_data_dirs get configured for the Praefect virtual storage - # Address is Internal Load Balancer for Praefect - # Token is praefect_external_token + # Gitaly Cluster + ## git_data_dirs get configured for the Praefect virtual storage + ## Address is Internal Load Balancer for Praefect + ## Token is praefect_external_token git_data_dirs({ "default" => { "gitaly_address" => "tcp://10.6.0.40:2305", # internal load balancer IP @@ -1904,20 +1906,17 @@ To configure the Sidekiq nodes, on each one: } }) - ####################################### - ### Postgres ### - ####################################### + # PostgreSQL gitlab_rails['db_host'] = '10.6.0.20' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '<postgresql_user_password>' gitlab_rails['db_adapter'] = 'postgresql' gitlab_rails['db_encoding'] = 'unicode' - # Prevent database migrations from running on upgrade automatically + ## Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false - ####################################### - ### Sidekiq configuration ### - ####################################### + # Sidekiq + sidekiq['enable'] = true sidekiq['listen_address'] = "0.0.0.0" # Set number of Sidekiq queue processes to the same number as available CPUs @@ -1926,9 +1925,7 @@ To configure the Sidekiq nodes, on each one: # Set number of Sidekiq threads per queue process to the recommend number of 10 sidekiq['max_concurrency'] = 10 - ####################################### - ### Monitoring configuration ### - ####################################### + # Monitoring consul['enable'] = true consul['monitoring_service_discovery'] = true @@ -1936,16 +1933,13 @@ To configure the Sidekiq nodes, on each one: retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } - # Set the network addresses that the exporters will listen on + ## Set the network addresses that the exporters will listen on node_exporter['listen_address'] = '0.0.0.0:9100' - # Rails Status for prometheus + ## Add the monitoring node's IP address to the monitoring whitelist gitlab_rails['monitoring_whitelist'] = ['10.6.0.151/32', '127.0.0.0/8'] - ############################# - ### Object storage ### - ############################# - + # Object Storage # This is an example for configuring Object Storage on GCP # Replace this config with your chosen Object Storage provider as desired gitlab_rails['object_store']['connection'] = { @@ -1960,11 +1954,26 @@ To configure the Sidekiq nodes, on each one: gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-packages-bucket-name>" gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>" gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>" + + gitlab_rails['backup_upload_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } + gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" + ``` + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + +1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: + + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. + Only a single designated node should handle migrations as detailed in the + [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section. 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -1997,9 +2006,6 @@ On each node perform the following: 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to follow _only_ installation steps 1 and 2 on the page. -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. 1. Edit `/etc/gitlab/gitlab.rb` and use the following configuration. To maintain uniformity of links across nodes, the `external_url` @@ -2021,7 +2027,7 @@ On each node perform the following: }) ## Disable components that will not be on the GitLab application server - roles ['application_role'] + roles(['application_role']) gitaly['enable'] = false nginx['enable'] = true sidekiq['enable'] = false @@ -2094,9 +2100,15 @@ On each node perform the following: gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-packages-bucket-name>" gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>" gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>" + + gitlab_rails['backup_upload_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } + gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). 1. If you're using [Gitaly with TLS support](#gitaly-cluster-tls-support), make sure the `git_data_dirs` entry is configured with `tls` instead of `tcp`: @@ -2115,6 +2127,20 @@ On each node perform the following: sudo cp cert.pem /etc/gitlab/trusted-certs/ ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + +1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: + + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure + ``` + + Only a single designated node should handle migrations as detailed in the + [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section. + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. If you're [using NFS](#configure-nfs-optional): 1. If necessary, install the NFS client utility packages using the following commands: @@ -2154,7 +2180,7 @@ On each node perform the following: registry['gid'] = 9002 ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). 1. Confirm the node can connect to Gitaly: ```shell @@ -2218,52 +2244,30 @@ To configure the Monitoring node: 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to follow _only_ installation steps 1 and 2 on the page. -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - external_url 'http://gitlab.example.com' + roles(['monitoring_role', 'consul_role']) - # Disable all other services - alertmanager['enable'] = false - gitaly['enable'] = false - gitlab_exporter['enable'] = false - gitlab_workhorse['enable'] = false - nginx['enable'] = true - postgres_exporter['enable'] = false - postgresql['enable'] = false - redis['enable'] = false - redis_exporter['enable'] = false - sidekiq['enable'] = false - puma['enable'] = false - unicorn['enable'] = false - node_exporter['enable'] = false - gitlab_exporter['enable'] = false + external_url 'http://gitlab.example.com' - # Enable Prometheus - prometheus['enable'] = true + # Prometheus prometheus['listen_address'] = '0.0.0.0:9090' prometheus['monitor_kubernetes'] = false - # Enable Login form - grafana['disable_login_form'] = false - - # Enable Grafana - grafana['enable'] = true + # Grafana grafana['admin_password'] = '<grafana_password>' + grafana['disable_login_form'] = false # Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true consul['configuration'] = { retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } - # Prevent database migrations from running on upgrade automatically - gitlab_rails['auto_migrate'] = false + # Nginx - For Grafana access + nginx['enable'] = true ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). |