diff options
Diffstat (limited to 'doc/administration/high_availability')
| -rw-r--r-- | doc/administration/high_availability/README.md | 6 | ||||
| -rw-r--r-- | doc/administration/high_availability/database.md | 23 | ||||
| -rw-r--r-- | doc/administration/high_availability/gitaly.md | 25 | ||||
| -rw-r--r-- | doc/administration/high_availability/gitlab.md | 45 | ||||
| -rw-r--r-- | doc/administration/high_availability/monitoring_node.md | 67 | ||||
| -rw-r--r-- | doc/administration/high_availability/redis.md | 28 |
6 files changed, 192 insertions, 2 deletions
diff --git a/doc/administration/high_availability/README.md b/doc/administration/high_availability/README.md index d9c80b1ec59..0c4f926c579 100644 --- a/doc/administration/high_availability/README.md +++ b/doc/administration/high_availability/README.md @@ -65,6 +65,7 @@ larger one. - 1 Redis node - 1 NFS/Gitaly storage server - 2 or more GitLab application nodes (Unicorn, Workhorse, Sidekiq) +- 1 Monitoring node (Prometheus, Grafana) #### Installation Instructions @@ -76,6 +77,7 @@ you can continue with the next step. 1. [Redis](redis.md#redis-in-a-scaled-environment) 1. [Gitaly](gitaly.md) (recommended) or [NFS](nfs.md) 1. [GitLab application nodes](gitlab.md) +1. [Monitoring node (Prometheus and Grafana)](monitoring_node.md) ### Full Scaling @@ -90,6 +92,7 @@ in size, indicating that there is contention or not enough resources. - 2 or more NFS/Gitaly storage servers - 2 or more Sidekiq nodes - 2 or more GitLab application nodes (Unicorn, Workhorse) +- 1 Monitoring node (Prometheus, Grafana) ## High Availability Architecture Examples @@ -133,6 +136,7 @@ the contention. - 3 Consul/Sentinel nodes - 2 or more GitLab application nodes (Unicorn, Workhorse, Sidekiq, PGBouncer) - 1 NFS/Gitaly server +- 1 Monitoring node (Prometheus, Grafana)  @@ -192,6 +196,7 @@ with the added complexity of many more nodes to configure, manage and monitor. - 2 or more API nodes (All requests to `/api`) - 2 or more Web nodes (All other web requests) - 2 or more NFS/Gitaly servers +- 1 Monitoring node (Prometheus, Grafana)  @@ -205,4 +210,5 @@ separately: 1. [NFS Client and Host setup](nfs_host_client_setup.md) 1. [Configure the GitLab application servers](gitlab.md) 1. [Configure the load balancers](load_balancer.md) +1. [Monitoring node (Prometheus and Grafana)](monitoring_node.md) diff --git a/doc/administration/high_availability/database.md b/doc/administration/high_availability/database.md index 2c051b660ee..4db53353218 100644 --- a/doc/administration/high_availability/database.md +++ b/doc/administration/high_availability/database.md @@ -82,7 +82,8 @@ deploy the bundled PostgreSQL. 1. Note the PostgreSQL node's IP address or hostname, port, and plain text password. These will be necessary when configuring the GitLab application servers later. - +1. [Enable monitoring](#enable-monitoring) + Advanced configuration options are supported and can be added if needed. @@ -399,6 +400,7 @@ check the [Troubleshooting section](#troubleshooting) before proceeding. ``` 1. [Reconfigure GitLab] for the changes to take effect. +1. [Enable Monitoring](#enable-monitoring) > Please note: > @@ -1086,6 +1088,25 @@ the previous section: the `gitlab` database user 1. [Reconfigure GitLab] for the changes to take effect +## Enable Monitoring + +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0. + +If you enable Monitoring, it must be enabled on **all** database servers. + +1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration: + + ```ruby + # Enable service discovery for Prometheus + consul['monitoring_service_discovery'] = true + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + postgres_exporter['listen_address'] = '0.0.0.0:9187' + ``` + +1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. + ## Troubleshooting #### Consul and PostgreSQL changes not taking effect. diff --git a/doc/administration/high_availability/gitaly.md b/doc/administration/high_availability/gitaly.md index 1d8e6c999cb..90e5f71d835 100644 --- a/doc/administration/high_availability/gitaly.md +++ b/doc/administration/high_availability/gitaly.md @@ -19,3 +19,28 @@ Continue configuration of other components by going back to: - [Scaled Architectures](README.md#scalable-architecture-examples) - [High Availability Architectures](README.md#high-availability-architecture-examples) + +## Enable Monitoring + +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0. + + 1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration: + + ```ruby + # Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + # Replace placeholders + # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z + # with the addresses of the Consul server nodes + consul['configuration'] = { + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z), + } + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + gitaly['prometheus_listen_addr'] = "0.0.0.0:9236" + ``` + + 1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. diff --git a/doc/administration/high_availability/gitlab.md b/doc/administration/high_availability/gitlab.md index 7e3ff741071..0e655e49922 100644 --- a/doc/administration/high_availability/gitlab.md +++ b/doc/administration/high_availability/gitlab.md @@ -76,6 +76,8 @@ registry['gid'] = 9002 ``` +1. [Enable monitoring](#enable-monitoring) + > **Note:** To maintain uniformity of links across HA clusters, the `external_url` on the first application server as well as the additional application servers should point to the external url that users will use to access GitLab. @@ -88,7 +90,8 @@ [Nginx documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) for more information. > - > **Note:** It is best to set the `uid` and `gid`s prior to the initial reconfigure of GitLab. Omnibus will not recursively `chown` directories if set after the initial reconfigure. + > **Note:** It is best to set the `uid` and `gid`s prior to the initial reconfigure + of GitLab. Omnibus will not recursively `chown` directories if set after the initial reconfigure. ## First GitLab application server @@ -129,6 +132,46 @@ need some extra configuration. 1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. +## Enable Monitoring + +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0. + +If you enable Monitoring, it must be enabled on **all** GitLab servers. + +1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration: + + ```ruby + # Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + # Replace placeholders + # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z + # with the addresses of the Consul server nodes + consul['configuration'] = { + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z), + } + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + gitlab_workhorse['prometheus_listen_addr'] = '0.0.0.0:9229' + sidekiq['listen_address'] = "0.0.0.0" + unicorn['listen'] = '0.0.0.0' + + # Add the monitoring node's IP address to the monitoring whitelist and allow it to scrape the NGINX metrics + # Replace placeholder + # monitoring.gitlab.example.com + # with the addresses gathered for the monitoring node + gitlab_rails['monitoring_whitelist'] = ['monitoring.gitlab.example.com'] + nginx['status']['options']['allow'] = ['monitoring.gitlab.example.com'] + ``` + +1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. + +> **Warning:** After changing `unicorn['listen']` in `gitlab.rb`, and running `sudo gitlab-ctl reconfigure`, + it can take an extended period of time for unicorn to complete reloading after receiving a `HUP`. + For more information, see the [issue](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/4401). + ## Troubleshooting - `mount: wrong fs type, bad option, bad superblock on` diff --git a/doc/administration/high_availability/monitoring_node.md b/doc/administration/high_availability/monitoring_node.md new file mode 100644 index 00000000000..d16bf7dc0f0 --- /dev/null +++ b/doc/administration/high_availability/monitoring_node.md @@ -0,0 +1,67 @@ +# Configuring a Monitoring node for Scaling and High Availability + +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0. + +## Standalone Monitoring node using GitLab Omnibus + +The GitLab Omnibus package can be used to configure a standalone Monitoring node running Prometheus and Grafana. +The monitoring node is not highly available. See [Scaling and High Availability](README.md) +for an overview of GitLab scaling and high availability options. + +The steps below are the minimum necessary to configure a Monitoring node running Prometheus and Grafana with +Omnibus: + +1. SSH into the Monitoring node. +1. [Download/install](https://about.gitlab.com/installation) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page. + - Do not complete any other steps on the download page. + +1. Edit `/etc/gitlab/gitlab.rb` and add the contents: + + ```ruby + external_url 'http://gitlab.example.com' + + # Enable Prometheus + prometheus['enable'] = true + prometheus['listen_address'] = '0.0.0.0:9090' + prometheus['monitor_kubernetes'] = false + + # Enable Grafana + grafana['enable'] = true + grafana['admin_password'] = 'toomanysecrets' + + # Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + # Replace placeholders + # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z + # with the addresses of the Consul server nodes + consul['configuration'] = { + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z), + } + + # Disable all other services + gitlab_rails['auto_migrate'] = false + alertmanager['enable'] = false + gitaly['enable'] = false + gitlab_monitor['enable'] = false + gitlab_workhorse['enable'] = false + nginx['enable'] = true + postgres_exporter['enable'] = false + postgresql['enable'] = false + redis['enable'] = false + redis_exporter['enable'] = false + sidekiq['enable'] = false + unicorn['enable'] = false + node_exporter['enable'] = false + ``` + +1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. + +## Migrating to Service Discovery + +Once monitoring using Service Discovery is enabled with `consul['monitoring_service_discovery'] = true`, +ensure that `prometheus['scrape_configs']` is not set in `/etc/gitlab/gitlab.rb`. Setting both +`consul['monitoring_service_discovery'] = true` and `prometheus['scrape_configs']` in `/etc/gitlab/gitlab.rb` +will result in errors. diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index 1aaa709fc8f..f61a8834af3 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -74,6 +74,7 @@ Omnibus: 1. Note the Redis node's IP address or hostname, port, and Redis password. These will be necessary when configuring the GitLab application servers later. +1. [Enable Monitoring](#enable-monitoring) Advanced configuration options are supported and can be added if needed. @@ -749,6 +750,33 @@ gitlab_rails['redis_sentinels'] = [ [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. +## Enable Monitoring + +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0. + + If you enable Monitoring, it must be enabled on **all** Redis servers. + + 1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration: + + ```ruby + # Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + # Replace placeholders + # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z + # with the addresses of the Consul server nodes + consul['configuration'] = { + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z), + } + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + redis_exporter['listen_address'] = '0.0.0.0:9121' + ``` + + 1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. + ## Advanced configuration Omnibus GitLab configures some things behind the curtains to make the sysadmins' |