summaryrefslogtreecommitdiff
path: root/doc/administration
diff options
context:
space:
mode:
authorAchilleas Pipinellis <axilleas@axilleas.me>2016-07-19 14:27:50 +0300
committerGabriel Mazetto <gabriel@gitlab.com>2016-08-04 18:55:37 +0200
commitca7c6cc7d660a7c17ce5002ff3a9e94dcc0421cb (patch)
treecbe8494ccc0cccd9308ac34939e1f30a6af943c8 /doc/administration
parenta5c1e2e55e6345192cf01dc422c07b894195a0d2 (diff)
downloadgitlab-ce-ca7c6cc7d660a7c17ce5002ff3a9e94dcc0421cb.tar.gz
Refactor Redis HA docs to also include Sentinel
Diffstat (limited to 'doc/administration')
-rw-r--r--doc/administration/high_availability/redis.md347
1 files changed, 245 insertions, 102 deletions
diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md
index 1adc681e98b..8a6c5869e70 100644
--- a/doc/administration/high_availability/redis.md
+++ b/doc/administration/high_availability/redis.md
@@ -1,163 +1,300 @@
# Configuring Redis for GitLab HA
-You can choose to install and manage Redis yourself, or you can use GitLab
-Omnibus packages to help.
+You can choose to install and manage Redis yourself, or you can use the one
+that comes bundled with GitLab Omnibus packages.
-## Experimental Redis Sentinel support
+> **Note:** Redis does not require authentication by default. See
+ [Redis Security](http://redis.io/topics/security) documentation for more
+ information. We recommend using a combination of a Redis password and tight
+ firewall rules to secure your Redis service.
+
+## Configure your own Redis server
+
+If you're hosting GitLab on a cloud provider, you can optionally use a
+managed service for Redis. For example, AWS offers a managed ElastiCache service
+that runs Redis.
+
+## Configure Redis using Omnibus
+
+If you don't want to bother setting up your own Redis server, you can use the
+one bundled with Omnibus. In this case, you should disable all services except
+Redis.
+
+1. Download/install GitLab Omnibus using **steps 1 and 2** from
+ [GitLab downloads](https://about.gitlab.com/downloads). Do not complete other
+ steps on the download page.
+1. Create/edit `/etc/gitlab/gitlab.rb` and use the following configuration.
+ Be sure to change the `external_url` to match your eventual GitLab front-end
+ URL:
+
+ ```ruby
+ external_url 'https://gitlab.example.com'
+
+ # Disable all services except Redis
+ redis['enable'] = true
+ bootstrap['enable'] = false
+ nginx['enable'] = false
+ unicorn['enable'] = false
+ sidekiq['enable'] = false
+ postgresql['enable'] = false
+ gitlab_workhorse['enable'] = false
+ mailroom['enable'] = false
+
+ # Redis configuration
+ redis['port'] = 6379
+ redis['bind'] = '0.0.0.0'
+
+ # If you wish to use Redis authentication (recommended)
+ redis['password'] = 'Redis Password'
+ ```
+
+1. Run `sudo gitlab-ctl reconfigure` to install and configure PostgreSQL.
+
+ > **Note**: This `reconfigure` step will result in some errors.
+ That's OK - don't be alarmed.
+
+1. Run `touch /etc/gitlab/skip-auto-migrations` to prevent database migrations
+ from running on upgrade. Only the primary GitLab application server should
+ handle migrations.
-Since 8.10 release, you can configure a list of Redis Sentinel servers that
+## Experimental Redis sentinel support
+
+> [Introduced][ce-1877] in GitLab 8.10.
+
+Since GitLab 8.10, you can configure a list of Redis Sentinel servers that
will monitor a group of Redis servers to provide you with a standard failover
support.
-There is currently one exception to the Sentinel support: **mail_room**, the
-component that process incoming emails.
-
-It doesn't support Sentinel yet, but we hope to integrate a future release
-that does support it.
+There is currently one exception to the Sentinel support: `mail_room`, the
+component that processes incoming emails. It doesn't support Sentinel yet, but
+we hope to integrate a future release that does support it.
To get a better understanding on how to correctly setup Sentinel, please read
-[Redis Sentinel documentation](http://redis.io/topics/sentinel) first, as
-faling to configure it correctly can lead to data-loss.
+the [Redis Sentinel documentation](http://redis.io/topics/sentinel) first, as
+failing to configure it correctly can lead to data loss.
+
+The configuration consists of three parts:
-### Redis setup
+- Redis setup
+- Sentinel setup
+- GitLab setup
+
+Read carefully how to configure those components below.
+
+### Sentinel - Redis setup
You must have at least 2 Redis servers: 1 Master, 1 or more Slaves.
They should be configured the same way and with similar server specs, as
in a failover situation, any Slave can be elected as the new Master by
-the Sentinels servers.
+the Sentinel servers.
-In a minimal setup, the only required change for the slaves in `redis.conf`
-is the addition of a `slaveof` line pointing to the initial master like this:
+In a minimal setup, the only required change for the slaves in `redis.conf`
+is the addition of a `slaveof` line pointing to the initial master.
+You can increase the security by defining a `requirepass` configuration in
+the master, and `masterauth` in slaves.
-```conf
-slaveof 192.168.1.1 6379
-```
+---
-You can increase the security by defining a `requirepass` configuration in
-the master:
+**Configuring your own Redis server**
-```conf
-requirepass "<password>
-```
+1. Add to the slaves' `redis.conf`:
-and adding this line to all the slave servers:
+ ```conf
+ # IP and port of the master Redis server
+ slaveof 10.10.10.10 6379
+ ```
-```conf
-masterauth "<password>"
-```
+1. Optionally, set up password authentication for increased security.
+ Add the following to master's `redis.conf`:
+
+ ```conf
+ # Optional password authentication for increased security
+ requirepass "<password>"
+ ```
+
+1. Then add this line to all the slave servers' `redis.conf`:
-> **Note** This setup is not safe to be used by a machine accessible by the
-internet. Use it in combination with tight firewall rules.
+ ```conf
+ masterauth "<password>"
+ ```
+
+1. Restart the Redis services for the changes to take effect.
+
+---
+
+**Using Redis via Omnibus**
+
+1. Edit `/etc/gitlab/gitlab.rb`:
+
+ ```ruby
+ # IP and port of the master Redis server
+ gitlab['redis']['master_ip'] = '10.10.10.10'
+ gitlab['redis']['master_port'] = 6379
+ ```
+
+1. Optionally, set up password authentication for increased security.
+ Add the following to master's `/etc/gitlab/gitlab.rb`:
+
+ ```conf
+ # Optional password authentication for increased security
+ gitlab['redis']['password'] = '<password>'
+ ```
+
+1. Then add this line to all the slave servers' `redis.conf`:
+
+ ```conf
+ gitlab['redis']['master_password'] = '<password>'
+ ```
+
+1. Reconfigure the GitLab for the changes to take effect.
+
+---
+
+Now that the Redis servers are all set up, let's configure the Sentinel
+servers.
### Sentinel setup
-The support for Sentinel in ruby have some [caveats](https://github.com/redis/redis-rb/issues/531).
-While you can give any name for the `master-group-name` part of the
-configuration, as in this example:
+The support for Sentinel in Ruby has some [caveats](https://github.com/redis/redis-rb/issues/531).
+While you can give any name for the `master-group-name` part of the
+configuration, as in this example:
```conf
-sentinel monitor <master-group-name> <ip> <port> <quorum>`
+sentinel monitor <master-group-name> <ip> <port> <quorum>
```
-For it to work in ruby, you have to use the "hostname" of the master redis
-server otherwhise you will get an error message like this one:
-`Redis::CannotConnectError: No sentinels available.`.
-
+,for it to work in Ruby, you have to use the "hostname" of the master Redis
+server, otherwise you will get an error message like:
+`Redis::CannotConnectError: No sentinels available.`. Read
+[Sentinel troubleshooting](#sentinel-troubleshooting) for more information.
Here is an example configuration file (`sentinel.conf`) for a Sentinel node:
-
+
```conf
port 26379
sentinel monitor master-redis.example.com 10.10.10.10 6379 1
sentinel down-after-milliseconds master-redis.example.com 10000
sentinel config-epoch master-redis.example.com 0
-sentinel leader-epoch locmaster-redis.example.comalhost 0
+sentinel leader-epoch master-redis.example.com 0
```
-### GitLab setup
+---
+
+The final part is to inform the main GitLab application server of the Redis
+master and the new sentinels servers.
+
+### Sentinel - GitLab setup
You can enable or disable sentinel support at any time in new or existing
-installs. From the GitLab application perspective, all it requires is
+installations. From the GitLab application perspective, all it requires is
the correct credentials for the Master redis and for a few Sentinels nodes.
It doesn't require a list of all sentinel nodes, as in case of a failure,
the application will need to query only one of them.
-For a source based install, you must change `/home/git/gitlab/config/resque.yml`,
-following the example in `/home/git/gitlab/config/resque.yml.example` and
-uncommenting the sentinels line, changing to the correct server credentials,
-and resstart GitLab.
-
-For a Omnibus install you have to add/change this lines from the
-`/etc/gitlab/gitlab.rb` configuration file:
-
-```ruby
-gitlab['gitlab-rails']['redis_host'] = "master-redis.example.com"
-gitlab['gitlab-rails']['redis_port'] = 6379
-gitlab['gitlab-rails']['redis_password'] = "redis-secure-password-here"
-gitlab['gitlab-rails']['redis_socket'] = nil
-gitlab['gitlab-rails']['redis_sentinels'] = [
- {'host' => '10.10.10.1', 'port' => 26379},
- {'host' => '10.10.10.2', 'port' => 26379},
- {'host' => '10.10.10.3', 'port' => 26379}
-]
-```
+>**Note:**
+The following steps should be performed in the [GitLab application server](gitlab.md).
+
+**For source based installations**
-After the change run the reconfigure command:
+1. Edit `/home/git/gitlab/config/resque.yml` following the example in
+ `/home/git/gitlab/config/resque.yml.example`i, and uncomment the sentinels
+ line, changing to the correct server credentials.
+1. Restart GitLab for the changes to take effect.
-```bash
-sudo gitlab-ctl reconfigure
+**For Omnibus installations**
+1. Edit `/etc/gitlab/gitlab.rb` and add/change the following lines:
+
+ ```ruby
+ gitlab-rails['redis_host'] = "master-redis.example.com"
+ gitlab-rails['redis_port'] = 6379
+ gitlab-rails['redis_password'] = "redis-secure-password-here"
+ gitlab-rails['redis_socket'] = nil
+ gitlab-rails['redis_sentinels'] = [
+ {'host' => '10.10.10.1', 'port' => 26379},
+ {'host' => '10.10.10.2', 'port' => 26379},
+ {'host' => '10.10.10.3', 'port' => 26379}
+ ]
+ ```
+
+1. [Reconfigure] the GitLab for the changes to take effect.
+
+### Sentinel troubleshooting
+
+If you get an error like: `Redis::CannotConnectError: No sentinels available.`,
+there may be something wrong with your configuration files or it can be related
+to [this issue][gh-531] ([pull request][gh-534] that should make things better).
+
+It's a bit rigid the way you have to config `resque.yml` and `sentinel.conf`,
+otherwise `redis-rb` will not work properly.
+
+The hostname ('my-primary-redis') of the primary Redis server (`sentinel.conf`)
+**must** match the one configured in GitLab (`resque.yml` for source installations
+or `gitlab-rails['redis_*']` in Omnibus) and it must be valid ex:
+
+```conf
+# sentinel.conf:
+sentinel monitor my-primary-redis 10.10.10.10 6379 1
+sentinel down-after-milliseconds my-primary-redis 10000
+sentinel config-epoch my-primary-redis 0
+sentinel leader-epoch my-primary-redis 0
```
-## Configure your own Redis server
+```yaml
+# resque.yaml
+production:
+ url: redis://my-primary-redis:6378
+ sentinels:
+ -
+ host: slave1
+ port: 26380 # point to sentinel, not to redis port
+ -
+ host: slave2
+ port: 26381 # point to sentinel, not to redis port
+```
-If you're hosting GitLab on a cloud provider, you can optionally use a
-managed service for Redis. For example, AWS offers a managed ElastiCache service
-that runs Redis.
+When in doubt, please read [Redis Sentinel documentation](http://redis.io/topics/sentinel)
-> **Note:** Redis does not require authentication by default. See
- [Redis Security](http://redis.io/topics/security) documentation for more
- information. We recommend using a combination of a Redis password and tight
- firewall rules to secure your Redis service.
+---
-## Configure using Omnibus
+To make sure your configuration is correct:
-1. Download/install GitLab Omnibus using **steps 1 and 2** from
- [GitLab downloads](https://about.gitlab.com/downloads). Do not complete other
- steps on the download page.
-1. Create/edit `/etc/gitlab/gitlab.rb` and use the following configuration.
- Be sure to change the `external_url` to match your eventual GitLab front-end
- URL.
+1. SSH into your GitLab application server
+1. Enter the Rails console:
+
+ ```
+ # For Omnibus installations
+ sudo gitlab-rails console
+
+ # For source installations
+ sudo -u git rails console RAILS_ENV=production
+ ```
+
+1. Run in the console:
```ruby
- external_url 'https://gitlab.example.com'
-
- # Disable all components except Redis
- redis['enable'] = true
- bootstrap['enable'] = false
- nginx['enable'] = false
- unicorn['enable'] = false
- sidekiq['enable'] = false
- postgresql['enable'] = false
- gitlab_workhorse['enable'] = false
- mailroom['enable'] = false
-
- # Redis configuration
- redis['port'] = 6379
- redis['bind'] = '0.0.0.0'
-
- # If you wish to use Redis authentication (recommended)
- redis['password'] = 'Redis Password'
+ redis = Redis.new(Gitlab::Redis.params)
+ redis.info
```
-1. Run `sudo gitlab-ctl reconfigure` to install and configure PostgreSQL.
+ Keep this screen open and try to simulate a failover below.
- > **Note**: This `reconfigure` step will result in some errors.
- That's OK - don't be alarmed.
-1. Run `touch /etc/gitlab/skip-auto-migrations` to prevent database migrations
- from running on upgrade. Only the primary GitLab application server should
- handle migrations.
+1. To simulate a failover on master Redis, SSH into the Redis server and run:
+
+ ```bash
+ # port must match your master redis port
+ redis-cli -h localhost -p 6379 DEBUG sleep 60
+ ```
+
+1. Then back in the Rails console from the first step, run:
+
+ ```
+ redis.info
+ ```
+
+ You should see a different port after a few seconds delay
+ (the failover/reconnect time).
---
@@ -167,3 +304,9 @@ Read more on high-availability configuration:
1. [Configure NFS](nfs.md)
1. [Configure the GitLab application servers](gitlab.md)
1. [Configure the load balancers](load_balancer.md)
+
+[ce-1877]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/1877
+[restart]: ../restart_gitlab.md#installations-from-source
+[reconfigure]: ../restart_gitlab.md#omnibus-gitlab-reconfigure
+[gh-531]: https://github.com/redis/redis-rb/issues/531
+[gh-534]: https://github.com/redis/redis-rb/issues/534