Support multiple Redis instances based on queue type

10 files changed, 272 insertions, 23 deletions

diff --git a/config/README.md b/config/README.md
index 0a5ea2424e0..2778d0d4f02 100644
--- a/config/README.md
+++ b/config/README.md
@@ -19,4 +19,132 @@ an ERB file and then loads the resulting YML as its configuration.
 
 This file is called `resque.yml` for historical reasons. We are **NOT**
 using Resque at the moment. It is used to specify Redis configuration
-values instead.
+values when a single database instance of Redis is desired.
+
+# Advanced Redis configuration files
+
+In more advanced configurations of Redis key-value storage, it is desirable
+to separate the keys by lifecycle and intended use to ease provisioning and
+management of scalable Redis clusters.
+
+These settings provide routing and other configuration data (such as sentinel,
+persistence policies, and other Redis customization) for connections
+to Redis single instances, Redis sentinel, and Redis clusters.
+
+If desired, the routing URL provided by these settings can be used with:
+1. Unix Socket
+    1. named socket for each Redis instance desired.
+    2. `database number` for each Redis instance desired.
+2. TCP Socket
+    1. `host name` or IP for each Redis instance desired
+    2. TCP port number for each Redis instance desired
+    3. `database number` for each Redis instance desired
+    
+## Example URL attribute formats for GitLab Redis `.yml` configuration files
+* Unix Socket, default Redis database (0)
+    * `url: unix:/path/to/redis.sock`
+    * `url: unix:/path/to/redis.sock?db=`
+* Unix Socket, Redis database 44
+    * `url: unix:/path/to/redis.sock?db=44`
+    * `url: unix:/path/to/redis.sock?extra=foo&db=44`
+* TCP Socket for Redis on localhost, port 6379, database 33
+    * `url: redis://:mynewpassword@localhost:6379/33`
+* TCP Socket for Redis on remote host `myserver`, port 6379, database 33
+    * `url: redis://:mynewpassword@myserver:6379/33`
+
+## redis.cache.yml
+
+If configured, `redis.cache.yml` overrides the
+`resque.yml` settings to configure the Redis database instance
+used for `Rails.cache` and other volatile non-persistent data which enhances
+the performance of GitLab.
+Settings here can be overridden by the environment variable
+`GITLAB_REDIS_CACHE_CONFIG_FILE` which provides
+an alternate location for configuration settings.
+
+The order of precedence for the URL used to connect to the Redis instance
+used for `cache` is:
+1. URL from a configuration file pointed to by the
+`GITLAB_REDIS_CACHE_CONFIG_FILE` environment variable
+2. URL from `redis.cache.yml`
+3. URL from a configuration file pointed to by the
+`GITLAB_REDIS_CONFIG_FILE` environment variable
+4. URL from `resque.yml`
+5. `redis://localhost:6380`
+
+The order of precedence for all other configuration settings for `cache`
+are selected from only the first of the following files found (if a setting
+is not provided in an earlier file, the remainder of the files are not
+searched):
+1. the configuration file pointed to by the
+`GITLAB_REDIS_CACHE_CONFIG_FILE` environment variable
+2. the configuration file `redis.cache.yml`
+3. the configuration file pointed to by the
+`GITLAB_REDIS_CONFIG_FILE` environment variable
+4. the configuration file `resque.yml`
+
+## redis.queues.yml
+
+If configured, `redis.queues.yml` overrides the
+`resque.yml` settings to configure the Redis database instance
+used for clients of `::Gitlab::Redis::Queues`.
+These queues are intended to be the foundation
+of reliable inter-process communication between modules, whether on the same
+host node, or within a cluster.   The primary clients of the queues are
+SideKiq, Mailroom, CI Runner, Workhorse, and push services.  Settings here can
+be overridden by the environment variable
+`GITLAB_REDIS_QUEUES_CONFIG_FILE` which provides an alternate location for
+configuration settings.
+
+The order of precedence for the URL used to connect to the Redis instance
+used for `queues` is:
+1. URL from a configuration file pointed to by the
+`GITLAB_REDIS_QUEUES_CONFIG_FILE` environment variable
+2. URL from `redis.queues.yml`
+3. URL from a configuration file pointed to by the
+`GITLAB_REDIS_CONFIG_FILE` environment variable
+4. URL from `resque.yml`
+5. `redis://localhost:6381`
+
+The order of precedence for all other configuration settings for `queues`
+are selected from only the first of the following files found (if a setting
+is not provided in an earlier file, the remainder of the files are not
+searched):
+1. the configuration file pointed to by the
+`GITLAB_REDIS_QUEUES_CONFIG_FILE` environment variable
+2. the configuration file `redis.queues.yml`
+3. the configuration file pointed to by the
+`GITLAB_REDIS_CONFIG_FILE` environment variable
+4. the configuration file `resque.yml`
+
+## redis.shared_state.yml
+
+If configured, `redis.shared_state.yml` overrides the
+`resque.yml` settings to configure the Redis database instance
+used for clients of `::Gitlab::Redis::SharedState` such as session state,
+and rate limiting.
+Settings here can be overridden by the environment variable
+`GITLAB_REDIS_SHARED_STATE_CONFIG_FILE` which provides
+an alternate location for configuration settings.
+
+The order of precedence for the URL used to connect to the Redis instance
+used for `shared_state` is:
+1. URL from a configuration file pointed to by the
+`GITLAB_REDIS_SHARED_STATE_CONFIG_FILE` environment variable
+2. URL from `redis.shared_state.yml`
+3. URL from a configuration file pointed to by the
+`GITLAB_REDIS_CONFIG_FILE` environment variable
+4. URL from `resque.yml`
+5. `redis://localhost:6382`
+
+The order of precedence for all other configuration settings for `shared_state`
+are selected from only the first of the following files found (if a setting
+is not provided in an earlier file, the remainder of the files are not
+searched):
+1. the configuration file pointed to by the
+`GITLAB_REDIS_SHARED_STATE_CONFIG_FILE` environment variable
+2. the configuration file `redis.shared_state.yml`
+3. the configuration file pointed to by the
+`GITLAB_REDIS_CONFIG_FILE` environment variable
+4. the configuration file `resque.yml`
+
diff --git a/config/application.rb b/config/application.rb
index 2f4e2624195..1c13cc81270 100644
--- a/config/application.rb
+++ b/config/application.rb
@@ -6,7 +6,9 @@ Bundler.require(:default, Rails.env)
 
 module Gitlab
   class Application < Rails::Application
-    require_dependency Rails.root.join('lib/gitlab/redis')
+    require_dependency Rails.root.join('lib/gitlab/redis/cache')
+    require_dependency Rails.root.join('lib/gitlab/redis/queues')
+    require_dependency Rails.root.join('lib/gitlab/redis/shared_state')
     require_dependency Rails.root.join('lib/gitlab/request_context')
 
     # Settings in config/environments/* take precedence over those specified here.
@@ -142,15 +144,15 @@ module Gitlab
       end
     end
 
-    # Use Redis caching across all environments
-    redis_config_hash = Gitlab::Redis.params
-    redis_config_hash[:namespace] = Gitlab::Redis::CACHE_NAMESPACE
-    redis_config_hash[:expires_in] = 2.weeks # Cache should not grow forever
+    # Use caching across all environments
+    caching_config_hash = Gitlab::Redis::Cache.params
+    caching_config_hash[:namespace] = Gitlab::Redis::Cache::CACHE_NAMESPACE
+    caching_config_hash[:expires_in] = 2.weeks # Cache should not grow forever
     if Sidekiq.server? # threaded context
-      redis_config_hash[:pool_size] = Sidekiq.options[:concurrency] + 5
-      redis_config_hash[:pool_timeout] = 1
+      caching_config_hash[:pool_size] = Sidekiq.options[:concurrency] + 5
+      caching_config_hash[:pool_timeout] = 1
     end
-    config.cache_store = :redis_store, redis_config_hash
+    config.cache_store = :redis_store, caching_config_hash
 
     config.active_record.raise_in_transactional_callbacks = true
 
diff --git a/config/initializers/7_redis.rb b/config/initializers/7_redis.rb
index ae2ca258df1..af4967521b8 100644
--- a/config/initializers/7_redis.rb
+++ b/config/initializers/7_redis.rb
@@ -1,3 +1,8 @@
-# Make sure we initialize a Redis connection pool before Sidekiq starts
-# multi-threaded execution.
-Gitlab::Redis.with { nil }
+# Make sure we initialize a Redis connection pool before multi-threaded
+# execution starts by
+# 1. Sidekiq
+# 2. Rails.cache
+# 3. HTTP clients
+Gitlab::Redis::Cache.with { nil }
+Gitlab::Redis::Queues.with { nil }
+Gitlab::Redis::SharedState.with { nil }
diff --git a/config/initializers/peek.rb b/config/initializers/peek.rb
index 65432caac2a..da8282ec924 100644
--- a/config/initializers/peek.rb
+++ b/config/initializers/peek.rb
@@ -1,4 +1,4 @@
-Rails.application.config.peek.adapter = :redis, { client: ::Redis.new(Gitlab::Redis.params) }
+Rails.application.config.peek.adapter = :redis, { client: ::Redis.new(Gitlab::Redis::Cache.params) }
 
 Peek.into Peek::Views::Host
 Peek.into Peek::Views::PerformanceBar
diff --git a/config/initializers/session_store.rb b/config/initializers/session_store.rb
index 8919f7640fe..e8213ac8ba4 100644
--- a/config/initializers/session_store.rb
+++ b/config/initializers/session_store.rb
@@ -19,12 +19,12 @@ cookie_key = if Rails.env.development?
 if Rails.env.test?
   Gitlab::Application.config.session_store :cookie_store, key: "_gitlab_session"
 else
-  redis_config = Gitlab::Redis.params
-  redis_config[:namespace] = Gitlab::Redis::SESSION_NAMESPACE
+  sessions_config = Gitlab::Redis::SharedState.params
+  sessions_config[:namespace] = Gitlab::Redis::SharedState::SESSION_NAMESPACE
 
   Gitlab::Application.config.session_store(
     :redis_store, # Using the cookie_store would enable session replay attacks.
-    servers: redis_config,
+    servers: sessions_config,
     key: cookie_key,
     secure: Gitlab.config.gitlab.https,
     httponly: true,
diff --git a/config/initializers/sidekiq.rb b/config/initializers/sidekiq.rb
index 3be4cd797aa..a1cc9655319 100644
--- a/config/initializers/sidekiq.rb
+++ b/config/initializers/sidekiq.rb
@@ -1,12 +1,12 @@
-# Custom Redis configuration
-redis_config_hash = Gitlab::Redis.params
-redis_config_hash[:namespace] = Gitlab::Redis::SIDEKIQ_NAMESPACE
+# Custom Queues configuration
+queues_config_hash = Gitlab::Redis::Queues.params
+queues_config_hash[:namespace] = Gitlab::Redis::Queues::SIDEKIQ_NAMESPACE
 
 # Default is to retry 25 times with exponential backoff. That's too much.
 Sidekiq.default_worker_options = { retry: 3 }
 
 Sidekiq.configure_server do |config|
-  config.redis = redis_config_hash
+  config.redis = queues_config_hash
 
   config.server_middleware do |chain|
     chain.add Gitlab::SidekiqMiddleware::ArgumentsLogger if ENV['SIDEKIQ_LOG_ARGUMENTS']
@@ -54,7 +54,7 @@ Sidekiq.configure_server do |config|
 end
 
 Sidekiq.configure_client do |config|
-  config.redis = redis_config_hash
+  config.redis = queues_config_hash
 
   config.client_middleware do |chain|
     chain.add Gitlab::SidekiqStatus::ClientMiddleware
diff --git a/config/mail_room.yml b/config/mail_room.yml
index 88d93d4bc6b..c3a5be8d38c 100644
--- a/config/mail_room.yml
+++ b/config/mail_room.yml
@@ -21,7 +21,7 @@
       :delivery_method: sidekiq
       :delivery_options:
         :redis_url: <%= config[:redis_url].to_json %>
-        :namespace: <%= Gitlab::Redis::SIDEKIQ_NAMESPACE %>
+        :namespace: <%= Gitlab::Redis::Queues::SIDEKIQ_NAMESPACE %>
         :queue: email_receiver
         :worker: EmailReceiverWorker
         <% if config[:sentinels] %>
@@ -36,7 +36,7 @@
       :arbitration_method: redis
       :arbitration_options:
         :redis_url: <%= config[:redis_url].to_json %>
-        :namespace: <%= Gitlab::Redis::MAILROOM_NAMESPACE %>
+        :namespace: <%= Gitlab::Redis::Queues::MAILROOM_NAMESPACE %>
         <% if config[:sentinels] %>
         :sentinels:
           <% config[:sentinels].each do |sentinel| %>
diff --git a/config/redis.cache.yml.example b/config/redis.cache.yml.example
new file mode 100644
index 00000000000..27478f0a93e
--- /dev/null
+++ b/config/redis.cache.yml.example
@@ -0,0 +1,38 @@
+# If you change this file in a Merge Request, please also create
+# a Merge Request on https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests
+#
+development:
+  url: redis://localhost:6379/10
+  #
+  # url: redis://localhost:6380
+  # sentinels:
+  #   -
+  #     host: localhost
+  #     port: 26380 # point to sentinel, not to redis port
+  #   -
+  #     host: slave2
+  #     port: 26380 # point to sentinel, not to redis port
+test:
+  url: redis://localhost:6379/10
+  #
+  # url: redis://localhost:6380
+production:
+  # Redis (single instance)
+  url: unix:/var/run/redis/redis.cache.sock
+  ##
+  # Redis + Sentinel (for HA)
+  #
+  # Please read instructions carefully before using it as you may lose data:
+  # http://redis.io/topics/sentinel
+  #
+  # You must specify a list of a few sentinels that will handle client connection
+  # please read here for more information: https://docs.gitlab.com/ce/administration/high_availability/redis.html
+  ##
+  # url: redis://master:6380
+  # sentinels:
+  #   -
+  #     host: slave1
+  #     port: 26380 # point to sentinel, not to redis port
+  #   -
+  #     host: slave2
+  #     port: 26380 # point to sentinel, not to redis port
diff --git a/config/redis.queues.yml.example b/config/redis.queues.yml.example
new file mode 100644
index 00000000000..dab1f26b096
--- /dev/null
+++ b/config/redis.queues.yml.example
@@ -0,0 +1,38 @@
+# If you change this file in a Merge Request, please also create
+# a Merge Request on https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests
+#
+development:
+  url: redis://localhost:6379/11
+  #
+  # url: redis://localhost:6381
+  # sentinels:
+  #   -
+  #     host: localhost
+  #     port: 26381 # point to sentinel, not to redis port
+  #   -
+  #     host: slave2
+  #     port: 26381 # point to sentinel, not to redis port
+test:
+  url: redis://localhost:6379/11
+  #
+  # url: redis://localhost:6381
+production:
+  # Redis (single instance)
+  url: unix:/var/run/redis/redis.queues.sock
+  ##
+  # Redis + Sentinel (for HA)
+  #
+  # Please read instructions carefully before using it as you may lose data:
+  # http://redis.io/topics/sentinel
+  #
+  # You must specify a list of a few sentinels that will handle client connection
+  # please read here for more information: https://docs.gitlab.com/ce/administration/high_availability/redis.html
+  ##
+  # url: redis://master:6381
+  # sentinels:
+  #   -
+  #     host: slave1
+  #     port: 26381 # point to sentinel, not to redis port
+  #   -
+  #     host: slave2
+  #     port: 26381 # point to sentinel, not to redis port
diff --git a/config/redis.shared_state.yml.example b/config/redis.shared_state.yml.example
new file mode 100644
index 00000000000..9371e3619b7
--- /dev/null
+++ b/config/redis.shared_state.yml.example
@@ -0,0 +1,38 @@
+# If you change this file in a Merge Request, please also create
+# a Merge Request on https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests
+#
+development:
+  url: redis://localhost:6379/12
+  #
+  # url: redis://localhost:6382
+  # sentinels:
+  #   -
+  #     host: localhost
+  #     port: 26382 # point to sentinel, not to redis port
+  #   -
+  #     host: slave2
+  #     port: 26382 # point to sentinel, not to redis port
+test:
+  url: redis://localhost:6379/12
+  #
+  # url: redis://localhost:6382
+production:
+  # Redis (single instance)
+  url: unix:/var/run/redis/redis.shared_state.sock
+  ##
+  # Redis + Sentinel (for HA)
+  #
+  # Please read instructions carefully before using it as you may lose data:
+  # http://redis.io/topics/sentinel
+  #
+  # You must specify a list of a few sentinels that will handle client connection
+  # please read here for more information: https://docs.gitlab.com/ce/administration/high_availability/redis.html
+  ##
+  # url: redis://master:6382
+  # sentinels:
+  #   -
+  #     host: slave1
+  #     port: 26382 # point to sentinel, not to redis port
+  #   -
+  #     host: slave2
+  #     port: 26382 # point to sentinel, not to redis port