diff options
Diffstat (limited to 'doc/development/redis.md')
| -rw-r--r-- | doc/development/redis.md | 147 |
1 files changed, 140 insertions, 7 deletions
diff --git a/doc/development/redis.md b/doc/development/redis.md index d5d42a3869e..d205082b9c6 100644 --- a/doc/development/redis.md +++ b/doc/development/redis.md @@ -1,10 +1,19 @@ # Redis guidelines -GitLab uses [Redis](https://redis.io) for three distinct purposes: +GitLab uses [Redis](https://redis.io) for the following distinct purposes: -- Caching via `Rails.cache`. +- Caching (mostly via `Rails.cache`). - As a job processing queue with [Sidekiq](sidekiq_style_guide.md). - To manage the shared application state. +- As a Pub/Sub queue backend for ActionCable. + +In most environments (including the GDK), all of these point to the same +Redis instance. + +On GitLab.com, we use [separate Redis +instances](../administration/redis/replication_and_failover.md#running-multiple-redis-clusters). +(We do not currently use [ActionCable on +GitLab.com](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/228)). Every application process is configured to use the same Redis servers, so they can be used for inter-process communication in cases where [PostgreSQL](sql.md) @@ -21,11 +30,11 @@ to key names to avoid collisions. Typically we use colon-separated elements to provide a semblance of structure at application level. An example might be `projects:1:somekey`. -Although we split our Redis usage into three separate purposes, and those may -map to separate Redis servers in a [Highly Available](../administration/high_availability/redis.md) -configuration, the default Omnibus and GDK setups share a single Redis server. -This means that keys should **always** be globally unique across the three -purposes. +Although we split our Redis usage by purpose into distinct categories, and +those may map to separate Redis servers in a Highly Available +configuration like GitLab.com, the default Omnibus and GDK setups share +a single Redis server. This means that keys should **always** be +globally unique across all categories. It is usually better to use immutable identifiers - project ID rather than full path, for instance - in Redis key names. If full path is used, the key will @@ -56,3 +65,127 @@ Currently, we validate this in the development and test environments with the [`RedisClusterValidator`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/instrumentation/redis_cluster_validator.rb), which is enabled for the `cache` and `shared_state` [Redis instances](https://docs.gitlab.com/omnibus/settings/redis.html#running-with-multiple-redis-instances).. + +## Redis in structured logging + +Our [structured logging](logging.md#use-structured-json-logging) for web +requests and Sidekiq jobs contains fields for the duration, call count, +bytes written, and bytes read per Redis instance, along with a total for +all Redis instances. For a particular request, this might look like: + +| Field | Value | +| --- | --- | +| `json.queue_duration_s` | 0.01 | +| `json.redis_cache_calls` | 1 | +| `json.redis_cache_duration_s` | 0 | +| `json.redis_cache_read_bytes` | 109 | +| `json.redis_cache_write_bytes` | 49 | +| `json.redis_calls` | 2 | +| `json.redis_duration_s` | 0.001 | +| `json.redis_read_bytes` | 111 | +| `json.redis_shared_state_calls` | 1 | +| `json.redis_shared_state_duration_s` | 0 | +| `json.redis_shared_state_read_bytes` | 2 | +| `json.redis_shared_state_write_bytes` | 206 | +| `json.redis_write_bytes` | 255 | + +As all of these fields are indexed, it is then straightforward to +investigate Redis usage in production. For instance, to find the +requests that read the most data from the cache, we can just sort by +`redis_cache_read_bytes` in descending order. + +### The slow log + +On GitLab.com, entries from the [Redis +slow log](https://redis.io/commands/slowlog) are available in the +`pubsub-redis-inf-gprd*` index with the [`redis.slowlog` +tag](https://log.gprd.gitlab.net/app/kibana#/discover?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-1d,to:now))&_a=(columns:!(json.type,json.command,json.exec_time),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:AWSQX_Vf93rHTYrsexmk,key:json.tag,negate:!f,params:(query:redis.slowlog),type:phrase),query:(match:(json.tag:(query:redis.slowlog,type:phrase))))),index:AWSQX_Vf93rHTYrsexmk)). +This shows commands that have taken a long time and may be a performance +concern. + +The +[fluent-plugin-redis-slowlog](https://gitlab.com/gitlab-org/fluent-plugin-redis-slowlog) +project is responsible for taking the slowlog entries from Redis and +passing to fluentd (and ultimately Elasticsearch). + +## Analyzing the entire keyspace + +The [Redis Keyspace +Analyzer](https://gitlab.com/gitlab-com/gl-infra/redis-keyspace-analyzer) +project contains tools for dumping the full key list and memory usage of a Redis +instance, and then analyzing those lists while elimating potentially sensitive +data from the results. It can be used to find the most frequent key patterns, or +those that use the most memory. + +Currently this is not run automatically for the GitLab.com Redis instances, but +is run manually on an as-needed basis. + +## Utility classes + +We have some extra classes to help with specific use cases. These are +mostly for fine-grained control of Redis usage, so they wouldn't be used +in combination with the `Rails.cache` wrapper: we'd either use +`Rails.cache` or these classes and literal Redis commands. + +`Rails.cache` or these classes and literal Redis commands. We prefer +using `Rails.cache` so we can reap the benefits of future optimizations +done to Rails. It is worth noting that Ruby objects are +[marshalled](https://github.com/rails/rails/blob/v6.0.3.1/activesupport/lib/active_support/cache/redis_cache_store.rb#L447) +when written to Redis, so we need to pay attention to not to store huge +objects, or untrusted user input. + +Typically we would only use these classes when at least one of the +following is true: + +1. We want to manipulate data on a non-cache Redis instance. +1. `Rails.cache` does not support the operations we want to perform. + +### `Gitlab::Redis::{Cache,SharedState,Queues}` + +These classes wrap the Redis instances (using +[`Gitlab::Redis::Wrapper`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/redis/wrapper.rb)) +to make it convenient to work with them directly. The typical use is to +call `.with` on the class, which takes a block that yields the Redis +connection. For example: + +```ruby +# Get the value of `key` from the shared state (persistent) Redis +Gitlab::Redis::SharedState.with { |redis| redis.get(key) } + +# Check if `value` is a member of the set `key` +Gitlab::Redis::Cache.with { |redis| redis.sismember(key, value) } +``` + +### `Gitlab::Redis::Boolean` + +In Redis, every value is a string. +[`Gitlab::Redis::Boolean`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/redis/boolean.rb) +makes sure that booleans are encoded and decoded consistently. + +### `Gitlab::Redis::HLL` + +The Redis [`PFCOUNT`](https://redis.io/commands/pfcount), +[`PFADD`](https://redis.io/commands/pfadd), and +[`PFMERGE`](https://redis.io/commands/pfmergge) commands operate on +HyperLogLogs, a data structure that allows estimating the number of unique +elements with low memory usage. (In addition to the `PFCOUNT` documentation, +Thoughtbot's article on [HyperLogLogs in +Redis](https://thoughtbot.com/blog/hyperloglogs-in-redis) provides a good +background here.) + +[`Gitlab::Redis::HLL`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/redis/hll.rb) +provides a convenient interface for adding and counting values in HyperLogLogs. + +### `Gitlab::SetCache` + +For cases where we need to efficiently check the whether an item is in a group +of items, we can use a Redis set. +[`Gitlab::SetCache`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/set_cache.rb) +provides an `#include?` method that will use the +[`SISMEMBER`](https://redis.io/commands/sismember) command, as well as `#read` +to fetch all entries in the set. + +This is used by the +[`RepositorySetCache`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/repository_set_cache.rb) +to provide a convenient way to use sets to cache repository data like branch +names. |