3 files changed, 94 insertions, 0 deletions
diff --git a/doc/operations/README.md b/doc/operations/README.md
new file mode 100644
index 00000000000..f1456c6c8e2
--- /dev/null
+++ b/doc/operations/README.md
@@ -0,0 +1,4 @@
+# GitLab operations
+
+- [Sidekiq MemoryKiller](sidekiq_memory_killer.md)
+- [Cleaning up Redis sessions](cleaning_up_redis_sessions.md)
diff --git a/doc/operations/cleaning_up_redis_sessions.md b/doc/operations/cleaning_up_redis_sessions.md
new file mode 100644
index 00000000000..93521e976d5
--- /dev/null
+++ b/doc/operations/cleaning_up_redis_sessions.md
@@ -0,0 +1,52 @@
+# Cleaning up stale Redis sessions
+
+Since version 6.2, GitLab stores web user sessions as key-value pairs in Redis.
+Prior to GitLab 7.3, user sessions did not automatically expire from Redis. If
+you have been running a large GitLab server (thousands of users) since before
+GitLab 7.3 we recommend cleaning up stale sessions to compact the Redis
+database after you upgrade to GitLab 7.3. You can also perform a cleanup while
+still running GitLab 7.2 or older, but in that case new stale sessions will
+start building up again after you clean up.
+
+In GitLab versions prior to 7.3.0, the session keys in Redis are 16-byte
+hexadecimal values such as '976aa289e2189b17d7ef525a6702ace9'. Starting with
+GitLab 7.3.0, the keys are
+prefixed with 'session:gitlab:', so they would look like
+'session:gitlab:976aa289e2189b17d7ef525a6702ace9'. Below we describe how to
+remove the keys in the old format.
+
+First we define a shell function with the proper Redis connection details.
+
+```
+rcli() {
+  # This example works for Omnibus installations of GitLab 7.3 or newer. For an
+  # installation from source you will have to change the socket path and the
+  # path to redis-cli.
+  sudo /opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket "$@"
+}
+
+# test the new shell function; the response should be PONG
+rcli ping
+```
+
+Now we do a search to see if there are any session keys in the old format for
+us to clean up.
+
+```
+# returns the number of old-format session keys in Redis
+rcli keys '*' | grep '^[a-f0-9]\{32\}$' | wc -l
+```
+
+If the number is larger than zero, you can proceed to expire the keys from
+Redis. If the number is zero there is nothing to clean up.
+
+```
+# Tell Redis to expire each matched key after 600 seconds.
+rcli keys '*' | grep '^[a-f0-9]\{32\}$' | awk '{ print "expire", $0, 600 }' | rcli
+# This will print '(integer) 1' for each key that gets expired.
+```
+
+Over the next 15 minutes (10 minutes expiry time plus 5 minutes Redis
+background save interval) your Redis database will be compacted. If you are
+still using GitLab 7.2, users who are not clicking around in GitLab during the
+10 minute expiry window will be signed out of GitLab.
diff --git a/doc/operations/sidekiq_memory_killer.md b/doc/operations/sidekiq_memory_killer.md
new file mode 100644
index 00000000000..867b01b0d5a
--- /dev/null
+++ b/doc/operations/sidekiq_memory_killer.md
@@ -0,0 +1,38 @@
+# Sidekiq MemoryKiller
+
+The GitLab Rails application code suffers from memory leaks. For web requests
+this problem is made manageable using
+[unicorn-worker-killer](https://github.com/kzk/unicorn-worker-killer) which
+restarts Unicorn worker processes in between requests when needed. The Sidekiq
+MemoryKiller applies the same approach to the Sidekiq processes used by GitLab
+to process background jobs.
+
+Unlike unicorn-worker-killer, which is enabled by default for all GitLab
+installations since GitLab 6.4, the Sidekiq MemoryKiller is enabled by default
+_only_ for Omnibus packages. The reason for this is that the MemoryKiller
+relies on Runit to restart Sidekiq after a memory-induced shutdown and GitLab
+installations from source do not all use Runit or an equivalent.
+
+With the default settings, the MemoryKiller will cause a Sidekiq restart no
+more often than once every 15 minutes, with the restart causing about one
+minute of delay for incoming background jobs.
+
+## Configuring the MemoryKiller
+
+The MemoryKiller is controlled using environment variables.
+
+- `SIDEKIQ_MEMORY_KILLER_MAX_RSS`: if this variable is set, and its value is
+  greater than 0, then after each Sidekiq job, the MemoryKiller will check the
+  RSS of the Sidekiq process that executed the job. If the RSS of the Sidekiq
+  process (expressed in kilobytes) exceeds SIDEKIQ_MEMORY_KILLER_MAX_RSS, a
+  delayed shutdown is triggered. The default value for Omnibus packages is set
+  [in the omnibus-gitlab
+  repository](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/attributes/default.rb).
+- `SIDEKIQ_MEMORY_KILLER_GRACE_TIME`: defaults 900 seconds (15 minutes). When
+  a shutdown is triggered, the Sidekiq process will keep working normally for
+  another 15 minutes.
+- `SIDEKIQ_MEMORY_KILLER_SHUTDOWN_WAIT`: defaults to 30 seconds. When the grace
+  time has expired, the MemoryKiller tells Sidekiq to stop accepting new jobs.
+  Existing jobs get 30 seconds to finish. After that, the MemoryKiller tells
+  Sidekiq to shut down, and an external supervision mechanism (e.g. Runit) must
+  restart Sidekiq.