From 71789468d306c7974fcf27442c83d7fc131e94fa Mon Sep 17 00:00:00 2001 From: Jacob Vosmaer Date: Mon, 15 Dec 2014 13:43:36 +0100 Subject: Add documentation for the Sidekiq MemoryKiller --- doc/operations/README.md | 3 +++ doc/operations/sidekiq_memory_killer.md | 38 +++++++++++++++++++++++++++++++++ doc/operations/sidekiq_restarter.md | 10 +++++++++ 3 files changed, 51 insertions(+) create mode 100644 doc/operations/README.md create mode 100644 doc/operations/sidekiq_memory_killer.md create mode 100644 doc/operations/sidekiq_restarter.md (limited to 'doc/operations') diff --git a/doc/operations/README.md b/doc/operations/README.md new file mode 100644 index 00000000000..31b1b583b0c --- /dev/null +++ b/doc/operations/README.md @@ -0,0 +1,3 @@ +# GitLab operations + +- [Sidekiq MemoryKiller](sidekiq_memory_killer.md) diff --git a/doc/operations/sidekiq_memory_killer.md b/doc/operations/sidekiq_memory_killer.md new file mode 100644 index 00000000000..867b01b0d5a --- /dev/null +++ b/doc/operations/sidekiq_memory_killer.md @@ -0,0 +1,38 @@ +# Sidekiq MemoryKiller + +The GitLab Rails application code suffers from memory leaks. For web requests +this problem is made manageable using +[unicorn-worker-killer](https://github.com/kzk/unicorn-worker-killer) which +restarts Unicorn worker processes in between requests when needed. The Sidekiq +MemoryKiller applies the same approach to the Sidekiq processes used by GitLab +to process background jobs. + +Unlike unicorn-worker-killer, which is enabled by default for all GitLab +installations since GitLab 6.4, the Sidekiq MemoryKiller is enabled by default +_only_ for Omnibus packages. The reason for this is that the MemoryKiller +relies on Runit to restart Sidekiq after a memory-induced shutdown and GitLab +installations from source do not all use Runit or an equivalent. + +With the default settings, the MemoryKiller will cause a Sidekiq restart no +more often than once every 15 minutes, with the restart causing about one +minute of delay for incoming background jobs. + +## Configuring the MemoryKiller + +The MemoryKiller is controlled using environment variables. + +- `SIDEKIQ_MEMORY_KILLER_MAX_RSS`: if this variable is set, and its value is + greater than 0, then after each Sidekiq job, the MemoryKiller will check the + RSS of the Sidekiq process that executed the job. If the RSS of the Sidekiq + process (expressed in kilobytes) exceeds SIDEKIQ_MEMORY_KILLER_MAX_RSS, a + delayed shutdown is triggered. The default value for Omnibus packages is set + [in the omnibus-gitlab + repository](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/attributes/default.rb). +- `SIDEKIQ_MEMORY_KILLER_GRACE_TIME`: defaults 900 seconds (15 minutes). When + a shutdown is triggered, the Sidekiq process will keep working normally for + another 15 minutes. +- `SIDEKIQ_MEMORY_KILLER_SHUTDOWN_WAIT`: defaults to 30 seconds. When the grace + time has expired, the MemoryKiller tells Sidekiq to stop accepting new jobs. + Existing jobs get 30 seconds to finish. After that, the MemoryKiller tells + Sidekiq to shut down, and an external supervision mechanism (e.g. Runit) must + restart Sidekiq. diff --git a/doc/operations/sidekiq_restarter.md b/doc/operations/sidekiq_restarter.md new file mode 100644 index 00000000000..ab28c9def18 --- /dev/null +++ b/doc/operations/sidekiq_restarter.md @@ -0,0 +1,10 @@ +# Sidekiq MemoryKiller + +The GitLab Rails application code suffers from memory leaks. For web requests +this problem is made manageable using +[unicorn-worker-killer](https://github.com/kzk/unicorn-worker-killer) which +restarts Unicorn worker processes in between requests when needed. The Sidekiq +MemoryKiller applies the same approach to the Sidekiq processes used by GitLab +to process background jobs. + + -- cgit v1.2.1