diff options
author | Dmitriy Zaporozhets <dzaporozhets@gitlab.com> | 2014-12-16 11:23:52 +0000 |
---|---|---|
committer | Dmitriy Zaporozhets <dzaporozhets@gitlab.com> | 2014-12-16 11:23:52 +0000 |
commit | 8d71e4e0d0868de0659047f18cad08924a3ba49e (patch) | |
tree | b88d7e53b360b224a53f9b229a7ae9130e72e8fd | |
parent | f023db2b474ab53056bdff3d01675fb6388265cd (diff) | |
parent | d117927131351e319db947e83ef4b9ec228d5030 (diff) | |
download | gitlab-ce-8d71e4e0d0868de0659047f18cad08924a3ba49e.tar.gz |
Merge branch 'doc/sidekiq_memory_killer' into 'master'
Add documentation for the Sidekiq MemoryKiller
We already merged the code, and we are already using it on gitlab.com. This MR adds documentation.
See merge request !1325
-rw-r--r-- | doc/README.md | 1 | ||||
-rw-r--r-- | doc/operations/README.md | 3 | ||||
-rw-r--r-- | doc/operations/sidekiq_memory_killer.md | 38 |
3 files changed, 42 insertions, 0 deletions
diff --git a/doc/README.md b/doc/README.md index 896224fe930..3c8f8ad3d03 100644 --- a/doc/README.md +++ b/doc/README.md @@ -23,6 +23,7 @@ - [Welcome message](customization/welcome_message.md) Add a custom welcome message to the sign-in page. - [Issue closing](customization/issue_closing.md) Customize how to close an issue from commit messages. - [Libravatar](customization/libravatar.md) Use Libravatar for user avatars. +- [Operations](operations/README.md) Keeping GitLab up and running ## Contributor documentation diff --git a/doc/operations/README.md b/doc/operations/README.md new file mode 100644 index 00000000000..31b1b583b0c --- /dev/null +++ b/doc/operations/README.md @@ -0,0 +1,3 @@ +# GitLab operations + +- [Sidekiq MemoryKiller](sidekiq_memory_killer.md) diff --git a/doc/operations/sidekiq_memory_killer.md b/doc/operations/sidekiq_memory_killer.md new file mode 100644 index 00000000000..867b01b0d5a --- /dev/null +++ b/doc/operations/sidekiq_memory_killer.md @@ -0,0 +1,38 @@ +# Sidekiq MemoryKiller + +The GitLab Rails application code suffers from memory leaks. For web requests +this problem is made manageable using +[unicorn-worker-killer](https://github.com/kzk/unicorn-worker-killer) which +restarts Unicorn worker processes in between requests when needed. The Sidekiq +MemoryKiller applies the same approach to the Sidekiq processes used by GitLab +to process background jobs. + +Unlike unicorn-worker-killer, which is enabled by default for all GitLab +installations since GitLab 6.4, the Sidekiq MemoryKiller is enabled by default +_only_ for Omnibus packages. The reason for this is that the MemoryKiller +relies on Runit to restart Sidekiq after a memory-induced shutdown and GitLab +installations from source do not all use Runit or an equivalent. + +With the default settings, the MemoryKiller will cause a Sidekiq restart no +more often than once every 15 minutes, with the restart causing about one +minute of delay for incoming background jobs. + +## Configuring the MemoryKiller + +The MemoryKiller is controlled using environment variables. + +- `SIDEKIQ_MEMORY_KILLER_MAX_RSS`: if this variable is set, and its value is + greater than 0, then after each Sidekiq job, the MemoryKiller will check the + RSS of the Sidekiq process that executed the job. If the RSS of the Sidekiq + process (expressed in kilobytes) exceeds SIDEKIQ_MEMORY_KILLER_MAX_RSS, a + delayed shutdown is triggered. The default value for Omnibus packages is set + [in the omnibus-gitlab + repository](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/attributes/default.rb). +- `SIDEKIQ_MEMORY_KILLER_GRACE_TIME`: defaults 900 seconds (15 minutes). When + a shutdown is triggered, the Sidekiq process will keep working normally for + another 15 minutes. +- `SIDEKIQ_MEMORY_KILLER_SHUTDOWN_WAIT`: defaults to 30 seconds. When the grace + time has expired, the MemoryKiller tells Sidekiq to stop accepting new jobs. + Existing jobs get 30 seconds to finish. After that, the MemoryKiller tells + Sidekiq to shut down, and an external supervision mechanism (e.g. Runit) must + restart Sidekiq. |