diff options
author | GitLab Bot <gitlab-bot@gitlab.com> | 2021-12-20 13:37:47 +0000 |
---|---|---|
committer | GitLab Bot <gitlab-bot@gitlab.com> | 2021-12-20 13:37:47 +0000 |
commit | aee0a117a889461ce8ced6fcf73207fe017f1d99 (patch) | |
tree | 891d9ef189227a8445d83f35c1b0fc99573f4380 /doc/administration/troubleshooting/sidekiq.md | |
parent | 8d46af3258650d305f53b819eabf7ab18d22f59e (diff) | |
download | gitlab-ce-aee0a117a889461ce8ced6fcf73207fe017f1d99.tar.gz |
Add latest changes from gitlab-org/gitlab@14-6-stable-eev14.6.0-rc42
Diffstat (limited to 'doc/administration/troubleshooting/sidekiq.md')
-rw-r--r-- | doc/administration/troubleshooting/sidekiq.md | 35 |
1 files changed, 35 insertions, 0 deletions
diff --git a/doc/administration/troubleshooting/sidekiq.md b/doc/administration/troubleshooting/sidekiq.md index 7a8ac8c3dbe..a606a3712ba 100644 --- a/doc/administration/troubleshooting/sidekiq.md +++ b/doc/administration/troubleshooting/sidekiq.md @@ -85,6 +85,27 @@ several `WARN` level messages. Here's an example of a single thread's backtrace: In some cases Sidekiq may be hung and unable to respond to the `TTIN` signal. Move on to other troubleshooting methods if this happens. +## Ruby profiling with `rbspy` + +[rbspy](https://rbspy.github.io) is an easy to use and low-overhead Ruby profiler that can be used to create +flamegraph-style diagrams of CPU usage by Ruby processes. + +No changes to GitLab are required to use it and it has no dependencies. To install it: + +1. Download the binary from the [`rbspy` releases page](https://github.com/rbspy/rbspy/releases). +1. Make the binary executable. + +To profile a Sidekiq worker for one minute, run: + +```shell +sudo ./rbspy record --pid <sidekiq_pid> --duration 60 --file /tmp/sidekiq_profile.svg +``` + +![Example rbspy flamegraph](img/sidekiq_flamegraph.png) + +In this example of a flamegraph generated by `rbspy`, almost all of the Sidekiq process's time is spent in `rev_parse`, a native C +function in Rugged. In the stack, we can see `rev_parse` is being called by the `ExpirePipelineCacheWorker`. + ## Process profiling with `perf` Linux has a process profiling tool called `perf` that is helpful when a certain @@ -358,3 +379,17 @@ has number of drawbacks, as mentioned in [Why Ruby's Timeout is dangerous (and T > - in any of your code, regardless of whether it could have possibly raised an exception before > > Nobody writes code to defend against an exception being raised on literally any line. That's not even possible. So Thread.raise is basically like a sneak attack on your code that could result in almost anything. It would probably be okay if it were pure-functional code that did not modify any state. But this is Ruby, so that's unlikely :) + +## Disable Rugged + +Calls into Rugged, Ruby bindings for `libgit2`, [lock the Sidekiq processes's GVL](https://silverhammermba.github.io/emberb/c/#c-in-ruby-threads), +blocking all jobs on that worker from proceeding. If Rugged calls performed by Sidekiq are slow, this can cause significant delays in +background task processing. + +By default, Rugged is used when Git repository data is stored on local storage or on an NFS mount. +[Using Rugged is recommened when using NFS](../nfs.md#improving-nfs-performance-with-gitlab), but if +you are using local storage, disabling Rugged can improve Sidekiq performance: + +```shell +sudo gitlab-rake gitlab:features:disable_rugged +``` |