summaryrefslogtreecommitdiff
path: root/doc/administration/troubleshooting/sidekiq.md
diff options
context:
space:
mode:
authorGitLab Bot <gitlab-bot@gitlab.com>2021-12-20 13:37:47 +0000
committerGitLab Bot <gitlab-bot@gitlab.com>2021-12-20 13:37:47 +0000
commitaee0a117a889461ce8ced6fcf73207fe017f1d99 (patch)
tree891d9ef189227a8445d83f35c1b0fc99573f4380 /doc/administration/troubleshooting/sidekiq.md
parent8d46af3258650d305f53b819eabf7ab18d22f59e (diff)
downloadgitlab-ce-aee0a117a889461ce8ced6fcf73207fe017f1d99.tar.gz
Add latest changes from gitlab-org/gitlab@14-6-stable-eev14.6.0-rc42
Diffstat (limited to 'doc/administration/troubleshooting/sidekiq.md')
-rw-r--r--doc/administration/troubleshooting/sidekiq.md35
1 files changed, 35 insertions, 0 deletions
diff --git a/doc/administration/troubleshooting/sidekiq.md b/doc/administration/troubleshooting/sidekiq.md
index 7a8ac8c3dbe..a606a3712ba 100644
--- a/doc/administration/troubleshooting/sidekiq.md
+++ b/doc/administration/troubleshooting/sidekiq.md
@@ -85,6 +85,27 @@ several `WARN` level messages. Here's an example of a single thread's backtrace:
In some cases Sidekiq may be hung and unable to respond to the `TTIN` signal.
Move on to other troubleshooting methods if this happens.
+## Ruby profiling with `rbspy`
+
+[rbspy](https://rbspy.github.io) is an easy to use and low-overhead Ruby profiler that can be used to create
+flamegraph-style diagrams of CPU usage by Ruby processes.
+
+No changes to GitLab are required to use it and it has no dependencies. To install it:
+
+1. Download the binary from the [`rbspy` releases page](https://github.com/rbspy/rbspy/releases).
+1. Make the binary executable.
+
+To profile a Sidekiq worker for one minute, run:
+
+```shell
+sudo ./rbspy record --pid <sidekiq_pid> --duration 60 --file /tmp/sidekiq_profile.svg
+```
+
+![Example rbspy flamegraph](img/sidekiq_flamegraph.png)
+
+In this example of a flamegraph generated by `rbspy`, almost all of the Sidekiq process's time is spent in `rev_parse`, a native C
+function in Rugged. In the stack, we can see `rev_parse` is being called by the `ExpirePipelineCacheWorker`.
+
## Process profiling with `perf`
Linux has a process profiling tool called `perf` that is helpful when a certain
@@ -358,3 +379,17 @@ has number of drawbacks, as mentioned in [Why Ruby's Timeout is dangerous (and T
> - in any of your code, regardless of whether it could have possibly raised an exception before
>
> Nobody writes code to defend against an exception being raised on literally any line. That's not even possible. So Thread.raise is basically like a sneak attack on your code that could result in almost anything. It would probably be okay if it were pure-functional code that did not modify any state. But this is Ruby, so that's unlikely :)
+
+## Disable Rugged
+
+Calls into Rugged, Ruby bindings for `libgit2`, [lock the Sidekiq processes's GVL](https://silverhammermba.github.io/emberb/c/#c-in-ruby-threads),
+blocking all jobs on that worker from proceeding. If Rugged calls performed by Sidekiq are slow, this can cause significant delays in
+background task processing.
+
+By default, Rugged is used when Git repository data is stored on local storage or on an NFS mount.
+[Using Rugged is recommened when using NFS](../nfs.md#improving-nfs-performance-with-gitlab), but if
+you are using local storage, disabling Rugged can improve Sidekiq performance:
+
+```shell
+sudo gitlab-rake gitlab:features:disable_rugged
+```