summaryrefslogtreecommitdiff
path: root/doc/administration/gitaly/troubleshooting.md
diff options
context:
space:
mode:
Diffstat (limited to 'doc/administration/gitaly/troubleshooting.md')
-rw-r--r--doc/administration/gitaly/troubleshooting.md44
1 files changed, 43 insertions, 1 deletions
diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md
index ff136b44340..516af4ca469 100644
--- a/doc/administration/gitaly/troubleshooting.md
+++ b/doc/administration/gitaly/troubleshooting.md
@@ -554,7 +554,7 @@ Is [some cases](index.md#known-issues) the Praefect database can get out of sync
a given repository is fully synced on all nodes, run the [`gitlab:praefect:replicas` Rake task](../raketasks/praefect.md#replica-checksums)
that checksums the repository on all Gitaly nodes.
-The [Praefect dataloss](recovery.md#check-for-data-loss) command only checks the state of the repo in the Praefect database, and cannot
+The [Praefect dataloss](recovery.md#check-for-data-loss) command only checks the state of the repository in the Praefect database, and cannot
be relied to detect sync problems in this scenario.
### Relation does not exist errors
@@ -610,3 +610,45 @@ Possible solutions:
- Provision larger VMs to gain access to larger network traffic allowances.
- Use your cloud service's monitoring and logging to check that the Praefect nodes are not exhausting their traffic allowances.
+
+## Profiling Gitaly
+
+Gitaly exposes several of Golang's built-in performance profiling tools on the Prometheus listen port. For example, if Prometheus is listening
+on port `9236` of the GitLab server:
+
+- Get a list of running `goroutines` and their backtraces:
+
+ ```shell
+ curl --output goroutines.txt "http://<gitaly_server>:9236/debug/pprof/goroutine?debug=2"
+ ```
+
+- Run a CPU profile for 30 seconds:
+
+ ```shell
+ curl --output cpu.bin "http://<gitaly_server>:9236/debug/pprof/profile"
+ ```
+
+- Profile heap memory usage:
+
+ ```shell
+ curl --output heap.bin "http://<gitaly_server>:9236/debug/pprof/heap"
+ ```
+
+- Record a 5 second execution trace. This will impact Gitaly's performance while running:
+
+ ```shell
+ curl --output trace.bin "http://<gitaly_server>:9236/debug/pprof/trace?seconds=5"
+ ```
+
+On a host with `go` installed, the CPU profile and heap profile can be viewed in a browser:
+
+```shell
+go tool pprof -http=:8001 cpu.bin
+go tool pprof -http=:8001 heap.bin
+```
+
+Execution traces can be viewed by running:
+
+```shell
+go tool trace heap.bin
+```