diff options
Diffstat (limited to 'doc/administration/gitaly/troubleshooting.md')
| -rw-r--r-- | doc/administration/gitaly/troubleshooting.md | 44 |
1 files changed, 43 insertions, 1 deletions
diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md index ff136b44340..516af4ca469 100644 --- a/doc/administration/gitaly/troubleshooting.md +++ b/doc/administration/gitaly/troubleshooting.md @@ -554,7 +554,7 @@ Is [some cases](index.md#known-issues) the Praefect database can get out of sync a given repository is fully synced on all nodes, run the [`gitlab:praefect:replicas` Rake task](../raketasks/praefect.md#replica-checksums) that checksums the repository on all Gitaly nodes. -The [Praefect dataloss](recovery.md#check-for-data-loss) command only checks the state of the repo in the Praefect database, and cannot +The [Praefect dataloss](recovery.md#check-for-data-loss) command only checks the state of the repository in the Praefect database, and cannot be relied to detect sync problems in this scenario. ### Relation does not exist errors @@ -610,3 +610,45 @@ Possible solutions: - Provision larger VMs to gain access to larger network traffic allowances. - Use your cloud service's monitoring and logging to check that the Praefect nodes are not exhausting their traffic allowances. + +## Profiling Gitaly + +Gitaly exposes several of Golang's built-in performance profiling tools on the Prometheus listen port. For example, if Prometheus is listening +on port `9236` of the GitLab server: + +- Get a list of running `goroutines` and their backtraces: + + ```shell + curl --output goroutines.txt "http://<gitaly_server>:9236/debug/pprof/goroutine?debug=2" + ``` + +- Run a CPU profile for 30 seconds: + + ```shell + curl --output cpu.bin "http://<gitaly_server>:9236/debug/pprof/profile" + ``` + +- Profile heap memory usage: + + ```shell + curl --output heap.bin "http://<gitaly_server>:9236/debug/pprof/heap" + ``` + +- Record a 5 second execution trace. This will impact Gitaly's performance while running: + + ```shell + curl --output trace.bin "http://<gitaly_server>:9236/debug/pprof/trace?seconds=5" + ``` + +On a host with `go` installed, the CPU profile and heap profile can be viewed in a browser: + +```shell +go tool pprof -http=:8001 cpu.bin +go tool pprof -http=:8001 heap.bin +``` + +Execution traces can be viewed by running: + +```shell +go tool trace heap.bin +``` |