diff options
author | Jacob Vosmaer <jacob@gitlab.com> | 2019-08-26 12:02:15 +0200 |
---|---|---|
committer | Jacob Vosmaer <jacob@gitlab.com> | 2019-08-26 12:02:15 +0200 |
commit | b510659e601bd01d28eadd9f6a3ad6f1a39d372e (patch) | |
tree | f34fa8273b6be7cd0b0b09dfee24bc71e9a4a48b | |
parent | 116b3c186d8e6a54dbf5d4bfa78a4a195514cf35 (diff) | |
download | gitlab-ce-docs-jv-gitaly-concurrency-limiter.tar.gz |
Move new section above troubleshootingdocs-jv-gitaly-concurrency-limiter
-rw-r--r-- | doc/administration/gitaly/index.md | 82 |
1 files changed, 41 insertions, 41 deletions
diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index 0b5becc00d8..ac6a1070617 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -474,6 +474,47 @@ One current feature of GitLab that still requires a shared directory (NFS) is There is [work in progress](https://gitlab.com/gitlab-org/gitlab-pages/issues/196) to eliminate the need for NFS to support GitLab Pages. +## Limiting RPC concurrency + +It can happen that CI clone traffic puts a large strain on your Gitaly +service. The bulk of the work gets done in the SSHUploadPack (for Git +SSH) and PostUploadPack (for Git HTTP) RPC's. To prevent such workloads +from overcrowding your Gitaly server you can set concurrency limits in +Gitaly's configuration file. + +```ruby +# in /etc/gitlab/gitlab.rb + +gitaly['concurrency'] = [ + { + 'rpc' => "/gitaly.SmartHTTPService/PostUploadPack", + 'max_per_repo' => 20 + }, + { + 'rpc' => "/gitaly.SSHService/SSHUploadPack", + 'max_per_repo' => 20 + } +] +``` +This will limit the number of in-flight RPC calls for the given RPC's. +The limit is applied per repository. In the example above, each on the +Gitaly server can have at most 20 simultaneous PostUploadPack calls in +flight, and the same for SSHUploadPack. If another request comes in for +a repository that hase used up its 20 slots, that request will get +queued. + +You can observe the behavior of this queue via the Gitaly logs and via +Prometheus. In the Gitaly logs, you can look for the string (or +structured log field) `acquire_ms`. Messages that have this field are +reporting about the concurrency limiter. In Prometheus, look for the +`gitaly_rate_limiting_in_progress`, `gitaly_rate_limiting_queued` and +`gitaly_rate_limiting_seconds` metrics. + +The name of the Prometheus metric is not quite right because this is a +concurrency limiter, not a rate limiter. If a client makes 1000 requests +in a row in a very short timespan, the concurrency will not exceed 1, +and this mechanism (the concurrency limiter) will do nothing. + ## Troubleshooting Gitaly ### Commits, pushes, and clones return a 401 @@ -615,44 +656,3 @@ To fix this problem, confirm that your [`gitlab-secrets.json` file](#3-gitaly-se on the Gitaly node matches the one on all other nodes. If it doesn't match, update the secrets file on the Gitaly node to match the others, then [reconfigure the node](../restart_gitlab.md#omnibus-gitlab-reconfigure). - -## Limiting RPC concurrency - -It can happen that CI clone traffic puts a large strain on your Gitaly -service. The bulk of the work gets done in the SSHUploadPack (for Git -SSH) and PostUploadPack (for Git HTTP) RPC's. To prevent such workloads -from overcrowding your Gitaly server you can set concurrency limits in -Gitaly's configuration file. - -```ruby -# in /etc/gitlab/gitlab.rb - -gitaly['concurrency'] = [ - { - 'rpc' => "/gitaly.SmartHTTPService/PostUploadPack", - 'max_per_repo' => 20 - }, - { - 'rpc' => "/gitaly.SSHService/SSHUploadPack", - 'max_per_repo' => 20 - } -] -``` -This will limit the number of in-flight RPC calls for the given RPC's. -The limit is applied per repository. In the example above, each on the -Gitaly server can have at most 20 simultaneous PostUploadPack calls in -flight, and the same for SSHUploadPack. If another request comes in for -a repository that hase used up its 20 slots, that request will get -queued. - -You can observe the behavior of this queue via the Gitaly logs and via -Prometheus. In the Gitaly logs, you can look for the string (or -structured log field) `acquire_ms`. Messages that have this field are -reporting about the concurrency limiter. In Prometheus, look for the -`gitaly_rate_limiting_in_progress`, `gitaly_rate_limiting_queued` and -`gitaly_rate_limiting_seconds` metrics. - -The name of the Prometheus metric is not quite right because this is a -concurrency limiter, not a rate limiter. If a client makes 1000 requests -in a row in a very short timespan, the concurrency will not exceed 1, -and this mechanism (the concurrency limiter) will do nothing. |