diff options
Diffstat (limited to 'doc/administration/gitaly/configure_gitaly.md')
| -rw-r--r-- | doc/administration/gitaly/configure_gitaly.md | 105 |
1 files changed, 15 insertions, 90 deletions
diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md index 0b22df5a115..5e8cbac42c1 100644 --- a/doc/administration/gitaly/configure_gitaly.md +++ b/doc/administration/gitaly/configure_gitaly.md @@ -217,10 +217,7 @@ disable enforcement. For more information, see the documentation on configuring 1. Edit `/etc/gitlab/gitlab.rb`: - <!-- - updates to following example must also be made at - https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab - --> + <!-- Updates to following example must also be made at https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab --> ```ruby # Avoid running unnecessary services on the Gitaly server @@ -267,10 +264,7 @@ disable enforcement. For more information, see the documentation on configuring 1. Append the following to `/etc/gitlab/gitlab.rb` for each respective Gitaly server: - <!-- - updates to following example must also be made at - https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab - --> + <!-- Updates to following example must also be made at https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab --> On `gitaly1.internal`: @@ -595,10 +589,7 @@ To configure Gitaly with TLS: 1. Edit `/etc/gitlab/gitlab.rb` and add: - <!-- - updates to following example must also be made at - https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab - --> + <!-- Updates to following example must also be made at https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab --> ```ruby gitaly['tls_listen_addr'] = "0.0.0.0:9999" @@ -693,12 +684,8 @@ To configure Gitaly with TLS: ### Observe type of Gitaly connections -[Prometheus](../monitoring/prometheus/index.md) can be used observe what type of connections Gitaly -is serving a production environment. Use the following Prometheus query: - -```prometheus -sum(rate(gitaly_connections_total[5m])) by (type) -``` +For information on observing the type of Gitaly connections being served, see the +[relevant documentation](index.md#useful-queries). ## `gitaly-ruby` @@ -790,20 +777,8 @@ repository. In the example above: - If another request comes in for a repository that has used up its 20 slots, that request gets queued. -You can observe the behavior of this queue using the Gitaly logs and Prometheus: - -- In the Gitaly logs, look for the string (or structured log field) `acquire_ms`. Messages that have - this field are reporting about the concurrency limiter. -- In Prometheus, look for the following metrics: - - - `gitaly_rate_limiting_in_progress`. - - `gitaly_rate_limiting_queued`. - - `gitaly_rate_limiting_seconds`. - -NOTE: -Although the name of the Prometheus metric contains `rate_limiting`, it's a concurrency limiter, not -a rate limiter. If a Gitaly client makes 1,000 requests in a row very quickly, concurrency doesn't -exceed 1, and the concurrency limiter has no effect. +You can observe the behavior of this queue using the Gitaly logs and Prometheus. For more +information, see the [relevant documentation](index.md#monitor-gitaly). ## Background Repository Optimization @@ -857,30 +832,11 @@ server" and "Gitaly client" refers to the same machine. ### Verify authentication monitoring -Before rotating a Gitaly authentication token, verify that you can monitor the authentication -behavior of your GitLab installation using Prometheus. Use the following Prometheus query: +Before rotating a Gitaly authentication token, verify that you can +[monitor the authentication behavior](index.md#useful-queries) of your GitLab installation using +Prometheus. -```prometheus -sum(rate(gitaly_authentications_total[5m])) by (enforced, status) -``` - -In a system where authentication is configured correctly and where you have live traffic, you -see something like this: - -```prometheus -{enforced="true",status="ok"} 4424.985419441742 -``` - -There may also be other numbers with rate 0. We care only about the non-zero numbers. - -The only non-zero number should have `enforced="true",status="ok"`. If you have other non-zero -numbers, something is wrong in your configuration. - -The `status="ok"` number reflects your current request rate. In the example above, Gitaly is -handling about 4000 requests per second. - -Now that you have established that you can monitor the Gitaly authentication behavior of your GitLab -installation, you can begin the rest of the procedure. +You can then continue the rest of the procedure. ### Enable "auth transitioning" mode @@ -955,7 +911,7 @@ result as you did at the start. For example: {enforced="true",status="ok"} 4424.985419441742 ``` -Note that `enforced="true"` means that authentication is being enforced. +`enforced="true"` means that authentication is being enforced. ## Pack-objects cache **(FREE SELF)** @@ -1079,7 +1035,7 @@ cache hit and the average amount of storage used by cache files. Entries older than `max_age` get evicted from the in-memory metadata store, and deleted from disk. -Note that eviction does not interfere with ongoing requests, so it is OK +Eviction does not interfere with ongoing requests, so it is OK for `max_age` to be less than the time it takes to do a fetch over a slow connection. This is because Unix filesystems do not truly delete a file until all processes that are reading the deleted file have @@ -1087,9 +1043,8 @@ closed it. ### Observe the cache -The cache can be observed in logs and using metrics. - -#### Logs +The cache can be observed [using metrics](index.md#monitor-gitaly) and in the following logged +information: |Message|Fields|Description| |:---|:---|:---| @@ -1149,33 +1104,3 @@ Example: "time":"2021-03-25T14:57:53.543Z" } ``` - -#### Metrics - -The following cache metrics are available. - -|Metric|Type|Labels|Description| -|:---|:---|:---|:---| -|`gitaly_pack_objects_cache_enabled`|gauge|`dir`,`max_age`|Set to `1` when the cache is enabled via the Gitaly configuration file| -|`gitaly_pack_objects_cache_lookups_total`|counter|`result`|Hit/miss counter for cache lookups| -|`gitaly_pack_objects_generated_bytes_total`|counter||Number of bytes written into the cache| -|`gitaly_pack_objects_served_bytes_total`|counter||Number of bytes read from the cache| -|`gitaly_streamcache_filestore_disk_usage_bytes`|gauge|`dir`|Total size of cache files| -|`gitaly_streamcache_index_entries`|gauge|`dir`|Number of entries in the cache| - -Some of these metrics start with `gitaly_streamcache` -because they are generated by the "streamcache" internal library -package in Gitaly. - -Example: - -```plaintext -gitaly_pack_objects_cache_enabled{dir="/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache",max_age="300"} 1 -gitaly_pack_objects_cache_lookups_total{result="hit"} 2 -gitaly_pack_objects_cache_lookups_total{result="miss"} 1 -gitaly_pack_objects_generated_bytes_total 2.618649e+07 -gitaly_pack_objects_served_bytes_total 7.855947e+07 -gitaly_streamcache_filestore_disk_usage_bytes{dir="/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache"} 2.6200152e+07 -gitaly_streamcache_filestore_removed_total{dir="/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache"} 1 -gitaly_streamcache_index_entries{dir="/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache"} 1 -``` |