summaryrefslogtreecommitdiff
path: root/doc/administration/gitaly
diff options
context:
space:
mode:
authorGitLab Bot <gitlab-bot@gitlab.com>2023-05-17 16:05:49 +0000
committerGitLab Bot <gitlab-bot@gitlab.com>2023-05-17 16:05:49 +0000
commit43a25d93ebdabea52f99b05e15b06250cd8f07d7 (patch)
treedceebdc68925362117480a5d672bcff122fb625b /doc/administration/gitaly
parent20c84b99005abd1c82101dfeff264ac50d2df211 (diff)
downloadgitlab-ce-43a25d93ebdabea52f99b05e15b06250cd8f07d7.tar.gz
Add latest changes from gitlab-org/gitlab@16-0-stable-eev16.0.0-rc4216-0-stable
Diffstat (limited to 'doc/administration/gitaly')
-rw-r--r--doc/administration/gitaly/configure_gitaly.md560
-rw-r--r--doc/administration/gitaly/gitaly_geo_capabilities.md41
-rw-r--r--doc/administration/gitaly/index.md28
-rw-r--r--doc/administration/gitaly/monitoring.md15
-rw-r--r--doc/administration/gitaly/praefect.md558
-rw-r--r--doc/administration/gitaly/recovery.md72
-rw-r--r--doc/administration/gitaly/reference.md42
-rw-r--r--doc/administration/gitaly/troubleshooting.md58
8 files changed, 904 insertions, 470 deletions
diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md
index 143f7dca7d3..f3eb2de3f1d 100644
--- a/doc/administration/gitaly/configure_gitaly.md
+++ b/doc/administration/gitaly/configure_gitaly.md
@@ -6,25 +6,26 @@ info: To determine the technical writer assigned to the Stage/Group associated w
# Configure Gitaly **(FREE SELF)**
-The Gitaly service itself is configured by using a [TOML configuration file](reference.md).
+Configure Gitaly in one of two ways:
-To change Gitaly settings:
+::Tabs
-**For Omnibus GitLab**
+:::TabTitle Linux package (Omnibus)
1. Edit `/etc/gitlab/gitlab.rb` and add or change the
[Gitaly settings](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/1dd07197c7e5ae23626aad5a4a070a800b670380/files/gitlab-config-template/gitlab.rb.template#L1622-1676).
1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
-**For installations from source**
+:::TabTitle Self-compiled (source)
1. Edit `/home/git/gitaly/config.toml` and add or change the [Gitaly settings](https://gitlab.com/gitlab-org/gitaly/blob/master/config.toml.example).
1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source).
+::EndTabs
+
The following configuration options are also available:
- Enabling [TLS support](#enable-tls-support).
-- Configuring the [number of `gitaly-ruby` workers](#configure-number-of-gitaly-ruby-workers).
- Limiting [RPC concurrency](#limit-rpc-concurrency).
## About the Gitaly token
@@ -138,15 +139,25 @@ Gitaly and GitLab use two shared secrets for authentication:
- _Gitaly token_: used to authenticate gRPC requests to Gitaly
- _GitLab Shell token_: used for authentication callbacks from GitLab Shell to the GitLab internal API
-**For Omnibus GitLab**
+Configure authentication in one of two ways:
+
+::Tabs
+
+:::TabTitle Linux package (Omnibus)
To configure the _Gitaly token_, edit `/etc/gitlab/gitlab.rb`:
```ruby
- gitaly['auth_token'] = 'abc123secret'
+ gitaly['configuration'] = {
+ # ...
+ auth: {
+ # ...
+ token: 'abc123secret',
+ },
+ }
```
-There are two ways to configure the _GitLab Shell token_.
+Configure the _GitLab Shell token_ in one of two ways.
Method 1 (recommended):
@@ -161,7 +172,7 @@ Edit `/etc/gitlab/gitlab.rb`:
gitlab_shell['secret_token'] = 'shellsecret'
```
-**For installations from source**
+:::TabTitle Self-compiled (source)
1. Copy `/home/git/gitlab/.gitlab_shell_secret` from the Gitaly client to the same path on the
Gitaly servers (and any other Gitaly clients).
@@ -183,19 +194,26 @@ Edit `/etc/gitlab/gitlab.rb`:
1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source).
-#### Configure Gitaly server
-
-**For Omnibus GitLab**
+::EndTabs
-1. Edit `/etc/gitlab/gitlab.rb`:
+#### Configure Gitaly server
<!--
Updates to example must be made at:
+
- https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab
- https://gitlab.com/gitlab-org/gitlab/blob/master/doc/administration/gitaly/index.md#gitaly-server-configuration
-- all reference architecture pages
+- All reference architecture pages
-->
+Configure Gitaly server in one of two ways:
+
+::Tabs
+
+:::TabTitle Linux package (Omnibus)
+
+1. Edit `/etc/gitlab/gitlab.rb`:
+
```ruby
# Avoid running unnecessary services on the Gitaly server
postgresql['enable'] = false
@@ -230,14 +248,21 @@ Updates to example must be made at:
# Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from Gitaly client to Gitaly server.
gitlab_rails['internal_api_url'] = 'https://gitlab.example.com'
- # Make Gitaly accept connections on all network interfaces. You must use
- # firewalls to restrict access to this address/port.
- # Comment out following line if you only want to support TLS connections
- gitaly['listen_addr'] = "0.0.0.0:8075"
-
- # Authentication token to ensure only authorized servers can communicate with
- # Gitaly server
- gitaly['auth_token'] = 'AUTH_TOKEN'
+ gitaly['configuration'] = {
+ # ...
+ #
+ # Make Gitaly accept connections on all network interfaces. You must use
+ # firewalls to restrict access to this address/port.
+ # Comment out following line if you only want to support TLS connections
+ listen_addr: '0.0.0.0:8075',
+ auth: {
+ # ...
+ #
+ # Authentication token to ensure only authorized servers can communicate with
+ # Gitaly server
+ token: 'AUTH_TOKEN',
+ },
+ }
```
1. Append the following to `/etc/gitlab/gitlab.rb` for each respective Gitaly server:
@@ -247,24 +272,33 @@ Updates to example must be made at:
On `gitaly1.internal`:
```ruby
- git_data_dirs({
- 'default' => {
- 'path' => '/var/opt/gitlab/git-data'
- },
- 'storage1' => {
- 'path' => '/mnt/gitlab/git-data'
- },
- })
+ gitaly['configuration'] = {
+ # ...
+ storage: [
+ {
+ name: 'default',
+ path: '/var/opt/gitlab/git-data',
+ },
+ {
+ name: 'storage1',
+ path: '/mnt/gitlab/git-data',
+ },
+ ],
+ }
```
On `gitaly2.internal`:
```ruby
- git_data_dirs({
- 'storage2' => {
- 'path' => '/srv/gitlab/git-data'
- },
- })
+ gitaly['configuration'] = {
+ # ...
+ storage: [
+ {
+ name: 'storage2',
+ path: '/srv/gitlab/git-data',
+ },
+ ],
+ }
```
1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
@@ -272,7 +306,7 @@ Updates to example must be made at:
- For GitLab 15.3 and later, run `sudo /opt/gitlab/embedded/bin/gitaly check /var/opt/gitlab/gitaly/config.toml`.
- For GitLab 15.2 and earlier, run `sudo /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml`.
-**For installations from source**
+:::TabTitle Self-compiled (source)
1. Edit `/home/git/gitaly/config.toml`:
@@ -323,6 +357,8 @@ Updates to example must be made at:
- For GitLab 15.3 and later, run `sudo /opt/gitlab/embedded/bin/gitaly check /var/opt/gitlab/gitaly/config.toml`.
- For GitLab 15.2 and earlier, run `sudo /opt/gitlab/embedded/bin/gitaly-hooks check /var/opt/gitlab/gitaly/config.toml`.
+::EndTabs
+
WARNING:
If directly copying repository data from a GitLab server to Gitaly, ensure that the metadata file,
default path `/var/opt/gitlab/git-data/repositories/.gitaly-metadata`, is not included in the transfer.
@@ -359,7 +395,11 @@ You can't define Gitaly servers with some as a local Gitaly server
server (with `gitaly_address`) unless you use
[mixed configuration](#mixed-configuration).
-**For Omnibus GitLab**
+Configure Gitaly clients in one of two ways:
+
+::Tabs
+
+:::TabTitle Linux package (Omnibus)
1. Edit `/etc/gitlab/gitlab.rb`:
@@ -393,7 +433,7 @@ server (with `gitaly_address`) unless you use
sudo gitlab-ctl tail gitaly
```
-**For installations from source**
+:::TabTitle Self-compiled (source)
1. Edit `/home/git/gitlab/config/gitlab.yml`:
@@ -429,6 +469,8 @@ server (with `gitaly_address`) unless you use
tail -f /home/git/gitlab/log/gitaly.log
```
+::EndTabs
+
When you tail the Gitaly logs on your Gitaly server, you should see requests coming in. One sure way
to trigger a Gitaly request is to clone a repository from GitLab over HTTP or HTTPS.
@@ -461,17 +503,28 @@ example:
git_data_dirs({
'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' },
# Address of the GitLab server that also has Gitaly running on it
- 'storage1' => { 'gitaly_address' => 'tcp://gitlab.internal:8075', 'path' => '/mnt/gitlab/git-data' },
+ 'storage1' => { 'gitaly_address' => 'tcp://gitlab.internal:8075' },
'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' },
})
-# Make Gitaly accept connections on all network interfaces
-gitaly['listen_addr'] = "0.0.0.0:8075"
-
-# Or for TLS
-gitaly['tls_listen_addr'] = "0.0.0.0:9999"
-gitaly['certificate_path'] = "/etc/gitlab/ssl/cert.pem"
-gitaly['key_path'] = "/etc/gitlab/ssl/key.pem"
+gitaly['configuration'] = {
+ # ...
+ #
+ # Make Gitaly accept connections on all network interfaces
+ listen_addr: '0.0.0.0:8075',
+ # Or for TLS
+ tls_listen_addr: '0.0.0.0:9999',
+ tls: {
+ certificate_path: '/etc/gitlab/ssl/cert.pem',
+ key_path: '/etc/gitlab/ssl/key.pem',
+ },
+ storage: [
+ {
+ name: 'storage1',
+ path: '/mnt/gitlab/git-data',
+ },
+ ],
+}
```
`path` can be included only for storage shards on the local Gitaly server.
@@ -499,9 +552,11 @@ Disabling Gitaly on the GitLab instance makes sense only when you run GitLab in
Gitaly runs on a separate machine from the GitLab instance. Disabling Gitaly on all machines in the cluster is not
a valid configuration (some machines much act as Gitaly servers).
-To disable Gitaly on a GitLab server:
+Disable Gitaly on a GitLab server in one of two ways:
+
+::Tabs
-**For Omnibus GitLab**
+:::TabTitle Linux package (Omnibus)
1. Edit `/etc/gitlab/gitlab.rb`:
@@ -511,7 +566,7 @@ To disable Gitaly on a GitLab server:
1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
-**For installations from source**
+:::TabTitle Self-compiled (source)
1. Edit `/etc/default/gitlab`:
@@ -521,10 +576,9 @@ To disable Gitaly on a GitLab server:
1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source).
-## Enable TLS support
+::EndTabs
-> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/22602) in GitLab 11.8.
-> - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/3160) in GitLab 13.6, outgoing TLS connections to GitLab provide client certificates if configured.
+## Enable TLS support
Gitaly supports TLS encryption. To communicate with a Gitaly instance that listens for secure
connections, use the `tls://` URL scheme in the `gitaly_address` of the corresponding
@@ -543,6 +597,8 @@ Additionally, the certificate (or its certificate authority) must be installed o
- Gitaly servers.
- Gitaly clients that communicate with it.
+If you use a load balancer, it must be able to negotiate HTTP/2 using the ALPN TLS extension.
+
### Certificate requirements
- The certificate must specify the address you use to access the Gitaly server. You must add the hostname or IP address as a Subject Alternative Name to the certificate.
@@ -553,9 +609,11 @@ Additionally, the certificate (or its certificate authority) must be installed o
### Configure Gitaly with TLS
-To configure Gitaly with TLS:
+Configure Gitaly with TLS in one of two ways:
+
+::Tabs
-**For Omnibus GitLab**
+:::TabTitle Linux package (Omnibus)
1. Create certificates for Gitaly servers.
1. On the Gitaly clients, copy the certificates (or their certificate authority) into
@@ -600,21 +658,26 @@ To configure Gitaly with TLS:
<!-- Updates to following example must also be made at https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab -->
```ruby
- gitaly['tls_listen_addr'] = "0.0.0.0:9999"
- gitaly['certificate_path'] = "/etc/gitlab/ssl/cert.pem"
- gitaly['key_path'] = "/etc/gitlab/ssl/key.pem"
+ gitaly['configuration'] = {
+ # ...
+ tls_listen_addr: '0.0.0.0:9999',
+ tls: {
+ certificate_path: '/etc/gitlab/ssl/cert.pem',
+ key_path: '/etc/gitlab/ssl/key.pem',
+ },
+ }
```
1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
1. Verify Gitaly traffic is being served over TLS by
[observing the types of Gitaly connections](#observe-type-of-gitaly-connections).
1. Optional. Improve security by:
- 1. Disabling non-TLS connections by commenting out or deleting `gitaly['listen_addr']` in
+ 1. Disabling non-TLS connections by commenting out or deleting `gitaly['configuration'][:listen_addr]` in
`/etc/gitlab/gitlab.rb`.
1. Saving the file.
1. [Reconfiguring GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
-**For installations from source**
+:::TabTitle Self-compiled (source)
1. Create certificates for Gitaly servers.
1. On the Gitaly clients, copy the certificates into the system trusted certificates:
@@ -690,66 +753,12 @@ To configure Gitaly with TLS:
1. Saving the file.
1. [Restarting GitLab](../restart_gitlab.md#installations-from-source).
+::EndTabs
+
### Observe type of Gitaly connections
For information on observing the type of Gitaly connections being served, see the
-[relevant documentation](monitoring.md#useful-queries).
-
-## `gitaly-ruby`
-
-Gitaly was developed to replace the Ruby application code in GitLab.
-
-To save time and avoid the risk of rewriting existing application logic, we chose to copy some
-application code from GitLab into Gitaly.
-
-To be able to run that code, `gitaly-ruby` was created, which is a "sidecar" process for the main
-Gitaly Go process. Some examples of things that are implemented in `gitaly-ruby` are:
-
-- RPCs that deal with wikis.
-- RPCs that create commits on behalf of a user, such as merge commits.
-
-We recommend:
-
-- At least 300 MB memory per worker.
-- No more than one worker per core.
-
-NOTE:
-`gitaly-ruby` is planned to be eventually removed. To track progress, see the
-[Remove the Gitaly-Ruby sidecar](https://gitlab.com/groups/gitlab-org/-/epics/2862) epic.
-
-### Configure number of `gitaly-ruby` workers
-
-`gitaly-ruby` has much less capacity than Gitaly implemented in Go. If your Gitaly server has to handle lots of
-requests, the default setting of having just one active `gitaly-ruby` sidecar might not be enough.
-
-If you see `ResourceExhausted` errors from Gitaly, it's very likely that you have not enough
-`gitaly-ruby` capacity.
-
-You can increase the number of `gitaly-ruby` processes on your Gitaly server with the following
-settings:
-
-**For Omnibus GitLab**
-
-1. Edit `/etc/gitlab/gitlab.rb`:
-
- ```ruby
- # Default is 2 workers. The minimum is 2; 1 worker is always reserved as
- # a passive stand-by.
- gitaly['ruby_num_workers'] = 4
- ```
-
-1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
-
-**For installations from source**
-
-1. Edit `/home/git/gitaly/config.toml`:
-
- ```toml
- [gitaly-ruby]
- num_workers = 4
- ```
-
-1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source).
+[relevant documentation](monitoring.md#queries).
## Limit RPC concurrency
@@ -775,21 +784,23 @@ example:
```ruby
# in /etc/gitlab/gitlab.rb
-
-gitaly['concurrency'] = [
- {
- 'rpc' => "/gitaly.SmartHTTPService/PostUploadPackWithSidechannel",
- 'max_per_repo' => 20,
- 'max_queue_time' => "1s",
- 'max_queue_size' => 10
- },
- {
- 'rpc' => "/gitaly.SSHService/SSHUploadPackWithSidechannel",
- 'max_per_repo' => 20
- 'max_queue_time' => "1s",
- 'max_queue_size' => 10
- }
-]
+gitaly['configuration'] = {
+ # ...
+ concurrency: [
+ {
+ rpc: '/gitaly.SmartHTTPService/PostUploadPackWithSidechannel',
+ max_per_repo: 20,
+ max_queue_time: '1s',
+ max_queue_size: 10,
+ },
+ {
+ rpc: '/gitaly.SSHService/SSHUploadPackWithSidechannel',
+ max_per_repo: 20,
+ max_queue_time: '1s',
+ max_queue_size: 10,
+ },
+ ],
+}
```
- `rpc` is the name of the RPC to set a concurrency limit for per repository.
@@ -834,7 +845,7 @@ performance.
You can use control groups (cgroups) in Linux to impose limits on how much memory and CPU can be consumed by Gitaly processes.
See the [`cgroups` Linux man page](https://man7.org/linux/man-pages/man7/cgroups.7.html) for more information.
-cgroups can be useful for protecting the system against unexpected resource exhaustion because of over consumption of memory and CPU.
+cgroups can help protect the system against unexpected resource exhaustion because of over consumption of memory and CPU.
Some Git operations can consume notable resources up to the point of exhaustion in situations such as:
@@ -861,43 +872,60 @@ When these limits are reached, performance may be reduced and users may be disco
### Configure repository cgroups (new method)
-> This method of configuring repository cgroups was introduced in GitLab 15.1.
+> - This method of configuring repository cgroups was introduced in GitLab 15.1.
+> - `cpu_quota_us`[introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5422) in GitLab 15.10.
To configure repository cgroups in Gitaly using the new method, use the following settings for the new configuration method
-to `gitaly['cgroups']` in `/etc/gitlab/gitlab.rb`:
+to `gitaly['configuration'][:cgroups]` in `/etc/gitlab/gitlab.rb`:
-- `cgroups_mountpoint` is where the parent cgroup directory is mounted. Defaults to `/sys/fs/cgroup`.
-- `cgroups_hierarchy_root` is the parent cgroup under which Gitaly creates groups, and
+- `mountpoint` is where the parent cgroup directory is mounted. Defaults to `/sys/fs/cgroup`.
+- `hierarchy_root` is the parent cgroup under which Gitaly creates groups, and
is expected to be owned by the user and group Gitaly runs as. Omnibus GitLab
creates the set of directories `mountpoint/<cpu|memory>/hierarchy_root`
when Gitaly starts.
-- `cgroups_memory_bytes` is the total memory limit that is imposed collectively on all
+- `memory_bytes` is the total memory limit that is imposed collectively on all
Git processes that Gitaly spawns. 0 implies no limit.
-- `cgroups_cpu_shares` is the CPU limit that is imposed collectively on all Git
+- `cpu_shares` is the CPU limit that is imposed collectively on all Git
processes that Gitaly spawns. 0 implies no limit. The maximum is 1024 shares,
which represents 100% of CPU.
-- `cgroups_repositories_count` is the number of cgroups in the cgroups pool. Each time a new Git
+- `cpu_quota_us` is the [`cfs_quota_us`](https://docs.kernel.org/scheduler/sched-bwc.html#management)
+ to throttle the cgroups' processes if they exceed this quota value. We set
+ `cfs_period_us` to `100ms` so 1 core is `100000`. 0 implies no limit.
+- `repositories.count` is the number of cgroups in the cgroups pool. Each time a new Git
command is spawned, Gitaly assigns it to one of these cgroups based
on the repository the command is for. A circular hashing algorithm assigns
Git commands to these cgroups, so a Git command for a repository is
always assigned to the same cgroup.
-- `cgroups_repositories_memory_bytes` is the total memory limit imposed on all Git processes contained in a repository cgroup.
- 0 implies no limit. This value cannot exceed that of the top level `cgroups_memory_bytes`.
-- `cgroups_repositories_cpu_shares` is the CPU limit that is imposed on all Git processes contained in a repository cgroup.
+- `repositories.memory_bytes` is the total memory limit imposed on all Git processes contained in a repository cgroup.
+ 0 implies no limit. This value cannot exceed that of the top level `memory_bytes`.
+- `repositories.cpu_shares` is the CPU limit that is imposed on all Git processes contained in a repository cgroup.
0 implies no limit. The maximum is 1024 shares, which represents 100% of CPU.
- This value cannot exceed that of the top level`cgroups_cpu_shares`.
+ This value cannot exceed that of the top level`cpu_shares`.
+- `repositories.cpu_quota_us` is the [`cfs_quota_us`](https://docs.kernel.org/scheduler/sched-bwc.html#management)
+ that is imposed on all Git processes contained in a repository cgroup. A Git
+ process can't use more then the given quota. We set
+ `cfs_period_us` to `100ms` so 1 core is `100000`. 0 implies no limit.
For example:
```ruby
# in /etc/gitlab/gitlab.rb
-gitaly['cgroups_mountpoint'] = "/sys/fs/cgroup"
-gitaly['cgroups_hierarchy_root'] => "gitaly"
-gitaly['cgroups_memory_bytes'] = 64424509440, # 60gb
-gitaly['cgroups_cpu_shares'] = 1024
-gitaly['cgroups_repositories_count'] => 1000,
-gitaly['cgroups_repositories_memory_bytes'] => 32212254720 # 20gb
-gitaly['cgroups_repositories_cpu_shares'] => 512
+gitaly['configuration'] = {
+ # ...
+ cgroups: {
+ mountpoint: '/sys/fs/cgroup',
+ hierarchy_root: 'gitaly',
+ memory_bytes: 64424509440, # 60gb
+ cpu_shares: 1024,
+ cpu_quota_us: 400000 # 4 cores
+ repositories: {
+ count: 1000,
+ memory_bytes: 32212254720, # 20gb
+ cpu_shares: 512,
+ cpu_quota_us: 200000 # 2 cores
+ },
+ },
+}
```
### Configure repository cgroups (legacy method)
@@ -953,28 +981,38 @@ This strategy has two main benefits:
to 3 child cgroups can concurrently burst up to their max. In general, all
1000 cgroups would use much less than the 20 GB.
-## Background Repository Optimization
+## Background repository optimization
Empty directories and unneeded configuration settings may accumulate in a repository and
slow down Git operations. Gitaly can schedule a daily background task with a maximum duration
to clean up these items and improve performance.
WARNING:
-This is an experimental feature and may place significant load on the host while running.
+Background repository optimization is an experimental feature and may place significant load on the host while running.
Make sure to schedule this during off-peak hours and keep the duration short (for example, 30-60 minutes).
-**For Omnibus GitLab**
+Configure background repository optimization in one of two ways:
+
+::Tabs
+
+:::TabTitle Linux package (Omnibus)
Edit `/etc/gitlab/gitlab.rb` and add:
```ruby
-gitaly['daily_maintenance_start_hour'] = 4
-gitaly['daily_maintenance_start_minute'] = 30
-gitaly['daily_maintenance_duration'] = '30m'
-gitaly['daily_maintenance_storages'] = ["default"]
+gitaly['configuration'] = {
+ # ...
+ daily_maintenance: {
+ # ...
+ start_hour: 4,
+ start_minute: 30,
+ duration: '30m',
+ storages: ['default'],
+ },
+}
```
-**For installations from source**
+:::TabTitle Self-compiled (source)
Edit `/home/git/gitaly/config.toml` and add:
@@ -986,6 +1024,8 @@ duration = '30m'
storages = ["default"]
```
+::EndTabs
+
## Rotate Gitaly authentication token
Rotating credentials in a production environment often requires downtime, causes outages, or both.
@@ -1006,7 +1046,7 @@ server" and "Gitaly client" refers to the same machine.
### Verify authentication monitoring
Before rotating a Gitaly authentication token, verify that you can
-[monitor the authentication behavior](monitoring.md#useful-queries) of your GitLab installation using
+[monitor the authentication behavior](monitoring.md#queries) of your GitLab installation using
Prometheus.
You can then continue the rest of the procedure.
@@ -1018,7 +1058,13 @@ transitioning" mode as follows:
```ruby
# in /etc/gitlab/gitlab.rb
-gitaly['auth_transitioning'] = true
+gitaly['configuration'] = {
+ # ...
+ auth: {
+ # ...
+ transitioning: true,
+ },
+}
```
After you have made this change, your [Prometheus query](#verify-authentication-monitoring)
@@ -1038,8 +1084,13 @@ To update to a new Gitaly authentication token, on each Gitaly client **and** Gi
```ruby
# in /etc/gitlab/gitlab.rb
-
- gitaly['auth_token'] = '<new secret token>'
+ gitaly['configuration'] = {
+ # ...
+ auth: {
+ # ...
+ token: '<new secret token>',
+ },
+ }
```
1. Restart Gitaly:
@@ -1069,7 +1120,13 @@ your Gitaly servers as follows:
```ruby
# in /etc/gitlab/gitlab.rb
-gitaly['auth_transitioning'] = false
+gitaly['configuration'] = {
+ # ...
+ auth: {
+ # ...
+ transitioning: false,
+ },
+}
```
WARNING:
@@ -1088,17 +1145,11 @@ result as you did at the start. For example:
## Pack-objects cache **(FREE SELF)**
-> - [Introduced](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/372) in GitLab 13.11.
-> - It's enabled on GitLab.com.
-> - It's recommended for production use.
-
[Gitaly](index.md), the service that provides storage for Git
repositories, can be configured to cache a short rolling window of Git
fetch responses. This can reduce server load when your server receives
lots of CI fetch traffic.
-### Overview
-
The pack-objects cache wraps `git pack-objects`, an internal part of
Git that gets invoked indirectly via the PostUploadPack and
SSHUploadPack Gitaly RPCs. Gitaly runs PostUploadPack when a
@@ -1131,31 +1182,40 @@ automatically keep parts of the pack-objects cache files in RAM,
making it faster.
Because the pack-objects cache can lead to a significant increase in
-disk write IO, it is off by default.
+disk write IO, it is off by default. In GitLab 15.11 and later,
+the write workload is approximately 50% lower, but the cache is still disabled by default.
### Configure the cache
-These are the configuration settings for the pack-objects cache. Each
-setting is discussed in greater detail below.
+These configuration settings are available for the pack-objects cache. Each setting is discussed in greater detail
+below.
-|Setting|Default|Description|
-|:---|:---|:---|
-|`enabled`|`false`|Turns on the cache. When off, Gitaly runs a dedicated `git pack-objects` process for each request. |
-|`dir`|`<PATH TO FIRST STORAGE>/+gitaly/PackObjectsCache`|Local directory where cache files get stored.|
-|`max_age`|`5m` (5 minutes)|Cache entries older than this get evicted and removed from disk.|
+| Setting | Default | Description |
+|:----------|:---------------------------------------------------|:---------------------------------------------------------------------------------------------------|
+| `enabled` | `false` | Turns on the cache. When off, Gitaly runs a dedicated `git pack-objects` process for each request. |
+| `dir` | `<PATH TO FIRST STORAGE>/+gitaly/PackObjectsCache` | Local directory where cache files get stored. |
+| `max_age` | `5m` (5 minutes) | Cache entries older than this get evicted and removed from disk. |
+| `min_occurrences` | 1 | Minimum times a key must occur before a cache entry is created. |
In `/etc/gitlab/gitlab.rb`, set:
```ruby
-gitaly['pack_objects_cache_enabled'] = true
-## gitaly['pack_objects_cache_dir'] = '/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache'
-## gitaly['pack_objects_cache_max_age'] = '5m'
+gitaly['configuration'] = {
+ # ...
+ pack_objects_cache: {
+ # ...
+ enabled: true,
+ # dir: '/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache',
+ # max_age: '5m',
+ # min_occurrences: 1,
+ },
+}
```
#### `enabled` defaults to `false`
-The cache is disabled by default. This is because in some cases, it
-can create an [extreme increase](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4010#note_534564684)
+The cache is disabled by default because in some cases, it can create an
+[extreme increase](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4010#note_534564684)
in the number of bytes written to disk. On GitLab.com, we have verified
that our repository storage disks can handle this extra workload, but
we felt we cannot assume this is true everywhere.
@@ -1198,21 +1258,37 @@ The amount of space required depends on:
- The size of the `max_age` cache eviction window.
If your users pull 100 MB/s and you use a 5 minute window, then on average you have
-`5*60*100MB = 30GB` of data in your cache directory. This is an expected average, not
+`5*60*100MB = 30GB` of data in your cache directory. This average is an expected average, not
a guarantee. Peak size may exceed this average.
#### Cache eviction window `max_age`
The `max_age` configuration setting lets you control the chance of a
cache hit and the average amount of storage used by cache files.
-Entries older than `max_age` get evicted from the in-memory metadata
-store, and deleted from disk.
+Entries older than `max_age` get deleted from the disk.
+
+Eviction does not interfere with ongoing requests. It is OK for `max_age` to be less than the time it takes to do a
+fetch over a slow connection because Unix filesystems do not truly delete a file until all processes that are reading
+the deleted file have closed it.
+
+#### Minimum key occurrences `min_occurrences`
+
+> [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/2222) in GitLab 15.11.
-Eviction does not interfere with ongoing requests, so it is OK
-for `max_age` to be less than the time it takes to do a fetch over a
-slow connection. This is because Unix filesystems do not truly delete
-a file until all processes that are reading the deleted file have
-closed it.
+The `min_occurrences` setting controls how often an identical request
+must occur before we create a new cache entry. The default value is `1`,
+meaning that unique requests do not get written into the cache.
+
+If you:
+
+- Increase this number, your cache hit rate goes down and the
+cache uses less disk space.
+- Decrease this number, your cache hit
+rate goes up and the cache uses more disk space.
+
+You should set `min_occurrences` to `1`. On GitLab.com,
+going from 0 to 1 saved us 50% cache disk space while barely affecting
+the cache hit rate.
### Observe the cache
@@ -1315,7 +1391,32 @@ process repositories that do not pass consistency checks.
For Omnibus GitLab installations, edit `/etc/gitlab/gitlab.rb` and set the
following keys (in this example, to disable the `hasDotgit` consistency check):
-- In [GitLab 15.3](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6800) and later:
+- In [GitLab 15.10](https://gitlab.com/gitlab-org/gitaly/-/issues/4754) and later:
+
+ ```ruby
+ ignored_blobs = "/etc/gitlab/instance_wide_ignored_git_blobs.txt"
+
+ gitaly['configuration'] = {
+ # ...
+ git: {
+ # ...
+ config: [
+ # Populate a file with one unabbreviated SHA-1 per line.
+ # See https://git-scm.com/docs/git-config#Documentation/git-config.txt-fsckskipList
+ { key: "fsck.skipList", value: ignored_blobs },
+ { key: "fetch.fsck.skipList", value: ignored_blobs },
+ { key: "receive.fsck.skipList", value: ignored_blobs },
+
+ { key: "fsck.hasDotgit", value: "ignore" },
+ { key: "fetch.fsck.hasDotgit", value: "ignore" },
+ { key: "receive.fsck.hasDotgit", value: "ignore" },
+ { key: "fsck.missingSpaceBeforeEmail", value: "ignore" },
+ ],
+ },
+ }
+ ```
+
+- In [GitLab 15.3](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6800) to GitLab 15.9:
```ruby
ignored_blobs = "/etc/gitlab/instance_wide_ignored_git_blobs.txt"
@@ -1407,10 +1508,14 @@ By default, Gitaly doesn't sign commits made using GitLab UI. For example, commi
- Web IDE.
- Merge requests.
-You can configure Gitaly to sign commits made using GitLab UI. The commits show as unverified and signed by an unknown user. Support for improvements is
-proposed in issue [19185](https://gitlab.com/gitlab-org/gitlab/-/issues/19185).
+You can configure Gitaly to sign commits made with the GitLab UI. The commits show as unverified and signed by an unknown
+user. Support for improvements is proposed in [issue 19185](https://gitlab.com/gitlab-org/gitlab/-/issues/19185).
+
+Configure Gitaly to sign commits made with the GitLab UI in one of two ways:
+
+::Tabs
-**For Omnibus GitLab**
+:::TabTitle Linux package (Omnibus)
1. [Create a GPG key](../../user/project/repository/gpg_signed_commits/index.md#create-a-gpg-key)
and export it. For optimal performance, consider using an EdDSA key.
@@ -1423,12 +1528,18 @@ proposed in issue [19185](https://gitlab.com/gitlab-org/gitlab/-/issues/19185).
1. Edit `/etc/gitlab/gitlab.rb` and configure `gitaly['gpg_signing_key_path']`:
```ruby
- gitaly['gpg_signing_key_path'] = "/etc/gitlab/gitaly/signing_key.gpg"
+ gitaly['configuration'] = {
+ # ...
+ git: {
+ # ...
+ signing_key: '/etc/gitlab/gitaly/signing_key.gpg',
+ },
+ }
```
1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
-**For installations from source**
+:::TabTitle Self-compiled (source)
1. [Create a GPG key](../../user/project/repository/gpg_signed_commits/index.md#create-a-gpg-key)
and export it. For optimal performance, consider using an EdDSA key.
@@ -1446,3 +1557,60 @@ proposed in issue [19185](https://gitlab.com/gitlab-org/gitlab/-/issues/19185).
```
1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source).
+
+::EndTabs
+
+## Generate configuration using an external command
+
+> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/4828) in GitLab 15.11.
+
+You can generate parts of the Gitaly configuration using an external command. You might do this:
+
+- To configure nodes without having to distribute the full configuration to each of them.
+- To configure using auto-discovery of the node's settings. For example, using DNS entries.
+- To configure secrets at startup of the node, so that don't need to be visible in plain text.
+
+To generate configuration using an external command, you must provide a script that dumps the
+desired configuration of the Gitaly node in JSON format to its standard output.
+
+For example, the following command configures the HTTP password used to connect to the
+GitLab internal API using an AWS secret:
+
+```ruby
+#!/usr/bin/env ruby
+require 'json'
+JSON.generate({"gitlab": {"http_settings": {"password": `aws get-secret-value --secret-id ...`}}})
+```
+
+You must then make the script path known to Gitaly in one of two ways:
+
+::Tabs
+
+:::TabTitle Linux package (Omnibus)
+
+Edit `/etc/gitlab/gitlab.rb` and configure the `config_command`:
+
+```ruby
+gitaly['configuration'] = {
+ config_command: '/path/to/config_command',
+}
+```
+
+:::TabTitle Self-compiled (source)
+
+Edit `/home/git/gitaly/config.toml` and configure `config_command`:
+
+```toml
+config_command = "/path/to/config_command"
+```
+
+::EndTabs
+
+After configuration, Gitaly executes the command on startup and parses its
+standard output as JSON. The resulting configuration is then merged back into
+the other Gitaly configuration.
+
+Gitaly fails to start up if either:
+
+- The configuration command fails.
+- The output produced by the command cannot be parsed as valid JSON.
diff --git a/doc/administration/gitaly/gitaly_geo_capabilities.md b/doc/administration/gitaly/gitaly_geo_capabilities.md
new file mode 100644
index 00000000000..e4147eec162
--- /dev/null
+++ b/doc/administration/gitaly/gitaly_geo_capabilities.md
@@ -0,0 +1,41 @@
+---
+stage: Systems
+group: Gitaly
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
+---
+
+# Gitaly and Geo capabilities
+
+It is common to want the most available, quickly recoverable, highly performant,
+and fully resilient solution for your data. However, there are tradeoffs.
+
+The following tables are intended to guide you to choose the right combination of capabilities based on your requirements.
+
+## Gitaly capabilities
+
+| Capability | Availability | Recoverability | Data Resiliency | Performance | Risks/Trade-offs|
+|------------|--------------|----------------|-----------------|-------------|-----------------|
+|Gitaly Cluster | Very high - tolerant of node failures | RTO for a single node of 10 s with no manual intervention | Data is stored on multiple nodes | Good - While writes may take slightly longer due to voting, read distribution improves read speeds | **Trade-off** - Slight decrease in write speed for redundant, strongly-consistent storage solution. **Risks** - [Does not support snapshot backups](../gitaly/index.md#snapshot-backup-and-recovery-limitations), GitLab backup task can be slow for large data sets |
+|Gitaly Shards | Single storage location is a single point of failure | Would need to restore only shards which failed | Single point of failure | Good - can allocate repositories to shards to spread load | **Trade-off** - Need to manually configure repositories into different shards to balance loads / storage space **Risks** - Single point of failure relies on recovery process when single-node failure occurs |
+|Gitaly + NFS | Single storage location is a single point of failure | Single node failure requires restoration from backup | Single point of failure | Average - NFS is not ideally suited to large quantities of small reads / writes which can have a detrimental impact on performance | **Trade-off** - Familiar administration though NFS is not ideally suited to Git demands **Risks** - Many instances of NFS compatibility issues which provide very poor customer experiences |
+
+## Geo capabilities
+
+If your availability needs to span multiple zones or multiple locations, read about [Geo](../geo/index.md).
+
+| Capability | Availability | Recoverability | Data Resiliency | Performance | Risks/Trade-offs|
+|------------|--------------|----------------|-----------------|-------------|-----------------|
+|Geo| Depends on the architecture of the Geo site. It is possible to deploy secondaries in single and multiple node configurations. | Eventually consistent. Recovery point depends on replication lag, which depends on a number of factors such as network speeds. Geo supports failover from a primary to secondary site using manual commands that are scriptable. | Geo replicates 100% of planned data types and verifies 50%. See [limitations table](../geo/replication/datatypes.md#limitations-on-replicationverification) for more detail. | Improves read/clone times for users of a secondary. | Geo is not intended to replace other backup/restore solutions. Because of replication lag and the possibility of replicating bad data from a primary, customers should also take regular backups of their primary site and test the restore process. |
+
+## Scenarios for failure modes and available mitigation paths
+
+The following table outlines failure modes and mitigation paths for the product offerings detailed in the tables above. Note - Gitaly Cluster install assumes an odd number replication factor of 3 or greater
+
+| Gitaly Mode | Loss of Single Gitaly Node | Application / Data Corruption | Regional Outage (Loss of Instance) | Notes |
+| ----------- | -------------------------- | ----------------------------- | ---------------------------------- | ----- |
+| Single Gitaly Node | Downtime - Must restore from backup | Downtime - Must restore from Backup | Downtime - Must wait for outage to end | |
+| Single Gitaly Node + Geo Secondary | Downtime - Must restore from backup, can perform a manual failover to secondary | Downtime - Must restore from Backup, errors could have propagated to secondary | Manual intervention - failover to Geo secondary | |
+| Sharded Gitaly Install | Partial Downtime - Only repositories on impacted node affected, must restore from backup | Partial Downtime - Only repositories on impacted node affected, must restore from backup | Downtime - Must wait for outage to end | |
+| Sharded Gitaly Install + Geo Secondary | Partial Downtime - Only repositories on impacted node affected, must restore from backup, could perform manual failover to secondary for impacted repositories | Partial Downtime - Only repositories on impacted node affected, must restore from backup, errors could have propagated to secondary | Manual intervention - failover to Geo secondary | |
+| Gitaly Cluster Install* | No Downtime - swaps repository primary to another node after 10 seconds | Not applicable; All writes are voted on by multiple Gitaly Cluster nodes | Downtime - Must wait for outage to end | Snapshot backups for Gitaly Cluster nodes not supported at this time |
+| Gitaly Cluster Install* + Geo Secondary | No Downtime - swaps repository primary to another node after 10 seconds | Not applicable; All writes are voted on by multiple Gitaly Cluster nodes | Manual intervention - failover to Geo secondary | Snapshot backups for Gitaly Cluster nodes not supported at this time |
diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md
index 405c56284cf..359d4ef90dc 100644
--- a/doc/administration/gitaly/index.md
+++ b/doc/administration/gitaly/index.md
@@ -69,7 +69,7 @@ the current status of these issues, refer to the referenced issues and epics.
| Issue | Summary | How to avoid |
|:--------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------|
-| Gitaly Cluster + Geo - Issues retrying failed syncs | If Gitaly Cluster is used on a Geo secondary site, repositories that have failed to sync could continue to fail when Geo tries to resync them. Recovering from this state requires assistance from support to run manual steps. | No known solution prior to GitLab 15.0. In GitLab 15.0 to 15.2, enable the [`gitaly_praefect_generated_replica_paths` feature flag](#praefect-generated-replica-paths-gitlab-150-and-later). In GitLab 15.3, the feature flag is enabled by default. |
+| Gitaly Cluster + Geo - Issues retrying failed syncs | If Gitaly Cluster is used on a Geo secondary site, repositories that have failed to sync could continue to fail when Geo tries to resync them. Recovering from this state requires assistance from support to run manual steps. | No known solution prior to GitLab 15.0. In GitLab 15.0 to 15.2, enable the [`gitaly_praefect_generated_replica_paths` feature flag](#praefect-generated-replica-paths-gitlab-150-and-later) on your Geo primary site. In GitLab 15.3, the feature flag is enabled by default. |
| Praefect unable to insert data into the database due to migrations not being applied after an upgrade | If the database is not kept up to date with completed migrations, then the Praefect node is unable to perform standard operation. | Make sure the Praefect database is up and running with all migrations completed (For example: `/opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-migrate-status` should show a list of all applied migrations). Consider [requesting upgrade assistance](https://about.gitlab.com/support/scheduling-upgrade-assistance/) so your upgrade plan can be reviewed by support. |
| Restoring a Gitaly Cluster node from a snapshot in a running cluster | Because the Gitaly Cluster runs with consistent state, introducing a single node that is behind results in the cluster not being able to reconcile the nodes data and other nodes data | Don't restore a single Gitaly Cluster node from a backup snapshot. If you must restore from backup:<br/><br/>1. [Shut down GitLab](../read_only_gitlab.md#shut-down-the-gitlab-ui).<br/>2. Snapshot all Gitaly Cluster nodes at the same time.<br/>3. Take a database dump of the Praefect database. |
| Rebuilding or replacing an existing Gitaly Cluster node | There is no way to replace existing nodes in place because the Praefect database is relied on to determine the current state of each Gitaly node. Changing how Gitaly Cluster stores repositories is proposed in issue [4218](https://gitlab.com/gitlab-org/gitaly/-/issues/4218). Turning on [repository verification](praefect.md#repository-verification) is proposed in issue [4429](https://gitlab.com/gitlab-org/gitaly/-/issues/4429) so Praefect can detect that data is missing and needs to replicated to a new Gitaly node. | No known solution at this time. |
@@ -78,7 +78,7 @@ the current status of these issues, refer to the referenced issues and epics.
Gitaly Cluster does not support snapshot backups. Snapshot backups can cause issues where the Praefect database becomes
out of sync with the disk storage. Because of how Praefect rebuilds the replication metadata of Gitaly disk information
-during a restore, we recommend using the [official backup and restore Rake tasks](../../raketasks/backup_restore.md).
+during a restore, you should use the [official backup and restore Rake tasks](../../raketasks/backup_restore.md).
The [incremental backup method](../../raketasks/backup_gitlab.md#incremental-repository-backups)
can be used to speed up Gitaly Cluster backups.
@@ -100,7 +100,7 @@ These SSDs should have a throughput of at least:
- 2,000 IOPS for write operations.
These IOPS values are initial recommendations, and may be adjusted to greater or lesser values
-depending on the scale of your environment's workload. If you’re running the environment on a
+depending on the scale of your environment's workload. If you're running the environment on a
cloud provider, refer to their documentation about how to configure IOPS correctly.
For repository data, only local storage is supported for Gitaly and Gitaly Cluster for performance and consistency reasons.
@@ -363,10 +363,10 @@ For example, `@cluster/repositories/6f/96/54771`.
The last component of the replica path, `54771`, is the repository ID. This can be used to identify the repository on the disk.
-`<xx>/<xx>` are the first four hex digits of the SHA256 hash of the string representation of the repository ID. This is used to balance
-the repositories evenly into subdirectories to avoid overly large directories that might cause problems on some file
-systems. In this case, `54771` hashes to `6f960ab01689464e768366d3315b3d3b2c28f38761a58a70110554eb04d582f7` so the
-first four digits are `6f` and `96`.
+`<xx>/<xx>` are the first four hex digits of the SHA256 hash of the string representation of the repository ID. These
+digits are used to balance the repositories evenly into subdirectories to avoid overly large directories that might
+cause problems on some file systems. In this case, `54771` hashes to
+`6f960ab01689464e768366d3315b3d3b2c28f38761a58a70110554eb04d582f7` so the first four digits are `6f` and `96`.
#### Identify repositories on disk
@@ -387,8 +387,9 @@ Follow the [instructions in hashed storage's documentation](../repository_storag
Gitaly Cluster uses the PostgreSQL metadata store with the storage layout to ensure atomicity of repository creation,
deletion, and move operations. The disk operations can't be atomically applied across multiple storages. However, PostgreSQL guarantees
the atomicity of the metadata operations. Gitaly Cluster models the operations in a manner that the failing operations always leave
-the metadata consistent. The disks may contain stale state even after successful operations. This is expected and the leftover state
-does not interfere with future operations but may use up disk space unnecessarily until a clean up is performed.
+the metadata consistent. The disks may contain stale state even after successful operations. This situation is expected and
+the leftover state does not interfere with future operations but may use up disk space unnecessarily until a clean up is
+performed.
There is on-going work on a [background crawler](https://gitlab.com/gitlab-org/gitaly/-/issues/3719) that cleans up the leftover
repositories from the storages.
@@ -397,7 +398,7 @@ repositories from the storages.
When creating repositories, Praefect:
-1. Reserves a repository ID from PostgreSQL. This is atomic and no two creations receive the same ID.
+1. Reserves a repository ID from PostgreSQL, which is atomic and no two creations receive the same ID.
1. Creates replicas on the Gitaly storages in the replica path derived from the repository ID.
1. Creates metadata records after the repository is successfully created on disk.
@@ -470,9 +471,12 @@ _Up to date_ in this context means that:
The primary node is chosen to serve the request if:
-- There are no up to date nodes.
+- No up-to-date nodes exist.
- Any other error occurs during node selection.
+If you have a large, heavily modified repository (like a multi-gigabyte monorepo), the primary node can service most or all requests if changes come in faster than Praefect
+can replicate to the secondaries. When this occurs, CI/CD jobs and other repository traffic are bottlenecked by the capacity of the primary node.
+
You can [monitor distribution of reads](monitoring.md#monitor-gitaly-cluster) using Prometheus.
#### Strong consistency
@@ -677,7 +681,7 @@ Direct Git access is:
For the sake of removing complexity, we must remove direct Git access in GitLab. However, we can't
remove it as long some GitLab installations require Git repositories on NFS.
-There are two facets to our efforts to remove direct Git access in GitLab:
+Two facets of our efforts to remove direct Git access in GitLab are:
- Reduce the number of inefficient Gitaly queries made by GitLab.
- Persuade administrators of fault-tolerant or horizontally-scaled GitLab instances to migrate off
diff --git a/doc/administration/gitaly/monitoring.md b/doc/administration/gitaly/monitoring.md
index 0fd34d5c89f..00d0499faa2 100644
--- a/doc/administration/gitaly/monitoring.md
+++ b/doc/administration/gitaly/monitoring.md
@@ -54,6 +54,9 @@ You can observe the status of [control groups (cgroups)](configure_gitaly.md#con
- `gitaly_cgroups_cpu_usage`, a gauge that measures CPU usage per cgroup.
- `gitaly_cgroup_procs_total`, a gauge that measures the total number of
processes Gitaly has spawned under the control of cgroups.
+- `gitaly_cgroup_cpu_cfs_periods_total`, a counter that for the value of [`nr_periods`](https://docs.kernel.org/scheduler/sched-bwc.html#statistics).
+- `gitaly_cgroup_cpu_cfs_throttled_periods_total`, a counter for the value of [`nr_throttled`](https://docs.kernel.org/scheduler/sched-bwc.html#statistics).
+- `gitaly_cgroup_cpu_cfs_throttled_seconds_total`, a counter for the value of [`throttled_time`](https://docs.kernel.org/scheduler/sched-bwc.html#statistics) in seconds.
## `pack-objects` cache
@@ -86,9 +89,9 @@ gitaly_streamcache_filestore_removed_total{dir="/var/opt/gitlab/git-data/reposit
gitaly_streamcache_index_entries{dir="/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache"} 1
```
-## Useful queries
+## Queries
-The following are useful queries for monitoring Gitaly:
+The following are some queries for monitoring Gitaly:
- Use the following Prometheus query to observe the
[type of connections](configure_gitaly.md#enable-tls-support) Gitaly is serving a production
@@ -130,8 +133,8 @@ The following are useful queries for monitoring Gitaly:
## Monitor Gitaly Cluster
-To monitor Gitaly Cluster (Praefect), you can use these Prometheus metrics. There are two separate metrics
-endpoints from which metrics can be scraped:
+To monitor Gitaly Cluster (Praefect), you can use these Prometheus metrics. Two separate metrics endpoints are
+available from which metrics can be scraped:
- The default `/metrics` endpoint.
- `/db_metrics`, which contains metrics that require database queries.
@@ -153,6 +156,7 @@ The following metrics are available from the `/metrics` endpoint:
- `gitaly_praefect_replication_delay_bucket`, a histogram measuring how much time passes between
when the replication job is created and when it starts. Available in GitLab 12.10 and later.
- `gitaly_praefect_connections_total`, the total number of connections to Praefect. [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4220) in GitLab 14.7.
+- `gitaly_praefect_method_types`, a count of accessor and mutator RPCs per node.
To monitor [strong consistency](index.md#strong-consistency), you can use the following Prometheus metrics:
@@ -179,9 +183,6 @@ To monitor [repository verification](praefect.md#repository-verification), use t
- `gitaly_praefect_stale_verification_leases_released_total`, the number of stale verification leases
released.
-The `/metrics` endpoint also provides all the metrics available under the `/db_metrics` endpoint. Using `/metrics` for `/db_metrics` metrics
-is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/390266) in GitLab 15.9 and will be removed in GitLab 16.0.
-
You can also monitor the [Praefect logs](../logs/index.md#praefect-logs).
### Database metrics `/db_metrics` endpoint
diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md
index 440bd7427ae..51201ec442f 100644
--- a/doc/administration/gitaly/praefect.md
+++ b/doc/administration/gitaly/praefect.md
@@ -142,7 +142,7 @@ with secure tokens as you complete the setup process.
1. `PRAEFECT_EXTERNAL_TOKEN`: repositories hosted on your Praefect cluster can
only be accessed by Gitaly clients that carry this token.
1. `PRAEFECT_INTERNAL_TOKEN`: this token is used for replication traffic inside
- your Praefect cluster. This is distinct from `PRAEFECT_EXTERNAL_TOKEN`
+ your Praefect cluster. This token is distinct from `PRAEFECT_EXTERNAL_TOKEN`
because Gitaly clients must not be able to access internal nodes of the
Praefect cluster directly; that could lead to data loss.
1. `PRAEFECT_SQL_PASSWORD`: this password is used by Praefect to connect to
@@ -269,11 +269,16 @@ The database used by Praefect is now configured.
You can now configure Praefect to use the database:
```ruby
-praefect['database_host'] = POSTGRESQL_HOST
-praefect['database_port'] = 5432
-praefect['database_user'] = 'praefect'
-praefect['database_password'] = PRAEFECT_SQL_PASSWORD
-praefect['database_dbname'] = 'praefect_production'
+praefect['configuration'] = {
+ # ...
+ database: {
+ # ...
+ host: POSTGRESQL_HOST,
+ port: 5432,
+ password: PRAEFECT_SQL_PASSWORD,
+ dbname: 'praefect_production',
+ }
+}
```
If you see Praefect database errors after configuring PostgreSQL, see
@@ -285,19 +290,27 @@ Praefect performance can be improved by additionally configuring the `database_d
settings:
```ruby
-praefect['database_direct_host'] = POSTGRESQL_HOST
-praefect['database_direct_port'] = 5432
-
-# Use the following to override parameters of direct database connection.
-# Comment out where the parameters are the same for both connections.
-
-praefect['database_direct_user'] = 'praefect'
-praefect['database_direct_password'] = PRAEFECT_SQL_PASSWORD
-praefect['database_direct_dbname'] = 'praefect_production'
-#praefect['database_direct_sslmode'] = '...'
-#praefect['database_direct_sslcert'] = '...'
-#praefect['database_direct_sslkey'] = '...'
-#praefect['database_direct_sslrootcert'] = '...'
+praefect['configuration'] = {
+ # ...
+ database: {
+ # ...
+ session_pooled: {
+ # ...
+ host: POSTGRESQL_HOST,
+ port: 5432
+
+ # Use the following to override parameters of direct database connection.
+ # Comment out where the parameters are the same for both connections.
+ user: 'praefect',
+ password: PRAEFECT_SQL_PASSWORD,
+ dbname: 'praefect_production',
+ # sslmode: '...',
+ # sslcert: '...',
+ # sslkey: '...',
+ # sslrootcert: '...',
+ }
+ }
+}
```
When configured, this connection is automatically used for the
@@ -313,8 +326,8 @@ reads distribution caching is enabled by configuration
#### Use PgBouncer
-To reduce PostgreSQL resource consumption, we recommend setting up and configuring
-[PgBouncer](https://www.pgbouncer.org/) in front of the PostgreSQL instance. However, PgBouncer isn't required because
+To reduce PostgreSQL resource consumption, you should set up and configure [PgBouncer](https://www.pgbouncer.org/) in
+front of the PostgreSQL instance. However, PgBouncer isn't required because
Praefect makes a low number of connections. If you choose to use PgBouncer, you can use the same PgBouncer instance for
both the GitLab application database and the Praefect database.
@@ -322,15 +335,21 @@ To configure PgBouncer in front of the PostgreSQL instance, you must point Praef
parameters on Praefect configuration:
```ruby
-praefect['database_host'] = PGBOUNCER_HOST
-praefect['database_port'] = 6432
-praefect['database_user'] = 'praefect'
-praefect['database_password'] = PRAEFECT_SQL_PASSWORD
-praefect['database_dbname'] = 'praefect_production'
-#praefect['database_sslmode'] = '...'
-#praefect['database_sslcert'] = '...'
-#praefect['database_sslkey'] = '...'
-#praefect['database_sslrootcert'] = '...'
+praefect['configuration'] = {
+ # ...
+ database: {
+ # ...
+ host: PGBOUNCER_HOST,
+ port: 6432,
+ user: 'praefect',
+ password: PRAEFECT_SQL_PASSWORD,
+ dbname: 'praefect_production',
+ # sslmode: '...',
+ # sslcert: '...',
+ # sslkey: '...',
+ # sslrootcert: '...',
+ }
+}
```
Praefect requires an additional connection to the PostgreSQL that supports the
@@ -341,12 +360,12 @@ It is not supported in `transaction` pool mode (`pool_mode = transaction`).
To configure the additional connection, you must either:
- Configure a new PgBouncer database that uses to the same PostgreSQL database endpoint,
- but with different pool mode. That is, `pool_mode = session`.
+ but with different pool mode (`pool_mode = session`).
- Connect Praefect directly to PostgreSQL and bypass PgBouncer.
#### Configure a new PgBouncer database with `pool_mode = session`
-We recommend using PgBouncer with `session` pool mode. You can use the
+You should use PgBouncer with `session` pool mode. You can use the
[bundled PgBouncer](../postgresql/pgbouncer.md) or use an external PgBouncer and
[configure it manually](https://www.pgbouncer.org/config.html).
@@ -399,23 +418,30 @@ praefect_production_direct = host=POSTGRESQL_HOST auth_user=pgbouncer dbname=pra
Now you can configure Praefect to use PgBouncer for both connections:
```ruby
-praefect['database_host'] = PGBOUNCER_HOST
-praefect['database_port'] = 6432
-praefect['database_user'] = 'praefect'
-# `PRAEFECT_SQL_PASSWORD` is the plain-text password of
-# Praefect user. Not to be confused with `PRAEFECT_SQL_PASSWORD_HASH`.
-praefect['database_password'] = PRAEFECT_SQL_PASSWORD
-
-praefect['database_dbname'] = 'praefect_production'
-praefect['database_direct_dbname'] = 'praefect_production_direct'
-
-# There is no need to repeat the following. Parameters of direct
-# database connection will fall back to the values above.
-
-#praefect['database_direct_host'] = PGBOUNCER_HOST
-#praefect['database_direct_port'] = 6432
-#praefect['database_direct_user'] = 'praefect'
-#praefect['database_direct_password'] = PRAEFECT_SQL_PASSWORD
+praefect['configuration'] = {
+ # ...
+ database: {
+ # ...
+ host: PGBOUNCER_HOST,
+ port: 6432,
+ user: 'praefect',
+ # `PRAEFECT_SQL_PASSWORD` is the plain-text password of
+ # Praefect user. Not to be confused with `PRAEFECT_SQL_PASSWORD_HASH`.
+ password: PRAEFECT_SQL_PASSWORD,
+ dbname: 'praefect_production',
+ session_pooled: {
+ # ...
+ dbname: 'praefect_production_direct',
+ # There is no need to repeat the following. Parameters of direct
+ # database connection will fall back to the values above.
+ #
+ # host: PGBOUNCER_HOST,
+ # port: 6432,
+ # user: 'praefect',
+ # password: PRAEFECT_SQL_PASSWORD,
+ },
+ },
+}
```
With this configuration, Praefect uses PgBouncer for both connection types.
@@ -428,25 +454,34 @@ configuration option is set. For more details, consult the PgBouncer documentati
#### Configure Praefect to connect directly to PostgreSQL
-As an alternative to configuring PgBouncer with `session` pool mode, Praefect can be configured to use different connection parameters for direct access
-to PostgreSQL. This is the connection that supports the `LISTEN` feature.
+As an alternative to configuring PgBouncer with `session` pool mode, Praefect can be configured to use different
+connection parameters for direct access to PostgreSQL. This connection supports the `LISTEN` feature.
An example of Praefect configuration that bypasses PgBouncer and directly connects to PostgreSQL:
```ruby
-praefect['database_direct_host'] = POSTGRESQL_HOST
-praefect['database_direct_port'] = 5432
-
-# Use the following to override parameters of direct database connection.
-# Comment out where the parameters are the same for both connections.
-
-praefect['database_direct_user'] = 'praefect'
-praefect['database_direct_password'] = PRAEFECT_SQL_PASSWORD
-praefect['database_direct_dbname'] = 'praefect_production'
-#praefect['database_direct_sslmode'] = '...'
-#praefect['database_direct_sslcert'] = '...'
-#praefect['database_direct_sslkey'] = '...'
-#praefect['database_direct_sslrootcert'] = '...'
+praefect['configuration'] = {
+ # ...
+ database: {
+ # ...
+ session_pooled: {
+ # ...
+ host: POSTGRESQL_HOST,
+ port: 5432,
+
+ # Use the following to override parameters of direct database connection.
+ # Comment out where the parameters are the same for both connections.
+ #
+ user: 'praefect',
+ password: PRAEFECT_SQL_PASSWORD,
+ dbname: 'praefect_production',
+ # sslmode: '...',
+ # sslcert: '...',
+ # sslkey: '...',
+ # sslrootcert: '...',
+ },
+ },
+}
```
### Praefect
@@ -501,30 +536,42 @@ Updates to example must be made at:
`/etc/gitlab/gitlab.rb`:
```ruby
- praefect['listen_addr'] = '0.0.0.0:2305'
+ praefect['configuration'] = {
+ # ...
+ listen_addr: '0.0.0.0:2305',
+ }
```
1. Configure Prometheus metrics by editing
`/etc/gitlab/gitlab.rb`:
```ruby
- # Enable Prometheus metrics access to Praefect. You must use firewalls
- # to restrict access to this address/port.
- # The default metrics endpoint is /metrics
- praefect['prometheus_listen_addr'] = '0.0.0.0:9652'
-
- # Some metrics run queries against the database. Enabling separate database metrics allows
- # these metrics to be collected when the metrics are
- # scraped on a separate /db_metrics endpoint.
- praefect['separate_database_metrics'] = true
+ praefect['configuration'] = {
+ # ...
+ #
+ # Enable Prometheus metrics access to Praefect. You must use firewalls
+ # to restrict access to this address/port.
+ # The default metrics endpoint is /metrics
+ prometheus_listen_addr: '0.0.0.0:9652',
+ # Some metrics run queries against the database. Enabling separate database metrics allows
+ # these metrics to be collected when the metrics are
+ # scraped on a separate /db_metrics endpoint.
+ prometheus_exclude_database_from_default_metrics: true,
+ }
```
-1. Configure a strong `auth_token` for **Praefect** by editing
- `/etc/gitlab/gitlab.rb`. This is needed by clients outside the cluster
+1. Configure a strong authentication token for **Praefect** by editing
+ `/etc/gitlab/gitlab.rb`, which is needed by clients outside the cluster
(like GitLab Shell) to communicate with the Praefect cluster:
```ruby
- praefect['auth_token'] = 'PRAEFECT_EXTERNAL_TOKEN'
+ praefect['configuration'] = {
+ # ...
+ auth: {
+ # ...
+ token: 'PRAEFECT_EXTERNAL_TOKEN',
+ },
+ }
```
1. Configure **Praefect** to [connect to the PostgreSQL database](#postgresql). We
@@ -533,19 +580,32 @@ Updates to example must be made at:
If you want to use a TLS client certificate, the options below can be used:
```ruby
- # Connect to PostgreSQL using a TLS client certificate
- # praefect['database_sslcert'] = '/path/to/client-cert'
- # praefect['database_sslkey'] = '/path/to/client-key'
-
- # Trust a custom certificate authority
- # praefect['database_sslrootcert'] = '/path/to/rootcert'
+ praefect['configuration'] = {
+ # ...
+ database: {
+ # ...
+ #
+ # Connect to PostgreSQL using a TLS client certificate
+ # sslcert: '/path/to/client-cert',
+ # sslkey: '/path/to/client-key',
+ #
+ # Trust a custom certificate authority
+ # sslrootcert: '/path/to/rootcert',
+ },
+ }
```
By default, Praefect refuses to make an unencrypted connection to
PostgreSQL. You can override this by uncommenting the following line:
```ruby
- # praefect['database_sslmode'] = 'disable'
+ praefect['configuration'] = {
+ # ...
+ database: {
+ # ...
+ # sslmode: 'disable',
+ },
+ }
```
1. Configure the **Praefect** cluster to connect to each Gitaly node in the
@@ -573,29 +633,37 @@ Updates to example must be made at:
NOTE:
When adding additional Gitaly nodes to a virtual storage, all storage names
- within that virtual storage must be unique. Additionally, all Gitaly node
+ in that virtual storage must be unique. Additionally, all Gitaly node
addresses referenced in the Praefect configuration must be unique.
```ruby
# Name of storage hash must match storage name in git_data_dirs on GitLab
- # server ('default') and in git_data_dirs on Gitaly nodes ('gitaly-1')
- praefect['virtual_storages'] = {
- 'default' => {
- 'nodes' => {
- 'gitaly-1' => {
- 'address' => 'tcp://GITALY_HOST_1:8075',
- 'token' => 'PRAEFECT_INTERNAL_TOKEN',
- },
- 'gitaly-2' => {
- 'address' => 'tcp://GITALY_HOST_2:8075',
- 'token' => 'PRAEFECT_INTERNAL_TOKEN'
+ # server ('default') and in gitaly['configuration'][:storage][INDEX][:name] on Gitaly nodes ('gitaly-1')
+ praefect['configuration'] = {
+ # ...
+ virtual_storage: [
+ {
+ # ...
+ name: 'default',
+ node: [
+ {
+ storage: 'gitaly-1',
+ address: 'tcp://GITALY_HOST_1:8075',
+ token: 'PRAEFECT_INTERNAL_TOKEN'
+ },
+ {
+ storage: 'gitaly-2',
+ address: 'tcp://GITALY_HOST_2:8075',
+ token: 'PRAEFECT_INTERNAL_TOKEN'
+ },
+ {
+ storage: 'gitaly-3',
+ address: 'tcp://GITALY_HOST_3:8075',
+ token: 'PRAEFECT_INTERNAL_TOKEN'
+ },
+ ],
},
- 'gitaly-3' => {
- 'address' => 'tcp://GITALY_HOST_3:8075',
- 'token' => 'PRAEFECT_INTERNAL_TOKEN'
- }
- }
- }
+ ],
}
```
@@ -662,7 +730,7 @@ for secure connections, you must:
Additionally the certificate, or its certificate authority, must be installed on all Gitaly servers
and on all Praefect clients that communicate with it following the procedure described in
-[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates) (and repeated below).
+[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl/index.html#install-custom-public-certificates) (and repeated below).
Note the following:
@@ -681,7 +749,14 @@ Note the following:
This allows you to do a gradual transition from unencrypted to encrypted traffic, if
necessary.
- To disable the unencrypted listener, set `praefect['listen_addr'] = nil`.
+ To disable the unencrypted listener, set:
+
+ ```ruby
+ praefect['configuration'] = {
+ # ...
+ listen_addr: nil,
+ }
+ ```
To configure Praefect with TLS:
@@ -702,9 +777,15 @@ To configure Praefect with TLS:
1. Edit `/etc/gitlab/gitlab.rb` and add:
```ruby
- praefect['tls_listen_addr'] = "0.0.0.0:3305"
- praefect['certificate_path'] = "/etc/gitlab/ssl/cert.pem"
- praefect['key_path'] = "/etc/gitlab/ssl/key.pem"
+ praefect['configuration'] = {
+ # ...
+ tls_listen_addr: '0.0.0.0:3305',
+ tls: {
+ # ...
+ certificate_path: '/etc/gitlab/ssl/cert.pem',
+ key_path: '/etc/gitlab/ssl/key.pem',
+ },
+ }
```
1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure).
@@ -790,6 +871,125 @@ To configure Praefect with TLS:
1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source).
+#### Service discovery
+
+> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/8971) in GitLab 15.10.
+
+Prerequisites:
+
+- A DNS server.
+
+GitLab uses service discovery to retrieve a list of Praefect hosts. Service
+discovery involves periodic checks of a DNS A or AAAA record, with the IPs
+retrieved from the record serving as the addresses of the target nodes.
+Praefect does not support service discovery by SRV record.
+
+By default, the minimum time between checks is 5 minutes, regardless of the
+records' TTLs. Praefect does not support customizing this interval. When clients
+receive an update, they:
+
+- Establish new connections to the new IP addresses.
+- Keep existing connections to intact IP addresses.
+- Drop connections to removed IP addresses.
+
+In-flight requests on to-be-removed connections are still handled until they
+finish. Workhorse has a 10-minute timeout, while other clients do not specify a
+graceful timeout.
+
+The DNS server should return all IP addresses instead of load-balancing itself.
+Clients can distribute requests to IP addresses in a round-robin fashion.
+
+Before updating client configuration, ensure that DNS service discovery works
+correctly. It should return the list of IP addresses correctly. `dig` is a good
+tool to use to verify.
+
+```console
+❯ dig A praefect.service.consul @127.0.0.1
+
+; <<>> DiG 9.10.6 <<>> A praefect.service.consul @127.0.0.1
+;; global options: +cmd
+;; Got answer:
+;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29210
+;; flags: qr aa rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
+
+;; OPT PSEUDOSECTION:
+; EDNS: version: 0, flags:; udp: 4096
+;; QUESTION SECTION:
+;praefect.service.consul. IN A
+
+;; ANSWER SECTION:
+praefect.service.consul. 0 IN A 10.0.0.3
+praefect.service.consul. 0 IN A 10.0.0.2
+praefect.service.consul. 0 IN A 10.0.0.1
+
+;; Query time: 0 msec
+;; SERVER: ::1#53(::1)
+;; WHEN: Wed Dec 14 12:53:58 +07 2022
+;; MSG SIZE rcvd: 86
+```
+
+##### Configure service discovery
+
+By default, Praefect delegates DNS resolution to the operating system. In such
+cases, the Gitaly address can be set in either of these formats:
+
+- `dns:[host]:[port]`
+- `dns:///[host]:[port]` (note the three slashes)
+
+You can also appoint an authoritative name server by setting it in this format:
+
+- `dns://[authority_host]:[authority_port]/[host]:[port]`
+
+::Tabs
+
+:::TabTitle Linux package (Omnibus)
+
+1. Edit `/etc/gitlab/gitlab.rb` and add:
+
+ ```ruby
+ praefect['consul_service_name'] = 'praefect'
+ ```
+
+1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure).
+1. On the Praefect clients (except Gitaly servers), edit `git_data_dirs` in
+`/etc/gitlab/gitlab.rb` as follows. Replace `PRAEFECT_SERVICE_DISCOVERY_ADDRESS`
+with Praefect service discovery address, such as `praefect.service.consul`.
+
+ ```ruby
+ git_data_dirs({
+ "default" => {
+ "gitaly_address" => 'dns:PRAEFECT_SERVICE_DISCOVERY_ADDRESS:2305',
+ "gitaly_token" => 'PRAEFECT_EXTERNAL_TOKEN'
+ }
+ })
+ ```
+
+1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
+
+:::TabTitle Self-compiled (source)
+
+1. Install a DNS service discovery service. Register all Praefect nodes with the service.
+1. On the Praefect clients (except Gitaly servers), edit `storages` in
+ `/home/git/gitlab/config/gitlab.yml` as follows:
+
+ ```yaml
+ gitlab:
+ repositories:
+ storages:
+ default:
+ gitaly_address: dns:PRAEFECT_SERVICE_DISCOVERY_ADDRESS:2305
+ path: /some/local/path
+ ```
+
+ NOTE:
+ `/some/local/path` should be set to a local folder that exists, however no
+ data is stored in this folder. [Issue 375254](https://gitlab.com/gitlab-org/gitlab/-/issues/375254)
+ proposes to remove this requirement.
+
+1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source).
+
+::EndTabs
+
### Gitaly
NOTE:
@@ -813,12 +1013,11 @@ because we rely on Praefect to route operations correctly.
Particular attention should be shown to:
-- The `gitaly['auth_token']` configured in this section must match the `token`
- value under `praefect['virtual_storages']['nodes']` on the Praefect node. This was set
- in the [previous section](#praefect). This document uses the placeholder
- `PRAEFECT_INTERNAL_TOKEN` throughout.
-- The storage names in `git_data_dirs` configured in this section must match the
- storage names under `praefect['virtual_storages']` on the Praefect node. This
+- The `gitaly['configuration'][:auth][:token]` configured in this section must match the `token`
+ value under `praefect['configuration'][:virtual_storage][<index>][:node][<index>][:token]` on the Praefect node. This value was
+ set in the [previous section](#praefect). This document uses the placeholder `PRAEFECT_INTERNAL_TOKEN` throughout.
+- The storage names in `gitaly['configuration'][:storage]` configured in this section must match the
+ storage names under `praefect['configuration'][:virtual_storage]` on the Praefect node. This
was set in the [previous section](#praefect). This document uses `gitaly-1`,
`gitaly-2`, and `gitaly-3` as Gitaly storage names.
@@ -834,7 +1033,7 @@ For more information on Gitaly server configuration, see our
1. Disable all other services by editing `/etc/gitlab/gitlab.rb`:
```ruby
- # Disable all other services on the Praefect node
+ # Disable all other services on the Gitaly node
postgresql['enable'] = false
redis['enable'] = false
nginx['enable'] = false
@@ -859,22 +1058,31 @@ For more information on Gitaly server configuration, see our
`/etc/gitlab/gitlab.rb`:
```ruby
- # Make Gitaly accept connections on all network interfaces.
- # Use firewalls to restrict access to this address/port.
- gitaly['listen_addr'] = '0.0.0.0:8075'
-
- # Enable Prometheus metrics access to Gitaly. You must use firewalls
- # to restrict access to this address/port.
- gitaly['prometheus_listen_addr'] = '0.0.0.0:9236'
+ gitaly['configuration'] = {
+ # ...
+ #
+ # Make Gitaly accept connections on all network interfaces.
+ # Use firewalls to restrict access to this address/port.
+ listen_addr: '0.0.0.0:8075',
+ # Enable Prometheus metrics access to Gitaly. You must use firewalls
+ # to restrict access to this address/port.
+ prometheus_listen_addr: '0.0.0.0:9236',
+ }
```
1. Configure a strong `auth_token` for **Gitaly** by editing
- `/etc/gitlab/gitlab.rb`. This is needed by clients to communicate with
+ `/etc/gitlab/gitlab.rb`, which is needed by clients to communicate with
this Gitaly nodes. Typically, this token is the same for all Gitaly
nodes.
```ruby
- gitaly['auth_token'] = 'PRAEFECT_INTERNAL_TOKEN'
+ gitaly['configuration'] = {
+ # ...
+ auth: {
+ # ...
+ token: 'PRAEFECT_INTERNAL_TOKEN',
+ },
+ }
```
1. Configure the GitLab Shell secret token, which is needed for `git push` operations. Either:
@@ -903,13 +1111,13 @@ For more information on Gitaly server configuration, see our
gitlab_rails['internal_api_url'] = 'http://GITLAB_HOST'
```
-1. Configure the storage location for Git data by setting `git_data_dirs` in
+1. Configure the storage location for Git data by setting `gitaly['configuration'][:storage]` in
`/etc/gitlab/gitlab.rb`. Each Gitaly node should have a unique storage name
(such as `gitaly-1`).
- Instead of configuring `git_data_dirs` uniquely for each Gitaly node, it is
+ Instead of configuring `gitaly['configuration'][:storage]` uniquely for each Gitaly node, it is
often easier to have include the configuration for all Gitaly nodes on every
- Gitaly node. This is supported because the Praefect `virtual_storages`
+ Gitaly node. You can do this because the Praefect `virtual_storage`
configuration maps each storage name (such as `gitaly-1`) to a specific node, and
requests are routed accordingly. This means every Gitaly node in your fleet
can share the same configuration.
@@ -918,17 +1126,23 @@ For more information on Gitaly server configuration, see our
# You can include the data dirs for all nodes in the same config, because
# Praefect will only route requests according to the addresses provided in the
# prior step.
- git_data_dirs({
- "gitaly-1" => {
- "path" => "/var/opt/gitlab/git-data"
- },
- "gitaly-2" => {
- "path" => "/var/opt/gitlab/git-data"
- },
- "gitaly-3" => {
- "path" => "/var/opt/gitlab/git-data"
- }
- })
+ gitaly['configuration'] = {
+ # ...
+ storage: [
+ {
+ name: 'gitaly-1',
+ path: '/var/opt/gitlab/git-data',
+ },
+ {
+ name: 'gitaly-2',
+ path: '/var/opt/gitlab/git-data',
+ },
+ {
+ name: 'gitaly-3',
+ path: '/var/opt/gitlab/git-data',
+ },
+ ],
+ }
```
1. Save the changes to `/etc/gitlab/gitlab.rb` and
@@ -967,10 +1181,7 @@ scope of the GitLab documentation.
NOTE:
The load balancer must be configured to accept traffic from the Gitaly nodes in
-addition to the GitLab nodes. Some requests handled by
-[`gitaly-ruby`](configure_gitaly.md#gitaly-ruby) sidecar processes call into the main Gitaly
-process. `gitaly-ruby` uses the Gitaly address set in the GitLab server's
-`git_data_dirs` setting to make this connection.
+addition to the GitLab nodes.
We hope that if you're managing fault-tolerant systems like GitLab, you have a load balancer
of choice already. Some examples include [HAProxy](https://www.haproxy.org/)
@@ -980,8 +1191,8 @@ Big-IP LTM, and Citrix Net Scaler. This documentation outlines what ports
and protocols you need configure.
NOTE:
-We recommend the equivalent of HAProxy `leastconn` load-balancing strategy because long-running operations (for example,
-clones) keep some connections open for extended periods.
+You should use the equivalent of HAProxy `leastconn` load-balancing strategy because long-running operations (for
+example, clones) keep some connections open for extended periods.
| LB Port | Backend Port | Protocol |
|:--------|:-------------|:---------|
@@ -995,12 +1206,12 @@ To complete this section you need:
- [Configured Gitaly nodes](#gitaly)
The Praefect cluster needs to be exposed as a storage location to the GitLab
-application. This is done by updating the `git_data_dirs`.
+application, which is done by updating the `git_data_dirs`.
Particular attention should be shown to:
- the storage name added to `git_data_dirs` in this section must match the
- storage name under `praefect['virtual_storages']` on the Praefect nodes. This
+ storage name under `praefect['configuration'][:virtual_storage]` on the Praefect nodes. This
was set in the [Praefect](#praefect) section of this guide. This document uses
`default` as the Praefect storage name.
@@ -1219,12 +1430,16 @@ You can configure:
The configuration is added to the `/etc/gitlab/gitlab.rb` file:
```ruby
- praefect['virtual_storages'] = {
- 'default' => {
- 'default_replication_factor' => 1,
+ praefect['configuration'] = {
# ...
- }
- }
+ virtual_storage: [
+ {
+ # ...
+ name: 'default',
+ default_replication_factor: 1,
+ },
+ ],
+ }
```
- A replication factor for an existing repository using the `set-replication-factor` sub-command.
@@ -1313,29 +1528,50 @@ interval is configurable with any valid [Go duration string](https://pkg.go.dev/
To verify the metadata every three days:
```ruby
-praefect['background_verification_verification_interval'] = '72h'
+praefect['configuration'] = {
+ # ...
+ background_verification: {
+ # ...
+ verification_interval: '72h',
+ },
+}
```
Values of 0 and below disable the background verifier.
```ruby
-praefect['background_verification_verification_interval'] = '0'
+praefect['configuration'] = {
+ # ...
+ background_verification: {
+ # ...
+ verification_interval: '0',
+ },
+}
```
#### Enable deletions
+> - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/4080) and disabled by default in GitLab 15.0
+> - [Default enabled](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5321) in GitLab 15.9.
+
WARNING:
-Deletions are disabled by default due to a race condition with repository renames that can cause incorrect
-deletions. This is especially prominent in Geo instances as Geo performs more renames than instances without Geo.
-You should enable deletions only if the [`gitaly_praefect_generated_replica_paths` feature flag](index.md#praefect-generated-replica-paths-gitlab-150-and-later) is enabled.
+Deletions were disabled by default prior to GitLab 15.9 due to a race condition with repository renames
+that can cause incorrect deletions, which is especially prominent in Geo instances as Geo performs more renames
+than instances without Geo. In GitLab 15.0 to 15.5, you should enable deletions only if the [`gitaly_praefect_generated_replica_paths` feature flag](index.md#praefect-generated-replica-paths-gitlab-150-and-later) is enabled. The feature flag was removed in GitLab 15.6 making deletions always safe to enable.
-By default, the worker does not delete invalid metadata records but logs them and outputs Prometheus
-metrics for them.
+By default, the worker deletes invalid metadata records. It also logs the deleted records and outputs Prometheus
+metrics.
-You can enable deleting invalid metadata records with:
+You can disable deleting invalid metadata records with:
```ruby
-praefect['background_verification_delete_invalid_records'] = true
+praefect['configuration'] = {
+ # ...
+ background_verification: {
+ # ...
+ delete_invalid_records: false,
+ },
+}
```
### Prioritize verification manually
@@ -1370,10 +1606,10 @@ The output includes the number of replicas that were marked unverified.
## Automatic failover and primary election strategies
-Praefect regularly checks the health of each Gitaly node. This is used to automatically fail over
+Praefect regularly checks the health of each Gitaly node, which is used to automatically fail over
to a newly-elected primary Gitaly node if the current primary node is found to be unhealthy.
-We recommend using [repository-specific primary nodes](#repository-specific-primary-nodes). This is
+You should use [repository-specific primary nodes](#repository-specific-primary-nodes). This is
[the only available election strategy](https://gitlab.com/gitlab-org/gitaly/-/issues/3574) from GitLab 14.0.
### Repository-specific primary nodes
diff --git a/doc/administration/gitaly/recovery.md b/doc/administration/gitaly/recovery.md
index 1207d7af3e7..7d27a633512 100644
--- a/doc/administration/gitaly/recovery.md
+++ b/doc/administration/gitaly/recovery.md
@@ -74,7 +74,7 @@ Cluster:
### Unavailable repositories
> - From GitLab 13.0 through 14.0, repositories became read-only if they were outdated on the primary but fully up to date on a healthy secondary. `dataloss` sub-command displays read-only repositories by default through these versions.
-> - Since GitLab 14.1, Praefect contains more responsive failover logic which immediately fails over to one of the fully up to date secondaries rather than placing the repository in read-only mode. Since GitLab 14.1, the `dataloss` sub-command displays repositories which are unavailable due to having no fully up to date copies on healthy Gitaly nodes.
+> - From GitLab 14.1, Praefect contains more responsive failover logic which immediately fails over to one of the fully up to date secondaries rather than placing the repository in read-only mode. From GitLab 14.1, the `dataloss` sub-command displays repositories which are unavailable due to having no fully up to date copies on healthy Gitaly nodes.
A repository is unavailable if all of its up to date replicas are unavailable. Unavailable repositories are
not accessible through Praefect to prevent serving stale data that may break automated tooling.
@@ -100,7 +100,7 @@ The following parameters are available:
some assigned copies that are not available.
NOTE:
-`dataloss` is still in [Beta](../../policy/alpha-beta-support.md#beta-features) and the output format is subject to change.
+`dataloss` is still in [Beta](../../policy/alpha-beta-support.md#beta) and the output format is subject to change.
To check for repositories with outdated primaries or for unavailable repositories, run:
@@ -277,15 +277,33 @@ The reconciliation frequency can be changed via the configuration. The value can
Examples:
```ruby
-praefect['reconciliation_scheduling_interval'] = '5m' # the default value
+praefect['configuration'] = {
+ # ...
+ reconciliation: {
+ # ...
+ scheduling_interval: '5m', # the default value
+ },
+}
```
```ruby
-praefect['reconciliation_scheduling_interval'] = '30s' # reconcile every 30 seconds
+praefect['configuration'] = {
+ # ...
+ reconciliation: {
+ # ...
+ scheduling_interval: '30s', # reconcile every 30 seconds
+ },
+}
```
```ruby
-praefect['reconciliation_scheduling_interval'] = '0' # disable the feature
+praefect['configuration'] = {
+ # ...
+ reconciliation: {
+ # ...
+ scheduling_interval: '0', # disable the feature
+ },
+}
```
### Manual reconciliation
@@ -334,16 +352,21 @@ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.t
sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml remove-repository -virtual-storage <virtual-storage> -repository <repository> -apply
```
-- `-virtual-storage` is the virtual storage the repository is located in. Virtual storages are configured in `/etc/gitlab/gitlab.rb` under `praefect['virtual_storages]` and looks like the following:
+- `-virtual-storage` is the virtual storage the repository is located in. Virtual storages are configured in `/etc/gitlab/gitlab.rb` under `praefect['configuration']['virtual_storage]` and looks like the following:
```ruby
- praefect['virtual_storages'] = {
- 'default' => {
- ...
- },
- 'storage-1' => {
- ...
- }
+ praefect['configuration'] = {
+ # ...
+ virtual_storage: [
+ {
+ # ...
+ name: 'default',
+ },
+ {
+ # ...
+ name: 'storage-1',
+ },
+ ],
}
```
@@ -415,16 +438,21 @@ The `track-repository` Praefect sub-command adds repositories on disk to the Pra
sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml track-repository -virtual-storage <virtual-storage> -authoritative-storage <storage-name> -repository <repository> -replicate-immediately
```
-- `-virtual-storage` is the virtual storage the repository is located in. Virtual storages are configured in `/etc/gitlab/gitlab.rb` under `praefect['virtual_storages]` and looks like the following:
+- `-virtual-storage` is the virtual storage the repository is located in. Virtual storages are configured in `/etc/gitlab/gitlab.rb` under `praefect['configuration'][:virtual_storage]` and looks like the following:
```ruby
- praefect['virtual_storages'] = {
- 'default' => {
- ...
- },
- 'storage-1' => {
- ...
- }
+ praefect['configuration'] = {
+ # ...
+ virtual_storage: [
+ {
+ # ...
+ name: 'default',
+ },
+ {
+ # ...
+ name: 'storage-1',
+ },
+ ],
}
```
@@ -473,7 +501,7 @@ The command validates that all entries:
- Are formatted correctly and contain required fields.
- Correspond to a valid Git repository on disk.
-- Are not currently tracked in the Praefect tracking database.
+- Are not tracked in the Praefect tracking database.
If any entry fails these checks, the command aborts prior to attempting to track a repository.
diff --git a/doc/administration/gitaly/reference.md b/doc/administration/gitaly/reference.md
index 1516b82a906..81b3faf859e 100644
--- a/doc/administration/gitaly/reference.md
+++ b/doc/administration/gitaly/reference.md
@@ -8,11 +8,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w
Gitaly is configured via a [TOML](https://github.com/toml-lang/toml)
configuration file. Unlike installations from source, in Omnibus GitLab, you
-would not edit this file directly.
+would not edit this file directly. For Omnibus GitLab installations, the default file location is `/var/opt/gitlab/gitaly/config.toml`.
-The configuration file is passed as an argument to the `gitaly`
-executable. This is usually done by either Omnibus GitLab or your
-[init](https://en.wikipedia.org/wiki/Init) script.
+The configuration file is passed as an argument to the `gitaly` executable, which is usually done by either Omnibus
+GitLab or your [init](https://en.wikipedia.org/wiki/Init) script.
An [example configuration file](https://gitlab.com/gitlab-org/gitaly/blob/master/config.toml.example)
can be found in the Gitaly project.
@@ -42,7 +41,7 @@ prometheus_listen_addr = "localhost:9236"
### Authentication
Gitaly can be configured to reject requests that do not contain a
-specific bearer token in their headers. This is a security measure to
+specific bearer token in their headers, which is a security measure to
be used when serving requests over TCP:
```toml
@@ -70,7 +69,7 @@ Remember to disable `transitioning` when you are done
changing your token settings.
All authentication attempts are counted in Prometheus under
-the [`gitaly_authentications_total` metric](monitoring.md#useful-queries).
+the [`gitaly_authentications_total` metric](monitoring.md#queries).
### TLS
@@ -156,36 +155,6 @@ Prometheus query to see the hit rate:
sum(rate(gitaly_catfile_cache_total{type="hit"}[5m])) / sum(rate(gitaly_catfile_cache_total{type=~"(hit)|(miss)"}[5m]))
```
-### `gitaly-ruby`
-
-A Gitaly process uses one or more `gitaly-ruby` helper processes to
-execute RPCs implemented in Ruby instead of Go. The `[gitaly-ruby]`
-section of the configuration file contains settings for these helper processes.
-
-These processes are known to occasionally suffer from memory leaks.
-Gitaly restarts its `gitaly-ruby` helpers when their memory exceeds the
-`max_rss` limit.
-
-| Name | Type | Required | Description |
-| ---- | ---- | -------- | ----------- |
-| `dir` | string | yes | Path to where `gitaly-ruby` is installed (needed to boot the process).|
-| `max_rss` | integer | no | Resident set size limit that triggers a `gitaly-ruby` restart, in bytes. Default is `200000000` (200 MB). |
-| `graceful_restart_timeout` | string | no | Grace period before a `gitaly-ruby` process is forcibly terminated after exceeding `max_rss`. Default is `10m` (10 minutes).|
-| `restart_delay` | string | no |Time that `gitaly-ruby` memory must remain high before a restart. Default is `5m` (5 minutes).|
-| `num_workers` | integer | no |Number of `gitaly-ruby` worker processes. Try increasing this number in case of `ResourceExhausted` errors. Default is `2`, minimum is `2`.|
-| `linguist_languages_path` | string | no | Override for dynamic `languages.json` discovery. Defaults to an empty string (use of dynamic discovery).|
-
-Example:
-
-```toml
-[gitaly-ruby]
-dir = "/home/git/gitaly/ruby"
-max_rss = 200000000
-graceful_restart_timeout = "10m"
-restart_delay = "5m"
-num_workers = 2
-```
-
### GitLab Shell
For historical reasons
@@ -233,7 +202,6 @@ The following values configure logging in Gitaly under the `[logging]` section.
| `level` | string | no | Log level: `debug`, `info`, `warn`, `error`, `fatal`, or `panic`. Default: `info`. |
| `sentry_dsn` | string | no | Sentry DSN (Data Source Name) for exception monitoring. |
| `sentry_environment` | string | no | [Sentry Environment](https://docs.sentry.io/product/sentry-basics/environments/) for exception monitoring. |
-| `ruby_sentry_dsn` | string | no | Sentry DSN (Data Source Name) for `gitaly-ruby` exception monitoring. |
While the main Gitaly application logs go to `stdout`, there are some extra log
files that go to a configured directory, like the GitLab Shell logs.
diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md
index 5f05b6322b0..dbbd348556c 100644
--- a/doc/administration/gitaly/troubleshooting.md
+++ b/doc/administration/gitaly/troubleshooting.md
@@ -66,7 +66,7 @@ for details.
### Client side gRPC logs
Gitaly uses the [gRPC](https://grpc.io/) RPC framework. The Ruby gRPC
-client has its own log file which may contain useful information when
+client has its own log file which may contain helpful information when
you are seeing Gitaly errors. You can control the log level of the
gRPC client with the `GRPC_LOG_LEVEL` environment variable. The
default level is `WARN`.
@@ -85,7 +85,7 @@ check for an SSL or TLS problem:
```
Check whether `Verify return code` field indicates a
-[known Omnibus GitLab configuration problem](https://docs.gitlab.com/omnibus/settings/ssl.html).
+[known Omnibus GitLab configuration problem](https://docs.gitlab.com/omnibus/settings/ssl/index.html).
If `openssl` succeeds but `gitlab-rake gitlab:gitaly:check` fails,
check [certificate requirements](configure_gitaly.md#certificate-requirements) for Gitaly.
@@ -123,27 +123,6 @@ sudo cat /proc/$PID/environ | tr '\0' '\n' | grep ^CORRELATION_ID=
This method isn't reliable for `git cat-file` processes, because Gitaly
internally pools and re-uses those across RPCs.
-### Observing `gitaly-ruby` traffic
-
-[`gitaly-ruby`](configure_gitaly.md#gitaly-ruby) is an internal implementation detail of Gitaly,
-so, there's not that much visibility into what goes on inside
-`gitaly-ruby` processes.
-
-If you have Prometheus set up to scrape your Gitaly process, you can see
-request rates and error codes for individual RPCs in `gitaly-ruby` by
-querying `grpc_client_handled_total`.
-
-All gRPC calls made by `gitaly-ruby` itself are internal calls from the main Gitaly process to one of its `gitaly-ruby`
-sidecars.
-
-Assuming your `grpc_client_handled_total` counter only observes Gitaly,
-the following query shows you RPCs are (most likely) internally
-implemented as calls to `gitaly-ruby`:
-
-```prometheus
-sum(rate(grpc_client_handled_total[5m])) by (grpc_method) > 0
-```
-
### Repository changes fail with a `401 Unauthorized` error
If you run Gitaly on its own server and notice these conditions:
@@ -345,7 +324,7 @@ You might see the following in Gitaly and Praefect logs:
}
```
-This is a gRPC call
+This information in the logs is a gRPC call
[error response code](https://grpc.github.io/grpc/core/md_doc_statuscodes.html).
If this error occurs, even though
@@ -358,7 +337,7 @@ server to keep them synchronized.
### Gitaly not listening on new address after reconfiguring
-When updating the `gitaly['listen_addr']` or `gitaly['prometheus_listen_addr']` values, Gitaly may
+When updating the `gitaly['configuration'][:listen_addr]` or `gitaly['configuration'][:prometheus_listen_addr]` values, Gitaly may
continue to listen on the old address after a `sudo gitlab-ctl reconfigure`.
When this occurs, run `sudo gitlab-ctl restart` to resolve the issue. This should no longer be
@@ -401,6 +380,12 @@ push, which causes a significant delay.
If Git pushes are too slow when Dynatrace is enabled, disable Dynatrace.
+### `gitaly check` fails with `401` status code
+
+`gitaly check` can fail with `401` status code if Gitaly can't access the internal GitLab API.
+
+One way to resolve this is to make sure the entry is correct for the GitLab internal API URL configured in `gitlab.rb` with `gitlab_rails['internal_api_url']`.
+
## Gitaly fails to fork processes stored on `noexec` file systems
Because of changes [introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5999) in GitLab 14.10, applying the `noexec` option to a mount
@@ -492,7 +477,7 @@ in sync so the token check succeeds.
This check helps identify the root cause of `permission denied`
[errors being logged by Praefect](#permission-denied-errors-appearing-in-gitaly-or-praefect-logs-when-accessing-repositories).
-For offline environments where access to public [`pool.ntp.org`](https://pool.ntp.org) servers is not possible, the Praefect `check` sub-command fails this
+For offline environments where access to public `pool.ntp.org` servers is not possible, the Praefect `check` sub-command fails this
check with an error message similar to:
```plaintext
@@ -514,9 +499,9 @@ Here are common errors and potential causes:
- 500 response code
- `ActionView::Template::Error (7:permission denied)`
- - `praefect['auth_token']` and `gitlab_rails['gitaly_token']` do not match on the GitLab server.
+ - `praefect['configuration'][:auth][:token]` and `gitlab_rails['gitaly_token']` do not match on the GitLab server.
- `Unable to save project. Error: 7:permission denied`
- - Secret token in `praefect['storage_nodes']` on GitLab server does not match the
+ - Secret token in `praefect['configuration'][:virtual_storage]` on GitLab server does not match the
value in `gitaly['auth_token']` on one or more Gitaly servers.
- 503 response code
- `GRPC::Unavailable (14:failed to connect to all addresses)`
@@ -530,7 +515,7 @@ Here are common errors and potential causes:
Some common reasons for the Praefect database to experience elevated CPU usage include:
- Prometheus metrics scrapes [running an expensive query](https://gitlab.com/gitlab-org/gitaly/-/issues/3796). If you have GitLab 14.2
- or above, set `praefect['separate_database_metrics'] = true` in `gitlab.rb`.
+ or above, set `praefect['configuration'][:prometheus_exclude_database_from_default_metrics] = true` in `gitlab.rb`.
- [Read distribution caching](praefect.md#reads-distribution-caching) is disabled, increasing the number of queries made to the
database when user traffic is high. Ensure read distribution caching is enabled.
@@ -544,9 +529,8 @@ To determine the primary node of a repository:
- With legacy election strategies in GitLab 13.12 and earlier, the primary was the same for all repositories in a virtual storage.
To determine the current primary Gitaly node for a specific virtual storage:
- - Use the `Shard Primary Election` [Grafana chart](praefect.md#grafana) on the
+ - (Recommended) Use the `Shard Primary Election` [Grafana chart](praefect.md#grafana) on the
[`Gitlab Omnibus - Praefect` dashboard](https://gitlab.com/gitlab-org/grafana-dashboards/-/blob/master/omnibus/praefect.json).
- This is recommended.
- If you do not have Grafana set up, use the following command on each host of each
Praefect node:
@@ -650,7 +634,7 @@ If the supplied value for `-virtual-storage` is incorrect, the command returns t
get metadata: rpc error: code = NotFound desc = repository not found
```
-The documented examples specify `-virtual-storage default`. Check the Praefect server setting `praefect['virtual_storages']` in `/etc/gitlab/gitlab.rb`.
+The documented examples specify `-virtual-storage default`. Check the Praefect server setting `praefect['configuration'][:virtual_storage]` in `/etc/gitlab/gitlab.rb`.
### Check that repositories are in sync
@@ -669,7 +653,7 @@ However, the Praefect database tables are not created on initial reconfigure and
errors that relations do not exist if either:
- The `gitlab-ctl reconfigure` command isn't executed.
-- There are errors during the execution.
+- Errors occur during the execution.
For example:
@@ -693,7 +677,7 @@ praefect sql-migrate: OK (applied 21 migrations)
This indicates that the virtual storage name used in the
[Praefect configuration](praefect.md#praefect) does not match the storage name used in
-[`git_data_dirs` setting](praefect.md#gitaly) for GitLab.
+[`gitaly['configuration'][:storage][<index>][:name]` setting](praefect.md#gitaly) for GitLab.
Resolve this by matching the virtual storage names used in Praefect and GitLab configuration.
@@ -715,9 +699,13 @@ Possible solutions:
- Provision larger VMs to gain access to larger network traffic allowances.
- Use your cloud service's monitoring and logging to check that the Praefect nodes are not exhausting their traffic allowances.
+### `gitlab-ctl reconfigure` fails with error: `STDOUT: praefect: configuration error: error reading config file: toml: cannot store TOML string into a Go int`
+
+This error occurs when `praefect['database_port']` or `praefect['database_direct_port']` are configured as a string instead of an integer.
+
## Profiling Gitaly
-Gitaly exposes several of the Golang built-in performance profiling tools on the Prometheus listen port. For example, if Prometheus is listening
+Gitaly exposes several of the Go built-in performance profiling tools on the Prometheus listen port. For example, if Prometheus is listening
on port `9236` of the GitLab server:
- Get a list of running `goroutines` and their backtraces:
View Lorry Controller Status