diff options
Diffstat (limited to 'doc/administration/gitaly')
| -rw-r--r-- | doc/administration/gitaly/img/gitlab_gitaly_version_mismatch_v12_4.png | bin | 0 -> 21779 bytes | |||
| -rw-r--r-- | doc/administration/gitaly/index.md | 114 | ||||
| -rw-r--r-- | doc/administration/gitaly/praefect.md | 154 | ||||
| -rw-r--r-- | doc/administration/gitaly/reference.md | 16 |
4 files changed, 205 insertions, 79 deletions
diff --git a/doc/administration/gitaly/img/gitlab_gitaly_version_mismatch_v12_4.png b/doc/administration/gitaly/img/gitlab_gitaly_version_mismatch_v12_4.png Binary files differnew file mode 100644 index 00000000000..4d2c5cdb00c --- /dev/null +++ b/doc/administration/gitaly/img/gitlab_gitaly_version_mismatch_v12_4.png diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index 82283650070..9218ffa4006 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -221,6 +221,8 @@ Git operations in GitLab will result in an API error. ```toml listen_addr = '0.0.0.0:8075' + internal_socket_dir = '/var/opt/gitlab/gitaly' + [auth] token = 'abc123secret' @@ -332,7 +334,10 @@ When you tail the Gitaly logs on your Gitaly server you should see requests coming in. One sure way to trigger a Gitaly request is to clone a repository from your GitLab server over HTTP. -DANGER: **Danger:** If you have [custom server-side Git hooks](../custom_hooks.md#custom-server-side-git-hooks) configured, either per repository or globally, you must move these to the Gitaly node. If you have multiple Gitaly nodes, copy your custom hook(s) to all nodes. +DANGER: **Danger:** +If you have [custom server-side Git hooks](../custom_hooks.md) configured, +either per repository or globally, you must move these to the Gitaly node. +If you have multiple Gitaly nodes, copy your custom hook(s) to all nodes. ### Disabling the Gitaly service in a cluster environment @@ -714,8 +719,115 @@ result as you did in the beginning: Note that `enforced="true"`, meaning that authentication is being enforced. +## Direct Git access in GitLab Rails + +Also known as "the Rugged patches". + +### History + +Before Gitaly existed, the things that are now Gitaly clients used to +access Git repositories directly. Either on a local disk in the case of +e.g. a single-machine Omnibus GitLab installation, or via NFS in the +case of a horizontally scaled GitLab installation. + +Besides running plain `git` commands, in GitLab Rails we also used to +use a Ruby gem (library) called +[Rugged](https://github.com/libgit2/rugged). Rugged is a wrapper around +[libgit2](https://libgit2.org/), a stand-alone implementation of Git in +the form of a C library. + +Over time it has become clear to use that Rugged, and particularly +Rugged in combination with the [Unicorn](https://bogomips.org/unicorn/) +web server, is extremely efficient. Because libgit2 is a *library* and +not an external process, there was very little overhead between GitLab +application code that tried to look up data in Git repositories, and the +Git implementation itself. + +Because Rugged+Unicorn was so efficient, GitLab's application code ended +up with lots of duplicate Git object lookups (like looking up the +`master` commmit a dozen times in one request). We could write +inefficient code without being punished for it. + +When we migrated these Git lookups to Gitaly calls, we were suddenly +getting a much higher fixed cost per Git lookup. Even when Gitaly is +able to re-use an already-running `git` process to look up e.g. a commit +you still have the cost of a network roundtrip to Gitaly, and within +Gitaly a write/read roundtrip on the Unix pipes that connect Gitaly to +the `git` process. + +Using GitLab.com performance as our yardstick, we pushed down the number +of Gitaly calls per request until the loss of Rugged's efficiency was no +longer felt. It also helped that we run Gitaly itself directly on the +Git file severs, rather than via NFS mounts: this gave us a speed boost +that counteracted the negative effect of not using Rugged anymore. + +Unfortunately, some *other* deployments of GitLab could not ditch NFS +like we did on GitLab.com and they got the worst of both worlds: the +slowness of NFS and the increased inherent overhead of Gitaly. + +As a performance band-aid for these stuck-on-NFS deployments, we +re-introduced some of the old Rugged code that got deleted from +GitLab Rails during the Gitaly migration project. These pieces of +re-introduced code are informally referred to as "the Rugged patches". + +### Activation of direct Git access in GitLab Rails + +The Ruby methods that perform direct Git access are hidden behind [feature +flags](../../development/gitaly.md#legacy-rugged-code). These feature +flags are off by default. It is not good if you need to know about +feature flags to get the best performance so in a second iteration, we +added an automatic mechanism that will enable direct Git access. + +When GitLab Rails calls a function that has a Rugged patch it performs +two checks. The result of both of these checks is cached. + +1. Is the feature flag for this patch set in the database? If so, do + what the feature flag says. +1. If the feature flag is not set (i.e. neither true nor false), try to + see if we can access filesystem underneath the Gitaly server + directly. If so, use the Rugged patch. + +To see if GitLab Rails can access the repo filesystem directly, we use +the following heuristic: + +- Gitaly ensures that the filesystem has a metadata file in its root + with a UUID in it. +- Gitaly reports this UUID to GitLab Rails via the `ServerInfo` RPC. +- GitLab Rails tries to read the metadata file directly. If it exists, + and if the UUID's match, assume we have direct access. + +Because of the way the UUID check works, and because Omnibus GitLab will +fill in the correct repository paths in the GitLab Rails config file +`config/gitlab.yml`, **direct Git access in GitLab Rails is on by default in +Omnibus**. + +### Plans to remove direct Git access in GitLab Rails + +For the sake of removing complexity it is desirable that we get rid of +direct Git access in GitLab Rails. For as long as some GitLab installations are stuck +with Git repositories on slow NFS, however, we cannot just remove them. + +There are two prongs to our efforts to remove direct Git access in GitLab Rails: + +1. Reduce the number of (inefficient) Gitaly queries made by + GitLab Rails. +1. Persuade everybody who runs a Highly Available / horizontally scaled + GitLab installation to move off of NFS. + +The second prong is the only real solution. For this we need [Gitaly +HA](https://gitlab.com/groups/gitlab-org/-/epics?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=Gitaly%20HA), +which is still under development as of December 2019. + ## Troubleshooting Gitaly +### Checking versions when using standalone Gitaly nodes + +When using standalone Gitaly nodes, you must make sure they are the same version +as GitLab to ensure full compatibility. Check **Admin Area > Gitaly Servers** on +your GitLab instance and confirm all Gitaly Servers are `Up to date`. + + + ### `gitaly-debug` The `gitaly-debug` command provides "production debugging" tools for Gitaly and Git diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index 83c9aa3f013..6193a40ac4f 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -20,21 +20,19 @@ for updates and roadmap. ### Architecture -For this document, the following network topology is assumed: +The most common architecture for Praefect is simplified in the diagram below: ```mermaid graph TB - GitLab --> Gitaly; GitLab --> Praefect; - Praefect --> Praefect-Gitaly-1; - Praefect --> Praefect-Gitaly-2; - Praefect --> Praefect-Gitaly-3; + Praefect --> Gitaly-1; + Praefect --> Gitaly-2; + Praefect --> Gitaly-3; ``` Where `GitLab` is the collection of clients that can request Git operations. -`Gitaly` is a Gitaly server before using Praefect. The Praefect node has three -storage nodes attached. Praefect itself doesn't store data, but connects to -three Gitaly nodes, `Praefect-Gitaly-1`, `Praefect-Gitaly-2`, and `Praefect-Gitaly-3`. +The Praefect node has threestorage nodes attached. Praefect itself doesn't +store data, but connects to three Gitaly nodes, `Gitaly-1`, `Gitaly-2`, and `Gitaly-3`. Praefect may be enabled on its own node or can be run on the GitLab server. In the example below we will use a separate server, but the optimal configuration @@ -49,41 +47,43 @@ purposes. In this setup guide we will start by configuring Praefect, then its child Gitaly nodes, and lastly the GitLab server configuration. +#### Secrets + +We need to manage the following secrets and make them match across hosts: + +1. `GITLAB_SHELL_SECRET_TOKEN`: this is used by Git hooks to make + callback HTTP API requests to GitLab when accepting a Git push. This + secret is shared with GitLab Shell for legacy reasons. +1. `PRAEFECT_EXTERNAL_TOKEN`: repositories hosted on your Praefect + cluster can only be accessed by Gitaly clients that carry this + token. +1. `PRAEFECT_INTERNAL_TOKEN`: this token is used for replication + traffic inside your Praefect cluster. This is distinct from + `PRAEFECT_EXTERNAL_TOKEN` because Gitaly clients must not be able to + access internal nodes of the Praefect cluster directly; that could + lead to data loss. + #### Praefect On the Praefect node we disable all other services, including Gitaly. We list each -Gitaly node that will be connected to Praefect under `praefect['storage_nodes']`. +Gitaly node that will be connected to Praefect as members of the `praefect` hash in `praefect['virtual_storages']`. -In the example below, the Gitaly nodes are named `praefect-gitaly-N`. Note that one +In the example below, the Gitaly nodes are named `gitaly-N`. Note that one node is designated as primary by setting the primary to `true`. -`praefect['auth_token']` is the token used to authenticate with the GitLab server, -just like `gitaly['auth_token']` is used for a standard Gitaly server. - -The `token` field under each storage listed in `praefect['storage_nodes']` is used -to authenticate each child Gitaly node with Praefect. - ```ruby -# /etc/gitlab/gitlab.rb +# /etc/gitlab/gitlab.rb on praefect server # Avoid running unnecessary services on the Gitaly server postgresql['enable'] = false redis['enable'] = false nginx['enable'] = false prometheus['enable'] = false +grafana['enable'] = false unicorn['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false gitaly['enable'] = false -``` - -##### Set up Praefect and its Gitaly nodes - -In the example below, the Gitaly nodes are named `praefect-git-X`. Note that one node is designated as -primary, by setting the primary to `true`: - -```ruby -# /etc/gitlab/gitlab.rb # Prevent database connections during 'gitlab-ctl reconfigure' gitlab_rails['rake_cache_clear'] = false @@ -95,25 +95,27 @@ praefect['enable'] = true # firewalls to restrict access to this address/port. praefect['listen_addr'] = '0.0.0.0:2305' -# virtual_storage_name must match the same storage name given to praefect in git_data_dirs -praefect['virtual_storage_name'] = 'praefect' - -# Authentication token to ensure only authorized servers can communicate with -# Praefect server -praefect['auth_token'] = 'praefect-token' -praefect['storage_nodes'] = { - 'praefect-gitaly-1' => { - 'address' => 'tcp://praefect-git-1.internal:8075', - 'token' => 'praefect-gitaly-token', - 'primary' => true - }, - 'praefect-gitaly-2' => { - 'address' => 'tcp://praefect-git-2.internal:8075', - 'token' => 'praefect-gitaly-token' - }, - 'praefect-gitaly-3' => { - 'address' => 'tcp://praefect-git-3.internal:8075', - 'token' => 'praefect-gitaly-token' +# Replace PRAEFECT_EXTERNAL_TOKEN with a real secret +praefect['auth_token'] = 'PRAEFECT_EXTERNAL_TOKEN' + +# Replace each instance of PRAEFECT_INTERNAL_TOKEN below with a real +# secret, distinct from PRAEFECT_EXTERNAL_TOKEN. +# Name of storage hash must match storage name in git_data_dirs on GitLab server. +praefect['virtual_storages'] = { + 'praefect' => { + 'gitaly-1' => { + 'address' => 'tcp://gitaly-1.internal:8075', + 'token' => 'PRAEFECT_INTERNAL_TOKEN', + 'primary' => true + }, + 'gitaly-2' => { + 'address' => 'tcp://gitaly-2.internal:8075', + 'token' => 'PRAEFECT_INTERNAL_TOKEN' + }, + 'gitaly-3' => { + 'address' => 'tcp://gitaly-3.internal:8075', + 'token' => 'PRAEFECT_INTERNAL_TOKEN' + } } } ``` @@ -126,37 +128,40 @@ Next we will configure each Gitaly server assigned to Praefect. Configuration f is the same as a normal standalone Gitaly server, except that we use storage names and auth tokens from Praefect instead of GitLab. -Below is an example configuration for `praefect-gitaly-1`, the only difference for the +Below is an example configuration for `gitaly-1`, the only difference for the other Gitaly nodes is the storage name under `git_data_dirs`. -Note that `gitaly['auth_token']` matches the `token` value listed under `praefect['storage_nodes']` +Note that `gitaly['auth_token']` matches the `token` value listed under `praefect['virtual_storages']` on the Praefect node. ```ruby -# /etc/gitlab/gitlab.rb +# /etc/gitlab/gitlab.rb on gitaly node inside praefect cluster # Avoid running unnecessary services on the Gitaly server postgresql['enable'] = false redis['enable'] = false nginx['enable'] = false prometheus['enable'] = false +grafana['enable'] = false unicorn['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false +prometheus_monitoring['enable'] = false # Prevent database connections during 'gitlab-ctl reconfigure' gitlab_rails['rake_cache_clear'] = false gitlab_rails['auto_migrate'] = false +# Replace GITLAB_SHELL_SECRET_TOKEN below with real secret +gitlab_shell['secret_token'] = 'GITLAB_SHELL_SECRET_TOKEN' + # Configure the gitlab-shell API callback URL. Without this, `git push` will # fail. This can be your 'front door' GitLab URL or an internal load # balancer. -# Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server. gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' -# Authentication token to ensure only authorized servers can communicate with -# Gitaly server -gitaly['auth_token'] = 'praefect-gitaly-token' +# Replace PRAEFECT_INTERNAL_TOKEN below with a real secret. +gitaly['auth_token'] = 'PRAEFECT_INTERNAL_TOKEN' # Make Gitaly accept connections on all network interfaces. You must use # firewalls to restrict access to this address/port. @@ -164,16 +169,13 @@ gitaly['auth_token'] = 'praefect-gitaly-token' gitaly['listen_addr'] = "0.0.0.0:8075" git_data_dirs({ - "praefect-gitaly-1" => { + "gitaly-1" => { "path" => "/var/opt/gitlab/git-data" } }) ``` -Note that just as with a standard Gitaly server, `/etc/gitlab/gitlab-secrets.json` must -be copied from the GitLab server to the Gitaly node for authentication purposes. - -For more information on Gitaly server configuration, see our [gitaly documentation](index.md#3-gitaly-server-configuration). +For more information on Gitaly server configuration, see our [Gitaly documentation](index.md#3-gitaly-server-configuration). #### GitLab @@ -182,25 +184,29 @@ is done through setting the `git_data_dirs`. Assuming the default storage is present, there should be two storages available to GitLab: ```ruby +# /etc/gitlab/gitlab.rb on gitlab server + +# Replace PRAEFECT_EXTERNAL_TOKEN below with real secret. git_data_dirs({ "default" => { - "gitaly_address" => "tcp://gitaly.internal" + "path" => "/var/opt/gitlab/git-data" }, "praefect" => { - "gitaly_address" => "tcp://praefect.internal:2305" + "gitaly_address" => "tcp://praefect.internal:2305", + "gitaly_token" => 'PRAEFECT_EXTERNAL_TOKEN' } }) -gitlab_rails['gitaly_token'] = 'praefect-token' +# Replace GITLAB_SHELL_SECRET_TOKEN below with real secret +gitlab_shell['secret_token'] = 'GITLAB_SHELL_SECRET_TOKEN' ``` Note that the storage name used is the same as the `praefect['virtual_storage_name']` set on the Praefect node. -Also, the `gitlab_rails['gitaly_token']` matches the value of `praefect['auth_token']` -on Praefect. +Save your changes and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -Restart GitLab using `gitlab-ctl restart` on the GitLab node. +Run `gitlab-rake gitlab:gitaly:check` to confirm that GitLab can reach Praefect. ### Testing Praefect @@ -208,10 +214,18 @@ To test Praefect, first set it as the default storage node for new projects using **Admin Area > Settings > Repository > Repository storage**. Next, create a new project and check the "Initialize repository with a README" box. -If you receive a 503 error, check `/var/log/gitlab/gitlab-rails/production.log`. -A `GRPC::Unavailable (14:failed to connect to all addresses)` error indicates -that GitLab was unable to connect to Praefect. - -If the project is created but the README is not, then ensure that the -`/etc/gitlab/gitlab-secrets.json` file from the GitLab server has been copied -to the Gitaly servers. +If you receive an error, check `/var/log/gitlab/gitlab-rails/production.log`. + +Here are common errors and potential causes: + +- 500 response code + - **ActionView::Template::Error (7:permission denied)** + - `praefect['auth_token']` and `gitlab_rails['gitaly_token']` do not match on the GitLab server. + - **Unable to save project. Error: 7:permission denied** + - Secret token in `praefect['storage_nodes']` on GitLab server does not match the + value in `gitaly['auth_token']` on one or more Gitaly servers. +- 503 response code + - **GRPC::Unavailable (14:failed to connect to all addresses)** + - GitLab was unable to reach Praefect. + - **GRPC::Unavailable (14:all SubCons are in TransientFailure...)** + - Praefect cannot reach one or more of its child Gitaly nodes. diff --git a/doc/administration/gitaly/reference.md b/doc/administration/gitaly/reference.md index fe88ef13958..2c5e54743c3 100644 --- a/doc/administration/gitaly/reference.md +++ b/doc/administration/gitaly/reference.md @@ -134,7 +134,7 @@ A lot of Gitaly RPCs need to look up Git objects from repositories. Most of the time we use `git cat-file --batch` processes for that. For better performance, Gitaly can re-use these `git cat-file` processes across RPC calls. Previously used processes are kept around in a -["git cat-file cache"](https://about.gitlab.com/blog/2019/07/08/git-performance-on-nfs/#enter-cat-file-cache). +["Git cat-file cache"](https://about.gitlab.com/blog/2019/07/08/git-performance-on-nfs/#enter-cat-file-cache). In order to control how much system resources this uses, we have a maximum number of cat-file processes that can go into the cache. @@ -165,11 +165,11 @@ Gitaly restarts its `gitaly-ruby` helpers when their memory exceeds the | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | -| `dir` | string | yes | Path to where gitaly-ruby is installed (needed to boot the process).| -| `max_rss` | integer | no | Resident set size limit that triggers a gitaly-ruby restart, in bytes. Default is `200000000` (200MB). | -| `graceful_restart_timeout` | string | no | Grace period before a gitaly-ruby process is forcibly terminated after exceeding `max_rss`. Default is `10m` (10 minutes).| -| `restart_delay` | string | no |Time that gitaly-ruby memory must remain high before a restart. Default is `5m` (5 minutes).| -| `num_workers` | integer | no |Number of gitaly-ruby worker processes. Try increasing this number in case of `ResourceExhausted` errors. Default is `2`, minimum is `2`.| +| `dir` | string | yes | Path to where `gitaly-ruby` is installed (needed to boot the process).| +| `max_rss` | integer | no | Resident set size limit that triggers a `gitaly-ruby` restart, in bytes. Default is `200000000` (200MB). | +| `graceful_restart_timeout` | string | no | Grace period before a `gitaly-ruby` process is forcibly terminated after exceeding `max_rss`. Default is `10m` (10 minutes).| +| `restart_delay` | string | no |Time that `gitaly-ruby` memory must remain high before a restart. Default is `5m` (5 minutes).| +| `num_workers` | integer | no |Number of `gitaly-ruby` worker processes. Try increasing this number in case of `ResourceExhausted` errors. Default is `2`, minimum is `2`.| | `linguist_languages_path` | string | no | Override for dynamic `languages.json` discovery. Defaults to an empty string (use of dynamic discovery).| Example: @@ -231,11 +231,11 @@ The following values configure logging in Gitaly under the `[logging]` section. | `level` | string | no | Log level: `debug`, `info`, `warn`, `error`, `fatal`, or `panic`. Default: `info`. | | `sentry_dsn` | string | no | Sentry DSN for exception monitoring. | | `sentry_environment` | string | no | [Sentry Environment](https://docs.sentry.io/enriching-error-data/environments/) for exception monitoring. | -| `ruby_sentry_dsn` | string | no | Sentry DSN for gitaly-ruby exception monitoring. | +| `ruby_sentry_dsn` | string | no | Sentry DSN for `gitaly-ruby` exception monitoring. | While the main Gitaly application logs go to stdout, there are some extra log files that go to a configured directory, like the GitLab Shell logs. -Gitlab Shell does not support `panic` or `trace` level logs. `panic` will fall +GitLab Shell does not support `panic` or `trace` level logs. `panic` will fall back to `error`, while `trace` will fall back to `debug`. Any other invalid log levels will default to `info`. |
