diff options
Diffstat (limited to 'doc/administration/gitaly/troubleshooting.md')
| -rw-r--r-- | doc/administration/gitaly/troubleshooting.md | 30 |
1 files changed, 14 insertions, 16 deletions
diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md index 1c5f0d43864..7edce840396 100644 --- a/doc/administration/gitaly/troubleshooting.md +++ b/doc/administration/gitaly/troubleshooting.md @@ -41,17 +41,6 @@ The `gitaly-debug` command provides "production debugging" tools for Gitaly and performance. It is intended to help production engineers and support engineers investigate Gitaly performance problems. -If you're using GitLab 11.6 or newer, this tool should be installed on -your GitLab or Gitaly server already at `/opt/gitlab/embedded/bin/gitaly-debug`. -If you're investigating an older GitLab version you can compile this -tool offline and copy the executable to your server: - -```shell -git clone https://gitlab.com/gitlab-org/gitaly.git -cd cmd/gitaly-debug -GOOS=linux GOARCH=amd64 go build -o gitaly-debug -``` - To see the help page of `gitaly-debug` for a list of supported sub-commands, run: ```shell @@ -144,9 +133,8 @@ If you have Prometheus set up to scrape your Gitaly process, you can see request rates and error codes for individual RPCs in `gitaly-ruby` by querying `grpc_client_handled_total`. -- In theory, this metric does not differentiate between `gitaly-ruby` and other RPCs. -- In practice from GitLab 11.9, all gRPC calls made by Gitaly itself are internal calls from the - main Gitaly process to one of its `gitaly-ruby` sidecars. +All gRPC calls made by `gitaly-ruby` itself are internal calls from the main Gitaly process to one of its `gitaly-ruby` +sidecars. Assuming your `grpc_client_handled_total` counter only observes Gitaly, the following query shows you RPCs are (most likely) internally @@ -335,7 +323,7 @@ You might see the following in Gitaly and Praefect logs: } ``` -This is a GRPC call +This is a gRPC call [error response code](https://grpc.github.io/grpc/core/md_doc_statuscodes.html). If this error occurs, even though @@ -359,7 +347,7 @@ necessary because [this issue](https://gitlab.com/gitlab-org/gitaly/-/issues/252 If this error occurs even though file permissions are correct, it's likely that the Gitaly node is experiencing [clock drift](https://en.wikipedia.org/wiki/Clock_drift). -Please ensure that the GitLab and Gitaly nodes are synchronized and use an NTP time +Ensure that the GitLab and Gitaly nodes are synchronized and use an NTP time server to keep them synchronized if possible. ### Health check warnings @@ -604,6 +592,16 @@ For each replica, the following metadata is available: | `Valid Primary` | Indicates whether the replica is fit to serve as the primary node. If the repository's primary is not a valid primary, a failover occurs on the next write to the repository if there is another replica that is a valid primary. A replica is a valid primary if:<br><br>- It is stored on a healthy Gitaly node.<br>- It is fully up to date.<br>- It is not targeted by a pending deletion job from decreasing replication factor.<br>- It is assigned. | | `Verified At` | Indicates last successful verification of the replica by the [verification worker](praefect.md#repository-verification). If the replica has not yet been verified, `unverified` is displayed in place of the last successful verification time. Introduced in GitLab 15.0. | +#### Command fails with 'repository not found' + +If the supplied value for `-virtual-storage` is incorrect, the command returns the following error: + +```plaintext +get metadata: rpc error: code = NotFound desc = repository not found +``` + +The documented examples specify `-virtual-storage default`. Check the Praefect server setting `praefect['virtual_storages']` in `/etc/gitlab/gitlab.rb`. + ### Check that repositories are in sync Is [some cases](index.md#known-issues) the Praefect database can get out of sync with the underlying Gitaly nodes. To check that |