diff options
Diffstat (limited to 'doc/administration/gitaly')
-rw-r--r-- | doc/administration/gitaly/configure_gitaly.md | 7 | ||||
-rw-r--r-- | doc/administration/gitaly/faq.md | 6 | ||||
-rw-r--r-- | doc/administration/gitaly/index.md | 2 | ||||
-rw-r--r-- | doc/administration/gitaly/monitoring.md | 2 | ||||
-rw-r--r-- | doc/administration/gitaly/praefect.md | 4 | ||||
-rw-r--r-- | doc/administration/gitaly/recovery.md | 73 | ||||
-rw-r--r-- | doc/administration/gitaly/reference.md | 2 | ||||
-rw-r--r-- | doc/administration/gitaly/troubleshooting.md | 22 |
8 files changed, 72 insertions, 46 deletions
diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md index bfd252a9f42..e4aef2db9a8 100644 --- a/doc/administration/gitaly/configure_gitaly.md +++ b/doc/administration/gitaly/configure_gitaly.md @@ -1,7 +1,7 @@ --- stage: Systems group: Gitaly -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Configure Gitaly **(FREE SELF)** @@ -555,12 +555,15 @@ Additionally, the certificate (or its certificate authority) must be installed o - Gitaly servers. - Gitaly clients that communicate with it. -Note the following: +### Certificate requirements - The certificate must specify the address you use to access the Gitaly server. You must add the hostname or IP address as a Subject Alternative Name to the certificate. - You can configure Gitaly servers with both an unencrypted listening address `listen_addr` and an encrypted listening address `tls_listen_addr` at the same time. This allows you to gradually transition from unencrypted to encrypted traffic if necessary. +- The certificate's Common Name field is ignored. + +### Configure Gitaly with TLS To configure Gitaly with TLS: diff --git a/doc/administration/gitaly/faq.md b/doc/administration/gitaly/faq.md deleted file mode 100644 index f6571295200..00000000000 --- a/doc/administration/gitaly/faq.md +++ /dev/null @@ -1,6 +0,0 @@ ---- -redirect_to: 'index.md' -remove_date: '2022-08-24' ---- - -This document was moved to [another location](index.md). diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index c7f7c4c58a5..96447239116 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -1,7 +1,7 @@ --- stage: Systems group: Gitaly -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Gitaly and Gitaly Cluster **(FREE SELF)** diff --git a/doc/administration/gitaly/monitoring.md b/doc/administration/gitaly/monitoring.md index a435fff74fc..9b7acd536b3 100644 --- a/doc/administration/gitaly/monitoring.md +++ b/doc/administration/gitaly/monitoring.md @@ -1,7 +1,7 @@ --- stage: Systems group: Gitaly -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Monitoring Gitaly and Gitaly Cluster diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index bd03aa1bdbc..e3b198d1012 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -1,7 +1,7 @@ --- stage: Systems group: Gitaly -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Configure Gitaly Cluster **(FREE SELF)** @@ -1320,7 +1320,7 @@ Deletions are disabled by default due to a race condition with repository rename deletions. This is especially prominent in Geo instances as Geo performs more renames than instances without Geo. You should enable deletions only if the [`gitaly_praefect_generated_replica_paths` feature flag](index.md#praefect-generated-replica-paths-gitlab-150-and-later) is enabled. -By default, the worker does not delete invalid metadata records but simply logs them and outputs Prometheus +By default, the worker does not delete invalid metadata records but logs them and outputs Prometheus metrics for them. You can enable deleting invalid metadata records with: diff --git a/doc/administration/gitaly/recovery.md b/doc/administration/gitaly/recovery.md index 4bbf25d7cdd..b51454aa44e 100644 --- a/doc/administration/gitaly/recovery.md +++ b/doc/administration/gitaly/recovery.md @@ -1,14 +1,15 @@ --- stage: Systems group: Gitaly -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# Recovery options +# Gitaly Cluster recovery options and tools -Gitaly Cluster can [recover from certain types of failure](recovery.md). +Gitaly Cluster can recover from primary-node failure and unavailable repositories. Gitaly Cluster can perform data +recovery and has Praefect tracking database tools. -## Primary Node Failure +## Primary node failure Gitaly Cluster recovers from a failing primary Gitaly node by promoting a healthy secondary as the new primary. @@ -29,14 +30,12 @@ To minimize data loss in GitLab 13.0 to 14.0, Gitaly Cluster: > - Introduced in GitLab 13.0 as [generally available](../../policy/alpha-beta-support.md#generally-available-ga). > - Between GitLab 13.0 and GitLab 13.2, read-only mode applied to the whole virtual storage and occurred whenever failover occurred. -> - [In GitLab 13.3 and later](https://gitlab.com/gitlab-org/gitaly/-/issues/2862), read-only mode applies on a per-repository basis and only occurs if a new primary is out of date. -new primary. If the failed primary contained unreplicated writes, [data loss can occur](#check-for-data-loss). +> - [In GitLab 13.3 and later](https://gitlab.com/gitlab-org/gitaly/-/issues/2862), read-only mode applies on a per-repository basis and only occurs if a new primary is out of date. If the failed primary contained unreplicated writes, [data loss can occur](#check-for-data-loss). > - Removed in GitLab 14.1. Instead, repositories [become unavailable](#unavailable-repositories). -When Gitaly Cluster switches to a new primary in GitLab 13.0 to 14.0, repositories enter -read-only mode if they are out of date. This can happen after failing over to an outdated -secondary. Read-only mode eases data recovery efforts by preventing writes that may conflict -with the unreplicated writes on other nodes. +When Gitaly Cluster switches to a new primary in GitLab 13.0 to 14.0, repositories enter read-only mode if they are +out-of-date. This can happen after failing over to an outdated secondary. Read-only mode eases data recovery efforts by +preventing writes that may conflict with the unreplicated writes on other nodes. To enable writes again in GitLab 13.0 to 14.0, an administrator can: @@ -46,7 +45,7 @@ To enable writes again in GitLab 13.0 to 14.0, an administrator can: [accept data loss](#enable-writes-or-accept-data-loss) if necessary, depending on the version of GitLab. -## Unavailable repositories +### Unavailable repositories > - From GitLab 13.0 through 14.0, repositories became read-only if they were outdated on the primary but fully up to date on a healthy secondary. `dataloss` sub-command displays read-only repositories by default through these versions. > - Since GitLab 14.1, Praefect contains more responsive failover logic which immediately fails over to one of the fully up to date secondaries rather than placing the repository in read-only mode. Since GitLab 14.1, the `dataloss` sub-command displays repositories which are unavailable due to having no fully up to date copies on healthy Gitaly nodes. @@ -252,7 +251,7 @@ These tools reconcile the outdated repositories to bring them fully up to date a > [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/2717) in GitLab 13.4. Praefect automatically reconciles repositories that are not up to date. By default, this is done every -five minutes. For each outdated repository on a healthy Gitaly node, the Praefect picks a +five minutes. For each outdated repository on a healthy Gitaly node, Praefect picks a random, fully up-to-date replica of the repository on another healthy Gitaly node to replicate from. A replication job is scheduled only if there are no other replication jobs pending for the target repository. @@ -296,7 +295,7 @@ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.t > - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3767) in GitLab 14.3. > - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4054) in GitLab 14.6, support for dry-run mode. -> - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4715) in GitLab 15.3, support for removing repositories from Praefect's database. +> - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4715) in GitLab 15.3, support for removing repositories from the Praefect tracking database. The `remove-repository` Praefect sub-command removes a repository from a Gitaly Cluster, and all state associated with a given repository including: @@ -311,9 +310,9 @@ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.t - Replace `<virtual-storage>` with the name of the virtual storage containing the repository. - Replace `<repository>` with the relative path of the repository to remove. -- In GitLab 15.3 and later, add `-db-only` to remove the Praefect database entry without removing the on-disk repository. Use this option to remove orphaned database entries and to +- In GitLab 15.3 and later, add `-db-only` to remove the Praefect tracking database entry without removing the on-disk repository. Use this option to remove orphaned database entries and to protect on-disk repository data from deletion when a valid repository is accidentally specified. If the database entry is accidentally deleted, re-track the repository with the - [`track-repository` command](#manually-track-repositories). + [`track-repository` command](#manually-add-a-single-repository-to-the-tracking-database). - In GitLab 14.6 and later, add `-apply` to run the command outside of dry-run mode and remove the repository. For example: ```shell @@ -349,7 +348,11 @@ Parts of the repository can continue to exist after running `remove-repository`. If this occurs, run `remove-repository` again. -### Manually list untracked repositories +## Praefect tracking database maintenance + +Common maintenance tasks on the Praefect tracking database are documented in this section. + +### List untracked repositories > - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3926) in GitLab 14.4. > - `older-than` option added in GitLab 15.0. @@ -357,10 +360,14 @@ If this occurs, run `remove-repository` again. The `list-untracked-repositories` Praefect sub-command lists repositories of the Gitaly Cluster that both: - Exist for at least one Gitaly storage. -- Aren't tracked in the Praefect database. +- Aren't tracked in the Praefect tracking database. + +Add the `-older-than` option to avoid showing repositories that: -Add the `-older-than` option to avoid showing repositories that are the process of being created and for which a record doesn't yet exist in the -Praefect database. Replace `<duration>` with a time duration (for example, `5s`, `10m`, or `1h`). Defaults to `6h`. +- Are in the process of being created. +- For which a record doesn't yet exist in the Praefect tracking database. + +Replace `<duration>` with a time duration (for example, `5s`, `10m`, or `1h`). Defaults to `6h`. ```shell sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml list-untracked-repositories -older-than <duration> @@ -382,12 +389,12 @@ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.t {"virtual_storage":"default","storage":"gitaly-1","relative_path":"@hashed/ab/cd/abcd123456789012345678901234567890123456789012345678901234567891.git"} ``` -### Manually track repositories +### Manually add a single repository to the tracking database > - [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5658) in GitLab 14.4. > - [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5789) in GitLab 14.6, support for immediate replication. -The `track-repository` Praefect sub-command adds repositories on disk to the Praefect database to be tracked. +The `track-repository` Praefect sub-command adds repositories on disk to the Praefect tracking database to be tracked. ```shell sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml track-repository -virtual-storage <virtual-storage> -authoritative-storage <storage-name> -repository <repository> -replicate-immediately @@ -427,15 +434,17 @@ The command outputs: This command fails if: -- The repository is already being tracked by the Praefect database. +- The repository is already being tracked by the Praefect tracking database. - The repository does not exist on disk. -### Manually track large numbers of repositories +### Manually add many repositories to the tracking database > [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/6319) in GitLab 15.4. -The `track-repositories` Praefect sub-command adds large batches of on-disk repositories to the Praefect database for tracking. This can -be useful when migrating an existing instance to new infrastructure and ingesting all existing repositories into a fresh Gitaly Cluster. +Migrations using the API automatically add repositories to the Praefect tracking database. + +If you are instead manually copying repositories over from existing infrastructure, you can use the `track-repositories` +Praefect subcommand. This subcommand adds large batches of on-disk repositories to the Praefect tracking database. ```shell # Omnibus GitLab install @@ -449,21 +458,21 @@ The command validates that all entries: - Are formatted correctly and contain required fields. - Correspond to a valid Git repository on disk. -- Are not currently tracked in the Praefect database. +- Are not currently tracked in the Praefect tracking database. If any entry fails these checks, the command aborts prior to attempting to track a repository. - `input-path` is the path to a file containing a list of repositories formatted as newline-delimited JSON objects. Objects must contain the following keys: - - `relative_path`: corresponds with `repository` in [track-repositories](#manually-track-repositories). + - `relative_path`: corresponds with `repository` in [`track-repository`](#manually-add-a-single-repository-to-the-tracking-database). - `authoritative-storage`: the storage Praefect is to treat as the primary. - `virtual-storage`: the virtual storage the repository is located in. - For example: + For example: - ```json - {"relative_path":"@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b.git","authoritative_storage":"gitaly-1","virtual_storage":"default"} - {"relative_path":"@hashed/f8/9f/f89f8d0e735a91c5269ab08d72fa27670d000e7561698d6e664e7b603f5c4e40.git","authoritative_storage":"gitaly-2","virtual_storage":"default"} - ``` + ```json + {"relative_path":"@hashed/f5/ca/f5ca38f748a1d6eaf726b8a42fb575c3c71f1864a8143301782de13da2d9202b.git","authoritative_storage":"gitaly-1","virtual_storage":"default"} + {"relative_path":"@hashed/f8/9f/f89f8d0e735a91c5269ab08d72fa27670d000e7561698d6e664e7b603f5c4e40.git","authoritative_storage":"gitaly-2","virtual_storage":"default"} + ``` - `-replicate-immediately`, causes the command to replicate the repository to its secondaries immediately. Otherwise, replication jobs are scheduled for execution in the database and are picked up by a Praefect background process. diff --git a/doc/administration/gitaly/reference.md b/doc/administration/gitaly/reference.md index 2542848c7a8..3bf1e3136c0 100644 --- a/doc/administration/gitaly/reference.md +++ b/doc/administration/gitaly/reference.md @@ -1,7 +1,7 @@ --- stage: Systems group: Gitaly -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Gitaly reference **(FREE SELF)** diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md index 285ec3d2631..1c5f0d43864 100644 --- a/doc/administration/gitaly/troubleshooting.md +++ b/doc/administration/gitaly/troubleshooting.md @@ -1,7 +1,7 @@ --- stage: Systems group: Gitaly -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Troubleshooting Gitaly and Gitaly Cluster **(FREE SELF)** @@ -67,6 +67,13 @@ remote: GitLab: 401 Unauthorized You need to sync your `gitlab-secrets.json` file with your GitLab application nodes. +### 500 and `fetching folder content` errors on repository pages + +`Fetching folder content`, and in some cases `500`, errors indicate +connectivity problems between GitLab and Gitaly. +Consult the [client-side gRPC logs](#client-side-grpc-logs) +for details. + ### Client side gRPC logs Gitaly uses the [gRPC](https://grpc.io/) RPC framework. The Ruby gRPC @@ -81,6 +88,19 @@ You can run a gRPC trace with: sudo GRPC_TRACE=all GRPC_VERBOSITY=DEBUG gitlab-rake gitlab:gitaly:check ``` +If this command fails with a `failed to connect to all addresses` error, +check for an SSL or TLS problem: + +```shell +/opt/gitlab/embedded/bin/openssl s_client -connect <gitaly-ipaddress>:<port> -verify_return_error +``` + +Check whether `Verify return code` field indicates a +[known Omnibus GitLab configuration problem](https://docs.gitlab.com/omnibus/settings/ssl.html). + +If `openssl` succeeds but `gitlab-rake gitlab:gitaly:check` fails, +check [certificate requirements](configure_gitaly.md#certificate-requirements) for Gitaly. + ### Server side gRPC logs gRPC tracing can also be enabled in Gitaly itself with the `GODEBUG=http2debug` |