diff options
author | GitLab Bot <gitlab-bot@gitlab.com> | 2021-02-18 10:34:06 +0000 |
---|---|---|
committer | GitLab Bot <gitlab-bot@gitlab.com> | 2021-02-18 10:34:06 +0000 |
commit | 859a6fb938bb9ee2a317c46dfa4fcc1af49608f0 (patch) | |
tree | d7f2700abe6b4ffcb2dcfc80631b2d87d0609239 /doc/administration/geo/replication | |
parent | 446d496a6d000c73a304be52587cd9bbc7493136 (diff) | |
download | gitlab-ce-13.9.0-rc42.tar.gz |
Add latest changes from gitlab-org/gitlab@13-9-stable-eev13.9.0-rc42
Diffstat (limited to 'doc/administration/geo/replication')
20 files changed, 207 insertions, 562 deletions
diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md index 5adf256f6fa..154815efa51 100644 --- a/doc/administration/geo/replication/configuration.md +++ b/doc/administration/geo/replication/configuration.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Geo configuration **(PREMIUM ONLY)** +# Geo configuration **(PREMIUM SELF)** ## Configuring a new **secondary** node @@ -229,7 +229,7 @@ keys must be manually replicated to the **secondary** node. gitlab-rake gitlab:geo:check ``` -Once added to the admin panel and restarted, the **secondary** node will automatically start +Once added to the Geo administration page and restarted, the **secondary** node will automatically start replicating missing data from the **primary** node in a process known as **backfill**. Meanwhile, the **primary** node will start to notify each **secondary** node of any changes, so that the **secondary** node can act on those notifications immediately. @@ -299,7 +299,7 @@ Currently, this is what is synced: ## Selective synchronization -Geo supports selective synchronization, which allows admins to choose +Geo supports selective synchronization, which allows administrators to choose which projects should be synchronized by **secondary** nodes. A subset of projects can be chosen, either by group or by storage shard. The former is ideal for replicating data belonging to a subset of users, while the diff --git a/doc/administration/geo/replication/database.md b/doc/administration/geo/replication/database.md deleted file mode 100644 index 099a73e93d8..00000000000 --- a/doc/administration/geo/replication/database.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: '../setup/database.md' ---- - -This document was moved to [another location](../setup/index.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index 0d2922c3347..f2913dd55ce 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Geo data types support **(PREMIUM ONLY)** +# Geo data types support **(PREMIUM SELF)** A Geo data type is a specific class of data that is required by one or more GitLab features to store relevant information. @@ -35,21 +35,21 @@ verification methods: | Git | Object pools for forked project deduplication | Geo with Gitaly | _Not implemented_ | | Git | Project Snippets | Geo with Gitaly | _Not implemented_ | | Git | Personal Snippets | Geo with Gitaly | _Not implemented_ | -| Blobs | User uploads _(filesystem)_ | Geo with API | _Not implemented_ | +| Blobs | User uploads _(file system)_ | Geo with API | _Not implemented_ | | Blobs | User uploads _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | -| Blobs | LFS objects _(filesystem)_ | Geo with API | _Not implemented_ | +| Blobs | LFS objects _(file system)_ | Geo with API | _Not implemented_ | | Blobs | LFS objects _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | -| Blobs | CI job artifacts _(filesystem)_ | Geo with API | _Not implemented_ | +| Blobs | CI job artifacts _(file system)_ | Geo with API | _Not implemented_ | | Blobs | CI job artifacts _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | -| Blobs | Archived CI build traces _(filesystem)_ | Geo with API | _Not implemented_ | +| Blobs | Archived CI build traces _(file system)_ | Geo with API | _Not implemented_ | | Blobs | Archived CI build traces _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | -| Blobs | Container registry _(filesystem)_ | Geo with API/Docker API | _Not implemented_ | +| Blobs | Container registry _(file system)_ | Geo with API/Docker API | _Not implemented_ | | Blobs | Container registry _(object storage)_ | Geo with API/Managed/Docker API (*2*) | _Not implemented_ | -| Blobs | Package registry _(filesystem)_ | Geo with API | _Not implemented_ | +| Blobs | Package registry _(file system)_ | Geo with API | _Not implemented_ | | Blobs | Package registry _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | -| Blobs | Versioned Terraform State _(filesystem)_ | Geo with API | _Not implemented_ | +| Blobs | Versioned Terraform State _(file system)_ | Geo with API | _Not implemented_ | | Blobs | Versioned Terraform State _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | -| Blobs | External Merge Request Diffs _(filesystem)_ | Geo with API | _Not implemented_ | +| Blobs | External Merge Request Diffs _(file system)_ | Geo with API | _Not implemented_ | | Blobs | External Merge Request Diffs _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | - (*1*): Redis replication can be used as part of HA with Redis sentinel. It's not used between Geo nodes. @@ -63,14 +63,14 @@ is responsible for allowing access and operations on the locally stored Git repo on a machine with a single disk, multiple disks mounted as a single mount-point (like with a RAID array), or using LVM. -It requires no special filesystem and can work with NFS or a mounted Storage Appliance (there may be -performance limitations when using a remote filesystem). +It requires no special file system and can work with NFS or a mounted Storage Appliance (there may be +performance limitations when using a remote file system). Communication is done via Gitaly's own gRPC API. There are three possible ways of synchronization: - Using regular Git clone/fetch from one Geo node to another (with special authentication). - Using repository snapshots (for when the first method fails or repository is corrupt). -- Manual trigger from the Admin UI (a combination of both of the above). +- Manual trigger from the Admin Area (a combination of both of the above). Each project can have at most 3 different repositories: @@ -88,13 +88,13 @@ Both types will be synced to a secondary node. GitLab stores files and blobs such as Issue attachments or LFS objects into either: -- The filesystem in a specific location. +- The file system in a specific location. - An [Object Storage](../../object_storage.md) solution. Object Storage solutions can be: - Cloud based like Amazon S3 Google Cloud Storage. - Hosted by you (like MinIO). - A Storage Appliance that exposes an Object Storage-compatible API. -When using the filesystem store instead of Object Storage, you need to use network mounted filesystems +When using the file system store instead of Object Storage, you need to use network mounted file systems to run GitLab when using more than one server. With respect to replication and verification: @@ -144,9 +144,9 @@ The replication for some data types is behind a corresponding feature flag: > - They're enabled on GitLab.com. > - They can't be enabled or disabled per-project. > - They are recommended for production use. -> - For GitLab self-managed instances, GitLab administrators can opt to [disable them](#enable-or-disable-replication-for-some-data-types). **(CORE ONLY)** +> - For GitLab self-managed instances, GitLab administrators can opt to [disable them](#enable-or-disable-replication-for-some-data-types). **(FREE SELF)** -#### Enable or disable replication (for some data types) **(CORE ONLY)** +#### Enable or disable replication (for some data types) Replication for some data types are released behind feature flags that are **enabled by default**. [GitLab administrators with access to the GitLab Rails console](../../feature_flags.md) can opt to disable it for your instance. You can find feature flag names of each of those data types in the notes column of the table below. @@ -173,7 +173,7 @@ successfully, you must replicate their data using some other means. | Feature | Replicated (added in GitLab version) | Verified (added in GitLab version) | Object Storage replication (see [Geo with Object Storage](object_storage.md)) | Notes | |:---------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------|:----------------------------------------------------------|:-------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [Application data in PostgreSQL](../../postgresql/index.md) | **Yes** (10.2) | **Yes** (10.2) | No | | -| [Project repository](../../..//user/project/repository/) | **Yes** (10.2) | **Yes** (10.7) | No | | +| [Project repository](../../../user/project/repository/) | **Yes** (10.2) | **Yes** (10.7) | No | | | [Project wiki repository](../../../user/project/wiki/) | **Yes** (10.2) | **Yes** (10.7) | No | | [Group wiki repository](../../../user/group/index.md#group-wikis) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/208147) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/208147) | No | | | [Uploads](../../uploads.md) | **Yes** (10.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | No | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing the output between them. | @@ -186,13 +186,13 @@ successfully, you must replicate their data using some other means. | [Container Registry](../../packages/container_registry.md) | **Yes** (12.3) | No | No | Disabled by default. See [instructions](docker_registry.md) to enable. | | [Content in object storage (beta)](object_storage.md) | **Yes** (12.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/13845) | No | | | [Project designs repository](../../../user/project/issues/design_management.md) | **Yes** (12.7) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/32467) | Via Object Storage provider if supported. Native Geo support (Beta). | | -| [NPM Registry](../../../user/packages/npm_registry/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | -| [Maven Repository](../../../user/packages/maven_repository/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | -| [Conan Repository](../../../user/packages/conan_repository/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | -| [NuGet Repository](../../../user/packages/nuget_repository/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | -| [PyPI Repository](../../../user/packages/pypi_repository/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | -| [Composer Repository](../../../user/packages/composer_repository/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | -| [Generic packages](../../../user/packages/generic_packages/index.md) | **Yes** (13.5) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | +| [Package Registry for npm](../../../user/packages/npm_registry/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | +| [Package Registry for Maven](../../../user/packages/maven_repository/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | +| [Package Registry for Conan](../../../user/packages/conan_repository/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | +| [Package Registry for NuGet](../../../user/packages/nuget_repository/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | +| [Package Registry for PyPI](../../../user/packages/pypi_repository/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | +| [Package Registry for Composer](../../../user/packages/composer_repository/index.md) | **Yes** (13.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | +| [Package Registry for generic packages](../../../user/packages/generic_packages/index.md) | **Yes** (13.5) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | | [Versioned Terraform State](../../terraform_state.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_terraform_state_version_replication`, enabled by default | | [External merge request diffs](../../merge_request_diffs.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_merge_request_diff_replication`, enabled by default | | [Versioned snippets](../../../user/snippets.md#versioned-snippets) | [**Yes** (13.7)](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2810) | No | | @@ -201,4 +201,4 @@ successfully, you must replicate their data using some other means. | [GitLab Pages](../../pages/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/589) | No | No | | | [CI Pipeline Artifacts](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/pipeline_artifact.rb) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Persists additional artifacts after a pipeline completes | | [Dependency proxy images](../../../user/packages/dependency_proxy/index.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/259694) | No | No | Blocked on [Geo: Secondary Mimicry](https://gitlab.com/groups/gitlab-org/-/epics/1528). Note that replication of this cache is not needed for Disaster Recovery purposes because it can be recreated from external sources. | -| [Vulnerability Export](../../../user/application_security/security_dashboard/#export-vulnerabilities) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Not planned because they are ephemeral and sensitive. They can be regenerated on demand. | +| [Vulnerability Export](../../../user/application_security/vulnerability_report/#export-vulnerabilities) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Not planned because they are ephemeral and sensitive. They can be regenerated on demand. | diff --git a/doc/administration/geo/replication/disable_geo.md b/doc/administration/geo/replication/disable_geo.md index e8767a56266..a26adcefef5 100644 --- a/doc/administration/geo/replication/disable_geo.md +++ b/doc/administration/geo/replication/disable_geo.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Disabling Geo **(PREMIUM ONLY)** +# Disabling Geo **(PREMIUM SELF)** If you want to revert to a regular Omnibus setup after a test, or you have encountered a Disaster Recovery situation and you want to disable Geo momentarily, you can use these instructions to disable your diff --git a/doc/administration/geo/replication/docker_registry.md b/doc/administration/geo/replication/docker_registry.md index f92a878662e..1669abbc52a 100644 --- a/doc/administration/geo/replication/docker_registry.md +++ b/doc/administration/geo/replication/docker_registry.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Docker Registry for a secondary node **(PREMIUM ONLY)** +# Docker Registry for a secondary node **(PREMIUM SELF)** You can set up a [Docker Registry](https://docs.docker.com/registry/) on your **secondary** Geo node that mirrors the one on the **primary** Geo node. diff --git a/doc/administration/geo/replication/external_database.md b/doc/administration/geo/replication/external_database.md deleted file mode 100644 index dfaf6819fa0..00000000000 --- a/doc/administration/geo/replication/external_database.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: '../setup/external_database.md' ---- - -This document was moved to [another location](../setup/external_database.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/geo/replication/faq.md b/doc/administration/geo/replication/faq.md index ab8199c3717..73a36f5e674 100644 --- a/doc/administration/geo/replication/faq.md +++ b/doc/administration/geo/replication/faq.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Geo Frequently Asked Questions **(PREMIUM ONLY)** +# Geo Frequently Asked Questions **(PREMIUM SELF)** ## What are the minimum requirements to run Geo? diff --git a/doc/administration/geo/replication/geo_validation_tests.md b/doc/administration/geo/replication/geo_validation_tests.md index 2d701458e66..f050d3e708c 100644 --- a/doc/administration/geo/replication/geo_validation_tests.md +++ b/doc/administration/geo/replication/geo_validation_tests.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Geo validation tests **(PREMIUM ONLY)** +# Geo validation tests **(PREMIUM SELF)** The Geo team performs manual testing and validation on common deployment configurations to ensure that Geo works when upgrading between minor GitLab versions and major PostgreSQL database versions. @@ -53,7 +53,7 @@ The following are GitLab upgrade validation tests we performed. - Outcome: Partial success because we did not run the looping pipeline during the demo to validate zero-downtime. - Follow up issues: - - [Clarify hup Puma/Unicorn should include deploy node](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5460) + - [Clarify how Puma/Unicorn should include deploy node](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5460) - [Investigate MR creation failure after upgrade to 12.9.10](https://gitlab.com/gitlab-org/gitlab/-/issues/223282) Closed as false positive. ### February 2020 @@ -128,7 +128,7 @@ The following are PostgreSQL upgrade validation tests we performed. PostgreSQL 12 with a database cluster on the primary is not recommended until the issues are resolved. - Known issues for PostgreSQL clusters: - [Ensure Patroni detects PostgreSQL update](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5423) - - [Allow configuring permanent replication slots in patroni](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5628) + - [Allow configuring permanent replication slots in Patroni](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5628) ### August 2020 diff --git a/doc/administration/geo/replication/high_availability.md b/doc/administration/geo/replication/high_availability.md deleted file mode 100644 index 1da4246db1f..00000000000 --- a/doc/administration/geo/replication/high_availability.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: 'multiple_servers.md' ---- - -This document was moved to [another location](multiple_servers.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/geo/replication/index.md b/doc/administration/geo/replication/index.md deleted file mode 100644 index 955e05491d2..00000000000 --- a/doc/administration/geo/replication/index.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: '../index.md' ---- - -This document was moved to [another location](../index.md). - -<!-- This redirect file can be deleted after February 1, 2021. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/geo/replication/location_aware_git_url.md b/doc/administration/geo/replication/location_aware_git_url.md index f2cf8c9bda0..0eea792d374 100644 --- a/doc/administration/geo/replication/location_aware_git_url.md +++ b/doc/administration/geo/replication/location_aware_git_url.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Location-aware Git remote URL with AWS Route53 **(PREMIUM ONLY)** +# Location-aware Git remote URL with AWS Route53 **(PREMIUM SELF)** You can provide GitLab users with a single remote URL that automatically uses the Geo node closest to them. This means users don't need to update their Git diff --git a/doc/administration/geo/replication/multiple_servers.md b/doc/administration/geo/replication/multiple_servers.md index 0ab972c9077..f83e0e14e54 100644 --- a/doc/administration/geo/replication/multiple_servers.md +++ b/doc/administration/geo/replication/multiple_servers.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Geo for multiple nodes **(PREMIUM ONLY)** +# Geo for multiple nodes **(PREMIUM SELF)** This document describes a minimal reference architecture for running Geo in a multi-node configuration. If your multi-node setup differs from the one diff --git a/doc/administration/geo/replication/object_storage.md b/doc/administration/geo/replication/object_storage.md index ce791d66b34..ad419f999b3 100644 --- a/doc/administration/geo/replication/object_storage.md +++ b/doc/administration/geo/replication/object_storage.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Geo with Object storage **(PREMIUM ONLY)** +# Geo with Object storage **(PREMIUM SELF)** Geo can be used in combination with Object Storage (AWS S3, or other compatible object storage). @@ -29,7 +29,7 @@ WARNING: This is a [**beta** feature](https://about.gitlab.com/handbook/product/#beta) and is not ready yet for production use at any scale. The main limitations are a lack of testing at scale and no verification of any replicated data. **Secondary** nodes can replicate files stored on the **primary** node regardless of -whether they are stored on the local filesystem or in object storage. +whether they are stored on the local file system or in object storage. To enable GitLab replication, you must: diff --git a/doc/administration/geo/replication/remove_geo_node.md b/doc/administration/geo/replication/remove_geo_node.md index 519b93b447b..26150a6f536 100644 --- a/doc/administration/geo/replication/remove_geo_node.md +++ b/doc/administration/geo/replication/remove_geo_node.md @@ -5,15 +5,15 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Removing secondary Geo nodes **(PREMIUM ONLY)** +# Removing secondary Geo nodes **(PREMIUM SELF)** -**Secondary** nodes can be removed from the Geo cluster using the Geo admin page of the **primary** node. To remove a **secondary** node: +**Secondary** nodes can be removed from the Geo cluster using the Geo administration page of the **primary** node. To remove a **secondary** node: 1. Navigate to **Admin Area > Geo** (`/admin/geo/nodes`). 1. Click the **Remove** button for the **secondary** node you want to remove. 1. Confirm by clicking **Remove** when the prompt appears. -Once removed from the Geo admin page, you must stop and uninstall the **secondary** node: +Once removed from the Geo administration page, you must stop and uninstall the **secondary** node: 1. On the **secondary** node, stop GitLab: diff --git a/doc/administration/geo/replication/security_review.md b/doc/administration/geo/replication/security_review.md index 7c15662340c..c31881910bc 100644 --- a/doc/administration/geo/replication/security_review.md +++ b/doc/administration/geo/replication/security_review.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Geo security review (Q&A) **(PREMIUM ONLY)** +# Geo security review (Q&A) **(PREMIUM SELF)** The following security review of the Geo feature set focuses on security aspects of the feature as they apply to customers running their own GitLab instances. The review @@ -78,7 +78,7 @@ from [owasp.org](https://owasp.org/). ### Who has administrative capabilities in the application? - Nothing Geo-specific. Any user where `admin: true` is set in the database is - considered an admin with super-user privileges. + considered an administrator with super-user privileges. - See also: [more granular access control](https://gitlab.com/gitlab-org/gitlab/-/issues/18242) (not Geo-specific). - Much of Geo’s integration (database replication, for instance) must be @@ -194,7 +194,7 @@ from [owasp.org](https://owasp.org/). administrator via SSH, and then back out to the **secondary** node in the same manner. In particular, this includes the PostgreSQL replication credentials and a secret key (`db_key_base`) which is used to decrypt certain columns in the database. - The `db_key_base` secret is stored unencrypted on the filesystem, in + The `db_key_base` secret is stored unencrypted on the file system, in `/etc/gitlab/gitlab-secrets.json`, along with a number of other secrets. There is no at-rest protection for them. @@ -217,7 +217,7 @@ from [owasp.org](https://owasp.org/). - **Secondary** nodes and **primary** nodes interact via HTTP/HTTPS (secured with JSON web tokens) and via PostgreSQL streaming replication. -- Within a **primary** node or **secondary** node, the SSOT is the filesystem and the database +- Within a **primary** node or **secondary** node, the SSOT is the file system and the database (including Geo tracking database on **secondary** node). The various internal components are orchestrated to make alterations to these stores. @@ -231,7 +231,7 @@ from [owasp.org](https://owasp.org/). ### What data is or may need to be encrypted and what key management requirements have been defined? -- Neither **primary** nodes or **secondary** nodes encrypt Git repository or filesystem data at +- Neither **primary** nodes or **secondary** nodes encrypt Git repository or file system data at rest. A subset of database columns are encrypted at rest using the `db_otp_key`. - A static secret shared across all hosts in a GitLab deployment. - In transit, data should be encrypted, although the application does permit @@ -287,5 +287,5 @@ from [owasp.org](https://owasp.org/). ### What application auditing requirements have been defined? How are audit and debug logs accessed, stored, and secured? -- Structured JSON log is written to the filesystem, and can also be ingested +- Structured JSON log is written to the file system, and can also be ingested into a Kibana installation for further analysis. diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index 2c0160ed02a..3b1f60a0f3f 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Troubleshooting Geo **(PREMIUM ONLY)** +# Troubleshooting Geo **(PREMIUM SELF)** Setting up Geo requires careful attention to details and sometimes it's easy to miss a step. @@ -219,7 +219,7 @@ sudo gitlab-rake gitlab:geo:check ``` - Ensure that you have added the secondary node in the Admin Area of the **primary** node. - - Ensure that you entered the `external_url` or `gitlab_rails['geo_node_name']` when adding the secondary node in the admin are of the **primary** node. + - Ensure that you entered the `external_url` or `gitlab_rails['geo_node_name']` when adding the secondary node in the Admin Area of the **primary** node. - Prior to GitLab 12.4, edit the secondary node in the Admin Area of the **primary** node and ensure that there is a trailing `/` in the `Name` field. 1. Check returns `Exception: PG::UndefinedTable: ERROR: relation "geo_nodes" does not exist` @@ -396,6 +396,69 @@ In GitLab 13.4, a seed project is added when GitLab is first installed. This mak on a new Geo secondary node. There is an [issue to account for seed projects](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5618) when checking the database. +### Message: `Synchronization failed - Error syncing repository` + +WARNING: +If large repositories are affected by this problem, +their resync may take a long time and cause significant load on your Geo nodes, +storage and network systems. + +If you get the error `Synchronization failed - Error syncing repository` along with the following log messages, this indicates that the expected `geo` remote is not present in the `.git/config` file +of a repository on the secondary Geo node's filesystem: + +```json +{ + "created": "@1603481145.084348757", + "description": "Error received from peer unix:/var/opt/gitlab/gitaly/gitaly.socket", + … + "grpc_message": "exit status 128", + "grpc_status": 13 +} +{ … + "grpc.request.fullMethod": "/gitaly.RemoteService/FindRemoteRootRef", + "grpc.request.glProjectPath": "<namespace>/<project>", + … + "level": "error", + "msg": "fatal: 'geo' does not appear to be a git repository + fatal: Could not read from remote repository. …", +} +``` + +To solve this: + +1. Log into the secondary Geo node. + +1. Back up [the `.git` folder](../../repository_storage_types.md#translating-hashed-storage-paths). + +1. Optional: [Spot-check](../../troubleshooting/log_parsing.md#find-all-projects-affected-by-a-fatal-git-problem)) + a few of those IDs whether they indeed correspond + to a project with known Geo replication failures. + Use `fatal: 'geo'` as the `grep` term and the following API call: + + ```shell + curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/<first_failed_geo_sync_ID>" + ``` + +1. Enter the [Rails console](../../troubleshooting/navigating_gitlab_via_rails_console.md) and run: + + ```ruby + failed_geo_syncs = Geo::ProjectRegistry.failed.pluck(:id) + failed_geo_syncs.each do |fgs| + puts Geo::ProjectRegistry.failed.find(fgs).project_id + end + ``` + +1. Run the following commands to reset each project's + Geo-related attributes and execute a new sync: + + ```ruby + failed_geo_syncs.each do |fgs| + registry = Geo::ProjectRegistry.failed.find(fgs) + registry.update(resync_repository: true, force_to_redownload_repository: false, repository_retry_count: 0) + Geo::RepositorySyncService.new(registry.project).execute + end + ``` + ### Very large repositories never successfully synchronize on the **secondary** node GitLab places a timeout on all repository clones, including project imports @@ -678,19 +741,19 @@ sudo /opt/gitlab/embedded/bin/gitlab-pg-ctl promote GitLab 12.9 and later are [unaffected by this error](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5147). -### Two-factor authentication is broken after a failover +### Message: `ERROR - Replication is not up-to-date` during `gitlab-ctl promotion-preflight-checks` + +In GitLab 13.7 and earlier, if you have a data type with zero items to sync, +this command reports `ERROR - Replication is not up-to-date` even if +replication is actually up-to-date. This bug was fixed in GitLab 13.8 and +later. -The setup instructions for Geo prior to 10.5 failed to replicate the -`otp_key_base` secret, which is used to encrypt the two-factor authentication -secrets stored in the database. If it differs between **primary** and **secondary** -nodes, users with two-factor authentication enabled won't be able to log in -after a failover. +### Message: `ERROR - Replication is not up-to-date` during `gitlab-ctl promote-to-primary-node` -If you still have access to the old **primary** node, you can follow the -instructions in the -[Upgrading to GitLab 10.5](../replication/version_specific_updates.md#updating-to-gitlab-105) -section to resolve the error. Otherwise, the secret is lost and you'll need to -[reset two-factor authentication for all users](../../../security/two_factor_authentication.md#disabling-2fa-for-everyone). +In GitLab 13.7 and earlier, if you have a data type with zero items to sync, +this command reports `ERROR - Replication is not up-to-date` even if +replication is actually up-to-date. If replication and verification output +shows that it is complete, you can add `--skip-preflight-checks` to make the command complete promotion. This bug was fixed in GitLab 13.8 and later. ## Expired artifacts @@ -717,7 +780,7 @@ node's URL matches its external URL. ## Fixing common errors -This section documents common errors reported in the Admin UI and how to fix them. +This section documents common errors reported in the Admin Area and how to fix them. ### Geo database configuration file is missing diff --git a/doc/administration/geo/replication/tuning.md b/doc/administration/geo/replication/tuning.md index 183180e5ade..a4aad3dec68 100644 --- a/doc/administration/geo/replication/tuning.md +++ b/doc/administration/geo/replication/tuning.md @@ -5,11 +5,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Tuning Geo **(PREMIUM ONLY)** +# Tuning Geo **(PREMIUM SELF)** ## Changing the sync/verification capacity values -In the Geo admin page at **Admin Area > Geo** (`/admin/geo/nodes`), +In **Admin Area > Geo** (`/admin/geo/nodes`), there are several variables that can be tuned to improve performance of Geo: - Repository sync capacity diff --git a/doc/administration/geo/replication/updating_the_geo_nodes.md b/doc/administration/geo/replication/updating_the_geo_nodes.md index ff3a1e8689b..0c68adf162d 100644 --- a/doc/administration/geo/replication/updating_the_geo_nodes.md +++ b/doc/administration/geo/replication/updating_the_geo_nodes.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Updating the Geo nodes **(PREMIUM ONLY)** +# Updating the Geo nodes **(PREMIUM SELF)** WARNING: Read these sections carefully before updating your Geo nodes. Not following diff --git a/doc/administration/geo/replication/using_a_geo_server.md b/doc/administration/geo/replication/using_a_geo_server.md index 743b1b384db..7578b6bf61a 100644 --- a/doc/administration/geo/replication/using_a_geo_server.md +++ b/doc/administration/geo/replication/using_a_geo_server.md @@ -7,7 +7,7 @@ type: howto <!-- Please update EE::GitLab::GeoGitAccess::GEO_SERVER_DOCS_URL if this file is moved) --> -# Using a Geo Server **(PREMIUM ONLY)** +# Using a Geo Server **(PREMIUM SELF)** After you set up the [database replication and configure the Geo nodes](../index.md#setup-instructions), use your closest GitLab node as you would a normal standalone GitLab instance. diff --git a/doc/administration/geo/replication/version_specific_updates.md b/doc/administration/geo/replication/version_specific_updates.md index 0d8eba555ab..be2ce0ac2c0 100644 --- a/doc/administration/geo/replication/version_specific_updates.md +++ b/doc/administration/geo/replication/version_specific_updates.md @@ -5,23 +5,37 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Version specific update instructions +# Version-specific update instructions -Check this document if it includes instructions for the version you are updating. -These steps go together with the [general steps](updating_the_geo_nodes.md#general-update-steps) +Review this page for update instructions for your version. These steps +accompany the [general steps](updating_the_geo_nodes.md#general-update-steps) for updating Geo nodes. +## Updating to GitLab 13.7 + +We've detected an issue with the `FetchRemove` call used by Geo secondaries. +This causes performance issues as we execute reference transaction hooks for +each updated reference. Delay any upgrade attempts until this is in the +[13.7.5 patch release.](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3002). +More details are available [in this issue](https://gitlab.com/gitlab-org/git/-/issues/79). + ## Updating to GitLab 13.5 -In GitLab 13.5, there is a [regression that prevents viewing a list of container repositories and registries](https://gitlab.com/gitlab-org/gitlab/-/issues/285475) on Geo secondaries. This issue is fixed in GitLab 13.6.1 and later. +GitLab 13.5 has a [regression that prevents viewing a list of container repositories and registries](https://gitlab.com/gitlab-org/gitlab/-/issues/285475) +on Geo secondaries. This issue is fixed in GitLab 13.6.1 and later. ## Updating to GitLab 13.3 -In GitLab 13.3, Geo removed the PostgreSQL [Foreign Data Wrapper](https://www.postgresql.org/docs/11/postgres-fdw.html) dependency for the tracking database. +In GitLab 13.3, Geo removed the PostgreSQL [Foreign Data Wrapper](https://www.postgresql.org/docs/11/postgres-fdw.html) +dependency for the tracking database. -The FDW server, user, and the extension will be removed during the upgrade process on each **secondary** node. The GitLab settings related to the FDW in the `/etc/gitlab/gitlab.rb` have been deprecated and can be safely removed. +The FDW server, user, and the extension will be removed during the upgrade +process on each secondary node. The GitLab settings related to the FDW in the +`/etc/gitlab/gitlab.rb` have been deprecated and can be safely removed. -There are some scenarios like using an external PostgreSQL instance for the tracking database where the FDW settings must be removed manually. Enter the PostgreSQL console of that instance and remove them: +There are some scenarios like using an external PostgreSQL instance for the +tracking database where the FDW settings must be removed manually. Enter the +PostgreSQL console of that instance and remove them: ```shell DROP SERVER gitlab_secondary CASCADE; @@ -36,7 +50,10 @@ upgrade to GitLab 13.4 or later. ## Updating to GitLab 13.2 -In GitLab 13.2, promoting a secondary node to a primary while the secondary is paused fails. **Do not pause replication before promoting a secondary.** If the node is paused, please resume before promoting. To avoid this issue, upgrade to GitLab 13.4 or later. +In GitLab 13.2, promoting a secondary node to a primary while the secondary is +paused fails. Do not pause replication before promoting a secondary. If the +node is paused, be sure to resume before promoting. To avoid this issue, +upgrade to GitLab 13.4 or later. ## Updating to GitLab 13.0 @@ -59,12 +76,13 @@ GitLab 12.9.0 through GitLab 12.9.3 are affected by [a bug that stops repository verification](https://gitlab.com/gitlab-org/gitlab/-/issues/213523). The issue is fixed in GitLab 12.9.4. Upgrade to GitLab 12.9.4 or later. -By default, GitLab 12.9 will attempt to automatically update the embedded -PostgreSQL server to 10.12 from 9.6, which requires downtime on secondaries -while reinitializing streaming replication. For the recommended procedure, see the +By default, GitLab 12.9 attempts to update the embedded PostgreSQL server +version from 9.6 to 10.12, which requires downtime on secondaries while +reinitializing streaming replication. For the recommended procedure, see the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). -This can be temporarily disabled by running the following before updating: +You can temporarily disable this behavior by running the following before +updating: ```shell sudo touch /etc/gitlab/disable-postgresql-upgrade @@ -72,12 +90,13 @@ sudo touch /etc/gitlab/disable-postgresql-upgrade ## Updating to GitLab 12.8 -By default, GitLab 12.8 will attempt to automatically update the embedded -PostgreSQL server to 10.12 from 9.6, which requires downtime on secondaries -while reinitializing streaming replication. For the recommended procedure, see -the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). +By default, GitLab 12.8 attempts to update the embedded PostgreSQL server +version from 9.6 to 10.12, which requires downtime on secondaries while +reinitializing streaming replication. For the recommended procedure, see the +[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). -This can be temporarily disabled by running the following before updating: +You can temporarily disable this behavior by running the following before +updating: ```shell sudo touch /etc/gitlab/disable-postgresql-upgrade @@ -87,18 +106,17 @@ sudo touch /etc/gitlab/disable-postgresql-upgrade WARNING: Only upgrade to GitLab 12.7.5 or later. Do not upgrade to versions 12.7.0 -through 12.7.4 because there is [an initialization order -bug](https://gitlab.com/gitlab-org/gitlab/-/issues/199672) that causes Geo -**secondaries** to set the incorrect database connection pool size. [The -fix](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/24021) was +through 12.7.4 because there is [an initialization order bug](https://gitlab.com/gitlab-org/gitlab/-/issues/199672) that causes Geo secondaries to set the incorrect database connection pool size. +[The fix](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/24021) was shipped in 12.7.5. -By default, GitLab 12.7 will attempt to automatically update the embedded -PostgreSQL server to 10.9 from 9.6, which requires downtime on secondaries -while reinitializing streaming replication. For the recommended procedure, see -the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). +By default, GitLab 12.7 attempts to update the embedded PostgreSQL server +version from 9.6 to 10.9, which requires downtime on secondaries while +reinitializing streaming replication. For the recommended procedure, see the +[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). -This can be temporarily disabled by running the following before updating: +You can temporarily disable this behavior by running the following before +updating: ```shell sudo touch /etc/gitlab/disable-postgresql-upgrade @@ -106,12 +124,13 @@ sudo touch /etc/gitlab/disable-postgresql-upgrade ## Updating to GitLab 12.6 -By default, GitLab 12.6 will attempt to automatically update the embedded -PostgreSQL server to 10.9 from 9.6, which requires downtime on secondaries -while reinitializing streaming replication. For the recommended procedure, see -the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). +By default, GitLab 12.6 attempts to update the embedded PostgreSQL server +version from 9.6 to 10.9, which requires downtime on secondaries while +reinitializing streaming replication. For the recommended procedure, see the +[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). -This can be temporarily disabled by running the following before updating: +You can temporarily disable this behavior by running the following before +updating: ```shell sudo touch /etc/gitlab/disable-postgresql-upgrade @@ -119,12 +138,13 @@ sudo touch /etc/gitlab/disable-postgresql-upgrade ## Updating to GitLab 12.5 -By default, GitLab 12.5 will attempt to automatically update the embedded -PostgreSQL server to 10.9 from 9.6, which requires downtime on secondaries -while reinitializing streaming replication. For the recommended procedure, see -the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). +By default, GitLab 12.5 attempts to update the embedded PostgreSQL server +version from 9.6 to 10.9, which requires downtime on secondaries while +reinitializing streaming replication. For the recommended procedure, see the +[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). -This can be temporarily disabled by running the following before updating: +You can temporarily disable this behavior by running the following before +updating: ```shell sudo touch /etc/gitlab/disable-postgresql-upgrade @@ -132,12 +152,13 @@ sudo touch /etc/gitlab/disable-postgresql-upgrade ## Updating to GitLab 12.4 -By default, GitLab 12.4 will attempt to automatically update the embedded -PostgreSQL server to 10.9 from 9.6, which requires downtime on secondaries -while reinitializing streaming replication. For the recommended procedure, see -the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). +By default, GitLab 12.4 attempts to update the embedded PostgreSQL server +version from 9.6 to 10.9, which requires downtime on secondaries while +reinitializing streaming replication. For the recommended procedure, see the +[Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). -This can be temporarily disabled by running the following before updating: +You can temporarily disable this behavior by running the following before +updating: ```shell sudo touch /etc/gitlab/disable-postgresql-upgrade @@ -146,9 +167,9 @@ sudo touch /etc/gitlab/disable-postgresql-upgrade ## Updating to GitLab 12.3 WARNING: -If the existing PostgreSQL server version is 9.6.x, it is recommended to -upgrade to GitLab 12.4 or later. By default, GitLab 12.3 attempts to update the -embedded PostgreSQL server from 9.6 to 10.9. In certain circumstances, it will +If the existing PostgreSQL server version is 9.6.x, we recommend upgrading to +GitLab 12.4 or later. By default, GitLab 12.3 attempts to update the embedded +PostgreSQL server version from 9.6 to 10.9. In certain circumstances, it can fail. For more information, see the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). @@ -160,23 +181,23 @@ For the recommended procedure, see the ## Updating to GitLab 12.2 WARNING: -If the existing PostgreSQL server version is 9.6.x, it is recommended to -upgrade to GitLab 12.4 or later. By default, GitLab 12.2 attempts to update the -embedded PostgreSQL server from 9.6 to 10.9. In certain circumstances, it will +If the existing PostgreSQL server version is 9.6.x, we recommend upgrading to +GitLab 12.4 or later. By default, GitLab 12.2 attempts to update the embedded +PostgreSQL server version from 9.6 to 10.9. In certain circumstances, it can fail. For more information, see the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). -Additionally, if the PostgreSQL upgrade does not fail, a successful upgrade +Additionally, if the PostgreSQL upgrade doesn't fail, a successful upgrade requires downtime for secondaries while reinitializing streaming replication. For the recommended procedure, see the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). GitLab 12.2 includes the following minor PostgreSQL updates: -- To version `9.6.14` if you run PostgreSQL 9.6. -- To version `10.9` if you run PostgreSQL 10. +- To version `9.6.14`, if you run PostgreSQL 9.6. +- To version `10.9`, if you run PostgreSQL 10. -This update will occur even if major PostgreSQL updates are disabled. +This update occurs even if major PostgreSQL updates are disabled. Before [refreshing Foreign Data Wrapper during a Geo upgrade](https://docs.gitlab.com/omnibus/update/README.html#run-post-deployment-migrations-and-checks), restart the Geo tracking database: @@ -185,14 +206,15 @@ restart the Geo tracking database: sudo gitlab-ctl restart geo-postgresql ``` -The restart avoids a version mismatch when PostgreSQL tries to load the FDW extension. +The restart avoids a version mismatch when PostgreSQL tries to load the FDW +extension. ## Updating to GitLab 12.1 WARNING: -If the existing PostgreSQL server version is 9.6.x, it is recommended to -upgrade to GitLab 12.4 or later. By default, GitLab 12.1 attempts to update the -embedded PostgreSQL server from 9.6 to 10.9. In certain circumstances, it will +If the existing PostgreSQL server version is 9.6.x, we recommend upgrading to +GitLab 12.4 or later. By default, GitLab 12.1 attempts to update the embedded +PostgreSQL server version from 9.6 to 10.9. In certain circumstances, it can fail. For more information, see the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance). @@ -206,419 +228,11 @@ For the recommended procedure, see the WARNING: This version is affected by a [bug that results in new LFS objects not being replicated to Geo secondary nodes](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). -The issue is fixed in GitLab 12.1; be sure to upgrade to GitLab 12.1 or later. +The issue is fixed in GitLab 12.1. Be sure to upgrade to GitLab 12.1 or later. ## Updating to GitLab 11.11 WARNING: This version is affected by a [bug that results in new LFS objects not being replicated to Geo secondary nodes](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). -The issue is fixed in GitLab 12.1; be sure to upgrade to GitLab 12.1 or later. - -## Updating to GitLab 10.8 - -Before 10.8, broadcast messages would not propagate without flushing -the cache on the **secondary** nodes. This has been fixed in 10.8, but -requires one last cache flush on each **secondary** node: - -```shell -sudo gitlab-rake cache:clear -``` - -## Updating to GitLab 10.6 - -In 10.4, we started to recommend that you define a password for database user (`gitlab`). - -We now require this change as we use this password to enable the Foreign Data Wrapper, as a way to optimize -the Geo Tracking Database. We are also improving security by disabling the use of **trust** -authentication method. - -1. **(primary)** Login to your **primary** node and run: - - ```shell - gitlab-ctl pg-password-md5 gitlab - # Enter password: <your_password_here> - # Confirm password: <your_password_here> - # fca0b89a972d69f00eb3ec98a5838484 - ``` - - Copy the generated hash and edit `/etc/gitlab/gitlab.rb`: - - ```ruby - # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab` - postgresql['sql_user_password'] = '<md5_hash_of_your_password>' - - # Every node that runs Unicorn or Sidekiq needs to have the database - # password specified as below. - # This must be present in all application nodes. - gitlab_rails['db_password'] = '<your_password_here>' - ``` - - Still in the configuration file, locate and remove the `trust_auth_cidr_address`: - - ```ruby - postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','1.2.3.4/32'] # <- Remove this - ``` - -1. **(primary)** Reconfigure and restart: - - ```shell - sudo gitlab-ctl reconfigure - sudo gitlab-ctl restart - ``` - -1. **(secondary)** Login to all **secondary** nodes and edit `/etc/gitlab/gitlab.rb`: - - ```ruby - # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab` - postgresql['sql_user_password'] = '<md5_hash_of_your_password>' - - # Every node that runs Unicorn or Sidekiq needs to have the database - # password specified as below. - # This must be present in all application nodes. - gitlab_rails['db_password'] = '<your_password_here>' - - # Enable Foreign Data Wrapper - geo_secondary['db_fdw'] = true - - # Secondary address in CIDR format, for example '5.6.7.8/32' - postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32'] - ``` - - Still in the configuration file, locate and remove the `trust_auth_cidr_address`: - - ```ruby - postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','5.6.7.8/32'] # <- Remove this - ``` - -1. **(secondary)** Reconfigure and restart: - - ```shell - sudo gitlab-ctl reconfigure - sudo gitlab-ctl restart - ``` - -## Updating to GitLab 10.5 - -For Geo Disaster Recovery to work with minimum downtime, your **secondary** node -should use the same set of secrets as the **primary** node. However, setup instructions -prior to the 10.5 release only synchronized the `db_key_base` secret. - -To rectify this error on existing installations, you should **overwrite** the -contents of `/etc/gitlab/gitlab-secrets.json` on each **secondary** node with the -contents of `/etc/gitlab/gitlab-secrets.json` on the **primary** node, then run the -following command on each **secondary** node: - -```shell -sudo gitlab-ctl reconfigure -``` - -If you do not perform this step, you may find that two-factor authentication -[is broken following DR](troubleshooting.md#two-factor-authentication-is-broken-after-a-failover). - -To prevent SSH requests to the newly promoted **primary** node from failing -due to SSH host key mismatch when updating the **primary** node domain's DNS record -you should perform the step to [Manually replicate **primary** SSH host keys](configuration.md#step-2-manually-replicate-the-primary-nodes-ssh-host-keys) in each -**secondary** node. - -## Updating to GitLab 10.3 - -### Support for SSH repository synchronization removed - -In GitLab 10.2, synchronizing secondaries over SSH was deprecated. In 10.3, -support is removed entirely. All installations will switch to the HTTP/HTTPS -cloning method instead. Before updating, ensure that all your Geo nodes are -configured to use this method and that it works for your installation. In -particular, ensure that [Git access over HTTP/HTTPS is enabled](configuration.md#step-5-enable-git-access-over-httphttps). - -Synchronizing repositories over the public Internet using HTTP is insecure, so -you should ensure that you have HTTPS configured before updating. Note that -file synchronization is **also** insecure in these cases! - -## Updating to GitLab 10.2 - -### Secure PostgreSQL replication - -Support for TLS-secured PostgreSQL replication has been added. If you are -currently using PostgreSQL replication across the open internet without an -external means of securing the connection (e.g., a site-to-site VPN), then you -should immediately reconfigure your **primary** and **secondary** PostgreSQL instances -according to the [updated instructions](../setup/database.md). - -If you *are* securing the connections externally and wish to continue doing so, -ensure you include the new option `--sslmode=prefer` in future invocations of -`gitlab-ctl replicate-geo-database`. - -### HTTPS repository sync - -Support for replicating repositories and wikis over HTTP/HTTPS has been added. -Replicating over SSH has been deprecated, and support for this option will be -removed in a future release. - -To switch to HTTP/HTTPS replication, log into the **primary** node as an admin and visit -**Admin Area > Geo** (`/admin/geo/nodes`). For each **secondary** node listed, -press the "Edit" button, change the "Repository cloning" setting from -"SSH (deprecated)" to "HTTP/HTTPS", and press "Save changes". This should take -effect immediately. - -Any new secondaries should be created using HTTP/HTTPS replication - this is the -default setting. - -After you've verified that HTTP/HTTPS replication is working, you should remove -the now-unused SSH keys from your secondaries, as they may cause problems if the -**secondary** node if ever promoted to a **primary** node: - -1. **(secondary)** Login to **all** your **secondary** nodes and run: - - ```ruby - sudo -u git -H rm ~git/.ssh/id_rsa ~git/.ssh/id_rsa.pub - ``` - -### Hashed Storage - -WARNING: -Hashed storage is in **Alpha**. It is considered experimental and not -production-ready. See [Hashed Storage](../../repository_storage_types.md) for more detail. - -If you previously enabled Hashed Storage and migrated all your existing -projects to Hashed Storage, disabling hashed storage will not migrate projects -to their previous project based storage path. As such, once enabled and -migrated we recommend leaving Hashed Storage enabled. - -## Updating to GitLab 10.1 - -WARNING: -Hashed storage is in **Alpha**. It is considered experimental and not -production-ready. See [Hashed Storage](../../repository_storage_types.md) for more detail. - -[Hashed storage](../../repository_storage_types.md) was introduced in -GitLab 10.0, and a [migration path](../../raketasks/storage.md) for -existing repositories was added in GitLab 10.1. - -## Updating to GitLab 10.0 - -In GitLab 10.0 and later, we require all **Geo** systems to [use SSH key lookups via -the database](../../operations/fast_ssh_key_lookup.md) to avoid having to maintain consistency of the -`authorized_keys` file for SSH access. Failing to do this will prevent users -from being able to clone via SSH. - -Note that in older versions of Geo, attachments downloaded on the **secondary** -nodes would be saved to the wrong directory. We recommend that you do the -following to clean this up. - -On the **secondary** Geo nodes, run as root: - -```shell -mv /var/opt/gitlab/gitlab-rails/working /var/opt/gitlab/gitlab-rails/working.old -mkdir /var/opt/gitlab/gitlab-rails/working -chmod 700 /var/opt/gitlab/gitlab-rails/working -chown git:git /var/opt/gitlab/gitlab-rails/working -``` - -You may delete `/var/opt/gitlab/gitlab-rails/working.old` any time. - -Once this is done, we advise restarting GitLab on the **secondary** nodes for the -new working directory to be used: - -```shell -sudo gitlab-ctl restart -``` - -## Updating from GitLab 9.3 or older - -If you started running Geo on GitLab 9.3 or older, we recommend that you -resync your **secondary** PostgreSQL databases to use replication slots. If you -started using Geo with GitLab 9.4 or 10.x, no further action should be -required because replication slots are used by default. However, if you -started with GitLab 9.3 and updated later, you should still follow the -instructions below. - -When in doubt, it doesn't hurt to do a resync. The easiest way to do this in -Omnibus is the following: - -1. Make sure you have Omnibus GitLab on the **primary** server. -1. Run `gitlab-ctl reconfigure` and `gitlab-ctl restart postgresql`. This will enable replication slots on the **primary** database. -1. Check the steps about defining `postgresql['sql_user_password']`, `gitlab_rails['db_password']`. -1. Make sure `postgresql['max_replication_slots']` matches the number of **secondary** Geo nodes locations. -1. Install GitLab on the **secondary** server. -1. Re-run the [database replication process](../setup/database.md#step-3-initiate-the-replication-process). - -## Updating to GitLab 9.0 - -> **IMPORTANT**: -With GitLab 9.0, the PostgreSQL version is updated to 9.6 and manual steps are -required to update the **secondary** nodes and keep the Streaming Replication -working. Downtime is required, so plan ahead. - -The following steps apply only if you update from a 8.17 GitLab version to -9.0+. For previous versions, update to GitLab 8.17 first before attempting to -update to 9.0+. - ---- - -Make sure to follow the steps in the exact order as they appear below and pay -extra attention in what node (either **primary** or **secondary**) you execute them! Each step -is prepended with the relevant node for better clarity: - -1. **(secondary)** Log in to **all** your **secondary** nodes and stop all services: - - ```ruby - sudo gitlab-ctl stop - ``` - -1. **(secondary)** Make a backup of the `recovery.conf` file on **all** - **secondary** nodes to preserve PostgreSQL's credentials: - - ```shell - sudo cp /var/opt/gitlab/postgresql/data/recovery.conf /var/opt/gitlab/ - ``` - -1. **(primary)** Update the **primary** node to GitLab 9.0 following the - [regular update docs](../../../update/README.md). At the end of the - update, the **primary** node will be running with PostgreSQL 9.6. - -1. **(primary)** To prevent a de-synchronization of the repository replication, - stop all services except `postgresql` as we will use it to re-initialize the - **secondary** node's database: - - ```shell - sudo gitlab-ctl stop - sudo gitlab-ctl start postgresql - ``` - -1. **(secondary)** Run the following steps on each of the **secondary** nodes: - - 1. **(secondary)** Stop all services: - - ```shell - sudo gitlab-ctl stop - ``` - - 1. **(secondary)** Prevent running database migrations: - - ```shell - sudo touch /etc/gitlab/skip-auto-migrations - ``` - - 1. **(secondary)** Move the old database to another directory: - - ```shell - sudo mv /var/opt/gitlab/postgresql{,.bak} - ``` - - 1. **(secondary)** Update to GitLab 9.0 following the [regular update docs](../../../update/README.md). - At the end of the update, the node will be running with PostgreSQL 9.6. - - 1. **(secondary)** Make sure all services are up: - - ```shell - sudo gitlab-ctl start - ``` - - 1. **(secondary)** Reconfigure GitLab: - - ```shell - sudo gitlab-ctl reconfigure - ``` - - 1. **(secondary)** Run the PostgreSQL upgrade command: - - ```shell - sudo gitlab-ctl pg-upgrade - ``` - - 1. **(secondary)** See the stored credentials for the database that you will - need to re-initialize the replication: - - ```shell - sudo grep -s primary_conninfo /var/opt/gitlab/recovery.conf - ``` - - 1. **(secondary)** Save the snippet below in a file, let's say `/tmp/replica.sh`. Modify the - embedded paths if necessary: - - ```shell - #!/bin/bash - - PORT="5432" - USER="gitlab_replicator" - echo --------------------------------------------------------------- - echo WARNING: Make sure this script is run from the secondary server - echo --------------------------------------------------------------- - echo - echo Enter the IP or FQDN of the primary PostgreSQL server - read HOST - echo Enter the password for $USER@$HOST - read -s PASSWORD - echo Enter the required sslmode - read SSLMODE - - echo Stopping PostgreSQL and all GitLab services - sudo service gitlab stop - sudo service postgresql stop - - echo Backing up postgresql.conf - sudo -u postgres mv /var/opt/gitlab/postgresql/data/postgresql.conf /var/opt/gitlab/postgresql/ - - echo Cleaning up old cluster directory - sudo -u postgres rm -rf /var/opt/gitlab/postgresql/data - - echo Starting base backup as the replicator user - echo Enter the password for $USER@$HOST - sudo -u postgres /opt/gitlab/embedded/bin/pg_basebackup -h $HOST -D /var/opt/gitlab/postgresql/data -U gitlab_replicator -v -x -P - - echo Writing recovery.conf file - sudo -u postgres bash -c "cat > /var/opt/gitlab/postgresql/data/recovery.conf <<- _EOF1_ - standby_mode = 'on' - primary_conninfo = 'host=$HOST port=$PORT user=$USER password=$PASSWORD sslmode=$SSLMODE' - _EOF1_ - " - - echo Restoring postgresql.conf - sudo -u postgres mv /var/opt/gitlab/postgresql/postgresql.conf /var/opt/gitlab/postgresql/data/ - - echo Starting PostgreSQL - sudo service postgresql start - ``` - - 1. **(secondary)** Run the recovery script using the credentials from the - previous step: - - ```shell - sudo bash /tmp/replica.sh - ``` - - 1. **(secondary)** Reconfigure GitLab: - - ```shell - sudo gitlab-ctl reconfigure - ``` - - 1. **(secondary)** Start all services: - - ```shell - sudo gitlab-ctl start - ``` - - 1. **(secondary)** Repeat the steps for the remaining **secondary** nodes. - -1. **(primary)** After all **secondary** nodes are updated, start all services in - **primary** node: - - ```shell - sudo gitlab-ctl start - ``` - -### Update tracking database on **secondary** node - -After updating a **secondary** node, you might need to run migrations on the -tracking database. The tracking database was added in GitLab 9.1, and is -required in GitLab 10.0 and later. - -1. Run database migrations on tracking database: - - ```shell - sudo gitlab-rake geo:db:migrate - ``` - -1. Repeat this step for each **secondary** node. +The issue is fixed in GitLab 12.1. Be sure to upgrade to GitLab 12.1 or later. |