diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2020-08-20 18:42:06 +0000 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2020-08-20 18:42:06 +0000 |
| commit | 6e4e1050d9dba2b7b2523fdd1768823ab85feef4 (patch) | |
| tree | 78be5963ec075d80116a932011d695dd33910b4e /doc/administration/geo | |
| parent | 1ce776de4ae122aba3f349c02c17cebeaa8ecf07 (diff) | |
| download | gitlab-ce-6e4e1050d9dba2b7b2523fdd1768823ab85feef4.tar.gz | |
Add latest changes from gitlab-org/gitlab@13-3-stable-ee
Diffstat (limited to 'doc/administration/geo')
18 files changed, 377 insertions, 410 deletions
diff --git a/doc/administration/geo/disaster_recovery/background_verification.md b/doc/administration/geo/disaster_recovery/background_verification.md index 88218d24b2f..89a93e45b1d 100644 --- a/doc/administration/geo/disaster_recovery/background_verification.md +++ b/doc/administration/geo/disaster_recovery/background_verification.md @@ -58,14 +58,14 @@ Feature.enable('geo_repository_verification') ## Repository verification -Navigate to the **{admin}** **Admin Area >** **{location-dot}** **Geo** dashboard on the **primary** node and expand +Navigate to the **Admin Area > Geo** dashboard on the **primary** node and expand the **Verification information** tab for that node to view automatic checksumming status for repositories and wikis. Successes are shown in green, pending work in gray, and failures in red.  -Navigate to the **{admin}** **Admin Area >** **{location-dot}** **Geo** dashboard on the **secondary** node and expand +Navigate to the **Admin Area > Geo** dashboard on the **secondary** node and expand the **Verification information** tab for that node to view automatic verification status for repositories and wikis. As with checksumming, successes are shown in green, pending work in gray, and failures in red. @@ -92,7 +92,7 @@ data. The default and recommended re-verification interval is 7 days, though an interval as short as 1 day can be set. Shorter intervals reduce risk but increase load and vice versa. -Navigate to the **{admin}** **Admin Area >** **{location-dot}** **Geo** dashboard on the **primary** node, and +Navigate to the **Admin Area > Geo** dashboard on the **primary** node, and click the **Edit** button for the **primary** node to customize the minimum re-verification interval: @@ -141,7 +141,7 @@ sudo gitlab-rake geo:verification:wiki:reset If the **primary** and **secondary** nodes have a checksum verification mismatch, the cause may not be apparent. To find the cause of a checksum mismatch: -1. Navigate to the **{admin}** **Admin Area >** **{overview}** **Overview > Projects** dashboard on the **primary** node, find the +1. Navigate to the **Admin Area > Overview > Projects** dashboard on the **primary** node, find the project that you want to check the checksum differences and click on the **Edit** button:  diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md index ef0e67434d0..a0cf263a762 100644 --- a/doc/administration/geo/disaster_recovery/planned_failover.md +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -110,7 +110,7 @@ The maintenance window won't end until Geo replication and verification is completely finished. To keep the window as short as possible, you should ensure these processes are close to 100% as possible during active use. -Navigate to the **{admin}** **Admin Area >** **{location-dot}** **Geo** dashboard on the **secondary** node to +Navigate to the **Admin Area > Geo** dashboard on the **secondary** node to review status. Replicated objects (shown in green) should be close to 100%, and there should be no failures (shown in red). If a large proportion of objects aren't yet replicated (shown in gray), consider giving the node more @@ -135,8 +135,8 @@ This [content was moved to another location](background_verification.md). ### Notify users of scheduled maintenance -On the **primary** node, navigate to **{admin}** **Admin Area >** **{bullhorn}** **Messages**, add a broadcast -message. You can check under **{admin}** **Admin Area >** **{location-dot}** **Geo** to estimate how long it +On the **primary** node, navigate to **Admin Area > Messages**, add a broadcast +message. You can check under **Admin Area > Geo** to estimate how long it will take to finish syncing. An example message would be: > A scheduled maintenance will take place at XX:XX UTC. We expect it to take @@ -181,7 +181,7 @@ access to the **primary** node during the maintenance window. connection. 1. Disable non-Geo periodic background jobs on the **primary** node by navigating - to **{admin}** **Admin Area >** **{monitor}** **Monitoring > Background Jobs > Cron**, pressing `Disable All`, + to **Admin Area > Monitoring > Background Jobs > Cron**, pressing `Disable All`, and then pressing `Enable` for the `geo_sidekiq_cron_config_worker` cron job. This job will re-enable several other cron jobs that are essential for planned failover to complete successfully. @@ -190,11 +190,11 @@ access to the **primary** node during the maintenance window. 1. If you are manually replicating any data not managed by Geo, trigger the final replication process now. -1. On the **primary** node, navigate to **{admin}** **Admin Area >** **{monitor}** **Monitoring > Background Jobs > Queues** +1. On the **primary** node, navigate to **Admin Area > Monitoring > Background Jobs > Queues** and wait for all queues except those with `geo` in the name to drop to 0. These queues contain work that has been submitted by your users; failing over before it is completed will cause the work to be lost. -1. On the **primary** node, navigate to **{admin}** **Admin Area >** **{location-dot}** **Geo** and wait for the +1. On the **primary** node, navigate to **Admin Area > Geo** and wait for the following conditions to be true of the **secondary** node you are failing over to: - All replication meters to each 100% replicated, 0% failures. @@ -202,7 +202,7 @@ access to the **primary** node during the maintenance window. - Database replication lag is 0ms. - The Geo log cursor is up to date (0 events behind). -1. On the **secondary** node, navigate to **{admin}** **Admin Area >** **{monitor}** **Monitoring > Background Jobs > Queues** +1. On the **secondary** node, navigate to **Admin Area > Monitoring > Background Jobs > Queues** and wait for all the `geo` queues to drop to 0 queued and 0 running jobs. 1. On the **secondary** node, use [these instructions](../../raketasks/check.md) to verify the integrity of CI artifacts, LFS objects, and uploads in file diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md index 3d08ed81700..74fa8e3b8f2 100644 --- a/doc/administration/geo/replication/configuration.md +++ b/doc/administration/geo/replication/configuration.md @@ -191,17 +191,16 @@ keys must be manually replicated to the **secondary** node. gitlab-ctl reconfigure ``` -1. Visit the **primary** node's **{admin}** **Admin Area >** **{location-dot}** **Geo** +1. Visit the **primary** node's **Admin Area > Geo** (`/admin/geo/nodes`) in your browser. 1. Click the **New node** button. -  +  1. Fill in **Name** with the `gitlab_rails['geo_node_name']` in `/etc/gitlab/gitlab.rb`. These values must always match *exactly*, character for character. 1. Fill in **URL** with the `external_url` in `/etc/gitlab/gitlab.rb`. These values must always match, but it doesn't matter if one ends with a `/` and the other doesn't. -1. **Do NOT** check the **This is a primary node** checkbox. 1. Optionally, choose which groups or storage shards should be replicated by the **secondary** node. Leave blank to replicate all. Read more in [selective synchronization](#selective-synchronization). @@ -238,7 +237,7 @@ You can login to the **secondary** node with the same credentials as used for th Using Hashed Storage significantly improves Geo replication. Project and group renames no longer require synchronization between nodes. -1. Visit the **primary** node's **{admin}** **Admin Area >** **{settings}** **Settings > Repository** +1. Visit the **primary** node's **Admin Area > Settings > Repository** (`/admin/application_settings/repository`) in your browser. 1. In the **Repository storage** section, check **Use hashed storage paths for newly created and renamed projects**. @@ -255,7 +254,7 @@ on the **secondary** node. ### Step 6. Enable Git access over HTTP/HTTPS Geo synchronizes repositories over HTTP/HTTPS, and therefore requires this clone -method to be enabled. Navigate to **{admin}** **Admin Area >** **{settings}** **Settings** +method to be enabled. Navigate to **Admin Area > Settings** (`/admin/application_settings/general`) on the **primary** node, and set `Enabled Git access protocols` to `Both SSH and HTTP(S)` or `Only HTTP(S)`. @@ -264,7 +263,7 @@ method to be enabled. Navigate to **{admin}** **Admin Area >** **{settings}** ** Your **secondary** node is now configured! You can login to the **secondary** node with the same credentials you used for the -**primary** node. Visit the **secondary** node's **{admin}** **Admin Area >** **{location-dot}** **Geo** +**primary** node. Visit the **secondary** node's **Admin Area > Geo** (`/admin/geo/nodes`) in your browser to check if it's correctly identified as a **secondary** Geo node and if Geo is enabled. diff --git a/doc/administration/geo/replication/database.md b/doc/administration/geo/replication/database.md index 02f51e79907..0bc37ce6438 100644 --- a/doc/administration/geo/replication/database.md +++ b/doc/administration/geo/replication/database.md @@ -37,8 +37,7 @@ recover. See below for more details. The following guide assumes that: - You are using Omnibus and therefore you are using PostgreSQL 11 or later - which includes the [`pg_basebackup` tool](https://www.postgresql.org/docs/11/app-pgbasebackup.html) and improved - [Foreign Data Wrapper](https://www.postgresql.org/docs/11/postgres-fdw.html) support. + which includes the [`pg_basebackup` tool](https://www.postgresql.org/docs/11/app-pgbasebackup.html). - You have a **primary** node already set up (the GitLab server you are replicating from), running Omnibus' PostgreSQL (or equivalent version), and you have a new **secondary** server set up with the same versions of the OS, @@ -167,6 +166,11 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o corresponding to the given address. See [the PostgreSQL documentation](https://www.postgresql.org/docs/11/runtime-config-connection.html) for more details. + NOTE: **Note:** + If you need to use `0.0.0.0` or `*` as the listen_address, you will also need to add + `127.0.0.1/32` to the `postgresql['md5_auth_cidr_addresses']` setting, to allow Rails to connect through + `127.0.0.1`. For more information, see [omnibus-5258](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5258). + Depending on your network configuration, the suggested addresses may not be correct. If your **primary** node and **secondary** nodes connect over a local area network, or a virtual network connecting availability zones like @@ -341,10 +345,10 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o Ensure that the contents of `~gitlab-psql/data/server.crt` on the **primary** node match the contents of `~gitlab-psql/.postgresql/root.crt` on the **secondary** node. -1. Configure PostgreSQL to enable FDW support: +1. Configure PostgreSQL: This step is similar to how we configured the **primary** instance. - We need to enable this, to enable FDW support, even if using a single node. + We need to enable this, even if using a single node. Edit `/etc/gitlab/gitlab.rb` and add the following, replacing the IP addresses with addresses appropriate to your network configuration: @@ -369,11 +373,6 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ## postgresql['sql_user_password'] = '<md5_hash_of_your_password>' gitlab_rails['db_password'] = '<your_password_here>' - - ## - ## Enable FDW support for the Geo Tracking Database (improves performance) - ## - geo_secondary['db_fdw'] = true ``` For external PostgreSQL instances, see [additional instructions](external_database.md). @@ -385,15 +384,12 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o gitlab-ctl reconfigure ``` -1. Restart PostgreSQL for the IP change to take effect and reconfigure again: +1. Restart PostgreSQL for the IP change to take effect: ```shell gitlab-ctl restart postgresql - gitlab-ctl reconfigure ``` - This last reconfigure will provision the FDW configuration and enable it. - ### Step 3. Initiate the replication process Below we provide a script that connects the database on the **secondary** node to @@ -468,48 +464,6 @@ high-availability configuration with a cluster of nodes supporting a Geo **primary** node and another cluster of nodes supporting a Geo **secondary** node. For more information, see [High Availability with Omnibus GitLab](../../postgresql/replication_and_failover.md). -For a Geo **secondary** node to work properly with PgBouncer in front of the database, -it will need a separate read-only user to make [PostgreSQL FDW queries](https://www.postgresql.org/docs/11/postgres-fdw.html) -work: - -1. On the **primary** Geo database, enter the PostgreSQL on the console as an - admin user. If you are using an Omnibus-managed database, log onto the **primary** - node that is running the PostgreSQL database (the default Omnibus database name is `gitlabhq_production`): - - ```shell - sudo \ - -u gitlab-psql /opt/gitlab/embedded/bin/psql \ - -h /var/opt/gitlab/postgresql gitlabhq_production - ``` - -1. Then create the read-only user: - - ```sql - -- NOTE: Use the password defined earlier - CREATE USER gitlab_geo_fdw WITH password 'mypassword'; - GRANT CONNECT ON DATABASE gitlabhq_production to gitlab_geo_fdw; - GRANT USAGE ON SCHEMA public TO gitlab_geo_fdw; - GRANT SELECT ON ALL TABLES IN SCHEMA public TO gitlab_geo_fdw; - GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO gitlab_geo_fdw; - - -- Tables created by "gitlab" should be made read-only for "gitlab_geo_fdw" - -- automatically. - ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON TABLES TO gitlab_geo_fdw; - ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON SEQUENCES TO gitlab_geo_fdw; - ``` - -1. On the **secondary** nodes, change `/etc/gitlab/gitlab.rb`: - - ```ruby - geo_postgresql['fdw_external_user'] = 'gitlab_geo_fdw' - ``` - -1. Save the file and reconfigure GitLab for the changes to be applied: - - ```shell - gitlab-ctl reconfigure - ``` - ## Troubleshooting Read the [troubleshooting document](troubleshooting.md). diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index 5636ff79189..b8d01e80371 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Geo data types support +# Geo data types support **(PREMIUM ONLY)** A Geo data type is a specific class of data that is required by one or more GitLab features to store relevant information. @@ -126,6 +126,33 @@ these epics/issues: - [Unreplicated Data Types](https://gitlab.com/groups/gitlab-org/-/epics/893) - [Verify all replicated data](https://gitlab.com/groups/gitlab-org/-/epics/1430) +### Replicated data types behind a feature flag + +The replication for some data types is behind a corresponding feature flag: + +> - They're deployed behind a feature flag, enabled by default. +> - They're enabled on GitLab.com. +> - They can't be enabled or disabled per-project. +> - They are recommended for production use. +> - For GitLab self-managed instances, GitLab administrators can opt to [disable them](#enable-or-disable-replication-for-some-data-types-core-only). **(CORE ONLY)** + +#### Enable or disable replication (for some data types) **(CORE ONLY)** + +Replication for some data types are released behind feature flags that are **enabled by default**. +[GitLab administrators with access to the GitLab Rails console](../../feature_flags.md) can opt to disable it for your instance. You can find feature flag names of each of those data types in the notes column of the table below. + +To disable, such as for package file replication: + +```ruby +Feature.disable(:geo_package_file_replication) +``` + +To enable, such as for package file replication: + +```ruby +Feature.enable(:geo_package_file_replication) +``` + DANGER: **Danger:** Features not on this list, or with **No** in the **Replicated** column, are not replicated on the **secondary** node. Failing over without manually @@ -151,12 +178,12 @@ successfully, you must replicate their data using some other means. | [Elasticsearch integration](../../../integration/elasticsearch.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/1186) | No | | | [GitLab Pages](../../pages/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/589) | No | | | [Container Registry](../../packages/container_registry.md) | **Yes** (12.3) | No | | -| [NPM Registry](../../../user/packages/npm_registry/index.md) | **Yes** (13.2) | No | | -| [Maven Repository](../../../user/packages/maven_repository/index.md) | **Yes** (13.2) | No | | -| [Conan Repository](../../../user/packages/conan_repository/index.md) | **Yes** (13.2) | No | | -| [NuGet Repository](../../../user/packages/nuget_repository/index.md) | **Yes** (13.2) | No | | -| [PyPi Repository](../../../user/packages/pypi_repository/index.md) | **Yes** (13.2) | No | | -| [Composer Repository](../../../user/packages/composer_repository/index.md) | **Yes** (13.2) | No | | +| [NPM Registry](../../../user/packages/npm_registry/index.md) | **Yes** (13.2) | No | Behind feature flag `geo_package_file_replication`, enabled by default | | +| [Maven Repository](../../../user/packages/maven_repository/index.md) | **Yes** (13.2) | No | Behind feature flag `geo_package_file_replication`, enabled by default | | +| [Conan Repository](../../../user/packages/conan_repository/index.md) | **Yes** (13.2) | No | Behind feature flag `geo_package_file_replication`, enabled by default | | +| [NuGet Repository](../../../user/packages/nuget_repository/index.md) | **Yes** (13.2) | No | Behind feature flag `geo_package_file_replication`, enabled by default | | +| [PyPi Repository](../../../user/packages/pypi_repository/index.md) | **Yes** (13.2) | No | Behind feature flag `geo_package_file_replication`, enabled by default | | +| [Composer Repository](../../../user/packages/composer_repository/index.md) | **Yes** (13.2) | No | Behind feature flag `geo_package_file_replication`, enabled by default | | | [External merge request diffs](../../merge_request_diffs.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/33817) | No | | | [Terraform State](../../terraform_state.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/3112)(*3*) | No | | | [Vulnerability Export](../../../user/application_security/security_dashboard/#export-vulnerabilities) | [No](https://gitlab.com/groups/gitlab-org/-/epics/3111)(*3*) | No | | | diff --git a/doc/administration/geo/replication/disable_geo.md b/doc/administration/geo/replication/disable_geo.md index f53b4c1b796..aed8e5fc3bc 100644 --- a/doc/administration/geo/replication/disable_geo.md +++ b/doc/administration/geo/replication/disable_geo.md @@ -33,7 +33,7 @@ in order to do that. ## Remove the primary node from the UI -1. Go to **{admin}** **Admin Area >** **{location-dot}** **Geo** (`/admin/geo/nodes`). +1. Go to **Admin Area > Geo** (`/admin/geo/nodes`). 1. Click the **Remove** button for the **primary** node. 1. Confirm by clicking **Remove** when the prompt appears. diff --git a/doc/administration/geo/replication/docker_registry.md b/doc/administration/geo/replication/docker_registry.md index c34732cba67..5403edc69da 100644 --- a/doc/administration/geo/replication/docker_registry.md +++ b/doc/administration/geo/replication/docker_registry.md @@ -122,7 +122,7 @@ generate a short-lived JWT that is pull-only-capable to access the ### Verify replication -To verify Container Registry replication is working, go to **{admin}** **Admin Area >** **{location-dot}** **Geo** +To verify Container Registry replication is working, go to **Admin Area > Geo** (`/admin/geo/nodes`) on the **secondary** node. The initial replication, or "backfill", will probably still be in progress. You can monitor the synchronization process on each Geo node from the **primary** node's **Geo Nodes** dashboard in your browser. diff --git a/doc/administration/geo/replication/external_database.md b/doc/administration/geo/replication/external_database.md index e85a82fa414..d860a3dd368 100644 --- a/doc/administration/geo/replication/external_database.md +++ b/doc/administration/geo/replication/external_database.md @@ -183,9 +183,6 @@ to grant additional roles to your tracking database user (by default, this is - Amazon RDS requires the [`rds_superuser`](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.html#Appendix.PostgreSQL.CommonDBATasks.Roles) role. - Azure Database for PostgreSQL requires the [`azure_pg_admin`](https://docs.microsoft.com/en-us/azure/postgresql/howto-create-users#how-to-create-additional-admin-users-in-azure-database-for-postgresql) role. -The tracking database requires an [FDW](https://www.postgresql.org/docs/11/postgres-fdw.html) -connection with the **secondary** replica database for improved performance. - If you have an external database ready to be used as the tracking database, follow the instructions below to use it: @@ -224,7 +221,6 @@ the tracking database on port 5432. geo_secondary['db_host'] = '<tracking_database_host>' geo_secondary['db_port'] = <tracking_database_port> # change to the correct port - geo_secondary['db_fdw'] = true # enable FDW geo_postgresql['enable'] = false # don't use internal managed instance ``` @@ -236,48 +232,3 @@ the tracking database on port 5432. gitlab-rake geo:db:create gitlab-rake geo:db:migrate ``` - -1. Configure the [PostgreSQL FDW](https://www.postgresql.org/docs/11/postgres-fdw.html) - connection and credentials: - - Save the script below in a file, ex. `/tmp/geo_fdw.sh` and modify the connection - parameters to match your environment. Execute it to set up the FDW connection. - - ```shell - #!/bin/bash - - # Secondary Database connection params: - DB_HOST="<public_ip_or_vpc_private_ip>" - DB_NAME="gitlabhq_production" - DB_USER="gitlab" - DB_PASS="<your_password_here>" - DB_PORT="5432" - - # Tracking Database connection params: - GEO_DB_HOST="<public_ip_or_vpc_private_ip>" - GEO_DB_NAME="gitlabhq_geo_production" - GEO_DB_USER="gitlab_geo" - GEO_DB_PORT="5432" - - query_exec () { - gitlab-psql -h $GEO_DB_HOST -U $GEO_DB_USER -d $GEO_DB_NAME -p $GEO_DB_PORT -c "${1}" - } - - query_exec "CREATE EXTENSION postgres_fdw;" - query_exec "CREATE SERVER gitlab_secondary FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '${DB_HOST}', dbname '${DB_NAME}', port '${DB_PORT}');" - query_exec "CREATE USER MAPPING FOR ${GEO_DB_USER} SERVER gitlab_secondary OPTIONS (user '${DB_USER}', password '${DB_PASS}');" - query_exec "CREATE SCHEMA gitlab_secondary;" - query_exec "GRANT USAGE ON FOREIGN SERVER gitlab_secondary TO ${GEO_DB_USER};" - ``` - - NOTE: **Note:** - The script template above uses `gitlab-psql` as it's intended to be executed from the Geo machine, - but you can change it to `psql` and run it from any machine that has access to the database. We also recommend using - `psql` for AWS RDS. - -1. Save the file and [restart GitLab](../../restart_gitlab.md#omnibus-gitlab-restart) -1. Populate the FDW tables: - - ```shell - gitlab-rake geo:db:refresh_foreign_tables - ``` diff --git a/doc/administration/geo/replication/geo_validation_tests.md b/doc/administration/geo/replication/geo_validation_tests.md index 0255e5c9883..323f6f367b1 100644 --- a/doc/administration/geo/replication/geo_validation_tests.md +++ b/doc/administration/geo/replication/geo_validation_tests.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Geo validation tests +# Geo validation tests **(PREMIUM ONLY)** The Geo team performs manual testing and validation on common deployment configurations to ensure that Geo works when upgrading between minor GitLab versions and major PostgreSQL database versions. @@ -27,6 +27,13 @@ The following are GitLab upgrade validation tests we performed. - [Investigate why `reconfigure` and `hup` cause downtime on multi-node Geo deployments](https://gitlab.com/gitlab-org/gitlab/-/issues/228898) - [Geo multi-node deployment upgrade: investigate order when upgrading non-deploy nodes](https://gitlab.com/gitlab-org/gitlab/-/issues/228954) +[Switch from repmgr to Patroni on a Geo primary site](https://gitlab.com/gitlab-org/gitlab/-/issues/224652): + +- Description: Tested switching from repmgr to Patroni on a multi-node Geo primary site. Used [the orchestrator tool](https://gitlab.com/gitlab-org/gitlab-orchestrator) to deploy a Geo installation with 3 database nodes managed by repmgr. With this approach, we were also able to address a related issue for [verifying a Geo installation with Patroni and PostgreSQL 11](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5113). +- Outcome: Partial success. We enabled Patroni on the primary site and set up database replication on the secondary site. However, we found that Patroni would delete the secondary site's replication slot whenever Patroni was restarted. Another issue is that when Patroni elects a new leader in the cluster, the secondary site will fail to automatically follow the new leader. Until these issues are resolved, we cannot officially support and recommend Patroni for Geo installations. +- Follow up issues/actions: + - [Investigate permanent replication slot for Patroni with Geo single node secondary](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5528) + ### June 2020 [Upgrade Geo multi-node installation](https://gitlab.com/gitlab-org/gitlab/-/issues/223284): @@ -107,6 +114,19 @@ The following are GitLab upgrade validation tests we performed. The following are PostgreSQL upgrade validation tests we performed. +### August 2020 + +[Verify Geo installation with PostgreSQL 12](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5453): + +- Description: Prior to PostgreSQL 12 becoming available as an opt-in version in GitLab 13.3, + we tested fresh installations of GitLab 13.3 with PostgreSQL 12 enabled and Geo installed. +- Outcome: Setting up a Geo secondary required manual intervention because the `recovery.conf` file + is no longer supported in PostgreSQL 12. We do not recommend deploying Geo with PostgreSQL 12 until + the appropriate changes have been made to Omnibus and verified. +- Follow up issues: + - [Update `replicate-geo-database` to support PostgreSQL 12](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5575) + - [Remove PostgreSQL 12 check in `replicate-geo-database` for 14.0](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5576) + ### April 2020 [PostgreSQL 11 upgrade procedure for Geo installations](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4975): diff --git a/doc/administration/geo/replication/img/adding_a_secondary_node.png b/doc/administration/geo/replication/img/adding_a_secondary_node.png Binary files differdeleted file mode 100644 index e33b690da18..00000000000 --- a/doc/administration/geo/replication/img/adding_a_secondary_node.png +++ /dev/null diff --git a/doc/administration/geo/replication/img/adding_a_secondary_node_v13_3.png b/doc/administration/geo/replication/img/adding_a_secondary_node_v13_3.png Binary files differnew file mode 100644 index 00000000000..e43ace99666 --- /dev/null +++ b/doc/administration/geo/replication/img/adding_a_secondary_node_v13_3.png diff --git a/doc/administration/geo/replication/index.md b/doc/administration/geo/replication/index.md index 51c831abaf3..f1cc9f0df8b 100644 --- a/doc/administration/geo/replication/index.md +++ b/doc/administration/geo/replication/index.md @@ -117,7 +117,7 @@ The following are required to run Geo: The following operating systems are known to ship with a current version of OpenSSH: - [CentOS](https://www.centos.org) 7.4+ - [Ubuntu](https://ubuntu.com) 16.04+ -- PostgreSQL 11+ with [FDW](https://www.postgresql.org/docs/11/postgres-fdw.html) support and [Streaming Replication](https://wiki.postgresql.org/wiki/Streaming_Replication) +- PostgreSQL 11+ with [Streaming Replication](https://wiki.postgresql.org/wiki/Streaming_Replication) - Git 2.9+ - All nodes must run the same GitLab version. @@ -166,7 +166,6 @@ The tracking database instance is used as metadata to control what needs to be u - Fetch changes from a repository that has recently been updated. Because the replicated database instance is read-only, we need this additional database instance for each **secondary** node. -The tracking database requires the `postgres_fdw` extension. ### Geo Log Cursor diff --git a/doc/administration/geo/replication/multiple_servers.md b/doc/administration/geo/replication/multiple_servers.md index d8f04e07fb0..31d1ea2cd2b 100644 --- a/doc/administration/geo/replication/multiple_servers.md +++ b/doc/administration/geo/replication/multiple_servers.md @@ -137,7 +137,7 @@ documentation: synchronized from the **primary** node. NOTE: **Note:** -[NFS](../../high_availability/nfs.md) can be used in place of Gitaly but is not +[NFS](../../nfs.md) can be used in place of Gitaly but is not recommended. ### Step 2: Configure the main read-only replica PostgreSQL database on the **secondary** node @@ -196,9 +196,27 @@ the **primary** database. Use the following as a guide. geo_postgresql['enable'] = false ## - ## Disable `geo_logcursor` service so Rails doesn't get configured here + ## Disable all other services that aren't needed. Note that we had to enable + ## geo_secondary_role to cause some configuration changes to postgresql, but + ## the role enables single-node services by default. ## + alertmanager['enable'] = false + consul['enable'] = false geo_logcursor['enable'] = false + gitaly['enable'] = false + gitlab_exporter['enable'] = false + gitlab_workhorse['enable'] = false + nginx['enable'] = false + node_exporter['enable'] = false + pgbouncer_exporter['enable'] = false + prometheus['enable'] = false + redis['enable'] = false + redis_exporter['enable'] = false + repmgr['enable'] = false + sidekiq['enable'] = false + sidekiq_cluster['enable'] = false + puma['enable'] = false + unicorn['enable'] = false ``` After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect. @@ -242,10 +260,8 @@ Configure the tracking database. geo_postgresql['sql_user_password'] = '<tracking_database_password_md5_hash>' ## - ## Configure FDW connection to the replica database + ## Configure PostgreSQL connection to the replica database ## - geo_secondary['db_fdw'] = true - geo_postgresql['fdw_external_password'] = '<replica_database_password_plaintext>' geo_postgresql['md5_auth_cidr_addresses'] = ['<replica_database_ip>/32'] gitlab_rails['db_host'] = '<replica_database_ip>' @@ -253,11 +269,11 @@ Configure the tracking database. gitlab_rails['auto_migrate'] = false ## - ## Disable all other services that aren't needed, since we don't have a role - ## that does this. + ## Ensure unnecessary services are disabled ## alertmanager['enable'] = false consul['enable'] = false + geo_logcursor['enable'] = false gitaly['enable'] = false gitlab_exporter['enable'] = false gitlab_workhorse['enable'] = false @@ -270,7 +286,9 @@ Configure the tracking database. redis_exporter['enable'] = false repmgr['enable'] = false sidekiq['enable'] = false + sidekiq_cluster['enable'] = false puma['enable'] = false + unicorn['enable'] = false ``` After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect. @@ -284,9 +302,9 @@ In the architecture overview, there are two machines running the GitLab application services. These services are enabled selectively in the configuration. -Configure the application servers following -[Configuring GitLab for multiple nodes](../../high_availability/gitlab.md), then make the -following modifications: +Configure the GitLab Rails application servers following the relevant steps +outlined in the [reference architectures](../../reference_architectures/index.md), +then make the following modifications: 1. Edit `/etc/gitlab/gitlab.rb` on each application server in the **secondary** cluster, and add the following: @@ -373,7 +391,7 @@ application servers. In this topology, a load balancer is required at each geographic location to route traffic to the application servers. -See [Load Balancer for GitLab with multiple nodes](../../high_availability/load_balancer.md) for +See [Load Balancer for GitLab with multiple nodes](../../load_balancer.md) for more information. ### Step 6: Configure the backend application servers on the **secondary** node @@ -417,6 +435,7 @@ application servers above, with some changes to run only the `sidekiq` service: redis_exporter['enable'] = false repmgr['enable'] = false puma['enable'] = false + unicorn['enable'] = false ## ## The unique identifier for the Geo node. diff --git a/doc/administration/geo/replication/object_storage.md b/doc/administration/geo/replication/object_storage.md index 3cc0ade414e..159e2524198 100644 --- a/doc/administration/geo/replication/object_storage.md +++ b/doc/administration/geo/replication/object_storage.md @@ -33,7 +33,7 @@ whether they are stored on the local filesystem or in object storage. To enable GitLab replication, you must: -1. Go to **{admin}** **Admin Area >** **{location-dot}** **Geo**. +1. Go to **Admin Area > Geo**. 1. Press **Edit** on the **secondary** node. 1. Enable the **Allow this secondary node to replicate content on Object Storage** checkbox. diff --git a/doc/administration/geo/replication/remove_geo_node.md b/doc/administration/geo/replication/remove_geo_node.md index 539132776b3..2b01231241c 100644 --- a/doc/administration/geo/replication/remove_geo_node.md +++ b/doc/administration/geo/replication/remove_geo_node.md @@ -9,7 +9,7 @@ type: howto **Secondary** nodes can be removed from the Geo cluster using the Geo admin page of the **primary** node. To remove a **secondary** node: -1. Navigate to **{admin}** **Admin Area >** **{location-dot}** **Geo** (`/admin/geo/nodes`). +1. Navigate to **Admin Area > Geo** (`/admin/geo/nodes`). 1. Click the **Remove** button for the **secondary** node you want to remove. 1. Confirm by clicking **Remove** when the prompt appears. diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index c2f8aa35d2d..b8172322c10 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -14,7 +14,6 @@ Here is a list of steps you should take to attempt to fix problem: - Perform [basic troubleshooting](#basic-troubleshooting). - Fix any [replication errors](#fixing-replication-errors). -- Fix any [Foreign Data Wrapper](#fixing-foreign-data-wrapper-errors) errors. - Fix any [common](#fixing-common-errors) errors. ## Basic troubleshooting @@ -26,7 +25,7 @@ Before attempting more advanced troubleshooting: ### Check the health of the **secondary** node -Visit the **primary** node's **{admin}** **Admin Area >** **{location-dot}** **Geo** (`/admin/geo/nodes`) in +Visit the **primary** node's **Admin Area > Geo** (`/admin/geo/nodes`) in your browser. We perform the following health checks on each **secondary** node to help identify if something is wrong: @@ -44,6 +43,8 @@ For information on how to resolve common errors reported from the UI, see If the UI is not working, or you are unable to log in, you can run the Geo health check manually to get this information as well as a few more details. +#### Health check Rake task + This Rake task can be run on an app node in the **primary** or **secondary** Geo nodes: @@ -62,8 +63,6 @@ This machine's Geo node name matches a database record ... yes, found a secondar GitLab Geo secondary database is correctly configured ... yes Database replication enabled? ... yes Database replication working? ... yes -GitLab Geo tracking database is configured to use Foreign Data Wrapper? ... yes -GitLab Geo tracking database Foreign Data Wrapper schema is up-to-date? ... yes GitLab Geo HTTP(S) connectivity ... * Can connect to the primary node ... yes HTTP/HTTPS repository cloning is enabled ... yes @@ -77,6 +76,8 @@ All projects are in hashed storage? ... yes Checking Geo ... Finished ``` +#### Sync status Rake task + Current sync information can be found manually by running this Rake task on any **secondary** app node: @@ -128,8 +129,7 @@ Geo finds the current machine's Geo node name in `/etc/gitlab/gitlab.rb` by: - Using the `gitlab_rails['geo_node_name']` setting. - If that is not defined, using the `external_url` setting. -This name is used to look up the node with the same **Name** in -**{admin}** **Admin Area >** **{location-dot}** **Geo**. +This name is used to look up the node with the same **Name** in **Admin Area > Geo**. To check if the current machine has a node name that matches a node in the database, run the check task: @@ -205,7 +205,7 @@ sudo gitlab-rake gitlab:geo:check - Verify the correct password is set for `gitlab_rails['db_password']` that was used when creating the hash in `postgresql['sql_user_password']` by running `gitlab-ctl pg-password-md5 gitlab` and entering the password. -1. Check returns not a secondary node +1. Check returns `not a secondary node` ```plaintext Checking Geo ... @@ -252,12 +252,12 @@ sudo gitlab-rake gitlab:geo:check When performing a PostgreSQL major version (9 > 10) update this is expected. Follow: - [initiate-the-replication-process](database.md#step-3-initiate-the-replication-process) - - [Geo database has an outdated FDW remote schema](troubleshooting.md#geo-database-has-an-outdated-fdw-remote-schema-error) ## Fixing replication errors The following sections outline troubleshooting steps for fixing replication -errors. +errors (indicated by `Database replication working? ... no` in the +[`geo:check` output](#health-check-rake-task). ### Message: `ERROR: replication slots can only be used if max_replication_slots > 0`? @@ -390,13 +390,13 @@ to respect the CIDR format (i.e. `1.2.3.4/32`). GitLab places a timeout on all repository clones, including project imports and Geo synchronization operations. If a fresh `git clone` of a repository -on the **primary** takes more than a few minutes, you may be affected by this. +on the **primary** takes more than the default three hours, you may be affected by this. To increase the timeout, add the following line to `/etc/gitlab/gitlab.rb` on the **secondary** node: ```ruby -gitlab_rails['gitlab_shell_git_timeout'] = 10800 +gitlab_rails['gitlab_shell_git_timeout'] = 14400 ``` Then reconfigure GitLab: @@ -405,7 +405,7 @@ Then reconfigure GitLab: sudo gitlab-ctl reconfigure ``` -This will increase the timeout to three hours (10800 seconds). Choose a time +This will increase the timeout to four hours (14400 seconds). Choose a time long enough to accommodate a full clone of your largest repositories. ### New LFS objects are never replicated @@ -504,11 +504,63 @@ to start again from scratch, there are a few steps that can help you: gitlab-ctl start ``` -1. Refresh Foreign Data Wrapper tables +## Fixing errors during a PostgreSQL upgrade or downgrade - ```shell - gitlab-rake geo:db:refresh_foreign_tables - ``` +### Message: `ERROR: psql: FATAL: role "gitlab-consul" does not exist` + +When +[upgrading PostgreSQL on a Geo instance](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance), you might encounter the +following error: + +```plaintext +$ sudo gitlab-ctl pg-upgrade --target-version=11 +Checking for an omnibus managed postgresql: OK +Checking if postgresql['version'] is set: OK +Checking if we already upgraded: NOT OK +Checking for a newer version of PostgreSQL to install +Upgrading PostgreSQL to 11.7 +Checking if PostgreSQL bin files are symlinked to the expected location: OK +Waiting 30 seconds to ensure tasks complete before PostgreSQL upgrade. +See https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server for details +If you do not want to upgrade the PostgreSQL server at this time, enter Ctrl-C and see the documentation for details + +Please hit Ctrl-C now if you want to cancel the operation. +..............................Detected an HA cluster. +Error running command: /opt/gitlab/embedded/bin/psql -qt -d gitlab_repmgr -h /var/opt/gitlab/postgresql -p 5432 -c "SELECT name FROM repmgr_gitlab_cluster.repl_nodes WHERE type='master' AND active != 'f'" -U gitlab-consul +ERROR: psql: FATAL: role "gitlab-consul" does not exist +Traceback (most recent call last): + 10: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `<main>' + 9: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load' + 8: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in `<top (required)>' + 7: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in `run' + 6: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in `block in add_command_under_category' + 5: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:171:in `block in load_file' + 4: from /opt/gitlab/embedded/service/omnibus-ctl-ee/lib/repmgr.rb:248:in `is_master?' + 3: from /opt/gitlab/embedded/service/omnibus-ctl-ee/lib/repmgr.rb:100:in `execute_psql' + 2: from /opt/gitlab/embedded/service/omnibus-ctl-ee/lib/repmgr.rb:113:in `cmd' + 1: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/mixlib-shellout-3.0.9/lib/mixlib/shellout.rb:287:in `error!' +/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/mixlib-shellout-3.0.9/lib/mixlib/shellout.rb:300:in `invalid!': Expected process to exit with [0], but received '2' (Mixlib::ShellOut::ShellCommandFailed) +---- Begin output of /opt/gitlab/embedded/bin/psql -qt -d gitlab_repmgr -h /var/opt/gitlab/postgresql -p 5432 -c "SELECT name FROM repmgr_gitlab_cluster.repl_nodes WHERE type='master' AND active != 'f'" -U gitlab-consul ---- +STDOUT: +STDERR: psql: FATAL: role "gitlab-consul" does not exist +---- End output of /opt/gitlab/embedded/bin/psql -qt -d gitlab_repmgr -h /var/opt/gitlab/postgresql -p 5432 -c "SELECT name FROM repmgr_gitlab_cluster.repl_nodes WHERE type='master' AND active != 'f'" -U gitlab-consul ---- +Ran /opt/gitlab/embedded/bin/psql -qt -d gitlab_repmgr -h /var/opt/gitlab/postgresql -p 5432 -c "SELECT name FROM repmgr_gitlab_cluster.repl_nodes WHERE type='master' AND active != 'f'" -U gitlab-consul returned 2 +``` + +If you are upgrading the PostgreSQL read-replica of a Geo secondary node, and +you are not using `consul` or `repmgr`, you may need to disable `consul` and/or +`repmgr` services in `gitlab.rb`: + +```ruby +consul['enable'] = false +repmgr['enable'] = false +``` + +Then reconfigure GitLab: + +```shell +sudo gitlab-ctl reconfigure +``` ## Fixing errors during a failover or when promoting a secondary to a primary node @@ -551,6 +603,35 @@ or `gitlab-ctl promote-to-primary-node`, either: bug](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/22021) was fixed. +If the above does not work, another possible reason is that you have paused replication +from the original primary node before attempting to promote this node. + +To double check this, you can do the following: + +- Get the current secondary node's ID using: + + ```shell + sudo gitlab-rails runner 'puts GeoNode.current_node.id' + ``` + +- Double check that the node is active through the database by running the following + on the secondary node, replacing `ID_FROM_ABOVE`: + + ```shell + sudo gitlab-rails dbconsole + + SELECT enabled FROM geo_nodes WHERE id = ID_FROM_ABOVE; + ``` + +- If the above returned `f` it means that the replication was paused. + You can re-enable it through an `UPDATE` statement in the database: + + ```shell + sudo gitlab-rails dbconsole + + UPDATE geo_nodes SET enabled = 't' WHERE id = ID_FROM_ABOVE; + ``` + ### Message: ``NoMethodError: undefined method `secondary?' for nil:NilClass`` When [promoting a **secondary** node](../disaster_recovery/index.md#step-3-promoting-a-secondary-node), @@ -593,231 +674,6 @@ sudo /opt/gitlab/embedded/bin/gitlab-pg-ctl promote GitLab 12.9 and later are [unaffected by this error](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5147). -## Fixing Foreign Data Wrapper errors - -This section documents ways to fix potential Foreign Data Wrapper errors. - -### "Foreign Data Wrapper (FDW) is not configured" error - -When setting up Geo, you might see this warning in the `gitlab-rake -gitlab:geo:check` output: - -```plaintext -GitLab Geo tracking database Foreign Data Wrapper schema is up-to-date? ... foreign data wrapper is not configured -``` - -There are a few key points to remember: - -1. The FDW settings are configured on the Geo **tracking** database. -1. The configured foreign server enables a login to the Geo - **secondary**, read-only database. - -By default, the Geo secondary and tracking database are running on the -same host on different ports. That is, 5432 and 5431 respectively. - -#### Checking configuration - -NOTE: **Note:** -The following steps are for Omnibus installs only. Using Geo with source-based installs was **deprecated** in GitLab 11.5. - -To check the configuration: - -1. SSH into an app node in the **secondary**: - - ```shell - sudo -i - ``` - - Note: An app node is any machine running at least one of the following services: - - - `puma` - - `unicorn` - - `sidekiq` - - `geo-logcursor` - -1. Enter the database console: - - If the tracking database is running on the same node: - - ```shell - gitlab-geo-psql - ``` - - Or, if the tracking database is running on a different node, you must specify - the user and host when entering the database console: - - ```shell - gitlab-geo-psql -U gitlab_geo -h <IP of tracking database> - ``` - - You will be prompted for the password of the `gitlab_geo` user. You can find - it in plaintext in `/etc/gitlab/gitlab.rb` at: - - ```ruby - geo_secondary['db_password'] = '<geo_tracking_db_password>' - ``` - - This password is normally set on the tracking database during - [Step 3: Configure the tracking database on the secondary node](multiple_servers.md#step-3-configure-the-tracking-database-on-the-secondary-node), - and it is set on the app nodes during - [Step 4: Configure the frontend application servers on the secondary node](multiple_servers.md#step-4-configure-the-frontend-application-servers-on-the-secondary-node). - -1. Check whether any tables are present with the following statement: - - ```sql - SELECT * from information_schema.foreign_tables; - ``` - - If everything is working, you should see something like this: - - ```plaintext - gitlabhq_geo_production=# SELECT * from information_schema.foreign_tables; - foreign_table_catalog | foreign_table_schema | foreign_table_name | foreign_server_catalog | foreign_server_name - -------------------------+----------------------+-------------------------------------------------+-------------------------+--------------------- - gitlabhq_geo_production | gitlab_secondary | abuse_reports | gitlabhq_geo_production | gitlab_secondary - gitlabhq_geo_production | gitlab_secondary | appearances | gitlabhq_geo_production | gitlab_secondary - gitlabhq_geo_production | gitlab_secondary | application_setting_terms | gitlabhq_geo_production | gitlab_secondary - gitlabhq_geo_production | gitlab_secondary | application_settings | gitlabhq_geo_production | gitlab_secondary - <snip> - ``` - - However, if the query returns with `0 rows`, then continue onto the next steps. - -1. Check that the foreign server mapping is correct via `\des+`. The - results should look something like this: - - ```plaintext - gitlabhq_geo_production=# \des+ - List of foreign servers - -[ RECORD 1 ]--------+------------------------------------------------------------ - Name | gitlab_secondary - Owner | gitlab-psql - Foreign-data wrapper | postgres_fdw - Access privileges | "gitlab-psql"=U/"gitlab-psql" + - | gitlab_geo=U/"gitlab-psql" - Type | - Version | - FDW Options | (host '0.0.0.0', port '5432', dbname 'gitlabhq_production') - Description | - ``` - - NOTE: **Note:** - Pay particular attention to the host and port under - FDW options. That configuration should point to the Geo secondary - database. - - If you need to experiment with changing the host or password, the - following queries demonstrate how: - - ```sql - ALTER SERVER gitlab_secondary OPTIONS (SET host '<my_new_host>'); - ALTER SERVER gitlab_secondary OPTIONS (SET port 5432); - ``` - - If you change the host and/or port, you will also have to adjust the - following settings in `/etc/gitlab/gitlab.rb` and run `gitlab-ctl - reconfigure`: - - - `gitlab_rails['db_host']` - - `gitlab_rails['db_port']` - -1. Check that the user mapping is configured properly via `\deu+`: - - ```plaintext - gitlabhq_geo_production=# \deu+ - List of user mappings - Server | User name | FDW Options - ------------------+------------+-------------------------------------------------------------------------------- - gitlab_secondary | gitlab_geo | ("user" 'gitlab', password 'YOUR-PASSWORD-HERE') - (1 row) - ``` - - Make sure the password is correct. You can test that logins work by running `psql`: - - ```shell - # Connect to the tracking database as the `gitlab_geo` user - sudo \ - -u git /opt/gitlab/embedded/bin/psql \ - -h /var/opt/gitlab/geo-postgresql \ - -p 5431 \ - -U gitlab_geo \ - -W \ - -d gitlabhq_geo_production - ``` - - If you need to correct the password, the following query shows how: - - ```sql - ALTER USER MAPPING FOR gitlab_geo SERVER gitlab_secondary OPTIONS (SET password '<my_new_password>'); - ``` - - If you change the user or password, you will also have to adjust the - following settings in `/etc/gitlab/gitlab.rb` and run `gitlab-ctl - reconfigure`: - - - `gitlab_rails['db_username']` - - `gitlab_rails['db_password']` - - If you are using [PgBouncer in front of the secondary - database](database.md#pgbouncer-support-optional), be sure to update - the following settings: - - - `geo_postgresql['fdw_external_user']` - - `geo_postgresql['fdw_external_password']` - -#### Manual reload of FDW schema - -If you're still unable to get FDW working, you may want to try a manual -reload of the FDW schema. To manually reload the FDW schema: - -1. On the node running the Geo tracking database, enter the PostgreSQL console via - the `gitlab_geo` user: - - ```shell - sudo \ - -u git /opt/gitlab/embedded/bin/psql \ - -h /var/opt/gitlab/geo-postgresql \ - -p 5431 \ - -U gitlab_geo \ - -W \ - -d gitlabhq_geo_production - ``` - - Be sure to adjust the port and hostname for your configuration. You - may be asked to enter a password. - -1. Reload the schema via: - - ```sql - DROP SCHEMA IF EXISTS gitlab_secondary CASCADE; - CREATE SCHEMA gitlab_secondary; - GRANT USAGE ON FOREIGN SERVER gitlab_secondary TO gitlab_geo; - IMPORT FOREIGN SCHEMA public FROM SERVER gitlab_secondary INTO gitlab_secondary; - ``` - -1. Test that queries work: - - ```sql - SELECT * from information_schema.foreign_tables; - SELECT * FROM gitlab_secondary.projects limit 1; - ``` - -### "Geo database has an outdated FDW remote schema" error - -GitLab can error with a `Geo database has an outdated FDW remote schema` message. - -For example: - -```plaintext -Geo database has an outdated FDW remote schema. It contains 229 of 236 expected tables. Please refer to Geo Troubleshooting. -``` - -To resolve this, run the following command on the **secondary**: - -```shell -sudo gitlab-rake geo:db:refresh_foreign_tables -``` - ## Expired artifacts If you notice for some reason there are more artifacts on the Geo @@ -835,7 +691,7 @@ If you are able to log in to the **primary** node, but you receive this error when attempting to log into a **secondary**, you should check that the Geo node's URL matches its external URL. -1. On the primary, visit **{admin}** **Admin Area >** **{location-dot}** **Geo**. +1. On the primary, visit **Admin Area > Geo**. 1. Find the affected **secondary** and click **Edit**. 1. Ensure the **URL** field matches the value found in `/etc/gitlab/gitlab.rb` in `external_url "https://gitlab.example.com"` on the frontend server(s) of @@ -896,13 +752,6 @@ If you are using Omnibus GitLab installation, something might have failed during - Run `sudo gitlab-ctl reconfigure`. - Manually trigger the database migration by running: `sudo gitlab-rake geo:db:migrate` as root on the **secondary** node. -### Geo database is not configured to use Foreign Data Wrapper - -This error means the Geo Tracking Database doesn't have the FDW server and credentials -configured. - -See ["Foreign Data Wrapper (FDW) is not configured" error?](#foreign-data-wrapper-fdw-is-not-configured-error). - ### GitLab indicates that more than 100% of repositories were synced This can be caused by orphaned records in the project registry. You can clear them diff --git a/doc/administration/geo/replication/tuning.md b/doc/administration/geo/replication/tuning.md index 63a8f81eecb..fe0b189863c 100644 --- a/doc/administration/geo/replication/tuning.md +++ b/doc/administration/geo/replication/tuning.md @@ -7,18 +7,25 @@ type: howto # Tuning Geo **(PREMIUM ONLY)** -## Changing the sync capacity values +## Changing the sync/verification capacity values -In the Geo admin page at **{admin}** **Admin Area >** **{location-dot}** **Geo** (`/admin/geo/nodes`), +In the Geo admin page at **Admin Area > Geo** (`/admin/geo/nodes`), there are several variables that can be tuned to improve performance of Geo: -- Repository sync capacity. -- File sync capacity. +- Repository sync capacity +- File sync capacity +- Container repositories sync capacity +- Verification capacity -Increasing these values will increase the number of jobs that are scheduled. +Increasing capacity values will increase the number of jobs that are scheduled. However, this may not lead to more downloads in parallel unless the number of available Sidekiq threads is also increased. For example, if repository sync capacity is increased from 25 to 50, you may also want to increase the number of Sidekiq threads from 25 to 50. See the [Sidekiq concurrency documentation](../../operations/extra_sidekiq_processes.md#number-of-threads) for more details. + +## Repository re-verification + +See +[Automatic background verification](../disaster_recovery/background_verification.md). diff --git a/doc/administration/geo/replication/version_specific_updates.md b/doc/administration/geo/replication/version_specific_updates.md index 777de715a8c..900d09bdd34 100644 --- a/doc/administration/geo/replication/version_specific_updates.md +++ b/doc/administration/geo/replication/version_specific_updates.md @@ -11,6 +11,35 @@ Check this document if it includes instructions for the version you are updating These steps go together with the [general steps](updating_the_geo_nodes.md#general-update-steps) for updating Geo nodes. +## Updating to GitLab 13.3 + +In GitLab 13.3, Geo removed the PostgreSQL [Foreign Data Wrapper](https://www.postgresql.org/docs/11/postgres-fdw.html) dependency for the tracking database. + +The FDW server, user, and the extension will be removed during the upgrade process on each **secondary** node. The GitLab settings related to the FDW in the `/etc/gitlab/gitlab.rb` have been deprecated and can be safely removed. + +There are some scenarios like using an external PostgreSQL instance for the tracking database where the FDW settings must be removed manually. Enter the PostgreSQL console of that instance and remove them: + +```shell +DROP SERVER gitlab_secondary CASCADE; +DROP EXTENSION IF EXISTS postgres_fdw; +``` + +## Updating to GitLab 13.0 + +Upgrading to GitLab 13.0 requires GitLab 12.10 to already be using PostgreSQL +version 11. Please see +[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) +for the recommended procedure. + +## Updating to GitLab 12.10 + +GitLab 12.10 does not attempt to automatically update the embedded PostgreSQL +server when using Geo, because the PostgreSQL upgrade requires downtime on +secondaries while reinitializing streaming replication. It must be upgraded +manually. Please see +[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) +for the recommended procedure. + ## Updating to GitLab 12.9 CAUTION: **Warning:** @@ -18,6 +47,32 @@ GitLab 12.9.0 through GitLab 12.9.3 are affected by [a bug that stops repository verification](https://gitlab.com/gitlab-org/gitlab/-/issues/213523). The issue is fixed in GitLab 12.9.4. Please upgrade to GitLab 12.9.4 or later. +By default, GitLab 12.9 will attempt to automatically update the embedded +PostgreSQL server to 10.12 from 9.6, which requires downtime on secondaries +while reinitializing streaming replication. Please see +[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) +for the recommended procedure. + +This can be temporarily disabled by running the following before updating: + +```shell +sudo touch /etc/gitlab/disable-postgresql-upgrade +``` + +## Updating to GitLab 12.8 + +By default, GitLab 12.8 will attempt to automatically update the embedded +PostgreSQL server to 10.12 from 9.6, which requires downtime on secondaries +while reinitializing streaming replication. Please see +[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) +for the recommended procedure. + +This can be temporarily disabled by running the following before updating: + +```shell +sudo touch /etc/gitlab/disable-postgresql-upgrade +``` + ## Updating to GitLab 12.7 DANGER: **Danger:** @@ -28,8 +83,92 @@ bug](https://gitlab.com/gitlab-org/gitlab/-/issues/199672) that causes Geo fix](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/24021) was shipped in 12.7.5. +By default, GitLab 12.7 will attempt to automatically update the embedded +PostgreSQL server to 10.9 from 9.6, which requires downtime on secondaries while +reinitializing streaming replication. Please see +[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) +for the recommended procedure. + +This can be temporarily disabled by running the following before updating: + +```shell +sudo touch /etc/gitlab/disable-postgresql-upgrade +``` + +## Updating to GitLab 12.6 + +By default, GitLab 12.6 will attempt to automatically update the embedded +PostgreSQL server to 10.9 from 9.6, which requires downtime on secondaries while +reinitializing streaming replication. Please see +[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) +for the recommended procedure. + +This can be temporarily disabled by running the following before updating: + +```shell +sudo touch /etc/gitlab/disable-postgresql-upgrade +``` + +## Updating to GitLab 12.5 + +By default, GitLab 12.5 will attempt to automatically update the embedded +PostgreSQL server to 10.9 from 9.6, which requires downtime on secondaries while +reinitializing streaming replication. Please see +[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) +for the recommended procedure. + +This can be temporarily disabled by running the following before updating: + +```shell +sudo touch /etc/gitlab/disable-postgresql-upgrade +``` + +## Updating to GitLab 12.4 + +By default, GitLab 12.4 will attempt to automatically update the embedded +PostgreSQL server to 10.9 from 9.6, which requires downtime on secondaries while +reinitializing streaming replication. Please see +[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) +for the recommended procedure. + +This can be temporarily disabled by running the following before updating: + +```shell +sudo touch /etc/gitlab/disable-postgresql-upgrade +``` + +## Updating to GitLab 12.3 + +DANGER: **Danger:** +If the existing PostgreSQL server version is 9.6.x, it is recommended to +upgrade to GitLab 12.4 or newer. By default, GitLab 12.3 will attempt to +automatically update the embedded PostgreSQL server to 10.9 from 9.6. In +certain circumstances, it will fail. Please see +[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) +for more information. + +Additionally, if the PostgreSQL upgrade does not fail, a successful upgrade +requires downtime on secondaries while reinitializing streaming replication. +Please see +[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) +for the recommended procedure. + ## Updating to GitLab 12.2 +DANGER: **Danger:** +If the existing PostgreSQL server version is 9.6.x, it is recommended to +upgrade to GitLab 12.4 or newer. By default, GitLab 12.2 will attempt to +automatically update the embedded PostgreSQL server to 10.9 from 9.6. In +certain circumstances, it will fail. Please see +[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) +for more information. + +Additionally, if the PostgreSQL upgrade does not fail, a successful upgrade +requires downtime on secondaries while reinitializing streaming replication. +Please see +[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) +for the recommended procedure. + GitLab 12.2 includes the following minor PostgreSQL updates: - To version `9.6.14` if you run PostgreSQL 9.6. @@ -48,17 +187,20 @@ The restart avoids a version mismatch when PostgreSQL tries to load the FDW exte ## Updating to GitLab 12.1 -By default, GitLab 12.1 will attempt to automatically update the -embedded PostgreSQL server to 10.7 from 9.6. Please see -[the omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) +DANGER: **Danger:** +If the existing PostgreSQL server version is 9.6.x, it is recommended to +upgrade to GitLab 12.4 or newer. By default, GitLab 12.1 will attempt to +automatically update the embedded PostgreSQL server to 10.9 from 9.6. In +certain circumstances, it will fail. Please see +[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) +for more information. + +Additionally, if the PostgreSQL upgrade does not fail, a successful upgrade +requires downtime on secondaries while reinitializing streaming replication. +Please see +[the Omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) for the recommended procedure. -This can be temporarily disabled by running the following before updating: - -```shell -sudo touch /etc/gitlab/disable-postgresql-upgrade -``` - ## Updating to GitLab 12.0 CAUTION: **Warning:** @@ -214,7 +356,7 @@ Replicating over SSH has been deprecated, and support for this option will be removed in a future release. To switch to HTTP/HTTPS replication, log into the **primary** node as an admin and visit -**{admin}** **Admin Area >** **{location-dot}** **Geo** (`/admin/geo/nodes`). For each **secondary** node listed, +**Admin Area > Geo** (`/admin/geo/nodes`). For each **secondary** node listed, press the "Edit" button, change the "Repository cloning" setting from "SSH (deprecated)" to "HTTP/HTTPS", and press "Save changes". This should take effect immediately. |
