diff options
Diffstat (limited to 'doc/administration/postgresql')
| -rw-r--r-- | doc/administration/postgresql/database_load_balancing.md | 5 | ||||
| -rw-r--r-- | doc/administration/postgresql/external.md | 22 | ||||
| -rw-r--r-- | doc/administration/postgresql/multiple_databases.md | 91 | ||||
| -rw-r--r-- | doc/administration/postgresql/pgbouncer.md | 21 | ||||
| -rw-r--r-- | doc/administration/postgresql/replication_and_failover.md | 2 |
5 files changed, 113 insertions, 28 deletions
diff --git a/doc/administration/postgresql/database_load_balancing.md b/doc/administration/postgresql/database_load_balancing.md index 15129770888..d5cf93a135a 100644 --- a/doc/administration/postgresql/database_load_balancing.md +++ b/doc/administration/postgresql/database_load_balancing.md @@ -235,3 +235,8 @@ operation retries up to 3 times using an exponential back-off. When using load balancing, you should be able to safely restart a database server without it immediately leading to errors being presented to the users. + +### Development guide + +For detailed development guide on database load balancing, +see [the development documentation](../../development/database/load_balancing.md). diff --git a/doc/administration/postgresql/external.md b/doc/administration/postgresql/external.md index 8605ee94255..8f664f9809e 100644 --- a/doc/administration/postgresql/external.md +++ b/doc/administration/postgresql/external.md @@ -17,7 +17,7 @@ If you use a cloud-managed service, or provide your own PostgreSQL instance: 1. Set up PostgreSQL according to the [database requirements document](../../install/requirements.md#database). -1. Set up a `gitlab` user with a password of your choice, create the `gitlabhq_production` database, and make the user an owner of the database. You can see an example of this setup in the [installation from source documentation](../../install/installation.md#6-database). +1. Set up a `gitlab` user with a password of your choice, create the `gitlabhq_production` database, and make the user an owner of the database. You can see an example of this setup in the [installation from source documentation](../../install/installation.md#7-database). 1. If you are using a cloud-managed service, you may need to grant additional roles to your `gitlab` user: - Amazon RDS requires the [`rds_superuser`](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.html#Appendix.PostgreSQL.CommonDBATasks.Roles) role. @@ -30,18 +30,18 @@ If you use a cloud-managed service, or provide your own PostgreSQL instance: 1. Configure the GitLab application servers with the appropriate connection details for your external PostgreSQL service in your `/etc/gitlab/gitlab.rb` file: - ```ruby - # Disable the bundled Omnibus provided PostgreSQL - postgresql['enable'] = false + ```ruby + # Disable the bundled Omnibus provided PostgreSQL + postgresql['enable'] = false - # PostgreSQL connection details - gitlab_rails['db_adapter'] = 'postgresql' - gitlab_rails['db_encoding'] = 'unicode' - gitlab_rails['db_host'] = '10.1.0.5' # IP/hostname of database server - gitlab_rails['db_password'] = 'DB password' - ``` + # PostgreSQL connection details + gitlab_rails['db_adapter'] = 'postgresql' + gitlab_rails['db_encoding'] = 'unicode' + gitlab_rails['db_host'] = '10.1.0.5' # IP/hostname of database server + gitlab_rails['db_password'] = 'DB password' + ``` - For more information on GitLab multi-node setups, refer to the [reference architectures](../reference_architectures/index.md). + For more information on GitLab multi-node setups, refer to the [reference architectures](../reference_architectures/index.md). 1. Reconfigure for the changes to take effect: diff --git a/doc/administration/postgresql/multiple_databases.md b/doc/administration/postgresql/multiple_databases.md index 836736fb73f..857fd4fc9c5 100644 --- a/doc/administration/postgresql/multiple_databases.md +++ b/doc/administration/postgresql/multiple_databases.md @@ -1,6 +1,6 @@ --- stage: Data Stores -group: Pods +group: Tenant Scale info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- @@ -15,23 +15,83 @@ By default, GitLab uses a single application database, referred to as the `main` To scale GitLab, you can configure GitLab to use multiple application databases. -Due to [known issues](#known-issues), configuring GitLab with multiple databases is in [**Alpha**](../../policy/alpha-beta-support.md#alpha-features). +Due to [known issues](#known-issues), configuring GitLab with multiple databases is an [Experiment](../../policy/alpha-beta-support.md#experiment). + +After you have set up multiple databases, GitLab uses a second application database for +[CI/CD features](../../ci/index.md), referred to as the `ci` database. + +All tables have exactly the same structure in both the `main`, and `ci` +databases. Some examples: + +- When multiple databases are configured, the `ci_pipelines` table exists in + both the `main` and `ci` databases, but GitLab reads and writes only to the + `ci_pipelines` table in the `ci` database. +- Similarly, the `projects` table exists in + both the `main` and `ci` databases, but GitLab reads and writes only to the + `projects` table in the `main` database. +- For some tables (such as `loose_foreign_keys_deleted_records`) GitLab reads and writes to both the `main` and `ci` databases. See the + [development documentation](../../development/database/multiple_databases.md#gitlab-schema) ## Known issues -- Migrating data from the `main` database to the `ci` database is not supported or documented yet. - Once data is migrated to the `ci` database, you cannot migrate it back. -## Set up multiple databases +## Migrate existing installations -Use the following content to set up multiple databases with a new GitLab installation. +To migrate existing data from the `main` database to the `ci` database, you can +copy the database across. -There is no documentation for existing GitLab installations yet. +### Existing source installation -After you have set up multiple databases, GitLab uses a second application database for -[CI/CD features](../../ci/index.md), referred to as the `ci` database. For -example, GitLab reads and writes to the `ci_pipelines` table in the `ci` -database. +1. Stop GitLab, except for PostgreSQL: + + ```shell + sudo service gitlab stop + sudo service postgresql start + ``` + +1. Dump the `main` database: + + ```shell + sudo -u git pg_dump -f gitlabhq_production.sql gitlabhq_production + ``` + +1. Create the `ci` database, and copy the data from the previous dump: + + ```shell + sudo -u postgres psql -d template1 -c "CREATE DATABASE gitlabhq_production_ci OWNER git;" + sudo -u git psql -f gitlabhq_production.sql gitlabhq_production_ci + ``` + +1. Configure GitLab to [use multiple databases](#set-up-multiple-databases). + +### Existing Omnibus installation + +1. Stop GitLab, except for PostgreSQL: + + ```shell + sudo gitlab-ctl stop + sudo gitlab-ctl start postgresql + ``` + +1. Dump the `main` database: + + ```shell + sudo -u gitlab-psql /opt/gitlab/embedded/bin/pg_dump -h /var/opt/gitlab/postgresql -f gitlabhq_production.sql gitlabhq_production + ``` + +1. Create the `ci` database, and copy the data from the previous dump: + + ```shell + sudo -u gitlab-psql /opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/postgresql -d template1 -c "CREATE DATABASE gitlabhq_production_ci OWNER gitlab;" + sudo -u gitlab-psql /opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/postgresql -f gitlabhq_production.sql gitlabhq_production_ci + ``` + +1. Configure GitLab to [use multiple databases](#set-up-multiple-databases). + +## Set up multiple databases + +To configure GitLab to use multiple application databases, follow the instructions below for your installation type. WARNING: You must stop GitLab before setting up multiple databases. This prevents @@ -40,6 +100,9 @@ the other way around. ### Installations from source +1. For existing installations, + [migrate the data](#migrate-existing-installations) first. + 1. [Back up GitLab](../../raketasks/backup_restore.md) in case of unforeseen issues. @@ -70,7 +133,7 @@ the other way around. 1. Update the service files to set the `GITLAB_ALLOW_SEPARATE_CI_DATABASE` environment variable to `true`. -1. Create the `gitlabhq_production_ci` database: +1. For new installations only. Create the `gitlabhq_production_ci` database: ```shell sudo -u postgres psql -d template1 -c "CREATE DATABASE gitlabhq_production OWNER git;" @@ -91,6 +154,9 @@ the other way around. ### Omnibus GitLab installations +1. For existing installations, + [migrate the data](#migrate-existing-installations) first. + 1. [Back up GitLab](../../raketasks/backup_restore.md) in case of unforeseen issues. @@ -116,7 +182,8 @@ the other way around. sudo gitlab-ctl reconfigure ``` -1. Optional. Reconfiguring GitLab should create the `gitlabhq_production_ci`. If it did not, manually create the `gitlabhq_production_ci`: +1. Optional, for new installations only. Reconfiguring GitLab should create the + `gitlabhq_production_ci` database if it does not exist. If the database is not created automatically, create it manually: ```shell sudo gitlab-ctl start postgresql diff --git a/doc/administration/postgresql/pgbouncer.md b/doc/administration/postgresql/pgbouncer.md index 5dd0aad7162..59aac9141a4 100644 --- a/doc/administration/postgresql/pgbouncer.md +++ b/doc/administration/postgresql/pgbouncer.md @@ -174,10 +174,12 @@ ote_pid | tls ## Procedure for bypassing PgBouncer +### Omnibus installations + Some database changes have to be done directly, and not through PgBouncer. -Read more about the affected tasks: [database restores](../../raketasks/backup_restore.md#back-up-and-restore-for-installations-using-pgbouncer) -and [GitLab upgrades](../../update/zero_downtime.md#postgresql). +The main affected tasks are [database restores](../../raketasks/backup_restore.md#back-up-and-restore-for-installations-using-pgbouncer) +and [GitLab upgrades with database migrations](../../update/zero_downtime.md#postgresql). 1. To find the primary node, run the following on a database node: @@ -195,7 +197,7 @@ and [GitLab upgrades](../../update/zero_downtime.md#postgresql). sudo gitlab-ctl reconfigure ``` -Once you've performed the tasks or procedure, switch back to using PgBouncer: +After you've performed the tasks or procedure, switch back to using PgBouncer: 1. Change back `/etc/gitlab/gitlab.rb` to point to PgBouncer. 1. Run reconfigure: @@ -204,6 +206,19 @@ Once you've performed the tasks or procedure, switch back to using PgBouncer: sudo gitlab-ctl reconfigure ``` +### Helm chart installations + +High-availability deployments also need to bypass PgBouncer for the same reasons as Omnibus-based ones. +For this type of installation: + +- Database backup and restore tasks are performed by the toolbox container. +- Migration tasks are performed by the migrations container. + +You should override the PostgreSQL port on each subchart, so these tasks can execute and connect to PostgreSQL directly: + +- [Toolbox](https://gitlab.com/gitlab-org/charts/gitlab/-/blob/master/charts/gitlab/charts/toolbox/values.yaml#L40) +- [Migrations](https://gitlab.com/gitlab-org/charts/gitlab/-/blob/master/charts/gitlab/charts/migrations/values.yaml#L46) + ## Fine tuning PgBouncer's default settings suit the majority of installations. diff --git a/doc/administration/postgresql/replication_and_failover.md b/doc/administration/postgresql/replication_and_failover.md index 46890b0b2ca..8ac3ac40a75 100644 --- a/doc/administration/postgresql/replication_and_failover.md +++ b/doc/administration/postgresql/replication_and_failover.md @@ -1255,8 +1255,6 @@ To do the switch on **all** PgBouncer nodes: ``` 1. Run `gitlab-ctl reconfigure`. -1. You must also run `rm /var/opt/gitlab/consul/config.d/watcher_postgresql.json`. - This is a [known issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/7293). #### Clean up |